Abstract
For most people, recalling information about familiar items in a visual scene is an effortless task, but it is one that depends on coordinated interactions of multiple, distributed neural components. We leveraged the high spatiotemporal resolution of direct intracranial recordings to better delineate the network dynamics underpinning visual scene recognition. We present a dataset of recordings from a large cohort of humans while they identified images of famous landmarks (50 individuals, 52 recording sessions, 6,775 electrodes, 6,541 trials). This dataset contains local field potential recordings derived from subdural and penetrating electrodes covering broad areas of cortex across both hemispheres. We provide this pre-processed data with behavioural metrics (correct/incorrect, response times) and electrode localisation in a population-normalised cortical surface space. This rich dataset will allow further investigation into the spatiotemporal progression of multiple neural processes underlying visual processing, scene recognition and cued memory recall.
Measurement(s) | electrophysiology data |
Technology Type(s) | Electrocorticography |
Factor Type(s) | visual scenes |
Sample Characteristic - Organism | Homo sapiens |
Machine-accessible metadata file describing the reported data: https://doi.org/10.6084/m9.figshare.16727317
Similar content being viewed by others
Background & Summary
Analysing and identifying previously encountered scenes and landmarks requires an interplay of activity between visual and memory regions of the brain. Category selective scene processing pathways exist across higher visual cortex1,2,3 and specialised, distributed memory networks exist for recall of scene and location specific information4,5,6,7,8. Neurobiological studies of higher-level image and scene processing in humans are usually derived using fMRI which, given its relatively low temporal resolution, provides a representation of the locations of neural processes but not their dynamics. Visual recognition and identification is accomplished by a combination of feedforward and feedback interactions between multiple substrates. While the traditional view of visual processing follows a hierarchical feedforward structure from low level visual features to high level category representations9, recurrent inputs from high level regions modulate activity in early visual regions enhancing sensitivity to certain higher order features10,11,12. Therefore, information about the temporal progression of feature sensitivity is required to disentangle these properties within visual processing networks and refine existing models of scene processing.
Given the special circumstances that it takes to obtain them, intracranial recordings remain a relatively rare but unique window into local and interactional cortical dynamics. We used a famous landmark identification task to probe multiple aspects of scene processing pathways; including visual processing of the scene stimuli, memory recall of the specific item and articulation of the response. These data were collected in a large population of patients (52 datasets) with a large number of electrodes implanted across both hemispheres (n = 6,775) allowing broad coverage across multiple cortical regions. Some of these data have previously been used to elaborate category selective cortical organisation in the ventral occipitotemporal cortex13,14,15, and memory recall in medial parietal cortex and medial temporal lobe15. Given the very rich nature of this dataset there remains multiple aspects of vision processing and memory recall that remain unaddressed and can be probed using these data.
Methods
Participants
50 patients (24 male, 18–51 years, 11 left-handed) participated in the experiments after giving written informed consent. Participants consented for anonymised data to be distributed for research use. All participants were semi-chronically implanted with intracranial electrodes for seizure localisation of pharmaco-resistant epilepsy. Two patients participated in the experiments twice (TA632/TA632C, TS060B/TS060C), during two separate electrode implantations (52 implantations total), with a separation of approximately two years between implants. Participants with significant additional neurological history (e.g. previous resections, MR imaging abnormalities such as malformations or hypoplasia) were excluded. All experimental procedures were reviewed and approved by the Committee for the Protection of Human Subjects (CPHS) of the University of Texas Health Science Center at Houston as Protocol Number HSC-MS-06-0385.
Electrode implantation and data recording
Data were acquired from either subdural grid electrodes (SDEs; 10 patients) or stereotactically placed depth electrodes (sEEGs; 42 patients)15. SDEs were subdural platinum-iridium electrodes embedded in a silicone elastomer sheet (PMT Corporation; top-hat design; 3 mm diameter cortical contact), and were surgically implanted via a craniotomy16,17,18. sEEGs were implanted using a Robotic Surgical Assistant (ROSA; Medtech, Montpellier, France)19,20. Each sEEG probe (PMT corporation, Chanhassen, Minnesota) was 0.8 mm in diameter and had 8–16 electrode contacts. Each contact was a platinum-iridium cylinder, 2.0 mm in length and separated from the adjacent contact by 1.5–2.43 mm. Each patient had 12–20 such probes implanted. Following surgical implantation, electrodes were localised by co-registration of pre-operative anatomical 3 T MRI and post-operative CT scans in AFNI21. Electrode positions were projected onto a cortical surface model generated in FreeSurfer22, and displayed on the cortical surface model for visualisation17. Population-level cortical maps were generated using a surface-based co-registration to address sparse-sampling, as well as intra- and inter-subject data variability, including the topological variability arising from each subject’s own complex cortical geometry. In brief, individual cortical surface models were inflated and warped to match the folding patterns on a spherical template mesh, derived from a population atlas (Colin 27), and then resampled to a new standardised mesh providing one-to-one vertex correspondence between vertex indices and anatomical locations across subjects23,24.
Intracranial data were collected during research experiments starting on the first day after electrode implantation for sEEGs and two days after implantation for SDEs. Data were digitised at 2 kHz using the NeuroPort recording system (Blackrock Microsystems, Salt Lake City, Utah; acquisition filter 0.3–500 Hz), imported into Matlab, initially referenced to the white matter channel used as a reference for the clinical acquisition system and visually inspected for line noise, artifacts, and epileptic activity. Electrodes with excessive line noise or localised to sites of seizure onset were excluded. Trials contaminated by inter-ictal epileptic spikes are marked.
Stimuli and experimental design
Stimuli were presented using Python v2.7 at a size of 500 × 500 pixels on a 2,880 × 1,800, 15.4” LCD screen positioned at eye-level, 2–3′ from the patient (~7.5° visual angle). Each stimulus was displayed for 2,000 ms with an inter-stimulus interval of 6,000 ms. Participants were presented with colour photos of famous landmarks (scenes) and asked to verbally recall their location (Fig. 1a)13,14,15. Stimuli were presented in one recording session, containing presentation of 140–160 images, consisting of a mix of coherent images and their spatially scrambled versions in a pseudorandom order. In trials with scrambled scenes the patients were asked to respond with “scrambled”.
Audio recordings
Continuous audio recordings were carried out during all experiments with an omnidirectional microphone (30–20,000 Hz response, 73 dB SNR, Audio Technica U841A) placed within 2 feet of the patient, and adjacent to the presentation laptop. These recordings were analysed offline to manually isolate articulatory onsets and assess patient responses. As voice recordings are biometric data protected under HIPAA, this dataset is released only with the manually picked articulation onset times and a binary correct/incorrect variable to denote the accuracy of the answer given. Incorrect responses included either an absence of speech, stating they did not recognise the object or incorrectly identifying the object.
Signal analysis
Example analyses presented here were performed by first bandpass filtering raw data of each electrode into broadband gamma activity (BGA; 70–150 Hz). A frequency domain bandpass Hilbert transform (paired sigmoid flanks with half-width 1.5 Hz) was applied and the analytic amplitude was smoothed (Savitzky - Golay finite impulse response, 3rd order, frame length 301 ms). BGA is presented here as percentage change from baseline level, defined as the period −500 to −100 ms before each stimulus presentation.
Data Records
The dataset25 was released in the Neurodata Without Borders (NWB)26,27 Version 2.2.5 format and is available online from the Data Archive for the BRAIN Initiative (DABI). The dataset is organised according to iEEG-BIDS28. Data from each recording session, including their associated pre-processing metadata and electrode localisations were saved in a single HDF5 file (.nwb). Standard NWB file structure was used which comprises a hierarchical structure29. Top level data groups include subject information, electrode information, time series data and trial information (Table 1).
Subject information (general_subject)
This data group contains basic patient demographics including age, sex and implant type.
Cortical surfaces (general_subject.corticalsurfaces)
In order to allow cross-individual consolidation of data, all electrodes have been mapped onto a standardised cortical surface. Each electrode is assigned to a vertex of the standard population surface. We have provided both inflated (std_inflated) and pial (std_pial) versions of this standard surface. By knowing the hemisphere and vertex assignment of an electrode then the 3D location can be determined for either representation. The cortical surface map contains the vertices and faces of each hemisphere. For each vertex we provide a Human Connectome Project parcellation map assignment30 (HCP) and the local curvature value of the pial surface (curv).
Electrode information (general_extracellular_ephys_electrodes)
For each electrode we provide a 3D location in standard inflated space (x,y,z). The assigned vertex of each electrode (vertex) and its hemispheric assignment (hemi) are also included. Based on this vertex assignment each electrode is given an HCP label (HCP, location). Electrodes with excessive line noise or localised to sites of seizure onset were marked (good = 0) and the time series data excluded.
Time series data
Each participant has continuous time series data, recorded from each of the electrodes for the duration of the experimental session. Time series data is provided referenced to the original clinical white matter reference electrode (either electrode 5 or 6) and grounded to electrode 7. We have included a virtual channel representing the common average of the good channels (labelled CAR) which can be subtracted from the raw data to allow easy re-referencing to a common average reference scheme.
Trial information (intervals_trials)
The continuous data can be epoched using event markers (start_time). These markers index either stimulus onset times (articulation = 0) or articulation onset times (articulation = 1). Trials contaminated by inter-ictal epileptic spikes or disrupted with behavioural distractions are marked (good = 0). Trials with successful identification of the presented landmark were marked (correct = 1). Stimuli presented were either coherent images (scrambled = 0) or scrambled images (scrambled = 1) and the exact stimulus presented is noted (stimulus). JPEG versions of the presented stimuli are available for download alongside the datasets.
Technical Validation
Behavioural analysis
Mean identification accuracy (±s.d.) was 56 ± 17% for scenes and 97 ± 5% for scrambled images. All patients included in this dataset correctly identified at least 20 of the scene stimuli and responded to at least 80% of the scrambled trials, providing enough trials for both successful and unsuccessful identification analysis. Articulation latencies for this cohort were 1,495 ± 330 ms for scenes and 1,491 ± 341 ms for scrambled images (Fig. 2b).
Spatiotemporal mapping of cortical activations
We have previously used some patients from this dataset to create a 4D movie of whole brain cortical activation, contrasting correctly answered coherent scenes vs. scrambled images15 as a broad map of cortical activity during the task. Here, we present plots of BGA for correctly answered, incorrectly answered and scrambled trials across six exemplar ROIs, using HCP parcellations in early visual cortex, medial parietal cortex, and medial temporal lobe (Fig. 3). These ROIs display diverse response properties in their responses to coherent vs. scrambled images and correctly vs. incorrectly identified stimuli.
Usage Notes
Electrodes are named according to clinical convention within our site, based on the broad location covered (SDEs) or the approximate distal electrode target (sEEGs). This nomenclature scheme is for clinical purposes and may not always reflect actual electrode location. sEEG electrodes are numbered sequentially along the probe with the most distal (deepest in the brain) contact being electrode 1.
Articulation times have only been included for correctly answered trials. Pairing stimulus onset and articulation onset markers should be performed by finding matching stimuli, as each stimulus was shown only once per patient. Baselining for articulation aligned data should be performed using the pre-stimulus baseline period rather than the pre-articulation period.
Electrical time series traces are provided as output from the NeuroPort recording system with no additional filtering applied beyond the acquisition filter (0.3–500 Hz). To convert the traces into μV the values should be divided by 4, after conversion from integers to floating-point values.
Users who wish to publish results of analysis performed on this data must include the following in their acknowledgements: “Data used to perform this analysis were collected with support from the National Institutes of Health under award numbers DC014589 and NS098981 and were accessed from the Data Archive for the BRAIN Initiative with support from the National Institutes of Health under Award Number R24MH114796.”
Access to the data is contingent on creating an account with DABI and agreeing to the terms of the data usage agreement (Supplementary Information).
Code availability
Example scripts are provided with the dataset25. They contain code for reading and plotting the neural and behavioural data. Code examples are provided in Matlab and Python Jupyter Notebooks.
References
Aguirre, G. K., Zarahn, E. & D’Esposito, M. An area within human ventral cortex sensitive to ‘building’ stimuli: Evidence and implications. Neuron 21, 373–383 (1998).
Haxby, J. V. et al. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science (80-.). 293, 87–96 (2013).
Epstein, R. A. & Baker, C. I. Scene Perception in the Human Brain. Annu. Rev. Vis. Sci. 5, 373–397 (2019).
Ranganath, C. & Ritchey, M. Two cortical systems for memory-guided behaviour. Nat. Rev. Neurosci. 13, 713–726 (2012).
Silson, E. H., Steel, A., Kidder, A., Gilmore, A. W. & Baker, C. I. Distinct subdivisions of human medial parietal cortex support recollection of people and places. Elife 8 (2019).
Morton, N. W., Zippi, E. L., Noh, S. & Preston, A. R. Semantic knowledge of famous people and places is represented in hippocampus and distinct cortical networks. J. Neurosci. https://doi.org/10.1523/jneurosci.2034-19.2021 (2021).
Martin, C. B., McLean, D. A., O’Neil, E. B. & Kohler, S. Distinct Familiarity-Based Response Patterns for Faces and Buildings in Perirhinal and Parahippocampal Cortex. J. Neurosci. 33, 10915–10923 (2013).
Baldassano, C., Esteva, A., Fei-Fei, L. & Beck, D. M. Two distinct Scene-Processing networks connecting vision and memory. eNeuro 3 (2016).
Groen, I. I. A., Silson, E. H. & Baker, C. I. Contributions of low- and high-level properties to neural processing of visual scenes in the human brain. Philos. Trans. R. Soc. B Biol. Sci. 372 (2017).
Tang, H. et al. Spatiotemporal Dynamics Underlying Object Completion in Human Ventral Visual Cortex. Neuron 83, 736–748 (2014).
Woolnough, O. et al. Spatiotemporal dynamics of orthographic and lexical processing in the ventral visual pathway. Nat. Hum. Behav. 5, 389–398 (2021).
Heilbron, M., Richter, D., Ekman, M., Hagoort, P. & de Lange, F. P. Word contexts enhance the neural representation of individual letters in early visual cortex. Nat. Commun. 11 (2020).
Kadipasaoglu, C. M. et al. Development of grouped icEEG for the study of cognitive processing. Front. Psychol. 6 (2015).
Kadipasaoglu, C. M., Conner, C. R., Whaley, M. L., Baboyan, V. G. & Tandon, N. Category-selectivity in human visual cortex follows cortical topology: A grouped icEEG study. PLoS One 11 (2016).
Woolnough, O. et al. Category Selectivity for Face and Scene Recognition in Human Medial Parietal Cortex. Curr. Biol. 30, 2707–2715 (2020).
Tandon, N. Mapping of human language. in Clinical Brain Mapping (eds. Yoshor, D. & Mizrahi, E.) 203–218 (McGraw Hill Education, 2012).
Pieters, T. A., Conner, C. R. & Tandon, N. Recursive grid partitioning on a cortical surface model: an optimized technique for the localization of implanted subdural electrodes. J. Neurosurg. 118, 1086–1097 (2013).
Tong, B. A., Esquenazi, Y., Johnson, J., Zhu, P. & Tandon, N. The Brain is Not Flat: Conformal Electrode Arrays Diminish Complications of Subdural Electrode Implantation, A Series of 117 Cases. World Neurosurg. 144, e734–e742 (2020).
Tandon, N. et al. Analysis of Morbidity and Outcomes Associated With Use of Subdural Grids vs Stereoelectroencephalography in Patients With Intractable Epilepsy. JAMA Neurol. 76, 672–681 (2019).
Rollo, P. S., Rollo, M. J., Zhu, P., Woolnough, O. & Tandon, N. Oblique trajectory angles in robotic stereo-electroencephalography. J. Neurosurg. 135, 245–254 (2020).
Cox, R. W. AFNI: Software for Analysis and Visualization of Functional Magnetic Resonance Neuroimages. Comput. Biomed. Res. 29, 162–173 (1996).
Dale, A. M., Fischl, B. & Sereno, M. I. Cortical Surface-Based Analysis: I. Segmentation and Surface Reconstruction. Neuroimage 9, 179–194 (1999).
Kadipasaoglu, C. M. et al. Surface-based mixed effects multilevel analysis of grouped human electrocorticography. Neuroimage 101, 215–224 (2014).
Saad, Z. S. & Reynolds, R. C. Suma. Neuroimage 62, 768–773 (2012).
Tandon, N. & Crone, N. E. A unified cognitive network model of language. DABI https://doi.org/10.18120/vn47-d626 (2021).
Rübel, O. et al. NWB:N 2.0: An Accessible Data Standard for Neurophysiology. bioRxiv https://doi.org/10.1101/523035 (2019).
Rübel, O., Tritt, A., Ly, R., Dichter, B. K. & Ghosh, S. The Neurodata Without Borders ecosystem for neurophysiological data science. bioRxiv (2021).
Holdgraf, C. et al. iEEG-BIDS, extending the Brain Imaging Data Structure specification to human intracranial electrophysiology. Sci. Data 6, 1–6 (2019).
Chandravadia, N. et al. A NWB-based dataset and processing pipeline of human single-neuron activity during a declarative memory task. Sci. Data 7, 1–12 (2020).
Glasser, M. F. et al. A multi-modal parcellation of human cerebral cortex. Nature 536, 171–178 (2016).
Acknowledgements
We express our gratitude to all the patients who participated in this study; the neurologists at the Texas Comprehensive Epilepsy Program who participated in the care of these patients; and the nurses and technicians in the Epilepsy Monitoring Unit at Memorial Hermann Hospital who helped make this research possible. This work was supported by the National Institute for Deafness and other Communication Disorders DC014589 and National Institute of Neurological Disorders and Stroke NS098981. Continued development of the NWB Neurophysiology data standard was supported by the BRAIN Initiative supplement grant NS098971-03S1. Data presented here are stored in the Data Archive for the BRAIN Initiative (DABI) with support from the National Institutes of Health under Award Number R24MH114796. We especially thank Ben Dichter for assistance in NWB debugging and the continued development of NWB resources. We also thank Rachael Garner for help integrating our data into the DABI database and Latane Bullock for beta testing our example scripts.
Author information
Authors and Affiliations
Contributions
Data Collection and Curation: O.W., C.M.K., C.R.C., K.J.F., P.S.R., M.J.R., V.G.B.; Software: O.W., C.M.K.; Formal Analysis and Visualisation: O.W.; Writing – Original Draft: O.W.; Writing – Review and Editing: O.W., C.M.K., N.T.; Funding Acquisition: N.T.; Neurosurgical Procedures: N.T.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.
The Creative Commons Public Domain Dedication waiver http://creativecommons.org/publicdomain/zero/1.0/ applies to the metadata files associated with this article.
About this article
Cite this article
Woolnough, O., Kadipasaoglu, C.M., Conner, C.R. et al. Dataset of human intracranial recordings during famous landmark identification. Sci Data 9, 28 (2022). https://doi.org/10.1038/s41597-022-01125-8
Received:
Accepted:
Published:
DOI: https://doi.org/10.1038/s41597-022-01125-8
- Springer Nature Limited