Abstract
When interacting with the visual world using saccadic eye movements (saccades), the perceived location of visual stimuli becomes biased, a phenomenon called perisaccadic mislocalization. However, the neural mechanism underlying this altered visuospatial perception and its potential link to other perisaccadic perceptual phenomena have not been established. Using the electrophysiological recording of extrastriate areas in four male macaque monkeys, combined with a computational model, we were able to quantify spatial bias around the saccade target (ST) based on the perisaccadic dynamics of extrastriate spatiotemporal sensitivity captured by a statistical model. This approach could predict the perisaccadic spatial bias around the ST, consistent with behavioral data, and revealed the precise neuronal response components underlying representational bias. These findings also establish the crucial role of increased sensitivity near the ST for neurons with receptive fields far from the ST in driving the ST spatial bias. Moreover, we showed that, by allocating more resources for visual target representation, visual areas enhance their representation of the ST location, even at the expense of transient distortions in spatial representation. This potential neural basis for perisaccadic ST representation also supports a general role for extrastriate neurons in creating the perception of stimulus location.
Similar content being viewed by others
Introduction
Saccades are rapid eye movements that shift the center of gaze to a new location in the visual field. Changes in visual perception occur around the time of saccades1,2. For example, our subjective experience of the visual scene remains stable across the abrupt changes of the retinal image during saccades. This phenomenon is called visual stability, and many studies have attempted to explain the mechanism behind it3. Several other perceptual phenomena which occur around the time of saccades have also been studied psychophysically. For example, there is a general reduction in visual sensitivity during saccades, a phenomenon called saccadic suppression or saccadic omission, that has been reported in both macaques and humans4,5,6,7. Saccadic eye movements also alter our perception of time8. Another phenomenon is perisaccadic mislocalization, in which the perceived location of a visual stimulus appearing near the time of a saccade is biased. Perisaccadic mislocalization was first discovered as a perisaccadic shift, a unidirectional mislocalization parallel to the saccade, when the experiments were done in darkness9,10,11,12. Later studies have demonstrated perisaccadic compression13,14, which is mislocalization towards the saccade target (ST), when the subjects make saccades with background illumination and visual references15,16,17,18.
Perisaccadic visual perception in macaques is qualitatively similar to humans4, and many studies have investigated the neurophysiology of perisaccadic visual perception in nonhuman primates19,20,21,22,23,24,25,26. Some neurons in the extrastriate visual areas and prefrontal cortex show a sensitivity shift to the postsaccadic receptive field (RF) even before the saccade, a phenomenon often referred to as future field remapping or forward remapping27,28. There is also another phenomenon, called ST remapping or convergent remapping, in which neural RFs shift towards the ST around the time of saccade29,30,31,32,33,34,35,36,37,38. Both future field and ST remapping can be observed in the same experiments in the same group of neurons39,40,41,42. It has been suggested that RF remapping is associated with perisaccadic mislocalization19,43,44, and some studies have used computational approaches to predict perisaccadic perception of space based on neural responses44,45,46,47. Although these studies have generated insightful experiments, theories, and hypotheses, they usually start with assumptions about the function of visual areas or have a limited precision in accounting for the time-varying relationship between neural modulations and perceptual alterations on the millisecond timescale of saccades. By quantifying the statistical dependencies of spiking responses on several behavioral (e.g., eye movement) or external (e.g., visual stimuli) variables, point process statistical models provide a powerful means to capture the encoding and decoding of visual information as continuously varying with eye movements, with no assumption about the function of neurons. To investigate the neural basis of perisaccadic mislocalization, we used a time-varying generalized linear model (GLM) framework capable of capturing the fast spatiotemporal dynamics of neural sensitivity around the time of saccades42,48, and examined the link between perisaccadic visual responses and visuospatial perception.
In this study, we first measured perisaccadic mislocalization behaviorally and found significant mislocalization against the saccade direction for probes appearing near the ST perisaccadically. To investigate the possible neural correlates of the observed mislocalization, we used a combined experimental and computational approach built upon neuronal responses in the middle temporal (MT) cortex and area V4 of rhesus macaque monkeys. We assessed each neuron’s sensitivity to each location of visual space across time relative to the saccade (the neuron’s kernels) using a statistical model fitted on the recorded spiking data during a visually guided saccade task with visual stimulation. We quantified the representational spatial bias using the spatiotemporal kernels of populations of neurons, based on the similarity in neural sensitivity to neighboring probe locations, without assumptions about the downstream readout mechanisms. We then used this measure of spatial bias to identify the perisaccadic changes in sensitivity that drive it and linked them to neural responses.
We found that neurons with RFs close to the ST do not contribute to spatial bias. In contrast, perisaccadic spatial bias in the direction opposite to the saccade vector can be accounted for by neurons with RFs farther from the ST. These neurons show perisaccadic and postsaccadic sensitivity changes near the ST that contribute to spatial bias. Unexpectedly, we found that the time course and response components of the spatial bias match that of another perisaccadic perceptual phenomenon, namely the enhancement of neural sensitivity around the ST. This representational ST enhancement could be related to the presaccadically enhanced ST perception reported in psychophysical studies35,37,49 and to the presaccadic increase in stimulus selectivity24,26,50,51 or ST remapping38,39 observed in neurophysiological studies. The shared neural response components underlying the ST representational enhancement and spatial bias suggest that the brain potentially prioritizes ST representation with consequent biases in location perception.
Taken together, our findings highlight a potential neural basis for perisaccadic mislocalization, supporting a role for extrastriate neurons in the perception of stimulus location and linking enhanced sensitivity near the ST to perisaccadic spatial bias.
Results
We designed a behavioral paradigm called the Perisaccadic Localization Task to measure perisaccadic mislocalization. Two monkeys performed the task while their eye movements were monitored with a high-resolution eye-tracking system (see “Methods” section). During the task, the monkey made a first saccade from the fixation point (FP) to the ST. A probe stimulus appeared for 50 ms at one out of nine possible locations at a random time during fixation or around the time of the first saccade (Fig. 1a). After landing on the ST, the monkey then made a saccade to the probe location, and the endpoint of that saccade was taken as the reported location. Mislocalization was defined as the horizontal distance between the reported location of each stimulus when presented around the time of the first saccade compared to when presented during fixation (150:250 ms before the onset of the first saccade). The average time course of mislocalization for probes near the ST shows mislocalization against the saccade direction for probes presented perisaccadically (Fig. 1b). We also measured the localization error: the distance between the reported location of each stimulus from the actual location of the probe presented. The horizontal localization error for probes presented perisaccadically (−100:0 ms) was significantly more negative—i.e., larger error opposite the saccade direction—compared to those presented during fixation for both monkeys (Monkey 1: p = 2.57e−14, Monkey 2: p = 0.01) (Fig. 1c). We also divided the trials by the median reaction time of the monkey in each session and found no difference between the horizontal localization errors for perisaccadic probes based on reaction time (p = 0.10; Supplementary Fig. 1a); thus, mislocalization appears to be related to saccade execution rather than factors such as arousal or motor planning. Averaging across the localization errors for probes presented around the ST in each session, we observed significant horizontal localization error for perisaccadic probes compared to fixation (p = 8.22e−05; Supplementary Fig. 1b), but no significant vertical localization error (p = 0.48; Supplementary Fig. 1c).
To examine the neural basis of perisaccadic spatial biases in perception, we recorded the responses of extrastriate neurons (see ”Methods” section). We analyzed the activity of 300 neurons from MT and 147 neurons from area V4, recorded while monkeys performed a visually-guided saccade task (Fig. 2a). In this task, the trial began when the monkey fixated on a central FP, upon which an ST appeared 13 degrees of visual angle (dva) away either to the left or the right horizontally, while the monkey held the fixation. When the FP disappeared, the monkey made a saccade to the ST and maintained fixation on the ST during the second fixation period. A series of probe stimuli were presented throughout the task while the monkey fixated and made a saccade. Only one stimulus was presented at a time, selected from a 9×9 grid of possible locations, and each stimulus appeared for 7 ms. The probe grid position and spacing were adjusted to cover the FP, ST, and estimated RF of the neuron. In order to computationally investigate the mechanism of spatial bias, we developed an encoding model that quantitatively characterizes the neuron’s input-output relationship and captures the neuron’s sensitivity map with high temporal precision throughout the eye movement task (see “Methods” section and ref. 48 for details). The model traces the time-varying dynamics of a neuron’s sensitivity across saccades with high-dimensional spatiotemporal kernels. First, we discretized the space into 81 locations in a two-dimensional plane (representing the 81 probe locations) and discretized all times of the neural response relative to saccade onset and delays (times of the stimulus relative to each response time) into 7-ms bins, to form spatiotemporal units (STUs) in a 4-dimensional space (Fig. 2b). These STUs were used to construct the spatiotemporal kernels of each neuron, which represent the neuron’s sensitivity across locations, times, and delays. The neuron’s spatiotemporal kernels were constituted by a weighted linear combination of STUs, where the weight by which each STU contributed to the kernel was estimated using a model that could capture the neuron’s spiking response. Figure 2c shows the STU weights at an example probe location around the RF of an example neuron across time and delay. When responding to a probe stimulus at that location, the STUs form kernels that represent how a neuron’s sensitivity changes across time from saccade onset and across delays. We used the Sparse Variable Generalized Linear Model (SVGLM)48 to estimate the STU weights and the resulting kernels by fitting the model to the neuron’s spiking responses (see “Methods” section and Supplementary Fig. 2). A signal representing the stimulus time and location passes through spatiotemporal kernels (representative of the neuron’s time-varying spatiotemporal sensitivity) and is added to the time-varying baseline neural activity relative to saccade onset (captured by an offset kernel) and the feedback signal generated using a post-spike filter (representing the effects of spiking history). The combined signal is then passed through a sigmoidal nonlinearity capturing the spike generation. The resulting firing rates are used to generate spikes with the Poisson spike generator. These spikes are then fed back to the circuit through the abovementioned post-spike history (Supplementary Fig. 2a). All the model components are learned via an optimization process to directly estimate the recorded spiking activity (see ref. 48 for details). The kernels estimated from the model reflect the time-varying temporal sensitivity of the neuron across space (Fig. 2c).
Next, we developed a procedure to measure spatial bias based on the neurons’ estimated kernels. We made the assumption that similar neural sensitivity to probes appearing at neighboring locations could create uncertainty in a readout of the stimulus location by a downstream area, which would lead to a bias in spatial perception. In other words, if during a saccade the population response to one probe becomes similar to that of a neighboring probe, we can assume a representational bias toward that neighboring location, without specifically modeling downstream readout mechanisms. To examine the neural basis of spatial bias, we analyzed the similarity between the spatiotemporal kernels at pairs of probe locations in a population of neurons. For the sensitivity analysis at the population level, we divided the neurons into ensembles based on their RF locations. Neurons recorded with the same RF, ST, and grid arrangements were grouped as an ensemble, and each ensemble had a minimum of 10 neurons. Figure 2d shows the kernel maps at 9 probe locations around the RF for an example ensemble of neurons. For each ensemble, we measured the similarity between the neural sensitivity at neighboring locations by assessing correlations between the kernel weights at the center probe and the kernel weights at each of its eight neighboring probes using a cosine similarity measure. The cosine similarity was measured at each time and delay using the corresponding kernel weights. For the rest of the paper, correlation always refers to cosine similarity between the kernel weights for neighboring probes across neurons in an ensemble. In this study, we focused on examining the spatial bias around the ST because prior psychophysics often reported perisaccadic mislocalization close to the ST15,16. We defined a spatial bias measure based on relative ensemble sensitivity to probes near the ST. Figure 2e shows the correlations of kernel weights between an example probe close to the ST and its 8 neighboring probes, for an example ensemble of 53 neurons at time 100 ms and delay 110 ms. The correlation coefficient between the central probe and its neighboring probes (central polar plot) indicates the similarity of neural sensitivity in each direction at that central probe location (Fig. 2e brown arrows). We then averaged over the eight vectors at each probe location to get one vector (Fig. 2e purple arrow). Since the saccade direction was either to the left or the right horizontally, we focused on the horizontal projections of the average vectors, which we defined as the spatial bias. In order to combine data from sessions with leftward and rightward saccades, values were normalized according to saccade direction so that positive always means the same direction as the saccade, and negative means the opposite direction from the saccade. This spatial bias measurement allowed us to predict potential mislocalization of stimuli based on the kernels of the SVGLM fit to neural data.
Next, we examined how spatial bias changed over time relative to the saccade and the stimulus onset. Each kernel map has its own time and delay dimensions (e.g. Fig. 2d), so we measured spatial bias maps across time and delay for each of the 7×7 non-edge probe locations for each ensemble. Figure 3a shows the spatial bias over delay, at response time 100 ms, at a probe location close to the ST for an example ensemble, and Fig. 3b shows the spatial bias over time at delay 110 ms for the same ensemble and probe location. In this study, we focused on the probe locations around the ST. For each ensemble, we selected 6 locations around the ST and averaged their bias maps. We then averaged the bias maps for 15 ensembles (447 neurons) (Fig. 3c), where the individual maps were first normalized so that each ensemble had bias values ranging from −1 to 1 (see “Methods” section). Taking the mean over delays of 50:100 ms, we observed a significant negative bias (which means a representational shift in the direction opposite to the saccade direction) of −0.14 ± 0.06 around the ST for responses ~50:150 ms after saccade onset compared to during fixation (Fig. 3d and Supplementary Table 1). We also measured the vertical projection of spatial bias and found no difference between fixation and perisaccadic windows (Supplementary Fig. 3a); all subsequent analysis is therefore restricted to horizontal bias. A second method of measuring spatial bias based on optimal decoding also predicted perisaccadic mislocalization in the direction opposite the saccade (Supplementary Fig. 4). Altogether, the neural results were consistent with the behavioral results (Fig. 1). To find out which population of neurons contributes to perisaccadic mislocalization, we grouped ensembles of neurons based on d, the distance between the RF center and the ST (Fig. 3e). Ensembles with d < 11 dva show very little bias (0.06 ± 0.12). To examine the variability of neurons within each ensemble and its possible effect on the amount of bias, we resampled 90% of the neurons in each ensemble to compute 100 samples of spatial bias for each ensemble. The mean spatial bias in the perisaccadic response window of 50:150 ms demonstrates that most of the ensembles with RFs closer to the ST show less spatial bias, and the standard error of the mean shows that the phenomenon within each ensemble is consistent (Fig. 3f). Thus, our population of neurons demonstrate perisaccadic spatial bias opposite to the saccade direction, primarily driven by neurons with RFs far from the ST.
The above results show that the perisaccadic changes in the spatiotemporal sensitivity of MT and V4 neurons could account for changes in spatial perception during eye movements, but so far, we have focused on the representation at the population level and model-based neural sensitivity measurements. In order to find out which components of the neuronal response of which neurons account for the perisaccadic alteration in the readout of location, we used an unsupervised approach to search for response components that are specifically related to spatial bias. In this study, spatial bias was defined based on similarity in the population representation of neighboring probe stimuli, as captured by the neurons’ spatiotemporal kernels; since the kernels are comprised of STUs, manipulation of certain STU weights can change the kernels and thereby affect the similarity between the population sensitivity at neighboring locations. This assumption-free alteration in the model enables us to determine which STUs are necessary for creating perisaccadic spatial bias. Based on this rationale, we defined a bias index according to the difference between the center kernel and each neighboring kernel across times and delays. Nulling each modulated STU one by one, we quantified their effect on the kernel similarity using this bias index, and systematically identified the bias-relevant STUs (see “Methods” section; Supplementary Fig. 5). Using this unbiased search in the space of STUs, we found different phenomena for ensembles with different distances between the RF center and the ST (d), so we divided the ensembles into two groups (d < 11 dva and d ≥ 11 dva) to examine their bias-relevant STUs separately (Fig. 4a). For ensembles with d < 11 dva, there was a set of bias-relevant STUs around response time 60:90 ms and delay 80:110 ms. For ensembles with d ≥ 11, there are two areas of bias-relevant STUs—one around response time 60:100 ms and delay 60:110 ms, and the other one around response time 110:280 ms and delay 50:100 ms. After removing all the identified bias-relevant STUs, we recomputed the spatial bias over time, and the previously observed bias around 50:150 ms after saccade onset is significantly reduced (Fig. 4b; −0.03 ± 0.04; p = 0.04), confirming that the identified set of STUs drive this bias. Thus, by leveraging the capabilities of the model to decompose the spatiotemporal sensitivity of individual neurons, we were able to identify the specific changes in neural sensitivity that contribute to perisaccadic spatial bias.
To interpret how the saccade-related changes in STUs link neurophysiological activity to a biased readout of location information, we wanted to relate them back to the neural responses. The model allows us to generate responses to synthetic stimuli and compare the predicted neural response during fixation and perisaccadic windows. We first examined the model-predicted response for ensembles with d < 11 and transformed the time and delay of the bias-relevant STUs to a stimulus-aligned response (Fig. 5a). To investigate the neural response underlying the perisaccadic change in spatial bias, we looked at how neurons responded to probes on different sides of the ST. Data from ensembles recorded with leftward saccades have been flipped to be combined with those recorded with rightward saccades. Out of the six probes around the ST, we called the three probes closer to the FP the “near” probes and the other three probes the “far” probes (Fig. 5b). RFs of neurons in ensembles with d < 11 mostly cluster between FP and ST (see prevalence in Fig. 5b), resulting in the near probes falling close to the RFs. Based on Fig. 5a, we averaged the model response for near vs. far probes over fixation (−500:−100 ms) and perisaccadic (−20:10 ms) windows, and used the neurons’ responses from experimental recordings as validation (Fig. 5c). We specifically compared the responses in 60-ms windows of time from stimulus onset, around the peak of the fixation and perisaccadic responses (fixation: 50:110 ms, perisaccadic: 70:130 ms). During fixation, there is a greater model-predicted response to near probes vs. far probes (near = 1.05 ± 0.01, far = 1.01 ± 0.01; p = 1.25e−09; n = 94 neurons), consistent with the near probes being closer to the RF centers. There is an increase of model-predicted response perisaccadically for both near and far probes, but more of an increase for far probes, such that the perisaccadic responses end up being similar for near and far probes (near = 1.05 ± 0.02, far = 1.07 ± 0.02; p = 0.13; n = 94). The model closely predicts the response from actual neurons during both the fixation window (experimental values: near = 1.04 ± 0.01, far = 0.94 ± 0.01) and the perisaccadic window (experimental values: near = 1.20 ± 0.04, far = 1.20 ± 0.04). We measured the statistical difference between the actual firing rates of neurons in response to near vs. far probes (Fig. 5d), in 60-ms windows matched to their evoked responses. During the fixation period, from 50 to 110 ms after stimulus onset, the firing rate evoked by near probes is significantly higher than that for far probes (near = 38.57 ± 2.30 Hz, far = 35.21 ± 2.18 Hz; p = 5.38e−12). During the perisaccadic period, from 70 to 130 ms after stimulus onset, there is no statistically significant difference between the firing rates in response to near vs. far probes (near = 43.39 ± 2.63 Hz, far = 43.13 ± 2.57 Hz; p = 0.50). These neural responses show that neurons with RFs close to the ST responded more to near probes during fixation, but respond equally to both near and far probes around the time of saccades. The lack of difference in response indicates that there is no neural bias towards either side of the ST around the time of eye movements, which explains the absence of spatial bias for ensembles with d < 11.
Next, we examined the model-predicted and actual fixation and perisaccadic neural responses for ensembles with RFs far from the ST. Similar to Fig. 5a, we transformed the axes to examine the relationship between bias-relevant STUs and the model response for ensembles with d ≥ 11 (Fig. 6a). Results look similar for MT and V4 neurons (Supplementary Fig. 6). Since there are two regions of bias-relevant STUs for this group of neuronal ensembles (Fig. 4a bottom), the contours in Fig. 6a also illustrate two temporal regions of the model-predicted response that might contribute to spatial bias. Near and far probes were defined as in Fig. 5b; however, for these ensembles most of the neurons have RFs on the other side of the FP from the ST (see prevalence in Fig. 6b). Since there are two regions of the model-predicted response that could potentially contribute to spatial bias, we compared the model’s response at near vs. far probes during fixation (−500:−100 ms), perisaccadic (0:40 ms), and postsaccadic (70:200 ms) windows (Fig. 6c from top to bottom). We quantified responses in a 60 ms window covering the peak of each response (shown by the gray bars, fixation: 30:90 ms, perisaccadic: 40:100 ms, postsaccadic: 30:90 ms). During fixation, there is no response to either near or far probes (near = 0.99 ± 0.00, far = 0.99 ± 0.00; p = 0.86). Perisaccadically, responses were observed for both near and far probes, with a larger increase of response for near probes (near = 1.05 ± 0.01, far = 1.01 ± 0.01; p = 1.05e−12). Postsaccadically, there is a continued increase in response at near probes, but the response at far probes decreases back to the fixation level (near = 1.06 ± 0.01, far = 0.99 ± 0.00; p = 5.82e−22). Neuron’s responses mirror the model’s predictions in the fixation window (near = 0.98 ± 0.01, far = 0.10 ± 0.01), perisaccadic window (near = 1.07 ± 0.02, far = 1.03 ± 0.02), and postsaccadic windows (near = 1.12 ± 0.01, far = 1.05 ± 0.01) (Fig. 6c from top to bottom). Figure 6d demonstrates that there is no significant difference between firing rates at near vs. far probes during fixation (near = 26.27 ± 1.01 Hz, far = 26.33 ± 1.02 Hz; p = 0.31), but during the perisaccadic response window the neural firing rate for near probes is significantly higher than the firing rate for far probes (near = 28.51 ± 1.08 Hz, far = 27.21 ± 1.07 Hz; p = 9.44e−10) and continues during the postsaccadic response window (near = 29.27 ± 1.02 Hz, far = 27.27 ± 1.01 Hz; p = 1.23e−13). Neurons with RFs far from the ST do not respond to either near or far probes during fixation, but respond more to near probes perisaccadically and postsaccadically. Neurons respond more strongly to near-ST stimuli closer to the FP, reflecting the spatial bias opposite to the saccade direction in ensembles with d ≥ 11. These findings demonstrate how this systematic and unbiased search in the space of spatiotemporal sensitivity components can identify the neural basis for a biased representation of visual space during eye movements.
To gain a deeper understanding of the nature of perisaccadic mislocalization, we wanted to investigate how perisaccadic neural modulations are associated with the representation around the ST and how it might be related to the observed spatial bias. The neural firing rate for probes in the initial RF is higher during fixation (−400:−200 ms) than perisaccadically (50:150 ms), but higher perisaccadically than during fixation for probes near the ST, demonstrated by sample neurons (Fig. 7a) and the distributions of firing rate differences over the population (Fig. 7b; top: p = 1.32e−45, bottom: p = 2.73e−15). To assess the change of neuronal sensitivity around the ST in the corresponding time and delay windows as the spatial bias, we defined a ST sensitivity index using kernels averaged over delays of 50:100 ms. This sensitivity index quantifies the average changes in sensitivity for near-ST probes and can theoretically vary independently of the spatial bias measure. Out of the 6 ST probes, we divided the range of kernel weights by the mean kernel weight over all times relative to saccade onset to quantify the difference in sensitivity of a neuron to all probes presented around the ST area across time from saccade onset (Fig. 7c). We excluded 95 neurons with high kernel weights during the second fixation period (240:440 ms from saccade onset) comparing to the first fixation period (−441:−241 ms) (i.e., neurons whose postsaccadic RF included the near-ST probe locations) to reduce the interference of future field activity. In the same perisaccadic time window that we observed the spatial bias (50:150 ms shown by gray bar in Fig. 7c), there is an increase in the ST sensitivity index compared with the fixation window (−300:−150 ms; Fig. 7d; fixation = 3.27 ± 0.07, perisaccadic = 3.67 ± 0.09; p = 0.04). The increase indicates that the modulation of neurons’ spatiotemporal sensitivity around the time of saccades enhances the representation of the ST area. To examine the relationship between the spatial bias and enhanced ST representation, we measured the ST sensitivity index again with the reduced model in which bias-relevant STUs were nullified (Fig. 4b). In the same perisaccadic window, the ST sensitivity index in the reduced model is significantly smaller than in the full model (Fig. 7e; 3.31 ± 0.12; p = 3.18e−09). Thus, the enhanced ST sensitivity index around the ST relies on the bias-relevant STUs, and a computational manipulation that removes spatial bias leads to decreased sensitivity around the ST. This reveals that the perisaccadic spatial bias could be a result of the same changes in sensitivity, which enhance the ST representation around the time of saccades.
Discussion
How the location of visual stimuli is represented in the brain is not well understood. Imaging studies have suggested that the perceived location could be encoded in extrastriate visual areas along with other visual features52,53. Our perception of location changes around the times of saccades2,4, as do extrastriate responses27,41,54. We measured monkeys’ perception of stimulus location behaviorally and found perisaccadic mislocalization opposite to the saccade direction for stimuli near the ST. We then used a combined experimental and computational approach to examine how changes of sensitivity in MT and V4 could explain perisaccadic mislocalization. We quantified perisaccadic spatial bias around the ST and identified the STUs relevant for the observed bias, which reveals that neurons with RFs far from the ST contribute more to the perisaccadic spatial bias. We found perisaccadic changes in extrastriate sensitivity in the identified bias-relevant time and delay windows, supporting the hypothesis that location representation occurs in extrastriate visual areas. In addition, we demonstrated that the spatial bias is accompanied by the perisaccadic enhancement of neural sensitivity around the ST, with a matching time course and underlying neural response components. This enhancement of ST representation is consistent with previous behavioral and neural findings24,26,35,37,49,50,51, suggesting that the brain prioritizes ST representation at the expense of biases in location perception.
The spatial bias measure predicts perisaccadic mislocalization opposite the saccade direction; importantly, this matches the direction of mislocalization we measured behaviorally in monkeys under conditions similar to the neurophysiological recordings (Fig. 1). The previous psychophysics results have been mixed, but our behavioral and neurophysiological results are consistent with many aspects of the previous literature. In total darkness, Honda reported that mislocalization in human subjects starts in the same direction as saccade and then is reversed to the opposite direction, with the greatest mislocalization occurring ~50 ms after saccade onset9,10. In a double-saccade task, Jeffries et al. found that mislocalization in rhesus monkeys is in the direction opposite to the first saccade, with the maximum mislocalization around 100 ms after saccade onset11. Klingenhoefer and Krekelberg similarly reported that monkeys mislocalized stimuli between the FP and ST in the direction opposite to the saccade4. The magnitude of mislocalization we observed is smaller than that reported in the previous literature, which might be due to various differences in the paradigm, including a shorter saccade vector, smaller offset between probes, smaller range of possible probe locations, and limiting the probes to within 5 dva from the ST. Based on the model’s kernels, we observed spatial bias in the direction opposite the saccade, at a timing consistent with both the human and nonhuman primate studies (Fig. 3d); however, we cannot rule out the possibility that examining different RF or probe positions could reveal cases of spatial bias in the saccade direction. In addition to mislocalization parallel or opposite to the saccade direction, many studies have reported compression when conducting the experiments in a dimly lit room15,16,17, meaning that stimuli are perceived as closer to the ST (i.e., mislocalization opposite the saccade direction for stimuli past the ST, and in the saccade direction for others). In a computational study, Krekelberg et al. also predicted mislocalization in the direction of the saccade at a location close to the FP, and mislocalization in the opposite direction at locations near and past the ST19. They implemented a decoder using nonhuman primate neural data recorded from MT, the medial superior temporal area (MST), the ventral intraparietal area (VIP), and the lateral intraparietal area (LIP). Decoder-based predictions of how perceived location will be altered by various types of remapping vary qualitatively based on assumptions about the decoder; specifically, whether the same decoding algorithm is applied during the perisaccadic and fixation periods (‘unaware’ of the RF shift), or with knowledge of the perisaccadic responses changes (‘aware’). In a theoretical study done by Qian and colleagues, a decoder predicted that future field remapping would result in mislocalization opposite to the saccade direction when the decoder is unaware of the RF shift55. When there is ST remapping, the unaware decoder predicted divergent mislocalization, and the aware decoder predicted convergent mislocalization. Like most previous studies attempting to understand this connection, our approach for measuring spatial bias assumes the same decoding algorithm is used during fixation and around the time of saccades, with altered visual responses driving the perisaccadic perceptual changes. We only found spatial bias opposite to the saccade direction for stimuli around the ST; however, we are not ruling out the possibility of a compression phenomenon or convergent mislocalization, because in this study we did not measure spatial bias for stimuli at locations other than near the ST (nor were our probe positions optimized to make such systematic measurements across the rest of the visual field).
Our results substantiate the association between perisaccadic mislocalization and changes in perisaccadic sensitivity (i.e., RF remapping)19,43,44. Many studies have interpreted perisaccadic mislocalization as a flaw in the visual system while shifting the coordinate systems across saccades10,13,56, but it is not clear what the reason for this flaw is, or if it is the byproduct of another, beneficial, set of changes. Previous psychophysical studies have reported enhanced discrimination performance at the ST35,37 and neurophysiological studies have shown a perisaccadic increase in neural sensitivity at the ST24,26,38,39,50,51. The exact spatial extent of this improved perception near the saccade target is unknown, and locations farther from the saccade endpoint instead display an overall decrease in visual sensitivity around the time of saccades, known as saccadic suppression4,5,6,7,35. Our model-based measure shows increased sensitivity to perisaccadic stimuli within 5 dva of the ST. The saccade target theory has hypothesized that the brain biases toward representing the ST in order to maintain visual stability, and the representation of non-target locations is consequently reduced57,58. Our results demonstrate that removal of bias-relevant neural components is correlated with a reduction of perisaccadic sensitivity around ST (Fig. 7c). Based on our results, we suggest that spatial mislocalization could be a result of allocating more neural resources toward the ST representation. The spatial bias could therefore be interpreted as a tradeoff the brain makes to amplify the ST representation perisaccadically, consistent with the saccade target theory and ST remapping. It should be noted that future field remapping could also contribute to perisaccadic spatial bias, and our dataset was not optimized to definitively differentiate between these two forms of remapping. Figure 6c shows increased perisaccadic response around the ST that might be induced by ST remapping, and the increased postsaccadic response could reflect future field remapping. This possible link between future field remapping and mislocalization will require further investigation. We also cannot definitively state whether these spatial biases in responses arise first in MT and V4 or are inherited from upstream areas.
Our approach in this study also reinforces the feasibility of using a GLM framework to model higher visual areas. The classical GLM has been widely used for encoding and decoding neural responses in low-level visual areas (such as the retina, LGN, and V1)59,60, but they fall short in capturing the time-varying characteristics of higher-level visual areas. To model responses in these areas, nonstationary model frameworks that enable a time-varying extension of a GLM have been developed, which succeeded in characterizing the perisaccadic spatiotemporal changes of neural response and reading out perisaccadic stimulus information on the same timescale of saccadic eye movements42,48,61,62,63. In the present study, we took advantage of this GLM framework (SVGLM) and developed a procedure to measure spatial bias based on instantaneous neural sensitivity at various locations to identify the neural components contributing to spatial bias. Our results provide a potential explanation of the neural basis of mislocalization, which could be tested most definitively through experiments combining psychophysical measurements in macaques with causal manipulations of neural activity. These applications of the SVGLM framework demonstrate that a GLM-based approach is a viable way of studying the complex dynamics in higher-level visual areas, and could also be used to link specific aspects of neural sensitivity to different perceptual phenomena.
Methods
All experimental procedures complied with the National Institutes of Health Guide for the Care and Use of Laboratory Animals and the Society for Neuroscience Guidelines and Policies. The protocols for all experimental, surgical, and behavioral procedures were approved by Institutional Animal Care and Use Committee of the University of Utah.
Behavioral paradigm for Perisaccadic Localization Task
Two adult male rhesus macaques (Macaca mulatta; both 9 years old) performed the Perisaccadic Localization Task. To start a trial, the monkey held fixation on a central FP. After the monkey had held fixation for 500 ms, a ST appeared 10 dva away from the FP horizontally. In each recording session, there was only one saccade direction (leftward or rightward). At a random time during fixation or saccade execution, a 50-ms visual probe stimulus was presented in one of nine possible locations in a 3×3 grid. The probes in the grid were 4 dva apart from each other. After the monkey held fixation for 1000 ms, the FP disappeared, which was the go cue to saccade to the ST. The monkey then made a second saccade to report his perceived location of the probe stimulus. If the monkey reported within 4 dva radius of the correct location, feedback was given by a reappearing flash of the stimulus, and the monkey received a juice reward. The FP and ST were white circles (full contrast) with a radius of 0.25 dva, and the probe stimuli were red circles with a radius of 0.4 dva, against a black background. We collected behavioral data during 89 sessions for Monkey 1 and 40 sessions for Monkey 2. The Perisaccadic Localization Task is based on tasks previously used to characterize perisaccadic mislocalization in monkeys4,11,19; it was designed to be similar to the paradigm used for separate neurophysiological recordings in terms of ambient lighting conditions, probe size, background color, and saccade amplitude.
Combined behavioral and electrophysiological recording
We trained and recorded from four adult male rhesus macaques (Macaca mulatta; 6, 6, 11, and 13 years old). The behavioral task used in this study was a visually guided saccade task, with probe stimuli appearing at pseudorandom locations before, during, and after the saccade. To start a trial, the monkey held fixation on a central FP. While the monkey was holding fixation, a ST appeared 13 dva away from the FP horizontally. In each recording session, there was only one saccade direction (leftward or rightward). After a randomized time-interval (uniform distribution between 700 and 1100 ms), the FP disappeared, which was the go cue for the monkey to saccade to the ST. The monkey then held fixation on the ST for 560:750 ms to receive a juice reward. Throughout the length of each trial, a complete sequence of 81 probe stimuli flashed on the screen in pseudorandom order, one at a time for 7 ms each. The probe locations were selected pseudorandomly from a 9×9 grid of possible locations. Each probe stimulus was a white square (full contrast), 0.5×0.5 dva, against a black background. Each probe stimulus occurred at each time in the sequence with equal frequency across trials.
During each neurophysiological recording session, the grid of possible locations of the probes was placed and scaled to cover the estimated presaccadic and postsaccadic RF centers of the neurons recorded, the FP, and the ST. The probe grids varied in size horizontally from 24 to 48.79 (40.63 ± 5.93) dva, and vertically from 16 to 48.79 (39.78 ± 7.81) dva. The distance between two adjacent probe locations varied horizontally from 3 to 6.1 (5.07 ± 0.74) dva, and vertically from 2 to 6.1 (4.97 ± 0.97) dva.
While the monkey was performing the task, we monitored their eye movements with an infrared optical eye-tracking system (EyeLink 1000 Plus Eye Tracker, SR Research Ltd., Ottawa, CA) with a resolution of <0.01 dva (based on the manufacturer’s technical specifications), and a sampling frequency of 2 kHz. Presentation of the visual stimuli on the screen was controlled using the MonkeyLogic toolbox. In total, 332 neurons in the middle temporal (MT) cortex and 291 neurons in area V4 were recorded in 108 sessions, but only 300 MT and 147 V4 neurons were used in order to make ensembles of neurons with at least 10 neurons with a similar RF, ST, and grid position during recording. We recorded both spiking activity and the local field potential (LFP) from either MT or V4 using 16-channel linear array electrodes (V-probe, Plexon Inc., Dallas, TX; Central software v7.0.6 in Blackrock acquisition system and Cheetah v5.7.4 in Neuralynx acquisition systems) at a sampling rate of 32 KHz, and sorted neural waveforms offline using the Plexon offline spike sorter and Blackrock Offline Spike Sorter (BOSS) software.
Encoding model framework
The Sparse Variable Generalized Linear Model (SVGLM) used in this study was previously developed by Niknam et al.48, see this paper for more details of the model fitting. The SVGLM is a variant of the widely used GLM framework59,60,64 that tracks the fast dynamics of sparse spiking activity with high temporal precision and accuracy. The SVGLM can capture the neurons’ sensitivity varying over space and time with high temporal resolution by providing a parameterized representation of the neuron’s spatiotemporal kernels using discrete STU components. Through a dimensionality reduction process, the model selects STUs which make a statistically significant contribution to the neuron’s response to achieve sparsity for fitting the parameters to the data (see Supplementary Information). The fitted model also captures how much these STUs contribute quantitatively to generating spikes on a precise millisecond timescale during a saccade. The weighted combination of these STUs constitutes the spatiotemporal stimulus kernels. The SVGLM defines a conditional intensity function according to the equation,
where \(\lambda\) is the instantaneous firing rate of the neuron at time \(t\) in trial \(l\), \({s}_{x,y}^{\left(l\right)}\) is either 0 or 1 representing respectively the off or on condition in a sequence of probe stimuli presented on the screen at probe location (\(x,y\)) in trial \(l\). \({r}^{\left(l\right)}\left(t\right)\) denotes the spiking response of the neuron for trial \(l\) and time \(t\), \({k}_{x,y}\left(t,\tau \right)\) represent the stimulus kernels, \(h\left(\tau \right)\) indicates the post-spike kernel applied to the spike history which captures the refractory effect, \(b\left(t\right)\) is the offset kernel that reflects the change of baseline activity induced by saccades, the constant \({b}_{0}={f}^{-1}\left({r}_{0}\right)\) with \({r}_{0}\) as the measured mean firing rate (Hz) across all trials in the experimental session, and \(f\left(u\right)=\frac{{r}_{\max }}{1+{e}^{-u}}\) is a static sigmoidal function that describes the nonlinear properties of spike generation with \({r}_{\max }\) indicating the maximum firing rate of the neuron obtained empirically from the experimental data. The model was fitted using an optimization procedure in the point process maximum likelihood estimation framework65 at the level of single trials. The evaluation for model performance is described in supplementary information (Supplementary Fig. 2b–d). More details about the model structure, estimation, and evaluation can be found in ref. 48.
Measuring spatial bias
Neurons recorded with the same ST position and probe arrangements (grid positioning and spacing), and with similar RF locations were grouped as an ensemble. Fifteen ensembles were formed, each with a minimum of 10 neurons. Before any analysis, kernels of all neurons were smoothed by moving average using boxcar windows of length 50 ms across time \(t\) and 20 ms across delay \(\tau\) to reduce noise. Figure 2c shows 9 kernels for a sample probe at the center of a neuron’s RF and its 8 neighboring probes, stacked over neurons in an example ensemble. For each particular time and delay, we constructed two population kernel vectors consisting of the kernel values of center probe and a neighbor probe at that time and delay with all neurons in an ensemble. To measure the similarity between kernels at neighboring probe locations, we computed the correlation between the kernels of center probe and a neighbor probe with all neurons in an ensemble, and subtracted baseline correlation values during fixation (−441:−141 ms from saccade onset). The correlation was measured for each of the 8 neighboring probes and repeated for 7×7 probe locations (after excluding probes on the edges). Using correlation values as magnitudes, and the probe position relative to the center probe as directions, we formed 8 vectors at each probe location across time and delay and took the average of these 8 vectors at each of the 7×7 probe locations. The polar plot in Fig. 2e shows these vectors between a sample probe around ST and its 8 neighboring probes. Spatial bias at each location was defined as the horizontal projection of the average vector at that location, and it was computed for all 15 ensembles. These spatial bias values were used to construct spatial bias maps across time and delay for each of 7×7 probe locations. Figure 3a, b shows two cross-sections of an example bias map, associated with a sample probe location around ST, over particular time and delay windows. For each ensemble, we averaged the bias maps at the 6 probe locations closest to the ST, excluding probes that were within 2 dva from either the presaccadic or postsaccadic RF (Fig. 3c). Before averaging the bias maps of all 15 ensembles, the original spatial bias of each ensemble was near-symmetric around zero and normalized to range from −1 to 1 using the following formula: 2×(bias−min(bias))/(max(bias)−min(bias))−1. We used one-sided Wilcoxon signed-rank test to report p-values for all our statistical comparison analysis, if not mentioned specifically.
Identifying modulated STUs
To identify which components of the neurons’ spatiotemporal sensitivity drive the neuron’s response changes around the time of saccades, we first quantified the contribution of each STU. It was expected that out of all STUs, only some of them at specific times and delays contribute to the generation of the neural response (referred to as ‘contributing STUs’). These contributing STUs were identified using the dimensionality reduction process during the model fitting, based on a statistical significance criterion, which measures the contribution of individual STUs to capturing the stimulus-response relationship (see Supplementary Information, and Niknam et al48. for details).
We then defined the modulated STUs as those for which the fraction of contributing STUs in a 3×3 window around that STU’s time and delay for each STU’s spatial location is significantly different during perisaccadic period compared to fixation period. Mathematically speaking, the fraction of contributing STUs needs to fulfill the following condition:
with \(p\left({\tau }_{n},{t}_{m}\right)\) as the fraction of contributing STUs in a 3×3 window around the \(n\) th bin of delay and \(m\) th bin in time 1 < \(n\) < 30,1 < \(m\) < 156. \({p}_{1}\left({\tau }_{n}\right)\) is the fraction of contributing STUs during the first fixation period 540 to 120 ms before saccade in time bin 1 to 60 at \(n\) th bin of delay (1 < \(n\) < 30), and \({p}_{2}\left({\tau }_{n}\right)\) is the fraction of contributing STUs during the second fixation period 280 to 540 ms after saccade in time bin 120 to 156 at \(n\) th bin of delay (1 < \(n\) < 30). \(h\) is a significance threshold between 0 and 1, and was set to 0.3 for the analysis.
Identifying bias-relevant STUs
From the list of modulated STUs, we identified the ones that contribute to spatial bias specifically (termed bias-relevant STUs). The contribution of each modulated STU to the spatial bias was quantified by evaluating its impact on the difference between kernels at neighbor probes across a saccade, by removing each modulated STU one at a time and testing if the change in kernels difference is significant based on a bias index. Because spatial bias was measured from the correlations between kernels at neighbor probes for an ensemble of neurons, the difference between the stimulus kernels of two neighboring probes for individual neurons, may impact the resulting spatial bias read out from that ensemble. The absolute difference between each pair of stimulus kernels of the fitted models at two neighboring probe locations (\({x}_{0},{y}_{0}\)) and (\({x}_{i},{y}_{i}\)) and at each delay (\(\tau\)) across different times to the saccade (\(t\)), was quantified as
where \({{AUC}}_{i}\), represents the area under curve of the difference of kernels \({k}_{{x}_{0},{y}_{0}}\left(t,\tau \right)\) and \({k}_{{x}_{i},{y}_{i}}\left(t,\tau \right)\) over time and delay, between each center probe at (\({x}_{0},{y}_{0}\)) and each of the eight neighbor probes (\(i\in \{1,\ldots,8\}\)) at (\({x}_{i},{y}_{i}\)) (Supplementary Fig. 5a). Since the spatial bias was measured for 6 probe locations around ST, we also measured the difference AUCs at those same 6 center probes for the neurons in each ensemble. The average of difference AUCs over 8 center-surround probe pairs was used to compute the bias index associated with individual center probe in the following. For each neuron in each ensemble, we first measured the difference AUCs with the full model (no perturbation in the model estimated STUs). Next, we removed each of the modulated STUs one at a time from the full model by replacing that STU with zero and repeat the above steps so that we have a list of difference AUCs measured without each of the modulated STUs. We then defined the bias index for each modulated STU as the absolute difference between the AUC for full model and the AUC corresponding to removing each of the modulated STU from the model (Supplementary Fig. 5b). To identify bias-relevant STUs, we defined a threshold for this bias index as the 90th percentile of the cumulative distribution function of all the nonzero bias indices (bias index of 2.56) as the threshold (Supplementary Fig. 5c). A larger bias index means that nulling the weight of a particular STU results in a stronger change in kernel differences for the probes around the ST, so the STUs with a bias index above the threshold were classified as bias-relevant. The bias indices were specific to each of the 6 ST probes and for each neuron. Figure 4a shows the map of bias-relevant STUs which was generated based on the mean bias index across probes and neurons. To validate if the identified bias-relevant STUs using this procedure would actually contribute to the readout spatial bias from each ensemble, we removed the identified bias-relevant STUs from the model for each neuron, and recomputed the spatial bias (Fig. 4b).
Reporting summary
Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.
Data availability
Source data for figures are provided with this paper. Sample neurons from the data generated in this study have been deposited in the GitHub page here: https://github.com/nnategh/SVGLM/tree/master/assets/data. Source data are provided with this paper.
Code availability
Source codes for the model fitting are available at https://github.com/nnategh/SVGLM. Sample codes for the analysis of this study are available at https://github.com/nnategh/Neural_Correlates_of_Mislocalization.
References
Marino, A. C. & Mazer, J. A. Perisaccadic updating of visual representations and attentional states: linking behavior and neurophysiology. Front. Syst. Neurosci. 10, 3 (2016).
Schlag, J. & Schlag-Rey, M. Through the eye, slowly; delays and localization errors in the visual system. Nat. Rev. Neurosci. 3, 191–200 (2002).
Wurtz, R. H. Neuronal mechanisms of visual stability. Vis. Res. 48, 2070–2089 (2008).
Klingenhoefer, S. & Krekelberg, B. Perisaccadic visual perception. J. Vis. 17, 16 (2017).
Dodge, R. Visual perception during eye movement. Psychol. Rev. 7, 454–465 (1900).
Matin, E. Saccadic suppression: a review and an analysis. Psychol. Bull. 81, 899–917 (1974).
Campbell, F. W. & Wurtz, R. H. Saccadic omission: why we do not see a grey-out during a saccadic eye movement. Vis. Res. 18, 1297–1303 (1978).
Morrone, M. C., Ross, J. & Burr, D. Saccadic eye movements cause compression of time as well as space. Nat. Neurosci. 8, 950–954 (2005).
Honda, H. Perceptual localization of visual stimuli flashed during saccades. Percept. Psychophys. 45, 162–74 (1989).
Honda, H. The time courses of visual mislocalization and of extraretinal eye position signals at the time of vertical saccades. Vis. Res 31, 1915–1921 (1991).
Jeffries, S. M., Kusunoki, M., Bisley, J. W., Cohen, I. S. & Goldberg, M. E. Rhesus monkeys mislocalize saccade targets flashed for 100 ms around the time of a saccade. Vis. Res. 47, 1924–1934 (2007).
Van Wetter, S. M. C. I. & Van Opstal, A. J. Experimental test of visuomotor updating models that explain perisaccadic mislocalization. J. Vis. 8, 1–22 (2008).
Ross, J., Morrone, M. C. & Burr, D. C. Compression of visual space before saccades. Nature 386, 598–601 (1997).
Lappe, M., Awater, H. & Krekelberg, B. Postsaccadic visual references generate presaccadic compression of space. Nature 403, 892–895 (2000).
Honda, H. Saccade-contingent displacement of the apparent position of visual stimuli flashed on a dimly illuminated structured background. Vis. Res. 33, 709–716 (1993).
Morrone, M. C., Ross, J. & Burr, D. C. Apparent position of visual targets during real and simulated saccadic eye movements. J. Neurosci. 17, 7941–7953 (1997).
Kaiser, M. & Lappe, M. Perisaccadic mislocalization orthogonal to saccade direction. Neuron 41, 293–300 (2004).
Awater, H. & Lappe, M. Perception of visual space at the time of pro- and anti-saccades. J. Neurophysiol. 91, 2457–2464 (2004).
Krekelberg, B., Kubischik, M., Hoffmann, K.-P. & Bremmer, F. Neural correlates of visual localization and perisaccadic mislocalization. Neuron 37, 537–545 (2003).
Kusunoki, M. & Goldberg, M. E. The time course of perisaccadic receptive field shifts in the lateral intraparietal area of the monkey. J. Neurophysiol. 89, 1519–1527 (2003).
Churan, J., Guitton, D. & Pack, C. C. Perisaccadic remapping and rescaling of visual responses in macaque superior colliculus. PLoS ONE 7, e52195 (2012).
Bremmer, F., Kubischik, M., Hoffmann, K.-P. & Krekelberg, B. Neural dynamics of saccadic suppression. J. Neurosci. 29, 12374–12383 (2009).
Krock, R. M. & Moore, T. Visual sensitivity of frontal eye field neurons during the preparation of saccadic eye movements. J. Neurophysiol. 116, 2882–2891 (2016).
Moore, T. & Chang, M. H. Presaccadic discrimination of receptive field stimuli by area V4 neurons. Vis. Res. 49, 1227–1232 (2009).
Moore, T. Shape representations and visual guidance of saccadic eye movements. Science 285, 1914–1917 (1999).
Steinmetz, N. A. & Moore, T. Changes in the response rate and response variability of area V4 neurons during the preparation of saccadic eye movements. J. Neurophysiol. 103, 1171–1178 (2010).
Nakamura, K. & Colby, C. L. Updating of the visual representation in monkey striate and extrastriate cortex during saccades. Proc. Natl Acad. Sci. USA 99, 4026–4031 (2002).
Sommer, M. A. & Wurtz, R. H. Influence of the thalamus on spatial visual processing in frontal cortex. Nature 444, 374–377 (2006).
Zirnsak, M., Gerhards, R. G. K., Kiani, R., Lappe, M. & Hamker, F. H. Anticipatory saccade target processing and the presaccadic transfer of visual features. J. Neurosci. 31, 17887–17891 (2011).
Hall, N. J. & Colby, C. L. Remapping for visual stability. Philos. Trans. R. Soc. Lond. B Biol. Sci. 366, 528–539 (2011).
Cavanagh, P., Hunt, A. R., Afraz, A. & Rolfs, M. Visual stability based on remapping of attention pointers. Trends Cogn. Sci. 14, 147–153 (2010).
Breitmeyer, B. G., Kropfl, W. & Julesz, B. The existence and role of retinotopic and spatiotopic forms of visual persistence. Acta Psychol. (Amst.) 52, 175–196 (1982).
Blackmore, S. J., Brelstaff, G., Nelson, K. & Trościanko, T. Is the richness of our visual world an illusion? Transsaccadic memory for complex scenes. Perception 24, 1075–1081 (1995).
Irwin, D. E. & Robinson, M. M. Detection of stimulus displacements across saccades is capacity-limited and biased in favor of the saccade target. Front. Syst. Neurosci. 9, 161 (2015).
Deubel, H. & Schneider, W. X. Saccade target selection and object recognition: evidence for a common attentional mechanism. Vis. Res. 36, 1827–1837 (1996).
Peterson, M. S., Kramer, A. F. & Irwin, D. E. Covert shifts of attention precede involuntary eye movements. Percept. Psychophys. 66, 398–405 (2004).
Rolfs, M., Jonikaitis, D., Deubel, H. & Cavanagh, P. Predictive remapping of attention across eye movements. Nat. Neurosci. 14, 252–256 (2011).
Zirnsak, M., Steinmetz, N. A., Noudoost, B., Xu, K. Z. & Moore, T. Visual space is compressed in prefrontal cortex before eye movements. Nature 507, 504–507 (2014).
Neupane, S., Guitton, D. & Pack, C. C. Two distinct types of remapping in primate cortical area V4. Nat. Commun. 7, 10402 (2016).
Neupane, S., Guitton, D. & Pack, C. C. Dissociation of forward and convergent remapping in primate visual cortex. Curr. Biol. 26, R491–R492 (2016).
Neupane, S., Guitton, D. & Pack, C. C. Perisaccadic remapping: what? How? Why? Rev. Neurosci. 31, 505–520 (2020).
Akbarian, A., Clark, K., Noudoost, B. & Nategh, N. A sensory memory to preserve visual representations across eye movements. Nat. Commun. 12, 6449 (2021).
Jeffries, S. M., Kusunoki, M., Bisley, J. W., Cohen, I. S. & Goldberg, M. E. Rhesus monkeys mislocalize saccade targets flashed for 100ms around the time of a saccade. Vis. Res. 47, 1924–1934 (2007).
Hamker, F. H., Zirnsak, M., Ziesche, A. & Lappe, M. Computational models of spatial updating in peri-saccadic perception. Philos. Trans. R. Soc. B: Biol. Sci. 366, 554–571 (2011).
Hamker, F. H., Zirnsak, M., Calow, D. & Lappe, M. The peri-saccadic perception of objects and space. PLoS Comput. Biol. 4, e31 (2008).
Ziesche, A., Bergelt, J., Deubel, H. & Hamker, F. H. Pre- and post-saccadic stimulus timing in saccadic suppression of displacement – a computational model. Vis. Res. 138, 1–11 (2017).
Edwards, G., VanRullen, R. & Cavanagh, P. Decoding trans-saccadic memory. J. Neurosci. 38, 1114–1123 (2018).
Niknam, K. et al. Characterizing and dissociating multiple timevarying modulatory computations influencing neuronal activity. PLoS Comput. Biol. 15, e1007275 (2019).
Li, H.-H., Barbot, A. & Carrasco, M. Saccade preparation reshapes sensory tuning. Curr. Biol. 26, 1564–1570 (2016).
Moore, T., Tolias, A. S. & Schiller, P. H. Visual representations during saccadic eye movements. Proc. Natl Acad. Sci. USA 95, 8981–8984 (1998).
Merrikhi, Y. et al. Dissociable contribution of extrastriate responses to representational enhancement of gaze targets. J. Cogn. Neurosci. 33, 2167–2180 (2021).
Fischer, J., Spotswood, N. & Whitney, D. The emergence of perceived position in the visual system. J. Cogn. Neurosci. 23, 119–136 (2011).
Majima, K., Sukhanov, P., Horikawa, T. & Kamitani, Y. Position information encoded by population activity in hierarchical visual areas. eNeuro 4, ENEURO.0268-16.2017 (2017).
Tolias, A. S. et al. Eye movements modulate visual receptive fields of V4 neurons. Neuron 29, 757–767 (2001).
Qian, N., Goldberg, M. E. & Zhang, M. Tuning curves vs. population responses, and perceptual consequences of receptive-field remapping. Front. Comput. Neurosci. 16, 1060757 (2023).
Mateeff, S. Saccadic eye movements and localization of visual stimuli. Percept. Psychophys. 24, 215–224 (1978).
McConkie, G. W. & Currie, C. B. Visual stability across saccades while viewing complex pictures. J. Exp. Psychol. Hum. Percept. Perform. 22, 563–581 (1996).
Currie, C. B., McConkie, G. W., Carlson-Radvansky, L. A. & Irwin, D. E. The role of the saccade target object in the perception of a visually stable world. Percept. Psychophys. 62, 673–683 (2000).
Pillow, J. W., Paninski, L., Uzzell, V. J., Simoncelli, E. P. & Chichilnisky, E. J. Prediction and decoding of retinal ganglion cell responses with a probabilistic spiking model. J. Neurosci. 25, 11003–11013 (2005).
Pillow, J. W. et al. Spatio-temporal correlations and visual signalling in a complete neuronal population. Nature 454, 995–999 (2008).
Akbarian, A. et al. Developing a nonstationary computational framework with application to modeling dynamic modulations in neural spiking responses. IEEE Trans. Biomed. Eng. 65, 241–253 (2018).
Weng, G., Akbarian, A., Noudoost, B. & Nategh, N. Modeling the relationship between perisaccadic neural responses and location information. in 2022 56th Asilomar Conference on Signals, Systems, and Computers. p. 451–454 (2022).
Weng, G., Clark, K., Akbarian, A., Noudoost, B. & Nategh, N. Time-varying generalized linear models: characterizing and decoding neuronal dynamics in higher visual areas. Front. Comput. Neurosci. 18, 1273053 (2024).
Simoncelli, E., Paninski, J., Pillow, J. & Schwartz, O. Characterization of neural responses with stochastic stimuli. Cogn. Neurosci. 3, 327–338 (2004).
Paninski, L., Pillow, J. W. & Simoncelli, E. P. Maximum likelihood estimation of a stochastic integrate-and-fire neural encoding model. Neural Comput. 16, 2533–2561 (2004).
Acknowledgements
The authors would like to thank the animal care personnel at the University of Utah. We specifically thank Rochelle D. Moore and Dr. Tyler Davis for their assistance with the NHP experiments. This work was supported by NIH EY026924 & NIH NS113073 to B.N.; NIH EY031477 to N.N.; NIH EY014800 and an Unrestricted Grant from Research to Prevent Blindness, New York, NY, to the Department of Ophthalmology & Visual Sciences, University of Utah.
Author information
Authors and Affiliations
Contributions
N.N. and B.N. conceived the study. B.N. performed the surgical procedures. B.N., N.N., and A.A. designed the experiment. B.N., N.N., K.C., and G.W. designed the analysis. B.N. and A.A. performed the physiology experiments and acquired data. A.A. performed the modeling. G.W. performed the data analysis. G.W., K.C., N.N., and B.N. wrote the manuscript.
Corresponding authors
Ethics declarations
Competing interests
The authors declare no competing interests.
Peer review
Peer review information
Nature Communications thanks the anonymous reviewer(s) for their contribution to the peer review of this work. A peer review file is available.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Source data
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.
About this article
Cite this article
Weng, G., Akbarian, A., Clark, K. et al. Neural correlates of perisaccadic visual mislocalization in extrastriate cortex. Nat Commun 15, 6335 (2024). https://doi.org/10.1038/s41467-024-50545-0
Received:
Accepted:
Published:
DOI: https://doi.org/10.1038/s41467-024-50545-0
- Springer Nature Limited