Abstract
Integration across space and time is essential for the analysis of motion, low contrast, and many more stimuli. A crucial question is what determines the duration of integration. Based on classical models of decision-making, one might expect that integration terminates as soon as sufficient evidence about a stimulus is accumulated and a threshold is crossed. However, there is very little research on this question as most experimental paradigms cannot monitor processing following stimulus presentation. In particular, it is difficult to determine when processing terminates. Here, using the sequential metacontrast paradigm (SQM), in which information is mandatorily integrated along motion trajectories, we show that the processing load determines the extent of integration but that evidence accumulation does not. Further, the extent of integration is determined by absolute time instead of the number of elements presented. These results have important implications for understanding the time course and mechanisms of temporal integration.
Similar content being viewed by others
Introduction
Long-lasting visual feature integration is crucial for detecting and integrating motion signals as well as for solving many of the ill-posed problems of vision. However, very little research has attempted to closely characterize these processes as most experimental paradigms do not allow to control the time course of temporal integration. Once a stimulus is presented, the experimenter has little control over the subsequent neural processes. In particular, it is difficult to determine when stimulus processing terminates. The sequential metacontrast paradigm (SQM), to the contrary, has turned out to be a versatile tool for examining the dynamics underlying temporal integration.
In the SQM1, participants are presented with a central line followed by pairs of flanking lines, eliciting the percept of two motion streams diverging from the center (see Fig. 1 for illustration). Due to metacontrast masking, the central line is rendered invisible. If the line is offset (i.e., the lower segment is shifted either towards the right or left, relative to the upper segment; here referred to as ‘vernier’ offset or ‘V’), participants perceive the subsequent flanking lines as offset, even though they are, in fact, straight. If, in addition to the central line, one of the flanking lines is offset, the two offsets integrate. If both offsets are in the same direction ('provernier’ or ‘PV’), the offset is perceived at even higher performance levels. If, instead, they are in opposite directions (‘antivernier’ or ‘AV’), the offsets cancel each other out before reaching consciousness, even if vernier and anti-vernier are separated by up to 450 ms2. Observers are not able to report or perceive either vernier offset separately. Hence, integration in this paradigm is mandatory.
Here, we asked what terminates stimulus processing and temporal integration. To this end, we conducted three experiments with the SQM, as illustrated in Fig. 2, and compared the resulting integration windows across different experimental conditions.
Methods
Participants
Naïve participants were recruited from the École Polytechnique Fédérale de Lausanne and the University of Lausanne in Switzerland. A total of 11 participants successfully completed experiment 1 (3 females; age range: 19–24 years), 11 (8 new, 3 old) participants successfully completed experiment 2 (2 females; age range: 19–26 years), and 15 (14 new, 1 old) participants successfully completed experiment 3 (5 females; age range: 18-26 years) (sex/gender was not self-identified but determined by the experimenter). Prior to the experiment, the Freiburg Visual Acuity test3 was administered, and only those who reached a score of at least 1.0 binocularly (corresponding to normal or corrected-to-normal vision) were admitted to the main experiment. Participants signed an informed consent form in advance of the study and received monetary compensation (20 Swiss francs per hour) upon its completion. The experiments and procedures took place in accordance with the Declaration of Helsinki, except for preregistration, and were approved by the local ethics committee (Commission cantonale d’ethique de la recherche sur l’etre humain) of canton Vaud in Switzerland. The experiments and analyses were not preregistered.
Apparatus
Stimuli were displayed on a BenQ XL2540 LCD monitor with a resolution of 1920 × 1080 pixels, a screen size of 24.5”, and a refresh rate of 240 Hz. Stimuli were generated using MATLAB R2013a (MathWorks Inc., Natick, MA, USA) and Psychtoolbox4. Participants were seated 2.5 meters away from the screen, in a dimly lit room. Stimuli were white (100 cd/m²), presented on a black background with a luminance of 1 cd/m².
Stimuli
Following a fixation dot presented for 0.5 s, and a subsequent blank screen lasting for 0.5 s, the standard SQM stimulus sequence commenced with the presentation of a central line consisting of an upper and a lower segment, each of 26.5’ (arcmin) length and vertically separated by a gap of 2.3’. This central line was then followed by pairs of flanking lines—of the same length and vertical separation as the central line—appearing further and further away from the center. The individual lines had a width of 1.2’ and were horizontally separated by 3.5’. Each line was presented for 20.8 ms. The interstimulus interval (ISI) between the central and first flanking line was 29.2 ms, and the ISI between each of the subsequent flanking lines was 20.8 ms. The stimulus sequence comprised a total of one central and 14 flanking lines with the last flanking line being presented at 590.8 ms. With this stimulus sequence, two motion streams are perceived as diverging from the center towards the periphery.
Participants were asked to covertly attend to one of the streams (in the experiments reported here, the right stream). In the standard SQM configuration, the central line presented at frame 0 was offset (i.e., the lower segment of the line was shifted towards the right or the left, relative to the upper segment; referred to as ‘vernier’ offset or ‘V’), and one of the later lines in the attended stream (whose position was randomly determined to be between the 7th and 13th frame) was offset in the opposite direction (referred to as ‘anti-vernier’ or ‘AV’). Observers perceive only one integrated offset, independent of the number of offsets presented, i.e., observers cannot respond to a single offset and offsets integrate mandatorily2. The observers‘ task was to indicate whether they perceived a left or a right offset. Following the completion of the stimulus stream presentation, participants had 3 s to respond by clicking one of two hand-held buttons. Auditory feedback about the correctness of the response was provided in the calibration phase (see the section on calibration below) but not during the main experiment. The inter-trial interval was 0.5 s. The different stimulus configurations used in the three experiments reported in this paper are illustrated in Fig. 2.
In experiment 1, the standard condition is identical to the stimulus configuration introduced above. In the split-vernier condition, three verniers offset in the same direction were presented at frames 0, 3, and 6. The vernier offset sizes at positions 0, 3, and 6 were adjusted for each observer so that performance was equal to the single vernier in the baseline condition (see section on offset calibration).
In the high-load condition of experiment 2, an additional anti-vernier offset was presented at frame 4, and an additional vernier offset was presented at frame 5. This additional pair of anti-vernier and subsequent vernier cancels itself out, i.e., performance was aimed to be similar to the standard condition, but was added to increase the overall processing load.
In experiment 3, the standard condition is identical to the standard condition introduced above, except that a total of 15, instead of 14, pairs of flanking lines were presented and that positions 8 to 13 were sampled for the placement of the anti-vernier (instead of positions 7 to 13). In the short-ISI condition, the ISI was reduced from 20.8 ms to 4.2 ms (but kept at 29.2 ms for the ISI between the central and first flanking line, to keep the metacontrast masking) and the individual lines were horizontally separated by 140, instead of 210, arcsec. The number of flanking lines was increased to 24, and the anti-vernier was randomly presented at positions between 13 and 21.
Offset size calibration
As detailed above, in the standard SQM condition, a vernier offset is presented at the central line and an anti-vernier offset presented at a later flanking line. To ensure that the individual contributions of these two offsets are perceptually comparable, their offset sizes (i.e., the extent of the horizontal displacement between their top and bottom segment) were individually calibrated, prior to the start of the main experiment. To this end, sequences with a single offset at a given position were presented, and a parameter estimation by sequential testing (PEST) procedure5 was used to adaptively determine the offset sizes required to yield around 70% to 80% performance. Given inter-individual differences, this process was carried out for each participant individually. Results of pilot studies suggested that the offset sizes, which are required to reach a certain level of performance, are comparable across the later flanking positions at which anti-vernier offsets would appear (i.e., positions 7-13 in the standard SQM condition). Thus, not all of these positions were calibrated individually. Instead, the offset at frame 10 (i.e., the middle of 7-13) was calibrated and used as a proxy for all later offset positions (i.e., positions 7-13 in the standard condition).
In experiment 1, in the standard condition, offset sizes were calibrated for frames 0 and 10, respectively. In the split-vernier condition, instead of calibrating the offset size at frame 0, the offset sizes were calibrated simultaneously for frames 0, 3, and 6, to ensure that the three offsets appearing together would have the same perceptual strength as the single offset in the standard condition. In both conditions of experiment 2, offset sizes were calibrated for frames 0 and 10, respectively. The offset sizes for the additional pair of anti-vernier and vernier offsets, appearing at positions 4 and 5, were determined by taking the mean of the two calibrated offset sizes for positions 0 and 10. In the standard condition of experiment 3, similar to experiments 1 and 2, frames 0 and 10 were calibrated, respectively. In the short-ISI condition of experiment 3, frames 0 and 17 were calibrated, respectively, with frame 17 chosen to be in the middle of the to be sampled anti-vernier range from position 13 to 21. This range was defined in order to cover temporal coordinates similar to those in the standard condition.
Means and standard deviations of the calibrated offset values across individual observers can be found in Table 1.
Experimental procedure
In experiments 1 and 2, participants completed 8 blocks of 84 trials each, for each of the two conditions. In each block, the position at which the anti-vernier in the stimulus sequence appeared was randomly determined in the given range of positions (frames 7–13), while ensuring that each position appeared equally often (here, 12 times each). In experiment 3, participants completed 6 blocks of 84 trials each in the normal-ISI condition (in which across 6 different AV positions were sampled) and 9 blocks of 81 trials each in the short-ISI condition (in which across 9 different AV positions were sampled), to ensure that each position in each condition was repeated at least 80 times. The randomization was applied in order to reduce the potential impact of effects such as fatigue on our results.
Data analysis
For each experiment and condition, we extracted the anti-vernier dominance level (i.e., the fraction of responses that were in accordance with the direction of the anti-vernier offset) as a function of the position at which the anti-vernier appeared. If the anti-vernier is presented early and the vernier and anti-vernier offsets fall into the same integration window, the anti-vernier dominance would be expected to be close to 50%1,2. As the anti-vernier is presented at later and later positions and starting to fall into the next integration window, anti-vernier dominance is expected to increase, finally converging to the calibration level (approximately 75%). If an experimental manipulation were to prolong the duration of the integration window, the slope of the anti-vernier dominance as a function of anti-vernier position would be expected to be less steep. To compare these differences in slope across two experimental conditions, the two resulting data were normalized with regard to the dominance level observed at the earliest measured anti-vernier position (typically, position 7), where full integration occurred2.
Note that trials in which reaction times were below 800 ms (i.e., approx. 200 ms following the end of the motion stream) were rejected, as they are likely to have occurred before the end of the stimulus sequence was fully perceived. This procedure did not change the results strongly. The data distribution was assumed to be normal but this was not formally tested. Data analysis was carried out using MATLAB R2022b (MathWorks Inc., Natick, MA, USA).
For the Bayes factor analysis reported in experiment 1 (for MATLAB package, see6), the default prior (Cauchy width = 0.707) was chosen, revealing a moderate Bayes factor of 0.298. This result generalizes well across priors: a wide prior (Cauchy width = 1) would yield a moderate Bayes factor of 0.223 and an ultrawide prior (Cauchy width = 1.414) a moderate Bayes factor of 0.164. This analysis of prior dependence was carried out using JASP (version 0.17.2.1).
Power analysis
The SQM1 has a large effect size, with a Cohen’s d of usually around 1.5 or greater2,7. With this effect size, a power analysis, computed with the G*Power software, revealed that a modest sample size of less than 10 participants would be required for achieving a power of 90%. To be on the safe side, we here recruited more than 10 participants for each experiment.
Reporting summary
Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.
Results
Window duration is not dependent on the timing of stimulus evidence presentation
In experiment 1, either a large offset at the central line (standard condition) or 3 smaller offsets at lines 0, 3, and 6, all in the same direction (split-vernier condition), were presented (see Fig. 2 for illustration). Classical models of decision-making, in which sensory evidence is integrated until a decision threshold is reached and a motor response elicited, would predict that integration windows in the split-vernier condition may be longer, considering that evidence about the offset to be reported is presented later, as the vernier offset information is dispersed along the motion stream instead of being available right from the beginning. As can be seen in Fig. 3, if any, the increase in anti-vernier dominance would rather show the opposite effect. We fitted regression lines (fitting only the slope; keeping the intercept constant, so as to enforce an AV dominance increase of 0 at the first AV position) to the individual participants’ data, separately for both conditions, and found moderate evidence in favor of an absence of the effect (Bayes factor = 0.298, median of effect size = 0.015, 95% CI of effect size = [-0.512, 0.544] in a two-tailed paired-sample comparison of slopes; n = 11). Hence, evidence dispersed along the motion stream, rather than presented right from the beginning, does not prolong integration.
Higher processing load extends window duration
Relative to the standard condition, two further offsets at two additional lines were added in the high-load condition, which are expected to consciously cancel each other out but increase the overall unconscious processing load, due to there being more elements to be processed. As depicted in Fig. 4, the integration profiles across the two conditions differed significantly in their slope (t(10) = 3.10, p = .011, Cohen’s d = 1.250, 95% CI = [0.633, 3.860] in two-tailed paired-sample t-test for individual participants’ lines of best fits for the two conditions; n = 11), indicating that higher processing load extends the duration of temporal integration windows.
Integration windows depend on absolute time, rather than on the number of lines
In experiment 3, the inter-stimulus interval was reduced from 20.8 ms to 4.2 ms, leading to the presentation of more lines in the same amount of time. If, in the standard SQM setting, integration lasts for around 10 frames, corresponding to 425 ms, will integration in the short-ISI condition last for around 10 frames, corresponding to 275 ms, or will it last for around 16 frames, corresponding to 425 ms? In other words, is the extent of integration, as measured in the SQM, a temporal measure or does it depend on the number of discrete elements presented?
The left panel of Fig. 5 depicts the two conditions’ integration profiles when they are aligned with regard to the time at which the anti-vernier is presented. In contrast, the right panel of Fig. 5 depicts the integration profiles when they are aligned with regard to the position at which the anti-vernier is presented. The two curves depicted in temporal coordinates are significantly more aligned than those depicted in positional coordinates (t(14) = 2.98, p = 0.010, Cohen’s d = 1.07, 95% CI = [4.233, 26.098] in a two-tailed paired-sample t-test comparing differences between unnormalized AV dominance at position 13 (presented at 350 ms) of the short-ISI condition [mean = 58.9%] with the temporal (position 8, at ca. 350 ms) vs. the positional (position 13) equivalent in the standard condition [means = 57.2% and 72.3%, respectively]; n = 15). Thus, the absolute time, rather than the number of lines, determines window duration.
Discussion
Substantially-delayed conscious perception
Conscious perception is substantially delayed, preceded by extended periods of unconscious processing1,2,9,10 (but see discussion11,12,13). These periods are crucial, for example, for detecting and integrating motion signals, solving many of the ill-posed problems of vision, and integrating information in low-contrast scenarios. In this paper, we examined how long these periods of temporal integration last and what factors determine them. The SQM has turned out to be a versatile tool for this purpose as its mandatory integration allows measuring how information at specific time points affects integration1,2,7,14,15,16,17,18.
Processing load, and not stimulus evidence, determines integration duration
Here, we have shown that the duration of unconscious feature integration is not significantly affected by the evidence presented. Specifically, the duration of the integration window did not differ depending on whether strong evidence was presented right from the beginning or was dispersed along the motion stream. This result is surprising considering that in classical models of decision making, the stronger the evidence, the quicker a decision is reached and a motor response elicited19. We would have expected a similar result for the duration of a window of integration. In this respect, our results suggest that classical decision-making and the duration of integration are only weakly linked, subserving potentially distinct neural processes (see also the work of Rüter and colleagues20,21).
Instead of evidence-based integration termination, our second experiment revealed that processing load determines integration duration. When adding additional vernier and anti-vernier offsets that canceled each other out but increased the overall processing load, integration lasted longer. This result, we propose, can be accounted for by the brain continuously monitoring, and controlling for, the difficulty of tasks. This appears intuitively plausible: in a challenging perceptual scenario, such as a low-contrast situation, one benefits from extended integration windows. In our experiment specifically, the repetitive and block-wise presentation of SQM stimulus streams may have also helped to form expectations prior to the onset of a given trial about task difficulty. We believe that this proposal opens a new avenue for future research, not just focused on the brain’s processing but also on the monitoring of its own processing.
Integration windows depend on absolute time, not on the number of lines
Further, as revealed in our third experiment, it is the absolute time (i.e., a temporal measure quantifiable in milliseconds), and not the number of lines (i.e., a measure of the number of discrete elements in the motion stream), that determines the extent of unconscious feature integration. Interestingly, this finding differs from those that have been reported in some other paradigms: for instance, studies of semantic monitoring of rapidly-presented word lists have revealed that recovery from shift costs (i.e., recovery from performance drops following a cue to shift attention) occurs as a function of the number of presented items, rather than the absolute time, following the shift cue22,23.
Minimum integration periods
While our results reveal that an increase in the processing load extends the integration time, it is important to note that there appears to be a minimum integration period with the SQM paradigm, longer than around 350 ms. These long periods even exist when the stimuli themselves are presented for much shorter durations18. Interestingly, the long periods of unconscious integration observed in the SQM mirror results reported in other paradigms. For instance, it was shown that integration in RSVP paradigms can last longer than 200 ms24, and that the presentation of a cue that appeared up to 400 ms following a visual stimulus changed the perception of the latter25,26. Further, it is important to note that even if experiments reveal short effects of integration, conscious visual perception may nevertheless occur at a later point in time.
Limitations
While we here have examined the role of stimulus evidence, processing load, and the number of elements vs. absolute time on the duration of unconscious visual integration, there are many more relevant features that will need to be examined in future work to gain a more wholesome understanding of the dynamics underlying temporal integration mechanisms.
Conclusion
Taken together, based on the experiments with the SQM that we report in this paper, we propose that the extent of discrete windows of unconscious feature integration depends mainly on the processing load, not on the stimulus evidence, and can be measured in absolute time rather than the number of elements presented in a motion stream.
Data availability
The data underlying this study is available at https://doi.org/10.5281/zenodo.81138558. Please contact the lead author (Lukas Vogelsang) in case of any questions as well as for additional data.
Code availability
The analysis code is available at https://doi.org/10.5281/zenodo.81138558.
References
Otto, T. U., Ögmen, H. & Herzog, M. H. The flight path of the phoenix—The visible trace of invisible elements in human vision. J. Vis. 6, 7 (2006).
Drissi-Daoudi, L., Doerig, A. & Herzog, M. H. Feature integration within discrete time windows. Nat. Commun. 10, 1–8 (2019).
Bach, M. The Freiburg Visual Acuity Test-automatic measurement of visual acuity. Optometry Vis. Sci. 73, 49–53 (1996).
Brainard, D. H. & Vision, S. The psychophysics toolbox. Spatial Vis. 10, 433–436 (1997).
Taylor, M. & Creelman, C. D. PEST: Efficient estimates on probability functions. J. Acoust. Soc. Am. 41, 782–787 (1967).
Krekelberg, B. BayesFactor: Release 2022 (v2.3.0) (v2.3.0). Zenodo. (2022). https://doi.org/10.5281/zenodo.7006300.
Drissi-Daoudi, L., Öğmen, H., Herzog, M. H. & Cicchini, G. M. Object identity determines trans-saccadic integration. J. Vis. 20, 33 (2020).
Vogelsang, L., Drissi-Daoudi, L. & Herzog, M. (2023). Processing load, and not stimulus evidence, determines the duration of unconscious visual feature integration [Data set]. Zenodo. https://doi.org/10.5281/zenodo.8113855.
Herzog, M. H., Drissi-Daoudi, L. & Doerig, A. All in good time: long-lasting postdictive effects reveal discrete perception. Trends Cogn. Sci. 24, 826–837 (2020).
Herzog, M. H., Kammer, T. & Scharnowski, F. Time slices: what is the duration of a percept? PLoS Biol. 14, e1002433 (2016).
Hogendoorn, H. Perception in real-time: predicting the present, reconstructing the past. Trends Cogn. Sci. 26, 129 (2022).
Fekete, T., Van de Cruys, S., Ekroll, V. & van Leeuwen, C. In the interest of saving time: a critique of discrete perception. Neurosci. Consciousness 2018, niy003 (2018).
Doerig, A., Scharnowski, F. & Herzog, M. H. Building perception block by block: a response to Fekete et al. Neurosci. Consciousness 2019, niy012 (2019).
Otto, T. U., Ögmen, H. & Herzog, M. H. Feature integration across space, time, and orientation. J. Exp. Psychol.: Hum. Percept. Perform. 35, 1670 (2009).
Otto, T. U., Öğmen, H. & Herzog, M. H. Attention and non-retinotopic feature integration. J. Vis. 10, 8 (2010).
Drissi-Daoudi, L., Öğmen, H. & Herzog, M. H. Features integrate along a motion trajectory when object integrity is preserved. J. Vis. 21, 4 (2021).
Plomp, G., Mercier, M. R., Otto, T. U., Blanke, O. & Herzog, M. H. Non-retinotopic feature integration decreases response-locked brain activity as revealed by electrical neuroimaging. Neuroimage 48, 405–414 (2009).
Scharnowski, F. et al. Long-lasting modulation of feature integration by transcranial magnetic stimulation. J. Vis. 9, 1 (2009).
Ratcliff, R. A theory of memory retrieval. Psychol. Rev. 85, 59 (1978).
Rüter, J., Marcille, N., Sprekeler, H., Gerstner, W. & Herzog, M. H. Paradoxical evidence integration in rapid decision processes. PLoS Comput. Biol. 8, e1002382 (2012).
Rüter, J., Sprekeler, H., Gerstner, W. & Herzog, M. H. The silent period of evidence integration in fast decision making. PLoS ONE 8, e46525 (2013).
Alport, A., Styles, E. A. & Hsieh, S. Shifting intentional set: exploring the dynamic control of tasks. Attention and performance XV: Conscious and nonconscious information processing 15, 421 (1994).
Hsieh, S. & Allport, A. Shifting attention in a rapid visual search paradigm. Percept. Motor Skills 79, 315–335 (1994).
Akyürek, E. G. & Wolff, M. J. Extended temporal integration in rapid serial visual presentation: attentional control at Lag 1 and beyond. Acta Psychol. 168, 50–64 (2016).
Sergent, C. et al. Cueing attention after the stimulus is gone can retrospectively trigger conscious perception. Curr. Biol. 23, 150–155 (2013).
Thibault, L., Van den Berg, R., Cavanagh, P. & Sergent, C. Retrospective attention gates discrete conscious access to past sensory stimuli. PLoS ONE 11, e0148504 (2016).
Acknowledgements
We wish to thank Marc Repnow for his excellent technical support. This work was supported by the Swiss National Science Foundation grant ‘Basics of visual processing: the first half second’ (325130_204898). The funders had no role in study design, data collection, and analysis, the decision to publish or preparation of the manuscript.
Author information
Authors and Affiliations
Contributions
L.V., L.D.D., and M.H.H. designed the research; L.V. performed the experiments and analyzed the data; L.V., L.D.D., and M.H.H. wrote the paper.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Peer review
Peer review information
Communications Psychology thanks Pascal Mamassian and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Primary Handling Editors: Antonia Eisenkoeck, Marike Schiffer. A peer review file is available.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary information
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Vogelsang, L., Drissi-Daoudi, L. & Herzog, M.H. Processing load, and not stimulus evidence, determines the duration of unconscious visual feature integration. Commun Psychol 1, 8 (2023). https://doi.org/10.1038/s44271-023-00011-2
Received:
Accepted:
Published:
DOI: https://doi.org/10.1038/s44271-023-00011-2
- Springer Nature Limited