Abstract
Haptic technologies aim to simulate tactile or kinesthetic interactions with a physical or virtual environment in order to enhance user experience and/or performance. However, due to stringent communication and computational needs, the user experience is influenced by delayed haptic feedback. While delayed feedback is well understood in the visual and auditory modalities, little research has systematically examined the neural correlates associated with delayed haptic feedback. In this paper, we used electroencephalography (EEG) to study sensory and cognitive neural correlates caused by haptic delay during passive and active tasks performed using a haptic device and a computer screen. Results revealed that theta power oscillation was significantly higher at the midfrontal cortex under the presence of haptic delay. Sensory correlates represented by beta rebound were found to be similar in the passive task and different in the active task under the delayed and synchronous conditions. Additionally, the event related potential (ERP) P200 component is modulated under the haptic delay condition during the passive task. The P200 amplitude significantly reduced in the last 20% of trials during the passive task and in the absence of haptic delay. Results suggest that haptic delay could be associated with increased cognitive control processes including multi-sensory divided attention followed by conflict detection and resolution with an earlier detection during the active task. Additionally, haptic delay tends to generate greater perceptual attention that does not significantly decay across trials during the passive task.
Similar content being viewed by others
Introduction
Many systems that involve human–machine interaction, such as teleoperation or virtual reality, are incorporating haptic information to enhance perception and manipulation of the environment1. One perceptual attribute that provides an essential basis for haptic-visual integration is haptic delay due to stringent computational and communication needs associated with haptic data2. Haptic delay can be defined as the temporal difference between the actual haptic feedback and the expected one. Delayed haptic feedback can seriously disrupt many aspects of the interaction, such as the completion time of manipulation tasks3, quality of teleoperation4, and the perception of physical properties such as stiffness and friction5.
The perceptual consequences of haptic delay have received significant attention through psychophysical studies. Detection thresholds for haptic delays may vary substantially, between 20 and 200 ms, based on the application and the type of haptic interaction (force feedback versus tactile, discrete versus continuous force feedback, and active versus passive interaction)6. A haptic-visual study showed that a discrete haptic feedback is noticed as delayed if the delay exceeded 110 ms6. In a collaborative virtual environment where haptic information is bi-directionally communicated, haptic feedback delay could be perceived starting from around 50 ms7. It is also reported that humans do not perceive delays below 30 ms during continuous haptic interaction8. Understanding the experience of perceiving haptic delay is an essential consideration for haptic technology designers to optimize the realism of the haptic experience.
Neurohaptics is an emerging field that strives to understand the complex neural representations provoked in response to touch stimulation9. Neural imaging techniques such as fMRI and EEG offer the potential to examine brain activities associated with haptic delay to provide objective, real-time assessment of the haptic delay9,10. Compared to other neural imaging techniques such as fMRI, EEG is preferable due to the compatibility with electric devices, relatively low cost, and the ability to measure brain responses with high temporal resolution9. Previous studies utilized EEG to examine brain correlates associated with unexpected or mismatched visual stimulation. For instance, an event-related potential (ERP) study found that a negative potential around 200 ms (N200) is pronounced after seeing a visual stimulus that was not expected11. Furthermore, in an object selection task in a virtual environment, it was found that the prediction error negativity component of the ERP signal was more pronounced when the user’s hand had unrealistic representation12.
Converging EEG studies have unraveled the functional roles that the central and midfrontal areas, and in particular the theta frequency band, play in conflict analysis at the stimulus/response level13,14, at the semantic/cognitive level15,16 and during cross-modal prediction error processing17. An early study found that theta oscillation at the midfrontal region is closely related to cognitive control processes that are needed to evaluate the stream of information from perceived stimuli and prepare the brain’s response accordingly18. These processes include multisensory divided attention19, conflict detection and resolution and selective suppression20. In multimodal interaction such as audiovisual, increased central theta power was observed following incongruent audiovisual stimuli compared with congruent audiovisual stimuli21.
Studies on the haptic modality are scarce. A few studies reported that theta synchronization at the midfrontal cortex reflects conflict monitoring and resolution to visual stimuli involving a motor response20,22 or an initiated motor movement23. Theta oscillation was reported to be more pronounced under incongruent cross-modal stimulation as compared to congruent stimulation under visuotactile matching paradigm using Braille stimulator and a computer screen24. In a similar study, visuotactile congruency was tested using two LEDs (visual stimuli) and two vibrotactile motors (tactile stimuli) placed on the thumb and the forefinger; incongruent stimulation induced a significantly greater theta band activity during 300–500 ms after stimulation25.
This paper focuses on examining neural correlates associated with delayed force feedback during active and passive tasks. A delay value of 220 ms has been selected for this experiment based on the previously mentioned literature6,26; an easily recognizable delay is needed to clearly identify the neural correlates. This decision is complimented by a pilot study that was conducted to make sure the delay is clearly perceivable by the majority of participants (beyond 90% recognition accuracy). ERP and event-related spectral perturbation (ERSP) in the theta band are examined in the central and midfrontal areas, respectively. We also study the impact of delay on the sensory correlates at the contralateral sensorimotor cortex. Three hypotheses are examined: (1) haptic delay elicits theta oscillation that is related to multi-sensory divided attention and conflict-resolution processes, (2) haptic delay has no impact on the sensory correlates, represented by beta rebound, and (3) a learning effect due to the repetitive exposure to the haptic stimulus could be observed through ERP components related to attentional resources devoted to the haptic feedback.
Methods
Participants
Nineteen subjects participated in the experiment (ten females, nine males), where 17 of them are aged between 18 and 25 years, one is aged between 25 and 30 years, and one is aged between 30 and 40 years. All participants were right-handed and performed the experiment with their right hand. Also, participants had either a normal vision or corrected-to-normal vision. Around 74% of the participants did not use a haptic device before. Each participant received a compensation voucher worth 30 dollars (USD) for participation. The sample size of the study was not calculated priorly through statistical power analysis. Instead, sample size selection was based on previous EEG studies that investigated sensorimotor processes (Savoie et al.27, N = 15; Tan et al.28, N = 17; Torrecillos et al.29, N = 15; Perfetti et al.30, N = 17; Lin et al.31, N = 19). The study was carried out with an approved protocol by New York University Abu Dhabi Institutional Review Board (IRB: #HRPP-2019-120) and in accordance with the Declaration of Helsinki, following its guidelines and regulations. Written informed consent was obtained from all participants in accordance with the IRB ethics before enrolling in this study.
Experimental setup and task
Participants were seated on a chair around one meter in front of a computer screen and were asked to hold the stylus of a haptic device (Geomagic Touch, 3D systems, United States) with their right hand. Participants were told they would participate in a haptic-visual task in which the goal is to bounce a tennis ball using a racket that is controlled by the haptic device. The game was developed using Unity game engine version 2018.4.5f1 (Unity technologies, United States) and Openhaptics Unity toolkit (3D Systems, United States). Participants had to perform passive and active tasks under synchronous visual and haptic stimulation or asynchronous stimulation such that the haptic feedback is delayed by 220 ms. During the active task, participants moved the racket up to bounce the ball, which is initially held stationary above the racket. However, during the passive task, the racket is held passively, and a thumb button-press on the haptic device initiated a free-fall of the tennis ball to collide with the racket. Force feedback was felt upon the ball collision with the racket. The haptic collision was either provided synchronously with the visual collision or 220 ms delayed with respect to the visual collision. Participants were trained on using the haptic device with minimal body movement to avoid excessive motor EEG artifacts. During the active task, participants were asked to move their wrists up or down while the elbow and forearm are rested without movement. Figure 1 shows one of the participants correctly holding the stylus of the haptic device along with the experimental setup.
A single trial consisted of either a 1.5 or 2.5 s rest period (randomized) during which a blank screen was presented, followed by a single bouncing move; Fig. 2 shows the sequence of events in a single trial for both the passive and the active tasks. The experimental session consisted of ten runs, each of which had 20 trials. Five runs were conducted under the passive mode, while the other five runs were conducted under the active mode. Ten trials had delayed haptic feedback within a single run while the other ten had synchronous haptic feedback; trials were sequenced randomly. Thus, in total, each participant performed 200 trials equally divided between the four experimental conditions (passive/active, with/without haptic delay). The experimental runs’ order was counterbalanced across participants to avoid the order effect between the passive and active tasks. The physics of the experiment, including the ball’s weight (60 g) and its bounciness, were kept as usual and expected as possible. This is to provide a realistic haptic experience to the participant. The amplitude and duration of the force feedback were held to constant values, 0.6 N and 60 ms, respectively, in all trials to provide a fair comparison between the different conditions. To make sure participants are well-engaged and to have a general subjective measure of their performance in recognizing the synchronous from asynchronous trials, they were asked to answer if the haptic feedback was delayed or not in 30 trials out of 200 using a keypad. Participants were asked to fill a post-experiment questionnaire to capture their perceptual experience.
EEG data acquisition
EEG data were recorded at a 1 kHz sampling rate using an EEG amplifier (BrainAmps Standard, Brain Products, Germany) and the Brain Vision Recorder software (BVR; Version 1.21.0201 Brain Products, Germany). The EEG cap had a 64 Ag/AgCL based electrode set (actiCAP snap, Brain Products, Germany), that was placed according to the 10–20 international system such that Cz electrode is placed over the vertex of the participant’s head. The online reference electrode was positioned at the FCz location, while the ground electrode was positioned at the FPz location. Input impedance on each electrode was kept below 15 k\(\Omega\) to ensure high-quality signal recordings.
EEG data pre-processing
The EEG data were pre-processed and analyzed offline using MATLAB release 2019b (MathWorks, United States) and EEGLAB toolbox (v14.1.2)32. Four channels that are located at the sides of the head (FT9, FT10, TP9, and TP10) were excluded from the analysis. The data were first filtered between 0.1 and 35 Hz using a zero-phase Hamming windowed sinc FIR filter. The artifact subspace reconstruction (ASR)33 method was applied to remove high-amplitude artifacts, including eye blinks, muscle bursts, and physical movement, as well as to reject bad channels and correct them by channel interpolation. Channels were then re-referenced using the Common Average Referencing (CAR) method34 while restoring the online reference channel, FCz, to the dataset.
Due to the asynchronous nature of the experimental task and since the initiation of the task is participant-dependent (pressing the button or moving the racket), the EEG data were double epoched. The first epoching was time-locked to the event of visuals appearance on the screen. The time between the appearance of visuals and the visual collision (\(\Delta\)) was calculated for all trials, and its distribution was plotted. Trials that have \(\Delta\) more than four standard deviations away from the mean were discarded (passive: \(\Delta = 1105 + 4\times (744)\) ms; active: \(\Delta = 1139 + 4\times (928)\) ms); this is to maintain a uniform behavioral trend in the considered trials. The data were then epoched from \(-1\) to 6 s around the visuals’ appearance event. The second epoching is performed time-locked to the haptic feedback event, which is the event of interest, with an epoch period between \(-1\) and 2 s around the onset of the haptic feedback. Since the rest period is not exactly prior or close to the collision event, the rest period was not used for baseline correction (ERP) or baseline normalization (ERSPs). Additionally, and since we introduce a delay in the haptic feedback in half of the trials, the period preceding the haptic feedback cannot be used as a baseline either. This choice will lead to inconsistent baselines across conditions. Instead, the baseline was selected to be the period that exactly precedes the visual collision event. This choice will guarantee that the baseline is close enough to the onset (haptic collision) while being consistent across all conditions (the state of mind before the collision occurrence). Thus, the data were baseline corrected to the average potential recorded during the 100 ms preceding the visual collision onset (ERP) and baseline-normalized with respect to the 200 ms data that preceded the visual collision onset (ERSPs). Lastly, the infomax algorithm was applied to implement independent component analysis (ICA)35, a blind source separation technique that separates signal components by maximizing the statistical independence between them. A component was marked as artifactual if: (1) the time-course of the component showed bogus bursts of activity, and (2) the power spectrum of the component was found to increase with frequency; EEG spectral power is expected to decrease with frequency36 as opposed to muscle artifacts’ spectral power which increases with frequency37, or (3) the topography of the component was localized at the far edges of the scalp as these could either be eye or muscle artifacts. The clean components were then reflected back to the channel space.
Electrodes, bands and time-windows selection
For the EEG data analysis, the selection procedure of electrodes, frequency bands, and time windows are based on available literature. As per our hypothesis (1) stated earlier in the introduction, theta power at the midfrontal area plays an essential functional role in multisensory attention and conflict processing19,20. Therefore, EEG data were examined at the midfrontal region of interest (ROI) comprising the following electrodes: AFz, F1, Fz, F2, FC1, FCz, FC2. This selection is based on previous investigations of the midfrontal theta oscillations, where chosen electrodes are usually centered around Fz electrode19,20. Within the same midfrontal ROI, the ERP component of P200 is reported to be modulated in response to delayed auditory feedback38, visuo-haptic mismatch in virtual reality39 and in conflict resolution studies20. We report the P200 result for FCz electrode, partly because it was the electrode that showed the maximum P200 modulation with the introduction of haptic delay and partly because the previously mentioned EEG studies on delayed feedback and conflict processing focus mainly on this electrode. It was decided to consider the time window between 140 and 280 ms from the haptic feedback onset since it is reported that the P200 peak is observed within this time window40,41,42.
In addition to the midfrontal ROI, EEG data were examined at the contralateral sensorimotor ROI to observe sensory correlates, and the impact of haptic delay on them compared to the cognitive correlates at the midfrontal ROI. Particularly, sensorimotor tasks invoke a drop in beta power during the movement of the effector followed by a post-movement beta rebound43 observed over the sensorimotor area. We selected FC1, C1, C3, CP3 electrodes as they overlie the sensorimotor area and are known to best capture the post-movement beta rebound44,45.
EEG data analysis
For ERP analysis, epochs of the same condition were averaged out across all subjects. For time-frequency analysis, epoched EEG data were first spatially filtered; current source density (CSD) transformation was calculated using the surface Laplacian transform (m-constant: 4, head radius: 10 cm, smoothing constant: 10\(^{-5}\)) implemented as part of the CSD toolbox in Matlab (version 1.1)46. There is strong evidence that applying CSD transformation to the EEG activity improves the spatio-temporal features and resolution of the EEG dynamics. Additionally, applying CSD following ICA-based artifact rejection has been shown effective in eliminating undesired muscle-related, and electromyographic activity47,48. The second step was to apply Morlet Wavelet transformation49 with a cycles range logarithmically increasing from 4 to 20 to maintain better frequency resolution at higher frequencies50. The power was then calculated for all frequencies, timepoints, and channels by squaring the amplitude of the transformed signals. The mean was used as a central tendency measure to represent the ERSP data of each participant. The power time series was then normalized through decibel conversion. Finally, the power of the brainwaves in the frequency bands of interest was extracted: theta (4–9 Hz) and beta (13–30 Hz).
Statistical analyses
The non-parametric randomization test was used for the statistical analysis of EEG data. Randomization testing is a type of surrogate test widely used on EEG signals51; they are ideal for null hypothesis testing because they make no assumption on the data distribution (i.e., whether it follows a normal distribution or not). Additionally, randomization tests are relatively more general when compared to the traditional parametric statistical tests as they do not force a specific test statistic on which the statistical inference is based upon (i.e., t- or F-statistic)52. All the statistical tests across the manuscript are two-tailed, and the threshold of significance is set to 0.05. Statistical analysis on ERP data was computed by splitting the time window of interest to an equal number of intervals of 20 ms and performing the statistical test on each of the segments separately. For the ERSP data, on the other hand, no predefined time window was decided. Thus, the time between 0 and 600 ms and 0–800 ms was split into 30 and 40 equal intervals of 20 ms for the theta and beta band activations, respectively. Statistical analysis was conducted on each of the time bins to determine if there is a statistical significance between the delayed and synchronous conditions. For both ERP and ERSP, the statistic is computed at the group level such that data are averaged over all trials within the subject.
To limit the inflation of type 1 error due to multiple comparisons in the statistical analysis of both ERP and ERSP data, p values were corrected using the false discovery rate (FDR) method53. The number of comparisons, k, is equal to the number of time bins in each statistical test (ERP: k = 7; ERSP(theta band): k = 30 and ERSP(beta band): k = 40). For all analyses, we report the FDR corrected p values along with the time range during which the statistical significance was found.
Results
Sensory correlates
At the contralateral sensorimotor ROI, a post-movement beta rebound time-locked to the haptic collision event was observed. The time course of scalp topographies shown in Fig. 3 clearly depicts the beta rebound under the four conditions (passive/active, with/without haptic delay). Regardless of the condition, an examination of the time-frequency EEG data at the contralateral sensorimotor ROI revealed that beta rebound starts around 200 ms post the haptic collision, as shown in Fig. 4. During the passive task, beta rebound takes almost a similar form under the synchronous and the delayed conditions. Statistical tests confirm this, as the duration at which there is statistically significant difference in the rebound is relatively short (\(\textit{p}\) = 0.0284, randomization test, FDR corrected; \(t_{range}\) = [280–380 ms]). On the other hand and during the active task, beta rebound differs significantly between the synchronous and delayed conditions for a much longer duration as can be seen in Fig. 4f (\(\textit{p}\) = [0.0105–0.0452] , randomization test, FDR corrected; \(t_{range}\) = [360–800 ms]). It can be observed that even though the rebound is time-locked to the haptic collision event in the active task, the rate at which the beta rebound occurred in the synchronous condition is slower than that of the delayed condition.
Cognitive correlates
The time course of scalp topographies of theta band showed an evident activation that was modulated by the haptic delay localized and exhibited at the midfrontal ROI as shown in Fig. 5. An examination of the time-frequency EEG data at the midfrontal ROI revealed theta power synchronization in both the passive and active tasks following the haptic collision event, as shown in the ERSPs plots in Fig. 6a–d. From a spectral perspective and in the presence of delay, theta activation in the passive task extends from 4 to 9 Hz, while in the active task, the activation extends from theta to alpha band, reaching \(\sim 13\) Hz. Temporally, theta synchronization is delayed with the introduction of haptic delay, indicating an association with the haptic collision event. During the active task and in the presence of delay, theta activation peaks earlier than that in the passive task (passive: t = 125 ms; active: t = 60 ms) with respect to the haptic collision event. More specifically, in the active task, theta activation starts before the delayed haptic feedback occurs. In contrast, in the passive task, theta activation is strengthened almost when the delayed haptic feedback is delivered. In the absence of haptic delay (Fig. 6a,c), an activation centered around 200 ms in the theta band is observed regardless of the task type.
The time course of the mean theta power in the midfrontal ROI is presented in Fig. 6e,f. It can be observed that theta synchronization is significantly higher under the presence of the haptic delay, regardless of the task type (passive: \(\textit{p}_{range} =[0.016, 0.037]\), randomization test, FDR corrected; \(t_{range}\) = [0–180 ms] and active: p=0.021, randomization test, FDR corrected; \(t_{range}\) = [0–140 ms]).
The following analysis aims to investigate the ERP activations at the FCz electrode, which is part of the midfrontal ROI as well. At FCz, a sharp ERP peak is observed around 200 ms time-locked to the haptic collision. The P200 peak is delayed under the presence of the haptic delay; Fig. 7 shows the P200 peak at FCz during the passive task, with the onset being the haptic collision event. The P200 peak is commonly observed between 140 and 280 ms40,41,42 post the onset. It was found that the amplitude of the P200 is significantly larger in the presence of the haptic delay during the passive task (\(\textit{p}_{range} =[0.011, 0.048]\), randomization test, FDR corrected; \(t_{range}\) = [140–240 ms]). Additionally, the mean activation topography of the scalp from 140 to 240 ms is shown in Fig. 7 under the synchronous condition, the delayed condition, and the difference between them. The difference topography plot shows the channels with a P200 amplitude that is significantly different between the synchronous and delayed conditions. A negative peak (N100) precedes the P200 component at around 80 ms after the haptic collision; this peak, however, is not significantly modulated by the presence of the haptic delay. During the active task, no statistically significant difference was found between the P200 peak amplitude of the delayed condition and the synchronous condition.
The learning effect during the passive task and its impact on the P200 component has been analyzed in the synchronous and delayed conditions as shown in Fig. 8. The mean ERP of the first 20% trials of each condition per subject were compared against the mean ERP of the last 20% trials of the same condition per subject; P200 peak modulation has been examined. Figure 8 shows a significant reduction (\(\textit{p}_{range} =[0.017, 0.034]\), randomization test, FDR corrected; \(t_{range}\) = [220–280 ms]) in the P200 peak amplitude during the passive task under the absence of haptic delay. In the presence of delay, on the contrary, no significant modulation was observed. Hypothesis (3) is therefore accepted (\(\textit{p}_{range} =[0.400, 0.791]\), randomization test, FDR corrected; \(t_{range}\) = [220–280 ms]) indicating no statistical difference between the first and last 20% of the trials in the presence of haptic delay in the passive task. During the active task, no significant modulation of the P200 peak has been observed over the trials.
Performance evaluation
Participants were occasionally asked after trials if the haptic collision was delayed or not. This is to keep the participant engaged with the experimental task and avoid boredom which could occur due to the repetitive nature of the EEG experiments. In total, every participant was asked to answer this question 30 times (15 for the passive task and 15 for the active task) during the whole session out of 200 trials. The average percentage of misperceptions is 11.2% for the passive task and 11.5% for the active task, with no significant difference observed in misperception between the two tasks. Cumulatively, participants were accurate in their answers (and consequently their perception of the delay) on average 88.6% of the time (Min 66.6%, Max 100%, Median 93.3%). The majority of the participants (74%) were not familiar with using a haptic device, so the training session was essential. Around 58% of the participants were able to identify better the presence of the haptic delay in the passive task as compared to 42% who were able to identify better the presence of the haptic delay in the active task based on the questionnaire.
Discussion
Haptic delay is a vital perceptual attribute that can drastically affect the quality of experience in many human–machine interaction scenarios such as in virtual reality and haptic-skill transfer applications54. The goal of the present study was to investigate the sensory and cognitive neural correlates of the haptic delay during passive and active haptic tasks. The main finding is that haptic delay induced significantly different neural traces than the synchronous condition in both ERP and oscillatory activity (ERSPs), mainly at the midfrontal region. Processes involved include multisensory divided attention, conflict resolution, and perceptual attention.
With respect to sensory ERSP activity, beta rebound starting time was found to be time-locked to the haptic collision. Post movement beta rebound is generally characterized by a pronounced increase in beta power occurring within 1000 ms post the movement43. Beta rebound is reported to be most prominent over the sensorimotor cortical areas55 once the movement of the effector (e.g: limb) is terminated56. Several studies also showed that beta rebound is not only observed during active movements, it is also clearly exhibited during passive57,58,59 or imagined movements60,61. Indeed, in this study, we observed beta rebound under both conditions, the passive and active task. The functional role of beta rebound is not fully agreed upon in the literature and is still under debate; however, a widely accepted hypothesis is that it inhibits the motor network upon the movement termination62,63. The current findings show in the passive task, beta rebound is relatively similar between the synchronous and delayed conditions. Participants receive the haptic feedback while their hands are at rest with merely a 220 ms delay in time under the delayed condition. It is thus expected from a sensory perspective, both scenarios are equivalent, and beta rebound is not modulated. The active task is intrinsically different as the haptic feedback is delivered during the hand movement. Under the delay condition, the wrist stops moving around the time of the visual collision indicating a successful end to the trial from the participant’s perspective. Thus, the haptic feedback is delivered when the wrist is relatively stationary, mimicking the passive condition from a sensory perspective. However, in the synchronous case, the haptic collision is felt while the wrist is still moving and about to stop. Since the direction of the hand movement and the direction of the haptic collision are opposite, we believe that the inhibitory processes will occur at a decreased rate compared to the other three conditions, as shown in Fig. 4f.
In the midfrontal ROI, theta oscillation was found to peak in a relatively similar manner during the passive and active tasks in the absence of haptic delay as shown in Fig. 6e,f. When more than one modality is in use, theta oscillations are considered a marker of multisensory divided attention17,19. Previous studies reported a relationship between theta oscillations and various functions related to multisensory divided attention in audio–visual integration19,64. In the context of our study, haptic-visual modalities are both employed, which is a different form of multisensory activity from the studies mentioned above19,64. Despite the difference in the modalities involved, theta oscillation was found to increase after the onset of the haptic-visual event (synchronous condition). Introducing haptic delay to the ball-racket collision event caused a significant increase in the theta oscillation in the midfrontal cortex and an observed delay of the peak power as clearly shown in Fig. 6b,d. In this scenario, divided attention is still required due to the dual nature of the stimuli (visual-haptic); however, further cognitive processes are induced due to the incongruity between the two modalities. Indeed, theta oscillation was reported to be more pronounced under incongruent cross-modal stimulation as compared to congruent stimulation under visuotactile24,25 and audiovisual stimulations21.
These studies21,24,25 suggest that theta oscillation reflects conflict processing and resolution since only in the incongruent condition, a bimodal competition of stimuli was present. In our current study, participants typically would expect the haptic feedback to be delivered and felt at the moment of the visual collision; an absence of this anticipated sensation at the moment of collision followed by an unexpected haptic stimulation once the collision is over would initiate neural processes of conflict detection and resolution. In other words, it seems that theta band activation in the middle frontal cortex index perceived multisensory incongruence and induced conflict detection mechanisms accordingly. In this light, it is tempting to state that conflict processing resembled in theta oscillation was possible to observe due to the recognizable delay by the human brain (220 ms).
In the presence of delay and comparing theta oscillation during the passive and active tasks, it is observed from Fig. 6b,d that theta oscillation peaks earlier in the active task. Understanding the differences between the active and passive movements proves helpful in understanding the underlying neural differences during the two tasks. A major difference between the two lies in the mechanisms involved in proprioception, a sense of awareness of the spatial and mechanical status of the body65. During the passive task, a one-fold dominance of proprioception is in action (awareness of body position), while during the active task, two-fold proprioception is in action (awareness of body position and movement). We argue that the delay in theta oscillation is caused by the delay in the realization process of the conflict. This realization occurred earlier in the active task as compared to the passive task due to the additional mechanism in action in the active task under proprioception. The two-fold continuous proprioceptive feedback activated during the active task could have caused an earlier realization of the incongruity conflict between the two modalities. As argued in self-reporting studies, it is observed that the detection threshold of the haptic delay with respect to the corresponding visual stimulus is smaller during active tasks when compared to passive tasks6.
From a spectral perspective, it can be observed from Fig. 6b,d that theta activation is limited between 4 and 9 Hz during the passive task, while in the active task, the activation extends beyond the theta band to cover the alpha band as well (4–13 Hz). It is interesting to observe that this activation in the alpha band is not a separate form of activation; instead, the activation looks like a single body covering both theta and alpha band. A similar observation is found in another work that studied error processing, and stimuli congruence66 where participants were asked to prioritize accuracy in detecting incongruence. It is argued that alpha activation in the frontal region associates with top-down attentional control over lower sensory regions as a reorienting response67. When sustained attention is required, participants need to keep track of their attention, and in case of any lapses, refocus. This refocusing process is attributed to the observed alpha synchronization. In light of this study, and to explain the alpha activation in Fig. 6d, we speculate that there is a relationship between the nature of the active task and the need for higher attentional control over the sensory channels. Indeed, due to the motor movement involved during the active task, higher attentional control could be needed to detect the conflict caused by the haptic delay.
In the time domain, attention-related ERP component (P200 peak) was observed particularly at the midfrontal ROI. P200 peak is observed to be time-locked to the haptic stimulus; a delay in the haptic feedback caused the same delay to the P200 peak. A study that focused on studying passive ankle movements found that the P200 peak is a later component that is related to endogenous factors and could reflect cognitive processes such as attention and haptic stimulus localization or estimation68. In fact, P200 is reported to be an ERP component that is highly related to early attentional mechanisms69,70. In this study, a significant P200 modulation (\(p<0.05\), randomization test, FDR corrected) was observed under the presence of haptic delay in the passive task only. This suggests that haptic delay had a significant effect on attention towards the haptic feedback during the passive task, while the same is not true during the active task. A possible explanation is that participants are perceptually less occupied during the passive task, which makes them more sensitive and attentive towards the haptic feedback when delayed. The impact of the learning effect on attention and high order perception (i.e.: P200 amplitude) was examined. As mentioned earlier in the results section, P200 amplitude was significantly reduced in the last 20% trials during the passive task and in the absence of haptic delay. Participants exhibited a strong learning effect under the passive condition in the absence of the haptic delay, and their attention (P200 peak) is reduced. Conversely, participants did not exhibit a learning effect in the presence of haptic delay (passive task) and elicited almost the same attention at the start and towards the end of the trial set. During the active task, participants were less susceptible to the learning effect in the presence or absence of the haptic delay.
One limitation of this study is that a single haptic delay value (220 ms) was tested. However, it is crucial to start with an easily recognizable delay that will highlight the related neural correlates of delay, which can assist in understanding the neural traces for granulated values of delay. Future studies should build on the obtained results and understand the effect of haptic delay on a continuous scale. Additionally, jitter is another parameter that is quite important in altering the haptic experience and is a potential future direction. In sum, the present study provides evidence that haptic delay induces significant neural traces that could be observed over the central and the midfrontal ROI during active and passive tasks. More work is warranted to cover other perceptual attributes important during haptic communication.
Data availability
The EEG dataset analysed and discussed in this work can be obtained from the corresponding author upon reasonable request.
References
Jaimes, A. & Sebe, N. Multimodal human–computer interaction: A survey. Comput. Vision Image Understanding 108, 116–134 (2007).
Van Den Berg, D. et al. Challenges in haptic communications over the tactile internet. IEEE Access 5, 23502–23518 (2017).
Ferrell, W. R. Remote manipulation with transmission delay. IEEE Trans. Hum. Fact. Electron. HFE-6(1), 24-32 (1965).
Tatematsu, A., Ishibashi, Y., Fukushima, N. & Sugawara, S. QoE assessment in tele-operation with 3D video and haptic media. 2011 IEEE International Conference on Multimedia and Expo 1–6 (2011).
Knorlein, B., Di Luca, M. & Harders, M. Influence of visual and haptic delays on stiffness perception in augmented reality. 2009 8th IEEE International Symposium on Mixed and Augmented Reality 49–52 (2009).
Vogels, I. M. Detection of temporal delays in visual-haptic interfaces. Human Factors 46, 118–134 (2004).
Jay, C., Glencross, M. & Hubbold, R. Modeling the effects of delayed haptic and visual feedback in a collaborative virtual environment. ACM Trans. Comput.–Hum. Interact. (TOCHI) 14, 8-es (2007).
Colonnese, N., Siu, A. F., Abbott, C. M. & Okamura, A. M. Rendered and characterized closed-loop accuracy of impedance-type haptic displays. IEEE Trans. Haptics 8, 434–446 (2015).
Alsuradi, H., Park, W. & Eid, M. Eeg-based neurohaptics research: A literature review. IEEE Access 8, 49313–49328 (2020).
Schirmer, A. & McGlone, F. A touching sight: Eeg/erp correlates for the vicarious processing of affectionate touch. Cortex 111, 1–15 (2019).
Holroyd, C. B., Pakzad-Vaezi, K. L. & Krigolson, O. E. The feedback correct-related positivity: Sensitivity of the event-related brain potential to unexpected positive feedback. Psychophysiology 45, 688–697 (2008).
Singh, A. K. et al. Visual appearance modulates prediction error in virtual reality. IEEE Access 6, 24617–24624 (2018).
Cavanagh, J. F., Zambrano-Vazquez, L. & Allen, J. J. Theta lingua franca: A common mid-frontal substrate for action monitoring processes. Psychophysiology 49, 220–238 (2012).
Nigbur, R., Ivanova, G. & Stürmer, B. Theta power as a marker for cognitive interference. Clin. Neurophysiol. 122, 2185–2194 (2011).
Sauseng, P. et al. Are event-related potential components generated by phase resetting of brain oscillations? A critical discussion. Neuroscience 146, 1435–1444 (2007).
van Ackeren, M. J., Schneider, T. R., Müsch, K. & Rueschemeyer, S.-A. Oscillatory neuronal activity reflects lexical-semantic feature integration within and across sensory modalities in distributed cortical networks. J. Neurosci. 34, 14318–14323 (2014).
Keil, J. & Senkowski, D. Neural oscillations orchestrate multisensory processing. Neuroscientist 24, 609–626 (2018).
Cavanagh, J. F., Frank, M. J., Klein, T. J. & Allen, J. J. Frontal theta links prediction errors to behavioral adaptation in reinforcement learning. Neuroimage 49, 3198–3209 (2010).
Keller, A. S., Payne, L. & Sekuler, R. Characterizing the roles of alpha and theta oscillations in multisensory attention. Neuropsychologia 99, 48–63 (2017).
Cohen, M. X. & Donner, T. H. Midfrontal conflict-related theta-band power reflects neural oscillations that predict behavior. J. Neurophysiol. 110, 2752–2763 (2013).
Morís Fernández, L., Torralba, M. & Soto-Faraco, S. Theta oscillations reflect conflict processing in the perception of the mcgurk illusion. Eur. J. Neurosci. 48, 2630–2641 (2018).
Cohen, M. X. & Ridderinkhof, K. R. Eeg source reconstruction reveals frontal-parietal dynamics of spatial conflict processing. PloS One 8, e57293 (2013).
Arrighi, P. et al. Eeg theta dynamics within frontal and parietal cortices for error processing during reaching movements in a prism adaptation study altering visuo-motor predictive planning. PLoS One 11, e0150265 (2016).
Göschl, F., Friese, U., Daume, J., König, P. & Engel, A. K. Oscillatory signatures of crossmodal congruence effects: An eeg investigation employing a visuotactile pattern matching paradigm. Neuroimage 116, 177–186 (2015).
Kanayama, N. & Ohira, H. Multisensory processing and neural oscillatory responses: Separation of visuotactile congruency effect and corresponding electroencephalogram activities. Neuroreport 20, 289–293 (2009).
Mania, K., Adelstein, B. D., Ellis, S. R. & Hill, M. I. Perceptual sensitivity to head tracking latency in virtual environments with varying degrees of scene complexity. In Proceedings of the 1st Symposium on Applied Perception in Graphics and Visualization 39–47 (2004).
Savoie, F.-A., Thénault, F., Whittingstall, K. & Bernier, P.-M. Visuomotor prediction errors modulate eeg activity over parietal cortex. Sci. Rep. 8, 1–16 (2018).
Tan, H., Wade, C. & Brown, P. Post-movement beta activity in sensorimotor cortex indexes confidence in the estimations from internal models. J. Neurosci. 36, 1516–1528 (2016).
Torrecillos, F., Albouy, P., Brochier, T. & Malfait, N. Does the processing of sensory and reward-prediction errors involve common neural resources? Evidence from a frontocentral negative potential modulated by movement execution errors. J. Neurosci. 34, 4845–4856 (2014).
Perfetti, B. et al. Modulation of gamma and theta spectral amplitude and phase synchronization is associated with the development of visuo-motor learning. J. Neurosci. 31, 14810–14819 (2011).
Lin, C.-L., Shaw, F.-Z., Young, K.-Y., Lin, C.-T. & Jung, T.-P. Eeg correlates of haptic feedback in a visuomotor tracking task. NeuroImage 60, 2258–2273 (2012).
Delorme, A. & Makeig, S. Eeglab: An open source toolbox for analysis of single-trial eeg dynamics including independent component analysis. J. Neurosci. Methods 134, 9–21 (2004).
Kothe, C. A. E. & Jung, T.-P. Artifact removal techniques with signal reconstruction. US Patent App. 14/895440 (2016).
Lakshmi, M. R., Prasad, T. & Prakash, D. V. C. Survey on eeg signal processing methods. Int J Adv Res Comput Sci Software Eng. 4(1), (2014).
Jung, T.-P. et al. Removing electroencephalographic artifacts by blind source separation. Psychophysiology 37, 163–178 (2000).
Buzsaki, G. Rhythms of the Brain (Oxford University Press, 2006).
Goncharova, I. I., McFarland, D. J., Vaughan, T. M. & Wolpaw, J. R. Emg contamination of eeg: Spectral and topographical characteristics. Clin. Neurophysiol. 114, 1580–1593 (2003).
Pinheiro, A. P., Schwartze, M., Gutierrez, F. & Kotz, S. A. When temporal prediction errs: Erp responses to delayed action-feedback onset. Neuropsychologia 134, 107200 (2019).
Gehrke, L. et al. Detecting visuo-haptic mismatches in virtual reality using the prediction error negativity of event-related brain potentials. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems 1–11 (2019).
Montoya, P. & Sitges, C. Affective modulation of somatosensory-evoked potentials elicited by tactile stimulation. Brain Res. 1068, 205–212 (2006).
Garcia, L. N., Guerrero-Mosquera, C., Colomer, M. & Sebastian-Galles, N. Evoked and oscillatory eeg activity differentiates language discrimination in young monolingual and bilingual infants. Sci. Rep. 8, 1–9 (2018).
Gan, T., Fang, W. & Ge, L. Colours' impact on morality: Evidence from event-related potentials. Sci. Rep. 6, 1–9 (2016).
Kilavik, B. E., Zaepffel, M., Brovelli, A., MacKay, W. A. & Riehle, A. The ups and downs of beta oscillations in sensorimotor cortex. Exp. Neurol. 245, 15–26 (2013).
Van Elk, M., Van Schie, H. T., Van Den Heuvel, R. & Bekkering, H. Semantics in the motor system: Motor-cortical beta oscillations reflect semantic knowledge of end-postures for object use. Front. Hum. Neurosci. 4, 8 (2010).
Espenhahn, S. et al. Cortical beta oscillations are associated with motor performance following visuomotor learning. NeuroImage 195, 340–353 (2019).
Kayser, J. & Tenke, C. E. Principal components analysis of laplacian waveforms as a generic method for identifying erp generator patterns: I. Evaluation with auditory oddball tasks. Clin. Neurophysiol. 117, 348–368 (2006).
Vidal, F. et al. Linking eeg signals, brain functions and mental operations: Advantages of the laplacian transformation. Int. J. Psychophysiol. 97, 221–232 (2015).
Burle, B. et al. Spatial and temporal resolutions of eeg: Is it really black and white? A scalp current density view. Int. J. Psychophysiol. 97, 210–220 (2015).
Grossmann, A. & Morlet, J. Decomposition of hardy functions into square integrable wavelets of constant shape. SIAM J. Math. Anal. 15, 723–736 (1984).
Cohen, M. X. Analyzing Neural Time Series Data: Theory and Practice (MIT Press, 2014).
Päeske, L. et al. Surrogate data method requires end-matched segmentation of electroencephalographic signals to estimate non-linearity. Front. Physiol. 9, 1350 (2018).
Maris, E. & Oostenveld, R. Nonparametric statistical testing of eeg-and meg-data. J. Neurosci. Methods 164, 177–190 (2007).
Benjamini, Y. & Hochberg, Y. Controlling the false discovery rate: A practical and powerful approach to multiple testing. J. R. Stat. Soc. Series B (Methodol.) 57, 289–300 (1995).
Mireles, E. J. A., Zenzeri, J., Squeri, V., Morasso, P. & De Santis, D. Skill learning and skill transfer mediated by cooperative haptic interaction. IEEE Trans. Neural Syst. Rehabilit. Eng. 25, 832–843 (2017).
Salmelin, R., Hámáaláinen, M., Kajola, M. & Hari, R. Functional segregation of movement-related rhythmic activity in the human brain. Neuroimage 2, 237–243 (1995).
Neuper, C. & Pfurtscheller, G. Evidence for distinct beta resonance frequencies in human eeg related to specific sensorimotor cortical areas. Clin. Neurophysiol. 112, 2084–2097 (2001).
Alegre, M. et al. Beta electroencephalograph changes during passive movements: Sensory afferences contribute to beta event-related desynchronization in humans. Neurosci. Lett. 331, 29–32 (2002).
Müller, G. et al. Event-related beta eeg changes during wrist movements induced by functional electrical stimulation of forearm muscles in man. Neurosci. Lett. 340, 143–147 (2003).
Cassim, F. et al. Does post-movement beta synchronization reflect an idling motor cortex?. Neuroreport 12, 3859–3863 (2001).
Neuper, C., Schlögl, A. & Pfurtscheller, G. Enhancement of left-right sensorimotor eeg differences during feedback-regulated motor imagery. J. Clin. Neurophysiol. 16, 373–382 (1999).
Pfurtscheller, G., Neuper, C., Brunner, C. & Da Silva, F. L. Beta rebound after different types of motor imagery in man. Neurosci. Lett. 378, 156–159 (2005).
Solis-Escalante, T., Müller-Putz, G. R., Pfurtscheller, G. & Neuper, C. Cue-induced beta rebound during withholding of overt and covert foot movement. Clin. Neurophysiol. 123, 1182–1190 (2012).
Cao, L. & Hu, Y.-M. Beta rebound in visuomotor adaptation: Still the status quo?. J. Neurosci. 36, 6365–6367 (2016).
Sakowitz, O. W., Schürmann, M. & Başar, E. Oscillatory frontal theta responses are increased upon bisensory stimulation. Clin. Neurophysiol. 111, 884–893 (2000).
Stillman, B. C. Making sense of proprioception: The meaning of proprioception, kinaesthesia and related terms. Physiotherapy 88, 667–676 (2002).
van Driel, J., Ridderinkhof, K. R. & Cohen, M. X. Not all errors are alike: Theta and alpha eeg dynamics relate to differences in error-processing dynamics. J. Neurosci. 32, 16795–16806 (2012).
Corbetta, M. & Shulman, G. L. Control of goal-directed and stimulus-driven attention in the brain. Nat. Rev. Neurosci. 3, 201–215 (2002).
Spironelli, C. & Angrilli, A. Influence of body position on cortical pain-related somatosensory processing: An erp study. PloS One 6, e24932 (2011).
Luck, S. J. et al. Effects of spatial cuing on luminance detectability: Psychophysical and electrophysiological evidence for early selection. J. Exp. Psychol. Hum. Percept. Perform. 20, 887 (1994).
Ballesteros, S., Munoz, F., Sebastian, M., Garcia, B. & Reales, J. M. Erp evidence of tactile texture processing: Effects of roughness and movement. In World Haptics 2009-Third Joint EuroHaptics conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems 166–171 (IEEE, 2009).
Acknowledgements
This research was funded by the New York University Abu Dhabi PhD Fellowship Program.
Author information
Authors and Affiliations
Contributions
H.A. and W.P. designed the experimental protocol. H.A. conducted the experiment and analyzed the results. W.P. and M.E. reviewed the results. All authors wrote and reviewed the manuscript.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Additional information
Publisher's note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Alsuradi, H., Park, W. & Eid, M. Midfrontal theta oscillation encodes haptic delay. Sci Rep 11, 17074 (2021). https://doi.org/10.1038/s41598-021-95631-1
Received:
Accepted:
Published:
DOI: https://doi.org/10.1038/s41598-021-95631-1
- Springer Nature Limited
This article is cited by
-
EEG correlates to perceived urgency elicited by vibration stimulation of the upper body
Scientific Reports (2024)
-
Midfrontal theta power encodes the value of haptic delay
Scientific Reports (2022)