Keywords

1 Introduction

Mid-air buttons are commonly used in extended reality or holographic interactions. In a pandemic-touched and increasingly germophobic world, contactless interactions cause less hygiene concerns for multi-user buttons. While current technology allow their visual and audio feedback to resemble physical buttons, mid-air buttons have poor performance due to their lack of physicality [1, 8, 9]. The fingertip can rest on the surface of physical buttons, but mid-air buttons have no surface to rest on. Where physical buttons reach a hard barrier upon being fully pressed, mid-air buttons can be pressed far beyond their activation point as seen in Fig. 1. The physical barrier creates a natural pull-up sensation with physical buttons that is not present in mid-air buttons, leading to an increase in the press duration or failed button activations.

Fig. 1.
figure 1

A sample timeline from a participant performing a rapid tapping task with a 3D mid-air button. The timeline shows the finger and button movements. The local maximums show the fingertip displacements between button presses (inter-press displacement) while the local minimums show the displacement beyond the button activation point (press-through displacement). All measurements were captured using the Oculus Quest 2 hand tracking.

Full size image

Researchers have proposed various solutions to add haptic feedback in mid-air [4] including wearable devices like haptic gloves, encounter-type devices that can move to make contact with users, or airborne feedback. The latter approach is power efficient and does not require users to wear a device. Ultrasound feedback has been shown to improve user interactions with mid-air widgets. Adding ultrasound haptics to the gestural control of an automotive dashboard reduced the eyes-off-the-road time and the driver’s mental workload [12]. Martinez et al. showed that ultrasound haptics can increase users’ sense of agency over mid-air buttons [2]. Sand et al. studied text entry in virtual reality (VR) by adding an array of ultrasound transducers to a VR headset. They found that ultrasound feedback significantly reduced user report of temporal demand compared to no haptic feedback, but there was no significant difference in user performance [11]. Ito et al. presented a dual-layer button with varied ultrasound intensities and quality for the button press and activation phases, but they only showed that people could locate the two haptic layers [5]. The current solutions only apply ultrasound feedback to the surface of the hand that is in contact with the button. They report little performance improvements, possibly due to the limited intensity of ultrasound haptics.

We propose whole-hand haptic feedback for mid-air button presses and report user performance with it. When tapping a physical surface with the fingertip, the skin vibrations propagate down the hand [13]. As the intensity of ultrasound haptics is low, we induce the vibrations to the whole hand by moving the ultrasound’s focal point over the hand instead of relying on the strength of the fingertip feedback to propagate down the hand. The objective of the whole-hand haptics is to mitigate the lack of pull-up effect in mid-air. To achieve this, we consider the three following measurements as the main indicators of an improved pull-up effect: a decrease in the pull-up time after an activation, a decrease in the finger displacement beyond the activation point, and an increase in the amount of presses in a time frame. We ran a user study to evaluate user performance with mid-air buttons using our whole-hand haptic feedback approach. Participants interacted with 2D and 3D mid-air buttons in a VR environment while they received either localized haptic feedback at their fingertip or whole-hand feedback over their dominant hand. Our results show that whole-hand haptics affect the button press duration and increase the number of button activations in a time frame compared to the localized haptics. We discuss possible reasons for these improved results.

2 Mid-air Button Design

We divide the user interaction with buttons into four phases where the finger movement and haptic feedback from physical buttons are distinct [6]. 3D buttons include all four phases, while 2D buttons have no protrusion. The draw frequency, which is the number of times the pattern is drawn in a second, is indicated with Hz. The sensations are rendered using Spatiotemporal Modulation [3]. Figure 2 shows how the haptics are spread across the palmar side of the hand. We design whole-hand haptics for user interaction in these phases based on a pilot study:

  1. 1.

    Proximity: When the user’s finger is near the button without touching it. We set this value to 10 mm above the button and present a haptic sensation over the whole finger by moving the focal point between the distal, middle and proximal phalanx (70 Hz).

  2. 2.

    Protrusion: When the fingertip is pressing the button. We set the button depth to 10 mm and present a haptic effect that is a 10 mm in radius circle (Hz = 70). The sensation is focused on the fingertip to match the visual contact area of the button.

  3. 3.

    Activation: A transient phase starting from when the button is successfully pressed. The sensation is a 80 mm wide line that traverses from the fingertip to the root of the hand (90 Hz) in 100 ms.

  4. 4.

    Release: From the end of the activation phase until the button is unpressed. We provide no sensation in this phase.

Fig. 2.
figure 2

The whole-hand sensation set showing the spread of the haptic sensation in each of the phases. Picture of the palm is adapted for non-commercial use [10].

Full size image

We kept the visual feedback minimal to avoid interference with the haptic effect. Based on a pilot study, we opted for 10 mm press distance and used a 40 by 40 mm button to ensure participants could accurately hit it. The 2D button had no visual feedback for any phases. The 3D button’s visual protrusion followed the fingertip, indicating to users that it was activated or released when it no longer followed the fingertip.

3 User Study

To evaluate the impact of whole-hand haptic feedback on user performance with 2D and 3D buttons, we conducted a user study with two user tasks.

Participants. We recruited 20 participants (4F/16M) 23–57 years old (M = 31, SD = 9.93). No one reported any sensory impairment in their dominant hand. Seven users had prior experience with mid-air hapticsFootnote 1. The experiment took around 30 min and participants received a gift worth around 100 DKK.

Tasks. The first task was a rapid tapping task, where participants pressed the button as quickly as possible. This task was chosen to capture whether more presses are possible with whole-hand feedback and the effect it has on the press duration and finger movement. Each trial of the task lasted 20 s. The second task was to double-click the mid-air button as quickly as possible after hearing a randomly timed sound cue. This reaction task is a modified version of the moving target selection task [7] without the visual element. We require two presses (i.e., a double click) to capture measures on the inter-press duration.

Study Design. The experiment used a within-subjects design with the three independent variables dimension and haptic and trial. The button dimension was either 2D or 3D to study the effect of the protrusion phase on user performance. The haptic sensation was either the whole-hand haptics or a localized sensation used as a baseline. The localized haptics was based on the activation sensation by Martinez et al., that showed improvement to the sense of agency for mid-air buttons [2]. Instead of the five focal points they used to cover 1 cm\(^2\), we used a 200 ms transient version similar to the protrusion sensation in Fig. 2. The combination of button dimension and haptics yielded four conditions. Each condition was repeated three times for a total of 12 trials in each task.

Apparatus. We used the ultrasound device STRATOS Explore to induce haptic feedback on the palmar side of the hand. The participants wore an Oculus Quest 2 VR head-mounted display with a refresh rate 90 Hz to interact with the mid-air button. The button was calibrated to be approximately 20 cm above the ultrasound device’s surface, where the ultrasound focal point is the strongest. We used Oculus Quest 2’s hand tracking (60 Hz refresh rate) to measure the movements of all limbs in the hand as well as the visual representation of the hand. The displacement of the button followed the fingertip while being constrained to one dimension with an upper and lower limit. The Leap Motion’s hand tracking (120 Hz refresh rate) was used to position the haptic feedback only. Measurements were recorded 200 Hz.

Procedure. After signing a consent form, the participants were introduced to the mid-air technology by feeling common 2D shapes like circles and lines. They were informed of the two tasks and that they needed to press a variation of 2D or 3D virtual buttons with haptic sensations. They were not told how the sensations would differ. The participants were instructed to complete the press interaction as quickly as possible. They were told that the goal of the rapid tapping was to press as many times as possible and that the time from the sound cue to the release of the second click was important for the reaction task. Before the first trial of each condition, the participants could practice the button press for up to a minute to get conditioned to the button dimension and haptic feedback. During this training, the participants were given visual feedback after each interaction on their performance. After the training, each trial was started. There was a five to ten second break between each trial of the same condition and a two minute break between the two tasks. The condition order was counterbalanced. Pink noise was played to mask the sound of the device.

4 Results

We recorded the movements of the button and the index finger for both tasks. With the button movements we captured the number of times the button was successfully activated (i.e., the press count). We obtained the press duration as a sum of the down-press duration (from button contact until the button is activated) and the pull-up duration (from activation until the button is released). The press-through displacement quantifies how far the finger moves beyond the button activation point. The inter-press displacement is how far the finger moves away from its released state between each press. Additionally, the reaction task included the reaction time from the sound cue until contact with the button, the duration between the two presses (inter-press duration), and the success rate.

Rapid Tapping Task. A total of 16,609 successful presses were recorded. Outliers in each participant’s trials were removed using the IQR method, where outliers are values more than 1.5 times above or below the trial’s inter-quartile range.

Table 1. The differences in press count, durations and displacements (Disp.) between the localized and whole-hand haptics (top) and 2D and 3D buttons (bottom) in the rapid tapping task.
Full size table

We ran three-way repeated measures ANOVAs with the six measurements as the dependent variables and dimension, haptics, and trial as the within-subjects factors (Table 1). The results showed main effects of haptics for press count, press duration, and pull-up duration without any significant interaction effects with dimension nor trial. The participants pressed significantly quicker (9.52%) and had more successful button presses (4.76%) with whole-hand haptics. The pull-up duration was significantly decreased (10.87%). The press-through displacement showed an improvement of 12.19%, but this difference was not significant due to the high standard errors. The inter-press displacement was lower with localized haptics but not significantly.

The repeated measures ANOVA also showed a main effect of dimension for the press count, press duration and the inter-press displacement (Table 1). These values are significantly different for 2D buttons due to no down-press time in the dimension. Importantly, we found no interaction among dimension and haptics for these dependent variables (\(p > 0.713\)). Figure 3 shows the results for the six measurements under the four conditions.

Fig. 3.
figure 3

The mean and standard error of the localized and whole-hand conditions in the rapid tapping task. The number of presses in 20 s increased significantly with the whole-hand haptics. A significant decrease is evident in the press and pull-up duration, while the differences for the displacements were not significant.

Full size image

Reaction Task. We ran three-way repeated measures ANOVAs with the nine measurements as the dependent variables and dimension, haptics, and trial as the within-subjects factors. There were no significant effects of the haptic conditions on any of the dependent variables (\(p\ge 0.064\) for all the measures). The 2D condition showed a significant improvement in press-through displacement (\(p = 0.024\)) and inter-press displacement (\(p < 0.001\)).

5 Discussion and Conclusion

This paper set out to improve haptics for mid-air buttons, exploring the idea that whole–hand haptics could improve the interaction due to an enhanced pull-up effect. To support this idea we examined the pull-up duration, press depth, and press count as main indicators. The results suggest that it is possible to design haptic feedback that improves the performance for rapid presses. In the rapid tapping task, whole-hand haptics significantly improves the pull-up duration. Since the pull-up duration accounts for 94% of the overall press duration according to our measurements, it is a major factor in the improved performance. We believe the decrease in the press duration is the main factor behind the increase in the press count. Improving the press count shows that whole-hand haptics is an improvement to the overall press interaction, not just the release phase. While the reduction in press-through displacement was not significant, it did trend towards improvement. The standard error of the whole-hand condition (0.219 cm) is lower than the localized (0.289 cm), suggesting it is easier to control the finger movement with the whole-hand haptics.

There are multiple reasons for improved performance with the whole-hand haptics. One is the overall intensity felt on the hand. Some participants mentioned they perceived the whole-hand haptics as stronger compared to the localized, even though their focal point intensity is equal. The design of the haptic feedback upon activation can also be a reason. The sweep-back sensation from the fingertip to the root of the hand can signal users to follow that direction. An instant whole-hand sensation, like a clap, may not have the same effect on user performance.

The lack of significant improvements in the reaction task can be due to the high frequency of repetitions in the rapid tapping task compared to the controlled double clicks. As participants continuously press in the rapid tapping task, their motor system likely takes over. With each press, it tunes the motor command to infer the moment of activation [8]. The intensity of the rapid tapping task forces participants to rely more on the haptic feedback, whereas the reaction task allows them to rely on the visual feedback in the short period between each double click. The proximity sensation also did not seem to improve user performance in the reaction task. Our recordings show that the participants moved their fingers beyond the 2-cm mark where the proximity effect stops (Fig. 1). We hypothesize that removing the proximity sensation may improve user performance.

As touchless interactions are becoming more valued, it is important that performance is not lost. In this paper, we show how whole-hand haptics can significantly improve the performance of button pressing.