Introduction

Increased dexterity in multifunction prosthetic hands offers hope to upper limb (UL) amputees that have been previously using a single degree of freedom (DOF) prosthetic hand. At the same time, using multifunction hands begins with a journey of learning that can be cognitively demanding. This learning process can be explained using the model of skill acquisition proposed by Fitts and Posner, which includes three stages: cognitive, associative, and autonomous stages [1]. At the cognitive stage, the prosthesis users learn the technical parts of the device and how to control it. At the associative stage, attention may be focused on specific details, such as grip switching. At the autonomous stage, the user keeps practising until his/her performance enters an automatized routine. Prosthesis users gradually learn to use different prosthetic grips effectively to hold  everyday objects of different materials, weights and sizes.

Since the commercial introduction of the first multifunction prosthetic hand in 2007 and its first publication [2], significant advancements have been made in both functionality and aesthetics. These include features like intelligent finger sensing, improved grip options, a more robust movable thumb, and a multi-position wrist [3, 4]. While prosthesis users generally prefer the enhanced functions and appearance of multifunction hands over other terminal device options such as single DOF hands [5, 6], conflicting findings are reported regarding their functional benefits [5, 7,8,9]. Additionally, questions arise around the cost-effectiveness of multifunction hands compared to single DOF prosthetic hands [10]. It is crucial to underscore the importance of targeted training for users of multifunction hands to achieve optimal functional outcomes [5, 7, 10].

During training, therapists closely monitor the user's progress with the prosthesis and tailor the pace and direction of training to ensure success [11]. One way to help monitor the progress in learning multifunction hands is to use clinical outcome measures. Between 2017 and 2023, a number of observational or self-reported clinical tools have been used to assess functional outcomes of multifunction hands and control strategies (Table 1).

Table 1 Clinical outcome measures that have been used to assess multifunction hands between 2017 and 2023
Full size table

Some functional assessments were originally developed for single DOF prosthetic hands whereas others were developed for human hand function. It is thus questionable whether the study results were valid for multifunctional hands [5, 8, 9]. For example, the dexterity measures in one study did not observe any differences between multifunction hands and single DOF hands. Functional assessments for single DOF hands may not adequately capture the improved dexterity of multifunction hands. Objective measures that assess functional outcomes of multifunction hands are thus crucial for prosthetic rehabilitation. These measures, particularly those that capture cognitive or mental processing during the use of multifunction hands, provide valuable information for therapists to adjust training pace effectively.

In recent years, eye tracking has been suggested to reveal cognitive processes in neuroscience research [21]. The interpretation of gaze behavior is commonly based on the eye-mind hypothesis, in which it is assumed that the eyes fixate on the entity with which the mind is engaged [22, 23]. Eye gaze metrics such as longer fixation duration on special areas during task performance have been demonstrated to be signs for engagement or mental processing [24]. Saccade amplitude decreases as task difficulty increases and the need to gather more fine-grained information increases [25]. Eye tracking metrics hence have the potential to measure cognitive processes in learning to use multifunction hands. For example, switching between different grips in multifunction hands is not intuitive and can be cognitively demanding [26, 27].

A recent review suggested that eye tracking technology can be an effective tool to quantitatively assess visuomotor behaviour among single DOF prosthesis users and able-bodied controls [28]. Based on the 17 included studies, the review concluded that visual attention was directed more towards the prosthetic hand and less towards the target during object manipulation tasks. However, most of these studies used standardized tasks under experimental conditions, which limits the transferability of this finding to self-chosen everyday tasks. Therapists frequently evaluate prosthesis users using self-chosen tasks because the ultimate goal of prosthetic rehabilitation is to empower amputees to seamlessly incorporate the multifunction hand into their self-chosen everyday tasks.

In terms of gaze behavior and single DOF prostheses, several studies showed a large hand-focused gaze during prosthesis use [29,30,31] and it took longer to disengage gaze from manipulations to plan upcoming movements [30]. When training was given, the hand-focused gaze on the single DOF prosthesis decreased in one study but remained the same in another study [29, 31]. It is unknown whether hand-focused gaze would decrease in users who are learning to use multifunction hands due to the additional complexity of multiple grip patterns and grip switching methods.

With this study, we explore the use of eye tracking to track learning progress of multifunction hands at two different time points over one year. The following research questions are explored in this study: (i) can gaze measurements be used to track learning progress in multifunction hands? (ii) how do gaze and use behaviors differ in standardized versus self-chosen tasks?

Methods

Design

An exploratory design was used to explore gaze behavior in two different conditions: while performing standardized tasks and self-chosen tasks using multiple grip patterns in a multifunction prosthetic hand.

Subjects

A convenience sample of three UL amputees (2 right-handed, all male, mean age 47.6) were recruited from the Atlantic Clinic for Upper Limb Prosthetics clinic at the University of New Brunswick, Canada. Inclusion criteria were that subjects were using a myoelectric hand, either conventional (one DOF) or multifunction hand with any control system, that they were at least 19 years old, and that they were able to return to the clinic for follow-up testing. The Research Ethics Board of University of New Brunswick approved the study (REB 2019–009) and all subjects signed informed consent forms before participating in the study.

All subjects recruited for the study lost their dominant hand (2 right and 1 left) after workplace accidents (Appendix 1, Table 4). They had previously experienced direct control of a myoelectric prosthetic device before being enrolled in this study and all were being fitted with a new multifunction hand or a new control system for operating the hand. In this study, Subject 1 and 2 used gesture control for switching grips but Subject 2 used Coapt Pattern Recognition for hand open and close. Similarly, Subject 3 used Coapt Pattern Recognition to control multiple grip patterns of the hand as well as a powered wrist.

Training

All subjects went through basic skills training of learning to open and close the hand, and switch between three to four basic grips and use them to pick up and manipulate objects of varying shapes and sizes. The specific grips used depended on the type of hand being used and the control system being used. For instance, Subject 1 and 2 had the iLimb Quantum hand using gesture control where Subject 3 had a TASKA hand and used pattern recognition to control wrist rotation, hand open, and pinch, key, and power grip. Gesture control allows for grips to be assigned to four different directions of movement. The user must hold an open signal and then move in one of four directions; forward, back, left, or right to access the desired grip. If more options were available for control of the hand, more grips were trained during this initial skills training period.

Skills training included tasks such as: grasping a tennis ball with a spherical grip and putting it down, moving one-inch blocks using a tripod grip, opening a reusable plastic bag using a tripod or pinch grip, holding a drinking glass with a cylindrical grip while pouring water, picking up small objects with a tripod or pinch grip, holding a dinner plate with a lateral grip, and cutting food with fork and knife using a lateral grip (Fig. 1a–c).

Fig. 1
figure 1

a Subject 1: Learning to switch to power grip for a glass. b Subject 2: Learning to switch from tripod grip to power grip for a tennis ball. c Subject 3: Learning to switch from power grip to lateral grip to hold a knife

Full size image

Standardized tasks and self-chosen tasks

The standardized tasks are one part of the Southampton Hand Assessment Procedure (SHAP). It includes moving 12 abstract objects using six different hand grasps: spherical, tripod, power, lateral, tip and extension [17, 32] (Fig. 2). Performance of each task is timed by the subject. The SHAP is performed in a seated position with the multifunction hand in a neutral hand and wrist position at the start.

Fig. 2
figure 2

The standardized tasks—the two sets of abstract objects of different weights (wood and metal) in the Southampton Hand Assessment Procedure

Full size image

The self-chosen tasks are tasks that are used in the Assessment of Capacity for Myoelectric Control (ACMC) [11]. The ACMC assesses movement quality (instead of time completion) using self-chosen tasks. For this study, we only used the ACMC self-chosen tasks for observation purposes. The subjects are free to choose any grip to perform the self-chosen task. The objects are daily common ADL objects, such as utensils, milk bottle, light bulb, lamp stand etc. (Fig. 3a and b). The users are encouraged to perform the self-chosen task in their usual way and at their own pace.

Fig. 3
figure 3

a and b. The self-chosen tasks—breakfast task and assemble a lamp task

Full size image

Eye-tracking measurement

Binocular movements were recorded using Tobii Pro Glasses 2 at a sampling rate of 50 Hz (every 20 ms). The head-mounted eye tracker is a video-based eye-tracking system that records gaze movements of the wearer continuously during use. The system records the gaze positions on the x- and y axis at a sampling frequency of at 50 Hz, while the video is recorded with a 1920 × 1080 px resolution at 25 frames per second [33].

Procedure for gaze data collection

The subjects were asked to wear the Tobii Pro glasses with the appropriate nose pad. Calibration was performed using the built-in one-point target calibration procedure. After calibration, they performed the SHAP using the multifunction hand at baseline (after 1 day training) and at a follow-up visit within one year. All gaze measurements were taken in the same activity room without windows. A table was placed at the same location under a fluorescent ceiling lamp. During the SHAP, the patient was in a seated position with the table set to the appropriate height to allow the elbow to rest at 90 degrees on the table surface. During the self-chosen task, the subject performed the activity while moving around a room, reaching into cupboards and spaces of different heights. Lighting was not consistent during the self-chosen task. The self-chosen tasks were performed after the performance of SHAP.

Data analysis

The eye tracker recordings were imported in the Tobii Pro Lab [34]. The recordings were first inspected to remove unexpected eye movements due to sudden head movements. Area of interest (AOI) and time of interest (TOI) were identified. The multifunction hands and the objects were labelled as different AOIs. The TOI in the SHAP standardized tasks is the time to complete a SHAP object movement. The TOI in the ACMC self-chosen tasks is the time from reaching for one object to just before the prosthetic hand was reaching for the next object.

Four eye tracking metrics, that are commonly used to assess cognitive processes, were chosen to answer research question 1 [35]. Fixation duration measures the temporal aspect of visual behaviour, indicating how long attention is maintained on a specific target [35]. In contrast, spatial measures in eye tracking typically refer to characteristics such as saccade amplitudes (the distance between successive fixations) or fixation count distribution across different areas of interest. In Table 2, we presented the four chosen metrics definitions and their relevance to learning to use a multifunction hand. For research question 2, we used the time to first fixation on either the multifunction hand or the objects to indicate gaze behaviour between standardized and self-chosen tasks.

Table 2 Eye tracking metrics and their relevance in learning multifunction hands
Full size table

Using the Pro Lab, heat maps were generated to show fixation durations on the multifunction hand. Deep red represents long fixation duration; green represents short fixation duration.

Results

Control training

Subjects 1 and 2 learned four different grips whereas Subject 3 learned three different grips (Table 3). Key grip was an easy grip for Subject 1 and 3 whereas it was a difficult grip for Subject 2. Power grip was easy for Subject 2, but it was a difficult grip for Subject 1 and Subject 3. Various reasons contributed to their difficulty, such as Subject 1 had a problem remembering direction in gesture control, and Subject 3 was confused with the pinch grip and power grip in Pattern Recognition. Subject 2 had previously used an iLimb Ultra hand and developed a habit of manually positioning the thumb to change the grip from oppositional to lateral, so gesture control was a significant change in control strategy.

Prosthetic wear time

Subject 1 and 3 wore the prosthetic hand for around 2 and 3–4 h per day respectively (Table 4). Subject 2 wore the prosthetic hand for 9–10 h per day.

Performance time for standardized tasks

Table 5 (the first row) shows the performance time of the standardized tasks at two different time points. After 1 day of training, Subject 1 took 422.58 s to complete the twelve tasks. The power grip for the heavy metal ball took 75.08 s (the longest time). At the one-year follow-up, Subject 1 had an average of 35% decrease in performance time.

After 1 day of training, Subject 2 took 212.33 s to complete the twelve tasks and the key grip for the wooden box took 110.08 s (the longest time). At the one-year follow-up, Subject 2 had an average of 60% decrease in performance time.

Eye tracking metrics during standardized tasks

Table 5 (Appendix 1) also shows total and average gaze measurements during standardized tasks at two different time points. Figure 4 (a,b,c,d) shows the gaze measurements for individual SHAP objects  at two different points. After one day of training, Subject 1 had the longest fixation duration, highest fixation count and longest eye-hand latency among all three subjects. In terms of saccade amplitude, Subject 2 had the lowest saccade amplitude among all three subjects, indicating that cognitive load was highest in Subject 2 during the SHAP standardized tasks.

Fig. 4
figure 4

a–d Fixation duration, fixation count, saccade amplitude and eye hand latency of the 3 subjects. Each dot is one object in the standardised task (total 12 objects)

Full size image

At the one-year follow-up, both Subject 1 and 2 showed a decrease in fixation duration, fixation count and eye-hand latency. An increase in saccade amplitude was observed in both Subject 1 and 2, indicating that it was less mentally demanding for them to control the multifunction hand. Due to socket problem, Subject 3 was not able to operate the multifunction hand at the follow-up appointment and hence no data was collected at that visit.

For Subject 1, the longest fixation duration, highest fixation count, lowest saccade amplitude and longest eye-hand latency were during thinking about grip switching for power grip (for a spherical object), pinch grip (for a tip object) and tripod grip (for a triangle object). For example, the gaze fixation count (numbers of green dots) and fixation duration (size of the green dots) for the power grip are shown after one day training (Fig. 5a) and at one year (Fig. 5b) (also see the videos under supplementary material). After one day training, Subject 1 took 58 s with high numbers of fixation counts and fixation duration to think about switching to the power grip. His fixations were clustered on  the multifunction hand, which led to a relative low saccade amplitude.

Fig. 5
figure 5

a and b Subject 1- gaze behaviour of power grip after 1 day and at 1 year

Full size image

For Subject 2, the longest fixation duration, highest fixation count, lowest saccade amplitude and longest eye-hand latency were during grip switching into the lateral grip (Fig. 6a). Both Subject 1 and 2 have improved in accessing the difficult grips at 1 year follow up (Figs. 5b and 6b).

Fig. 6
figure 6

a and b Heat maps of Subject 2 thinking about switching to lateral grip for the wooden box after 1 day training (time taken: 15 s) and after 1 year (time taken: 4 s)

Full size image

For Subject 3, the longest fixation duration, highest fixation count, lowest saccade amplitude and longest eye-hand latency was during grip switching into the power grip (Fig. 7a) and during grip switching into a lateral grip (Fig. 7b).

Fig. 7
figure 7

a and b Heat maps of Subject 3 thinking about switching to power grip for the wooden ball (Time taken: 36 s) and switching to lateral grip for the wooden box (time taken: 4 s) after 1 day training

Full size image

Differences in gaze and use behaviors in standardized versus self-chosen tasks

Frequency of changing grips

In the standardized tasks, the subjects were required to switch from a neutral hand position to a desired grip for grasping two sets of 6 objects of different shapes. In the self-chosen tasks, the subjects were allowed to use any grip they liked. Subject 1 changed grips at two different times in the self-chosen task (once for grasping the cereal box with the power grip and once for grasping the tablecloth with the pinch grip). Subject 2 changed grips at 4 different times (pinch grip for the cable on two occasions, tripod grip for the light bulb on two different occasions). Subject 3 changed grips 21 times for grasping all different objects at different occasions.

Time to first fixation for hand versus objects

Table 6 (Appendix 1) shows the time to first fixation on the multifunction hand and  on the objects. During the standardized tasks, the first fixation of all three subjects were on the multifunction hand in all objects. During the self-chosen tasks, the first fixations were mostly on the objects first.

Discussion

The present study explored the use of both qualitative data from control training and quantitative eye tracking data from clinical standardized tasks to understand and measure cognitive processing and gaze behaviour in learning to control a multifunction hand. Two different multifunction hands and two kinds of control strategies were being used. Two subjects showed improvements, with a decrease in performance time, fixation duration, fixation count and eye hand latency.

To our best knowledge, this is the first multifunction hand study that combines the data from clinical control training and eye tracking data from clinical standardized tasks. Compared with the number of studies on prosthetic components, literature on learning to control a multifunction hand in a clinical setting is sparse. This detailed report of control training and clinical findings are from a therapist perspective. Traditionally, therapist researchers perform research in prosthetic training whereas engineers perform research in prosthetic technologies [38]. The findings of control training provide valuable information on the importance of control strategies for multifunction hands. For example, as shown in Fig. 5a, the high fixation counts, long fixation durations and low saccade amplitude of Subject 1 occurred when he was thinking about switching to the power grip using gesture control. His fixation counts and durations were similar after he had switched to the power grip. This may suggest that thinking about the control of the multifunction hand is more cognitively demanding than the action of moving the hand. Similarly, Subject 2 was thinking about switching between two different grips when he showed the lowest saccade amplitude. An intuitive control strategy may contribute to the ease of controlling the prosthetic hand with multiple  degree of freedom.

Previous studies using eye tracking metrics were investigated in able-bodied individuals or amputees using single DOF prostheses [28,29,30,31, 39]. In this study, we aim to delve deeper into the use of gaze measurements among users of prosthetic hands that have multiple DOF prosthetic hands. The findings in fixation duration during the standardized tasks are consistent with previous studies on single DOF prostheses [29,30,31]. After one day of training, the prosthetic users were still in the cognitive process of understanding the new hand and learning to control it. At the follow-up testing, both Subject 1 and 2 showed a 30% and 60% decrease in fixation duration respectively. This suggests that they were moving towards the autonomous stage, as proposed by the learning theory by Fitts and Posner [1].

In contrast to previous studies, our study is the first to explore cognitive processing using fixation duration and saccade amplitude on UL amputees. Previous studies had used fixation duration to indicate visual attention, skill level and spatiotemporal disruption in able-bodied individuals [29,30,31, 39]. In the neuroscience field, fixation duration has been suggested to surpass pupil size as a measure of memory load [24]. Pupil size was not a reliable measure for self-chosen real-life tasks because real life tasks are seldom performed in one position and the pupil size will be affected by different light conditions.

Saccade amplitude is the distance travelled by the eye between two fixation points and it increases if the difficulty decreases [25]. All subjects learned 3 to 4 grips during control training, and they had been practising in their home environment. Similarly, both subjects also showed a 24% and 138% increase respectively in saccade amplitude. After one day of training, both subjects stared at the hand (i.e. one position) during grip switching. At the one-year follow-up, however, they looked around at different objects and the hand during the SHAP standardized tasks. Although more research using other outcome measures with a large sample is needed, our findings on shorter fixation durations and larger saccade amplitudes at 1 year are likely attributed to reduced mental processing or memory load, indicating lesser difficulty in controlling the multifunction hands after one year. From a learning perspective, as proposed by Fitts and Posner [1], during the autonomous stage, the user keeps practising using their prosthesis in familiar self-chosen daily tasks until his/her performance enters an automatized routine. An earlier study showed that full time prosthetic users (8 h or more) acquired the highest ability to control single DOF prostheses [40]. Subject 1 used his multifunction hand for 2 h per day whereas Subject 2 used his hand for 9–10 h per day. Subject 1's relatively low improvement, compared to Subject 2, may be attributed to his limited 2 h of actual use of the prosthetic hand.

Eye-hand latency is well-investigated in developmental neuroscience because it provides valuable insights into the coordination between visual processing and motor responses. Similar to previous studies [30, 41], a decrease in eye-hand latency was observed in both Subject 1 and 2 during the one-year- follow up, suggesting that a stronger coupling between visual processing and prosthesis use existed at the one-year follow-up than after one day of training. When there is a strong coupling between visual processing and prosthesis use, it means that the subjects were effectively utilizing visual feedback to guide and adjust their prosthetic grips [30].

Fixation count on the prosthetic hand is a spatial measure that shows the distribution of fixations. From clinical experience, we know that new users look at their hands during operation and when they become experienced users, the need for visual feedback will be much less [42]. In our subjects, we observed high fixation counts with long fixation durations after one day of training, which may indicate sustained attention of the multifunction hand. A decrease in fixation count at the one-year follow-up suggests the need for visual feedback has decreased.

The second question we asked was “how do gaze and use behaviors differ in standardized versus self-chosen tasks?”. Previous studies using standardized tasks showed that the gaze of able-bodied individuals and prosthesis users of single DOF were mostly on the prosthesis [28,29,30,31, 41]. Our findings showed that users of multifunction hands first fixated on the objects in the self-chosen tasks whereas their fixation first on the hand in the standardized tasks. This gaze difference has an important implication for assessment purposes. Standardized tasks are good for research purposes but if the purpose is to measure real life functional benefits, self-chosen tasks will provide a more valid assessment in terms of visual attention.

In terms of use behaviour, the frequency of grip switching during self-chosen tasks was much lower in Subject 1 compared to Subject 2 and 3. One possible reason could be that Subject 1 used gesture control and previous research suggested that gesture control requires too much time for functional grip switching [27]. Subject 3 used pattern recognition as a control strategy and he switched grips multiple times during the self-chosen task. The pattern recognition control system offers the ability to control multiple movements in a relatively seamless manner [43] and it may contribute to the ease of switching grips in Subject 3, although a larger sample with a pre-post design using other outcome measures is needed to further investigate the benefit of different control strategies.

Clinical implications

Despite being an exploratory study with 3 subjects, we would like to address several implications here. Clinically, we notice patients tend to stick to a limited number of grips in real-life situations and self-chosen tasks. Gaze measurement data supports this observation, indicating that switching grips is more cognitively demanding. While standardized tasks may show improved grip switching with practice, in self-chosen tasks, patients often refrain from frequent grip changes. We speculate that prosthesis users often develop habits based on the control method or hand they learned first (for example, in Subject 2, he learned to manually position the thumb to switch grips), or that certain control strategies require more effort or are time-consuming (like gesture control), leading them to stick to one or two basic hand grips. Their focus appears to be more on completing a given task rather than controlling the hand in specific ways. This reflects a more accurate picture of how they use their hands at home.

Using eye tracking metrics, we have shown grip switching is cognitively demanding, yet this skill can improve with time. Grip switching is a new skill for amputees with experience of single DOF prostheses. Functional tests serve as valuable tools, offering a swift assessment of our clients’ abilities. However, functional tests have limitations, such as the SHAP test demonstrates floor effect [17], and eye tracking metrics may not be bounded by floor effect although more research with a diversity of prosthesis users are needed to investigate this.

Methodological considerations

This exploratory study had 3 subjects and we have demonstrated the use of eye tracking to measure changes in control of multifunction hands. While eye tracking glasses offer valuable insights into visual behavior and cognitive processes, they also have several limitations. Although the subjects wore the glasses in both standardized tasks and in self-chosen tasks, we chose to present fixation and saccade metrics in standardized tasks only. This was because it was difficult to obtain accurate fixation and saccade metrics when the subjects were walking around in the activity room.

Calibration errors or technical issues can affect the reliability of eye tracking data. We did not have any calibration errors; however, the Tobii glasses sometimes stopped in the middle of the recording for unknown reasons. The live view during recording was helpful to notice recording errors.

Conclusion

With this study, by analyzing gaze behaviors during operation of two types of multifunction hands, we gained insights into mental processing, memory load and attentional processes. By integrating eye tracking into prosthetic rehabilitation programs, clinicians can optimize device control, enhance user experience, and promote successful outcomes of multifunction hands.