Effect of Cutaneous Feedback on the Perception of Virtual Object Weight during Manipulation

Abstract

Haptic interface technologies for virtual reality applications have been developed to increase the reality and manipulability of a virtual object by creating a diverse tactile sensation. Most evaluation of the haptic technologies, however, have been limited to the haptic perception of the tactile stimuli via static virtual objects. Noting this, we investigated the effect of lateral cutaneous feedback, along with kinesthetic feedback on the perception of virtual object weight during manipulation. We modeled the physical interaction between a participant’s finger avatars and virtual objects. The haptic stimuli were rendered with custom-built haptic feedback systems that can provide kinesthetic and lateral cutaneous feedback to the participant. We conducted two virtual object manipulation experiments, 1. a virtual object manipulation with one finger, and 2. the pull-out and lift-up of a virtual object grasped with a precision grip. The results of Experiment 1 indicate that the participants felt the virtual object rendered with lateral cutaneous feedback significantly heavier than with only kinesthetic feedback (p < 0.05 for m_ref = 100 and 200 g). Similarly, the participants of Experiment 2 felt the virtual objects significantly heavier when lateral cutaneous feedback was available (p < 0.05 for m_ref = 100, 200, and 300 g). Therefore, the additional lateral cutaneous feedback to the force feedback led the participants to feel the virtual object heavier than without the cutaneous feedback. The results also indicate that the contact force applied to a virtual object during manipulation can be a function of the perceived object weight (p = 0.005 for Experiment 1 and p = 0.02 for Experiment 2).

Introduction

The recent growth of the virtual reality industry calls for the means to let a user vividly feel and naturally interact with the virtual environment. In this context, various haptic technologies have been developed to increase the reality and manipulability of a virtual object by creating a diverse tactile sensation. A typical way of providing a tactile sensation to the user is kinesthetic feedback, which applies an interaction force between the user and a virtual object with a robot arm, called a force feedback interface. Most of force feedback interfaces allow the user a 3-degree-of-freedom (3DOF) or 6-degree-of-freedom (6DOF) motion and provide kinesthetic feedback to a tool or a stylus/stylus that s/he is holding during virtual interaction^1,2,3,4. A specific form of the force feedback interface is a glove-type or exoskeleton force feedback interface, which can represent contact force to the user’s fingertips^5,6,7,8,9. This type of haptic interface has the advantage of letting the user directly feel the contact at his/her fingers while minimizing the constraint on hand movement. Another form of haptic interface is a cutaneous or fingertip haptic interface that stimulates the user’s skin at the fingertip¹⁰. This type of haptic device is developed to emulate the sensation of touching a real object by stimulating RA, SA1, and SA2 type mechanoreceptors^11,12,13,14. Thus, tactile cues imparted from a cutaneous interface include contact location information^15,16,17, the orientation of contact area^18,19, and the change of contact area^20,21. Most of the aforementioned haptic feedback interfaces have been evaluated in terms of improving the perception of a static virtual object’s tactile properties. On the contrary, little is known about the effect of haptic feedback on the tactile perception during object manipulation.

There are theoretical and experimental grounds that the addition of cutaneous stimulus can lead to a change in the perception of an object’s physical properties. According to the studies on multi-modal sensory integration, the addition of sensory cue with different modality can result in the shift of percept when the different types of information integrate optimally. Such optimal sensory integration is found in various combination of sensory cues, e.g., visual-haptic^22,23,24, intra-visual²⁵, or audio-visual integration^26,27. Narrowing the range of modalities to only haptic cues, additional cutaneous feedback, along with kinesthetic cues, are known to affect the haptic perception of physical properties such as friction²⁸ or surface stiffness^21,29 or object size³⁰, as well as hand kinesthesia^31,32. However, fewer studies have been conducted to evaluate the relative role of cutaneous information in object weight perception, which is a crucial component in manipulation^33,34,35.

Multiple studies indicate that lateral cutaneous feedback at fingertip provides information on an object’s weight. When one lifts an object with a precision grip, tangential torque is applied to fingertip skin depending on object weight^36,37,38. Noting that the load torque is a function of object weight, a strong correlation is expected between weight perception and tangential torque at the fingertip. Also, cutaneous feedback to the fingertip is known to create an illusory perception of force with or without kinesthetic feedback^39,40,41,42. Considering its strong correlation to force sensation, we can expect weight perception is affected by cutaneous feedback. The focus of the studies above was mainly on either the relation between object weight and tangential torque profile or the effect of lateral skin stretch feedback that substituted kinesthetic feedback. In the meanwhile, it is hard to find studies on the relative role of lateral cutaneous feedback on object weight perception during manipulation in the view of multi-modal sensory integration.

This study explores the extent that additional lateral cutaneous feedback to kinesthetic feedback affects the perception of virtual object weight during manipulation. Previous studies on haptic feedback mainly focused on its effect on haptic perception of static object’s surface properties. On the other hand, the present study analyzes the effect of cutaneous feedback on the weight perception during object manipulation. If we assume that the tactile information from cutaneous and kinesthetic feedback integrate optimally, rendering both cutaneous and kinesthetic feedback can shift the perceived object weight from the one represented only with kinesthetic feedback. We, therefore, hypothesize that the perception of a virtual object weight during manipulation can be affected by the addition of lateral cutaneous feedback. Also, varying the rate of providing cutaneous feedback may affect the perception of virtual object weight. We tested our hypothesis by evaluating the effect of cutaneous feedback on the perceived weight of a virtual object. The experimental task is to judge the perceived weight of a virtual object, manipulated with one or two fingers (Fig. 1). A participant of the experiment could feel the weight of the virtual object rendered with both lateral cutaneous feedback and kinesthetic feedback. Then, we evaluated the effect of the additional lateral cutaneous feedback on the perceived weight. The next section describes the experimental setup for virtual object manipulation and weight perception.

Figure 1

Description of experimental tasks to evaluate the effect of cutaneous feedback on the perception of virtual object weight. (a) Lateral manipulation of a virtual object on the floor with one finger, and (b) Lifting a virtual object grasped with a precision grip.

Methods

The experimental tasks of the present study were to move a virtual object with an index finger laterally, and to pull-out and lift-up a virtual object grasped with a precision grip. Since tactile information imparted to a user differs by the type of manual task, we built two different experimental setups. This section describes the system overview and haptic rendering algorithm for the two experiments. Then, we illustrate the experimental paradigm and procedure to evaluate the effect of lateral cutaneous feedback on the perceived object weight.

All methods were previously approved by the KIST (Korea Institute of Science and Technology) Institutional Review Board (IRB approval No. 2019-007) and carried out in accordance with the Declaration of Helsinki for research involving human subjects. Informed written consent was obtained from all participants involved in the experiments.

Experiment 1: rendering haptic feedback during virtual object manipulation with one finger

For the experiment with one finger object manipulation in Fig. 1(a), we set up a virtual environment where a user can manipulate a virtual cube with one finger by feeling the contact force with a haptic system. Figure 2 shows the overall experimental system architecture, including the virtual environment, controller, and physical environment. The haptic interface consists of a force feedback interface and a custom-built cutaneous feedback interface (Fig. 2(a)). The cutaneous feedback interface is worn around the user’s index fingertip beneath which a contact plate is located. When the user laterally moves a virtual object, the fingertip skin is deformed by moving the contact plate with a servo motor (model HV75K, MKS Servo Tech, Ilan, Taiwan). The rotation of a joint of the cutaneous interface or the motor is tracked with a magnetic rotary encoder (RM08, RLS, Slovenia). Being attached to the force feedback interface, the user can feel both cutaneous and kinesthetic feedback simultaneously.

Figure 2

The haptic system architecture for Experiment 1. The interaction force between the user’s fingertip avatar and virtual environment is calculated with a virtual interaction model, including the lateral friction (Virtual Environment). Next, the controller transfers adjusted stimuli values to the haptic interface (Controller). Then, a user can perceive the haptic stimuli from the haptic interface providing lateral cutaneous and kinesthetic feedback (Physical Environment). (a) A cutaneous interface renders lateral friction by moving a contact plate placed beneath the user’s fingertip. (b) The virtual interaction model calculates the contact force (F_n and F_f) and cutaneous feedback (d_f). (c) Experimental apparatus to provide the cutaneous and kinesthetic feedback to a participant’s fingertip during object manipulation with an index finger.

During the experiment, a user’s fingertip is mapped to a virtual environment and represented as a fingertip avatar (Fig. 2(b)). When the fingertip avatar touches a virtual object, the normal contact force F_n is calculated based on a virtual proxy based haptic rendering model^43,44 as follows:

$${{\boldsymbol{F}}}_{n}=K({{\boldsymbol{x}}}_{p}-{{\boldsymbol{x}}}_{f}),$$

(1)

where K, x_p, and x_f denote virtual object surface stiffness, fingertip proxy contact position, and the fingertip contact position, respectively. Haptic rendering of the lateral friction force is calculated from a contact model based on a Coulomb friction law. A direct application of the Coulomb friction model results in a sudden change of friction force and thus can cause the haptic feedback system unstable. Thus, we modeled the state of static contact and sliding by considering the Coulomb friction model and the velocity of the fingertip based on the piecewise linear approximate Coulomb friction model^45,46. We calculate the relative lateral speed of fingertip v_fo,t to the virtual object at timestamp t as

$${{\boldsymbol{v}}}_{fo,t}={{\boldsymbol{v}}}_{f,t}-{{\boldsymbol{v}}}_{o,t-1},$$

(2)

where v_f,t and v_fo,t−1 are fingertip speed at timestamp t, and virtual object speed at timestamp t − 1, respectively. By incorporating the Eqs. 1 and 2 with a piecewise linear approximate Coulomb friction model, the following friction state model is derived as:

$${{\boldsymbol{F}}}_{f}=\left\{\begin{array}{lll}-{\mu }_{a,s}|{{\boldsymbol{F}}}_{n}|\frac{{{\boldsymbol{v}}}_{fo,t}}{{v}_{th}} & |{{\boldsymbol{v}}}_{fo,t}| < {v}_{th} & ({\rm{Static}}\,{\rm{Contact}})\\ -{\mu }_{a,d}|{{\boldsymbol{F}}}_{n}|sgn({{\boldsymbol{v}}}_{fo,t}) & {\rm{otherwise}} & ({\rm{Sliding}})\end{array}\right.,$$

(3)

where ${\mu }_{a,s}$, ${\mu }_{a,d}$, and ${v}_{th}$ are static and dynamic friction coefficients between the virtual fingertip, and state switching threshold speed, respectively. We modeled the tangential force between the virtual object and the virtual surface based on the Coulomb friction law as follows:

$${{\boldsymbol{F}}}_{o}=\left\{\begin{array}{ll}-{{\boldsymbol{F}}}_{f} & {\rm{if}}\,{{\boldsymbol{v}}}_{O}=0\,{\rm{and}}|{{\boldsymbol{F}}}_{f}|\le |{\mu }_{b}({m}_{obj}{\boldsymbol{g}}-{{\boldsymbol{F}}}_{n})|\\ -|{\mu }_{b}({m}_{obj}{\boldsymbol{g}}-{{\boldsymbol{F}}}_{n})|sgn({{\boldsymbol{F}}}_{f}) & {\rm{otherwise}}\end{array}\right..$$

(4)

where m_obj is the virtual object weight. Equations 3 and 4 decide whether the virtual objects moves or not. When the virtual object moves, its position is updated along with its speed. The force to be exerted by the force feedback, ${{\boldsymbol{F}}}_{kin}={{\boldsymbol{F}}}_{n}+{{\boldsymbol{F}}}_{f}$ is a vector summation of the lateral and normal contact force, calculated as

$${{\boldsymbol{F}}}_{kin}={{\boldsymbol{F}}}_{n}+{{\boldsymbol{F}}}_{f}.$$

(5)

On each servo loop, the calculated lateral contact force F_kin is fed to the controller and the force feedback interface. The desired displacement of the contact plate on the cutaneous interface is calculated as ${d}_{f}={l}_{f}|{{\boldsymbol{F}}}_{f}|sgn({{\boldsymbol{F}}}_{{\boldsymbol{f}}})$. Then, the contact plate is positioned to the desired displacement with a PID controller. The output torque from the controller is converted to a PWM signal generated by a multifunction board (Model 826, Sensoray Co. Inc., OR, U.S.A) and is sent to a motor controller, connected to the servo motor of the cutaneous interface.

Figure 2(c) shows the experimental apparatus that can render both lateral cutaneous feedback and force feedback, as a user moves a virtual object with an index finger. The interaction force due to the friction between a fingertip avatar and a virtual object was rendered for force and cutaneous feedback (Fig. 2(c)). The force feedback interface directly applied the calculated force F_kin with a commercially available force feedback interface (PHANToM Premium 1.0, 3D Systems, SC, USA). The custom-built haptic interface (Fig. 2(a)) rendered lateral cutaneous feedback by moving a contact plate placed beneath a participant’s fingertip.

Experiment 2: rendering haptic feedback during virtual object manipulation with a precision grip

For the experiment with two-finger object manipulation in Fig. 1(b), we set up a virtual environment where a user can feel when lifting a virtual cuboid grasped with a precision grip. Figure 3 shows the overall haptic system architecture, including virtual environment, controller, and physical environment. Two force feedback interfaces provides a user with kinesthetic feedback as s/he grasps a virtual object with a precision grip. A cutaneous feedback interface (Fig. 3(a)) was attached to the end-effector of each force feedback interface and a contact skin rotates fingertip skin, which emulates the torsional skin deformation as an object rotates due to its weight. A magnetic rotary encoder (RM08, RLS, Slovenia) with the resolution of +0.3 deg is attached at the bottom of the contact disk to track the disk’s rotation angle. A pair of springs tightly fixes the haptic interface to a user’s fingertip at the distal interphalangeal (DIP) joint.

Figure 3

The haptic system architecture for Experiment 2. The interaction force between the user’s fingertip avatars and virtual environment is calculated with a virtual interaction model, including the torsional friction (Virtual Environment). Next, the controller transfers adjusted stimuli values to the haptic interface (Controller). Then, a user can perceive the haptic stimuli from the haptic interface providing torsional cutaneous and kinesthetic feedback (Physical Environment). (a) A cutaneous interface renders torsional friction by moving a contact disk touching the user’s fingertip. (b) The virtual interaction model calculates the contact force (F_kin) and the cutaneous feedback generated by rotating the contact disk. (c) Experimental apparatus to provide the cutaneous and kinesthetic feedback to a participant’s fingertip. A contact plate on the cutaneous interface rotates to provide torsional friction feedback to the fingertip as the virtual object rotates.

Haptic rendering of the force and cutaneous feedback for manipulating an object with a precision grip was calculated in a similar way to Experiment 1. When a user holds a cuboid with his/her index finger and thumb (Fig. 3(b)), we can calculate the frictional torque at the surface of the contact point (red circle in the figure) as ${\tau }_{f}=\frac{2}{3}{\mu }_{s}{r}_{disk}|{{\boldsymbol{F}}}_{n}|$ (μ_s: friction coefficient, r_disk: rotating disk radius, and F_n: normal contact force)⁴⁷. Then, applying the Coulomb friction model⁴⁸ on the torque, we get the friction model as follows:

$${{\boldsymbol{\tau }}}_{f}=\left\{\begin{array}{ll}-{{\boldsymbol{\tau }}}_{e} & {\rm{if}}\,\omega =0\,{\rm{and}}|{{\boldsymbol{\tau }}}_{e}|\le \frac{2}{3}{\mu }_{s}{r}_{disk}|{{\boldsymbol{F}}}_{n}|\\ -\frac{2}{3}{\mu }_{s}{r}_{disk}|{{\boldsymbol{F}}}_{n}|sgn({{\boldsymbol{\tau }}}_{e}) & {\rm{otherwise}}\end{array}\right.,$$

(6)

where τ_e and x_r are the torque due to the object weight and the displacement vector from the contact point to the center of mass, respectively. F_n is the normal contact force in Eq. 1. Then, total torque at the fingertip contact point τ_tot is

$${{\boldsymbol{\tau }}}_{tot}={{\boldsymbol{\tau }}}_{e}+{{\boldsymbol{\tau }}}_{f}\cdot $$

(7)

We can calculate the rotation of the contact plate from ${{\boldsymbol{\tau }}}_{tot}=I{\boldsymbol{\alpha }}$ (I: the moment of inertia and α: angular acceleration), and ω with Euler integration. The rotation of the contact plate is then updated as follows:

$${\theta }_{i+1}={\theta }_{i}+\omega \delta t,$$

(8)

where i denotes the timestamp. The force to be exerted by each force feedback device in the vertical direction is calculated as ${{\boldsymbol{F}}}_{v}=(0.5{m}_{obj}-{m}_{tip}){\boldsymbol{g}}({m}_{obj}:{\rm{virtual}}\,{\rm{object}}\,{\rm{weight}},\,{\rm{and}}\,{m}_{cut}:{\rm{cutaneous}}\,{\rm{feedback}}\,{\rm{interface}}\,{\rm{weight}})$ to avoid the effect of haptic weight interface on the perception of virtual object weight. Then, total force to be exerted by each force feedback interface F_kin is,

$${{\boldsymbol{F}}}_{kin}={{\boldsymbol{F}}}_{v}+{{\boldsymbol{F}}}_{n},$$

(9)

where F_n is the normal contact force calculated by the Eq. 1.

Figure 3(c) shows the experimental apparatus that can provide kinesthetic feedback and cutaneous feedback to the participant. Two commercially available force feedback interfaces (Touch, 3D Systems, SC, USA) directly rendered the force vector F_kin as a vector sum of normal-directional contact force F_n and the weight of the virtual cuboid, F_v. A cutaneous interface (Fig. 3(a)) is attached to the end-effector of each force feedback interface to provide cutaneous feedback by rotating contact plate.

Friction coefficients of real object surfaces

We measured the friction coefficients between real objects to estimate the degree of slipperiness/roughness of the virtual friction coefficient for the experiments. The friction coefficient was measured for three surface materials (metal, plastic, and silicon), and two sliding object materials (metal, and silicon). We employed a typical inclined slope method to measure the static friction coefficient, where the friction coefficient was decided as a tangent of the angle that the object begins to slide. For the sake of measurement accuracy, we attached markers on the sides of the slope, video-recorded the sliding, and derived the sliding angle by analyzing the image at the moment of sliding. The measurement was conducted three times for each condition. The static friction coefficients between two materials were 0.21 ± 0.01 (metal-metal), 0.16 ± 0.03 (metal-plastic), 0.65 ± 0.12 (metal-silicon), 0.43 ± 0.03 (silicon-plastic), and 0.55 ± 0.11 (silicon-silicon).

Experimental procedure

For the experiments, we employed a one-up one-down adaptive procedure to match the perceived weight of a virtual object rendered with both cutaneous and kinesthetic feedback to that rendered only with kinesthetic feedback⁴⁹. The adaptive procedure estimates the PSE of comparison stimuli that is perceived to be equivalent to the reference stimulus.

Each participant conducted four (six) experimental runs by reference object weight and l_f for Experiment 1 (μ_s for Experiment 2). The order of reference stimuli was randomized for each participant. On each trial, a participant was presented with two virtual objects: a reference object rendered both with cutaneous and kinesthetic feedback and a comparison object rendered only with force feedback. If the participant responded that the reference object felt heavier than the comparison plane, the weight of the comparison plane decreased. Otherwise, the weight of the comparison plane increased. The initial weight was 400 g and 300 g for Experiments 1 and 2, respectively, considering the stability of the force feedback interface used for each experiment. The step size of increasing/decreasing the weight changed from 100 g to 25 g after the first three reversals of the responses. Each experimental run terminated after 12 reversals of the responses at the smaller step size. The total number of trials for one experimental run typically ranged between 30 and 40.

Before the main experiment, a participant was seated in front of a computer and asked to wear noise-canceling headphones (MDR10RNC, Sony, Tokyo, Japan). Then, s/he inserted his/her index finger inside cutaneous interface (Figs. 2(a) and 3(a)). Then, a training session was initiated where the participant could feel the weight of a virtual object rendered with both cutaneous and force feedback or only with force feedback. Virtual fingertips were visually displayed as green spheres along with the virtual object. When the participant felt ready for the main experiment, the training session terminated.

In the main experiment, the order of presenting reference and comparison stimuli was randomly decided on each trial. In the beginning, fingertip avatars and a virtual object were displayed on the screen. In Experiment 1, a participant was asked to laterally move the virtual object with an index finger and feel the object weight. In Experiment 2, a participant was asked to grasp and lift up the cuboid laid on a virtual plane with a precision grip and feel the object weight. During the main experiment, white noise was played on headphones to block possible audio cues from the haptic interface. By hitting the enter key, the participant moved to the next phase to feel the other stimulus. Then, the participant was asked to indicate which stimulus felt heavier. For each trial, the magnitude of comparison stimulus, the answer, collision depth to derive the contact force, and trial time were recorded. After each experimental run, the participant took a 5-min break to minimize the fatigue from the experiment.

Results

We conducted two psychophysical experiments by how to manipulate the object (Fig. 1), to investigate the effect of the cutaneous feedback on the perceived object weight. In the first experiment, participants moved a virtual object on a floor sideways by feeling cutaneous and kinesthetic feedback, to perceive the object weight. The second experiment evaluated the effect of cutaneous feedback on the perceived weight of a virtual object grasped and lifted with a precision grip. For both experiments, we also investigated how the rate of providing cutaneous feedback affects the perceived weight of the virtual object. The following subsections describe the experimental setup and report experimental results.

Experiment 1: Perceived weight of a virtual object manipulated with one finger

The goal of this experiment is to evaluate the effect of cutaneous feedback on the perceived weight of a virtual object manipulated with an index finger. Total 12 subjects participated in the experiment which matched the perceived weight of a virtual object rendered both with cutaneous and kinesthetic feedback to the one rendered with kinesthetic feedback only.

In the main experiment, a participant laterally manipulated a virtual cube (dimension: 3 cm × 3 cm × 3 cm) with a weight of 100 and 200 g, which is about as heavy as a hand-held device, with an index finger. We decided the weight of the virtual object to be around that of a handheld device while conserving the stability of the force feedback interface. The friction coefficients were 0.74, 0.57, and 0.25 for fingertip-object static friction (μ_a,s), fingertip-object dynamic friction (μ_a,d), and object-floor friction respectively (μ_b). We intentionally chose the friction coefficients for fingertip-object (∼silicon-metal) to be higher than that for object-floor (∼metal-metal). Then, the object can slide when sufficient normal and lateral force is applied. For the rendering of cutaneous feedback, we define l_F, which is the ratio of contact plate displacement to the lateral force. We used two l_F values, 0.005 and 0.01 m/N for the experiment to evaluate the effect of cutaneous feedback intensity on the perceived weight of a virtual object.

A one-up one-down adaptive procedure was employed to derive the point of subjective equality (PSE) for a reference stimulus rendered with cutaneous and kinesthetic feedback, matched to the weight of a virtual object rendered only with kinesthetic feedback. Both cutaneous and kinesthetic feedback was provided to a participant for the reference stimuli while only kinesthetic feedback was provided for the comparison stimuli.

Figure 4(a) shows the average PSE estimate of a virtual object weight plotted as a function of reference object weight by two l_F values. We compared the PSE estimate to the reference object weight, to see the effect of cutaneous feedback on the perceived object weight. The results of one-sample t-test (${H}_{0}:{\mu }_{PSE} > {m}_{ref}$ where m_ref means the reference weight) indicate that the PSE estimates are significantly larger than the reference object weight for all experimental conditions [$t(11)=3.73,p=0.0017$ for l_F = 0.005 m/N, m_ref = 100 g; $t(11)=2.69,p=0.011$ for l_F = 0.005 m/N, m_ref = 200 g; $t(11)=2.23,p=0.023$ for l_F = 0.01 m/N and m_ref = 100 g; $t(11)=2.0,p=0.035$ for l_F = 0.01 m/N, m_ref = 200 g]. Thus, the cutaneous feedback to the fingertip led the participants to feel virtual object weight heavier than when there is only kinesthetic feedback. When we conducted a two-way repeated measures ANOVA on the PSE estimates with the factors of reference weight and l_F, only m_ref had a significant effect on the PSE estimates [$F(1,11)=0.223,p=0.646,{\eta }^{2}=0.02$ for l_F; $F(1,11)=90.95,p < 0.001,$${\eta }^{2}=0.892$ for m_ref]. No significance interaction between the two factors was found [$F(1,11)=1.447,$$p=0.254,{\eta }^{2}=0.116$]. Overall, the addition of lateral cutaneous feedback led the participants to feel a virtual object heavier than the one rendered with only kinesthetic feedback. The modulation of l_F did not affect the perception of virtual object weight.

Figure 4

The results of Experiment 1. (a) Mean estimated PSE of perceived virtual object weight by l_F and reference weight. (b) Mean of the maximum normal-directional contact force by l_F, and reference weight. (c) Mean of the maximum lateral contact force by l_F, and reference weight. Error bars indicate standard errors.

Figure 4(b) shows the mean of the maximum normal-directional contact force by l_F as a function of reference object weight, m_ref. We conducted a two-way repeated measure ANOVA to determine the effects of l_F and m_ref on the maximum normal-directional contact force. We found no significant interaction of the two factors [$F(1,11)=0.31,p=0.589$, ${\eta }^{2}=0.027$]. m_ref had a significant main effect on the maximum normal-direction directional contact force [$F(1,11)=11.94,p=0.005,{\eta }^{2}=0.52$] meaning increasing trend of the maximum normal-directional contact force with the increase of m_ref. In the meanwhile, the effect of l_F was insignificant [$F(1,11)=1.14,p=0.31,{\eta }^{2}=0.09$]. In Fig. 4(c), the mean of the maximum lateral contact force is shown by l_F as a function of reference object weight. When we conducted a two-way repeated measure ANOVA with the factors l_F and m_ref, no significant two-way interaction was found between the two factors [$F(1,11)=0.046,$$p=0.84,{\eta }^{2}=0.004$]. The effect of m_ref was significant on the maximum lateral contact force [$F(1,11)=7.37,$$p=0.02,{\eta }^{2}=0.4$], which implies that the maximum lateral contact force increased with the increase of m_ref. l_F had no significant main effect [$F(1,11)=1.19,p=0.3,{\eta }^{2}=0.111$].

Overall, the addition of lateral cutaneous feedback to force feedback resulted in the increase of the perceived weight of a virtual object during one-finger manipulation. It is also notable that the reference object weight significantly affected both the normal and lateral force contact force. We discuss the implication of the findings in the discussion section.

Experiment 2: Perceived weight of a virtual object grasped and lifted with a precision grip

The goal of this experiment is to examine the effect of torsional cutaneous feedback on the weight perception of a virtual object held with precision grip during object pull-out and lift-up task. A new group of 12 subjects participated in the experiment, which matched the perceived weight of a virtual cuboid with the dimension of 6 cm × 5 cm × 14 cm, rendered both with cutaneous and kinesthetic feedback to the one rendered with kinesthetic feedback only. On each trial of the main experiment, a participant was asked to pull out a virtual cuboid attached to a virtual wall, by grasping the object with a precision grip. Then, the virtual object tilted forward, and the participant could feel its weight rendered with kinesthetic and cutaneous feedback.

The weight of the virtual cuboid is 100, 200, and 300 g, which are about as heavy as a hand-held electronics including a smartphone or a tablet. To see the effect of varying cutaneous feedback on the perceived object weight, we used two values for the friction coefficient μ_s between a participant’s fingertip and the cuboid, 0.35 (∼silicon-plastic) and 0.7 (∼silicon-metal). As was in the first experiment, we used a one-up one-down adaptive procedure to derive the point of subjective equality (PSE) for a reference stimulus rendered with cutaneous and kinesthetic feedback, matched to the weight of a virtual object rendered only with kinesthetic feedback.

In Fig. 5(a) are shown the average PSE estimates of a virtual object weight plotted as a function of reference object weight by the two μ_s values. When we compared the PSE estimates to the reference object weight with one-sample t-test (${H}_{0}:{\mu }_{PSE} > {m}_{ref}$), the PSE estimates are significantly larger than the reference object weight for all μ_s values and reference weight [$t(11)=7.72,p < 0.0001$ for ${\mu }_{s}=0.02$ and m_ref = 100 g; $t(11)=4.79$ and $p=0.0002$ for ${\mu }_{s}=0.02$ and m_ref = 200 g; $t(11)=2.43$ and $p=0.017$ for μ_s = 0.02 and m_ref = 300 g; $t(11)=7.61$ and $p < 0.0001$ for μ_s = 0.04 and m_ref = 100 g; $t(11)=5.09$ and $p=0.0002$ for ${\mu }_{s}=0.04$ and m_ref = 200 g; $t(11)=2.43$ and $p=0.001$ for μ_s = 0.04 and m_ref = 300 g]. The result is, therefore, consistent with Exp. 1, in that the cutaneous feedback to the fingertip led the participants to feel virtual object weight heavier than when there is only kinesthetic feedback. When a two-way repeated measures ANOVA on the PSE estimates with the factors μ_s and m_ref on the PSE estimates, a significant interaction was found between the two factors [$F(2,22)=3.65,p=0.04,{\eta }^{2}=0.25$]. Simple main effects analysis shows that if m_ref = 100 g, the participants felt the virtual object heavier for μ_s = 0.02 than μ_s = 0.04 [p = 0.009]. Overall, the addition of torsional cutaneous feedback led the participants to feel a virtual object heavier than the one rendered with only kinesthetic feedback. The modulation of μ_s affected the perception of virtual object weight when a virtual object light (m_ref = 100 g).

Figure 5

The results of Experiment 2. (a) Mean estimated PSE of perceived virtual object weight by μ_s and reference weight. (b) Mean of the maximum normal-directional contact force by μ_s, and reference weight.

Figure 5(b) shows the mean of the maximum grip force by the type of haptic feedback as a function of reference object weight. When we conducted a three-way repeated measure ANOVA with the factors μ_s and, m_ref, on the mean normal-direction contact force, a significant two-way interaction was found [$F(2,22)=4.69,$$p=0.02,{\eta }^{2}=0.3$]. The result of a simple main effects analysis indicates that there was a significant difference in the maximum grip force between the two m_s values when m_ref = 100 g [p = 0.02].

The results of Experiment 2 show a similar trend to those of Experiment 1. Additional torsion feedback led the participants to feel the weight of a virtual object heavier than when there was only kinesthetic feedback, during the pull-out and lift-up task. It is notable that varying μ_s resulted in the change of both perceived weight, and the normal contact force at m_ref = 100 g. We discuss the implications of the findings in the next section.

Discussion

The results of the experiments show that the addition of cutaneous feedback to kinesthetic feedback during manipulation led participants to feel virtual object weight as being significantly heavier than that rendered with kinesthetic feedback only. We varied the rate of providing cutaneous feedback for the two experiments. We modulated the rate of the contact plate’s lateral motion to the calculated force for Experiment 1. For Experiment 2, we varied the friction coefficient of rotating the contact plate during manipulation. The experimental results indicate that the effect of varying the rate of cutaneous feedback was insignificant except for the reference weight of 100 g in Experiment 2. We also observed that the grip force was affected by the rate of cutaneous feedback for the reference weight of 100 g.

One common feature of the two experiments’s results is that the addition of cutaneous feedback to a participant’s fingertip led the participants to feel a virtual object heavier during manipulation, than with kinesthetic feedback only. This result can be partly explained with the optimal integration model for multi-modal sensory input. Then, the weight perception can be estimated with the following model:

$${\hat{S}}_{weight}={w}_{K}{\hat{S}}_{K}+{w}_{C}\hat{C},$$

(10)

where ${\hat{S}}_{K}$ and ${\hat{S}}_{C}$ are the estimates of weight from kinesthetic and cutaneous information, respectively, and w_K and w_C are their respective weights. If we assume the distribution of ${\hat{S}}_{C}$ is larger than that of ${\hat{S}}_{K}$, ${\hat{S}}_{weight}$ with the addition of cutaneous feedback will be larger than ${\hat{S}}_{K}$, as is the case with the present study. To substantiate the model, however, additional experimental grounds are needed to show that ${\hat{S}}_{C}$ > ${\hat{S}}_{K}$.

The effect of the friction coefficient on the perceived object weight in Experiment 2 can be explained with the results of previous studies that investigated the perception of fingertip slip perception during tangential loading. Barrea et al. demonstrated that the participants of their experiment could perceive the slippage at the fingertip with the partial skin deformation due to passive tangential loading⁵⁰. Similarly, Srinivasan et al. showed that the tactile afferents could encode surface slippage at the fingertip⁵¹. In other words, $\hat{C}$ in 10 is a function of the cutaneous feedback. Then, their results suggest that the cutaneous feedback in the present study generated the tactile information indicating the existence of tangential loading. For Experiment 2, the slippage at the fingertip due to cutaneous feedback could have been more significant for the lower friction coefficient of μ_s = 0.35 because the virtual object would have rotated more easily. In the meanwhile, the difference of the perceived object between the two friction coefficient was insignificant for larger reference weight, m_ref = 200 and 300 g. This result can ascribe to the insensitivity to smaller cutaneous stimulus change as the overall perceived stimulus, ${\hat{S}}_{weight}$ increases with the increase of the reference weight by Weber’s law.

The experimental results suggest that the force applied to virtual object during manipulation can be a function of perceived object weight. The maximum contact force in Experiment 1 shows an increasing trend along with the increase of the perceived weight of the virtual object. Also, when we compare the maximum contact force of Experiment 2 (Fig. 3(b)) to the PSE estimates (Fig. 3(a)), the effect of varying μ_s shows the same trend that the difference is significant for low m_ref value of 100 g. Previous studies^36,38 indicate that when one is manipulating an object with a precision grip, s/he applies normal force being proportional to tangential torque at the fingertip. Their findings coincide with the results of Experiment 2 where participants felt an object heavier with larger torsional cutaneous feedback (lower μ_s) and thus exerted larger grip force. Thus, the contact force during object manipulation can be modeled as a function of the perceived object weight, and thus depend on kinesthetic and cutaneous information as follows:

$${F}_{contact}=f({\hat{S}}_{weight})=f({I}_{K},{I}_{C}),$$

(11)

where ${\hat{S}}_{weight}$, I_K, and I_C are the perceived weight, kinesthetic, and cutaneous information, respectively.

To summarize, the present study provides grounds that the addition of lateral cutaneous feedback to kinesthetic feedback during object manipulation can increase the perceived weight of the object. Also, the experimental results suggest that the contact force during object manipulation can be a function of the perceived object weight.