J Neurophysiol 107: 433– 441, 2012.

First published October 19, 2011; doi:10.1152/jn.00738.2011.

The role of haptic feedback when manipulating nonrigid objects

Frédéric Danion,1 Jonathan S. Diamond,2 and J. Randall Flanagan2,3

1
Institute of Movement Sciences, Centre National de la Recherche Scientifique, and Université de la Méditerranée, Marseille,
France; and 2Centre for Neuroscience Studies and 3Department of Psychology, Queen’s University, Kingston,
Ontario, Canada
Submitted 8 August 2011; accepted in final form 17 October 2011

Danion F, Diamond JS, Flanagan JR. The role of haptic feedback al. 2004). Such learning can be challenging, because the
when manipulating nonrigid objects. J Neurophysiol 107: 433– 441, motion of the controlled object (e.g., the ball of a paddleball)
2012. First published October 19, 2011; doi:10.1152/jn.00738.2011.— is governed indirectly through the interaction of the motion of
Humans can learn to manipulate objects with complex dynamics,
including nonrigid objects with internal degrees of freedom. The first the hand with the internal dynamics of the object and is not

Downloaded from http://jn.physiology.org/ by 10.220.33.1 on August 10, 2017

aim of this study was to assess the contribution of haptic feedback yoked to hand motion.
when learning to manipulate a nonrigid object. The second aim was to Moving nonrigid objects normally involves interaction
evaluate how learning without haptic feedback influences subsequent forces at the hand. However, in virtual environments, ranging
learning with haptic feedback and vice versa. The task involved from video games to surgical simulators, such forces are often
moving a simulated mass—attached to a grasped handle via a simu- absent, and therefore, appropriate haptic feedback is unavail-
lated, damped spring—to a target as quickly as possible. In the haptic
able. Previous studies have shown that people can learn to
plus vision (HV) condition, appropriate forces were applied to the
handle, which was attached to a robot. In the vision only (V) condi- control nonrigid objects both when appropriate haptic feedback
tion, these forces were turned off. Participants completed 80 trials in is provided (Dingwell et al. 2002, 2004; Nagengast et al. 2009)
each condition, with one-half starting with the HV condition. Both and when it is not (Mah and Mussa-Ivaldi 2003; Mehta and
groups exhibited significant learning, as measured by movement time, Schaal 2002). However, it remains unclear whether appropriate
in both conditions. For the condition performed first, initial perfor- haptic feedback improves learning and control. The primary
mance, learning rate, and final performance were better with haptic objective of the current study was to assess the influence of
feedback. Prior experience in the HV condition led to faster learning haptic feedback in learning to control a nonrigid object. On the
and better final performance in the V condition. However, prior
one hand, we might expect the provision of appropriate haptic
experience in the V condition led to slower learning and worse final
performance in the HV condition. In the V condition, all participants feedback to improve learning and control, because such feed-
tended to keep the mass close to the hand. In the HV condition, back should improve sensory estimates of the state of the
participants who started with the HV condition allowed the mass to object through multisensory integration (Ernst and Banks
move away from the hand, whereas participants who started with the 2002; Ernst and Bülthoff 2004; van Beers et al. 2002). More-
V condition continued to keep the mass close to the hand. We over, it has been argued that the ability to predict the conse-
conclude that haptic feedback as well as prior experience with haptic quences of one’s actions—an important component of senso-
feedback enhance the ability to control nonrigid objects and that rimotor control (Shadmehr et al. 2010; Wolpert and Flanagan
training without haptic feedback can lead to persisting detrimental 2001)— can be impaired when efference copy and sensory
effects when subsequently dealing with haptic feedback. feedback are inconsistent with a particular movement context
object manipulation; motor learning; vision; human (Blakemore et al. 1998). On the other hand, the benefit of
haptic feedback is not obligatory. With the use of a task in
which participants had to control the position of a ball on a
SKILLFUL OBJECT MANIPULATION, including tool use, requires rotating beam, Huang and colleagues (2006) found virtually no
knowledge of the dynamics of the object relating applied force benefit of haptic information. Specifically, participants with
to motion (Flanagan et al. 2006; Johansson and Flanagan vision only (V) began and finished training with comparable
2009). People are highly skilled at predicting and controlling performance with those who also had haptic feedback. In
the motion of grasped objects with familiar dynamics, includ- addition, because interaction forces must be taken into account
ing rigid objects with inertial and elastic loads, where the force to achieve the desired motion of the hand, it is also possible
applied to the hand is proportional to the acceleration and that the provision of appropriate haptic feedback degrades
position of the object, respectively (Flanagan and Wing 1993, hand-motion control (and thus object control). Altogether,
1995, 1997; Westling and Johansson 1984). People are also whether haptic feedback helps, hinders, or simply does not
capable of effectively manipulating many nonrigid objects with affect the control of nonrigid objects remains an open question.
internal degrees of freedom, such as a yoyo or a paddleball We were also interested in how learning to control a non-
consisting of a ball attached to a paddle by a spring. It has been rigid object without haptic feedback influences motor learning
postulated that to formulate an appropriate strategy for con- and control when haptic feedback is subsequently provided and
trolling such an object, the operator must learn an internal vice versa. From an applied perspective, this question was
model of the object’s dynamics, specifying the mapping be- motivated by the fact that provision of haptic feedback in
tween forces applied to the object and its motion (Dingwell et virtual reality (VR) simulators is often difficult and costly. As
Address for reprint requests and other correspondence: F. Danion, Institute
a consequence, determining how training without haptic feed-
of Movement Sciences, 163 Ave. de Luminy, 13009 Marseille, France (e-mail: back (i.e., with vision only) influences performance when
[email protected]). subsequently dealing with haptic feedback is important. From
www.jn.org 0022-3077/12 Copyright © 2012 the American Physiological Society 433
434 HAPTIC FEEDBACK AND MANIPULATION OF NONRIGID OBJECTS

a theoretical perspective, we reasoned that if participants learn motion of the hand to a horizontal plane (Fig. 1). All signals were
a direct mapping between arm motor commands and object recorded at a 1,000 Hz sampling frequency.
motion, we might expect (bidirectional) negative transfer, be- The positions of the hand (i.e., the grasped object), the virtual
mass-spring object, the start position, and the target were all displayed
cause this mapping will change depending on whether inter- in the horizontal plane of the center of the grasped object using a
action forces are present. However, it is also possible that visual display system (Fig. 1). This system consisted of a 30-inch
participants could learn two distinct mappings when learning to monitor positioned horizontally above a mirror located half-way
control the object: a mapping between hand motion and object between the monitor and plane of hand movement. Participants
motion, which does not depend on the presence or absence of viewed the visual scene displayed on the monitor via the mirror,
interaction forces applied to the hand, and a mapping between which blocked vision of the actual hand and grasped object. Filled
circles were used to represent the positions of the hand (blue, 10 mm
arm motor commands and hand motion, which does depend on
in diameter), object (yellow, 20 mm in diameter), start position (green,
interaction force. In the scenario, it is possible that bidirec- 20 mm in diameter), and target (green, 40 mm in diameter). The
tional, positive transfer would occur, because the two condi- simulated properties of the mass-spring object were the following:
tions share a common mapping. However, bidirectional, neg- mass ⫽ 3 Kg, stiffness ⫽ 120 N/m, damping ⫽ 1 N/m/s, resting
ative transfer could also occur, because the mapping between length ⫽ 0 m. Note that these parameter settings are similar to those
arm motor commands and hand motion differs between tasks. used in previous studies using this task (Dingwell et al. 2002, 2004;

Downloaded from http://jn.physiology.org/ by 10.220.33.1 on August 10, 2017

Finally, independently of learning dynamics, it is possible that Nagengast et al. 2009). The resonant frequency of this mass-spring
participants adopt different control strategies depending on the system was close to 1 Hz. The dynamics of the mass-spring object
initial feedback condition they experience. If they persist with were specified as two-dimensional, meaning that lateral displacements
a given strategy when the feedback condition changes, then of the hand and object were also taken into account for the simulation.
Procedure. At the beginning of each trial, the participant had to
transfer between conditions may either be positive or negative
first position the hand circle, without the mass-spring attached, over
depending on the efficacy of the strategies selected. the start position for 300 ms. At this point in time, the target was
We examined the role of haptic feedback using a task in displayed 15 cm away from the start position, the mass-spring object
which participants were asked to move a mass, attached to a was displayed, and the simulation of the mass-spring was initiated.
hand-held handle via a slightly damped spring, to a target as Initially, the mass-spring object was aligned with the hand position
quickly as possible (Dingwell et al. 2002, 2004; Nagengast et and the latter displayed on top of the object. Depending on the
al. 2009). The object was simulated using a VR setup, allowing experimental condition, simulated interaction forces were either ap-
us to remove appropriate haptic feedback by turning off the plied to the hand or not (see below). The participants were instructed
forces applied to the handle (but not the simulated mass). Two to bring the circles representing the hand and object over the target
groups of participants performed the task with visual and circle as quickly as possible. However, they were free to initiate their
movement when ready, once the target was presented. To complete a
haptic feedback (HV task) and with visual feedback only (V
trial successfully, both the hand and object had to be within the target
task) but in different orders. We assessed initial performance, with their speed below 2 cm/s for at least 150 ms (Dingwell et al.
learning rate, and final performance in both naïve participants, 2002). At the end of each trial, the resulting movement time (MT) was
experiencing one of the two tasks for the first time, and displayed on the right side of the target. MT was defined as the time
participants who previously experienced the other task. interval between the instant that hand speed first exceeded 2 cm/s and
the instant at which the trial was completed. As soon as the trial was
completed, the simulated mass-spring object was removed from view
MATERIALS AND METHODS and turned off. Participants were given a maximum time of 10 s to
Participants. Twenty self-proclaimed, right-handed participants (six complete each trial, after which, the trial was aborted. They were
males and 14 females) took part in this study. Participants were split in encouraged to explore various movements in an effort to minimize
two groups (group HV-V: age ⫽ 20.7 ⫾ 2.1 yrs, height ⫽ 1.70 ⫾ MT. However, no specific suggestions were provided.
0.11 m, mass ⫽ 58.6 ⫾ 10.5 kg; group V-HV: age ⫽ 21.5 ⫾ 2.5 yrs, All participants started the experiment with 10 trials, without the
height ⫽ 1.69 ⫾ 0.09 m, mass ⫽ 65.6 ⫾ 14.8 kg) who performed the mass-spring object attached to the hand so as to become familiar with
same experimental conditions but in a different order. All participants
were healthy and gave informed consent prior to the study. A local
university ethics board approved the experiments, which complied with
the Declaration of Helsinki.
Apparatus. In a fully illuminated room, participants were required
to move a cylindrical object held between the tips of the thumb and
the index finger of their right hand. The object had two parallel,
horizontal grip surfaces (2.5 cm in diameter), located 6.4 cm apart,
and was instrumented with two, six-axis force transducers (Nano
force/torque, ATI Industrial Automation, Apex, NC), which measured
forces (0.05 N resolution) and torques applied by the thumb and index
finger in three dimensions. The two grip surfaces were covered with
sandpaper and were free to spin about the long axis of the object,
which was attached to a lightweight robotic manipulator (Phantom 3.0
haptic interface, Sensable, Wilmington, MA) via a joint that allowed
rotation about all axes except the long axis of the object. Thus the
combination of this joint and the spinning grip surfaces allowed free
rotation of the object in three dimensions. Three optical encoders,
placed on the three motors of the manipulandum, measured the
object’s position in three dimensions (0.1 mm resolution). Cuffs Fig. 1. Schematic drawing of the experimental setup (see MATERIALS AND
mounted on air sleds supported the wrist and forearm and restricted METHODS for further information).

J Neurophysiol • doi:10.1152/jn.00738.2011 • www.jn.org

 HAPTIC FEEDBACK AND MANIPULATION OF NONRIGID OBJECTS 435

the goal of the task, the location, and timing of the targets and To assess learning more specifically, we fit an exponential of the
receiving feedback about MT. Afterwards, participants in the HV-V form y ⫽ aebx ⫹ c to the MT data. To test for the effects of group,
group completed a block of 80 trials, in which both haptic and visual task, and interaction between group and task on the parameters of the
feedback was provided (HV task). Following a short break of 2–3 exponential fits, we used nonlinear regression with dummy variables
min, these participants performed another block of 80 trials, in which to code for group and task. The full model (including main effects and
haptic feedback was removed, but visual feedback was preserved (V interactions) is given by
task). Participants of the V-HV group performed the same two blocks
of trials but in the opposite order with a similar break in between. MT ⫽ (a0 ⫹ a1T ⫹ a2G ⫹ a3TG)e(b0⫹b1T⫹b2G⫹b3TG)block
Before each block, the participant was informed about the nature of ⫹ (c0 ⫹ c1T ⫹ c2G ⫹ c3TG)
the task and the sensory context (haptic/no haptic). Overall, each
participant performed a total of 170 trials (10 ⫹ 80 ⫹ 80), which on where T and G are dummy variables coding for task (T ⫽ 0 for the HV
average, took about 45 min. Participants could request additional task; T ⫽ 1 for the V task) and group (G ⫽ 0 for the HV-V group;
breaks at any time, but most of them only took the break offered G ⫽ 1 for the V-HV group), respectively. Note that this 12-parameter
between the two versions of the task. model corresponds to fitting separate, three-parameter exponentials to
Data analysis. All kinematic and kinetic signals were low-pass each combination of group and condition. Benchmarking was used to
filtered at 20 Hz (fourth-order, no-lag, dual-pass Butterworth filter). determine the best-fit model (see RESULTS).
For each trial or movement, we computed the MT, the average

Downloaded from http://jn.physiology.org/ by 10.220.33.1 on August 10, 2017

distance between the hand and object in the Y direction (aligned with
the vector from the start position to the target), and the number of
RESULTS
crossings between the Y positions of the hand and object. To assess
the participants’ ability to predict the dynamics of the mass-spring
MT analysis. There was no significant difference between
system in the HV task, we measured the coefficient of correlation
between grip force and load force (Danion and Sarlegna 2007; the two groups of participants in terms of the mean MT
Flanagan and Wing 1997). Grip force was computed as the average of computed over the 10 practice trials (F1, 18 ⫽ 0.09; P ⫽
the normal forces at the two grip surfaces. To compute load force, we 0.75). This result suggests that the two groups were similar
first determined, for each grip surface, the resultant of the two in overall motor skill.
tangential forces, and we then summed these resultant forces. Figure 2 shows MT as a function of trial block for each group
ANOVA was used to assess the effects of task, group, and trial block and condition. Each point represents the mean MT, averaged
(using eight blocks of 10 trials for each task). The Newman-Keuls
technique was used for post hoc t-tests to correct for multiple com-
across trials within a block, for a single participant. Benchmarking
parisons. Since correlation coefficients do not follow a normal distri- revealed that the best-fit exponential model (fit to all of the data
bution, z scores (Fisher transformation) were used for statistical from both groups and both tasks), in which all parameters were
analysis. A 0.05 significance threshold was used for all analyses. significantly different (P ⬍ 0.05) than zero, was

Fig. 2. Movement time (MT) as a function of trial

block. Data are presented on separate panels for each
group and experimental condition. Open circles repre-
sent individual participant means for each block of 10
trials. The dashed and solid lines correspond to expo-
nential curve fits of the data using the full and best-fit
models, respectively. HV, haptic plus vision; V, vision
only.

J Neurophysiol • doi:10.1152/jn.00738.2011 • www.jn.org

436 HAPTIC FEEDBACK AND MANIPULATION OF NONRIGID OBJECTS

MT ⫽ (a0 ⫹ a1T ⫹ a2G)e(b0⫹b2G)block ⫹ (c0 ⫹ c1T ⫹ c2G) Consistent with the best-fit exponential model described
above, repeated measures ANOVA revealed that for each
(see MATERIALS AND METHODS). This exponential model revealed combination of group and condition, MT decreased signifi-
main effects of task (P ⬍ 0.001) and group (P ⫽ 0.029) on the cantly from the first to the last block of 10 trials (F1, 9 ⬎ 15.9;
leading value (a), main effects of task (P ⬍ 0.001) and group P ⬍ 0.003 in all four cases). Across conditions and groups,
(P ⫽ 0.035) on the asymptote (c) of the exponential, and a main mean MT dropped by 37% (see Figs. 3, A and B).
effect of group (P ⫽ 0.010) on the learning rate (b). There was no Two-way (task-by-group) ANOVA on MT during the
main effect of task on the learning rate and no interaction between first trial block (Fig. 3C) revealed a significant effect of task
task and group for any of the three parameters. (F1, 18 ⫽ 60.17; P ⬍ 0.001) but no effect of group (F1, 18 ⫽
This best-fit model can be partitioned into the following four 0.10; P ⫽ 0.75) and no interaction (F1, 18 ⫽ 0.98; P ⫽ 0.34).
models for each combination of group and condition Thus initial performance on either the HV or V task was not
group HV ⫺ V, task HV: MT ⫽ a0eb0x affected by previous experience with the other task. Concern-
⫹ c0 ⫽ 3.222e⫺0.963x ⫹ 1.822 ing final performance, two-way ANOVA on the last trial block
(Fig. 3D) showed a significant effect of task (F1, 18 ⫽ 79.5;
group HV ⫺ V, task V: MT ⫽ (a0 ⫹ a1)eb0x P ⬍ 0.001). On average, MT was 63% longer when haptic

Downloaded from http://jn.physiology.org/ by 10.220.33.1 on August 10, 2017

⫹ (c0 ⫹ c1) ⫽ 5.321e⫺0.963x ⫹ 2.947 feedback was absent (V task: M ⫽ 3.17 s) than when haptic
feedback was present (HV task: M ⫽ 1.95 s). In addition, there
group V ⫺ HV, task HV: MT ⫽ (a0 ⫹ a2)e(b0⫹b2)x was a significant effect of group (F1, 18 ⫽ 6.71; P ⫽ 0.018),
⫹ (c0 ⫹ c2) ⫽ 1.104e⫺0.455x ⫹ 2.104 demonstrating that prior experience influenced the ability of
participants to manipulate the object. Specifically, the HV-V
group V ⫺ HV, task V: MT ⫽ (a0 ⫹ a1 ⫹ a2)e(b0⫹b2)block group, which started with full feedback, performed better
⫹ (c0 ⫹ c1 ⫹ c2) ⫽ 3.203e⫺0.455 ⫹ 3.229 overall (M ⫽ 2.33 s) than the V-HV group, which started with
visual feedback only (M ⫽ 2.79 s). Finally, there was no
These exponential fits are shown in Fig. 2, and for compar- task-by-group interaction, suggesting that starting with haptic
ison, the figure shows the exponential fits for the full model feedback provided a final performance advantage in both tasks.
(see MATERIALS AND METHODS). Consistent with the regression Trajectory analysis. Figure 4 shows two representative hand
analysis, the additional parameters of the full model make little and object trajectories taken from the last 10 trials of the first
difference to the overall fit. In both cases, we found that final session. That is, a participant in the HV-V group performed the
or asymptotic performance (i.e., MT at the end of learning) was trial from the HV task and a different participant in the V-HV
significantly better in the HV task than in the V task and also group performed the trial from the V task. The figure shows the
that performance was significantly better for the HV-V group Y positions of the hand and mass (i.e., along the axis aligned
than for the V-HV group. A similar order effect was observed with the start position and target), the Y distance between mass
for the learning rate, which was approximately two times faster and object, and the Y velocities of the hand and mass. When
for the HV-V group than the V-HV group. haptic feedback was not available, the participant moved the

Fig. 3. Mean MT results. A: mean MT as a function of

trial block and group in the HV task. B: same as A but
for the V task. A and B: the white and black circles
represent means averaged across participants in the
V-HV and HV-V groups, respectively. C: mean MT in
the first block of trials as a function of group and task.
D: same as C but for the last block of trials. C and
D: the white and black bars correspond to the V-HV
and HV-V groups, respectively. For all panels, error
bars correspond to the SE.

J Neurophysiol • doi:10.1152/jn.00738.2011 • www.jn.org

 HAPTIC FEEDBACK AND MANIPULATION OF NONRIGID OBJECTS 437

Fig. 4. Examples of representative trials in each task. A and

B: hand and mass positions (Pos) along the y axis (main axis of
movement) as a function of time. C and D: Y distance (Dist)
between hand and mass as a function of time. E and F: hand and
mass velocities (Vel) along the y axis as a function of time. A,

Downloaded from http://jn.physiology.org/ by 10.220.33.1 on August 10, 2017

B, E, F: thick and thin lines refer to hand and mass kinematics,
respectively. Dashed lines represent criteria that were used to
assess when a trial was completed successfully (see MATERIALS
AND METHODS for further information). Each trial was per-
formed by a different participant and was extracted from the
last block of 10 trials. Note the different movement strategies
used to displace the mass in the HV and V tasks.

hand slowly—making a series of submovements characterized task, both measures changed significantly across blocks of
by distinct hand velocity maxima (Fig. 4F)—and kept the mass trials (F7, 126 ⬎ 5.19; P ⬍ 0.001). The average absolute
close to the hand (Fig. 4D). In contrast, when haptic feedback distance between the hand and mass tended to increase across
was available, the participant moved the hand more quickly blocks in the HV task, whereas it tended to decrease across
(Fig. 4E), and the trajectory of the object was quite different blocks in the V task. The number of crossings tended to
than that of the hand (Fig. 4C). decrease across blocks in both tasks.
Figure 5 shows mean hand and object Y positions (Fig. 5, A Concerning final performance, two-way (task-by-group)
and B) and the mean hand-to-mass Y distance (Fig. 5C) as a ANOVA of the last trial block showed main effects of task
function of normalized time for each combination of task and (F1, 18 ⫽ 11.2; P ⫽ 0.004) and group (F1, 18 ⫽ 11.8; P ⫽ 0.003) on
group. These mean functions are based on participant means the average absolute distance between hand and mass (Fig. 7A).
averaged over the last 10 trials. In both the HV and V tasks, the Although the effect of group was larger for the HV task, the
deviation between the hand and mass was greater for partici- interaction between task and group did not reach significance
pants in the HV-V group compared with participants in the (F1, 18 ⫽ 4.15; P ⫽ 0.056). Thus in the first session, participants
V-HV group. However, the difference between groups was far who received haptic feedback tended to allow the mass to
greater in the HV task than in the V task, revealing a strong move away from the hand, whereas participants who only
effect of prior experience. Overall, the deviation between the received visual feedback tended to keep the mass closer to the
hand and the mass was greater in the HV task, in which haptic hand. However, the effect of haptic feedback on the average
feedback was provided, than in the V task. However, quite absolute hand-to-mass distance was much smaller when con-
similar deviations were seen in the HV task performed by the sidering the second session. The number of crossing was about
V-HV group and the V task performed by the HV-V group two times greater in the V task compared with the HV task
(Fig. 5C), again revealing the strong effect of prior experience. (F1, 18 ⫽ 75.7; P ⬍ 0.001; 5.6 vs. 2.5). There was also an effect
Figure 6 shows mean values of the average absolute Y of group (F1, 18 ⫽ 7.48; P ⫽ 0.014) and a significant task-by-
distance between the hand and mass over the movement, as group interaction (F1, 18 ⫽ 7.09; P ⫽ 0.015), due to the fact that
well as the number of crossings between the hand and mass Y the number of crossings was smaller for the HV-V group but
positions. (As a reference, in Fig. 4, there are two and seven only for the V task (corrected t-test; P ⫽ 0.001).
crossings for the trials shown for the HV and V tasks, respec- Grip–load force coupling. To examine the coupling of grip
tively.) Repeated measures ANOVA revealed that for each force and load force in the HV task, in which substantial load
J Neurophysiol • doi:10.1152/jn.00738.2011 • www.jn.org
438 HAPTIC FEEDBACK AND MANIPULATION OF NONRIGID OBJECTS

are consistent with earlier observations showing that MT was

lower in the HV-V group than the V-HV group and that MT
decreased across trials blocks.

DISCUSSION
The current study yielded several key findings. First, in
agreement with earlier studies, in which haptic feedback was
provided (Dingwell et al. 2002, 2004; Nagengast et al. 2009),
we found that people could improve their ability to control the
movement of a nonrigid object. Second, improvements in
control also occurred in the absence of appropriate haptic
feedback, arising from interaction forces between the hand and
object. Third, haptic feedback allowed greater dexterity in
manipulating nonrigid objects as measured by MTs. Fourth,
previous experience influenced the way participants manipu-

Downloaded from http://jn.physiology.org/ by 10.220.33.1 on August 10, 2017

lated the nonrigid object. Although positive transfer was found
when appropriate haptic feedback was removed, quite unex-
pectedly, negative transfer was found when appropriate haptic
feedback was added. Fifth, we found that participants appeared
to use distinct control strategies when moving the object with
and without haptic feedback. With haptic feedback, consider-
able stretching of the spring, linking the hand and mass, was
found in naïve participants. In contrast, without haptic feed-
back, the mass was kept close to the hand in naïve participants.
Finally, we found that in the haptic version of the task, the
coupling between grip force and load force improved across
trials, suggesting that participants acquired an increasingly
accurate representation of the interaction forces between the
hand and mass.
Contributions of haptic and visual feedback. On the one
hand, one might expect haptic feedback related to interaction
forces to improve control, because such feedback should allow
more accurate sensory estimates of the state of the object
through multisensory integration (Ernst and Banks 2002; Ernst
and Bülthoff 2004; van Beers et al. 2002). On the other hand,
because interaction forces perturb the hand, one might expect
haptic feedback to degrade control (Dingwell et al. 2002;
Fig. 5. Mean trajectories in the last block of trials. A: mean hand (solid lines) Huang et al. 2006). Although previous experiments have ex-
and mass (dashed lines) positions as a function of normalized time for each
group in the HV task. B: same as A but for the V task. C: mean hand-to-mass
amined the contributions of haptic and visual feedback in
distance as a function of normalized time for each group and task. For all performing skilled object manipulation tasks (e.g., Sternad et
panels, the mean trajectories are based on participant means averaged over the al. 2001), to our knowledge, only one study has addressed this
last 10 trials. Prior to averaging, trials were normalized with respect to MT. issue explicitly in the context of learning to manipulate a
Shaded areas correspond to 1 SE. nonrigid object. As mentioned in the introduction, Huang and
colleagues (2006), who examined a task in which participants
forces acted on the object, we computed the coefficient of had to control the position of a ball on a beam, found virtually
correlation between grip force and load force for each trial. no benefit of haptic information; participants who received
Figure 8 shows mean correlation coefficients, averaged across appropriate haptic feedback began and finished their training
participant means, as a function of trial block for each group. session with comparable performance with participants who
A two-way (trial block-by-group) ANOVA revealed significant received visual feedback only (see Table 3 in Huang et al.
effects of trial block (F7, 126 ⫽ 4.14; P ⬍ 0.001) and group 2006). In contrast, we found that haptic feedback facilitates
(F1, 18 ⫽ 5.01; P ⫽ 0.04). The coupling between grip force and both initial and final performance in naïve participants. In the
load force tended to increase with trial block and was consis- absence of haptic feedback, initial and final MTs increased by
tently greater when haptic feedback was provided in the first 75% and 92%, respectively, and the number of crossings
session (HV-V group; R ⫽ 0.66) than when it was provided in between the hand and mass increased by 260%. This compar-
the second session (V-HV group; R ⫽ 0.46). We also deter- ison between these two studies suggests that the contribution of
mined the maximum load force in each trial of the HV task. A haptic information to the manipulation of objects with complex
two-way (trial block-by-group) ANOVA revealed that the dynamics can vary substantially depending on the properties of
maximum load force increased across blocks (F7, 126 ⫽ 2.06; the object and task. However, our observation that haptic
P ⫽ 0.05) and was greater for the HV-V group than the V-HV feedback is helpful when manipulating a mass-spring system is
group (2.9 vs. 1.1 N; F1, 18 ⬎ 10.48; P ⫽ 0.005). These effects consistent with another study by Huang and colleagues (2007),
J Neurophysiol • doi:10.1152/jn.00738.2011 • www.jn.org
 HAPTIC FEEDBACK AND MANIPULATION OF NONRIGID OBJECTS 439

Fig. 6. Means of kinematic variables. A: average

absolute (Avg Abs) hand-to-mass distance in the
HV task as a function of trial block and group.
B: same as A but for the V task. C and D: same as
A and B but for the number of crossings between
hand and mass along the y axis. In all 4 panels,
black and white circles correspond to the HV-V
and V-HV groups, respectively. Error bars corre-

Downloaded from http://jn.physiology.org/ by 10.220.33.1 on August 10, 2017

spond to 1 SE.

in which they examined the ability to oscillate and tune a Although we found that haptic feedback improved perfor-
mass-spring system at its resonance frequency. Future work mance, our participants were still able to complete the task and
will be required to determine which aspects and/or features of improve performance across trials when only visual feedback
a mass-spring system make haptic feedback beneficial. was provided. Overall, participants in the V task reduced their
MT by ⬃40% between the first and the last block of trials—a
reduction that was similar to that observed in the HV task. The
fact that participants could control the mass-spring system
based solely on visual information extends earlier observations
made in the context of bouncing a ball on a racket (Sternad et
al. 2001) or balancing an inverted pole (Mehta and Schaal
2002).
Learning internal models of nonrigid objects. It has been
argued that learning to control a nonrigid mass-spring object
involves the acquisition of an internal model of the object’s
dynamics (Dingwell et al. 2004). Internal models of object
dynamics capture the mapping between the force applied to the

Fig. 7. Means of kinematic variables in the last trial block. A: average Fig. 8. Grip–load force coupling in the HV task. Average correlation coeffi-
hand-to-mass distance in the HV task as a function of task and group. B: same cients of the correlation between grip force and load force as a function of trial
as A but for the number of crossings between and mass along the y axis. For block for both the HV-V (black circles) and V-HV (white circles) groups. Each
all panels, black bars correspond to the HV-V group, and white bars corre- point represents the mean of participants averaged across trials within the
spond to the V-HV group. Error bars correspond to 1 SE. block. Error bars correspond to 1 SE.

J Neurophysiol • doi:10.1152/jn.00738.2011 • www.jn.org

440 HAPTIC FEEDBACK AND MANIPULATION OF NONRIGID OBJECTS

object and the motion of the object (or the mapping between the VH task will be poorer than the transfer in the opposite
arm motor commands and object motion). Such models enable direction, it does not predict whether the transfer will be
the sensorimotor system to predict the consequences of motor positive or negative. Based on the above considerations, it
commands, in which case, they are referred to as forward seems difficult to explain the asymmetric transfer effects we
models (Wolpert and Flanagan 2001; Wolpert and Ghahramani obtained in the context of learning mappings or internal
2000). Our results pertaining to the coupling between grip and models.
load force support the idea that participants learned a forward As an alternative, we suggest that the asymmetric effects of
model of the mass-spring system. To assess participants’ abil- previous experience can be understood in the context of control
ity to predict dynamics of the mass-spring system in the HV strategies, namely the process of optimizing/selecting move-
task, we measured the correlation between grip force and load ment trajectories, which is distinct from learning dynamics. We
force (Danion and Sarlegna 2007; Flanagan and Wing 1997). observed that the hand-to-mass distance varied substantially
We found that correlation coefficients (R values) were initially depending on the experimental conditions. In the HV task,
low but increased substantially over the first three blocks of naïve participants allowed the mass to move quite far away
trials. Interestingly, roughly parallel changes were observed in from the hand. This resulted in movements characterized by a
MT, which decreased substantially over the first three blocks relatively low number of hand velocity peaks and crossings
(Fig. 3A). Indeed, across trial blocks in the HV task, the between the hand and mass. In contrast, in the V task, naive

Downloaded from http://jn.physiology.org/ by 10.220.33.1 on August 10, 2017

correlation between mean R values and mean MT was ⫺0.96 participants kept the mass close to the hand, which resulted in
(P ⬍ 0.001). The rather gradual improvement in grip–load movements featuring a large number of hand velocity peaks
coupling observed in the current study may be contrasted with and crossings between the hand and mass. Altogether, it seems
the rapid adaptation of grip forces reported for a task in which that in the HV task, naïve participants learned to control the
participants grasped and moved a rigid object with an unfa- degrees of freedom associated with the mass, whereas in the V
miliar load (Flanagan et al. 2003). task, naïve participants tended to do the opposite and froze the
Transfer between tasks: internal models and movement mass. Thus naïve participants used what might be referred to as
strategies. One aim of our study was to determine how learning a lead-lag strategy when haptic feedback was provided and a
to control a nonrigid object without haptic feedback influences dragging strategy when haptic feedback was not available. The
control when haptic feedback is provided and vice versa. Our main advantage of the lead-lag strategy is that it allows faster
results provide clear evidence that previous experience in one hand movements. However, this strategy requires the ability to
version of the task influenced the ability to perform the other quickly damp the resulting oscillations of the mass. In contrast,
version of the task. However, those transfer effects were not the dragging strategy does not allow fast hand movements, but
symmetrical. Although earlier practice with haptic feedback the ability to damp terminal oscillations is less crucial, since
improved the ability to perform the task without haptic feed- these oscillations are likely to be small. Last but not least,
back, earlier practice without haptic feedback was detrimental visual feedback processing is relatively slow compared with
to performing the task with haptic feedback. As noted above, somatosensory feedback control (200 vs. 80 ms), and this may
there is evidence that learning to control a nonrigid object have contributed to slower hand movements in the V task.
involves acquiring an internal model of the object’s dynamics. Our results indicate that when the two groups of participants
We reasoned that if participants learned a direct mapping exchanged tasks, they did not fully exchange strategies. In-
between arm motor commands and object motion, then be- stead, they tended to stick, at least partially, with their initial
cause this mapping depends on whether interaction forces strategy. As a consequence, both groups ended up adopting a
between the hand and object are present, negative transfer rather similar, intermediate strategy. In other words, when
should occur between the HV task and the V task and vice participants had prior experience with haptic feedback, they
versa. Alternatively, we proposed that in principle, participants allowed the mass to move more freely in the V task than naïve
could learn two mappings: the mapping between motion of the participants. Conversely, when participants had prior experi-
hand and motion of the mass (i.e., between the states of the ence in the V task, they were reluctant to free the mass in the
hand and mass) and the mapping between arm motor com- HV task, compared with naïve participants. Because fast com-
mands and hand motion. If so, it is not clear whether transfer pletion times are possible only if the mass is moved away from
would be positive or negative. On the one hand, because the the hand, this carryover effect (i.e., tendency to stick with the
former mapping is independent of interaction forces and is the initial strategy) was beneficial to participants who started with
same in both tasks, we might have expected positive transfer haptic feedback, whereas it was detrimental to those who
between the HV and V tasks in both directions. On the other started without.
hand, because the latter mapping changes between tasks, neg- Implications for training in VR simulators. Our finding that
ative transfer might be expected between the HV and V tasks haptic feedback as well as prior experience with haptic feed-
in both directions. However, none of these scenarios was back enhance the ability to control a nonrigid object has
supported, because negative transfer from the V task to the HV implications for the design of teleoperation devices and VR
task was observed, whereas positive transfer occurred from the simulators, such as those used in surgical training. This is
HV task to the V task. particularly obvious for surgical simulators, because trainees
Asymmetric transfer of learning could arise if participants will use these devices to learn how to handle a wide variety of
adapted their arm motor commands to interaction forces more soft tissues (Basafa and Farahmand 2010; Lim et al. 2009). A
slowly than they de-adapted their arm motor commands when typical issue when designing VR simulators is whether the
interaction forces were removed (Davidson and Wolpert 2004; provision of haptic feedback, which can be both difficult and
Shadmehr et al. 1998; Smith et al. 2006). However, although costly, is necessary or justified. In line with our results, a
this scheme predicts that transfer of learning from the V task to number of studies have found that haptic feedback enhances
J Neurophysiol • doi:10.1152/jn.00738.2011 • www.jn.org
 HAPTIC FEEDBACK AND MANIPULATION OF NONRIGID OBJECTS 441

performance in surgical simulator training (Panait et al. 2009; Flanagan JR, Bowman MC, Johansson RS. Control strategies in object
Ström et al. 2006; van der Meijden and Schijven 2009). manipulation tasks. Curr Opin Neurobiol 16: 650 – 659, 2006.
Flanagan JR, Vetter P, Johansson RS, Wolpert DM. Prediction precedes
However, a novel finding brought by our study is that training control in motor learning. Curr Biol 13: 146 –150, 2003.
without haptic feedback (i.e., with V) can lead to persisting Flanagan JR, Wing AM. Modulation of grip force with load force during
detrimental effects if operators subsequently have to work with point-to-point arm movements. Exp Brain Res 95: 131–143, 1993.
haptic feedback. This would typically be the case, when after Flanagan JR, Wing AM. The role of internal models in motion planning and
control: evidence from grip force adjustments during movements of hand-
simulated training, operators would then have to perform the held loads. J Neurosci 17: 1519, 1997.
real task. Flanagan JR, Wing AM. The stability of precision grip forces during cyclic
arm movements with a hand-held load. Exp Brain Res 105: 455– 464, 1995.
Huang FC, Gillespie RB, Kuo AD. Human adaptation to interaction forces in
ACKNOWLEDGMENTS visuo-motor coordination. IEEE Trans Neural Syst Rehabil Eng 14: 390 –
We thank Martin York, Sean Hickman, and Diane Fleming for technical 397, 2006.
support. We also thank James Ingram and Ian Howard for technical assistance. Huang FC, Gillespie RB, Kuo AD. Visual and haptic feedback contributeto
The comments provided by the expert referees were greatly appreciated. tuning and online control during object manipulation. J Mot Behav 39:
179 –193, 2007.
Johansson RS, Flanagan JR. Coding and use of tactile signals from the
GRANTS fingertips in object manipulation tasks. Nat Rev Neurosci 10: 345–359,

Downloaded from http://jn.physiology.org/ by 10.220.33.1 on August 10, 2017

2009.
Support for this work was provided by grants from the Canadian Institutes Lim YJ, Deo D, Singh TP, Jones DB, De S. In situ measurement and
of Health Research and the Canadian Foundation of Innovation. modeling of biomechanical response of human cadaveric soft tissues for
physics-based surgical simulation. Surg Endosc 23: 1298 –1307, 2009.
Mah CD, Mussa-Ivaldi FA. Evidence for a specific internal representation of
DISCLOSURES
motion-force relationships during object manipulation. Biol Cybern 88:
No conflicts of interest, financial or otherwise, are declared by the author(s). 60 –72, 2003.
Mehta B, Schaal S. Forward models in visuomotor control. J Neurophysiol
88: 942–953, 2002.
AUTHOR CONTRIBUTIONS Nagengast AJ, Braun DA, Wolpert DM. Optimal control predicts human
performance on objects with internal degrees of freedom. PLoS Comput Biol
Author contributions: F.D. and J.R.F. conception and design of research; 5: e1000419, 2009.
J.S.D. performed experiments; F.D. and J.R.F. analyzed data; F.D. and J.R.F. Panait L, Akkary E, Bell RL, Roberts KE, Dudrick SJ, Duffy AJ. The role
interpreted results of experiments; F.D. and J.R.F. prepared figures; F.D. and of haptic feedback in laparoscopic simulation training. J Surg Res 156:
J.R.F. drafted manuscript; F.D. and J.R.F. edited and revised manuscript; F.D. 312–316, 2009.
and J.R.F. approved final version of manuscript. Shadmehr R, Brandt J, Corkin S. Time-dependent motor memory processes
in amnesic subjects. J Neurophysiol 80: 1590 –1597, 1998.
REFERENCES Shadmehr R, Smith MA, Krakauer JW. Error correction, sensory predic-
tion, and adaptation in motor control. Annu Rev Neurosci 33: 89 –108, 2010.
Basafa E, Farahmand F. Real-time simulation of the nonlinear visco-elastic Smith MA, Ghazizadeh A, Shadmehr R. Interacting adaptive processes with
deformations of soft tissues. Int J Comput Assist Radiol Surg 6: 297–307, different timescales underlie short-term motor learning. PLoS Biol 4: e179,
2010. 2006.
Blakemore SJ, Goodbody SJ, Wolpert DM. Predicting the consequences of Sternad D, Duarte M, Katsumata H, Schaal S. Bouncing a ball: tuning into
our own actions: the role of sensorimotor context estimation. J Neurosci 18: dynamic stability. J Exp Psychol Hum Percept Perform 27: 1163–1184,
7511–7518, 1998. 2001.
Danion F, Sarlegna FR. Can the human brain predict the consequences of arm Ström P, Hedman L, Särna˚ L, Kjellin A, Wredmark T, Felländer-Tsai L.
movement corrections when transporting an object? Hints from grip force Early exposure to haptic feedback enhances performance in surgical simu-
adjustments. J Neurosci 27: 12839 –12843, 2007. lator training: a prospective randomized crossover study in surgical resi-
Davidson PR, Wolpert DM. Scaling down motor memories: de-adaptation dents. Surg Endosc 20: 1383–1388, 2006.
after motor learning. Neurosci Lett 370: 102–107, 2004. van Beers RJ, Wolpert DM, Haggard P. When feeling is more important
Dingwell JB, Mah CD, Mussa-Ivaldi FA. Experimentally confirmed math- than seeing in sensorimotor adaptation. Curr Biol 12: 834 – 837, 2002.
ematical model for human control of a non-rigid object. J Neurophysiol 91: van der Meijden OA, Schijven MP. The value of haptic feedback in
1158 –1170, 2004. conventional and robot-assisted minimal invasive surgery and virtual reality
Dingwell JB, Mah CD, Mussa-Ivaldi FA. Manipulating objects with internal training: a current review. Surg Endosc 23: 1180 –1190, 2009.
degrees of freedom: evidence for model-based control. J Neurophysiol 88: Westling G, Johansson RS. Factors influencing the force control during
222–235, 2002. precision grip. Exp Brain Res 53: 277–284, 1984.
Ernst MO, Banks MS. Humans integrate visual and haptic information in a Wolpert DM, Flanagan JR. Motor prediction. Curr Biol 11: R729 –R732,
statistically optimal fashion. Nature 415: 429 – 433, 2002. 2001.
Ernst MO, Bülthoff HH. Merging the senses into a robust percept. Trends Wolpert DM, Ghahramani Z. Computational principles of movement neu-
Cogn Sci 8: 162–169, 2004. roscience.Nat Neurosci 3: 1212–1217, 2000.

J Neurophysiol • doi:10.1152/jn.00738.2011 • www.jn.org