Proceeding of the 2015 IEEE

International Conference on Information and Automation

Lijiang, China, August 2015

Teleoperated Robot Writing using EMG Signals

Chenguang Yang∗1,2 , Sai Chang1 , Peidong Liang1,3 , Zhijun Li2 and Chun-Yi Su2,4

Abstract— In this paper, we have developed a method to in an economical perspective, and therefore may have great
tele-control a robot arm to imitate human writing skills us- potentials in compliant human robot interactions especially
ing electromyography (EMG) signals. The control method is in some physically coupling scenarios, e.g., rehabilitation or
implemented on the Baxter R robot with a brush attached on
the endpoint of its arm to imitate human writing skills, and a daily cooperation tasks.
MYO sensor is used to measure the surface electromyographic Robots are now used for a wide range of different tasks in
(sEMG) signals generated by contractions of human muscles various situations, e.g., industry, medicine and research [5].
involved in writing tasks. A haptic device Sensable R Omni is Nowadays, automatic robot technology is well developed, but
used to control the motion of the Baxter R robot arm, and V-
there are still a lot of challenges for robots to deal with un-
Rep R is employed to simulate the movement of the arm and to
calculate the inverse kinematic of Baxter R robot arm. All the
certain environments. While artificial intelligence technology
communications for Baxter R robot, V-Rep simulator, Omni still could not reach a level that is able to substitute human
device and MYO are integrated in MATLAB /Simulink R .
R decision, human controlled robots are preferred for most
The main test is to make the Baxter R arm following the applications which need real time decisions. The process that
movement of a human subject when writing with Omni stylus, a human user controlling a slave robot remotely is called
and the EMG signals are processed using low pass filter and
moving average technique to extract the smoothed envelope teleoperation, which is important in many applications [6],
which is utilised to control the variation of position instead of e.g., remote robot is used to perform tasks in dangerous
variation of force along the vertical Z axis, so when the operator environment, for tasks that need to be done precisely, so that
writes with force variations, the Baxter R can draw lines with an accurate movement is important. Conventionally, input
variable thickness. This imitation system is successfully tested devices such as keyboard, mouse or joystick are actually
to make the Baxter R arm to follow the writing movement
of the operator’s hand and would have potential applications not following the instinct of human. Human users may need
in more complicated human robot teleoperation tasks such as training and practice to get used to it. The work in this
tele-rehabilitation and tele-surgery. paper attempts to control a robot arm by using both joystick
Keywords. Teleoperation, sEMG, V-Rep, Baxter
R robot. and EMG signals. The proposed method would be useful in
various applications. For example, in medicine surgery[7],
I. INTRODUCTION robot assisted surgery enhances the accuracy and dexterity
Research of human motor behaviors reveal that human of a surgeon by human adaptive muscle variations which can
arm can be stabilized mainly dependent on mechanical make the knots firm enough to keep the sutures compliant.
impedance control during interaction with dynamic envi- To extract the muscle variations needed for the action, there
ronment [1]. This control method minimizes the interaction are mainly two methods: one is to measure the force by a
force and performance errors. Inspired by these research re- force sensor, and the second is to exploit the EMG signals
sults, biomimetic learning controllers are proposed in [2], [3], which are generated when muscle activations occur. In this
[4] which are able to adapt trajectory, force and impedance paper, we extract smoothed EMG envelope to denote the
in the presence of unknown dynamics at same time. Com- variations of human muscle activity and apply it in human
pared to traditional robotic controllers, they are human- robot tele-handwriting.
like enabling robots to have some human motor features II. P RELIMINARY
This work was supported in part by EPSRC grants EP/L026856/1 and A. Baxter
R Robot
EP/J004561/1, National Natural Science Foundation of China (NSFC) under Baxter R is a two-arm robot built by Rethink Robotics[8]
Grant 61473120, Guangdong Provincial Natural Science Foundation of
China 2014A030313266, State Key Laboratory of Robotics and System shown in Fig. 1. It is 5 to 6 feet tall, 306 lbs weight with
(HIT) SKLRS-2014-MS-05, the Program for New Century Excellent Tal- pedestal and 7 DOFs per arm, and it is equipped with 360
ents in University under Grant NCET-12-0195, and National High-Tech degree sonar for human presence detection and changeable
Research and Development Program of China (863 Program) Grant No.
2015AA042303. end-effectors. It is used in simple industrial tasks such as
1 Center for Robotics and Neural Systems, Plymouth University PL4 8AA, loading, unloading, packaging and kitting, machine tending
UK or material handling. Different accessories can be used on
2 Key Laboratory of Autonomous System and Network Control, Col-
lege of Automation Science and Engineering, South China University of
Baxter R platform such as vacuum cup gripper, electric
Technology, Guangzhou, 510640, P. R. China parallel gripper and mobile pedestal.
3 State Key Lab of Robotics Systems, Harbin Institute of Technology,
The feature of the Baxter R lies in its intrinsic safety. Many
Harbin, 150001, P. R. China. existing robots used in industry are built to complete tasks
4 Department of Mechanical and Industrial Engineering, Concordia Uni-
versity, Montreal, H3G 1M8, Canada. rapidly, most of the components are moving in very fast
* Corresponding author. Email: [email protected] speed in order to increase efficiency, and lack of safety

978-1-4673-9104-7/15/$31.00 ©2015 IEEE 2264

 Fig. 2: Phantom Omni [10]: users can use this device to
tele-operate robot and transfer kinematic skills to robot.
Fig. 1: Baxter
R robot profile [9]: Baxter
R robot has 7 joints
with each arm and each joint is constructed as serial elastic
actuator (SEA). mapping process is described as the equation below:
   
Xs cos δ − sin δ 0
sensors to reduce costs, these make them not safe for workers
 Ys  =  sin δ cos δ 0  ×
to interact with. Compliant joints are used in Baxter , R its
Zs 0 0 1
     (1)
arms are driven by springs that driven by motor, this allow Sx Xm Tx
Baxter R reduce the impact when its arm hit something.  Sy   Ym  +  Ty 
Force sensors built in the arms, adjustable torque force and Sz Zm Tz
12 sonar sensors around the head of Baxter R can perform
detection of human movement and proximity also make where [Xs ,Ys , Zs ]T is the Cartesian coordinate of end effector
Baxter R is safer to be work nearby. One another feature
of Phantom Omni,[Xm ,Ym , Zm ]T is the Cartesian coordinate
of the Baxter R is that it is cost effectively. And software
of end effector of Baxter R arm, δ is the angle of the rotation

engineers are not necessary to used Baxter , R because it

angle about Z-axis of Baxter. [Sx , Sy , Sz ]T and [Tx , Ty , Tz ]T are
can be trained to complete its tasks, by moving its arms to scaling factors and translation about XYZ axis, respectively,
required position and let Baxter R to memorise it. Therefore,
the parameters of the equation above are also provided as
Baxter R is able to work without complicated programming,
below:
       
such that the cost of software development can be reduced S 0.0041 Tx 0.701
π  x  
[8]. δ = , Sy = 0.0040  ,  Ty  =  0.210  (2)
4
Sz 0.0041 Tz 0.129
B. PHANTOM Omni It should be noted that the direction of axis of Phantom
Omni is different with axis of Baxter, i.e., the Z-axis of
PHANTOM Omni (Fig. 2) is a haptic device produced
Phantom Omni is parallel to the ground and the Z-axis of
by SensAble . R It has 6 DOFs with the first three DOFs
Baxter is upward vertical, the axis of Phantom Omni has
define the Cartesian position of the end-effector, and the last
to be rearranged to match the axis of Baxter. In this paper,
three DOFs define the orientation of the end effector. Omni
the operator is facing the Baxter, and the X-axis and Y-axis
makes it possible for users to touch and control virtual 3D
should be also aligned so that the movement of the end-
modelled objects. It also helps to improve the experience of
effector match the sight of the operator.
scientific or medical simulation [10]. OpenHaptics toolkit
can be used to connect Omni with various softwares and
D. MYO armband
can be programmed for different purposes[11]. PHANTOM
Omni is a very popular haptic device in the research MYO (Fig. 4) is an armband designed to measure the EMG
communities, and many studies about Phantom Omni have signals on the forearm, in order to recognize the gestures
been done. In this paper, the model of Omni built in our perform by the hand. It consists 8 EMG sensors and 9-axis
previous work[12] is employed. IMU.

E. EMG signal processing

C. Workspace matching In this paper, low pass filter and moving average technique
described in (3) are used to track to muscle variation change
Since the workspaces between Phantom Omni and the with 8 channel raw EMG data (Fig. 5) using algorithm shown
arm of Baxter R are quite different. A workspace mapping in Fig. 6. Fig. 7 is an example of EMG signal smoothing with
method which can be used to match the workspace between 20 samples window, as the EMG signals vibrate in a fast
Phantom Omni and Baxter R was developed in our previous frequency. The 8 EMG signal readings are added together
work [12], and the matching result is shown in Fig. 3. The and put into a simple moving average function block and

2265
 (a) (b)

Fig. 3: Workspace matching. (a) Workspace matching result represented at the XY plane; (b) Workspace matching result
represented at the xz plane.

Fig. 6: Raw EMG data processing using moving average and

low pass filter envelope method.

Fig. 4: Myo armband profile [13]: Wearable MYO armband

with 8 channels and 200Hz EMG sampling rate is used to
detect EMG signal variation during human hand writing in
this paper.

low pass filter block with 10Hz cut-off, which is written in

MATLAB codes, and outputs the muscle activation signal.
Fig. 7: Example of EMG envelope smoothing using low pass
W −1 filter and moving average method.
1
f (At ) = ∑ EMG(At−k ) (3)
W k=0
F. V-Rep : Virtual robot simulator
where f (An ) is enveloped EMG amplitude, W is the moving
window size, EMG(An−k ) is the EMG signals at simple point V-Rep is a useful virtual robot simulator with many
k, t is the EMG sample time. convenient features and elaborate APIs. V-Rep is used to
development algorithm, simulate automatic factory, remote
monitoring and safety checking, etc. It is possible to build
virtual new robot in it, using many existed robot models,
create scenes with varies objects and robot to simulate
working environment. Many robot models with their built
in controller are provided in V-Rep, so that the robots can
be easily controlled without writing many scripts or modifing
many parameters. Each object or model in V-Rep can also
be individually controlled via a plugin, an embedded script,
a remote API client or a ROS node, to perform a custom
solution [14].
III. E XPERIMENTAL S YSTEM S ETUP
OpenHaptics toolkit provides rich information of the
Omni stylus, but there is a problem to be solved, in order
to be used to provide the position data for Baxter R to
Fig. 5: Raw EMG data from MYO device.
follow. The end-effector position provided by Omni is the
position of the crosspoint of the fifth joint and the sixth

2266
joint, which is not the position of the tip of the stylus.
Therefore the position information has to be modified to
extract the correct position of the Omni stylus tip.
Fig. 8 shows the overall design of the system used in this
paper. It is implemented according to the following steps:

• Experimental control bus design. At the centre

of the whole system, Simulink receives data and
sends command to the equipments. There are five
blocks composed of the central system: Omni device
module for sending omni joint/Cartesian position, EMG
detection module for receiving EMG signals from MYO
device, and EMG processing module for extracting
muscle activation reference from EMG signals, Baxter
robot bridge based on Robot Raconteur R developed by
Wason Technology, LLC[15] for managing data from
MATLAB and Baxter robot. V-Rep/Simulink module
for connecting MATLAB and and V-Rep.

• The user wears MYO armband and holds the stylus

of Phantom Omni to write via visual feedback, the
position information of the stylus will be transfered to
Fig. 8: Experimental system design: Human tutor wears
Simulink. The MYO armband read the EMG signals
MYO armband and use omni to teleoperate Baxter robot with
from the operator’s forearm, and sends the filtered
V-Rep inverse kinematic solutions and the system is mainly
EMG signals data to Simulink.
implemented in MATLAB/Simulink. The communication
between Baxter robot and simulink is based UDP.
• Simulink processes the EMG signals into muscle
activation data and adjusts the position data according
to muscle activation. It sends the processed position
along xyz-axis. By measurement, the distance between the
data to V-Rep to perform inverse kinematics, then
crosspoint mentioned before and stylus tip is:
receives the joint angles of the Baxter arm in V-Rep
T  T
simulation.

Dx Dy Dz 1 = 0 0 −40(mm) 1 (5)

• Finally it sends the joint angles to the Baxter robot, To test the effect of the error correction, the stylus tip is
Baxter will then writes on a paper and provide vision standing at a fixed point and the stylus is turned to different
feedback to the operator. orientation. As shown in Fig. 9, the position data varies
before correction, which is not desired because the stylus
A. Position calibration tip does not move.
The Cartesian position data provided by the OpenHaptics
toolkit is corresponding to the position of the crosspoint
between the 5th and the 6th joint axis, not the position of
the stylus tip. In order to obtain the position of the stylus
tip, orientation of the stylus is also needed, and this is also
provided by OpenHaptics toolkit API. The orientation of
the stylus can be found in the homogeneous transformation
matrix, in which the left top 3x3 sub matrix is the orientation
matrix of the stylus and the right top 3x1 vector is the
position of the crosspoint mentioned before. In order to
calculate the position of the stylus tip, it just need to put
the transformation matrix into a simple equation as shown Fig. 9: Position comparabilities before and after correction.
below:
 T  T
xt yt zt 1 = Tr × Dx Dy Dz 1 (4) B. Control implemented in Simulink
where xt , yt , zt are the Cartesian position of the stylus tip, Tr Fig. 10 shows the block diagram of the system used in the
is the transformation matrix and Dx , Dy , Dz are the distance paper, in which the block on the left is an S-Function used
between the crosspoint mentioned before and stylus tip to connect with the Phantom Omni by using OpenHaptic

2267
Tool-kit API. The five blocks in the middle are used to shown below:
perform the position correction and workspace mapping to [∼, ∼] = v.vrep.simxGetJointPosition(v.vrep.clientID,
the position and orientation data from the Omni S-Function. jointhandle, v.vrep.simxo pmodestreaming + al pha);
The three blocks on the top are used to receive and perform
moving average to the EMG signals from MYO armband. After this command is sent to V-Rep, the V-Rep server
The position data and muscle activation data are then fed will continuously executing that command and sent the joint
into the MATLAB function at the right, to calculate the position to MATLAB. In MATLAB, when the data need to
position for the Baxter end-effector and transmit to the be received, execute the following codes
V-Rep by sending it to the V-Rep S-Function. Finally [∼, ∼] = v.vrep.simxGetJointPosition(v.vrep.clientID,
the joint angles read from V-Rep simulation by V-Rep jointhandle, v.vrep.simxo pmodeb u f f er)
S-Function is sent to the Baxter S-Function. and the variable in MATLAB will be refreshed. This method
should minimize the delay cause by waiting for response.

IV. E XPERIMENTAL RESULT

After receiving joint angles of Baxter arm in V-Rep, the
position of Baxter end-effector in V-Rep can be solved by
using V-Rep inverse kinematics built-in solver, while to
ensure Baxter can imitate human writing compliantly, we
constrain baxter robot endpoint orientation in Z direction so
that the brush can be vertical to the desk. The position in
Z direction is controlled by activation coefficients estimated
Fig. 10: MYO armband profile [13]: Wearable MYO arm-
from EMG signals e.g., in Fig. 11. We use Eq. (6) to ensure
band with 8 channels and 200Hz EMG sampling rate is used
the position in Z direction changes in a stable boundary.
to detect EMG signal variation during human hand writing
As a result, the water paint brush set on the Baxter gripper
in this paper.
will write in different depth level according to the muscle
activation of the operator’s arm.
(A f − Amin
f )
C. Inverse kinematics based on V-Rep IK solver Z = (Z max − Z min ) + Z min (6)
(Amax min
f −Af )
The inverse kinematics (IK) function built in the V-Rep is
used in this paper. By setting all the joints of the Baxter arm where z is the desired position, Z max , Z min are Z position
into inverse kinematics, and set the brush as the end-effector boundaries, A f is estimated activation coefficient, Amax min
f , Af
of inverse kinematics, than send command to V-Rep to are estimated activation coefficient boundaries, Fig. 12 shows
make the brush moves as the Phantom Omni stylus, joint the position error between Phantom stylus and Baxter R end-
angles needed for the Baxter to perform the same action effector, the maximum error between them is about 0.08mm,
as the stylus can be found. Using the V-Rep connection and there is almost no time delay, this is acceptable as a
function in Robotics toolbox to remote V-Rep is simply, simulation result.
firstly put one command into V-Rep script to let it execute
once when the simulation start :
simExtRemoteApiStart(19999);
a single line of command in MATLAB can get the
connection between them:
v = V REP(path, directorytoV − Repprogram);
To prevent the delay of waiting for response from V-Rep,
function in Robotics Toolbox[16] should not be used. The
other way to do it is using the original API functions
provided by V-Rep, so that the mode of sending commands
can be selected. In fact, the Robotics Toolbox is also
using the original V-Rep API functions, providing a
Fig. 11: Muscle variations estimated from EMG signals.
simply method to do the connection, therefore some of
the V-Rep API functions such as reading force sensors,
are not assessable by only using the V-Rep object created In real implement of the system, the joints of Baxter R arm
by Robotics Toolbox. There are few operation modes of are all following the Baxter R in V-Rep simulation, the
sending command can be used, e.g. one shot, one shot Baxter R arm perform little damping when the movement
and wait, streaming, streaming and split, buffer. To prevent is fast and longer in distance. The height of the water paint
causing delay when reading data from V-Rep, streaming brush can be adjusted by holding with stronger muscle acti-
and buffer mode can be used, and a sample command is vation in hand, in order to write lines in different thickness.

2268
 Fig. 15: “BAX” “BAX”written by Baxter
R
Fig. 12: Position of styles and Baxter
R end-effector.

R EFERENCES
[1] E. Burdet, R. Osu, D. W Franklin, T. E Milner, and M. Kawato.
The central nervous system stabilizes unstable dynamics by learning
optimal impedance. Nature, 414(6862):446–449, 2001.
[2] E. Burdet, G. Ganesh, C. Yang, and A. Albu-Schäffer. Interaction
force, impedance and trajectory adaptation: By humans, for robots.
In Oussama Khatib, Vijay Kumar, and Gaurav Sukhatme, editors,
Experimental Robotics, volume 79 of Springer Tracts in Advanced
Robotics, pages 331–345. Springer Berlin Heidelberg, 2014.
[3] C. Yang, G. Ganesh, S. Haddadin, S. Parusel, A. Albu-Schaeffer, and
E. Burdet. Human-like adaptation of force and impedance in stable
and unstable interactions. Robotics, IEEE Transactions on, 27(5):918–
Fig. 13: Position error between omni and Baxter
R end- 930, Oct 2011.
[4] C. Yang, Z. Li, and E. Burdet. Human like learning algorithm for
effector. simultaneous force control and haptic identification. In Intelligent
Robots and Systems (IROS), 2013 IEEE/RSJ International Conference
on, pages 710–715, Nov 2013.
[5] A. M Okamura. Methods for haptic feedback in teleoperated
As shown in Fig. 14 below, the Baxter R can follow the
robot-assisted surgery. Industrial Robot: An International Journal,
operators hand to write a recognisable “BAX” with lines of 31(6):499–508, 2004.
different thickness. [6] N. Mavridis, G. Pierris, P. Gallina, Z. Papamitsiou, and U. Saad. On
the subjective difficulty of joystick-based robot arm teleoperation with
auditory feedback. In GCC Conference and Exhibition (GCCCE),
2015 IEEE 8th, pages 1–6. IEEE, 2015.
[7] F. Arai, M. Tanimoto, T. Fukuda, K. Shimojima, H. Matsuura, and
M. Negoro. Multimedia tele-surgery using high speed optical fiber
network and its application to intravascular neurosurgery - system
configuration and computer networked robotic implementation. In
Robotics and Automation, 1996. Proceedings., 1996 IEEE Interna-
tional Conference on, volume 1, pages 878–883 vol.1, Apr 1996.
Fig. 14: “BAX” pre-written to be follow. [8] Baxter robot profile. http://www.rethinkrobotics.com/
products/baxter/. Accessed: 2014-05-01.
[9] Z. Li P. Liang, C. Yang and R. Li. Writing skills transfer from human
to robot using stiffness extracted from semg. June 2015.
V. C ONCLUSION [10] haptic-phantom-omni profile. http://http://www.
dentsable.com/haptic-phantom-omni.htm. Accessed:
In this paper, a method to combine teleoperation joystick 2015-05-01.
Phantom Omni and the sEMG signals has been developed to [11] Geomagic openhaptics toolkit profile. http://www.geomagic.
com/en/products/open-haptics/overview/. Accessed:
control a robot arm writing recognisable words. The joystick 2015-05-01.
is used to control the motion on the horizontal 2D plan, [12] Z. Ju, C. Yang, Z. Li, L. Cheng, and H. Ma. Teleoperation of humanoid
while the muscle contraction activities extracted from sEMG baxter robot using haptic feedback. In Multisensor Fusion and Infor-
mation Integration for Intelligent Systems (MFI), 2014 International
are used to control the motion variation along the vertical Conference on, pages 1–6. IEEE, 2014.
direction to achieve variable thickness of writing with a [13] Myo in use. http://www.traceparts.com/
brush. In the presence of motion amplification when a master news-and-events/2014-press-releases/
pr01102014/. Accessed: 2014-05-01.
device’s workspace is much smaller than the workspace of [14] V-rep profile. http://www.coppeliarobotics.com/. Ac-
the slave robot, the most sensitive motion in vertical direction cessed: 2014-05-01.
is key to the success of writing task. Under the proposed [15] J.D. Wason and J.T. Wen. Robot raconteur: A communication archi-
tecture and library for robotic and automation systems. In Automation
method, the most sensitive motion in the vertical direction Science and Engineering (CASE), 2011 IEEE Conference on, pages
can be well controlled by the human operator using muscle 761–766, Aug 2011.
isometric contraction. For further improvement of the system, [16] Peter Corke. A robotics toolbox for matlab. Robotics & Automation
Magazine, IEEE, 3(1):24–32, 1996.
torque extraction from sEMG can be employed to ensure
force produced by robot arm is proportional to the activation
generated by the operator’s arm.

2269