VISVESVARAYA TECHNOLOGICAL UNIVERSITY

“Jnana Sangama”, Belagavi-590018, Karnataka

A Technical Seminar Report on

“VR Hand Controller Using IMU and Flex Sensor”

Submitted in partial fulfilment of the requirement for the award of degree of

Bachelor of Engineering in
Computer Science and Engineering

Submitted by
HARSHA B R (4NN16CS024)
Under the guidance of
Mr. Ajay A V
Assistant Professor,
Dept. of CSE,
NIEIT, Mysuru-18

Department of Computer Science and Engineering

NIE Institute of Technology, Mysuru
2021-22
 NIE INSTITUTE OF TECHNOLOGY
#50(part), Hootagalli Industrial area, Koorgalli Village Mysuru-18

Department of Computer Science and Engineering

CERTIFICATE

This is to certify that the seminar work entitled “VR Hand Controller Using IMU and Flex Sensor” is
carried out by HARSHA B R, USN 4NN16CS024 in partial fulfilment for the Eighth Semester of
Bachelor of Engineering in Computer Science and Engineering of Visvesvaraya Technological
University, Belagavi during the academic year 2021-22. The Technical Seminar report has been approved
as it satisfies the academic requirements with respect to seminar work prescribed forth Bachelor of
Engineering Degree.

Signature of the Guide Signature of the HOD

Mr. Ajay A V Smt. Usha M S
Asst. Professor, Assoc. Professor & Head,
Dept. of CS & Engineering, Dept. of CS & Engineering,
NIEIT, Mysuru NIEIT, Mysuru
 ACKNOWLEDGEMENT

The satisfaction and euphoria that accompany the successful completion of any task would be
incomplete without the mention of the people who made it possible, whose consistent guidance and
encouragement crowned our efforts with success.

We consider ourselves proud to be a part of NIE Institute of Technology Mysuru family, the institution
which stood by our side in all our endeavors.
We wish to express our gratitude to Dr. Rohini Nagapadma Principal, NIE Institute of Technology
Mysuru, for providing a congenial working environment.

We express our sincere thanks to Smt. Usha M S, Associate Professor and Head, Department of
Computer Science and Engineering, for her support and encouragement.

We would like to thank our guide, Mr. Ajay A V. Assistant Professor, Department of Computer Science
and Engineering, for his inspiration, guidance, constant supervision, direction and discussions in the
successful completion of this technical seminar.

We are thankful to the Department of Computer Science and Engineering staff members and non-staff
members for their co-operation extended towards this work.

Finally, our heartfelt gratitude to our family members, relatives and friends for their constant support,
motivation and encouragement throughout this project. Above all we would like to thank God Almighty,
for having showered his blessings on us.

HARSHA B R (4NN16CS024)

iii
 ABSTRACT

Virtual Reality is a new field on technology industry that has been growing rapidly for
the last decade. One of the improvements made from this industry is Virtual Reality Glove or
VR Glove which is an improvement for Virtual Reality Controller. Although the glove is
popular on PC market, there has yet to be any glove for the smartphone market, even though
both markets have the same potential.
The purpose of this paper is to make a glove controller for a VR application that can be
integrated into smartphone. Taking example from other papers, an Inertial Measurement Unit
(IMU) sensor can capture an object angular rotation and using said rotation, use it to form a
forward kinematic to determine the glove position. In Order to allow communication between
the devices and the target smartphone and to be able to configure its Connection, a Wi-Fi
Module and a small microcontroller board is required. A microcontroller board capable to do
both Wireless connection and act as a microprocessor is ESP32 board. The test resultsof this
project indicate a 1% error on the device rotation and also a delay in
Simulation of 10 times the time required to rotate the device.

iiii
 LIST OF CONTENTS

TITLE PAGE No.

Cover Page i
Certificate ii
Acknowledgement iii
ABSTRACT iv
LIST OF CONTENTS v
LIST OF FIGURES vi
LIST OF TABLES vii

1. INTRODUCTION
1.1 Overview of the Project 1

2. LITERATURE SURVEY
2-4

3. SYSTEM DESIGN
3.1 Proposed System
3.2 Quaternion 5–7

3.3 Madgwick Filter

3.4 Flex Sensor

4. ESP32 Connection 8–9

5. UNITY IMPLEMENTATION
5.1 Model and Environment 10 – 14
5.2 Forward and Inverse Kinematics
5.3 Rotation Test

6. CONCLUSION 15

REFERENCES
16

iv
 LIST OF FIGURES

Figure No. Name of the Figure Page No.

System Design 3
3.1

3.1.2 The Glove and the Armbands 4

4.1 Connection Flowchart 7

5.1 GUI in the Environment 8

5.1.2 Model in Environment 8

5.1.3 Input-output Diagram of the system 9

5.1.4 The First and Second model side by side 10

5.1.5 The First and Second model in the same coordinate 10

v
 LIST OF TABLES

Table No. Name of the Table Page No.

I EEPROM Data 6

II X AXIS ROTATION ON EULER ANGLE 11

III Y AXIS ROTATION ON EULER ANGLE 11

IV Z AXIS ROTATION ON EULER ANGLE 12

V ROTATION TIMING 12

vi
VR Hand Controller Using IMU and Flex Sensor 2021-2022

CHAPTER 1

INTRODUCTION

Virtual Reality Glove or VR Glove is an improvement on controller technology for Virtual

Reality Controller. However, the device has yet to be available on smartphone platform. This paper
mainly studies on how to make a glove shaped VR controller for a smartphone. Flex sensor will
be used to capture a finger’s bend and IMU(Inertial Measurement Unit) sensor willbe used to
calculate the rotation and then using said rotation to be used as inputs in forward kinematics to
find the hand position.
As a wearable device, the device requires a small microprocessor board and a WI-FI
module to communicate with a smartphone, both of which were available at ESP32. Engine that
will be used is the Unity engine because it capable of designing both 2D and 3D programs across
multiple platforms, including smartphones.
The purpose of this paper is to improve the motion tracking device to be able to also detect
the position of hand movement. Detecting hand movement directly is almost impossible if only
IMU (Inertial Measurement Unit) sensor used, however, it is possible to detect hand movement by
constructing a forward kinematic from angular rotation on shoulder, elbow and hand.

DEPT OF CSE 1 NIEIT, MYSURU

VR Hand Controller Using IMU and Flex Sensor 2021-2022

CHAPTER 2

LITERATURE SURVEY

A paper titled “Novel Upper-Limb Rehabilitation System Based on Attention Technology

for Post-Stroke Patients: A Preliminary Study” by Bor-Shing Lin, Jean-Lon Chen, and Hsu-Chi
Hsu is describing a design of motion tracking device which consisted of a battery, an Arduino pro
mini board, a Bluetooth module, a tri axial accelerometer, gyroscope and magnetometer. The
motion tracking device communicates to the VR Game through a Bluetooth module which
connected into the pc where the game is running. The design above was the inspiration for this
paper, however the method to obtain motion through tri axial accelerometer, gyroscope and
magnetometer has limitation that it only capture motion based on tilting and angular rotation.

Another design inspiring this paper is a paper titled “Virtual Reality Glove” by Baird
Eutsler, Jared Gaertner, Evan Pollino, Nina Robinson Mequaint Moges, SaiTadimeti and Kavya
Yerrabandi. The design of the device described in the paper is a glove shaped controller, it used
flex sensor to capture finger movement and require a joystick (Razer Hydra remote) in order to
operate. In this methodology, the function of the joy stick is replaced by an IMU (Inertial
Measurement Unit). The purpose of this methodology is to improve the motion tracking device to
be able to also detect the position of hand movement. Detecting hand movement directly is almost
impossible if only IMU(Inertial Measurement Unit) sensor used, however, it is possible to detect
hand movement by constructing a forward kinematic from angular rotation on shoulder, elbow and
hand. This issue was addressed in second methodology “A Practical Design and Implementation
of a Low Cost Platform for Remote Monitoring of Lower Limb Health of Amputees in the
Developing World” by Neha Mahtur, Greig Paul, James Irvine, Mohamed Abuhelala, Arjan Buis
and Ivan Glesk. The device in the paper used an IMU’s Angular Rotation in order to capture limb
motion of a patient. The IMU Angular Rotation method in this paper willbe used as the basis of
the forward kinematics.

DEPT OF CSE 2 NIEIT, MYSURU

 VR Hand Controller Using IMU and Flex Sensor 2021-2022

In a paper titled Data Glove With a Force Sensor. The objective of this paper is to
demonstrate the development stages of a data glove equipped with force sensors and bend sensors.
This paper demonstrates that using flex sensors mounted on the glove, the finger articulation can
be captured. In addition, using force sensors mounted on the glove, the force produced due to
gripping mechanism of the hand can be obtained and transferred through serial communication
port and replicated within a remote mechanical gripper. As a result, wearing the Glove, if an object
is held with a certain force input (i.e. 4N), a remote mechanical robotic arm should also produce
the same grasp force. As mentioned in the paper, this development can be used in robotics,
telecheric, biomechanics, rehabilitation and virtual reality applications. The following is the
prototypical design of the Data Glove used in the project.

Although this paper gives the detailed description of force data acquisition algorithm, it
however is not the focus of the capstone design project. Nonetheless, the following circuit topology
presented in this paper has been adapted for the capstone design project. The circuit is composed
of three subdivided opamps circuits known as the buffer, the adder and the amplifier. The buffer
circuit simply act as a voltage follower to output the voltage produced at the voltage divider circuit.
The adder circuit adds positive voltage value to the buffer circuit so that the low negative voltage
is offset by the comparably higher voltage of the adder output. Finally, the small voltage is
amplified through the amplifier circuit and gives a amplified voltage that falls between desired
voltage range.

In a paper titled Resistive flex sensors: a survey by Giovanni Saggio, Francesco Riillo, Laura
Sbernini and Lucia Rita Quitadamo. As sensors, RFSs operate on the same principle as strain gauges
(flexion of the material induces tension along one side of the bending radius and compression along
the other side), but with a substantially greater sensitivity. RFSs are fundamentally based on the
variable resistive characteristics of a conductive material, typically in an ink form, so as to become a
part of that conductive ink business market that had generated about $1.6 billion by the end of 2014,
growing to $2.8 billion by 2024 [20]. Recently, new forms such as powder, paste or film ones, have
been growing in importance, possibly mixed with a resin, and/or with fillers and/or binders and/or
additives, diluted in a solvent or a thinner if necessary, and engineered with different sorts of
substrates.
DEPT OF CSE 3 NIEIT, MYSURU
 VR Hand Controller Using IMU and Flex Sensor 2021-2022

Originally, this type of sensor was developed by AGE Company (Abrams/Gentile Entertainment,
New York) only and used in the so-called ‘PowerGlove’ (by Mattel), a sensory glove capable of
measuring finger flexion. For some years there was no way of obtaining some of those sensors unless
they were removed from the glove itself. Nowadays, RFSs have been commercialized by different
brands, among which are Spectra Symbol (www.spectrasymbol.com, with ‘FlexSensor4.5’
commercialized by the online retail store Sparkfun.com), Abrams-Gentile (www.ageinc.com),
Flexpoint Sensor Systems, Inc. (www.flexpoint.com, with the proprietary ‘Bend Sensors®’
carbon/polymer based ink) and Images SI Inc. (www.images.com, that furnishes the ‘Bi-Flex
Sensors™’ capable of bidirectional behavior). The RFSs mostly commonly adopted and reported in
the literature are those produced by Flexpoint Inc. and Image SI Inc. (figure 7). Nowadays, the typical
commercial RFS can be purchased for as little as tens of US dollars.

DEPT OF CSE 4 NIEIT, MYSURU

VR Hand Controller Using IMU and Flex Sensor 2021-2022

CHAPTER 3

SYSTEM DESIGN
3.1 Proposed System

This System Design section will explain how the device’s system was made. The system consisted
of two groups of devices; the first group consisted of a glove and two armbands and the other
group is the smartphone, as represented by the diagram below:

Figure 3.1.System Diagram

The diagram above shows the first group on top and the second group bellow. The first
group consisted of a sensor and their processor, which were the MPU-9250 for the IMU sensor
and ESP32 as the processor. The glove is relatively the same, but it also has flex sensors to capture
fingers movements.

DEPT OF CSE 5 NIEIT, MYSURU

VR Hand Controller Using IMU and Flex Sensor 2021-2022

The second group is the smartphone, especially its component, both the hardware and
software. It receives data from the first group through the phone’s wireless module, or even better
if the phone could host its own wireless. The data then processed into the Unity app while also
taking the phone’s IMU data and then showed the movement on the phone’s screen.
The system works by gathering the rotation from the IMUs and using the rotation on the
user’s hand, elbow and shoulder to move a model hand, elbow and shoulder. The user’s rotation
will form a forward kinematic movement toward the model’s hand, which will then be used as a
reference for the Unity’s inverse kinematic to move the elbow and shoulder of the model. The
user’s finger movement will also be captured through the use of the flex sensor. However to make
the system simpler, in this paper the device will only use two flex sensor, one to index finger and
the other to pinky finger. The ADC output of the sensor will be used as a reference on how much
the hand bends. Below is the hardware of this system.

Figure 3.1.2.The glove and the armbands.

3.2 Quaternion

Euler Angler are angles most familiar to a lot of people, however on Unity movement,
the Euler angle has a problem with gimbal lock, causing it to twitch and stop when two axis were
on the same plane. In order to avoid the gimbal lock, the angle must be represented in Quaternions
angle. Quaternions are a number system that extends the complex numbers. They were first
described by Irish mathematician William Rowan Hamilton in 1843 and applied to mechanics in
three-dimensional space.

DEPT OF CSE 6 NIEIT, MYSURU

VR Hand Controller Using IMU and Flex Sensor 2021-2022

A feature of quaternions is that multiplication of two quaternions is non-commutative.

Hamilton defined a quaternion as the quotient of two directed lines in a three-dimensional space
or equivalently as the quotient of two vectors. Quaternions are generally represented in the form:
a + bi + cj + dk Where a, b, c, and d are real numbers, and i, j, and k are the fundamental quaternion
units. It can also be represented by a 2x2 matrix.

3.3 Madgwick Filter

The Madgwick Filter is a filter made by Sebastian O.H. Madgwick using IMU’s outputs as
inputs [4]. It uses the IMU 3 axis rotation from gyroscope, 3 axis acceleration from accelerometer
and 3 axis magnetic fields from magnetometer into Pitch, Yaw and Roll. The filter is chosen
because it does the calculation using Quaternion. However because the output is in Pitch, Yaw and
Roll of Euler angle, a new function was made to take the Quaternion from the filter calculation.
Madgwick Filter uses three axis of gyroscope and accelerometer to calculate the orientation, then
adjust the sensor frame toward earth using Magnetometer. The resulthowever, was distorted from
the presence of ferromagnetic objects near the magnetometer, so itis required to be calibrating
and normalizing. Aside from Magnetometer, the gyroscope also hasa bias that will keep increasing
the longer the sensor work. The bias can be reduced by comparing the orientation from filter above
and orientation based on magnetometer. The comparison then will be used as a reference to reduce
bias on gyroscope.

3.4 Flex Sensor

Flex sensor is an analogue sensor and it work by increasing its resistance if it’s being bended. The
ADC result of a bended finger will be lower voltage than when the finger was straight. The sensor
was tested and recorded at which Threshold it bends.

DEPT OF CSE 7 NIEIT, MYSURU

VR Hand Controller Using IMU and Flex Sensor 2021-2022

CHAPTER 4

ESP32 CONNECTION

The Project will be using wireless connection to communicate between the device
microcontroller and smartphone. The microcontrollers used are ESP32 which already have
wireless module embedded to the board. The connection is using the UDP structure with all devices
connected to the same network. The first group devices will send its device ID to all devices in the
network, but only the second group device, the server, recognize it. Once a device is recognized,
the server will record the device IP and send a recognition string to the device. In return, the device
stops its ID broadcast and start broadcasting its IMU quaternion data into the server. In case of a
crash, especially due to the filter requiring a much faster processing speed, the system will reboot
and recognized a crash has happened. This is because of the ESP32 EEPROM, which retains the
data stored in it, even after a sudden shutdown or flashing. The data stored in the EEPROM are
listed in the following table.

TABLE I. EEPROM DATA

DEPT OF CSE 8 NIEIT, MYSURU

VR Hand Controller Using IMU and Flex Sensor 2021-2022

The interaction between the devices can be seen in the flowchart bellow.

Figure 4.1.Connection Flowchart

The figure above is a flowchart of the wireless connection. The Server is on the left and the first
group device is on the right. The Unity App will listen to any broadcast in the wireless network
that the smartphone connected to and if it receives any ID, it will record the IP and send
confirmation. It will then listen to the recorded IP for any IMU data sent into it.

DEPT OF CSE 9 NIEIT, MYSURU

VR Hand Controller Using IMU and Flex Sensor 2021-2022

CHAPTER 5
Unity Implementation

Below are the results of the Implementation of the above connection and sensor reading. The
data received from the IMU will be used in 3D Environment where it will be attached to a model
in order to move it.

A. Model and Environment

The Environment consisted of the GUI, the model and the moving skeleton.
Below is the first of the environment, the GUI.

Figure.5.1. GUI in the Environment

Figure.5.1.2. Model in Environment

The picture above showed the model and a representation of the moving skeleton; however the
skeleton might not visible in the final product. The quaternion of IMU rotations will be used in
determining the ball rotation using forward kinematics.

DEPT OF CSE 10 NIEIT, MYSURU

VR Hand Controller Using IMU and Flex Sensor 2021-2022

B. Forward and Inverse Kinematics

Forward kinematics is the usage of the kinematic equations to compute a position
of the end-effector from specified values for the joint parameters [6], which means that
inverse kinematic, is the opposite, a process to recover movement of an object from other
data in this case, the position of the end-effector from forward kinematics.
The inverse kinematics of Unity recovers the hand movement based on the red ball
position. It will then affect the model’s hand, then elbow and finally the model shoulder,
causing the model to imitate the movement done by user. Below is a diagram explaining
how the IMU’s output corresponds into the model rotation.

Figure.5.1.3 Input-Output Diagram of the system

The Three IMU quaternion rotations are used to move the skeleton, which represented by the
colored ball. The balls rotated based on their input, moving their child object in the process. The
hand (red ball) being the end of the parent-child link, rotate along the blue ball and along the green
ball. The arm of the model will follow wherever the red ball move due to the set inverse kinematic,
thus mimicking the arm movement of the skeleton.

DEPT OF CSE 11 NIEIT, MYSURU

VR Hand Controller Using IMU and Flex Sensor 2021-2022

C. Rotation Test
In order to test the accuracy of the rotation, a test is required, however due to
using quaternion, the rotation of the object shown in Unity inspector is not Euler angle, but
instead in quaternions. So, to be able to compare the result, a second model was used.

Figure.5.1.4 The First and Second Model side by side

Figure.5.1.5 The First and Second Model in the same coordinate

DEPT OF CSE 12 NIEIT, MYSURU

VR Hand Controller Using IMU and Flex Sensor 2021-2022

The two models will be placed in the same coordinate, causing the grey covering the other.
Using the unity editor window, the second model’s arm can be rotated during testing and the result
of the rotation will be shown as Euler angles. Using this method, we can determine the accuracy
of the rotation.
a) Rotation Test
The similar build of the device cause the reading of the IMU rotation similar to
others. So instead of three tests for each IMU, the result shown here will just be one IMU.

Table II. X Axis Rotation on Euler Angle

According to the table above, the rotation of X axis is rather accurate with error percentage
of 0.98%. The value of the rotation became negative because it move counterclockwise.

Table III. Y Axis Rotation on Euler Angle

According to the table above, the rotation of Y axis is rather accurate with average error
percentage of 1.00%. The value of the rotation is positive because it moves clockwise.

DEPT OF CSE 13 NIEIT, MYSURU

VR Hand Controller Using IMU and Flex Sensor 2021-2022

Table IV. Z Axis Rotation on Euler Angle

According to the table above, the biggest error is 1.25% when the device is lifted 180 degree,
but the simulation dictate that it stopped at 225.9 degree. The average error for this rotation is
1.05%.

b) Speed Test
The purpose of this test is to determine the required time of a rotation from 0 degree
moving up to 180 degree, both in real rotation and simulation rotation. Below is a table of
rotation compared to the real and simulation time required to reach the target rotations.
Table V. Rotation Timing

The result of X, Y and Z axis were similar, so the three rotations merged into one. As seen
in this table, the time required to move into the target rotation is 10 times slower than the real time.

DEPT OF CSE 14 NIEIT, MYSURU

VR Hand Controller Using IMU and Flex Sensor 2021-2022

CHAPTER 6
CONCLUSION

The model was moved by rotating three aligned balls (skeleton) which were rotated by
IMU data and moved by forward kinematics. The result of rotation showed 1% error average and
movement speed 10 times slower than the real movement. The slowing of the simulation might
have been caused by the ESP32 lack of processing speed which the Madgwick filter requires to
work faster.

DEPT OF CSE 15 NIEIT, MYSURU

VR Hand Controller Using IMU and Flex Sensor 2021-2022

REFERENCES

[1]. Bor-Shing Lin, Jean-Lon Chen, and Hsu-Chi Hsu, (2018), “Novel Upper-Limb
Rehabilitation System Based on Attention Technology for Post-Stroke Patients: A
Preliminary Study” in IEEE Access, Volume 6, pp 2720-2731. 18 December 2017
[2]. Neha Mathur, Greig Paul, (2016), “A Practical Design and Implementation of a Low Cost
Platform for Remote Monitoring of Lower Limb Health of Amputees in the Developing
World”.
[3]. Baird Eutsler, Jared Gaertner, (2015), “Virtual Reality Glove”.
[4]. Sebastian O. H. Madgwick, Andrew J. L. Harrison, Ravi Vaidyanathan, (2011), Estimation
of IMU and MARG orientation using a gradient descent algorithm. 2011 IEEE International
Conference on Rehabilitation Robotics. Zurich, Switzerland.

[5]. Quaternions Accessed: Online Available: https://en.wikipedia.org/wiki/Quaternion

[6]. Forward kinematics Accessed: https://en.wikipedia.org/wiki/Forward_kinematics

[7]. Inverse Kinematics. Accessed: https://en.wikipedia.org/wiki/Inverse_kinematics

[8]. Unity Inverse Kinematic: https://docs.unity3d.com/Manual/InverseKinematics.html

DEPT OF CSE 16 NIEIT, MYSURU