Volume 4, Issue 11, November – 2019 International Journal of Innovative Science and Research Technology

ISSN No:-2456-2165

Musculoskeletal Physiotherapy using Artificial

Intelligence and Machine Learning
Dr. Sachin. P. Godse Shalini Singh
Computer Engineering Computer Engineering
Sinhgad Academy of Engineering, Pune-411048 Sinhgad Academy of Engineering, Pune-411048

Sonal Khule Vedant Yadav Shubham Wakhare

Computer Engineering Computer Engineering Computer Engineering
Sinhgad Academy of Engineering Sinhgad Academy of Engineering Sinhgad Academy of Engineering
Pune-411048 Pune-411048 Pune-411048

Abstract:- Physiotherapy is the trending medication for It is observed that technology in the field of medical has
curing bones related injuries and pain. In many cases, due been proven very beneficial. Varied amount of applications are
to sudden jerks or accidents, the patient might suffer from being developed which are making life of doctors as well as
severe pain. Therefore, it is the miracle medication for patients easier. Technologies such as virtual reality, artificial
curing patients. Our aim here is to build a framework intelligence and machine learning are among the most popular
using Artificial Intelligence and Machine Learning for one in Medical applications. Exoskeletons is one of them
providing users with a digitalized system for which is use for arms, legs and hands videogames that
physiotherapy. Even though varied computer-aided motivate patients to move and motivates them to improve.
assessment of physiotherapy rehabilitation exist still recent
approaches for computer aided monitoring and With the help of applications like these medical
performance lack versatility and robustness. In our technology therapists are getting tools they need to give the
approach we have come up with proposition of an kind of care and support which is required by the patients in
application which will record user’s physiotherapy need. It is also decreasing workloads from the therapists and
exercises and also it will provide personalized advice based providing them means to treat more patients than before.
on user performance for refinement of therapy. By using Therefore, advancement of technology in medical with the
Open Pose Library our system will detect angle between help of cutting edge technology inspired us to develop a
the joints and depending upon the range of motion it will system which will be changing the lives of therapists as well as
guide patient in accomplishing physiotherapy at home. It patients everywhere.
will also suggest patients about different physio-exercises.
With the help of Open Pose it is possible to render the  Problem Statement
patient’s images or real-time video. People might have hurt their joints or for some other
reason and cannot use their limbs to the full extent, such as
Keyword:- Physiotherapy, Artificial Intelligence, Machine after a fall, a stroke, or an accident. There is a need to develop
Learning, Open Pose, Rendering. an application to distinguish normal person and affected
person using Artificial intelligence and Machine learning and
I. INTRODUCTION to provide the need of physiotherapy to the affected people.
We are proposing an idea of Automated system which will
 Motivation track human Range of motion while physiotherapy of patient.
Physiotherapists often work with other health
professionals to meet individual’s health care needs. These  Background
days demands of physiotherapist are increasing but there is a In recent years with increase in treatment of injury,
lack of supply. Also, there are many benefits of physiotherapy physical strength, functioning of body and overall movement
which includes avoiding surgery, improved mobility and of body with involvement of science of movement of body
development, management of age related problem and rather than involvement of drugs and surgery has evolved need
improved balance. With increase in demand of physiotherapy of physiotherapy. With increase in physical activity the
its demand at home is also rising. But we cannot deny the fact demand of physiotherapy has increased. As well as it has
there are some advantages of physiotherapy at home. At home increased the requirement of rehabilitation centers and demand
better health outcomes are observed when a person is of physiotherapists. Technology in the field of medical is now
surrounded by people with whom they feel connected and at acting as supporting system of workload on physiotherapists.
home they feel positive. Also, it is observed that at home
healing is done faster.

IJISRT19NOV347 www.ijisrt.com 592

Volume 4, Issue 11, November – 2019 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
Artificial Intelligence is technically defined as the rehabilitation that monitors the patient.
development of technology which is used to perform
technology operations which require involvement of human  Proposed System
intelligence. Recently Artificial intelligence is playing major Our proposed system consists of series of processing
role in advancement of technology. Machine learning is one of starting from recording of video at one end to evaluation of
the key component of artificial intelligence and it provides us pose detected at the other end. Our proposed architecture
with the ability of both supervised and unsupervised learning consists of different stages such as video recording, pose
for training our model. Also, there are copious amount of detection, estimation of human key points and lastly
model provided by machine learning which can be used for evaluation of physiotherapy for betterment of therapy. Given
better training and prediction of our system. AI technology below is the architecture of our system which includes
today can be in different forms such as software programs as pipeline of our proposed idea.
well as hardware interface to develop a system which is
capable of learning from their own datasets. In our project AI  Uploading Recorded Video
with machine learning can be used for posture detection and Initial step for the patient who will be using our system
then assessment of patients. will be to record video of the exercise patient will be
performing. In recording of video we are providing complete
Open Pose is an open source library developed in C++ flexibility to user. Patient will be given complete liberty to let
for the posture detection. It is used by importing open pose them crop and to choose which video they want to upload.
and later image is passed as NumPy matrices which is then Once patient will be done recording and uploading video, next
converted to human key points. It is combination of Caffee, step will involve our posture detection library. Posture
OpenCV and OpenCL which is used for rendering of passed detection library will be used for rendering and generating
image. It is used in different fields such as hand gesture human key points using Open pose which is built on C++ and
detection, Basketball games for prediction of basket throw and includes Cafee, OpenCV, OpenCL and Cuda.
various other sports activities. It is also easier to use because
of its capability to replace methods which involve high cost  Posture Detection
along with more equipment and time for processing. Our proposed system consists of series of processing
starting from recording of video at one end to evaluation of
 Existing System pose detected at the other end. Our proposed architecture
Our system don’t demand the need for any personal consists of different stages such as video recording, pose
physio- therapists to look after the patient and teach exercises. detection, estimation of human key points and lastly
It’s all online, and no hardware is required and inexpensive as evaluation of physiotherapy for betterment of therapy. Given
only pictures or videos are required for successful execution. below is the architecture of our system which includes pipeline
 Physera: Online application that provides patients with of our proposed idea.
excellent physiotherapists. The patient can have face- to-
face video chat with their PT and escape their pain.  Scoring Function
Personalized plans are available to buy. In our system after we are done with the detection of key
 Sword Health: Digitalized physiotherapy solution. Digital points then we execute our parser which uses key points as
physio therapists are available to check patient’s motion input. These set of key points is recorded in the form of x and
and provide them exercises. Patients can chat with their y coordinates where x will represent frame of recorded video
therapists to adopt the changes physically and emotionally. and y represents angle which is recorded. These set of inputs
That is, real- time bio-feedback is provided. Patients can which we will get after posture detection will be then
just stay at home and learn these physio-exercises and compared with already fed input key points of particular
improve their health. exercise performed by healthy person.
 Physitrack: Application that provides home solutions to
patients in areas such as Orthopedics, Neurology, etc. Later depending upon the values of current therapy key
Patients can view and learn from thousands of uploaded points and already fed key points error will be calculated. For
videos and can also upload their own PDFs and exercise calculation of scoring function we will take average of error
videos. It also allows to book appointment in nearby which will be normalized to range between 0 and 1. After the
clinics. calculation of score of therapy, current score will be then
 Vera Health- Family Doctor Clinic: Growing community of compared with previous score for evaluation of progress.
doctors. It provides services in Vera Care, Vera Wellness,
Vera Physio and Vera Community. It is an offline

IJISRT19NOV347 www.ijisrt.com 593

Volume 4, Issue 11, November – 2019 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
II. WORKFLOW

Fig 1:- Workflow of the Model

In brief complete workflow of our project consists of III. SYSTEM FUNCTIONALITY

steps of processing which will include uploading video of
exercise, pose estimation, scoring function, result and sending  Register:
of feedback to doctors for further suggestion. At the initial Interface that allows the "Doctor" or "User" actor to
stage users will be expected to register and depending on the provide credentials/information required for registering with
instructions of musculoskeletal exercises they will be required the application.
to record video of the exercise. At the later stage of processing
pose estimation will be executed and it will be analysed under  Login:
the machine learning algorithms. Interface for "Doctor" or "User" actor to sign into the
application. Credentials are necessary to log in.
In our proposed system along with video, few extra
inputs such as weight, height etc. will also be asked for  Input Video:
accuracy of analysis. As we are going to consider range of The raw media, such as image or video footage. Raw
motion for the analysis of physiotherapy therefore, there are video is pass to the rendered engine for further post-
different perspectives which are needed to consider in our processing.
model. Person with different weight have different joint angles
and if not taken into account can result in inaccuracy of  Image Processing:
desired results. The raw video is shredded into frames. These frames are
analyzed, then resized to a specific form factor and converted
Last stage of our model will present result in different to certain format with minimum quality loss. Frame
format such as tabular or graphical representation of results. processing is necessary since it reduces the load on processors
These reports will be then forwarded to doctors whom you during rendering as well as it eliminates bad frames during
need to consult, which will incorporate involvement of doctors sanity check.
in the process of your recovery. Inclusion of diet charts will
also add as extras in our system because it is an important step  Pose Estimation:
in recovery of any patient with little bit of motivation and Frames are rendered to estimate human posture. During
positivity. rendering task keypoints are generated for the estimated
posture. Keypoints are numeric values for 18 body parts ->

IJISRT19NOV347 www.ijisrt.com 594

Volume 4, Issue 11, November – 2019 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
"nose", "neck", "rshoulder", "relbow", "rwrist", "lshoulder", be then compared with previous error records to conclude
"lelbow", "lwrist", "rhip", "rknee", "rankle", "lhip", "lknee", about the performance of physiotherapy.
"lankle", "reye", "leye", "rear", "lear".
 Result:
 Performance Metrics: Based on previous calculations, a detailed report
Rendered key points are parsed to estimate range or consisting of rendered images and performance metrics is
motion, angle, measurement between two key points. generated for recipient, that is interactive as well as easier to
understand.
 Scoring Function:
For error calculation and analysis of exercises performed  Feedback:
by patients Scoring Function will be used. It will compare Remedy to improve posture. An exercise is suggested to
processed key points with already fed key points of healthy the recipient.
human. For comparison extra inputs that includes height,
weight, etc will be considered which will decide the dataset  Logout:
from which we need to compare. For every key points, error Interface for "Doctor" or "User" actor to log out of the
will ne calculated which will be normalized first and then application.
mean square error will be calculated. Current error record will

Fig 2:- Use Case Diagram

IJISRT19NOV347 www.ijisrt.com 595

Volume 4, Issue 11, November – 2019 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
IV. LITERATURE SURVEY

Classification Classification label Data source Classification Algorithm used

question Accuracy[%]
Ashinsky et al. (2017) Is pathology present Fracture (vs no X-ray 83 16 layer CNN
or not? fracture)
Olczak et al. (2017) Pfirmann grade Disc MRI 70.4 CNN
narrowing 75.4
Spondylolisthesis 95.4
Central canal 94.7
Stenosis

Jamaludin Cartilage mapping MRI 75 WND-CHRM

et al. (2017)

Bagarinao et al. Is pain neuromapping Structural MRI CPP 73 SVM

(2014) phenotype identifiable
with clinical
Ung et al. diagnosis? CLBP 76 SVM
(2012)
Robinson et al. FM 76 LR, MP, Bayes,
(2015) fMRI SVM, j48 DT

Callan et CLBP 92 SLR

al. (2014)

López-Solà et al. FM 93 SVM, LR

(2017)

Burns et al. (2018) Can successful Accurate exercise Inertial Sensor 99.4 CNN k-NN SVM
exercise performance performance 97.8
be identified? 94.1

Kianifar et al. (2017) Can risk of injury be Risk of injury with Inertial sensor 95 10F-CV
classified based upon movement:
movement quality? “high” vs “low”
72 10F-CV LOSO-CV
“high” vs “moderate”
“low” LOSO-CV
920
Accuracy of
performance: “poor” vs
“good” 60

“poor” vs “moderate”
vs “good”

Nijewemed’Hollosy et Can CLBP “Physiotherapy” vs HER 71.05 DT BT

al. subgroups be stratified “GP” vs 71.05
(2018) accurately? “self-management”

Sen Qiao Can different gestures Camera 83.3 2-Branch Multi-

Yilin Wang be described stage CNN
Jian Li precisely? Bezier Curve

IJISRT19NOV347 www.ijisrt.com 596

Volume 4, Issue 11, November – 2019 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
Steven Chen Can user’s exercise DTW Camera 73 OpenPose
Richard Yang pose be detected Multi-Stage CNN
accurately? CUDA
CuDNN
Yalin Liao Can patient’s DTW Motion Sensors 87 NN
performance in
Aleksandar Vakanski prescribed Human Movement

Min Xian rehabilitation Modelling Rehabilitation

exercises exercise
improved?
Masato Nakai Is ball entering the Real time diagnosis Camera 41 Logistic regression
Yoshihiko Tsunoda basket or not as a
Hisashi Hayashi binary target variable? Motion capture
Hideki Murakoshi analysis

Video motion
analysis

Gin´es Hidalgo Can whole body PAF COCO dataset 70 CNN

Mart´ınez pose be estimated?
Confidence map OpenPose foot Multi-task Learning
dataset
OpenPose hand
dataset
Table 1

V. CONCLUSION providing support for physiotherapy but it will also help in

maintaining emotional stability of the human for healing.
Our approach will be an end to end computer vision
application which will be using pose estimation technique and REFERENCES
visual geometry to provide personalize feedback on fitness
exercise. We are using Machine learning because it has [1]. Niall McGrane, Tara Cusack, Grainne O’Donoghue,
potential to involve physiotherapy practice through human Emma Stokes1, Motivational strategies for
level diagnostic, decision making and measurement. We are physiotherapists, Physical Therapy Reviews VOL. 19
also using a function to score which will map the performance NO. 2,2014
metric into movement quality scores and to generate metrics [2]. Masato Nakai, Yoshihiko Tsunoda, Hisashi Hayashi,
score for repetitions of exercises. We use output of pose Hideki Murakoshi, Prediction of Basketball Free Throw
estimation with the help of open pose to evaluate Shooting by Open- Pose, School of Industrial Technology,
physiotherapy videos to evaluate exercise through human pose Advanced Institute of Indus- trial Technology 1-10-40,
key points. We will be using this to for assessing and Higashiooi, Shinagawa-Ku, Tokyo, 140-0011, Japan,
improvement of rehabilitation exercise. October, 2018
[3]. Shabnam Sadeghi Esfahlani, Tommy Thompson,
In future scope we can also add a chat bot system which Intelligent Physiotherapy Through Procedural Content
will be solve the issue of reluctant nature of patients toward Generation, Department of Computing Technology
physiotherapy. Mostly patient’s don’t see improvement in Anglia Ruskin University Cambridge, UK, 2016
therapy because there are some set of exercises which are [4]. Zhe Cao, Student Member, IEEE, Gines Hidalgo, Student
needed to be performed with strict discipline. Most of the Member, IEEE, Tomas Simon, Shih-En Wei, and Yaser
patients don’t follow regularity in their therapy. Therefore, Sheikh, OpenPose: Realtime Multi- Person 2D Pose
this chat bot system will be using cognitive therapy for Estimation using Part Affinity Fields, 30 May 2019.
understanding of psychology of patients for faster healing of [5]. Christopher Tack, Artificial intelligence and machine
them and to maintain regularity. It will also be able to keep learning — applications in musculoskeletal
track of their routines and it will chat with them on regular physiotherapy, Guy’s and St Thomas’ NHS Foundation
basis for collecting more data for processing of psychology. Trust, Guy’s Hospital, Great Maze Pond, SE1 9RT,
Thus we will be able to make a system which will not only be London, UK, 2019.

IJISRT19NOV347 www.ijisrt.com 597

Volume 4, Issue 11, November – 2019 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
[6]. Daniil Osokin, Real-time 2D Multi-Person Pose
Estimation on CPU: Lightweight OpenPose,29 Nov 2018
[7]. Yalin Liao, Aleksandar Vakanski, Member, IEEE, and
Min Xian1, A Deep Learning Framework for Assessing
Physical Rehabilitation Exercises, 30 Jan 2019
[8]. Sumaira Ghazal, Umar S. Khan, Human Posture
Classification Using Skeleton Information, National
University of Sciences and Technology, H-12, Islamabad,
Pakistan,2018
[9]. Gin´es Hidalgo Mart´ınez, OpenPose: Whole-Body Pose
Estimation, April 2019
[10]. Steven Chen, Richard Yang, Pose Trainer: Correcting
Exercise Posture using Pose Estimation,Department of
Computer Science, Stanford University
[11]. Sen Qiao,Yilin Wang,Jian Li,Real-Time Human Gesture
Grading Based on OpenPose,2017
[12]. Junchen Shen, Yanlin Luo, Zhongke Wu, Yun Tian,
Qingqiong Deng, CUDA-based real-time hand gesture
interaction and visualization for CT volume dataset using
leap motion,Springer-Verlag Berlin Heidelberg 2016.

IJISRT19NOV347 www.ijisrt.com 598