MEE6099 – Master’s Thesis

First Review Presentation

Development of framework to enhance

learning with XR

BalaSeshan Pillalamarri - 19MCD1002

Dr. Janardhan Reddy K

(Internal)
 MEE6099 – Master’s Thesis
First Review Presentation

Contents
• Introduction
• Literature survey
• Research Gap
• Problem Definition
• Objectives
• Methodology
• Implementation
• Results and Discussion
• Future Scope
• References

School of Mechanical Engineering, VIT

07/27/2021 Chennai
 MEE6099 – Master’s Thesis
First Review Presentation

Introduction
• Virtual reality (VR) is a rich, 3-dimensional, 360-degree fully artificial
digital environment in which users can immerse themselves and
interact with.
• Augmented reality (AR) overlays virtual objects (sound, video, or
graphics) on the real-world environment (user’s physical
surroundings).
• Mixed reality(MR) is the merging of real and virtual worlds to
produce new environments and visualizations where physical and
digital objects co-exist and interact in real time.

07/27/2021 School of Mechanical Engineering, VIT Chennai

 MEE6099 – Master’s Thesis
First Review Presentation

Introduction
• XR technology has a wide scope in education and training, from
augmenting encounters in the physical world to making simulations
of things that are difficult to reach in the physical world
• Training has become vital in every enterprise before moving
employees on to the required task/project and also consumes their
active time and expenditure to have extensive manner.
• The order of day is to make the learning to happen equivalent to
physical training, there is a need for blending technologies with
digital knowledge.
• So, to meet this need, the extended reality [XR] would play a major
role as it can incorporate active experimental learning, and XR also
allow to share experiences and collaboration.

School of Mechanical Engineering, VIT

Chennai
 MEE6099 – Master’s Thesis
First Review Presentation

Literature Survey
Sl. Author (Year) Journal Name Remarks
No.
1 Rahul Prasanna Journal of • Clinical use of MR in workflow and
Kumar et al. Biomedical qualitative user evaluation
(2020) Informatics: X

2 Simon Kind et al. Procedia CIRP • Development method for haptic interaction
with force-feedback device for assembly
(2020) validation is presented.
• Focusing on development process of a mixed
prototype with virtual and physical elements

3 Mafkereseb Digital Applications • MxR Map to enable users to interact with

Kassahun Bekele in Archaeology and virtual objects via maps that are virtually
(2019) Cultural Heritage projected on the floor and viewable
through a MxR device, specifically the
Microsoft HoloLens

School of Mechanical Engineering, VIT

Chennai
 MEE6099 – Master’s Thesis
First Review Presentation

Literature Survey
Sl. Author (Year) Journal Remarks
No. Name
4 Nick Salamon et al. Acta • Virtual Reality (VR) simulation of
(2018) Astronautica Astronaut crews to maintain the steady
mental health for long space durations
5 Steffen Kunnen et Procedia CIRP • MR application in maintenance method by
al. (2020) comparing coordinates with
parameterized system and transferring the
data sets to Product DATA Model (PDM)
element
6 Marvin Czarki et al. Procedia • Mixed Reality implementation with a
(2020) Manufacturing cyber physical system based on
performance simulation with CFD
simulation is composed
7 Dan Li et al.(2018) Procedia • XR cases studies for 3 different scenarios
Manufacturing showing the technological advantages

School of Mechanical Engineering, VIT Chennai

 MEE6099 – Master’s Thesis
First Review Presentation

Literature Survey
Sl. Author (Year) Journal Name Remarks
No.
8 Ahmad Hadavi Automation in • Review paper which discusses most
et al. (2019) Construction recent XR technologies in Design and
construction industries
9 Huang T.L. and Internet Research • AR to examine the extent to which
Liu F.S. (2014) presence, media richness and narrative
experiences yield the highest experiential
value for online consumers

10 Tussyadiah I.P. et Tourism • VR based visitors experience interacting

al. (2018) Management with exhibits and, also provide a
theoretical reflection of this technology in
mediation and tourism attraction

School of Mechanical Engineering, VIT Chennai

 MEE6099 – Master’s Thesis
First Review Presentation

Lacunae in literature
• From the research papers its evident that XR has huge scope but this
technology has been restricted to high level research and can also be
implemented to University and training level to support in
pandemic situations
• The XR technology has huge advantages in Designing and
Innovation - in field of design communication which helps in
concurrent engineering
• This technology has been explored and implemented to maximum
extent in Advertising and Branding but its not used much in
training which is changing due to current situation

School of Mechanical Engineering, VIT

Chennai
 MEE6099 – Master’s Thesis
First Review Presentation

Problem Statement
• Non availability of XR usage in Technical Education and
training for field operations has to be addressed to
enhance Learning and improve the learning speed

School of Mechanical Engineering, VIT

Chennai
 MEE6099 – Master’s Thesis
First Review Presentation

Objectives
• The main work focuses on developing and evaluating of
framework with XR technology that can enhance
learning and training

School of Mechanical Engineering, VIT

Chennai
 MEE6099 – Master’s Thesis
First Review Presentation

Methodology / Work Plan

Section I: Section V:
Literature Survey Section III: Conclusion
Implementation
Literature on use of Extended Reality • Bridging the Gap • Scope for future work
in Training and Education • Case Study
• Augmented Reality • Creating Interactive
• Virtual Reality Model
• Extended Reality • Defining Constraints
Problem Identification

Section IV:
Section II: Results
Methodology
• Key Findings
• Formulation of Work flow and • Results and Discussion
toolset required • Evaluation of implemented framework
• Establishing CAD to XR Toolset

07/27/2021 School of Mechanical Engineering, VIT Chennai

 MEE6099 – Master’s Thesis
First Review Presentation

Blender for Development

 In development of both AR and VR Blender is used to convert the .wrl files to .fbx(for
Unity only)
 In blender we can easily group and fix the components and
 It allows us to stitch the surfaces together so there would be no data loss while
exporting into unity
 It allows to remesh and decrease the model size so that it can maintain low ploy count
for decreasing Final App size

07/27/2021 School of Mechanical Engineering, VIT Chennai

 MEE6099 – Master’s Thesis
First Review Presentation

Unity for development

 In AR development Unity is used for  In VR Development Unity is used for
1. gamification of 3D objects, 1. gamification of 3D objects,
2. rendering,(light baking) 2. rendering,(light baking)
3. animating and scripting 3. animating and scripting
4. UI elements (Interactions) 4. UI elements (Interactions)

07/27/2021 School of Mechanical Engineering, VIT Chennai

 MEE6099 – Master’s Thesis
First Review Presentation

Training Structural Model

 L – logical structure (rules, algorithms, methods,

approaches, directives etc.) – the essence of the system
 P – physical structure (Paper based instructions, on-job
supervision - the environment for (L) deployment)

Training Architecture (A)

Physical Structure (P)

07/27/2021 School of Mechanical Engineering, VIT Chennai

 MEE6099 – Master’s Thesis
First Review Presentation

Implementation
AR Development
Marker based AR object placement Markerless AR object placement
 It uses a specified marker (QR Code) to place  It uses Camera to detect the plane to place
objects objects
 Objects can be moved with Marker  Allows user to move around the object
 Interaction for complex assemblies is difficult  Interaction for Complex assemblies is easy
 Internal details of the Assemblies is difficult  User can easily understand the assemblies as it
unless predefined allows to go to point of assembly

Scan the Wait for Render 3D

Video Feed
region to user tap to objects in
from
detect Plan place the Video
Camera
surfaces 3D object Frame

Algorithm used in this work

(Image Source: Internet)

07/27/2021 School of Mechanical Engineering, VIT Chennai

 MEE6099 – Master’s Thesis
First Review Presentation

VR Development – (GoogleVR SDK)

 It is collection of useful scripts aid building VR solutions rapidly and easily
in Unity3d
 It covers a number of common solutions such as:
 Motion within virtual space.
 Interacting with Unity3d UI elements through pointers or touch.
 Body physics within virtual space.
 2D and 3D controls like buttons, levers, doors, drawers, etc.

Camera Position to align Head movement to

Rendered Room with 3D Head movement to
with user with Head- Interact with UI
Assembly move person
Mount Device elements(Buttons)

Concept in VR Development

07/27/2021 School of Mechanical Engineering, VIT Chennai

 MEE6099 – Master’s Thesis
First Review Presentation

Chosen input devices for VR/AR system's

development

Active, A Passive
Position tracking, B Tracking body parts, C
Buttons Optical Head
Switches Video-metric Hand
Fingers

Chosen output devices for VR/AR system's

development
Visual, D Auditory, E Reference , F Vestibular, G
Mobile Screens Headphones Ground referenced Motion base systems
VR head mounted displays Externals speakers Body referenced Motion platforms

07/27/2021 School of Mechanical Engineering, VIT Chennai

 MEE6099 – Master’s Thesis
First Review Presentation

Project Demonstration – Video File

Digital work Digital work
instructions in AR instructions in VR

07/27/2021 School of Mechanical Engineering, VIT Chennai

 MEE6099 – Master’s Thesis
First Review Presentation

Evaluation
Test – 1: AR Mode Test – 2: VR Mode
• To provide general • To provide depth perspective
understanding of procedure understanding
• A group of 20 subjects were • A group of 4 subjects were
taken part taken part
• Each one to perform task 3 • Each one to perform task 3
times times
• And Answer brief • And Answer brief
questionnaire questionnaire

07/27/2021 School of Mechanical Engineering, VIT Chennai

 MEE6099 – Master’s Thesis
First Review Presentation

Questionnaire and Observation

• An open questionnaire to gather impressions and thoughts of the
participants in both AR and VR modes.
• The following questions (SUS) were given to participants:
1. Do you have any previous experience with Augmented Reality and Virtual Reality?
2. Do you have any previous experience with motion sensors using head gestures?
3. Is it intuitive?
4. Does the gesture feel responsive enough?
5. Was the depth perceivable?
6. Was the button touch-sensitive in AR?
7. Does it feel frustrating or tiring?
8. Do you think it needs practice?
9. Did you felt very confident using the system.
10. Do you needed to learn a lot of things before you could get going with this
system.

07/27/2021 School of Mechanical Engineering, VIT Chennai

 MEE6099 – Master’s Thesis
First Review Presentation

SUS is a scoring method consists on

• take the odd questions, i.e. 1,3,5,7,9 and subtract one(-1)
from the chosen point in the scale, while in
• even questions, i.e. 2,4,6,8,10, we subtract the selected
answer from five(5), sum-up the ten results obtaining a
score in a scale of [0-40].
• Finally, to convert the range to [0-100] just multiply the
result by 2.5

07/27/2021 School of Mechanical Engineering, VIT Chennai

 MEE6099 – Master’s Thesis
First Review Presentation

Results and Discussion

Case Study was taken and AR and VR apps were developed
for same assembly process
– Interactive assembly Procedure Instructure {AR Smart-Phone Based app}
– Modifications on AR app based on Feedback as Prototype Evaluation
and Improvement
– Development of VR app for same Case Study

School of Mechanical Engineering, VIT

07/27/2021 Chennai
 MEE6099 – Master’s Thesis
First Review Presentation

Test – 1: AR Mode Test – 2: VR Mode

• Has two scenes A and B • Has only one Scene but has
• Scene A: Placing the Model gesture interactions
• Scene B: Interacting with Model • Physical distance travelled vs
Time taken to complete

Mean’s comparison of attempts from

scene A and B Time-Distance per User

07/27/2021 School of Mechanical Engineering, VIT Chennai

 MEE6099 – Master’s Thesis
First Review Presentation

Summary of SUS

AR Perception:
1. Depth Perception
from field of view of
users camera
2. Movements with
reference to button
touch
VR Perception:
3. Motions perception
from field of view of
camera
4. Gesture based
movements and
interactions
SUS results Table

07/27/2021 School of Mechanical Engineering, VIT Chennai

 MEE6099 – Master’s Thesis
First Review Presentation

Table User’s Previous

Experience

SUS Average Result per

Question

07/27/2021 School of Mechanical Engineering, VIT Chennai

 MEE6099 – Master’s Thesis
First Review Presentation

Discussion
• From results its is evident that the significance difference in time was
in VR mode and it can be improved by user’s interactions within the
virtual space
• One problem that affected the performance initially was the depth
perception of the users, where the point of view of the camera did not
help the user to perceive the exact location of the objects in the z-
axis(depth) and y-axis(height).
• On SUS results, the agreement with the learning effect is mutual
among the users that stated to improve at some level on each
attempt.
• On VR mode, there was occlusion not considered but little awareness
should be taken into consideration

School of Mechanical Engineering, VIT

07/27/2021 Chennai
 MEE6099 – Master’s Thesis
First Review Presentation

Future Scope

• Haptic hand interaction for VR

• Hand gestures for AR using Mano-motion SDK(Paid)
• WebAR and OpenGL usage for supporting all devices
without ARCore support

07/27/2021 School of Mechanical Engineering, VIT Chennai

 MEE6099 – Master’s Thesis
First Review Presentation

Difficulties in Implementation

• Since some SDK’s are paid (Manomotion and Vuforia)

and AR App is developed to suit only ARCore supported
Devices
• VR does not support direct player movement without any
high-end tech like Oculus, or HTC

07/27/2021 School of Mechanical Engineering, VIT Chennai

 MEE6099 – Master’s Thesis
First Review Presentation

References
1. SepehrAlizadehsalehi, Ahmad Hadavi ,Joseph Chuenhuei Huang “From BIM to extended reality in AEC
industry” Automation in Construction Volume 116, August 2020, 103254
2. Marvin Czarskia et al “A Mixed Reality application for studying the improvement of HV AC systems in learning
factories” 10th Conference on Leaming Factories, CLF2020
3. Rahul Prasanna Kumar , Egidijus Pelanis, Robin Bugge, Henrik Brun, Rafael Palomar, Davit L. Aghayan,
Åsmund Avdem Fretland, Bjørn Edwin, Ole Jakob Elle “Use of mixed reality for surgery planning: assessment
and development workflow” Journal of Biomedical Informatics: X
4. Simon Kind, Andreas Geiger, Nora Kießling, Michael Schmitz, Rainer Stark “Haptic Interaction in Virtual
Reality Environments for Manual Assembly Validation” Procedia CIRP 91 (2020) 802–807
5. Mafkereseb Kassahun Bekele “Walkable Mixed Reality Map as interaction interface for Virtual Heritage” Digital
Applications in Archaeology and Cultural Heritage
6. Nick Salamon, Jonathan M. Grimm, John M. Horack, Elizabeth K. Newton “Application of virtual reality for
crew mental health in extended-duration space missions” Acta Astronautica
7. Kunnen, Steffen; Adamenko, Dmytro; Pluhnau, Robin; Loibl, André; Nagarajah, Arun; “System-based
concept for a mixed reality supported maintenance phase of an industrial plant” CIRP Design 2020
8. Åsa Fast-Berglund, Liang Gong, Dan Li “Testing and validating Extended Reality (xR) technologies in
manufacturing” Procedia Manufacturing, 2018
9. Huang, T.-L. and Hsu Liu, F. "Formation of augmented-reality interactive technology's persuasive effects from the
perspective of experiential value", Internet Research, (2014)
10. I P. Tussyadiah Dan Wang Timothy H.Jung M.Claudiatom Dieck’ “Virtual reality, presence, and attitude
change: Empirical evidence from tourism”, Tourism Management, June 2018

School of Mechanical Engineering, VIT

07/27/2021 Chennai
 MEE6099 – Master’s Thesis
First Review Presentation

Any Questions

07/27/2021 School of Mechanical Engineering, VIT Chennai