White Paper VRAR
White Paper VRAR
White Paper
Executive summary
Education and training is evolving at an accelerated rate along with an increased level of integration
with ICT systems. Growth in this area will only continue. Education and training integrated with ICT
systems is progressing in two ways: one is in the use of virtual digital information without the use of
devices, and the other is in the use of both virtual digital information and sensor devices. Both types of
education and training will expand based on the advance of ICT including virtual reality (VR) and
augmented reality (AR). Note that mixed reality (MR) is a mix of real and virtual reality and is therefore
presumed to be included in the continuum of VR and AR in this paper.
Virtual education and training systems are typical applications for systems integration. They are based
on the technologies of VR and AR with sensors. Although education and training areas are diverse,
virtual education and training systems can have unique information modeling, as well as common
functionalities. Common teaching and learning technologies for virtual education and training systems
can be determined by categorizing use cases for education and training based on VR and AR.
Education and training has been integrated with digital information systems and the development of
education and training information systems is increasing. It is presumed that application-dependent
methods are being used since the areas of education are vast and simulating education and training
with a unified procedure may not be feasible. However, from the point of view of ICT integration and
information modeling, virtual education and training systems can be developed based on a systematic
design approach with standards. In this paper, a systematic standards-based design approach that can
be used as a guideline for developing virtual education and training systems in various areas is
provided.
Acknowledgements
This white paper has been prepared by the JTC 1 VR AR for education ad hoc group, whose members
include:
Myeong Won Lee (U. of Suwon), JTC 1 SC 24, Convenor of the JTC 1 VR AR for education ad hoc group
Table of Contents
List of abbreviations
3D 3-Dimensional
AR Augmented Reality
IT Information Technology
MR Mixed Reality
VR Virtual Reality
X3D Extensible 3D
1. Introduction
Virtual education and training systems are typical applications for systems integration. These systems
use virtual reality (VR) and augmented reality (AR) with sensors for education and training. Although
education and training areas are diverse, virtual education and training systems can have unique
information modelling requirements, as well as common functionalities. Common teaching and learning
technologies for virtual education and training systems can be determined by categorizing use cases for
education and training based on VR and AR.
VR and AR based virtual education and training systems have requirements related to learning and
teaching technology, representation, exchange, data, and device, in the following ways. First, existing
learning and teaching technology should be extended and integrated with the use of VR and AR
technology. Based on changes in learning and teaching methods when using VR and AR, requirements
for interaction and simulation should be added. Second, 3D visual and interactive representation should
be provided to enhance the effectiveness of education and training. 3D simulation is an important
aspect in representing educational information and in understanding education and training content.
Third, information for education and training should be exchangeable over heterogeneous computing
environments so that users can access education and training information anytime, anywhere. Fourth,
information data for education and training should be organized and transferred securely and without
delay. Fifth, interfaces for interacting with devices and sensors should be included. Safety should also be
a consideration when using larger devices, such as in industrial education.
This paper provides guidelines for developing VR and AR based education and training systems to meet
the requirements listed above. The guidelines include concepts, information modelling architecture,
standards based functional components, and implementation components for virtual education and
training.
2. Scope
This white paper describes guidelines for developing virtual education and training systems based on
VR and AR technology. It is intended to illustrate information modeling with standards that can be used
for virtual education and training systems. It provides procedures or methods for developing 3D virtual
education and training systems based on ISO/IEC JTC 1 standards. It also provides a systematic
approach to developing applications for systems integration areas. Virtual education and training
systems based on VR and AR are typical examples of systems integration applications.
This white paper includes several topics that relate to virtual education and training systems. First,
concepts of VR and AR based education and training are defined. Second, it describes software and
application development technology necessary for virtual education and training systems. Third,
standards for VR and AR information generation, transfer, and exchange when developing the systems
are described. The information model and architecture of virtual education and training systems are
specified as basic standards concepts. Fourth, components for organizing virtual education and training
systems are discussed based on the standards concepts. Fifth, standards based application development
technology and software interfaces between computers and devices are described. Finally,
implementation components for developing VR and AR based education and training systems are
included.
This white paper excludes device hardware technology for virtual education and training systems.
3. Concepts
Virtual education and training is a representative area for the application of systems integration
technology. It is based on MAR concepts[1] which include VR and AR. It includes computer simulated
education and training processes, and provides for repetitive and empirical learning in immersive
virtual environments.
Use cases for virtual education and training systems can be classified as two types, each defined slightly
differently. The first is virtual education, which facilitates education via digital information and devices
based on ICT including VR. Virtual education systems include online and offline learning and teaching
(Figure 3.1, 3.2). Online virtual learning and teaching are wholly accomplished using computing devices
in virtual environments, while offline virtual learning and teaching partially uses digital information
and virtual environments. The latter case corresponds to an environment where both students and
teacher are present in a classroom and where digital information with virtual environments is
sometimes used.
The second type refers to virtual training systems that make use of real world sensor devices along with
virtual environments. It is based on MAR virtual environments with training devices, sensor
information, and interactions with sensors (Figures 3.3, 3.4).
Virtual education and training systems are computer hardware and software integrated systems for
virtual education and training in virtual environments. They can include implementation of education
and training simulation in 3D virtual environments. When simulating education and training
information, sensor data from education or training devices in the real world can be imported to and be
represented in a 3D virtual environment. All virtual education and training systems can have similar
system functions, so systems integration methods for virtual education and training can be
standardized. As a result, standardized systems integration guidelines can be drawn on and used when
developing various virtual education and training applications.
Figure 3.5 shows the phases of systems integration necessary when developing virtual education and
training systems in various areas. The first phase represents a systems integration process including VR,
AR, and 3D simulation based on representing information for virtual education and training. In the
second phase, real world sensor information can be included and simulated. The third phase represents
the necessity of standardized systems integration so that common processes for integration can be
applied to all virtual education and training areas.
For the first phase, while virtual education and training systems is one typical application area amongst
many, these applications are, in fact, MAR applications. Figure 3.6 shows the whole range of mixed
reality, including augmented reality, and the relation between virtual education and training systems
and mixed and augmented reality. It shows that virtual education and training systems are the result of
implementing 3D simulation with sensors in MAR environments.
For the second phase, sensor based virtual education and training systems integrate physical sensor
devices (or just physical sensors) and their functionalities in 3D virtual environments. A physical sensor
device means a physical device that includes sensors such as a simulator. The appearance and physical
properties of a physical sensor can be represented in a 3D virtual world. In addition, the physical
properties and events for education and training can be controlled and managed in the virtual world.
In order to provide the capability to represent and simulate physical sensors in education and training
systems, the systems require the following: representation of physical sensors, visual and functional
properties of each physical sensor, physical properties of each physical sensor, control of a physical
sensor’s data stream, and an interface for controlling physical sensors in a 3D scene.
Figure 3.5 Virtual education and training systems, and systems integration
Figure 3.6 Mixed reality, and virtual education and training systems
Figure 3.7 shows a physical sensor representation model that can be used for implementing virtual
education and training systems in 3D virtual worlds. It consists of a sensor simulated MAR world, a
spatial mapper, and an event mapper. A simulated sensor MAR world organizes MAR content with
scene composition. ISO/IEC 18038 DIS describes details of the model[2].
The sensor representation model can be used as a basis for implementing virtual education and training
systems. The middle of the figure shows 3D scene information modeling for education and training,
while at the left is a spatial mapper that recognizes location and orientation. At the bottom is an event
mapper that sends sensor information to the 3D scene. Finally, at the right is the rendering and display
of the 3D scene.
The framework for virtual education and training systems consists of various components (Figure 3.8).
In the figure, the upper half represents various applications of virtual education and training, while the
lower half shows virtual training system components. The middle part in the lower half represents the
required technologies for the systems. Education and training content can be input to the systems via
the combination of a virtual simulation platform and an experience and knowledge database. Simulator
software should be provided for education and training. This includes a simulator control management
tool, operations management, and evaluation.
When developing virtual education and training systems based on the framework, the following
technologies, in addition to VR and AR, are needed:
Information modeling
Sensor representation
Content creation and manipulation should include definition, storage, retrieval, transfer, and interaction
with teaching and learning knowledge. Since learning and teaching technology can be combined with
knowledge databases, guidelines on how to create and manipulate knowledge databases of education
and training content should be provided. In addition, user interfaces for learning and teaching should be
standardized. Information modeling using VR and AR is necessary to be defined and implemented for
virtual education and training systems so that content in knowledge databases can be simulated in
virtual environments. How to define and represent education and training information in relation to VR
and AR should be analyzed. Visualization and simulation should include 3D representation, and
simulation methods or procedures for education and training. Sensor representation should include
real world sensor information used for education and training. Real world representation should
include real world physical objects represented in virtual environments. Graphical user interaction
should include types and usage of interfaces with virtual environments during education and training.
In Figure 4.2, the horizontal layers correspond to visual technology, visual information modeling, visual
information exchange, and visual information interaction from the technical point of view for ISO/IEC
JTC 1/SC 24 and SC29. For SC36, the layers correspond to education technology, education information
modeling, education information exchange, and education information interaction. When developing
systems integration applications, real world information, including various types of sensors, should be
included. In this case, another layer should be added to the horizontal layers. In Figure 4.3, the top most
layer represents information processing in relation to real word sensors and the facilities necessary for
systems integration applications. In order to process real world information at this layer, VR, AR, and
MR technologies should be used. The two rightmost columns represent these technologies. Amongst the
many applications in systems integration areas, virtual education and training systems are typical
examples. These reside in the systems integration area.
Figure 4.2 Architecture for integrating ICT visual information and education
Figure 4.3 Virtual education and training systems and systems integration
Regarding the development of virtual education and training systems, Figure 4.4 shows the role of each
working group in JTC 1/SC 24, Figure 4.5 shows the role of each working group in JTC 1/SC 29, and
Figure 4.6 shows the role of each working group in JTC 1/SC 36.
Figure 4.4 SC24 working and study groups for virtual education and training systems
Figure 4.5 SC29 working groups for virtual education and training systems
Figure 4.6 SC36 working groups for virtual education and training systems
First, virtual environments should be represented with a standardized data model and interface so that
they can be generated for common use and exchanged between applications. VR and AR scenes should
be created and managed for virtual education and training. Education and training information can be
represented in the virtual environments. Second, an interface should be defined for simulating
education and training information in virtual environments. The interface controls the virtual
environment so as to represent the change in the scene. Third, real world information from sensors
should be represented in virtual environments. In order to represent sensor information, it should be
able to be imported, represented and simulated according to learning and teaching procedures. ISO/IEC
JTC 1/SC 24 standards can be used for providing the above three functionalities. VR and AR information
for education and training should be able to be transmitted across heterogeneous computing
environments in a seamless manner. The information should be able to be exchanged and interacted
with by learners and teachers. ISO/IEC JTC 1/SC 29 standards can be used for transmission of VR AR
data. Lastly, knowledge databases for education and training should be able to be created and
manipulated in accordance with learning, education and teaching methods in virtual environments.
ISO/IEC JTC 1/SC 36 standards provide education information processing methods that can be
integrated into virtual education and training systems. Table 5.1 provides the classification of standards
necessary for developing the systems and the related functions and functional types of VR and AR.
Table 5.1 Classification of standards necessary for virtual education and training systems
Virtual environments for education and training should provide a standardized virtual 3D space that
can define and exchange education and training objects in heterogeneous computing environments. The
following ISO/IEC JTC 1/SC 24 standards can be used for representing and generating virtual
environments in which learning and training are displayed as 3D scenes:
VRML and X3D are standards used for generating and exchanging 3D scenes in various computing
environments, including the Internet[3]. They are implemented as 3D file formats that include all
graphical information necessary for representing and interacting with a scene in a virtual environment.
VRML describes fundamental 3D graphics scenes and interaction functions[4]. Figure 6.1 shows the
architecture of VRML systems when implementing VR applications. X3D is a 3D graphics file format
using XML and has many functions that are enhancements over VRML[5]. X3D establishes a world
coordinate space for all objects, and composes a set of 2D, 3D, and multimedia objects. Figure 6.2 shows
the architecture of X3D systems when generating VR worlds. X3D VRML and X3D can be used for
developing Web-based or general 3D applications with virtual environments.
The following standards provide 3D environmental data representation that can be used for generating
and defining a semantic real world in a virtual environment. All semantic information for representing
the real world can be defined using the standards. An environmental data dictionary can be provided in
relation to geographical information in virtual environments. Figure 6.3 shows the concepts of SEDRIS
(Synthetic Environment Data Representation and Interchange Specification).
SEDRIS is a standard for representing environmental data and for the interchange of environmental
data sets[6]. It offers a data representation model, augmented with its environmental data coding
specification (EDCS)[7] and spatial reference model (SRM)[8], so that one set of environmental data can
be articulated clearly, while also using the same representation model to understand another set of data
unambiguously. Therefore, the data representation aspect of SEDRIS is about capturing and
communicating meaning and semantics.
Figure 6.3 Synthetic Environment Data Representation and Interchange Specification (SEDRIS)
H-Anim specifies the structure and manipulation of H-Anim figures[9]. H-Anim figures are articulated
3D representations that depict animated characters. While H-Anim figures are intended to represent
human-like characters, they are based on a general concept that is not limited to the same number of
limbs, head, and other body parts that are typical of human beings. A single H-Anim figure is called a
humanoid. H-Anim Part 1 is an enhanced version of ISO/IEC 19774 which specifies the architecture of
humanoids based on medical terms[10]. H-Anim Part 2 specifies basic concepts of humanoid animation
and an overall procedure for defining humanoid animation using an H-Anim model and motion data[11].
Motion parameters necessary for generating humanoid animation are defined. Humanoid animation
data is organized using these parameters and the geometric data of a humanoid model. Figure 6.4
represents the joints and segments of a human figure with LOA (Level of Articulation) 3.
Figure 6.5 shows the procedure of generating H-Anim based humanoid animation. It specifies three
types of humanoid animation: keyframe animation, algorithm animation, and motion data animation.
These can be used for representing humanoid figures in virtual environments. Table 6.1 provides a
summary of standards that can be used for representing 3D virtual environments for virtual education
and training.
Functions necessary for Advantages for education and Standards that can be used
virtual environments training
3D virtual environments 3D environments modeling and VRML (ISO/IEC 14772)
representation rendering X3D (ISO/IEC 19775-1)
3D environments animation and
simulation
3D virtual object representation
3D data exchange format 3D modeling and animation data VRML (ISO/IEC 14772)
format X3D (ISO/IEC 19775-1)
Environment data 3D environment data SEDRIS DRM (ISO/IEC 18023-1)
representation representation (space,
atmosphere, terrain, and ocean)
Real world environment
simulation
Environment data coding Classification (naming, labelling, EDCS (ISO/IEC 18025)
specification identification) of environmental
objects
Semantic environment
representation
Spatial reference model for Description of geometrical SRM (ISO/IEC 18026)
environments properties such as position,
direction and distance
In order to educate and train in virtual environments, user interfaces for manipulating and interacting
with virtual environments should be provided. There are two levels of interfaces, categorized as
application programming interfaces and graphical user interfaces. The former provides developers with
interfaces so that they can generate various applications with standard data and functions, while the
latter is a graphical interactive facility that can be used to manipulate objects in virtual environments
directly. It can be used when visualizing, simulating, and managing virtual environments without the
need for programming. The former can be provided by the following standards.
X3D SAI provides a language-neutral representation of all actions that can be performed by an external
application across this interface[12]. Bindings to specific languages are defined in ISO/IEC 19777[13-
14]. The SAI forms a common interface that can be used for manipulating browsers and a scene graph
from either an external application or from inside the scene graph itself. X3D virtual environments for
education and training can be simulated and manipulated by using X3D SAI. X3D language bindings
provide a set of implementation-independent objects. These objects represent the possible interactions
with the X3D scene through the SAI. Access to the information stored in a SEDRIS transmittal is through
an API. The abstract specification for this API is defined in ISO/IEC 18023-1[6]. A part of ISO/IEC 18024
defines the binding of the abstract specification to a particular programming language[15].
Environmental data coding specification and spatial reference model can also be manipulated with a
specific programming language binding[16-17].
The latter graphical user interfaces for end users, are simulation tools for education and training. All
simulation parameters should be able to be represented and controlled by the tools so that users have
direct control. Standard interfaces can be obtained by developing them using the SAI and language
bindings specifications.
In order to facilitate interactions between users and between systems in education and training,
effective transmittal formats should be supported for virtual environments. VRML and X3D have
functions to provide users with the capability to interact with other users and systems in common
virtual environments through networks and Web-based applications. In addition, several transmittal
formats can also be used, as follows:
XML is used for self-validating X3D encoding. This X3D encoding provides a Web-compatible format
that maximizes interoperability with other Web languages[18]. X3D can also be encoded by classic
VRML. The syntax of X3D in terms of VRML encoding is specified in ISO/IEC 19776-2[19]. The
semantics of X3D is defined in ISO/IEC 19775-1. X3D compressed binary format is specified in ISO/IEC
19776-3[20]. This X3D encoding provides a compact transmission format that minimizes delivery size
and maximizes parsing speed while following the precepts of XML. SEDRIS abstract transmittal format
provides a platform independent interchange mechanism for SEDRIS data for virtual environments[21].
The abstract transmittal format is a conceptual file format that defines the organization of persistent
SEDRIS data. A transmittal encoded using the techniques defined in this part of ISO/IEC 18023 consists
of a set of files that collectively are termed an STF-encoded transmittal. STF is a file format that defines
the syntax of binary-encoded SEDRIS data[22].
Table 6.2 provides a summary of standards that can be used for accessing, manipulating, and interacting
with 3D virtual scenes.
Table 6.2 Standards for manipulating and interacting with 3D virtual objects and environments
Functions necessary for Advantages for education and Standards that can be used
simulation and interaction training
with virtual environments
3D virtual environments 3D virtual scene access and X3D Scene Authoring Interface
Virtual simulation with real world information including sensors should be able to be represented in
virtual environments. In addition, standardized interfaces with real word information should be
provided. In order to represent and manage the functions of sensors in virtual environments, sensor
information should be processed as functional items necessary for virtual education and training
systems. Because there are many types of sensors, and because they have different data, an abstract
data format that can be used for the sensors should be provided. Sensor information should be able to
be simulated visually and interactively in virtual environments. Interfaces for managing and controlling
sensors for learning and teaching should be provided. Some functions can be obtained by using the MAR
work items in progress. With this work on MAR representation sensor information can be simulated in
virtual environments.
Standardized MAR representation can be obtained with the following standards developed in JTC 1/SC
24:
ISO/IEC 18040 Live Actor and Entity Representation in Mixed and Augmented Reality
The MAR reference model has an architecture with the following functions: processing the content as
specified and expressed in the MAR scene, including additional media content provided in media assets;
processing the user input(s); processing the context provided by the sensors capturing real world
information; managing the presentation of the final result (aural, visual, haptic, and commands to
additional actuators); and managing communication with additional services (Figure 6.6)[23]. Sensor
Representation in MAR specifies how to represent physical sensors in 3D virtual environments (Figure
6.7). A scene graph and a system architecture with sensors are defined for organizing virtual
environments with sensors[24]. Interaction and interfaces to real and virtual worlds for sensors are
described. Live Actor and Entity Representation in MAR specifies system functions for representing real
actor video images in a MAR scene[25]. A live actor in a MAR scene can be captured from the physical
world, then represented in a 3D virtual world, and can interact with cameras, objects, and AR content in
the 3D virtual world according to an input of sensing information (Figure 6.8).
Table 6.3 provides a summary of standards that can be used for representing real world information in
virtual environments.
Table 6.3 Standards for representing and simulating real world information in MAR
Functions necessary for Advantages for education and Standards that can be used
real world information training
MAR object representation A reference model for MAR reference model (ISO/IEC
representing real and virtual 18039)
objects
Real world information
representation
Sensor information Sensor representation in 3D Sensor representation in MAR
representation in virtual virtual environments (ISO/IEC 18038)
environments Sensor simulation in 3D virtual
environments
Representation of human Representation of human video Live actor and entity representation
figure motion in virtual information in 3D virtual in MAR (ISO/IEC 18040)
environments environments
In order to achieve adequate performance for virtual education and training systems, all information
generated during education and training should be able to be transmitted across heterogeneous
computing environments and systems using standardized methods and compressed media formats. The
following standards can be used when transmitting and exchanging education and training information:
ISO/IEC 10918 Digital compression and coding of continuous-tone still images (JPEG)
ISO/IEC 15444-1 JPEG 2000 image coding system: Core coding system
ISO/IEC 19566-1 JPEG Systems – Part 1: Packaging of information using codestreams and file
formats
ISO/IEC CD 21794-1 Plenoptic image coding system (JPEG PLENO) – Part 1 Framework
ISO/IEC CD 21794-2 Plenoptic image coding systems (JPEG PLENO) – Part 2 Lightfield
JPEG specifies an image coding technology and incorporates many options for encoding photographic
images. It provides lossy compression for digital images[26]. JPEG 2000 is an image coding system that
uses state-of-the-art compression techniques based on wavelength technology and offers an extremely
high level of scalability and accessibility[27]. JPEG 2000 can be used with portable digital cameras all
the way to advanced pre-press, medical imaging, geospatial and other applications. JPEG systems define
an overall framework for the system layer structure of the JPEG standards to ensure interoperability
and functionality[28]. JPEG systems further intend to specify system layer extensions that support
interactivity protocols, High Dynamic Range (HDR) tools, privacy and security, metadata, 360 degree
images, augmented reality, and 3D. JPEG 360 defines a method to represent 360 degree images and to
add supplementary metadata and images[29]. JPEG Pleno aims to provide a standard framework for
representing new imaging modalities such as texture-plus-depth, light field, point cloud, and
holographic imaging. Such imaging should be understood as light representations inspired by the
plenoptic function, regardless of which model captured or created all or part of the content[30]. JPEG
Pleno standard tools will be designed together to consider their synergies and dependencies for the
whole to be effectively greater than the sum of its parts. To fully exploit this holistic approach, JPEG
Pleno is not just a set of efficient coding tools addressing compression efficiency. It is a representation
framework understood as a fully integrated system for providing advanced functionality support for
image manipulation, metadata, random access and interaction, and various file formats. In addition, it
should offer privacy protection, ownership rights, and security.
When transmitting multimedia files that contain integrated audio, video, and graphics data that can be
used for VR and AR applications, latency should be considered. The following standards can be used for
efficient transmission.
ISO/IEC 14496-11 Scene Description and Application Engine (BIFS: BInary Format for Scenes)
BIFS is a format that can be used for representing and transmitting multimedia scenes and their
application engine[31]. ISO Base Media File Format specifies a file format that can be used for
transmitting multimedia data[32]. AFX is a set of compression formats for 3D graphics objects[33].
Lightweight Application Scene Representation specifies an XML-based scene description and
interaction format[34]. 3D Graphics Compression Model specifies a model for compressing 3D graphics
data[35].
MAR information in a VR and AR based education and training system should be able to be transmitted
and exchanged through networks in a seamless way despite the large and composite data transfer. The
following standards can be used to transmit the MAR information.
As the performance of various display and capturing devices advances, immersive media is enabled
through improvements in multimedia data processing and transmission capabilities. Immersive media
standards have been developed focusing on data format, metadata, and data container, as follows:
Immersive Media Architectures specifies architectures and nomenclature for immersive media. They
define the elements of an immersive experience in the production and in its consumption. The
Technical Report will document minimum quality requirements as well as quality objectives for an
uncompressed immersive media experience as a whole[41]. Omnidirectional Media Format (Figure
6.10) specifies an omnidirectional media format for coding, delivery, and rendering of omnidirectional
media including video, images, audio, and timed text[42]. The user’s viewing perspective is from the
centre of a sphere looking towards the inside surface of the sphere. Immersive video specifies coded
representation of versatile video[43]. Immersive Audio specifies coded representation of immersive
audio[44]. Point Cloud Coding specifies coded representation of point cloud data[45]. Immersive Media
Metrics specifies a metric for immersive media[46]. Immersive Media Metadata specifies metadata for
immersive media[47]. Geometry-based Point Cloud Compression specifies lossless and lossy coding of
time-varying 3D point clouds with associated attributes such as color and material properties. This
technology is appropriate especially for sparse point clouds[48]. Carriage of Point Cloud Data is the file
format for coded representation of point cloud data[49].
Table 6.4 shows a summary of SC 29 standards that can be used for transmitting virtual education and
training information.
Functions necessary for Advantages for education and Standards that can be used
transmission training
Image and video data Fast transmission JPEG (ISO/IEC 15444)
compression Realtime interaction MPEG (ISO/IEC 23001)
Resource saving
Multimedia file format for Compressed file format MPEG-A (ISO/IEC 23000-9)
transmission MPEG-4 (ISO/IEC 14496)
Video and virtual world Media information adaptation MPEG-V (ISO/IEC 23005)
integration and control between video
and virtual worlds
360 video transmission Highly compressed HEVC (ISO/IEC 23008-8)
transmission of 360 VR data
Supports all commonly used
progressive scan picture
formats with arbitrary size
3D audio Automatically adapt audio 3D Audio (ISO/IEC 23008-3)
program material to the target
number of loudspeakers
Coded representation of A compressed data container MPEG-I (ISO/IEC 23090)
immersive media for AR and VR data
Supports compressed and
uncompressed cinematic open
source formats
Learning, education, and training (LET) methods and procedures should be able to be simulated in
virtual environments. It is necessary that LET information is defined in a standardized format when
organizing knowledge databases. LET should also provide users with interfaces that can manage and
control LET interaction in virtual environments.
Figure 6.11 illustrates the LET components necessary for VR based education and training. VR files can
be generated within an architecture that provides users with education and training capabilities in
virtual environments. A school’s VR LET environment can be obtained from learning and teaching
methods and procedures in virtual environments. Interfaces between the VR device, the VR engine, the
school’s VR LET environment, and VR controllers should be provided so that LET information can be
controlled and managed in a seamless way throughout all the components.
Information about LET methods, and knowledge bases, should be able to be uniquely defined and
accessed in heterogeneous computing environments. Interfaces to access the information should be
provided. VR based learning and teaching can be obtained and functionally achieved using the following
standards:
ISO/IEC 19788 specifies metadata elements and their attributes for the description of learning
resources[50]. It includes rules governing the identification of data elements and specification of their
attributes. ISO/IEC 20748 specifies a reference model that identifies the diverse IT system
requirements for learning analytics interoperability[51]. The reference model identifies relevant
terminology, user requirements, workflow, and a reference architecture for learning analytics.
ISO/IEC 20821 describes a learning environment profile to support the establishment of mobile
learning environments and defines a standard set of terms used to express device information and
learning environments for mobile learning[52]. It contains two methods. One is the profile expression
method that displays device information language and includes definitions of schema and vocabulary.
The other is the profile grouping method that displays terminal information language and includes a
group profile example.
ISO/IEC 24751 is intended to meet the needs of learners with disabilities and for teachers working with
these learners in an educational context[53]. It provides a common framework to describe and specify
learner needs and preferences on the one hand, and the corresponding description of the digital
learning resources on the other hand, so that individual learner preferences and needs can be matched
with the appropriate user interface tools and digital learning resources. ISO/IEC 18121 defines a
framework for IT standards and specifications for virtual experiments supporting IT-enhanced learning,
education, and training (ITLET)[54]. It is based on implementations of standards and specifications that
are used to support virtual experiment, development, evaluation, and management that rely on ITLET.
It promotes the appropriate design and application of virtual experiments. It also provides a sample
architecture of a virtual experiment system framework that can be used to support further
development of virtual experiments. ISO/IEC 19778 specifies a table-based approach for defining data
models[55]. It is used for specifying the collaborative workplace data model for describing data
elements and their relationships. It provides for the portability and reuse of data in integrated form and
allows data model instantiations to be interchanged, stored, retrieved, reused, or analyzed by a variety
of systems.
ISO/IEC 23988 Information technology -- A code of practice for the use of information
technology (IT) in the delivery of assessments
ISO/IEC 23988 gives recommendations on the use of information technology to deliver assessments to
candidates and to record and score their responses[56]. Its scope is defined in terms of three
dimensions: the type of assessment to which it applies, the stage of the assessment “lifecycle” to which
it applies, and the IT aspect on which it is focused.
Table 6.5 provides a summary of SC 36 standards that can be used for defining, manipulating and
managing virtual education and training information.
Table 6.5 Functions and standards for managing virtual education and training information
Functions necessary for Advantages for education and Standards that can be used
education information training
description
Methods of managing Provision for describing methods Metadata for learning resources
learning resources for learning resources and data (ISO/IEC 19788)
elements
Management of learning
resource databases
A reference model for ICT Reference to develop an ICT Learning analytics
education information system for learning analytics interoperability (ISO/IEC 20748)
systems Provision of methods for
managing human resources
A user interface tool for Provision of a common Individualized adaptability and
digital learning resources framework for learner needs and accessibility in learning, education
preferences and training (ISO/IEC 24751)
Provision of a tool for learners
with disabilities
Virtual experiments Provision of a framework for Virtual experiment framework
virtual experiments based on (ISO/IEC 18121)
ITLET
Collaborative workplace Provision of the definition of data Collaborative workplace (ISO/IEC
data model models for collaborative 19778)
workplaces
Assessment for validity Provision of security and fairness A code of practice for the use of IT
and reliability of IT-delivered assessments in the delivery of assessments
(ISO/IEC 23988)
The left column represents information simulation processing for education. Education information
flows from the top left and is simulated into the scene graph for virtual environments for education and
training. Education and training information should first be described with data structures. The data
structures include VR and AR objects and simulation objects for education information. The VR and AR
objects are represented in the scene, and the simulation objects are commands to simulate the virtual
objects for the purpose of education and training. Standards related to the simulation objects and their
interfaces with the VR and AR objects are necessary.
The middle column represents VR and AR scene information flow. Visual and sensor objects are
processed and sent to the scene graph. A 3D scene graph should integrate with the simulation objects
and real world information such as device sensor information. It should be maintained so as to manage
and control the simulation objects, VR and AR objects, and sensor objects that are imported from the
real world. The VR and AR objects are controlled by the simulation objects and the sensor objects and
are represented in the scene graph.
The right column represents external device and application interfaces, allowing any sensor
information to be imported to the scene. The sensor information imported from the real world using
external applications can be integrated with the VR and AR objects and the simulation objects. The
sensor information can change the scene graph when communicating with the VR and AR objects and
the simulation objects. Interfaces should be provided between the three types of objects for
communication.
The MAR scene graph manager should control education information, 3D visual data with VR and AR,
and real world sensor information. Based on the scene graph, education information is simulated with
real world information in a 3D virtual environment. The three types of data are managed through the
integrated scene graph.
Figure 7.1 Functional components for virtual education and training systems
An information modeling architecture for virtual education and training is defined. It is based
on general technologies of ICT integrated systems development. It combines ICT information
and real world information in order to implement virtual education and training using VR and
AR. Real world information includes sensor information imported from devices.
Standards based functional system components for virtual education and training using VR and
AR were introduced. The components necessary for developing virtual education and training
systems are classified by their functionalities and the related standards. The components
consist of 3D virtual environments representation, 3D virtual information simulation, 3D real
System implementation components for virtual education and training using VR and AR were
introduced. These organize a system framework for implementing virtual education and
training systems. Education and training information is incorporated into and simulated within
a virtual environment through a MAR scene graph.
The above systematic approach can provide a standards-based method for developing virtual education
and training systems. It can be enhanced with details of the system components and used for
developing standards related to VR AR based systems integration. Recommended future work is as
follows:
To enhance industry usage, it is necessary to provide examples of systems design using the
systematic approach provided by the guidelines.
The guidelines for developing virtual education and training systems can be applied and
expanded in standards based ICT integrated systems development using VR and AR in other
areas.
Standards for ICT integrated systems using VR and AR can be developed based on the
guidelines in this white paper. The following topics can be considered as new work items:
- A system integration methodology and a framework for virtual education and training
systems
Annex A
References
[1] ISO/IEC 18039:2019 Information technology – Computer Graphics, image processing and
environmental data representation -- Mixed and augmented reality (MAR) reference model
[2] ISO/IEC DIS 18038 Information technology -- Computer graphics, image processing and
environmental representation -- Sensor representation in mixed and augmented reality
[3] ISO/IEC 14772-1:1997 Information technology -- Computer graphics and image processing -- The
Virtual Reality Modeling Language -- Part 1: Functional specification and UTF-8 encoding
[4] ISO/IEC 14772-2:2004 Information technology -- Computer graphics and image processing -- The
Virtual Reality Language (VRML) -- Part 2: External authoring interface (EAI)
[5] ISO/IEC 19775-1:2013 Information technology -- Computer graphics, image processing and
environmental data representation -- Extensible 3D (X3D) -- Part 1: Architecture and base
components
[7] ISO/IEC 18025:2014 Information technology -- Environmental Data Coding Specification (EDCS)
[9] ISO/IEC 19774:2006 Information technology -- Computer graphics and image processing --
Humanoid Animation (H-Anim)
[10] ISO/IEC 19774-1 Information technology -- Computer graphics and image processing -- Humanoid
Animation (H-Anim) -- Part 1: Architecture
[11] ISO/IEC 19774-2 Information technology -- Computer graphics and image processing -- Humanoid
Animation (H-Anim) -- Part 2: Motion data animation
[12] ISO/IEC 19775-2 Information technology -- Computer graphics and image processing and
environmental data representation -- Extensible 3D (X3D) -- Part 2: Scene access interface (SAI)
[13] ISO/IEC 19777-1:2006 Information technology -- Computer graphics and image processing --
Extensible 3D (X3D) language bindings -- Part 1: ECMAScript
[14] ISO/IEC 19777-2:2006 Information technology -- Computer graphics and image processing --
Extensible 3D (X3D) language bindings -- Part 2: Java
[16] ISO/IEC 18041-4:2016 Information technology -- Computer graphics, image processing and
environmental data representation -- Environmental Data Coding Specification (EDCS) language
bindings -- Part 4: C
[17] ISO/IEC 18042-4: 2006 Information technology -- Computer graphics, image processing and
environmental data representation -- Spatial Reference Model (SRM) language bindings -- Part 4: C
[19] ISO/IEC 19776-2:2015 Information technology -- Computer graphics, image processing and
environmental data representation -- Extensible 3D (X3D) encodings -- Part 2: Classic VRML
encoding
[20] ISO/IEC 19776-3:2015 Information technology -- Computer graphics, image processing and
environmental data representation -- Extensible 3D (X3D) encodings -- Part 3: Compressed binary
encoding
[21] ISO/IEC 18023-2:2006 Information technology -- SEDRIS -- Part 2: Abstract transmittal format
[22] ISO/IEC 18023-3:2006 Information technology -- SEDRIS -- Part 3: Transmittal format binary
encoding
[23] ISO/IEC 18039 Information technology -- Computer graphics, information processing and
environmental representation -- Mixed and augmented reality (MAR) reference model
[24] ISO/IEC FDIS 18039 Information technology -- Computer graphics, information processing and
environmental representation -- Sensor representation in Mixed and augmented reality
[25] ISO/IEC 18040 Information technology -- Computer graphics, information processing and
environmental representation -- Live actor and entity representation in mixed and augmented
reality (MAR)
[26] ISO/IEC 10918-1: 1994 Information technology -- Digital compression and coding of continuous-
tone still images: Requirements and guidelines
[27] ISO/IEC 15444-1:2016 Information technology -- JPEG 2000 image coding system: Core coding
system
[28] ISO/IEC TR 19566-1: 2016 Information technology -- JPEG Systems -- Part 1: Packaging of
information using codestreams and file formats
[29] ISO/IEC FDIS 19566-6 Information technology -- JPEG Systems -- Part 6: JPEG 360
[30] ISO/IEC CD 21794-1 Information technology -- Plenoptic image coding system (JPEG PLENO) --
Part 1: Framework
[31] ISO/IEC 14496-1:2010 Information technology -- Coding of audio-visual objects -- Part 1: Systems
[32] ISO/IEC 14496-12:2015 Information technology -- Coding of audio-visual objects -- Part 12: ISO
base media file format
[33] ISO/IEC 14496-16:2011 Information technology -- Coding of audio-visual objects -- Part 16:
Animation Framework eXtension (AFX)
[34] ISO/IEC 14496-20:2008 Information technology -- Coding of audio-visual objects -- Part 20:
Lightweight Application Scene Representation (LASeR) and Simple Aggregation Format (SAF)
[35] ISO/IEC 14496-25:2011 Information technology -- Coding of audio-visual objects -- Part 25: 3D
Graphics Compression Model
[36] ISO/IEC 23000-13:2017 Information technology -- Multimedia application format (MPEG-A) -- Part
13: Augmented reality application format
[38] ISO/IEC 23005-1:2016 Information technology -- Media context and control -- Part 1: Architecture
[39] ISO/IEC 23008-2:2017 Information technology -- High efficiency coding and media delivery in
heterogeneous environments -- Part 2: High efficiency video coding
[40] ISO/IEC 23008-3:2019 Information technology -- High efficiency coding and media delivery in
heterogeneous environments -- Part 3: 3D audio
[41] ISO/IEC PDTR 23090-1 Coded representation of immersive media – Part 1: Immersive media
[42] ISO/IEC 23090-2:2019 Information technology -- Coded representation of immersive media -- Part
2: Omnidirectional media format
[43] ISO/IEC AWI 23090-3 Coded representation of immersive media -- Part 3: Versatile video coding
[44] ISO/IEC AWI 23090-4 Coded representation of immersive media -- Part 4: Immersive audio
[45] ISO/IEC CD 23090-5 Coded representation of immersive media -- Part 5: Video-based Point Cloud
Compression
[46] ISO/IEC WD 23090-6 Information technology -- Coded representation of immersive media -- Part 6:
Immersive media metrics
[47] ISO/IEC WD 23090-7 Information technolgy -- Coded representation of immersive media -- Part 7:
Immersive media metadata
[48] ISO/IEC CD 23090-9 Information technology -- Coded representation of immrsive media -- Part 9:
Geometry based -- PCC
[49] ISO/IEC WD 23090-10 Information technology -- Coded representation of immersive media -- Part
10: Carriage of point cloud data
[50] ISO/IEC 19788-1:2011 Information technology -- Learning, education and training -- Metadata for
learning resources -- Part 1: Framework
[51] ISO/IEC TR 20748-1:2016 Information technology for learning, education and training -- Learning
analytics interoperability -- Part 1: Reference model
[52] ISO/IEC TR 20821:2018 Information technology -- Learning, education and training -- Learning
environment components for automated contents adaptation
[54] ISO/IEC TR 18121:2015 Information technology -- Learning, education and training -- Virtual
experiment framework
[55] ISO/IEC 19778-1:2015 Information technology -- Learning, education and training -- Collaborative
technology -- Collaborative workplace -- Part 1: Collaborative workplace data model
[56] ISO/IEC 23988:2007 Information technology -- A code of practice for the use of information
technology (IT) in the delivery of assessments