AR Seminar | Hestia Hotel Europa | May 9, 2019

Mixed Reality in Professional Training and Higher Education:
The Emergence of Digital Twins

Aleksei Tepljakov$^{\star}$

$^{\star}$Head of Re:creation VR&AR Laboratory, Centre for Intelligent Systems
Department of Computer Systems, Tallinn University of Technology, Estonia

About the Speaker: Aleksei Tepljakov

  • Research Scientist at Alpha Control Systems Research Laboratory, Centre for Intelligent Systems, Department of Computer Systems
  • Head of Re:creation Virtual and Augmented Reality Laboratory, VR First laboratory coordinator
  • IEEE Senior Member, Chair of IEEE Estonia Young Professionals Affinity Group, Vice Chair of IEEE Estonia Education Chapter
  • E-mail: aleksei.tepljakov@ttu.ee, website https://starspirals.net/

The state of MR ($\approx$ VR $+$ AR) technologies

(Encouraging and motivating factors)

 
 

Furthermore

  • As of 2016, industrial applications of VR have a 14% market share and this number is expected to increase [1].
  • Heather Bellini of Goldman Sachs Research expects virtual and augmented reality to become an \$80 billion market by 2025 (\$45bn in hardware and \$35bn in software), roughly the size of the desktop PC market today [2].
  • A recent market report [3] claims that the global augmented and virtual reality market is expected to reach approximately \$814.7bn by 2025.

References for market research

  1. Grand View Research, Inc., “Virtual reality market size growth & analysis, VR industry report 2025,” 2017. [Online]. Available: https://www.grandviewresearch.com/industry-analysis/virtual-reality-vr-market.
  2. Goldman Sachs Group, Inc., “Virtual & augmented reality: Goldman Sachs report,” 2017. [Online]. Available: http://www.goldmansachs.com/our-thinking/pages/technology-driving-innovation-folder/virtual-and-augmented-reality/report.pdf.
  3. Zion Market Research, “Press release for report: Augmented and virtual reality market by device (head-mounted display (HMD), handheld device, head-up display (HUD), projector and display wall, gesture-tracking device, and others), by offering (hardware and software), by application (consumer, commercial, enterprise, and others), and by vertical (entertainment & media, gaming, healthcare, aerospace & defense, manufacturing, retail, education, and others): Global industry perspective, comprehensive analysis, and forecast, 2018–2025,” 2019. [Online]. Available: https://www.zionmarketresearch.com/report/augmented-and-virtual-reality-market.

Digital Twins: State-of-the-Art

The following discussion is largely based on F. Tao, H. Zhang, A. Liu, and A. Y. C. Nee, “Digital twin in industry: State-of-the-art,” IEEE Transactions on Industrial Informatics, vol. 15, no. 4, pp. 2405–2415, 2019.

Four definitions of digital twin (DT)

  1. (Grieves, 2003) A DT includes three parts: physical product, virtual product, and their connections.
  2. (NASA, 2012) DT is a multiphysics, multiscale, probabilistic, ultrafidelity simulation that reflects, in a timely manner, the state of a corresponding twin based on the historical data, real-time sensor data, and physical model.
  3. (Gabor, 2016) DT is a special simulation, built based on the expert knowledge and real data collected from the existing system, to realize a more accurate simulation in different scales of time and space.
  4. (Mauer, 2017) DT is a digital representation that can depict the production process and product performance.

Emergence of digital twins

Five-dimensional model of digital twins

Key components of digital twins

  1. DT modeling and simulation: the basis of implementing DTs in practical applications;
  2. Data fusion: combining massive data (physical, virtual, historical, etc.) into a coherent resource;
  3. Interaction and collaboration: including reactive and proactive behavior of DT and meaningful man-machine interactions;
  4. Service: Data-driven DTs can reinforce services such as structure monitoring, lifetime forecasting, predictive maintenance, etc.

Let us now observe how MR and DT are related in the context of education

Successful Educational Applications of VR and AR

We shall review these from the perspective of digital twins:

  • Industrial worksite and factory training;
  • Learning complex design;
  • Space simulation (VR);
  • High school and university education.

Example: Worksite training

Virtual Reality Training by Immersive Technologies. Image taken from their website.

Example: Automotive Design

The engineers can get inside of the car in the virtual world and gain a much better idea of how their designs will be used by their customers, thus delivering more appropriate models. Image taken from esi-group.com.

Example: Industrial Visualization

A fermentation reactor producing itaconic acid, with textual descriptions to visualize the intrinsic processes. Image from RWTH Aachen University webpage.

Example: Safety in Robotics

The safety system in a robotic cell can be tested in the VR environment by means of an avatar. Image courtesy of the VR lab in the Department of Mechanical and Industrial Engineering of TUT.

Example: Space Simulation

This direction of development is actively pursued by NASA. The particular visual you see here was generated in Space Engine which already has builtin VR support.

How does Re:creation VR&AR lab
fit into this picture?

Re:creation VR and AR Laboratory

  • Part of our research is devoted to developing educational applications in VR and AR.
  • The laboratory has grown out of Alpha (Intelligent) Control Systems Research Laboratory, the members of which have diverse academic backgrounds and experience, including, but not limited to...

The problem

Based on the specifics of our background and professional activities, we have outlined the following problem:

  • How to improve lab instruction for system theory and control systems knowing that hands-on experiments with control objects are immensely helpful in this process?

With the emergence of digital twins in VR, we seek the solution in the form of a suitable VR or AR application.

The current solution

  • To complement the MATLAB/Simulink laboratory exercises, we create interactive visualizations.
  • The models are running in real time and are connected to the visualization platform via UDP sockets.
  • Unreal Engine 4 is used to implement the visualization, interaction logic, and to finally deliver the Virtual Reality experience to the user.

The concept

Case study: modeling the MLS

An overview of the actual implementation in UE4

(Some brief technical details)

 

Using the UDPReceiver class to animate the levitated sphere

 

Implementation of interactions: UE4-MATLAB feedback loop

 

Video: MagLev System Case Study

What's next?

  • Evaluation of our solutions in control groups;
  • Integration of control object like digital twins into actual industrial processes;
  • Convergence between learning, training and actual practice: e.g., the worker can first practice using the digital twins and advanced 3D UI to tune control loops and then do the same on the real-life objects using the same UI.
  • Development of IIoT based solutions with industrial protocols and plug-and-play solutions;

Collaboration in TalTech: IVAR Lab

  • IVAR lab (Industrial Virtual and Augmented Reality) is part of Department of Mechanical and Industrial Engineering, School of Engineering, TalTech.
  • Primary research directions: digital twins of industrial robots and smart manufacturing.
 

Acknowledgements

Our lab is part of VR First (https://vrfirst.com/)—a growing network of laboratories involved in developing immersive technologies and applications.

 

Thank you for your attention!