Interaction between Real and Virtual Worlds

Virtual reality (VR) technology associated for a long time solely with video games, has since experienced a major boom, particularly in the field of training. The «democratisation» of VR headsets is no stranger to this. The number of headsets sold has exploded from 5 million units in 2014 to 68 million in 2020, their cost has dropped and the technology itself has evolved. We are now talking about immersive technologies including virtual, augmented and mixed realities (VR/AR/MR). UTC has been a pioneer since it introduced, as early as 2001, teaching in virtual reality and launched, within its Heudiasyc laboratory, research on both the fundamental and application levels. The interaction between the real and virtual worlds opens up immense fields of application, particularly in relation to robotics. For example, we can interact with a drone that maps the damage caused by a natural disaster in places that have become inaccessible. Obviously, these new possibilities can be used for malicious purposes, and this raises several ethical issues. UTC’s academics are aware of this.

Interaction between Real and Virtual Worlds

Scenery adaptation in VR (Virtual Reality)

Domitile Lourdeaux, a university lecturer at UTC, is a member of the Knowledge, Uncertainty, Data (CID) research team at the UTC-Heudiasyc Lab. Her investigations focus on personalised adaptive systems in VR (Virtual Reality).

VR is a field of research that she has been exploring since gaining her PhD. "I am interested in the adaptation of scripted content according to a dynamic user profile and, more particularly, in virtual environments dedicated to training. In as much as we are dealing with learners, I am working in particular on the adaptation of educational content and narration. In a word: how to stage learning situations in virtual reality (VR)," explains Domitile Lourdeaux.

The fields of application are varied and past or current projects attest to this. There was Victeams on the training of 'medical leaders', a project funded by the ANR and the Directorate General of the French Armed Forces (DGA) and ongoing programmes Orchestraa as well as Infinity¹, a European project involving eleven countries and twenty partners including Manzalab, coordinated by Airbus Defense & Space. The former was launched in November 2019, thanks to funding from the DGA and has among its partners Réviatech, Thalès as well as CASPOA, a NATO centre of excellence, the latter in June 2020; both for a duration of three years.

"Orchestraa aims to train air command leaders in military air operations centres. There are about forty people in the room coordinating military operations. The learner is given a VR headset and interacts with his team of autonomous virtual characters; these are linked to fighter pilots, helicopters, drones, etc. They will therefore play out a scenario planned several days in advance. They will therefore playout a scenario planned several days in advance, but on the day, there may be unforeseen circumstances that disrupt the original scenario and that they must be able to manage. These may be unforeseen requests for air support or a plane crashing, in which case the pilots must be rescued in the field. These hazards will require a reallocation of resources, ultimately creating a chain reaction impacting the entire operation. In this case, the adaptation concerns the difficulty of the scenario, which will increase according to what the learner is capable of managing. This will require them to demonstrate progressive skills," she explains.

Infinity concerns a completely different field. This large-scale European project aims to provide tools -AI (artificial intelligence), VR for data visualisation and analysis - to improve the collaboration of European police in investigations against cybercrime and terrorism. "We have three use cases: analysis of the behaviour of cyberattacks during an ongoing event, rapid analysis in the aftermath of a terrorist attack and finally hybrid threats, resulting from the convergence of cyberterrorism and terrorism, "explains Domitile Lourdeaux.

Infinity is a project in which URC-Heudiasyc is the leader of a monitoring tool on recommendations to guarantee the well-being of police officers using VR, a technology that indeed causes side effects due to the helmets (cybersickness, visual fatigue) and can also generate cognitive overload and stress related to the tasks to be performed in VR. "In this project, we are focusing on the side effects. We are trying to measure them in real time while the users have the headset on to diagnose their condition. We are interested in "three states" in particular: the cybersickness common to all VR headset users, the mental load related to the complexity of the task and stress and its multiple causes. To detect these side effects, we use physiological sensors ((ECGs) electrocardiograms, electrodermal activity, oculometry and pupillometry), behavioural data (task efficiency) and questionnaires", stresses Alexis Souchet, a post-doc student also at the UTC Heudiasyc Lab..

Virtual reality does, however, raise various legal and ethical problems. "Manufacturers are going to make VR tools available to people without taking into account the harmful impacts on their health. This is a real problem when you consider the explosion in the sale of headsets, which has risen from 5 million units in 2014 to 68 million in 2020," concludes Domitile Lourdeaux.