Extended Reality and Human Computer Interaction for Medicine, Cultural Heritage and Education
Contact: Lucio Tommaso De Paolis
Mail: lucio.depaolis@unisalento.it
Application field
Technology
Activity
Health, Medicine, Surgery, Rehabilitation, Cultural Heritage, Education
Virtual, Augmented and Mixed Reality, Touchless interaction, Haptic devices
Simulation, Interaction, Visualization
Keywords
Virtual Reality (VR), Augmented Reality (AR), Mixed Reality (MR), Human Computer Interaction (HCI), Touchless Interaction, Gestures, Spatial Augmented Reality (SAR), Serious Games, Presence, Immersion, Engagement, User Experience, Usability
Operation of the proposed solution
The research activity in medicine and surgery is focused on the use of Virtual and Augmented Reality technologies to support surgeons in the diagnosis, the preoperative planning and the intraoperative phase in minimally invasive surgery. In addition, the use of touchless interaction is being investigated to enable a simple and intuitive fruition of specific patient’s data in those environments, such as the operative theatre, where any contact with the devices should be avoided to provide aseptic interactions between computer systems and medical staff.
Another line of research concerns the use of virtual, augmented and mixed reality for the treatment of children with ADHD through game-oriented activities (serious games).
Software applications are being designed, also based on mixed reality environments and serious games, for monitoring and rehabilitating patients with motor problems using touchless devices for detecting postures and arm and hand movements.
The research activity in Cultural Heritage is investigating the use of Virtual and Augmented Reality technologies to recreate historic sites for educational purposes and showcase features at visitor centres. In particular, Virtual Reality allows opening up real places normally not accessible to people by providing a realistic computer-generated navigation and sense of presence.
Spatial augmented reality, also known as video mapping, turns a building into a display surface for video projection: it uses audio-visual language to tell the story of a building and its surroundings through animations and sounds.
Challenges
1. Guiding the surgeon during an operation by providing him with a sort of ”X-Ray” vision of the internal patient’s anatomy, enriched in real time with contextually relevant information
2. Reducing the surgeon’s mental workload to imagine the three-dimensional structure based on CT imagesdesigning systems that simplify the interaction with technology, making it as natural and intuitive as possible
3. Making rehabilitation activities enjoyable by transforming them into game activities according to the serious game paradigm
4. Using augmented visualization to create a bridge between the people of the ancient culture and the modern users
5. Exploiting the serious games paradigm to make the study and promotion of cultural heritage enjoyable
6. Using video mapping to enable a creative process aiming at emotionally engaging the public and involving them in an unrepeatable experience
Bibliography
De Paolis L.T., Vite S.T., Padilla Castañeda M.A., Domínguez Velasco C.F., Muscatello S., Hernández Valencia A.F., An Augmented Reality Platform with Hand Gestures-based Navigation for Applications in Image-Guided Surgery: Prospective Concept Evaluation by Surgeons, International Journal of Human–Computer Interaction, 2021;
Video