AI-Driven Gesture and Action Recognition for Learning Medicine Through Virtual Reality

Abstract

Virtual reality (VR) has emerged as a prominent immersive technology, demonstrating significant potential for enhancing education and has been shown as an effective and efficient tool specifically for medical education. VR requires the utilization of dedicated wearable devices that, when worn by students, facilitate the creation of an immersive virtual environment and enable them to interact with both other students and virtual objects within it. However, the use of these gadgets may produce a distortion of the gestures and actions actually performed by the student, causing errors within the environment due to the technology itself and not the student’s mistaken intention. In medical education, this effect of convoluting the student’s own intention with what the hardware interface might limit is of vital importance, as it may result in a, medically speaking, erroneous action or gesture in the virtual scenario. Therefore, in medical education, virtual reality applications should be smart and able to correct these technology-related issues and limitations. In this work, we present our analysis, strategy, and approach to propose AI-based solutions that minimize this limitation in VR-based medical education, describing some experiments that we will conduct with medical students at our University to improve the response of virtual reality in medical learning evaluation.