Leveraging sensor data for real time recognition of engagement in adaptive serious games for health

Abstract

The aim of the present Doctoral Thesis is the development of a novel conceptual framework for personalization in serious games (SGs) for health. The proposed framework leverages sensor data for the recognition of player engagement during interaction with SGs for health in real time. This approach aims to automatically generate game content based on player engagement, in-game performance, and health-related needs. In the present thesis, two novel SGs for health are designed and developed, aiming to promote self-health management in chronic health conditions and incorporating mechanics to facilitate procedural generation of content. A novel technique, based on a genetic algorithm, that employs heterogeneous data for procedural content generation (PCG) in SGs for health is presented. Two carefully designed experimental processes are implemented to investigate the feasibility of the proposed framework. The experimental processes collect data from sensors and interaction with the SG for health and investigate matters of player experience, educational value of the intervention, and efficiency of the proposed PCG technique. The two employed SGs for health aim to promote food and nutrition literacy and raise awareness and promote self-health management for obstructive sleep apnea, respectively. Analysis for sensor-based real-time recognition of engagement is conducted by approximating the ground truth, in terms of perceived engagement, through continuous annotations by participants. Results indicate that the educational value of the first SG for health is similar to a traditional intervention and demonstrate the predictive capacity of features extracted from the collected data towards perceived player engagement. In addition, statistically significant differences are revealed in terms of player experience in correlation with the generated content by the PCG. Furthermore, the PCG technique’s capacity to rely on clinically relevant sensor data to produce tailored game content is investigated in a pre-pilot study employing a SG for health for children with type 1 diabetes and/or obesity. Results display the PCG’s effectiveness in generating useful and relevant content, tailored to player needs, based on sensor and platform interaction data. Finally, the proposed PCG technique is evaluated with the use of deep reinforcement learning (DRL) agents for automated playing. Results indicate an overall superiority in DRL agents’ training when exposed to SG content produced by the proposed PCG technique. Overall findings included in the present Doctoral Thesis advocate towards the feasibility of the introduced conceptual framework. The insights gained from the experimental processes provide convincing arguments towards creating a closed real-time engagement feedback loop in adaptive SG for health, based on sensor and interaction data.