Mohamed Amir Omezzine and Pigni Federico (Grenoble Ecole de Management, France); Lucia Billeci (National Research Council of Italy, Italy)
Abstract
Artificial intelligence, immersive XR/VR environments, and wearable sensing technologies are increasingly converging in education. However, most adaptive learning systems still rely on behavioral and performance indicators such as correctness, completion, response time, and help-seeking. These indicators are useful but often too coarse to capture the cognitive and affective conditions under which learning unfolds. This paper develops a design-oriented framework for a State-Aware Digital Twin (SADT), defined as an instrumented XR/VR task environment in which adaptive decisions are conditioned on confidence-scored evidence about learner state. Grounded in Information Systems research, Flow Theory, Cognitive Load Theory, and NeuroIS, the framework specifies a five-layer architecture linking XR task telemetry, behavioral traces, wearable sensing, a state engine, and a bounded-action orchestrator. The paper contributes a conservative-adaptation principle, a reference architecture, and a staged validation protocol for testing when multimodal sensing adds intervention value beyond behavioral logs and XR telemetry.
Summary
This paper develops a State-Aware Digital Twin (SADT) framework for adapting extended and virtual reality (XR/VR) learning environments using evidence about learners’ cognitive and affective states. It addresses a central problem in adaptive learning: observable behaviors can have several explanations. For example, hesitation may reflect careful reasoning, confusion or fatigue, while rapid completion may indicate mastery or superficial engagement. Drawing on Information Systems research, Flow Theory, Cognitive Load Theory and NeuroIS, the framework proposes five interconnected layers: an instrumented XR task environment; task telemetry and behavioral traces; wearable sensing, including EEG and motion data; a state engine that combines these inputs into confidence-scored estimates; and an orchestrator that selects bounded, pre-authorized interventions. Its conservative-adaptation principle keeps behavioral evidence as the default basis for decisions and allows physiological signals to influence adaptation only when behavioral indicators are insufficient, conflicting or relevant to learner safety. These signals are treated as uncertain evidence requiring adequate quality, synchronization with task events and confidence thresholds calibrated during the pilot. Depending on the evidence, the system may simplify hints, adjust pacing or difficulty, introduce a short pause, or alert an instructor. When confidence is low or indicators conflict, it returns to behavioral adaptation or human review. The paper proposes three validation phases: passive instrumentation to assess data quality and synchronization; human decision support to evaluate whether state estimates improve instructors’ interpretations; and limited automatic adaptation to test intervention value. Evaluation would compare the framework with behavior-only systems using task completion, error recovery, help-seeking efficiency, perceived cognitive load, trust calibration and learner safety. Its contribution is therefore a design principle, reference architecture and validation protocol for determining when additional sensing improves adaptive decisions; its educational benefits remain to be empirically demonstrated.
