Towards Emotion-based Adaptive Games: Emotion recognition via input and performance features

During game play undesired emotions can arise, impeding players’ positive experiences. Adapting game features based on players’ emotions can help to address this problem, but necessitates a way to detect the current emotional state. We investigate using input parameters on a graphics tablet in combination with in-game performance to unobtrusively detect the players’ current emotional state. We conducted a user study with 48 participants to collect self-reported emotions, input data from the tablet and in-game performance in a serious game teaching players to write Japanese hiragana characters. We synchronized data, extracted 46 features, trained machine learning models, and evaluated their performance to predict levels of valence, arousal, and dominance modeled as a seven class problem. The analysis shows that random forests achieve good accuracies with F ₁ scores of .567 to .577 and AUC of .738 to .740, while using input features or in-game performance alone leads to highly decreased performance. Finally, we propose an architecture that developers can use to define undesired emotion levels to provide adaptive content generation in combination with emotion recognition.