Assessing users’ emotion at interaction time: a multimodal approach with multiple sensors

Abstract

Users’ emotional states influence decision making and are essential for the knowledge and explanation of users’ behavior with computer applications. However, collecting emotional states during the interaction time with users is a onerous task because it requires very careful handling of the empirical observation, leading researchers to carry out assessments of emotional responses only at the end of the interaction. This paper reports our research in assessing users’ behavior at interaction time and also describes the results of a case study which analyzed users’ emotional responses while interacting with a game. We argue that capturing emotions during interaction time can help us in making changes on users’ behavior (e.g., changing from stressed to a less stressed state) or even suggesting an user to have a break. This can be all possible if both (1) emotions are captured during interaction and (2) changes are suggested at runtime (e.g., through persuasion). The results of this study suggest that there are significant differences between emotional responses captured during the interaction and those declared at the end.