Development of Emotion Estimator for Dynamic States Using Electroencephalogram and Galvanic Skin Response

Ayato Narikawa, Wen Wen, Hiroyuki Hamada, Acer Chang, Yukio Honda, Shinji Kanda, Hiroshi Mizoguchi, Erina Kakehashi, Masahiro Nishio, Koji Makino, Atsushi Yamashita and Hajime Asama

For designing production systems, evaluation of mental states in dynamic situations is important. Although there are many studies that considered emotion estimation using questionnaire or physiological signals, few studies conducted experiments in dynamic situations and no method for evaluation of mental states during work involving motion has been established, yet. Therefore, whether it is possible to estimate mental states during dynamic work, based on physiological signals is not cleared. In this study, we developed emotion estimator for dynamic states using physiological signals. Specifically, our aim is to identify the psychological states during tasks involving movement using physiological measures. In the experiment, we conducted a task simulating order picking, where participants had to retrieve parts from boxes within a time limit and place them in a cart. Seven healthy participants were included (two males and five females; mean age = 27.3 ± 6.8 years). The experiment manipulated two factors: task difficulty (easy vs. difficult) and social-comparative feedback (negative vs. positive). During the experiment, questionnaire about three items (mental stress, a sense of achievement, and motivation) and electroencephalogram (EEG) and galvanic skin response (GSR) were measured. We used k-nearest neighbors (k-NN) machine learning for development of emotion estimator. As input data, we extracted wide frequency bands (θ: 4-8 Hz, α1: 8-10 Hz, α2: 10-13 Hz, α: 8-13 Hz, β1: 13-22 Hz, β2: 22-30, β: 13-30 Hz, and γ1: 30-47 Hz) from EEG and mean and maximum value of GSR. Then, we used independent variables of the experiment and subjective ratings through questionnaires as the ground truth labels. Across all the participants, we achieved high accuracy in classification of difficulty levels and mental stress levels: 87.9% in classifying difficulty levels and 96.5% in classifying mental stress levels on average across all the participants. This suggests that the experiment successfully influenced physiological signals. EEG and GSR have been widely used as indicators of autonomic nervous system activity and are likely to reflect difficulty and mental stress, leading to high accuracy.