31 4 2012 8 Chinese Journal of Biomedical Engineering Vol. 31 No. 4 August 2012 * 1 200240 2-200240 - R318. 08 A 0258-8021 2012 04-0595-12 A Survey on EEG Based Emotion Recognition NIE Dan WANG Xiao-Wei DUAN Ruo-Nan LV Bao-Liang * 1 Center for Brain-like Computing and Machine Intelligence Department of Computer Science and Engineering Shanghai Jiao Tong University Shanghai 200240 China 2 MOE-Microsoft Key Lab for Intelligent Computing and Intelligent Systems Shanghai Jiao Tong University Shanghai 200240 China Abstract Emotion is a state that comprehensively represents human feeling thought and behavior thus takes an important role in inter-personal human communication. Emotion recognition aims to automatically discriminate different emotional states by using physiological and non-physiological signals acquired from people in order to develop friendly and natural human-machine interface. EEG-based emotion recognition is one of the most common used and efficient methods for emotion recognition. This paper introduced the state-of-the-art techniques the read-world applications and some critical problems of EEG-based emotion recognition from the aspects of EEG signals recording signal preprocessing feature extraction feature selection and classification methods. Key words emotion emotion recognition EEG feature extraction machine learning - doi 10. 3969 / j. issn. 0258-8021. 2012. 04. 00 2012-02-11 2012-05-18 90820018 973 2009CB320901 * E-mail bllu@ sjtu. edu. cn
596 31 6 1 1. 1 6 1 Papez James Fig. 1 The Papez circuit theory of the functional 1884 neuroanatomy of emotion 6 1 1885 Lange 2 James-Lange James- 1. 2 Lange 1927 Cannon James 3 Cannon 7 James Bard Ekman Cannon-Bard Cannon-Bard Clynes Izard 1937 Papez Papez 1 6 Papez 1 2 Lange 2 8 3 4 Papez Maclean 1. 3 5 Papez-Maclean
4 597 11 2 Paul 12 13 2 Lange Fig. 2 The Lange theory of emotions 3 14-3 15 1 Lang 3 9 3 10 1. 4 3 1 Fig. 3 3 11 Stimuli distribution in the 2-D emotional space formed by the arousal and valence 11
598 31 2. 1 2 EEG 16-17 18-19 2. 2 1. 3 1 3 Tab. 1 Comparison of three technologies for emotion recognition 4 4 autonomic nervous system 2. 3 central nervous system Picard 8 20 Bartels 21 fmri Amuse EEG Infomax 21 4
4 599 Fig. 4 4 a b Comparison of the EEG signals before artifact removal a and after artifact removal b 2. 4 N170 26 2. 4. 2 3-2. 4. 1 delta 1 ~ 4 Hz theta 4 ~ 8 Hz alpha 8 ~ 13 Hz beta 13 ~ 30 Hz gamma 36 ~ 44 Hz 5 Khalil 5 22 Zhang 23 event-related potentials 24 27 Zouridakis 5 5 180 ms Eimer 28 Aftanas theta alpha beta 120 ms 29 Kothe 25 120 ~
600 31 34 Aftanas 30 2. 4. 3 Scotti 36 Zhang 23 McDonald theta alpha Lin gamma 32 37 5 4 Christos 4 P100 N100 P200 N200 P300 31 delta theta alpha 11 2. 5 Murugappan Shi Murugappan alpha 5 38 32 2. 4. 4 128 3 K 35 2011 32 64 0 Petrantonakis higher order crossings 33 Khosrowabadi 4 ~ 13 Hz PCA ICA
4 601 5 Fig. 6 Relationship between the emotion a b classification accuracies of six subjects and number of feature dimensions the Fig. 5 Comparison of the features before and after using the LDS approach. a The features before using the LDS approach. b The features after using the LDS approach CSP 6 6 100 Li CSP 62 5 310 2 4 20 40 40 Firpi 6 41 2. 6 39 Lin F F F F 31 Zhang 2. 6. 1 600
602 31 K K K Frantzidis Murugappan C 44 4 32 Khosrowabadi Ko 45 Wang 4 46 42 2. 6. 2 47 K Estepp 48 2. 7 23 31 39 K Schmidt 43 4 K K 1
4 603 49 Sarlo 52 4 53 3D alpha 54-50 Petrantonakis - asymmetry Index cross-correlation 4-51 3. 2 Li gamma gamma 40 50 55 3. 3 alpha beta gamma Matthews 7 39 Fig. 7 3-56 3. 4 57 4 7 Distribution of emotional features 4. 1 3. 1 -
604 31 58-205. emotions 1885 33-90. 3 Cannon W. examination and an alternative theory J Psychology 1927 39 106-124. 4. 2 Neuroscience 2004 5 7 583 9. 8 Lang PJ Bradley MM Cuthbert BN. 2010 14 3 589-597. 4. 3 manua M Emotion and Attention 2008. 11 505-510. M 13 Coan JA Allen JJB McKnight PE. 5 1 James W. What is an emotion J. Mind 1884 9 34 188 2 Lange CG. The emotions a psychophysiological study J. The The James-Lange theory of emotions a critical - differences J 1996 24 239-266. emotions J 1056.. American Journal of 4 James W Papez MD. A proposed mechanism of emotion J. Arch Neural Psychiatry 1937 38 4 725-743. 5 Paul D Maclean MD. Psychosomatic disease and the Visceral Brain -recent developments bearing on the Papez theory of emotion J. Psychosomatic Medicine 1949 11 338-353. 6 Dalgleish T. The emotional brain J. Nature Reviews 7 Healey J. Wearable and automotive systems for the recognition of affect from physiology D. Massachusetts MIT 2000. International Affective Picture System IAPS Technical Manual And Affective Ratings M. Gainesville University of Florida 1999. 9 Frantzidis CA Bratsas C et al. Toward emotion aware computing an integrated approach using multichannel neurophysiological recordings and affective visual stimuli J. IEEE Transactions on Information Technology in Biomedicine 10 Lang PL Bradley MM Cuthbert BN. International Affective Picture System Affective ratings of pictures and instruction. Gainesville NIMH Center for the Study of. J. 2006 14 4 12 Ekman P Friesen WV. The facial action coding system FACS a technique for the measurement of facial action. San Francisco Consulting Psychologists Press 1978. A capability model of individual differences in frontal EEG asymmetry J. Biological Psychology 2006 72 198-207. 14 Crawford HJ Clarke SW Melissa KT. Self - generated happy and sad emotions in low and highly hypnotizable persons during waking and hypnosis laterality and regional EEG activity. International Journal of Psychophysiology 15 Damasio AR Grabowski TJ Bechara A et al. Subcortical and cortical brain activity during the feeling of self-generated. Nature Neuroscience 2000 3 10 1049 -
4 605 16. J. 31 Lin Yuanpin Wang Chihong Jung Tzyyping et al. EEG-based 2006 10 1359-1368. 17 Anderson K McOwan PW. A real - time automated system for the recognition of human facial expressions J. IEEE Transaction on System Man and Cybernetics Part B Cybernetics 2006 36 1 96-105. 18 Ang J Dhillon R Krupski A et al. Prosodybased automatic detection of annoyance and frustration in human computer dialog C / / Proceedeings of International Conference on Spoken Language Processing. Colorado ISCA 2002 2037-2039. 197. 19. J. 2007 16 4 374-376. 34 Khosrowabadi R Wahab A Ang KK et al. Affective computation on EEG correlates of emotion from musical and vocal 20 Picard RW Vyzas E Healey J. Toward machine emotional stimuli C / / Proceedings of International Joint Conference on intelligence analysis of affective physiological state J. IEEE Neural Networks. Atlanta IEEE 2009 1590-1594. Transaction on Pattern Analysis and Machined Intelligence 35 Aftanas LI Lotova NV Koshkarova VI et al. Non - linear 2001 23 10 1175-1191. 21 Bartels G Shi LC Lu BL. Automatic artifact removal from EEG - a mixed approach based on double blind source separation analysis of emotion EEG calculation of Kolmogorov entropy and the principal Lyapunov exponent J. Neuroscience Letters 1997 226 13-16. and support vector machine C / / Proceedings of International 36 Scotti S Mauri M Barbieri R et al. Automatic quantitative conference of the IEEE Engineering in Medicine and Biology evaluation of emotions in e-learning applications C / / Society. Buenos Aires IEEE 2010 5383-5386. 22 Khalili1 Z Moradi MH. Emotion recognition system using brain and peripheral signals using correlation dimension to improve the results of EEG C / / International Joint Conference on Neural Networks. Atlanta IEEE 2009 1571-1575. Proceedings of International Conference of the IEEE Engineering in Medicine and Biology Society New York City IEEE 2006 1359-1362. 37 McDonald NJ Soussou W. QUASAR s QStates cognitive gauge performance in the cognitive state Assessment competition 2011 23 Zhang Qing Lee Minho. A hierarchical positive and negative emotion understanding system based on integrated analysis of visual and brain signals J. Neurocomputing 2010 73 3264-3272. 24. M. 2010. emotion recognition in music listening J. IEEE Transaction on Biomedical Engineering 2010 57 7 1798-1806. 32 Murugappan M Rizon M Nagarajan R et al. Time - frequency analysis of EEG signals for human emotion detection C / / Proceedings of International Conference on Biomedical Engineering. Kuala Lumpur Springer 2008 21 262-265. 33 Petrantonakis PC Hadjileontiadis LJ. Emotion recognition from EEG using higher order crossings J. IEEE Transaction on Information Technology in Biomedicine 2010 14 2 186 - C / / Proceedings of International Conference of the IEEE Engineering in Medicine and Biology Society Boston IEEE 2011 6542-6546. 38 Shi LC Lu BL. Off - line and on-line vigilance estimation based on linear dynamical system and manifold learning C / / Proceedings of International Conference of the IEEE Engineering 25 Eimer M Holmes A. An ERP study on the time course of in Medicine and Biology Society Buenos Aires IEEE 2010 emotional face processing J. Cognitive Neuroscience and Neuropsychology 2002 13 4 1-5. 6587-6590. 39 Nie D Wang XW Shi LC et al. EEG-based emotion 26 Martin E Amanda H. Event - related brain potential correlates recognition during watching movies C / / Proceedings of IEEE of emotional face processing J. Neuropsychologia 2007 45 EMBS Conference on Neural Engineering. Cancun IEEE 1 15-31. 2011 667-670. 40 Li Mu Lu BaoLing. Emotion classification based on gammaband 27. M. 2007. 28 Zouridakis G Patidar U Padhye NS et al. Spectral power of EEG C / / Proceedings of International Conference of the IEEE Engineering in Medicine and Biology Society. Minneapolis brain activity associated with emotion a pilot MEG study IEEE 2009 1223-1226. C / / Proceedings of International Conference on Biomagnetism 41 Firpi HA Vogelstein RJ. Particle swarm optimization-based Advances in Biomagnetism. Dubrovnik Springer 2010 354-357. feature selection for cognitive state detection C / / Proceedings of International Conference of the IEEE Engineering in Medicine 29 Aftanas LI Lotova NV Koshkarov VI et al. Non-linear and Biology Society. Boston IEEE 2011 6556-6559. dynamic complexity of the human EEG during evoked emotions 42 Khosrowabadi R Quek HC Wahab A et al. EEG-based J. International Journal of Psychophysiology 1998 28 63 - emotion recognition using self-organizing map for boundary 76. detection C / / Proceedings of International Conference on 30 Kothe CA Makeig S. Estimation of task workload from EEG Pattern Recognition. Istanbul IEEE 2010 4242-4245. data new and current tools and perspectives C / / Proceedings 43 AlZoubi O Calvo RA Stevens RH. Classification of EEG for of International Conference of the IEEE Engineering in medicine and Biology Socit. Boston IEEE 2011 6547-6552. affect recognition an adaptive approach J. AI 2009
606 31 Advances In Artificial Intelligence Lecture Notes in Computer Science 2009 5866 52-61. 44 Frantzidis CA Bratsas C Klados MA et al. On the classification of emotional biosignals evoked while viewing affective pictures an integrated data-mining-based approach for healthcare applications J. IEEE Transaction on Information Technology in Biomedicine 2010 14 2 309-318. 45 Ko KE Yang HC Sim KB. Emotion recognition using EEG signals with relative power values and Bayesian network J. International Journal of Control Automation and Systems 2009 7 5 865-870. 46 Wang Ziheng Hope RM Wang Zuoguan et al. An EEG workload classifier for multiple subjects C / / Proceedings of International Conference of the IEEE Engineering in Medicine and Biology Society. Boston IEEE 2011 6534-6538. 47 Albornoz EM Milone DH Rufiner HL. Multiple feature extraction and hierarchical classifiers for emotions recognition C / / Proceedings of Development of Multimodal Interfaces Active Listening and Synchrony. Dublin Springer 2010 242-254. 48 Estepp JR Klosterman SL Christensen JC. An assessment of non - stationarity in physiological cognitive state assessment using artificial neural networks C / / Proceedings of International Conference of the IEEE Engineering in Medicine and Biology Society. Boston Massachusetts 2011 6552-6556. 49 Schmidt LA Trainor LJ. Frontal brain electrical activity EEG distinguishes valence and intensity of musical emotions J. Cognition & Emotion 2001 15 4 487-500. 50 Sarlo M Buodo G Poli S et al. Changes in EEG alpha power to different disgust elicitors the specificity of mutilations J. Neuroscience Letters 2005 382 291-296. 51 Petrantonakis PC Hadjileontiadis LJ. A novel emotion elicitation index using frontal brain asymmetry for enhanced EEG-based emotion recognition J. IEEE Transactions on Information Technology in Biomedicine 2011 15 5 737-746. 52 Galán F Nuttin M Lew E et al. A brain-actuated wheelchair Asynchronous and non-invasive Brain-computer interfaces for continuous control of robots J. Clinical Neurophysiology 2008 119 9 2159-2169. 53 Tao Liu Leslie Goldberg Shangkai Gao et al. An online brain-computer interface using non-flashing visual evoked potentials J. Journal of Neural Engineering 2010 7 3 3-6. 54 Qibin Zhao Liqing Zhang Andrzej Cichocki. EEG-based asynchronous BCI control of a car in 3D virtual reality environments J. Chinese Science Bulletin 2009 54 1 78-87. 55. EEG J. 2009 28 4 589-596. 56 Matthews R Turner PJ McDonald NJ et al. Real time workload classification from an ambulatory wireless EEG system using hybrid EEG electrodes C / / Proceedings of International Conference of the IEEE Egineering in Medicine and Biology Society. Vancouver IEEE 2008 5871-5875. 57 Li Xiaowei Hu Bin Zhu Tingshao et al. Towards affective learning with an EEG feedback approach C / / Proceedings of the first ACM international workshop on Multimedia technologies for distance learning. Beijing ACM 2009 33-38 58. J. 2010 29 5 777-784.