Emotional Sophistication: Studies of Facial Expressions in Games

Abstract

Decision-making is a complex process. Monetary incentives constitute one of the forces driving it, however the motivational space of decision-makers is much broader. We care about other people, we experience emotional reactions, and sometimes we make mistakes. Such social motivations (Sanfey, 2007) drive our own decisions, as well as affect our beliefs about what motivates others' decisions. Behavioral and brain sciences have started addressing the role of social motivations in economic games (Camerer, 2004; Glimcher et al., 2009), however several aspects of social decisions, such as the process of thinking about others' emotional states - emotional sophistication - have been rarely investigated. The goal of this project is to use automatic measurements of dynamic facial expressions to investigate non-monetary motivations and emotional sophistication. The core of our approach is to use state-of-the-art computer vision techniques to extract facial actions from videos in real-time (based on the Facial Action Coding System of Ekman and Friesen (1978)), while participants are playing economic games. We will use powerful statistical machine learning techniques to make inferences about participants internal emotional states during these interactions. These inferences will be used (a) to predict behavior; (b) to explain why a decision is made in terms of the hidden forces driving it; and (c) to investigate the ways in which people construct their beliefs about other people's future actions. The contributions of this targeted interdisciplinary project are threefold. First, it develops new methodologies to study decision processes. Second, it uses these methods to test hypotheses about the role of first order beliefs about social motivations. Finally, our statistical approach sets the ground for "affectively aware" systems, that can use facial expressions to assess the internal states of their users, thus improving human-machine interactions.