Affect-as-Information Approach to a Sentiment Analysis Based Evaluation of Conversational Agents

In this paper we propose a novel method for automatic evaluation of conversational agents. The method is based on analyzing the user’s affect conveyed in utterances. From analyzing: the user’s general emotional engagement in the conversation and the emotion types conveyed by the user in the conversation, a simple psychological reasoning is derived about the user’s sentiment about the agent’s performance. The evaluation experiment on two Japanese-speaking conversational agents showed the same tendencies in the results returned by the system constructed on the proposed method and the user’s opinion about the two agents checked in the afterward survey. Thus the method can be used for evaluation of Japanese-speaking conversational agents.

• Type: Conference
• Year: 2008
• Where: CIMCA
• Authors: Michal Ptaszynski, Pawel Dybala, Shinsuke Higuchi, Rafal Rzepka, Kenji Araki
• Keywords: CIMCA 2008, Conversational Agents, Evaluation, Internet Technology, Japanese-speaking Conversational Agents