A framework to co-optimize task and social dialogue policies using Reinforcement Learning

Florian Pecune, Stacy Marsella

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

One of the main challenges for conversational agents is to select the optimal dialogue policy based on the state of the interaction. This challenge becomes even harder when the conversational agent not only has to achieve a specific task, but also aims at building rapport. Although some work already tried to tackle this challenge using a Reinforcement Learning (RL) approach, they tend to consider one single optimal policy for all the users, regardless of their conversational goals. In this work, we describe a framework that allows us to build a RL-based agent able to adapt its dialogue policy depending on its user's conversational goals. After we build a rule-based agent and a user simulator communicating at the dialog-act level, we crowdsource the surface sentences authoring for both the simulated users and the agent, which allow us to generate a dataset of interactions in natural language. Then, we annotate each of these interactions with a single rapport score and analyze the links between simulated users' conversational goals, agent conversational policies, and rapport. Our results show that rapport was higher when both or none of the interlocutors tried to build rapport. We use this result to inform the design of a social reward function, and we rely on this social reward function to train a RL-based agent using an hybrid approach of supervised learning and reinforcement learning. We evaluate our approach by comparing two different versions of our RL-based agent: one that takes users' conversational goals into account and another that does not. The results show that an agent adapting its dialogue policy depending on users' conversational goals performs better.

Original languageEnglish
Title of host publicationProceedings of the 20th ACM International Conference on Intelligent Virtual Agents, IVA 2020
PublisherAssociation for Computing Machinery, Inc
ISBN (Electronic)9781450375863
DOIs
Publication statusPublished - 2020 Oct 20
Externally publishedYes
Event20th ACM International Conference on Intelligent Virtual Agents, IVA 2020 - Virtual, Online, United Kingdom
Duration: 2020 Oct 202020 Oct 22

Publication series

NameProceedings of the 20th ACM International Conference on Intelligent Virtual Agents, IVA 2020

Conference

Conference20th ACM International Conference on Intelligent Virtual Agents, IVA 2020
Country/TerritoryUnited Kingdom
CityVirtual, Online
Period20/10/2020/10/22

Keywords

  • Conversational Agent
  • Dialogue Manager
  • Rapport
  • Reinforcement Learning
  • Socially-Aware

ASJC Scopus subject areas

  • Artificial Intelligence
  • Human-Computer Interaction

Fingerprint

Dive into the research topics of 'A framework to co-optimize task and social dialogue policies using Reinforcement Learning'. Together they form a unique fingerprint.

Cite this