Dialogue Acts and Emotions in Multimodal Dyadic Conversations

Research output: Chapter in Book/Report/Conference proceedingArticle in proceedingsResearchpeer-review

This paper addresses the relation between dialogue
acts and emotions in a Danish multimodal annotated corpus of
first encounters. Dialogue acts are semantic generalizations of the
communicative functions of speech and gestures. Certain emotion
types have been found to be strongly related to feedback in a
previous study, and therefore we wanted to investigate the relation
between emotion and dialogue acts further. Our analysis of the
most frequently occurring dialogue acts and the co-occurring
emotions in the corpus confirms that there is a strong relation
between some dialogue act types and specific emotion types
and the relation is is not only limited to feedback functions.
Two speech segment representations and emotion labels are used
as features in machine learning experiments in which various
classifiers were trained to identify the 15 most frequent dialogue
acts in the data. The results of the experiments show that using the
two speech segment representations as training data give state of-
the art results for dialogue act classification, that exclusively
relies on speech segments information. Adding information about
emotions improves classification when the classifiers are Logistic
Regression and a multilayer perceptron.
Original languageEnglish
Title of host publication12th IEEE International Conference on Cognitive Infocommunications : CogInfoCom 2021
PublisherIEEE
Publication date2021
Pages429-434
ISBN (Print)978-1-6654-2495-0
ISBN (Electronic)978-1-6654-2494-3
Publication statusPublished - 2021

ID: 289023502