Form and Function of Hand Gestures for Interpretation and Generation
Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review
Standard
Form and Function of Hand Gestures for Interpretation and Generation. / Navarretta, Costanza.
10th IEEE International Conference on Cognitive Infocommunications : CogInfoCom 2019 . IEEE, 2019. p. 215-220.Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review
Harvard
APA
Vancouver
Author
Bibtex
}
RIS
TY - GEN
T1 - Form and Function of Hand Gestures for Interpretation and Generation
AU - Navarretta, Costanza
PY - 2019
Y1 - 2019
N2 - This paper investigates the relation between the form and function of hand gestures in audio and video recordings of American and English political discourse of different type. Gestures have an important function in face-to-face communication contributing to the successful delivery of the messageby reinforcing what is expressed by speech or by adding new information to what is uttered. The relation between form and function of gestures has been described by some of the pioneers of gestural studies. However, since gestures are multifunctional and they must be interpreted in context, it is important to investigate to what extent the form of gestures can be used to interprettheir function automatically. Individuating the relation between form and function of gestures is also important for generating appropriate gestures in various communicative situations and this knowledge is vital for the integration of machine-human communicative and cognitive functions. In this paper we showthat the automatic classification of the semiotic types of hand gestures using their shape description is quite successful and this is an important step towards their interpretation in face-to-face communication as well as their interpretation and generation in advanced multimodal interactive systems. More specifically in thepresent work we annotated the semiotic types of hand gestures produced by five politicians in different contexts, adding this information to existing multimodal annotations. Then, we trained machine learning algorithms to identify the semiotic type of the hand gestures. The F1 score obtained by the best performingalgorithms on the classification of four semiotic types is 0.7 outperforming the F1 score of 0.59 obtained in a preceding pilot study which addressed the identification of three semiotic types of hand gestures produced by a speaker in a small dataset.
AB - This paper investigates the relation between the form and function of hand gestures in audio and video recordings of American and English political discourse of different type. Gestures have an important function in face-to-face communication contributing to the successful delivery of the messageby reinforcing what is expressed by speech or by adding new information to what is uttered. The relation between form and function of gestures has been described by some of the pioneers of gestural studies. However, since gestures are multifunctional and they must be interpreted in context, it is important to investigate to what extent the form of gestures can be used to interprettheir function automatically. Individuating the relation between form and function of gestures is also important for generating appropriate gestures in various communicative situations and this knowledge is vital for the integration of machine-human communicative and cognitive functions. In this paper we showthat the automatic classification of the semiotic types of hand gestures using their shape description is quite successful and this is an important step towards their interpretation in face-to-face communication as well as their interpretation and generation in advanced multimodal interactive systems. More specifically in thepresent work we annotated the semiotic types of hand gestures produced by five politicians in different contexts, adding this information to existing multimodal annotations. Then, we trained machine learning algorithms to identify the semiotic type of the hand gestures. The F1 score obtained by the best performingalgorithms on the classification of four semiotic types is 0.7 outperforming the F1 score of 0.59 obtained in a preceding pilot study which addressed the identification of three semiotic types of hand gestures produced by a speaker in a small dataset.
M3 - Article in proceedings
SN - 9781728147932
SP - 215
EP - 220
BT - 10th IEEE International Conference on Cognitive Infocommunications
PB - IEEE
ER -
ID: 231369987