Multimodality of face to face communication

Human communication is intrinsically multimodal. Gestures, facial expressions, eye gaze, are not 'non-verbal cues', detached from the content of the message, but fundamental components of communicative intentions, both in production and in comprehension. However, we still lack a coherent picture of how the different modalities interact in different languages and settings of communication. And yet, this is a necessary step for our understanding of language differences. In this regard, any approach is welcome, both theoretical, aiming at developing semiotic models of multimodality, or experimental and-or corpus-based, to implement with the help of the available state of art technology (automatic annotation, automatic movement detection, statistical models).