Project

AURA: Augmenting human musical interaction with embodied virtual avatars

Code
G0A3R25N
Duration
01 January 2025 → 31 December 2028
Funding
Research Foundation - Flanders (FWO)
Promotor-spokesperson
Research disciplines
  • Humanities and the arts
    • Musicology and ethnomusicology
    • Music performance
  • Social sciences
    • Artificial intelligence
    • Social perception and cognition
  • Engineering and technology
    • Interactive and intelligent systems
Keywords
Recognition and modeling of human nonverbal behavior and responses Human expression and emotion Embodied virtual avatars
 
Project description

In science and society as a whole, we witness rapid adoption of intelligent, AI-driven agents that can engage in various contexts of human conversation and collaboration. Complementary to human-agent interaction through speech and text, intelligent agents may visually appear in humanlike form, constituting so-called embodied virtual avatars (EVAs). By including the body and nonverbal behavior in the communication process, EVAs can add a profound social and emotional dimension to human-agent interaction. Yet, it is an ongoing challenge to reliably detect intentions and emotional states of human users, and to let EVAs respond accordingly through nonverbal behavior in socially contingent, intelligent and meaningful ways. In the proposed project, we bridge the domains of social AI, musicology, and social and cognitive science to meet this challenge in the domain of piano duet playing. Elaborating on an existing musical agent for expressive piano accompaniment (ACCompanion, https://www.youtube.com/watch?v=Wtxcqp-sQ_4), we aim developing EVAs, visually represented as three-dimensional humanlike holograms via mixed reality technology, that can intelligibly and emotionally engage in live piano playing with a human pianist. Next to their practical use in artistic-creative contexts, we take the concept and methodological potential of EVAs as a springboard to increase scientific knowledge on social action, cognition and emotion in humans.