Speech2rtMRI: Speech-Guided Diffusion Model for Real-time MRI Video of the Vocal Tract during Speech

Hong Nguyen, Sean Foley, Kevin Huang, Xuan Shi, Tiantian Feng, Shrikanth Narayanan
Signal Analysis and Interpretation Laboratory (SAIL) at University of Southern California
Summitted to ICASSP 2025
Paper Supplementary Code arXiv
MY ALT TEXT

Figure 1: Overview of our speech-2-rtMRI Diffusion modeling framework for generating vocal tract movement video during speech. Our modeling framework includes two main phases: training and sampling .

Abstract

Understanding speech production both visually and kinematically can inform second language learning system designs, as well as the creation of speaking characters in video games and animations. In this work, we introduce a data-driven method to visually represent articulator motion in Magnetic Resonance Imaging (MRI) videos of the human vocal tract during speech based on arbitrary audio or speech input. We leverage large pre-trained speech models, which are embedded with prior knowledge, to generalize the visual domain to unseen data using an speech-to-video diffusion model. Our findings demonstrate that the visual generation significantly benefits from the pre-trained speech representations. We also observed that evaluating phonemes in isolation is challenging but becomes more straightforward when assessed within the context of spoken words. Limitations of the current results include the presence of unsmooth tongue motion and video distortion when the tongue contacts the palate.

MY ALT TEXT
Figure 2: The speech chain from higher-level linguistic representations to acoustic output. Our focus in this work is on the low-level articulation with the aim to generate vocal tract movements conditioned on acoustic prompts.

Reference

@article{nguyen2024speech2rtmri,
        title={Speech2rtMRI: Speech-Guided Diffusion Model for Real-time MRI Video of the Vocal Tract during Speech},
        author={Nguyen, Hong and Foley, Sean and Huang, Kevin and Shi, Xuan and Feng, Tiantian and Narayanan, Shrikanth},
        journal={arXiv preprint arXiv:2409.15525},
        year={2024}
      }