Skip to content

Projects

Ongoing Projects

    Computational Speech Science: SPAN - Speech Production and Articulation Knowledge

      RI Core Medium: Structured variability in vocal tract articulation dynamics in speech

      Short Description:

      Sponsored by: NSF

      CompCog: Deep causal inference grounds the perception of cognitive objects in speech

      Short Description:

      Sponsored by: NSF

      Multilingualism as a factor of resilience to Alzheimer's disease and related dementias in India

      Short Description:

      Sponsored by: NIH

        SLP: Speech and Language Processing Technologies

          AI Conversational Interactions

          Short Description:

          Sponsored by: Walt Disney Company

            BioSP: Biosignal Sensing and Processing

              PRECOG: Multimodal integration of neural and biobehavioral signals for predicting preconscious responses

              Short Description: Objectively illuminate the complex interplay between multimodal neural and biobehavioral signals of preconscious processing (e.g., about a certain intent in response to an external stimulus), mental states (e.g., emotions), and well-being or risk therein (e.g., suicidal ideation)

              Sponsored by: DARPA NEAT

              EDA and Sensor Fusion for Fatigue/Affective State Detection

              Short Description: Sensor analysis and fusion for detecting human states and affect is a challenging open problem in ubiquitous computing research, as well as a popular milestone for the automobile industry. The use of driver assistance systems has become increasingly popular due to advances in Artificial Intelligence, with the aim of improving road safety and reducing the number of accidents caused by human error. Despite their great potential, the deployment of such technologies is still at infant stage, especially when considering the driver’s affective state, which can greatly impact driving performance. This project aims to address this issue by developing systems and improving the performance of affective state detection in driving with the use of multimodal biometric sensor information, such as EDA, ECG, PPG, and respiration.

              Sponsored by: Toyota Research Institute North America (TRINA)

              Detecting and mapping stress patterns across space and time: Multimodal modeling of individuals in real-world physical and social work environments

              Short Description:

              Sponsored by: NSF Smart Connected Health

                BSP: Behavioral Signal Processing

                  SCC-CIVIC-FA Track B: Everyday Respect: Measuring & Improving Police Officer Communication

                  Short Description: This project combines detailed survey and interview-based research into stakeholder perspectives and preferences on officer communication, a community-led process to refine research objectives in our analysis of communication during motor vehicle stops (Phase 1) with sophisticated human annotation and ML measurement of communication using audio, CPS-3 of 4 video, and personnel data that are rarely shared with outside researchers (Phase 2)

                  Sponsored by: NSF

                  SFARI: Multimodal, objective assessment of the ASD phenotype: Longitudinal stability and change across contexts

                  Short Description:

                  Sponsored by: Simons Foundations

                  The Development and Systematic Evaluation of an AI-Assisted Just-in-Time-Adaptive-Intervention System for Improving Child Mental Health

                  Short Description: The major goal of this project is to develop an evidence driven technology supported approach for mental health interventions.

                  Sponsored by: NIH through Colliga Apps Corp

                  Using Automated Methods to Classify Language Ability in Children with Autism

                  Short Description:

                  Sponsored by: Apple

                  Evaluating BOSCC and ELSA as Outcome Measures in the Context of a JASPER Intervention Trial

                  Short Description:

                  Sponsored by: Simons Foundation

                    CMI: Computational Media Intelligence

                      Early Detection of Influence Indicators with Machine Intelligence (EDIFICE)

                      Short Description: Develop methods for detecting emotions and other indicators of influence from social media data.

                      Sponsored by: DARPA INCAS

                      Rich and Efficient Media Content Processing – Supplement

                      Short Description: The goal is to develop multimedia signal informatics for discerning gender representations in media.

                      Sponsored by: Google (With Geena Davis Institute on Gender in Media)

                        TMI: Trustworthy Machine Intelligence (Foundations of Signal Analysis and Interpretation)

                          SADIRI: Stylometric Authorship Discernment and Interpretation for Realistic Inputs

                          Short Description: Developing methods for for effective, explainable, and multi-lingual authorship attribution and privacy protection

                          Sponsored by: IARPA HIATUS

                            SAIL Project Partnerships

                                Completed

                                MRI: Development of a High-Performance Low-Field MRI for Dynamic Imaging

                                Short Description:

                                Sponsored by: NSF

                                Rich and Efficient Media Content Processing

                                Short Description: The goal is to develop multimedia signal informatics for discerning gender representations in media.

                                Sponsored by: Google Giving through Geena Davis Institute

                                Speech Quality Estimation for ASR

                                Short Description:

                                Sponsored by: Disney Research

                                Dynamics of Vocal Tract Shaping

                                Short Description: The major goal of this project is to develop real-time MRI for vocal tract imaging and the study of articulatory control mechanisms.

                                Sponsored by: NIH/NIDCD

                                Federated Learning for Human-centered Experience and Perception Modeling with Biobehavioral Data

                                Short Description:

                                Sponsored by: Amazon-USC Center

                                Learning Robust Representations

                                Short Description: The aim of the project is to detect adversarial attacks and develop defense algorithms to make neural systems more resilient against such attacks for audio-based speaker identification.

                                Sponsored by: DARPA

                                TILES: Tracking IndividuaL performance with Sensors

                                Short Description: The Tracking Individual Performance with Sensors (TILES) is a project holding multimodal data sets for the analysis of stress, task performance, behavior, and other factors pertaining to professionals engaged in a high-stress workplace environments. Biological, environmental, and contextual data was collected from hospital nurses, staff, and medical residents both in the workplace and at home over time. Labels of human experience were collected using a variety of psychologically validated questionnaires sampled on a daily basis at different times during the day. The data sets are publicly available and we encourage researchers to use it for data mining and testing their own human behavior models.

                                Sponsored by: IARPA (MOSAIC Program)

                                Developing Scalable Measures of Behavior Change for ASD Treatments

                                Short Description: Develop quantitative metrics based on signal processing and machine learning as markers of behavior change in ASD.

                                Sponsored by: Simons Foundation (With Cornell Weil Medical College)

                                Exploiting Language Information for Situational Awareness (ELISA)

                                Short Description: Develop multilingual emotion and sentiment analysis techniques for disaster assistance/humanitarian relief scenarios.

                                Sponsored by: Department of Defense (DARPA LORELEI program)

                                Technologies for Assessing Behavioral and Cognitive Markers of Suicide Risk

                                Short Description: The goal is to develop behavioral signal processing methods to support suicide research and interventions.

                                Sponsored by: Department of Defense (with University of Utah)

                                Testing a family-based preventative intervention on childhood obesity in primary care

                                Short Description: The goal is to develop automated behavioral coding algorithms for a coach's behavior in obesity interventions.

                                Sponsored by: CDC (with Arizona State University, Northwestern, U Washington)

                                Implementation of Technology-based Evaluation of Motivational Interviewing

                                Short Description: The goal of this effort is to create the computational framework for analysis and modeling therapist fidelity in the Motivational Interviewing style of therapy that aims to change behavior of addiction patients.

                                Sponsored by: NIH/NIAAA (with University of Washington)

                                Development of a Standardized Measure of Social-Communicative Abilities for Children with Neurodevelopmental Disorders

                                Short Description: The goal of this project is create novel technology-facilitated measures of socio-communication abilities.

                                Sponsored by: NIH (with UCSF)

                                RI: Medium: Collaborative Research: Understanding Individual-level Speech Variability: From Novel Articulatory Data to Robust Speaker Recognition

                                Short Description: The goal of this effort is to investigate the articulatory basis of inter speaker variability.

                                Sponsored by: NSF (With MIT Lincoln Laboratory)

                                The Degree of Synchrony Across Physiological and Behavioral Indicators in Aggression

                                Short Description: The goal of this project is to develop computational techniques to study relationship dynamics using data scientific approaches.

                                Sponsored by: NSF

                                Dating Couple Aggression: Using Mobile Technology to Assess Emotions, Vocalizations, and Physiology

                                Short Description: The goal of this project is to develop and apply multimodal technologies for investigating aggressive behavior in dating couples.

                                Sponsored by: NSF

                                Human-Building Integration: Bio-Sensing Adaptive Environmental Control for Human Health and Sustainability

                                Short Description: The goal of this project is to develop signals and systems techniques to enable user adaptive environmental control.

                                Sponsored by: NSF

                                Verbal/non-verbal Asynchrony in Adolescents with High Functioning Autism

                                Short Description: The focus in this project to develop computational techniques to analyze and model expressive communication in voice and face.

                                Sponsored by: NIH NIDCD through Emerson College

                                Intelligent Coordination and Adaptive Classification for Naval Autonomous Systems

                                Short Description: We examine the role of machine intelligence and adaptive classification systems in the context of wireless communications and cooperation to provide a coherent global framework for real-time evaluation and decision making for system deployments, navigation, and adaptive search.

                                Sponsored by: Office of Naval Research (ONR)

                                An Integrated Approach to Creating Enriched Speech Translation Systems

                                Short Description: The proposed research focuses on creating robust, widely-deployable and cost-effective technology solutions for enabling and enhancing cross-lingual spoken interaction between people who do not share a common language. Our target application focuses on communication between healthcare personnel who speak English only and patients with limited-English proficiency.

                                Sponsored by: National Science Foundation (NSF)

                                Computational Behavioral Science: Modeling, Analysis, and Visualization of Social and Communicative Behavior

                                Short Description: We propose an Expedition to define and explore a new research area of Behavior Imaging— integrated technologies for multi-modal computational sensing and modeling to capture, measure, and understand human behaviors. Our motivating goal is to revolutionize the diagnosis and treatment of behavioral and developmental disorders.

                                Sponsored by: NSF: Expeditions in Computing

                                ISE: Be A Scientist! A project on informal science learning

                                Short Description: BE A SCIENTIST! is a new model in informal science education (ISE) for bringing cutting-edge science to the public through service learning, with applicability to different science, technology, engineering and mathematics (STEM) fields. The goal is to create a scalable, technology-based, ISE infrastructure to recruit more underserved families, encourage deeper participant understanding of the scientific process and cost-effectively assess long term program impact on the participants.

                                Sponsored by: National Science Foundation (NSF)

                                NeTS: Large: Collaborative Research: Exploration and Exploitation in Actuated Communication Networks

                                Short Description: The goals of the Exploration and Exploitation in Actuated Communication Networks project are to design networking tools for mobile underwater networks, develop novel navigation mechanisms for communication-constrained autonomous underwater vehicles and to ultimately integrate sensing and classification to provide solutions for the exploration-exploitation tradeoff.

                                Sponsored by: National Science Foundation (NSF)

                                Detection and Computational Analysis of Psychological Signals (DCAPS)

                                Short Description: Human vocal behavior–conveying both verbal and non verbal information–signifies a key vehicle for such computational modeling, and offers a promising venue for evidence driven behavioral assessment and diagnostics. Our team brings extensive experience in voice, speech, spoken language and associated nonverbal behavior analysis.

                                Sponsored by: Defense Advanced Research Projects Agency (DARPA)

                                Collaborators: Raytheon BBN Technologies

                                TATRC: Advancing Speech Recognition Technology to Support Training with Virtual Humans

                                Short Description: Human spoken language–conveying both verbal and non verbal information–is a key vehicle for not only determining “who” is saying “what” but offers a promising venue for capturing higher-level socio-emotional behavioral assessment and modeling. Specific problems being studied include: Automatic Speech Recognition Robust to Age, Gender, and Linguistic background-Robust Voice activity detection and Speaker Segmentation-Novel Speech Features and others.

                                Sponsored by: Army Research Office (ARO)

                                Targeted Robust Audio Processing System - TRAP Project

                                Short Description: Speech processing in low signal-to-noise (SNR) condition is challenging depending on data recording and transmission conditions. As the spectral characteristics change from one type of sound to another, the way their acoustic properties change in low SNR depends on the type of sound and the spectral characteristics of the noise.

                                Sponsored by: Defense Advanced Research Projects Agency (DARPA)

                                Collaborators: IBM TJ Watson

                                Personalized Socially-Assistive Human-Robot Interaction: Applications to Autism Spectrum Disorder

                                Short Description: This interdisciplinary proposal identifies a specific set of HRI research questions in socially assistive robotics, the study of robotic systems capable of providing help through social rather than physical interaction. The research focus is on two key issues: (1) the role of the robot’s physical embodiment in the interaction; and 2) the use of expressive embodied communication and user modeling toward personalized time-extended assistive interaction.

                                Sponsored by: National Science Foundation & Nancy Laurie Marks Family Foundation & Clinical and Translational Sciences Institute

                                Automating Behavioral Coding via Text-Mining and Speech Signal Processing

                                Short Description: Psychotherapy intervention studies collect audio and/or videotapes of therapy sessions to use in treatment adherence and process research. These tapes record the complex series of interactions between therapist and client, and as such, they encode the active ingredients of the therapy – when the therapy works, the tapes should tell us why.

                                Sponsored by: National Institute of Health (NIH): NIAAA

                                Collaborators: Univeristy of Washington & University of Callifornia Irvine

                                Model-Based Phenotyping of OSAS in Pediatric Obesity using Dynamic MR Imaging

                                Short Description: Obstructive sleep apnea syndrome (OSAS) has been reported to occur in approximately 50% in morbidly obese children, compared to only ~2% in the general pediatric population. This marked difference in prevalence rates becomes even larger if one were to include other forms of sleep-related breathing disorders (SRBD) such as hypoventilation and hypercapnia or hypoxemia without frank obstructions.

                                Sponsored by: National Institute of Health (NIH): NHLBI

                                Dynamics of Vocal Tract Shaping

                                Short Description: The long-term goal of this project is to wed state-of-the-art technology for imaging the vocal tract with a linguistically informed analysis of the speech tasks or goals requisite in the production of spoken language. Importantly, this includes furthering our understanding of the relations between vocal tract shaping and speech acoustics.

                                Sponsored by: National Institute on Deafness and Other Communication Disorders (NIH NIDCD)

                                Data Collection and Analysis of Vocal-Tract MRI: Speaker specific properties and biometrics

                                Short Description: The proposal aims to understand, as a function of vocal tract structure, (1) the ways speakers modulate the spatiotemporal organization of syllable articulation and (2) the ways such articulatory variations interact with speech acoustic properties.

                                Sponsored by: Department of Justice & MIT Lincoln Lab

                                Collaborators: MIT Lincoln Lab

                                Novel Machine Learning Techniques for Speech Recognition of Languages with Low-Density Resources

                                Short Description: We propose to achieve the Babel program goals with a three-pronged approach: we will develop language-independent architectures and systems for Reliable Keyword Search; we will push the boundaries on Learning with Limited Labeled Training Data; and we will greatly enhance the Robustness to Noise, Channel, and Speaker Variability of the underlying speech recognition systems for keyword search.

                                Sponsored by: IBM Machine Learning

                                Speech Prosody and Articulatory Dynamics in Spoken Language

                                Short Description: In this new phase of the research program, we outline a research strategy that approaches the investigation of the control and coordination of speech initiation and cessation within a dynamical systems model of speech production.

                                Sponsored by: National Institute on Deafness and Other Communication Disorders (NIH NIDCD)

                                Sensor Subset Processing: Robust processing of remote audio and speech

                                Short Description: Automatic Speech Recognition (ASR) can currently achieve very high performance given the right conditions: low noise, appropriately sized training data with the desired speaker population, and special hardware, such as wearable, directional microphones and push to talk buttons. Maintaining high performance while relaxing each of these conditions is ultimately necessary to achieve the full potential of this technology. Achieving robustness of performance in such systems hence is a major research goal to develop broadly usable speech technology systems especially in unconstrained interaction settings.

                                Sponsored by: Office of Naval Research (ONR)

                                Collaborators: BAE Systems

                                Emotional Speech Production: Analysis, Modeling and Synthesis

                                Short Description: Vocal expression of emotions is an integral part of human speech communication. Investigating how emotional information is encoded in speech is hence important from a variety of viewpoints both scientific (in advancing our understanding of human speech generation) and technological (in creating computational tools for the analysis, recognition and synthesis of emotional speech).

                                Sponsored by: National Institute of Health (NIH): NHLBI

                                Quantitative Observational Practice in Family Studies: The case of reactivity

                                Short Description: The goal of this proposal is to transform the current practice of observational behavior analysis methodologies by enabling a computational framework for the analysis and modeling of emotionally-rich human interactions through signal processing and machine learning technologies. Using two large existing audio visual corpora of real couple interactions with rich psychologist-specified annotations (from 134 couples and 130 families), we propose to design and test algorithms and tools for automated behavioral coding.

                                Sponsored by: National Institute of Health (NIH): NHLBI

                                Improving Accuracy, Speed and Usability of Multi-Modal Biometric Handheld Devices

                                Short Description: This research will develop multi-modal biometric fusion and supporting technologies to significantly increase the accuracy and usability of mobile handheld devices in a tactical environment. To be fully effective, mobile devices must be much faster, simpler to operate, and more consistent in acquiring quality biometrics relative to the current generation of systems.

                                Sponsored by: Army Research Lab

                                Collaborators: Cogent Systems

                                TransTac SpeechLinks

                                Short Description: The Transonics team will achieve the ground-breaking power of computer-mediated crosslanguage communication by applying our in depth understanding of multilingual speech recognition in acoustically challenging environments, speech-based machine translation, embedded systems technologies, and human dialogue and communication.

                                Sponsored by: Defense Advanced Research Projects Agency (DARPA) & BBN Technologies

                                Computing Research Infrastructure for Human-Robot Interaction and Socially Assistive Robotics

                                Short Description: Robotics is poised to enter many arenas of human everyday use with the potential of positively impacting quality of life, especially for growing populations with special needs. However, research in the field of human-robot interaction (HRI) and the new field of socially assistive robotics (SAR), the study of robotic systems capable of providing help through social rather than physical interaction, is currently limited by a lack of richly interactive robotic testbeds suitable for human-subjects experimental evaluation.

                                Sponsored by: National Science Foundation (NSF)

                                Computing Research Infrastructure for Human-Robot Interaction and Socially Assistive Robotics

                                Short Description: Robotics is poised to enter many arenas of human everyday use with the potential of positively impacting quality of life, especially for growing populations with special needs. However, research in the field of human-robot interaction (HRI) and the new field of socially assistive robotics (SAR), the study of robotic systems capable of providing help through social rather than physical interaction, is currently limited by a lack of richly interactive robotic testbeds suitable for human-subjects experimental evaluation.

                                Sponsored by: National Science Foundation (NSF)

                                Automating Early Assessment of Academic Standards for Very Young Native and Non-Native Speakers of American English

                                Sponsored by: National Science Foundation (NSF – ITR)

                                Responsive Virtual Human Museum Guides

                                Sponsored by: National Science Foundation (NSF)

                                Exploring Emotional Vocal Productions through the use of Real-time Magnetic Resource Imaging (SGER)

                                Short Description: How do singers produce different sounds with different emotions? New technology tools, such as fast magnetic resonance imaging, combined with novel computational capabilities, such as statistical machine learning, offer ways for gaining insights into, and measuring and modeling, these processes in ways that were not possible before.

                                Sponsored by: National Science Foundation (NSF)

                                Human-Like Speech Processing

                                Short Description: We introduce a radically new, unified approach to speech-to-text which replaces the HMM at all levels yet is firmly grounded in a powerful statistical machine learning framework. It is motivated by insights into human speech perception, specifically recent studies of human word recognition in everyday conversations. These indicate that large improvements in speech-to-text under diverse conditions cannot be obtained by changing only isolated system components.

                                Sponsored by: Office of Naval Research (ONR) & University of Washington

                                Joint Fires and Effects Trainer System

                                Sponsored by: United States Army (ARO)

                                Tactical Questioning

                                Sponsored by: United States Army (ARO)

                                Virtual Humans - SIPI Natural Language

                                Sponsored by: United States Army (ARO)

                                Dynamics of Vocal Tract Shaping

                                Short Description: The long-term goal of this project is to wed state-of-the-art technology for imaging the vocal tract with a linguistically informed analysis of the vocal tract constriction actions in order to understand the production and cognitive control of the compositional action units of spoken language. Our vision has been to develop and use real time MRI to illuminate the inherently dynamic speech production processes.

                                Sponsored by: National Institute on Deafness and Other Communication Disorders (NIH NIDCD)

                                Mobile Device Biomonitoring to Prevent and Treat Obesity in Underserved Minority Youth (KNOWME Network)

                                Short Description: The overarching goal is to create, implement, and validate individualized engineering systems for enabling evidence driven health care and management. These systems will empower the health care experts with just-in-time information so they can plan and implement individualized intervention. We will target three distinct stakeholders: children, practitioners, and the community.

                                Sponsored by: Qualcomm & National Institutes of Health (NIH)

                                Speech Prosody and Articulatory Dynamics in Spoken Language

                                Short Description: The complex messages and emotions of spoken language must be communicated by the precise choreography of the jaw, tongue, lips, larynx, and respiratory system. The long term objective of the proposed research program is to understand how linguistic structure conditions the spatiotemporal realization of articulatory movement during speaking.

                                Sponsored by: National Institute on Deafness and Other Communication Disorders (NIH NIDCD)

                                Career: Modeling and Optimizing User-Centric Mixed-Initiative Spoken Dialog Systems

                                Short Description: Conversational interfaces hold the promise of providing natural, easy and universal access to information. The research effort of this project targets two specific problems in conversational engineering under a unifying stochastic modeling framework: (1) dialog interaction modeling (2) user modeling in spoken dialog.

                                Sponsored by: National Science Foundation (NSF)

                                Modeling Creative and Emotive Improvisation in Theater Performance

                                Short Description: The proposed research will study creativity in improvisation in both standard theatrical techniques where a script’s interpretation, including the physical performance, is improvised by an actor, and "improv theatre” where entire scenes are created by actors in real-time through improvisation. This research will increase the state of knowledge about improvisation, creativity, and intelligent agent design, as well as contribute meaningfully to theoretical and academic understanding of creative practice in theatre.

                                Sponsored by: National Science Foundation (NSF)

                                Robotics and Speech Processing Technology for the Facilitation of Social Communication Training in Children with Autism

                                Short Description: The proposed study is expected to produce results that will enable future in-depth intervention studies of robot and computer-assisted therapies for communication and social skills for children with autism.

                                Sponsored by: Autism Speaks Foundation

                                CKID

                                Sponsored by: Department of Homeland Security

                                Deep Green

                                Sponsored by: Defense Advanced Research Projects Agency (DARPA) & BAE Systems

                                Virtual Sick Call

                                Sponsored by: United States Army (ARO)

                                Disseminating Scientific Information about Autism to the Latino Community

                                Short Description: This project utilizes theories of health communication and relationship-oriented social marketing strategies to develop and field-test a prototype for distilling scientific articles on ASD into consumer-friendly audience-defined Science Briefs and to effectively disseminate these informational products to Latino families.

                                Sponsored by: National Institute on Mental Health (NIH NIMH) & American Recovery and Reinvestment Act (ARRA)

                                Content-based Approach to Indexing, Query and Retrieval of Music

                                Short Description: The proposed research involves development of methods for content-based indexing of music databases using a combination of signal processing and knowledge-based methods, design of statistical algorithms for enabling queries using sung or hummed melodies, and design of robust search techniques for retrieving the queried information especially in the presence of uncertainty.

                                Sponsored by: National Science Foundation(NSF)-ITR

                                Training Superiority

                                Short Description: The objectives of this project are (1) to develop computer-based training systems for rapid acquisition of mission-oriented communication skills, targeted at multiple languages and missions, and (2) to develop a toolset that permits the rapid construction of new training systems.

                                Sponsored by: Defense Advanced Research Projects Agency (DARPA) & Office of Naval Research (ONR)

                                Robust Speech Processing for Immersive Interactions

                                Short Description: Our ultimate goal is to enable immersive conversations, where people can freely interact with one another and with the virtual agents without regard to the physical constraints of where they are. Through creation and implementation of robust speech recognition technology, we propose to advance the usabilityand usefulness of military training simulation environments.

                                Sponsored by: United States Army & STRICOM

                                Toward Natural Child Machine Interaction

                                Short Description: The goal is to establish a highly interdisciplinary research program focused on Communication, Technology, and Children to explore realistic child-machine interactions, with the aim of developing and improving spoken language and multimedia technologies for children. Specific initial goals aim to design, collect, and analyze pilot experimental data of young children interacting with machines using natural modalities of speech and gestures, as well as more traditional modalities (e.g., mouse, keyboard, & joystick).

                                Sponsored by: Zumberge Interdisciplinary Grant

                                Automated Synthesis of Expressive Speech for Military Training

                                Short Description: his project aims to make significant progress in effective voice synthesis for virtual humans, building on current TTS technology. Our goal is high-quality expressive speech, i.e., speech that is intended to covey or evoke a particular emotion and affect.

                                Sponsored by: United States Army & STRICOM