Logo: University of Southern California

Projects

 Ongoing Projects

 TILES: Tracking IndividuaL performancE with Sensors


Short Description:

Sponsored by: IARPA (MOSAIC Program)

 Developing Scalable Measures of Behavior Change for ASD Treatments


Short Description: Develop quantitative metrics based on signal processing and machine learning as markers of behavior change in ASD.

Sponsored by: Simons Foundation (With Cornell Weil Medical College)

 Rich and Efficient Media Content Processing


Short Description: The goal is to develop multimedia signal informatics for discerning gender representations in media.

Sponsored by: Google (With Geena Davis Institute on Gender in Media)

 Exploiting Language Information for Situational Awareness (ELISA)


Short Description: Develop multilingual emotion and sentiment analysis techniques for disaster assistance/humanitarian relief scenarios.

Sponsored by: Department of Defense (DARPA LORELEI program)

 Technologies for Assessing Behavioral and Cognitive Markers of Suicide Risk


Short Description: The goal is to develop behavioral signal processing methods to support suicide research and interventions.

Sponsored by: Department of Defense (with University of Utah)

 Testing a family-based preventative intervention on childhood obesity in primary care


Short Description: The goal is to develop automated behavioral coding algorithms for a coach’s behavior in obesity interventions.

Sponsored by: CDC (with Arizona State University, Northwestern, U Washington)

 Dynamics of Vocal Tract Shaping


Short Description: The major goal of this project is to develop real-time MRI for vocal tract imaging and the study of articulatory control mechanisms.

Sponsored by: NIH/NIDCD

 Implementation of Technology-based Evaluation of Motivational Interviewing


Short Description: The goal of this effort is to create the computational framework for analysis and modeling therapist fidelity in the Motivational Interviewing style of therapy that aims to change behavior of addiction patients.

Sponsored by: NIH/NIAAA (with University of Washington)

 Development of a Standardized Measure of Social-Communicative Abilities for Children with Neurodevelopmental Disorders


Short Description: The goal of this project is create novel technology-facilitated measures of socio-communication abilities.

Sponsored by: NIH (with UCSF)

 RI: Medium: Collaborative Research: Understanding Individual-level Speech Variability: From Novel Articulatory Data to Robust Speaker Recognition


Short Description: The goal of this effort is to investigate the articulatory basis of inter speaker variability.

Sponsored by: NSF (With MIT Lincoln Laboratory)

 The Degree of Synchrony Across Physiological and Behavioral Indicators in Aggression


Short Description: The goal of this project is to develop computational techniques to study relationship dynamics using data scientific approaches.

Sponsored by: NSF

 Dating Couple Aggression: Using Mobile Technology to Assess Emotions, Vocalizations, and Physiology


Short Description: The goal of this project is to develop and apply multimodal technologies for investigating aggressive behavior in dating couples.

Sponsored by: NSF

 Human-Building Integration: Bio-Sensing Adaptive Environmental Control for Human Health and Sustainability


Short Description: The goal of this project is to develop signals and systems techniques to enable user adaptive environmental control.

Sponsored by: NSF

 Verbal/non-verbal Asynchrony in Adolescents with High Functioning Autism


Short Description: The focus in this project to develop computational techniques to analyze and model expressive communication in voice and face.

Sponsored by: NIH NIDCD through Emerson College

 Speech Prosody and Articulatory Dynamics in Spoken Language


Short Description: The focus is to develop methods and algorithms to understand the signatures of speech prosody in language communication manifested in articulation.

Sponsored by: NIH/NIDCD

 Rich and Efficient Media Content Processing – Supplement


Short Description: The goal is to develop multimedia signal informatics for discerning gender representations in media.

Sponsored by: Google Giving through Geena Davis Institute

 Completed

 Intelligent Coordination and Adaptive Classification for Naval Autonomous Systems


Short Description: We examine the role of machine intelligence and adaptive classification systems in the context of wireless communications and cooperation to provide a coherent global framework for real-time evaluation and decision making for system deployments, navigation, and adaptive search.

Sponsored by: Office of Naval Research (ONR)

 An Integrated Approach to Creating Enriched Speech Translation Systems


Short Description: The proposed research focuses on creating robust, widely-deployable and cost-effective technology solutions for enabling and enhancing cross-lingual spoken interaction between people who do not share a common language. Our target application focuses on communication between healthcare personnel who speak English only and patients with limited-English proficiency.

Sponsored by: National Science Foundation (NSF)

 Computational Behavioral Science: Modeling, Analysis, and Visualization of Social and Communicative Behavior


Short Description: We propose an Expedition to define and explore a new research area of Behavior Imaging— integrated technologies for multi-modal computational sensing and modeling to capture, measure, and understand human behaviors. Our motivating goal is to revolutionize the diagnosis and treatment of behavioral and developmental disorders.
 
Sponsored by: NSF: Expeditions in Computing

 ISE: Be A Scientist! A project on informal science learning


Short Description: BE A SCIENTIST! is a new model in informal science education (ISE) for bringing cutting-edge science to the public through service learning, with applicability to different science, technology, engineering and mathematics (STEM) fields. The goal is to create a scalable, technology-based, ISE infrastructure to recruit more underserved families, encourage deeper participant understanding of the scientific process and cost-effectively assess long term program impact on the participants.
 
Sponsored by: National Science Foundation (NSF)

 NeTS: Large: Collaborative Research: Exploration and Exploitation in Actuated Communication Networks


Short Description: The goals of the Exploration and Exploitation in Actuated Communication Networks project are to design networking tools for mobile underwater networks, develop novel navigation mechanisms for communication-constrained autonomous underwater vehicles and to ultimately integrate sensing and classification to provide solutions for the exploration-exploitation tradeoff.
 
Sponsored by: National Science Foundation (NSF)

 Detection and Computational Analysis of Psychological Signals (DCAPS)


Short Description: Human vocal behavior--conveying both verbal and non verbal information--signifies a key vehicle for such computational modeling, and offers a promising venue for evidence driven behavioral assessment and diagnostics. Our team brings extensive experience in voice, speech, spoken language and associated nonverbal behavior analysis.

Sponsored by: Defense Advanced Research Projects Agency (DARPA)

Collaborators: Raytheon BBN Technologies

 TATRC: Advancing Speech Recognition Technology to Support Training with Virtual Humans


Short Description: Human spoken language--conveying both verbal and non verbal information--is a key vehicle for not only determining “who” is saying “what” but offers a promising venue for capturing higher-level socio-emotional behavioral assessment and modeling. Specific problems being studied include: Automatic Speech Recognition Robust to Age, Gender, and Linguistic background-Robust Voice activity detection and Speaker Segmentation-Novel Speech Features and others.
 
Sponsored by: Army Research Office (ARO)

 Targeted Robust Audio Processing System - TRAP Project


Short Description: Speech processing in low signal-to-noise (SNR) condition is challenging depending on data recording and transmission conditions. As the spectral characteristics change from one type of sound to another, the way their acoustic properties change in low SNR depends on the type of sound and the spectral characteristics of the noise.

Sponsored by: Defense Advanced Research Projects Agency (DARPA)
Collaborators: IBM TJ Watson

 Personalized Socially-Assistive Human-Robot Interaction: Applications to Autism Spectrum Disorder


Short Description: This interdisciplinary proposal identifies a specific set of HRI research questions in socially assistive robotics, the study of robotic systems capable of providing help through social rather than physical interaction. The research focus is on two key issues: (1) the role of the robot’s physical embodiment in the interaction; and 2) the use of expressive embodied communication and user modeling toward personalized time-extended assistive interaction.
 
Sponsored by: National Science Foundation & Nancy Laurie Marks Family Foundation & Clinical and Translational Sciences Institute

Automating Behavioral Coding via Text-Mining and Speech Signal Processing


Short Description: Psychotherapy intervention studies collect audio and/or videotapes of therapy sessions to use in treatment adherence and process research. These tapes record the complex series of interactions between therapist and client, and as such, they encode the active ingredients of the therapy – when the therapy works, the tapes should tell us why.
 
Sponsored by: National Institute of Health (NIH): NIAAA
Collaborators: Univeristy of Washington & University of Callifornia Irvine

 Model-Based Phenotyping of OSAS in Pediatric Obesity using Dynamic MR Imaging


Short Description: Obstructive sleep apnea syndrome (OSAS) has been reported to occur in approximately 50% in morbidly obese children, compared to only ~2% in the general pediatric population. This marked difference in prevalence rates becomes even larger if one were to include other forms of sleep-related breathing disorders (SRBD) such as hypoventilation and hypercapnia or hypoxemia without frank obstructions.
 
Sponsored by: National Institute of Health (NIH): NHLBI

 Dynamics of Vocal Tract Shaping


Short Description: The long-term goal of this project is to wed state-of-the-art technology for imaging the vocal tract with a linguistically informed analysis of the speech tasks or goals requisite in the production of spoken language. Importantly, this includes furthering our understanding of the relations between vocal tract shaping and speech acoustics.
 
Sponsored by: National Institute on Deafness and Other Communication Disorders (NIH NIDCD)
 

 Data Collection and Analysis of Vocal-Tract MRI: Speaker specific properties and biometrics


Short Description: The proposal aims to understand, as a function of vocal tract structure, (1) the ways speakers modulate the spatiotemporal organization of syllable articulation and (2) the ways such articulatory variations interact with speech acoustic properties.
 
Sponsored by: Department of Justice & MIT Lincoln Lab
Collaborators: MIT Lincoln Lab

 Novel Machine Learning Techniques for Speech Recognition of Languages with Low-Density Resources


Short Description: We propose to achieve the Babel program goals with a three-pronged approach: we will develop language-independent architectures and systems for Reliable Keyword Search; we will push the boundaries on Learning with Limited Labeled Training Data; and we will greatly enhance the Robustness to Noise, Channel, and Speaker Variability of the underlying speech recognition systems for keyword search.
 
Sponsored by: IBM Machine Learning

 Speech Prosody and Articulatory Dynamics in Spoken Language


Short Description: In this new phase of the research program, we outline a research strategy that approaches the investigation of the control and coordination of speech initiation and cessation within a dynamical systems model of speech production.
 
Sponsored by: National Institute on Deafness and Other Communication Disorders (NIH NIDCD)
 

 Sensor Subset Processing: Robust processing of remote audio and speech


Short Description: Automatic Speech Recognition (ASR) can currently achieve very high performance given the right conditions: low noise, appropriately sized training data with the desired speaker population, and special hardware, such as wearable, directional microphones and push to talk buttons. Maintaining high performance while relaxing each of these conditions is ultimately necessary to achieve the full potential of this technology. Achieving robustness of performance in such systems hence is a major research goal to develop broadly usable speech technology systems especially in unconstrained interaction settings.
 
Sponsored by: Office of Naval Research (ONR)
Collaborators: BAE Systems

 Emotional Speech Production: Analysis, Modeling and Synthesis


Short Description: Vocal expression of emotions is an integral part of human speech communication. Investigating how emotional information is encoded in speech is hence important from a variety of viewpoints both scientific (in advancing our understanding of human speech generation) and technological (in creating computational tools for the analysis, recognition and synthesis of emotional speech).
 
Sponsored by: National Institute of Health (NIH): NHLBI

 Quantitative Observational Practice in Family Studies: The case of reactivity


Short Description: The goal of this proposal is to transform the current practice of observational behavior analysis methodologies by enabling a computational framework for the analysis and modeling of emotionally-rich human interactions through signal processing and machine learning technologies. Using two large existing audio visual corpora of real couple interactions with rich psychologist-specified annotations (from 134 couples and 130 families), we propose to design and test algorithms and tools for automated behavioral coding.
 
Sponsored by: National Institute of Health (NIH): NHLBI

 Improving Accuracy, Speed and Usability of Multi-Modal Biometric Handheld Devices


Short Description: This research will develop multi-modal biometric fusion and supporting technologies to significantly increase the accuracy and usability of mobile handheld devices in a tactical environment. To be fully effective, mobile devices must be much faster, simpler to operate, and more consistent in acquiring quality biometrics relative to the current generation of systems.
 
Sponsored by: Army Research Lab
Collaborators: Cogent Systems

 Completed Projects

 TransTac SpeechLinks


Short Description: The Transonics team will achieve the ground-breaking power of computer-mediated crosslanguage communication by applying our in depth understanding of multilingual speech recognition in acoustically challenging environments, speech-based machine translation, embedded systems technologies, and human dialogue and communication.
 
Sponsored by: Defense Advanced Research Projects Agency (DARPA) & BBN Technologies

 Computing Research Infrastructure for Human-Robot Interaction and Socially Assistive Robotics


Short Description: Robotics is poised to enter many arenas of human everyday use with the potential of positively impacting quality of life, especially for growing populations with special needs. However, research in the field of human-robot interaction (HRI) and the new field of socially assistive robotics (SAR), the study of robotic systems capable of providing help through social rather than physical interaction, is currently limited by a lack of richly interactive robotic testbeds suitable for human-subjects experimental evaluation.
 
Sponsored by: National Science Foundation (NSF)

 Computing Research Infrastructure for Human-Robot Interaction and Socially Assistive Robotics


Short Description: Robotics is poised to enter many arenas of human everyday use with the potential of positively impacting quality of life, especially for growing populations with special needs. However, research in the field of human-robot interaction (HRI) and the new field of socially assistive robotics (SAR), the study of robotic systems capable of providing help through social rather than physical interaction, is currently limited by a lack of richly interactive robotic testbeds suitable for human-subjects experimental evaluation.
 
Sponsored by: National Science Foundation (NSF)

 Automating Early Assessment of Academic Standards for Very Young Native and Non-Native Speakers of American English


Sponsored by: National Science Foundation (NSF - ITR)

 Responsive Virtual Human Museum Guides


Sponsored by: National Science Foundation (NSF)

 Exploring Emotional Vocal Productions through the use of Real-time Magnetic Resource Imaging (SGER)


Short Description: How do singers produce different sounds with different emotions? New technology tools, such as fast magnetic resonance imaging, combined with novel computational capabilities, such as statistical machine learning, offer ways for gaining insights into, and measuring and modeling, these processes in ways that were not possible before.
 
Sponsored by: National Science Foundation (NSF)

 Human-Like Speech Processing


Short Description: We introduce a radically new, unified approach to speech-to-text which replaces the HMM at all levels yet is firmly grounded in a powerful statistical machine learning framework. It is motivated by insights into human speech perception, specifically recent studies of human word recognition in everyday conversations. These indicate that large improvements in speech-to-text under diverse conditions cannot be obtained by changing only isolated system components.
 
Sponsored by: Office of Naval Research (ONR) & University of Washington

 Joint Fires and Effects Trainer System


Sponsored by: United States Army (ARO)

 Tactical Questioning


Sponsored by: United States Army (ARO)

 Virtual Humans - SIPI Natural Language


Sponsored by: United States Army (ARO)

 Dynamics of Vocal Tract Shaping


Short Description:The long-term goal of this project is to wed state-of-the-art technology for imaging the vocal tract with a linguistically informed analysis of the vocal tract constriction actions in order to understand the production and cognitive control of the compositional action units of spoken language. Our vision has been to develop and use real time MRI to illuminate the inherently dynamic speech production processes.
 
Sponsored by: National Institute on Deafness and Other Communication Disorders (NIH NIDCD)

 Mobile Device Biomonitoring to Prevent and Treat Obesity in Underserved Minority Youth (KNOWME Network)


Short Description: The overarching goal is to create, implement, and validate individualized engineering systems for enabling evidence driven health care and management. These systems will empower the health care experts with just-in-time information so they can plan and implement individualized intervention. We will target three distinct stakeholders: children, practitioners, and the community.
 
Sponsored by: Qualcomm & National Institutes of Health (NIH)

 Speech Prosody and Articulatory Dynamics in Spoken Language


Short Description: The complex messages and emotions of spoken language must be communicated by the precise choreography of the jaw, tongue, lips, larynx, and respiratory system. The long term objective of the proposed research program is to understand how linguistic structure conditions the spatiotemporal realization of articulatory movement during speaking.
 
Sponsored by: National Institute on Deafness and Other Communication Disorders (NIH NIDCD)

Career: Modeling and Optimizing User-Centric Mixed-Initiative Spoken Dialog Systems


Short Description: Conversational interfaces hold the promise of providing natural, easy and universal access to information. The research effort of this project targets two specific problems in conversational engineering under a unifying stochastic modeling framework: (1) dialog interaction modeling (2) user modeling in spoken dialog.
 
Sponsored by: National Science Foundation (NSF)

Modeling Creative and Emotive Improvisation in Theater Performance


Short Description: The proposed research will study creativity in improvisation in both standard theatrical techniques where a script’s interpretation, including the physical performance, is improvised by an actor, and "improv theatre” where entire scenes are created by actors in real-time through improvisation. This research will increase the state of knowledge about improvisation, creativity, and intelligent agent design, as well as contribute meaningfully to theoretical and academic understanding of creative practice in theatre.
 
Sponsored by: National Science Foundation (NSF)

 Robotics and Speech Processing Technology for the Facilitation of Social Communication Training in Children with Autism


Short Description: The proposed study is expected to produce results that will enable future in-depth intervention studies of robot and computer-assisted therapies for communication and social skills for children with autism.
 
Sponsored by: Autism Speaks Foundation

 CKID


Sponsored by: Department of Homeland Security

 Deep Green


Sponsored by: Defense Advanced Research Projects Agency (DARPA) & BAE Systems

 Virtual Sick Call


Sponsored by: United States Army (ARO)

 Disseminating Scientific Information about Autism to the Latino Community


Short Description: This project utilizes theories of health communication and relationship-oriented social marketing strategies to develop and field-test a prototype for distilling scientific articles on ASD into consumer-friendly audience-defined Science Briefs and to effectively disseminate these informational products to Latino families.
 
Sponsored by: National Institute on Mental Health (NIH NIMH) & American Recovery and Reinvestment Act (ARRA)

 Content-based Approach to Indexing, Query and Retrieval of Music


Short Description: The proposed research involves development of methods for content-based indexing of music databases using a combination of signal processing and knowledge-based methods, design of statistical algorithms for enabling queries using sung or hummed melodies, and design of robust search techniques for retrieving the queried information especially in the presence of uncertainty.
 
Sponsored by: National Science Foundation(NSF)-ITR

 Training Superiority


Short Description: The objectives of this project are (1) to develop computer-based training systems for rapid acquisition of mission-oriented communication skills, targeted at multiple languages and missions, and (2) to develop a toolset that permits the rapid construction of new training systems.
 
Sponsored by: Defense Advanced Research Projects Agency (DARPA) & Office of Naval Research (ONR)

 Robust Speech Processing for Immersive Interactions


Short Description: Our ultimate goal is to enable immersive conversations, where people can freely interact with one another and with the virtual agents without regard to the physical constraints of where they are. Through creation and implementation of robust speech recognition technology, we propose to advance the usabilityand usefulness of military training simulation environments.
 
Sponsored by: United States Army & STRICOM

 Toward Natural Child Machine Interaction


Short Description: The goal is to establish a highly interdisciplinary research program focused on Communication, Technology, and Children to explore realistic child-machine interactions, with the aim of developing and improving spoken language and multimedia technologies for children. Specific initial goals aim to design, collect, and analyze pilot experimental data of young children interacting with machines using natural modalities of speech and gestures, as well as more traditional modalities (e.g., mouse, keyboard, & joystick).
 
Sponsored by: Zumberge Interdisciplinary Grant

 

 Automated Synthesis of Expressive Speech for Military Training


Short Description: his project aims to make significant progress in effective voice synthesis for virtual humans, building on current TTS technology. Our goal is high-quality expressive speech, i.e., speech that is intended to covey or evoke a particular emotion and affect.
 
Sponsored by: United States Army & STRICOM