The overarching goal of this effort is to employ and advance intuitive and mathematically-principled signal representations and engineering models for describing human behavior.

Behavioral Signal Processing Overview

Behavioral signal processing focuses on gathering, analyzing and modeling multimodal behavior signals, both overtly and covertly expressed. This builds upon foundational efforts in speech/language/audio/visual/physiological signal processing. Critically, in addition to processing objectively-specified behavioral content in richer ways (e.g., what someone said and did), behavioral signal processing entails automating a host of subjectively-specified entities such as those related to socio-emotional states of people (e.g., how negative or frustrated a person is; politeness; engagement etc).

highlighted publications

Couples Interaction BSP

Assessment and interventions based on direct observations of human social, communicative and affective behavior are central to many realms including mental and behavioral healthcare and management.
In this research project our focus application is distressed couple and family interactions.
We focus our work on a range of datasets from longitudinal studies collected by our psychology collaborators and team members to multimodal collections that use a MOCAP system and camera and microphone arrays.

highlighted publications
  • James Gibson, Athanasios Katsamanis, Francisco Romero, Bo Xiao, Panayiotis Georgiou, Shrikanth S. Narayanan, "Multiple Instance Learning for Behavioral Coding", IEEE Transactions on Affective Computing, 2016.
  • Md Nasir, Wei Xia, Bo Xiao, Brian Baucom, Shrikanth S. Narayanan, Panayiotis Georgiou, "Still Together?: The Role of Acoustic Features in Predicting Marital Outcome", Proceedings of Interspeech, 2015.
  • Chi-Chun Lee, Athanasios Katsamanis, Matthew P. Black, Brian Baucom, Andrew Christensen, Panayiotis Georgiou, Shrikanth S. Narayanan, "Computing Vocal Entrainment: A Signal-derived PCA-based Quantification Scheme with Application to Affect Analysis in Married Couple Interactions", Computer, Speech, and Language, vol. 28, no. 2, pp. 518-539, 2014.
  • Matthew P. Black, Athanasios Katsamanis, Brian Baucom, Chi-Chun Lee, Adam Lammert, Andrew Christensen, Panayiotis Georgiou, Shrikanth S. Narayanan, "Toward automating a human behavioral coding system for married couples' interactions using speech acoustic features", Speech Communication, vol. 55, no. 1, pp. 1-21, 2013.
  • Panayiotis Georgiou, Matthew P. Black, Adam Lammert, Brian Baucom, Shrikanth S. Narayanan, ""That's aggravating, very aggravating": Is it possible to classify behaviors in couple interactions using automatically derived lexical features?", Proceedings of Affective Computing and Intelligent Interaction (ACII), Lecture Notes in Computer Science, 2011.

Autism BSP

Autism Spectrum Disorder (ASD) is the fastest growing developmental disorder in the United States, affecting 1 in 68 children (CDC, 2014). ASD is a social-communicative disorder with neurogenetic origins. There is a vast heterogeneity in cause (etiology) and behavioral profile (phenotype), making assessment and intervention that much more challenging.
In the CARE group, our goal is to develop, implement and validate engineering methods and tools for discovery, assessment, behavior tracking, stratification, and intervention related to communicative and affective social behavior in people with Autism Spectrum Disorders. Not only do engineering technologies promise quantitative understanding of the complex human behavior in its vast individual and contextual heterogeneity, but enable us supporting behavioral interventions in real, day-to-day life settings. Imagine empowering researchers and clinicians with tools that can support a comprehensive, quantitative understanding of behavior that is reflective of natural life scenarios. Imagine the ability to provide personalized, just in time, contextualized behavioral support for a child in a pre-school setting or an adolescent in a social interaction setting with interfaces that are personalized to the needs of the individual.

highlighted publications
  • Daniel Bone, Chi-Chun Lee, Alexandros Potamianos, Shrikanth Narayanan, "An Investigation of Vocal Arousal Dynamics in Child-Psychologist Interactions using Synchrony Measures and a Conversation-based Model", Proceedings of Interspeech, 2014.
  • Theodora Chaspari, Matthew Goodwin, Oliver Wilder-Smith, Amanda Gulsrud, Charlotte Mucchetti, Connie Kasari, Shrikanth Narayanan, "A Non-Homogeneous Poisson Process Model of Skin Conductance Responses Integrated with Observed Regulatory Behaviors for Autism Intervention", Proceedings of IEEE International Conference on Audio, Speech and Signal Processing (ICASSP), 2014.
  • Angeliki Metallinou, Ruth Grossman, Shrikanth S. Narayanan, "Quantifying Atypicality In Affective Facial Expressions Of Children With Autism Spectrum Disorders", Proceedings of the IEEE International Conference on Multimedia & Expo (ICME), 2013.
  • Daniel Bone, Matthew Black, Chi-Chun Lee, Marian Williams, Pat Levitt, Sungbok Lee, Shrikanth S. Narayanan, "The Psychologist as an Interlocutor in Autism Spectrum Disorder Assessment: Insights from a Study of Spontaneous Prosody", Journal of Speech, Language, and Hearing Research, 2013.
  • Rahul Gupta, Chi-Chun Lee, Daniel Bone, Agata Rozga, Sungbok Lee, Shrikanth S. Narayanan, "Acoustical analysis of engagement behavior in children", Proceedings of Workshop on Child, Computer and Interaction (WOCCI 2012), 2012.

Addiction BSP

Observational coding of audio-video recordings of patient-therapist interactions is a key part of understanding the process and treatment adherence of psychotherapy interventions. In this collaborative project with researchers at U Washington and at UC Irvine, we plan to create and use advances in computational speech and spoken language processing to offer tools and methods for automating aspects of observational coding.
The specific theoretical framework for this project is based on Motivational Interviewing (MI). The focus of this BSP/BI effort is based on automating behavioral coding related to MI, namely the Motivational Interviewing Skills Code (MISC) and its derivative, the Motivational Interviewing Treatment Integrity (MITI) coding.

highlighted publications
  • Bo Xiao, Dogan Can, James Gibson, Zac E. Imel, David C. Atkins, Panayiotis Georgiou, Shrikanth S. Narayanan, "Behavioral Coding of Therapist Language in Addiction Counseling Using Recurrent Neural Networks", Proceedings of Interspeech, 2016.
  • James Gibson, Dogan Can, Bo Xiao, Zac E. Imel, David C. Atkins, Panayiotis Georgiou, Shrikanth S. Narayanan, "A Deep Learning Approach to Modeling Empathy in Addiction Counseling", Proceedings of Interspeech, 2016.
  • Bo Xiao, Zac E. Imel, Panayiotis Georgiou, David Atkins, Shrikanth S. Narayanan, ""Rate my therapist": Automated detection of empathy in drug and alcohol counseling via speech and language processing", PLoS ONE, 2015.
  • Dogan Can, David Atkins, Shrikanth S. Narayanan, "A Dialog Act Tagging Approach to Behavioral Coding: A Case Study of Addiction Counseling Conversations", Proceedings of Interspeech, 2015.
  • Bo Xiao, Zac E. Imel, David Atkins, Panayiotis Georgiou, Shrikanth S. Narayanan, "Analyzing Speech Rate Entrainment and Its Relation to Therapist Empathy in Drug Addiction Counseling", Proceedings of Interspeech, 2015.

Depression Disorders & PTSD

The goal is to contribute to the development of an integrated toolkit for text and voice analysis and informatics for social network based psychological health care. The specific goals center on using advanced speech and language processing in conjunction with statistical machine learning for detecting and characterizing psychological distress.

highlighted publications
  • Rahul Gupta, Nikolaos Malandrakis, Bo Xiao, Tanaya Guha, Maarten Van Segbroeck, Matthew P Black, Alexandros Potamianos, Shrikanth S Narayanan, "Multimodal prediction of affective dimensions and depression in human-computer interactions".

Obesity BSP

The technical goal is to use a multimodal approach to sense and model, and influence, physical activity and related behavior in real life settings. Our approach relies on a mobile body computing framework that exploits a variety of signal measurements including physiological cues (e.g., ECG, GSR etc), movement (accelerometry), and contextual information (GPS, audio/video of environment).

highlighted publications
  • Urbashi Mitra, Adar Emken, Sangwon Lee, Ming Li, Viktor Rozgic, Gautam Thatte, Harsh Vatsangam, Daphney Zois, Murali Annavaram, Shrikanth S. Narayanan, Donna Spruijt-Metz, Gaurav Sukhatme, "KNOWME: a Case Study in Wireless Body Area Sensor Network Design", IEEE Communications Magazine, vol. 50, no. 5, pp. 116-125, 2012.
  • Adar Emken, Ming Li, Gautam Thatte, Sangwon Lee, Murali Annavaram, Urbashi Mitra, Shrikanth S. Narayanan, Donna Spruijt-Metz, "Recognition of Physical Activities in Overweight Hispanic Youth Using KNOWME Networks", Journal of Physical Activity & Health, vol. 9, no. 3, pp. 432-441, 2012.
  • Gautam Thatte, Ming Li, Sangwon Lee, B Adar Emken, Murali Annavaram, Shrikanth Narayanan, Donna Spruijt-Metz, Urbashi Mitra, "Optimal time-resource allocation for energy-efficient physical activity detection", Signal Processing, IEEE Transactions on, IEEE, vol. 59, no. 4, pp. 1843-1857, 2011.
  • Ming Li, Viktor Rozgic, Gautam Thatte, Sangwon Lee, BA Emken, Murali Annavaram, Urbashi Mitra, Donna Spruijt-Metz, Shrikanth Narayanan, "Multimodal physical activity recognition by fusing temporal and cepstral information", Neural Systems and Rehabilitation Engineering, IEEE Transactions on, IEEE, vol. 18, no. 4, pp. 369-380, 2010.