The overarching goal of this effort is to employ and advance intuitive and mathematically-principled signal representations and engineering models for describing human behavior.

Advances in sensing and wireless systems have enabled gathering data about a person's communicative and social behavior in a variety of controlled and unstructured real life settings and that opens new avenues of analysis and interpretation of such data.

Behavioral signal processing focuses on gathering, analyzing and modeling multimodal behavior signals, both overtly and covertly expressed. This builds upon foundational efforts in speech/language/audio/visual/physiological signal processing. Critically, in addition to processing objectively-specified behavioral content in richer ways (e.g., what someone said and did), behavioral signal processing entails automating a host of subjectively-specified entities such as those related to socio-emotional states of people (e.g., how negative or frustrated a person is; politeness; engagement etc).

Concomitant with the data explosion are opportunities and, challenges, to effectively interpret these data to seek the right information from the ocean of data to suit the specific application. There is hence a critical need for automatically discovering and reconciling the rich patterns in these data for aiding human analysis and decision-making. Behavioral Informatics aims at extracting meaningful information using the multimodal behavioral signal observations that can benefit human, and even automated, decision making. An important aspect of our approach to Behavioral Informatics is in developing methods that rely on synergistic collaboration between human and machine processing of behavioral signals. A key element of this relies on seeking ways of combining domain knowledge and data in advantageous ways.

Threads:

We have a range of activities under this umbrella of Behavioral Signal Processing with projects in Autism, Addiction, Couples therapy, Depression Disorders and PTSD, Obesity, and Language Assessment.

The benefits of quantitative behavior assessment range from speedup and parallel observation capabilities to identifying large scale trends and micro-analysis that humans are as individuals not-ideal to handle. The significance of this research is clear: USA-10mil people receive psychotherapy every year, but the state of the art in psychology hasn't changed for decades.

Despite the diverse projects under Behavioral Signal Processing there are a lot commonalities from the engineering prospective. For instance analysis shares many common features across domains (e.g. spoken and body language use, interaction patterns). However each domain has it's specific requirements stemming both from engineering and domain needs (e.g. domain specific features such as reactivity for family and couple therapy). In simple terms our overall approach is the same:

• Observe what the experts observe
• Learn from, and augment their capabilities
• Support but not supplant!
• Work closely with domain experts.
• Combine human and machine expertise