Sandeep Nallan Chakravarthula, Brian R.W. Baucom, Shrikanth Narayanan, and Panayiotis Georgiou. An analysis of observation length requirements for machine understanding of human behaviors from spoken language. Computer Speech & Language, 66:101162, 2021.

Download

[PDF] 

Abstract

The task of quantifying human behavior by observing interaction cues is an important and useful one across a range of domains in psychological research and practice. Machine learning-based approaches typically perform this task by first estimating behavior based on cues within an observation window, such as a fixed number of words, and then aggregating the behavior over all the windows in that interaction. The length of this window directly impacts the accuracy of estimation by controlling the amount of information being used. The exact link between window length and accuracy, however, has not been well studied, especially in spoken language. In this paper, we investigate this link and present an analysis framework that determines appropriate window lengths for the task of behavior estimation. Our proposed framework utilizes a two-pronged evaluation approach: (a) extrinsic similarity between machine predictions and human expert annotations, and (b) intrinsic consistency between intra-machine and intra-human behavior relations. We apply our analysis to real-life conversations that are annotated for a large and diverse set of behavior codes and examine the relation between the nature of a behavior and how long it should be observed. We find that behaviors describing negative and positive affect can be accurately estimated from short to medium-length expressions whereas behaviors related to problem-solving and dysphoria require much longer observations and are difficult to quantify from language alone. These findings are found to be generally consistent across different behavior modeling approaches.

BibTeX Entry

@article{NALLANCHAKRAVARTHULA2021101162,
 abstract = {The task of quantifying human behavior by observing interaction cues is an important and useful one across a range of domains in psychological research and practice. Machine learning-based approaches typically perform this task by first estimating behavior based on cues within an observation window, such as a fixed number of words, and then aggregating the behavior over all the windows in that interaction. The length of this window directly impacts the accuracy of estimation by controlling the amount of information being used. The exact link between window length and accuracy, however, has not been well studied, especially in spoken language. In this paper, we investigate this link and present an analysis framework that determines appropriate window lengths for the task of behavior estimation. Our proposed framework utilizes a two-pronged evaluation approach: (a) extrinsic similarity between machine predictions and human expert annotations, and (b) intrinsic consistency between intra-machine and intra-human behavior relations. We apply our analysis to real-life conversations that are annotated for a large and diverse set of behavior codes and examine the relation between the nature of a behavior and how long it should be observed. We find that behaviors describing negative and positive affect can be accurately estimated from short to medium-length expressions whereas behaviors related to problem-solving and dysphoria require much longer observations and are difficult to quantify from language alone. These findings are found to be generally consistent across different behavior modeling approaches.},
 author = {Sandeep {Nallan Chakravarthula} and Brian R.W. Baucom and Shrikanth Narayanan and Panayiotis Georgiou},
 doi = {https://doi.org/10.1016/j.csl.2020.101162},
 issn = {0885-2308},
 journal = {Computer Speech & Language},
 keywords = {Human behavior, Spoken language, Observation window length, Machine learning},
 link = {http://sail.usc.edu/publications/files/Sandeep-CSL-2021.pdf},
 pages = {101162},
 title = {An analysis of observation length requirements for machine understanding of human behaviors from spoken language},
 url = {http://www.sciencedirect.com/science/article/pii/S0885230820300954},
 volume = {66},
 year = {2021}
}

Generated by bib2html.pl (written by Patrick Riley ) on Sat Nov 20, 2021 15:31:38