SAIL:User Modeling & Dialogue Modeling

User Modeling & Spoken Dialog Management

Phd. students: JongHo Shin, Abe Kazemzadeh, Viktor Rozgic

Conversational interfaces hold the promise of providing natural, easy and universal access to information. The research effort of this project targets two specific problems in conversational engineering under a unifying stochastic modeling framework:

Dialog interaction modeling

User modeling in spoken dialog(Human-to-Machine and Human-to-Human with machine mediated).

User models are essential in designing optimal dialog strategies, while discourse state information is essential for user behavior modeling. Conversational participants -- humans or machines -- are modeled as stochastic dynamical systems, interacting with one another over a noisy communication channel.

The approach to modeling user behavior is data-driven with an emphasis on behavior under error conditions and on the inclusion of automatic emotion tracking. An unified statistical framework provides a way for integrating multiple sources of information (e.g., acoustic, lexical, nonverbal and discourse) based on information-theoretic principles.

Current projects

Modeling and optimizing user-centric mixed-initiative spoken dialog systems (NSF-CAREER)

Speech Interface for Haptics (IMSC)

Transonics

Mission Rehearsal Exercise for virtual training simulations (ICT)

* NSF: National Science Foundation, IMSC: Integrated Media Systems Center, ICT: Institute for Creative Technologies.

Selected publication

A. Potamianos, S. Narayanan, and G. Riccardi. Adapative categorical understanding for spoken dialog systems. IEEE Trans. Speech and Audio Processing, 2004. URL: http://sail.usc.edu/publications/PoNaRi_understand_Review.pdf.

Robert Belvin, Win May, Shrikanth Narayanan, Panayiotis Georgiou, and Shadi Ganjavi. Creation of a doctor-patient dialogue corpus using standardized patients. In Proc. LREC, Lisbon, Portugal, 2004.

Narayanan, S. Ananthakrishnan, R. Belvin, E. Ettaile, S. Gandhe, S. Ganjavi, P. G. Georgiou, C. M. Hein, S. Kadambe, K. Knight, D. Marcu, H. E. Neely, N. Srinivasamurthy, D. Traum, and D. Wang. The transonics spoken dialogue translator: An aid for english-persian doctor-patient interviews. In AAAI Fall Symposium, 2004.

Abe Kazemzadeh, Sungbok Lee, and Shrikanth Narayanan. Acoustic correlates of user response to errors in human-computer dialogues. In Proc. IEEE ASRU, St. Thomas, U.S. Virgin Islands, December 2003. URL: http://sail.usc.edu/publications/asru_2003_abe_usc.pdf

Shrikanth Narayanan. Towards modeling user behavior in human-machine interactions: Efect of errors and emotions. In ISLE Tagging for multimodal dialogs Workshop, Edingburgh, UK, December 2002. URL: http://sail.usc.edu/publications/ISLE-shri.pdf.

JongHo Shin, Shri Narayanan, Laurie Gerber, Abe Kazemzadeh and Dani Byrd, "Analysis of user behavior under conditions in spoken dialogs", ICSLP, September 2002. - Presented at International Conference on Spoken Language Processing (ICSLP) 2002 and awarded the second best student paper (ps) .

Chul Min Lee, Shrikanth Narayanan, and Roberto Pieraccini. Classifying emotions in human-machine spoken dialogs. In Proc. of ICME, Lausanne, Switzerland, 2002. URL: http://sail.usc.edu/publications/icme_cml.pdf.

Ying Li, Shrikanth Narayanan, and C.-C. Jay Kuo. Identification of speakers in movie dialogs using audiovisual cues. In Proc. of ICASSP, Orlando, FL, 2002. URL: http://sail.usc.edu/publications/Icassp02_YL.pdf.

M. Walker, J. Aberdeen, J. Boland, E. Bratt, J. Garafolo, L. Hirschman, A. Le, S. Lee, S. Narayanan, K. Papineni, B. Pellom, J. Polifroni, A. Potamianos, P. Prabhu, A. Rudnicky, G. Sanders, S. Seneff, D. Stallard, and S. Whittaker. Darpa communicator dialog travel planning systems: The june 2000 data collection. In Proc. Eurospeech, pages 1371-1374, Aalborg, Denmark, 2001. URL: http://sail.usc.edu/publications/WalkerEtal_euro2001.pdf.

S. Narayanan, G. Di Fabbrizio, C. Kamm, J. Hubbell, B. Buntschuh, P. Ruscitti, and J. Wright. Effects of dialog initiative and multi-modal presentation strategies on large directory information access. In Proc. of the Intnl Conf. Spoken Lang. Processing, pages 636-639, Beijing, China, 2000. URL: http://sail.usc.edu/publications/NaEtal_mvpq_icslp2000.pdf.

E. Levin, S. Narayanan, R. Pieraccini, K. Biatov, E. Bocchieri, G. Di Fabbrizio, W. Eckert, S. Lee, A. Pokrovsky, M. Rahim, P.Ruscitti, and M. Walker. The at&t-darpa communicator mixed-initiative spoken dialog system. In Proc. of the Intnl Conf. Spoken Lang. Processing, pages 122-125, Beijing, China, 2000. URL: http://sail.usc.edu/publications/NaEtal_mvpq_icslp2000.pdf.

M. Rahim, R. Pieraccini, W. Eckert, E. Levin, G. Di Fabbrizio, C. Kamm, and S. Narayanan. A spoken dialog system for conference/workshop services. In Proc. of the Intnl Conf. Spoken Lang.?Processing, pages 736-739, Beijing, China, 2000.

R. Pieraccini, E. Levin, W. Eckert, and S. Narayanan. Spoken dialog systems: From theory to practice. In IEEE ASRU Workshop, Keystone, CO, December 1999.

G. di Fabbrizio, P. Ruscitti, S. Narayanan, and C. Kamm. Extending computer telephony and ip telephony standards for voice-enabled services in a multi-modal user interface environment. In Proceedings of Interactive Dialogue in Multi-modal systems, pages 9-12, Kloster Irsee, Germany, June 1999.

M. Walker, J. Fromer, and S. Narayanan. Learning optimal dialogue strategies: A case study of a spoken dialogue agent for email. In Proc. of ACL/COLING 98, pages 1345-1351, Montreal, Canada, 1998.

A. Potamianos and S. Narayanan. Spoken dialog systems for children. In ICASSP 98, volume 1, pages 197-200, Seattle, WA, May 1998.

C. Kamm, S. Narayanan, D. Dutton, and R. Ritenour. Evaluating spoken dialog systems for telecommunication services. In Proc.?EuroSpeech, volume 4, pages 2203-2206, Rhodes, Greece, September 1997.

C. Lin, S. Narayanan, and R. Ritenour. Database management and analysis for spoken dialog systems: Methodology and tools. In Proc.?EuroSpeech, volume 4, pages 2199-2202, Rhodes, Greece, September 1997.

Annotation scheme for tagging communicator data

- model tags