Introduction:

This project addresses the problem of natural child-machine communication specifically targeting preschool age (aged 2.5-4) and early elementary school children (aged 4-6). While the state of art in sensory interfaces to information (e.g., machine speech recognition & synthesis), is still not perfect for the adult population, the task of enabling and measuring the effect of sensory technologies for children poses even greater challenges. Children are a crucial segment of the society that will benefit from advances in multimedia information and communication technologies that educate and entertain. Three major topics addressed by this project are:

• Acoustic characterization of young children's speech
  • Quantify variation along dimensions of age and gender
  • Determine effects of age and signal bandwidth on speech signal features.
• Linguistic characterization of young children's conversational speech
  • analyze lexical/non-lexical discourse characteristics, speech acts, pacing strategies, and repairs.
  • Determine effects of age.
• Multimodality in Spoken Language
  • investigate the type of relationship that exists between different communicative modalities(speech versus gesture) that children employ in child-machine interaction.
  • determine effects of age
  
  
  
  

  Experimental Design and Data Collection:

  The major components of the proposed research were highlighted in figure above. The project is data driven and relies on spoken interaction data of children interacting with a computer agent. Toward that goal, we have initiated a Wizard of Oz experiment, where a series of age-appropriate cognitive challenges are implemented in an interactive game with a spoken dialog agent. [Further details]
  The experimental set-up has been constructed and equipped to accommodate this research, with test room observable from a laboratory through a one-way mirror. High quality audio recordings and video recordings (two cameras, one focus on the child's faced from the front and the other capturing the child and the computer from the side).
  
  

  
  
  

  Data Transcription and Annotation:

  The speech data from each session will be organized, transcribed and annotated, by a native English speaker using modified version of CHILDES format. A second researcher will independently verify transcriptions. The rich transcription includes orthography, any phonetic modifications of lexical items, details on non-lexical items (disfluencies, speech errors etc.), and speech act information. Speech Acts coding scheme is adapted from TRAINS and DAMSL coding system [Speech Act Coding System]
  A second stage of transcription will involve adding gestural information from video data, adapting existing tools and annotation techniques for synchronous analysis of our multimodal speech and video data. To enable an integrated analysis, a multi track annotation board is constructed using the ANVIL tool kit [M. Kipp, Eurospeech, 1367-1370 (2001)]. Along with speech transcriptions and acoustic analysis, non-lexical and discourse characteristics, and child's gesture (facial expressions, body movements, hand/head movements) are annotated in a synchronized multi layer system. [Annotation Details]
  
  

  
  
  
  

  Project Members:

  Faculty	Graduate Students
  S. Narayanan Associate Professor Director, Speech Analysis & Interpretation Laboratory (SAIL) Department of Electrical Engineering-Systems	S. Yildirim PhD Candidate Electrical Engineering-Systems,USC
  E. Andersen Professor Linguistics, Psychology, Neuroscience	S. Montanari PhD Student Linguistics, USC

  Publications:

  1. S. Yildirim, C. M. Lee, S. Lee, A.Potamianos, and S. Narayanan. "Detecting politeness and frustration state of a child in a conversational computer game", In Proc. of EUROSPEECH, Interspeech, Lisbon, Portugal, 2005. [PDF]
  2. S. Montanari, S. Yildirim, E. Andersen, and S. Narayanan, "Reference marking in children’s computer-directed speech: An integrated analysis of discourse and gesture", In Proceedings of ICSLP, Jeju, Korea, October 2004.[PDF]
  3. S. Montanari, S. Yildirim, S. Khurana, M. Landes, L. Lawyer, E. Andersen, and S. Narayanan. "Analyzing the interplay between spoken language and gestural cues in conversational child-machine interactions in pre/early literate age group", In Proceedings of InStil, Venice, Italy, July 2004.[PDF]
  4. S. Yildirim, S. Montanari, E. Andersen, and S. Narayanan. "An integrated analysis of speech and gestural characteristics in conversational child-computer interactions", J. Acoust. Soc. Am., 114:2394, 2003.
  5. S. Yildirim and S. Narayanan, "Acoustic analysis of preschool children's speech", Proc. ICPhS (Barcelona, Spain), August 2003. [PDF]
  6. S. Yildirim and S. Narayanan, ``An information-theoretic analysis of developmental changes in speech", Proc. ICASSP (Hong Kong), April 2003. [PDF]
  7. A. Potamianos, S. Narayanan and G. Riccardi, ``Adapative categorical understanding for spoken dialog systems,'' IEEE Trans. Speech and Audio Processing, 13(3):321-329, 2005.[PDF]
  8. A. Potamianos and S. Narayanan, "Robust recognition of children's speech ," IEEE Trans. Speech and Audio Processing, 11:603-616, Nov. 2003. [PDF]
  9. S. Narayanan and A. Potamianos, "Creating conversational interfaces for children," IEEE Trans. Speech and Audio Processing, vol. 10, no. 2, pp. 65-78, 2002. [PDF]
  10. S. Arunachalam, D. Gould, E. Andersen, D. Byrd and S. Narayanan, ``Politeness and frustration language in child-machine interactions", in Proc. Eurospeech, (Aalborg, Denmark), pp. 2675-2678, 2001. [PDF]
  11. S. Lee, A. Potamianos, and S. Narayanan, ``Acoustics of children's speech: Developmental changes of temporal and spectral parameters,'' J. Acoust. Soc. Am., vol. 105, pp. 1455-1468, Mar. 1999 (Selected Research Article). [PDF]
  12. A. Potamianos and S. Narayanan, ``Spoken dialog systems for children,'' in ICASSP 98, vol. 1, (Seattle, WA), pp. 197-200, May 1998.