SAM Corpus consists of speech segments from over 500 top-grossing hollywood movies spanning 5 years (2014-19). Easily scalable techniques are employed to align movie-audio to subtitles, which are then used to develop Speech Activity Detection models
Speech boundaries in movie audio are demarcated using an automatic alignment tool called gentle. Gentle aligns a text transcript to corresponding audio at phoneme-level, resulting in accurate word-boundaries for successfully aligned words. We use subtitles extracted from OpenSubtitles as text-input to gentle. Around 70% of all words in the transcript are successfully aligned using this method, resulting in over 650K audio-words.
IPUs are short segments of contiguous speech separated by pauses. An IPU is characterized by two threshold values: pause duration (PT), and segment length (ST). After gentle-alignment, consecutive aligned words are grouped together into IPUs which are atleast ST seconds long. Two consecutive words belong to the same IPU if they are no more than PT seconds apart. Construction of IPUs in such a fashion enables “segment” level processing. This is not only computationally efficient, but also captures context which is useful for several downstream tasks.
Data from over 500 top-grossing hollywood movies
225 hours of gentle-aligned speech
Over 275K IPUs of duration 1.25s or longer
CNN architectures for Speech Activity Detection [1]
On the right we see the visualization of the activation maps from the CNN-GAP model architecture. For speech segments, the model attends to lower frequency regions corresponding to the first few harmonics. On the other hand, for noise regions, the model attends to higher frequencies which correspond to non-speech sounds.
Automatic, easily scalable methods are used to extract reliable speech data from large number of movies for training SAD models. Using novel CNN architectures, we show state-of-the-art performance on two benchmark movie datasets
Speech activity detection in movies
[1]. Hebbar, R., Somandepalli, K., & Narayanan, S. “Robust speech activity detection in movie audio: Data resources and experimental evaluation” IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019.
