{"id":69,"date":"2019-08-28T07:21:40","date_gmt":"2019-08-28T07:21:40","guid":{"rendered":"https:\/\/sail.usc.edu\/~mica\/wordpress\/?p=69"},"modified":"2020-08-27T11:40:37","modified_gmt":"2020-08-27T11:40:37","slug":"project-sam","status":"publish","type":"post","link":"https:\/\/sail.usc.edu:\/ccmi\/project-sam\/","title":{"rendered":"Subtitle Aligned Movie (SAM) Corpus"},"content":{"rendered":"\n
\n
\n
\n

Project Description<\/h2>\n

SAM Corpus consists of speech segments from over 500 top-grossing hollywood movies spanning 5 years (2014-19). Easily scalable techniques are employed to align movie-audio to subtitles, which are then used to develop Speech Activity Detection models<\/p> <\/div>\n <\/div>\n <\/div>\n<\/section>\n\n\n\n\n

\n
\n
\n

Gentle alignment<\/h2>\n <\/header>\n
\n
\n \"\"\n <\/div>\n
\n

Speech boundaries in movie audio are demarcated using an automatic alignment tool called gentle<\/a>. Gentle aligns a text transcript to corresponding audio at phoneme-level, resulting in accurate word-boundaries for successfully aligned words. We use subtitles extracted from OpenSubtitles<\/a> as text-input to gentle. Around 70% of all words in the transcript are successfully aligned using this method, resulting in over 650K<\/b> audio-words. <\/p>\n <\/div>\n <\/div>\n <\/div>\n<\/section>\n\n\n\n

\n
\n
\n

Inter-Pausal Units (IPUs)<\/h2>\n <\/header>\n
\n
\n \"\"\n <\/div>\n
\n

IPUs are short segments of contiguous speech separated by pauses. An IPU is characterized by two threshold values: pause duration <\/b>(PT<\/sub>), and segment length<\/b> (ST<\/sub>). After gentle-alignment, consecutive aligned words are grouped together into IPUs which are atleast ST<\/sub> seconds long. Two consecutive words belong to the same IPU if they are no more than PT<\/sub> seconds apart. Construction of IPUs in such a fashion enables “segment” level processing. This is not only computationally efficient, but also captures context which is useful for several downstream tasks. <\/p>\n <\/div>\n <\/div>\n <\/div>\n<\/section>\n\n\n\n

\n
\n
\n

Dataset stats <\/h2>\n <\/header>\n
\n
\n

500+<\/h2>\n

Movies<\/h3>\n

Data from over 500 top-grossing hollywood movies<\/p>\n <\/div>\n

\n

225<\/h2>\n

Hours<\/h3>\n

225 hours of gentle-aligned speech<\/p>\n <\/div>\n

\n

275K+<\/h2>\n

IPUs<\/h3>\n

Over 275K IPUs of duration 1.25s or longer<\/p>\n <\/div>\n <\/div>\n <\/div>\n<\/section>\n\n\n\n

\n
\n
\n

Network Architecture<\/h2>\n

\"\"\n
CNN architectures for Speech Activity Detection [1]<\/p>\n <\/div>\n <\/div>\n<\/section>\n\n\n\n

\n
\n
\n

Visualization of CNN model<\/h2>\n <\/header>\n
\n
\n \"\"\n <\/div>\n
\n

On the right we see the visualization of the activation maps from the CNN-GAP model architecture. For speech segments, the model attends to lower frequency regions corresponding to the first few harmonics. On the other hand, for noise regions, the model attends to higher frequencies which correspond to non-speech sounds. <\/p>\n <\/div>\n <\/div>\n <\/div>\n<\/section>\n\n\n

\n
\n
\n

Results<\/h2>\n <\/header>\n
\n
\n