SEAR: Semantically-grounded Audio Representations

Rajat Hebbar
Digbalay Bose
Shrikanth Narayanan

Signal Analysis and Interpretation Laboratory (SAIL), University of Southern California

31st ACM International Conference on Multimedia

[Paper]
[Github]

Fig: Relationship between audio and visual elements in movies and how (a) visual captions can help tease out acoustically-relevant factors while (b) being invariant to acoustically-irrelevant factors. (c) Captions are also a concise yet efficient way to aggregate information across fast-paced transition in visual frames as often seen in movies.

Abstract

Audio supports visual story-telling in movies through the use of different sounds. These sounds are often tied to different visual ele- ments, including foreground entities, the interactions between them as well as background context. Visual captions provide a condensed view of an image, providing a natural language description of enti- ties and the relationships between them. In this work, we utilize visual captions to semantically ground audio representations in a self-supervised setup. We leverage state-of-the-art vision-language models to augment movie datasets with visual captions at scale to the order of 9.6M captions to learn audio representations from over 2500 hours of movie data. We evaluate the utility of the learned rep- resentations and show state-of-the art performance on two movie understanding tasks, genre and speaking-style classification, out- performing video based methods and audio baselines. We also show that the learned model can be transferred in a zero-shot man- ner through application in both movie understanding tasks and general action recognition

SEAR

Introduction

Audio profile in movies strongly linked to visual entities -- foreground objects, background location and actions. Visual captions can help tease out relevant visual elements and ignore invariant features (e.g., color), and can be extracted automatically. Cross-modal supervision can be used to learn robust audio representations
Fig: Schematic overview of SEAR. Audio Encoder is learned during pretraining phase and kept frozen for downstream movie understanding tasks

Pretraining Details

  • Condensed Movies + Full-Length movies
  • 10s audio segments used as input
  • BLIP [1] model from LAVIS library used for visual captioning at 1fps (T=10)
  • CLIP-text encoder [2] used for text features
  • Audio-encoder initialized with SSAST [3] (masked reconstruction) weights


Fig: Wordcloud of the movie captions in pretraining dataset

Results: LVU Movie Understanding Benchmark [4]

Long-form Video Understanding (LVU) provides a benchmark for 9 different tasks to study modeling techniques applied to long-form data such as movie clips. In this work, we evaluate our method on 3 LVU tasks: speaker-style, genre and scene classification. Here, the first task is primarily audio driven, the second is generally multimodal, i.e, both audio-visual modalities provide signal for genre classification, while scene classification is primarily visual. We hope to evaluate the effect of cross-modal distillation by evaluating across these tasks.


Fig: Details of LVU benchmark tasks

Table: Results of zero-shot/linear-probe classification on LVU


SEAR is the first audio-only approach to long-form content understanding (to the best of our knowledge). We present two previous state-of-the-art (SotA) video baselines and also evaluate one self-supervised (SSAST) and one fully-supervised (AST) models as audio baselines. In the above table, we see that SEAR clearly outperforms SotA in two tasks: speaking-style and genre classification. It also performs significantly outperforms both audio baselines.
We also present zero-shot classification results.

Zero-shot: A qualitative analysis

Fig: Top samples detected for zero-shot text queries

References

[1] Li et al., "BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language", ICML, 2022
[2] Radford et al., "Learning Transferable Visual Model From Natural Language Supervision", ICML, 2021
[3] Gong et al., "SSAST: Self-Supervised Audio Spectrogram Transformer", AAAI, 2022
[4] Wu et al., "Towards Long-Form Video Understanding", CVPR, 2021

Acknowledgements

This template was originally made by Phillip Isola and Richard Zhang for a colorful ECCV project; the code can be found here.