MM-AU: Towards multimodal understanding of advertisement videos

Digbalay Bose1
Rajat Hebbar1
Tiantian Feng1

Krishna Somandepalli2
Anfeng Xu1
Shrikanth Narayanan1

1Signal Analysis and Interpretation Laboratory (SAIL), University of Southern California
2Google Research

31st ACM International Conference on Multimedia

[Paper]
[Supplementary]

Figure 1: Schematic diagram showing illustrative examples of various tasks in the MM-AU (Multi-modal ads understanding) dataset. Multimodal understanding of ads along the lines of (a) Topic categorization (18 classes) (b) Tone transition (c) Social message detection, i.e. Absence/Presence of social message.

Abstract

Advertisement videos (ads) play an integral part in the domain of Internet e-commerce, as they amplify the reach of particular products to a broad audience or can serve as a medium to raise awareness about specific issues through concise narrative structures. The narrative structures of advertisements involve several elements like reasoning about the broad content (topic and the underlying message) and examining fine-grained details involving the transition of perceived tone due to the sequence of events and interaction among characters. In this work, to facilitate the understanding of advertisements along the three dimensions of topic categorization, perceived tone transition, and social message detection, we introduce a multimodal multilingual benchmark called MM-AU comprised of 8.4 K videos (147hrs) curated from multiple web-based sources. We explore multiple zero-shot reasoning baselines through the application of large language models on the ads transcripts. Further, we demonstrate that leveraging signals from multiple modalities, including audio, video, and text, in multimodal transformer-based supervised models leads to improved performance compared to unimodal approaches.

MM-AU dataset

Data Sources

As a primary source, we consider Ads-of-the-world (AOW) video hosting website since it contains a richly-curated catalog of ads in various formats like film, print, digital, and video spanning across multiple countries. As auxiliary sources, we consider additional videos from the Cannes Lion Film Festival archive and Video-Ads dataset. We filter the videos based on unique video ids associated with their public links to ensure no duplicates across three sources.
Figure 2: Share of different ad sources in MM-AU dataset. Ads of the world [1] (6304 videos), Cannes Lion [2] (1135), Video-Ads [3] dataset (960).

Tone transition and social message annotation

Figure 3: Outline of the tone transition annotation framework.
Figure 4: Outline of the social message annotation framework.

For the detection tasks of tone transition and social message, we use Amazon Mechanical Turk to obtain responses from a pool of 36 human annotators. For selecting a pool of workers with the requisite expertise, we hosted an initial pilot study where the workers are instructed to mark the tone transition labels and presence/absence of social message in the given set of videos. Further, in the final annotation phase, three annotators independently annotate each sample for the tone-transition and social message detection tasks.

Tone transition: The annotators are instructed to mark the perceived tone labels associated with the start, middle and ending segments of the advertisement videos. To reduce the burden associated with the task, no instructions are provided to mark the timestamps associated with the respective segments. An outline with tone definitions [4] is shown in Figure 3.

Social message: For social message detection, the annotators are instructed to check for the absence/presence of social messages in the given video. To simplify the annotation process, we ask the annotators to mark Yes/No for indicating the presence/absence of social messages in the videos instead of marking the exact categories An outline with definition of social message is shown in Figure 4.

Topic annotation

We annotate topic categories using the existing taxonomies from Ads-of-the-world (AOW) [1], Cannes Lions Film Festival (CC) [2], and Video-Ads (VA) [3] datasets. We extract the available tags associated with 6304 videos in Ads-of-the-world (AOW) and retain the top 40 tags based on frequency. Then we manually merge the filtered topic tags from AOW with similar labels in Cannes-coding (CC) and Video-Ads (VA) coding schemes. Some examples of merged labels from different sources are listed as follows with the final parent topic category: Final topic taxonomy consists of the following:

Dataset statistics
Table 1: Data statistics of MM-AU dataset. #social msg labels: total number of labels

Figure 5: Distribution of 18 topics in MM-AU dataset.

Figure 6: Distribution of majority tone transition labels in MM-AU dataset.
Figure 7: Distribution of majority social message absence/presence in MM-AU dataset.




(a) Start Segment

(b) Middle segment

(b) Ending segment

Figure 8: Distribution of majority perceived tone labels (among 3 annotators) across (a) start, (b) middle, and (c) ending segments in MM-AU dataset.
Figure 9: Distribution of topics wrt videos having social message (739 videos).
Figure 10: Distribution of tone transition wrt videos having social message (739 videos).

Multimodal fusion - Two stage approach

Figure 11: Outline of proposed two-stage multimodal fusion approach for audio-visual (AV) and text-visual (TV) transformers. Stage 1: Full finetuning. Stage 2: Weights are completely frozen. The text, visual and audio encoders are kept completely frozen. in MM-AU dataset.


Experiments

Experimental setup

For training, validation, and testing purposes, we consider a split of 5877 (70%), 830 (10%), and 1692 (20%) videos. The modality specfic setups are listed as follows:

Visual: Shot extraction using PySceneDetect [5] followed by frame wise feature extraction using CLIP [6]'s visual encoder (4 fps) and average pooling.

Audio: Use Audio Spectrogram Transformer [7] pretrained on AudioSet for extracting features at 10-sec intervals with the step-size of 512.

Text: Use multilingual Whisper [8] to extract transcripts and pretrained BERT [9] for text representations.

Language based reasoning

We explore zero-shot results based on application of language models for diverse tasks including topic categorization, social message and tone transition detection. We report the results on 1670 non-empty transcripts. Task specific prompts for GPT-4 [10] are listed as follows:

Social message: An advertisement video has a social message if it provides awareness about any social issue. Example of social issues: gender equality, drug abuse, police brutality, workplace harassment, domestic violence, child labor, environmental damage, homelessness, hate crimes, racial inequality etc. Based on the given text transcript, determine if the advertisement has any social message. Please provide answers in Yes and No

Tone transition: Based on the given text transcript from the advertisement, determine if the advertisement has any transitions in tones. Possible tone labels are: positive, negative, and neutral. Please respond by saying Transition and No transition

Topic categorization: Associate a single topic label with the transcript from the given set: <Topic list>

<Topic list> refers to the condensed list of 18 topic categories curated for MM-AU dataset.
Table 2: Zero shot performance comparison between various LLMs on MM-AU dataset. Tasks: SM: Social message detection, TT: Tone transition, Topic: Topic categorization. $NA^{*}:$ Information not available. F1: Macro-F1. #social msg labels: total number of labels

GPT-4 performs better than other large-language models across all tasks. Further there is a trend towards increased model performance with scaling, except for Alpaca.

Unimodal vs Multimodal

Table 3: Comparative results between different unimodal and multimodal models across different tasks: Social message and Tone transition, Topic categorization. CLIP-S: Shot level features extracted using CLIP. Modality: A: Audio, V: Visual, T: Text. Results are reported as an average of 5 runs with randomly selected seeds.

References

Acknowledgements

This template was originally made by Phillip Isola and Richard Zhang for a colorful ECCV project; the code can be found here.