{"id":54,"date":"2020-01-28T06:52:20","date_gmt":"2020-01-28T06:52:20","guid":{"rendered":"https:\/\/sail.usc.edu\/~mica\/wordpress\/?page_id=54"},"modified":"2021-10-07T04:26:27","modified_gmt":"2021-10-07T04:26:27","slug":"publications","status":"publish","type":"page","link":"https:\/\/sail.usc.edu:\/ccmi\/publications\/","title":{"rendered":"Publications"},"content":{"rendered":"\n
<\/a>
active speaker localization<\/a><\/span> advertisements<\/a><\/span> audio-visual event detection<\/a><\/span> Auschwitz<\/a><\/span> autoencoders<\/a><\/span> computational media understanding<\/a><\/span> computational narrative modeling<\/a><\/span> content analysis<\/a><\/span> coreference resolution<\/a><\/span> cross-modal learning<\/a><\/span> face clustering<\/a><\/span> face diarization<\/a><\/span> gender representation<\/a><\/span> gendered analysis<\/a><\/span> Media<\/a><\/span> movie<\/a><\/span> multimedia understanding<\/a><\/span> multimodal<\/a><\/span> multiple instance learning<\/a><\/span> multiview correlation<\/a><\/span> music representations<\/a><\/span> Professions<\/a><\/span> self-supervision<\/a><\/span> semantic role labeling<\/a><\/span> survivor testimonies<\/a><\/span> taxonomy curation<\/a><\/span> triplet loss<\/a><\/span> video character labeling<\/a><\/span> visual scene recognition<\/a><\/span> weakly supervised learning<\/a><\/span> <\/div>
\n

2023<\/h3>

Hebbar, Rajat; Bose, Digbalay; Narayanan, Shrikanth<\/p>

SEAR: Semantically-grounded Audio Representations Conference<\/span> <\/p>

ACM Multimedia , <\/span>2023<\/span>.<\/p>

BibTeX<\/a><\/span> | Tags: <\/span>computational media understanding<\/a>, multimodal<\/a>, self-supervision<\/a><\/p>

@conference{nokey,
\r\ntitle = {SEAR: Semantically-grounded Audio Representations},
\r\nauthor = {Rajat Hebbar and Digbalay Bose and Shrikanth Narayanan},
\r\nyear  = {2023},
\r\ndate = {2023-10-29},
\r\npublisher = {ACM Multimedia },
\r\nkeywords = {computational media understanding, multimodal, self-supervision},
\r\npubstate = {published},
\r\ntppubtype = {conference}
\r\n}
\r\n<\/pre><\/div>
Close<\/a><\/p><\/div><\/div><\/div>
 Bose, Digbalay;  Hebbar, Rajat;  Feng, Tiantian;  Somandepalli, Krishna;  Xu, Anfeng;  Narayanan, Shrikanth<\/p>
MM-AU: Towards Multimodal Understanding of Advertisement Videos Conference<\/span> <\/p>
ACM Multimedia , <\/span>2023<\/span>.<\/p>
BibTeX<\/a><\/span> | Tags: <\/span>advertisements<\/a>, computational media understanding<\/a>, content analysis<\/a>, multimedia understanding<\/a>, multimodal<\/a><\/p>
@conference{nokey,
\r\ntitle = {MM-AU: Towards Multimodal Understanding of Advertisement Videos},
\r\nauthor = {Digbalay Bose and Rajat Hebbar and Tiantian Feng and Krishna Somandepalli and Anfeng Xu and Shrikanth Narayanan },
\r\nyear  = {2023},
\r\ndate = {2023-10-29},
\r\nurldate = {2023-10-29},
\r\npublisher = {ACM Multimedia },
\r\nkeywords = {advertisements, computational media understanding, content analysis, multimedia understanding, multimodal},
\r\npubstate = {published},
\r\ntppubtype = {conference}
\r\n}
\r\n<\/pre><\/div>
Close<\/a><\/p><\/div><\/div><\/div>
 Sharma, Rahul;  Narayanan, Shrikanth<\/p>
Audio-Visual Activity Guided Cross-Modal Identity Association for Active Speaker Detection<\/a> Journal Article<\/span> <\/p>
In: <\/span>IEEE Open Journal of Signal Processing , <\/span>pp. 225-232, <\/span>2023<\/span>.<\/p>
Abstract<\/a><\/span> | Links<\/a><\/span> | BibTeX<\/a><\/span> | Tags: <\/span>active speaker localization<\/a>, computational media understanding<\/a>, cross-modal learning<\/a>, multimedia understanding<\/a><\/p>
@article{nokey,
\r\ntitle = {Audio-Visual Activity Guided Cross-Modal Identity Association for Active Speaker Detection},
\r\nauthor = {Rahul Sharma and Shrikanth Narayanan},
\r\ndoi = {10.1109\/OJSP.2023.3267269},
\r\nyear  = {2023},
\r\ndate = {2023-04-14},
\r\nurldate = {2023-04-14},
\r\njournal = {IEEE Open Journal of Signal Processing },
\r\npages = {225-232},
\r\nabstract = {Active speaker detection in videos addresses associating a source face, visible in the video frames, with the underlying speech in the audio modality. The two primary sources of information to derive such a speech-face relationship are i) visual activity and its interaction with the speech signal and ii) co-occurrences of speakers' identities across modalities in the form of face and speech. The two approaches have their limitations: the audio-visual activity models get confused with other frequently occurring vocal activities, such as laughing and chewing, while the speakers' identity-based methods are limited to videos having enough disambiguating information to establish a speech-face association. Since the two approaches are independent, we investigate their complementary nature in this work. We propose a novel unsupervised framework to guide the speakers' cross-modal identity association with the audio-visual activity for active speaker detection. Through experiments on entertainment media videos from two benchmark datasets\u2013the AVA active speaker (movies) and Visual Person Clustering Dataset (TV shows)\u2013we show that a simple late fusion of the two approaches enhances the active speaker detection performance.},
\r\nkeywords = {active speaker localization, computational media understanding, cross-modal learning, multimedia understanding},
\r\npubstate = {published},
\r\ntppubtype = {article}
\r\n}
\r\n<\/pre><\/div>
Close<\/a><\/p><\/div>
Active speaker detection in videos addresses associating a source face, visible in the video frames, with the underlying speech in the audio modality. The two primary sources of information to derive such a speech-face relationship are i) visual activity and its interaction with the speech signal and ii) co-occurrences of speakers' identities across modalities in the form of face and speech. The two approaches have their limitations: the audio-visual activity models get confused with other frequently occurring vocal activities, such as laughing and chewing, while the speakers' identity-based methods are limited to videos having enough disambiguating information to establish a speech-face association. Since the two approaches are independent, we investigate their complementary nature in this work. We propose a novel unsupervised framework to guide the speakers' cross-modal identity association with the audio-visual activity for active speaker detection. Through experiments on entertainment media videos from two benchmark datasets\u2013the AVA active speaker (movies) and Visual Person Clustering Dataset (TV shows)\u2013we show that a simple late fusion of the two approaches enhances the active speaker detection performance.<\/div>
Close<\/a><\/p><\/div>