Investigating and analyzing music’s place in film, television, and advertisement

Project Summary

Music is an integral part of enjoyment and consumption of media. This project looks at how music is used in film, television, and advertisements. By analyzing how music is used in these contexts, we can better understand how music is used as directive, driving further inquiry into human affective experiences and music’s broad-reaching applications in multi-modal contexts.

Music’s Use in Film

Music plays a pivotal role in film and media. Music played during ads, movies, and television shows gives narrative cues and suggestions to the audience. Different styles of music can also convey different affect, and movies of a particular genre can contain “cross-genre” scenes

Music and Visual Features

By combining the visual information contained in media (such as brightness and contrast) with the audio information contained in the same media (like musical features and dialogue), we can better capture what this media is trying to convey and impart on its audience.

Recent News

06/28/2021 – Our paper on music genre classification was accepted for presentation at Content Based Multimedia Indexing (CBMI)! The paper’s title, “Loss Function Approaches for Multi-label Music Tagging” is available starting today.

03/31/2021 – Our paper on movie genre classification using music in movies was accepted for publication at PLOSOne! The paper’s title, “A computational lens into how music characterizes genre in film,” is now available online.

03/18/2021 – We await to hear word on our work involving movie genre classification using musical signatures. If accepted, our work will appear in PLOSOne journal.

02/01/2021 – We await to hear word on our work involving music genre and mood classification. If accepted, our work will appear in Content Based Multimedia Indexing (CBMI).

11/25/2020 – Our team won an international music tagging challenge called MediaEval, where we classified the musical genre and mood of 55,000 songs better than competitors from all over the world. It is a great honor to have our work be recognized! The code that we used to train and run our best performing model is here.

Selected Publications

• Knox, D., Greer, T., Ma, B., Kuo, E., Somandepalli, K. & Narayanan, S. (June, 2021) “Loss Function Approaches for Multi-label Music Tagging.” Content Based Multimedia Indexing.
• Ma, B., Greer, T., Knox, D., & Narayanan, S. (March, 2021) “A computational lens into how music characterizes genre in film.” PLOSOne Journal.
• Knox, D., Greer, T., Ma, B., Kuo, E., & Narayanan, S. (November, 2020) “MediaEval 2020 Emotion and Theme Recognition in Music Task:Loss Function Approaches for Multi-label Music Tagging.” MediaEval 2020 Emotion and Theme Recognition in Music (Competition Winner).
• Greer, T., Mundnich, K., Sachs, M., & Narayanan, S. (2020, May). “The Role of Annotation Fusion Methods in the Study of Human-Reported Emotion Experience During Music Listening.” ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 776-780). IEEE.
• Greer, T., Ma, B., Sachs, M., Habibi, A., and Narayanan, S. (2019). “A Multimodal View into Music’s Effect on Human Neural, Physiological, and Emotional Experience.” In Proceedings of the 27th ACM International Conference on Multimedia (MM ’19). Association for Computing Machinery, New York, NY, USA, 167–175.

Highlights from Studying Music in Media

Best-performing prediction model’s f1-score: .65

It is possible to use the musical cues of a movie to accurately predict a movie’s genre(s)

Tonal Features

Were helpful in predicting movies with higher accuracy, especially horror films

6/7

Film genres could be better predicted by using both musical features and visual cues