Unsupervised discovery of characters in animated movies
Projects | | Links: code | dataset | poper
Our proposed system can automatically extract character exemplars from an animated movie..
What are character dictionaries for movies?
Automatic movie or multimedia content analysis requires audio-video diarization (i.e., "who" appeared? and "when"). For video diarization in particular, where the characters that appear on screen are not known, a key step is to provide a list of characters that form the who appeared component of the system. We refer to such a list of characters specific to each movie character dictionary.Why is it hard to get character dictionaries for animation movies?
Human face detection is the crux of character labeling methods for live action TV. Since human-characters can be uniquely identified by their faces, this method performs adequately well. But, such methods developed for human faces do not work for the digital animation genre. Animated characters, though mostly anthropomorphic (having human characteristics) are not always human-like in appearance. They can be fictional animals, inanimate objects or abstract in design.Our proposed system for animation movie character dictionaries:
We propose an unsupervised approach which requires no prior information about the characters in a movie.
- First, we use a deep neural network based object detector that is trained on natural photographs to identify a set of initial character candidates.
- Second, these candidates are further pruned using saliency constraints and object tracking.
- A character dictionary per movie is then generated from exemplars obtained by clustering these candidates.
Schematic diagram of the proposed system for discovery of characters in animation movies
We were able to identify both anthropomorphic and non-anthropomorphic characters in a dataset of forty-six animation movies with varying composition and character design.
See a few results below:
(Click on the images to enlarge them)
| Cars 2 (2011) | Free Birds (2014) |
|---|---|
| Frozen (2013) |
|---|
| Evaluation of the proposed system on forty-six movies |
|---|
SAIL Animation Movie Character Database (SAIL AMCD)
Results from our system and movie casts used for evaluation
Instructions to download the data:
First, fill out this Google form with the required information to receive the password.
Then, use the password to download the data at this link for the outputs and annotations used in the system evaluation of the paper
The database consists of the following directories (Please see README for more details):
