Automatic movie or multimedia content analysis requires audio-video diarization (i.e., “who” appeared? and “when”). For video diarization in particular, where the characters that appear on screen are not known, a key step is to provide a list of characters that form the who appeared component of the system. We refer to such a list of characters specific to each movie character dictionary. In this work we present an approach to identify them.

Table of contents

  1. Character dictionaries for movies
  2. Proposed system
  3. Results
    i. Example outputs
    ii. System evaluation
  4. Animation Movie Character Database

What are character dictionaries for movies? Automatic movie or multimedia content analysis requires audio-video diarization (i.e., “who” appeared? and “when”). For video diarization in particular, where the characters that appear on screen are not known, a key step is to provide a list of characters that form the who appeared component of the system. We refer to such a list of characters specific to each movie character dictionary.


Why is it hard to obtain character dictionaries for animation movies?
Human face detection is the crux of character labeling methods for live action TV. Since human-characters can be uniquely identified by their faces, this method performs adequately well. But, such methods developed for human faces do not work for the digital animation genre. Animated characters, though mostly anthropomorphic (having human characteristics) are not always human-like in appearance. They can be fictional animals, inanimate objects or abstract in design.


Our proposed system for animation movie character dictionaries alt text

Schematic diagram of the proposed system for discovery of characters in animation movies

We propose an unsupervised approach which requires no prior information about the characters in a movie. We first use a deep neural network based object detector that is trained on natural photographs to identify a set of initial character candidates. These candidates are further pruned using saliency constraints and object tracking. A character dictionary per movie is then generated from exemplars obtained by clustering these candidates. we were able to identify both anthropomorphic and non-anthropomorphic characters in a dataset of forty-six animation movies with varying composition and character design.


Results

Example outputs of animation movie character dictionaries

(Click on the images to enlarge them)
alt text alt text alt text


Evaluation of the proposed system on forty-six movies alt text


SAIL Animation Movie Character Database (SAIL AMCD)

Results from our system and movie casts used for evaluation
Instructions to download the data:
First, fill out this Google form with the required information to receive the password.
Then, use the password to download the data at this link for the outputs and annotations used in the system evaluation of the paper

The database consists of the following directories (Please see README for more details):
alt text

  1. A list of major characters in an animation movie (CAST)
  2. All detected objects in the first step of our model (all_detected_objects)
  3. Our system outputs, labeled (character_dictionaries_annotated)
  4. Mechanical Turk annotations used for our system evaluation (mturk_annotation)
  5. All objects detected in movie frames (bounding boxes, confidence and other info) in the form of json files (multibox_outputs)
  6. Outputs from our local tracking algorithm with objects initialized from the previous step (local_tracking_outputs)