Moviegoer: Character Details

This is part of a series describing the development of Moviegoer, a multi-disciplinary data science project with the lofty goal of teaching machines how to “watch” movies and interpret emotion and antecedents (behavioral cause/effect).

In Ford v Ferrari (2019), we infer from his face that Matt Damon’s character is white, male, and 39 years old. The actor is actually older, so we may have been fooled by his movie star looks.

With scenes and their details identified, we turn our attention to characters. Tracking characters (along with their screentime, dialogue, and emotional ups-and-downs) throughout the film is a big step toward understanding a film.

One tough task is identifying character names. Though we have clues, like other characters addressing them by name (“here’s your ticket, Anna”), this isn’t quite foolproof enough to name characters with confidence. But to be honest, I often have a hard time knowing characters’ names when I watch films. For now, we’ll just use faces to identify characters, much like I do when I watch a movie.

Since we’ve been able to identifying individual scenes, we can track characters throughout their scene appearances. Each time we see a face onscreen, we turn that face into facial encodings. These facial encodings are clustered together, and each character’s face is usually associated with a few clusters (it’s not perfect). But because we’ve been able to identify individual two-character dialogue scenes, we can link facial clusters together. For example, we know that a character remains in a scene, but the lighting on their face may have changed, affecting their facial encoding.

With each character being represented by groups of facial encoding clusters, we can take examples of these faces and use the DeepFace Python library to get demographic information such as predicated age, race, and gender. See this article’s first image and caption for an example.

Similar to what we did for scene-level details, we can take a look at each characters’ dialogue and try and find important pieces of information. Dialogue attribution isn’t perfect yet, though we can still try.

In Plus One (2019), we can list every time the character of Alice asks a question

Finally, since we can track each character’s emotions throughout the film, we can plot their emotions over the course of the film. We do this by counting up the negative emotions Sad and Angry, and tallying this as a percentage of overall emotions. This is plotted over eight equal-size portions of the film (not counting the credits).

In Plus One (2019), we can track Alice’s negative facial emotions throughout the film

Wanna see more?

• Repository: Moviegoer
• Project Intro: Can a Machine Watch a Movie?
• Previous Story: Scene Details