This is part of a series describing the development of Moviegoer, a multi-disciplinary data science project with the lofty goal of teaching machines how to “watch” movies and interpret emotion and antecedents (behavioral cause/effect).
We’ve improved our scene-boundary detection algorithm, and we’ve been able to detect two-character dialogue scenes throughout the film. With these scenes partitioned, we can apply some of the NLP-based and emotional analysis we’ve been developing the past few weeks. Our goal is to extract plot and character information from the dialogue.
Our algorithm identifies scenes that alternate back-and-forth between two characters speaking to one another. We can identify these from shot clusters, or groups of similar frames. These shots alternate in an A/B/A/B pattern, with A and B each representing a shot of a different character.
For each scene, we can run a variety of analyses to generate a best-guess summary of scene information. We conduct TF-IDF analysis for the scene against the rest of the film, and try and identify important noun groups. We can also detect first-person declarations and second-person addresses, which usually provide lots of plot information. We can also record direct questions and their responses, looking for questions asked by one character and answered by the other character. Sometimes these analyses are relevant, sometimes they aren’t, but this is a good start for retrieving information.
Finally, we can count up the facial emotion for each character. Each time a character is onscreen, we can calculate their emotion. We’ll tally this up for the entire scene and report the most common emotion as their primary emotion. This is the prelude to character tracking across the entire film; measuring a character’s ups and downs is a key goal of this project.
With this automated scene detection and analysis, we’re at a key point in the project — we’re ready to release a prototype. Stay tuned!
Wanna see more?
- Repository: Moviegoer
- Project Intro: Can a Machine Watch a Movie?
- Previous Story: Data Serialization
- Next Story: Character Details
