by alexattia

alexattia /SimpsonRecognition

Detect and recognize The Simpsons characters using Keras and Faster R-CNN

132 Stars 57 Forks Last release: Not found 37 Commits 0 Releases

Available items

No Items, yet!

The developer of this repository has not created any items for sale yet. Need a bug fixed? Help with integration? A different license? Create a request here:


Training a Convolutional Neural Network to recognize The Simpson TV Show characters using Keras (TensorFlow backend).
To have more detailed explanations, see the blog articles on Medium Part 1 and Part 2. If you like it, don't hesitate to recommend it (as for the dataset on Kaggle)

Part 0 : Collecting data

This part is about collecting and labeling Simpson pictures.
Most of the pictures are from Simpson video, analyzed frame by frame.


into a folder with Simpson episodes (.avi format) to analyze them and label frames.
You crop each frame (left part, right part, full-frame, nothing) and then label it.

You can find the dataset on Kaggle

Part 1 : Training with Keras

The first part is training the classification model, it's described in this post.
I aim to have 1000 pictures per class (for 20 classes), unfortunately some characters are not often on screen so I have fewer pictures for those characters. As you can see on the Jupyter notebook, I benchmark two models : 4 and 6 convolutional layers neural networks. Because of the small number of pictures (approx. 1k pictures per class), I use data augmentation.
Currently, I have 96% of accuracy (F1-Score) for 18 classes.

Part 2 : Faster R-CNN

The second part is described in this Medium post.
The second part is about upgrading the deep learning model to detect and recognize characters. I have to annotate data to get bounding boxes for characters for each picture in order to train a new model : Faster R-CNN (which is based on a Region Proposal Network). As usual, the annotation text file with the bounding boxes coordinates will be released soon.
The implementation on this network on Keras is from here by Yann Henon. I edited the code : remove parts which are useless for my purpose.

Files description

    : tools functions for notebooks + script to name characters from frames from .avi videos
    : point with mouse clicks to save bounding box coordinates on annotations text file (from already labeled pictures)
    : training simple convnet
  4. -p annotation.txt
    : training Faster R-CNN with data from the annotation text file
  5. -p path/test_data/
    : testing Faster R-CNN

Lisa picture

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.