Detect and recognize The Simpsons characters using Keras and Faster R-CNN
Training a Convolutional Neural Network to recognize The Simpson TV Show characters using Keras (TensorFlow backend).
To have more detailed explanations, see the blog articles on Medium Part 1 and Part 2. If you like it, don't hesitate to recommend it (as for the dataset on Kaggle)
This part is about collecting and labeling Simpson pictures.
Most of the pictures are from Simpson video, analyzed frame by frame.
python3 label_data.pyinto a folder with Simpson episodes (.avi format) to analyze them and label frames.
You can find the dataset on Kaggle
The first part is training the classification model, it's described in this post.
I aim to have 1000 pictures per class (for 20 classes), unfortunately some characters are not often on screen so I have fewer pictures for those characters. As you can see on the Jupyter notebook, I benchmark two models : 4 and 6 convolutional layers neural networks. Because of the small number of pictures (approx. 1k pictures per class), I use data augmentation.
Currently, I have 96% of accuracy (F1-Score) for 18 classes.
The second part is described in this Medium post.
The second part is about upgrading the deep learning model to detect and recognize characters. I have to annotate data to get bounding boxes for characters for each picture in order to train a new model : Faster R-CNN (which is based on a Region Proposal Network). As usual, the annotation text file with the bounding boxes coordinates will be released soon.
The implementation on this network on Keras is from here by Yann Henon. I edited the code : remove parts which are useless for my purpose.
label_data.py: tools functions for notebooks + script to name characters from frames from .avi videos
label_pointer.py: point with mouse clicks to save bounding box coordinates on annotations text file (from already labeled pictures)
train.py: training simple convnet
train_frcnn.py -p annotation.txt: training Faster R-CNN with data from the annotation text file
test_frcnn.py -p path/test_data/: testing Faster R-CNN