fake-face-detection

some collected paper and personal notes relevant to Fake Face Detetection

Challenge

• [Facebook] Deepfake Detection Challenge
  • unofficial github repo

Study

1. [arXiv 2019] Deep Learning for Deepfakes Creation and Detection
2. [ACM SIGSAC 2019] Poster: Towards Robust Open-World Detection of Deepfakes
3. [arXiv 2020] DeepFakes and Beyond: A Survey of Face Manipulation and Fake Detection
4. [arXiv 2019] Zooming into Face Forensics: A Pixel-level Analysis
5. [arXiv 2020] DeepFake Detection: Current Challenges and Next Steps
6. [arXiv 2020] DeepFakes Evolution: Analysis of Facial Regionsand Fake Detection Performance
7. [arXiv 2020] The Creation and Detection of Deepfakes: A Survey
8. [arXiv 2020] Preliminary Forensics Analysis of DeepFake Images

I. Dataset

1. FaceForensics++ Dataset

• benchmark
• paper: [ICCV 2019] FaceForensics++: Learning to Detect Manipulated Facial Images
  • 977 downloaded videos from youtube, 1000 original extracted sequences and its manipulated version
  • generated based on Deep-Fakes, Face2Face, FaceSwap and NeuralTextures
    

1. [Google] DeepFakeDetection Dataset

• homepage
  • over 363 original sequences from 28 paid actors in 16 different scenes
  • over 3000 manipulated videos using Deep-Fakes.

1. DeepFake Forensics (Celeb-DF) Dataset

• paper: [arXiv 2019] Celeb-DF: A New Dataset for DeepFake Forensics
  • real and DeepFake synthesized videos having similar visual quality on par with those circulated online
  • 408 original videos collected from YouTube with subjects of different ages, ethic groups and genders, and 795 DeepFake videos synthesized from these real videos.

1. [Facebook] Deepfake Detection Challenge (DFDC) Dataset

• paper : [arXiv 2019] The Deepfake Detection Challenge (DFDC) Preview Dataset

  • consisting of 5K videos featuring two facial modification algorithms.
  • a set of specific metrics to evaluate the performance have been defined and two existing models for detecting deepfakes have been tested to provide a reference perfor-mance baseline.

  

1. DeeperForensics-1.0 Dataset

• paper: [arXiv 2020] DeeperForensics-1.0: A Large-Scale Dataset for Real-World Face Forgery Detection
  • represents the largest face forgery detection dataset by far, with 60;000 videos constituted by a total of 17.6 million frames, including 50,000 original collected videos and 10,000 manipulated videos
  • fake videos are generated by a newly proposed end-to-end face swapping framework
• code

1. TAMFA (Tampered face) Dataset

• paper: [Expert Systems With Applications 2019] Face image manipulation detection based on a convolutional neural network
  • 8,950 facial images with unconstrained conditions such as pose, background cluttered, illumination change
  • 1,500 images labeled as “fake” and 7,450 images labeled as “normal”.

1. SwapMe and FaceSwap Dataset

• paper: [CVPRW 2017] Two-Stream Neural Networks for Tampered Face Detection
  • generated by using one iOS application called SwapMe and an open source face swap application called FaceSwap
  • contains 705 fake faces and 1,400 normal faces

1. Deep Fakes Dataset

   • [to be released]
   • paper: [arXiv 2019] FakeCatcher: Detection of Synthetic Portrait Videos using Biological Signals
     • more ''in the wild" portrait videos
     • totaling up to 142 videos, 32 minutes, and 30 GBs

2. Fake Faces in the Wild (FFW) Dataset

• paper: [BIOSIG 2018] Fake Face Detection Methods: Can They Be Generalized?
  • more than 53,000 images (from 150 videos)

1. Swapped Face Detection Dataset

• [to be released]
• paper: [arXiv 2019] Swapped Face Detection using Deep Learning and Subjective Assessment
  • A public dataset comprising 86 celebrities using 420,053 images.
  • This dataset is created using still images, different from other datasets created using video frames that may contain highly correlated images.

1. Video Forensics HQ

   • [to be released]
   • paper:[arXiv 2020] Video Forensics HQ: Detecting High-quality Manipulated Face Videos

2. FFIW10KDataset

   • [CVPR 2021] Face Forensics in the Wild
   • To take face forgery detection to a new level, we construct a novel large-scale dataset, called FFIW10K, which comprises 10,000 high-quality forgery videos, with an average of three human faces in each frame.

II. Current Work

(1) Special Artifact-Based

1. [CVPRW 2019] Protecting World Leaders Against Deep Fakes
   • note;
   • capture the distinct facial expression and movements of a specific person use Action Unit (AU)
2. [CVPRW 2019] Exposing DeepFake Videos By Detecting FaceWarping Artifacts
   • code; note;
   • improved version: DSP-FWA
   • current generated face have limited resolutions
3. [WIFS 2018] In Ictu Oculi: Exposing AI Created Fake Videos by Detecting Eye Blinking
   • code; note;
   • The lack of eye blinking indicates a synthesized video
4. [ICASSP 2019] EXPOSING DEEP FAKES USING INCONSISTENT HEAD POSES
   • note; code;
   • the mismatch between the landmarks at center and outer contour of faked faces is revealed as inconsistent 3D head poses estimated from central and whole facial landmarks
5. [arXiv 2019] FakeCatcher: Detection of Synthetic Portrait Videos using Biological Signals
   • note;
   • biological signals hidden in portrait videos can be used as an implicit descriptor of authenticity, because they are neither spatially nor temporally preserved in fake content.
6. [WACVW 2019] Exploiting Visual Artifacts to Expose Deepfakes and Face Manipulations
   • code; note;
   • detect each manipulation method according to corresponding artifacts(eye color inconsistency, hard shadow in nose/contour, missing details in teeth, etc.)
7. [ICCVW 2019] Deepfake Video Detection through Optical Flow Based CNN
   • we propose the adoption of optical flow fields to exploit possible inter-frame dissimilarities.
8. [IMVOP 2018] Detection of Deepfake Video Manipulation
   • To contribute to a solution, photo response non uniformity (PRNU) analysis is tested for its effectiveness at detecting Deepfake video manipulation
9. [CVPR 2020] Face X-ray for More General Face Forgery Detection
   • note
   • We observe that most existing face manipulation methods share a common step: blending the altered face into an existing background image.
   • The face X-ray of an input face image is a greyscale image that reveals whether the input image can be decomposed into the blending of two images from different sources.
10. [arXiv 2020] DeepFake Detection Based on DiscrepanciesBetween Faces and their Context
11. [arXiv 2020] DeepRhythm: Exposing DeepFakes with Attentional VisualHeartbeat Rhythms
12. [CVPR 2021] Lips Don't Lie: A Generalisable and Robust Approach To Face Forgery Detection

(2) CNN-Based

1. [ICCV 2019] FaceForensics++: Learning to Detect Manipulated Facial Images
   • code; note;
   • XceptionNet
2. [ISITC 2018] Forensics Face Detection From GANs Using Convolutional Neural Network
   • note;
   • VGGFace + 2-way FN
3. [ICASSP 2019] Capsule-forensics: Using Capsule Networks to Detect Forged Images and Videos
   • code; note;
   • image -> [face extracting & align] -> [VGG19] -> features -> [Capsule Network] -> fake/pristine
4. [arXiv 2019] Swapped Face Detection using Deep Learning and Subjective Assessment
   • ResNet18 pretrained on ImageNet

(3) Video forensics

1. [AVSS 2018] Deepfake Video Detection Using Recurrent Neural Networks
   • note;
   • CNN (InceptionV3) + LSTM
2. [CVPR 2019] Recurrent Convolutional Strategies for Face Manipulation Detection in Videos
   • note;
   • CNN (DenseNet) + bidirectional RNN
3. [arxiv 2020] Deepfakes Detection with Automatic Face Weighting
4. [arXiv 2020] Video Face Manipulation Detection Through Ensemble of CNNs
   • code
   • ensemble of CNNs & attention layers & siamese training
   • DFDC challenge performance: the final solution proposed by our team was an ensemble of the 4 proposed models, which led us to top3% on the leaderboard computed against the public test set.
5. [ACM MM 2020] Sharp Multiple Instance Learning for DeepFake Video Detection
6. [arXiv 2020] Dynamic texture analysis for detectingfake faces in video sequences
7. [ECCV 2020] Two-branch Recurrent Network for Isolating Deepfakes in Videos
8. [arXiv 2021] Detection of Deepfake Videos Using Long Distance Attention

(4) Two Stream

1. [CVPRW 2017] Two-Stream Neural Networks for Tampered Face Detection
   • note;
   • Face Classification stream(GoogLeNet) + Patch Triplet stream(Steganalysis feature)
2. [TIFS 2019] Attention-Based Two-Stream Convolutional Networks for Face Spoofing Detection
   • note;
   • RGB stream(contain texture details) + MSR stream(illumination invariant) & Attention-based fusion
3. [arXiv 2019] Complement Face Forensic Detection and Localization with Facial Landmarks
   • face landmark + RGB
4. [ICASSP 2020] SSTNet: Detecting Manipulated Faces Through Spatial, Steganalysis and Temporal Features

(5) Auto-encoder

1. [arXiv 2018] ForensicTransfer: Weakly-supervised Domain Adaptation for Forgery Detection
   • note;
   • input image -> [Encoder] -> Forensic Embedding -> [Decoder] -> reconstructed image
2. [BTAS 2019] Multi-task Learning for Detecting and Segmenting Manipulated Facial Images and Videos
   • code; note;
   • multi-task learning of classification(real/fake) and segmentation(locating manipulated regions in images)
3. [arXiv 2019] Towards Generalizable Forgery Detection with Locality-aware AutoEncoder
   • note
   • To bridge generalization gap, in this paper we propose Locality-aware AutoEcoder (LAE), which combines fine-grained representation learning and enforcing locality in a unified frame-work.
   • A key characteristic of LAE is the augmented local interpretability, which could be regularized using extra pixel wise forgery masks, in order to learn intrinsic and meaningful forgery representations.

(6) Frequency Domain

1. [arXiv 2019] Unmasking DeepFakes with simple Features
   • code; note
   • image -> [DFT] -> sinusoidal components of various frequencies -> [Azimuthal Average] -> 1D representation of FFT power spectrum -> [Classifier] -> Real/Fake
2. [arXiv 2020] Manipulated Face Detector: Joint Spatial and Frequency Domain Attention Network
3. [ECCV 2020] Thinking in Frequency: Face Forgery Detectionby Mining Frequency-aware Clues
4. [CVPR 2021] Spatial-Phase Shallow Learning: Rethinking Face Forgery Detection in Frequency Domain
5. [CVPR 2021] Generalizing Face Forgery Detection with High-frequency Features

(7) General image manipulation

1. [CVPR 2019] ManTraNet: Manipulation Tracing Network For Detection And Localization of Image Forgeries With Anomalous Features
   • code; note;
   • formulate the forgery localization problem as a local anomaly detection problem, design a Z-score feature to capture local anomaly, and propose a novel LSTM solution to assess local anomalies
2. [arXiv 2019] Hybrid LSTM and Encoder-Decoder Architecture for Detection of Image Forgeries
   • note
   • This paper proposes a high-confidence manipulation localization architecture which utilizes resampling features, Long-Short Term Memory (LSTM) cells, and encoder-decoder network to segment out manipulated regions from non-manipulated ones
3. [CVPR 2018] Learning Rich Features for Image Manipulation Detection
   • code
4. [arXiv 2019] Full-Image Full-Resolution End-to-End-Trainable CNN Framework for Image Forgery Detection
   • code

(8) Novel Network or Module

1. [WIFS 2018] MesoNet: a Compact Facial Video Forgery Detection Network
   • code; note;
   • MesoNet exploits features at a meso-scopic level leveraging Inception Module and Dilated Convolution
2. [Expert Systems With Applications 2019] Face image manipulation detection based on a convolutional neural network
   • note;
   • a customized convolutional neural network model for Manipulated Face (MANFA) & A hybrid framework (HF-MANFA) that uses Adaptive Boosting (AdaBoost) and eXtreme Gradient Boosting (XGBoost) to deal with the imbalanced dataset challenge
3. [arXiv 2019] On the Detection of Digital Face Manipulation
   • note
   • proposed a novel attention-based layer to improve classification performance and produce an attention map indicating the manipulated facial regions.
4. [arXiv 2019] Exploiting Human Social Cognition for the Detection of Fake and Fraudulent Faces via Memory Networks
   • note
   • We propose a Hierarchical Memory Network (HMN) architecture, which is able to successfully detect faked faces by utilizing knowledge stored in neural memories as well as visual cues to reason about the perceived face and anticipate its future semantic embeddings.
     
5. [arXiv 2020] Fake Face Detection via Adaptive Residuals Extraction Network
   • Novel residual extractor for residual feature extraction
6. [CVPR 2021] Multi-attentional Deepfake Detection
   • code repo
7. [CVPR 2021] Representative Forgery Mining for Fake Face Detection

(9) GAN-fake face detection

1. [CVPR 2020] Global Texture Enhancement for Fake Face Detection In the Wild
   • propose to introduce “Gram Block” into the CNN architecture and propose a novel architecture coined as Gram-Net as shown. The “Gram Block” captures the global image texture feature by calculating the Gram matrix in different semantic level
2. [WIFS 2019] AutoGAN : Detecting and Simulating Artifacts in GAN Fake Images
   • AutoGAN: which can simulate the artifacts produced by the common pipeline shared by several popular GAN models
3. [CVPR 2020] Watch your Up-Convolution: CNN Based Generative Deep Neural Networks are Failing to Reproduce Spectral Distributions
   • common up-sampling methods,i.e. known as up-convolution or transposed convolution, are causing the inability of such models to reproduce spectral distributions of natural training data correctly
4. [CVPR 2020] CNN-generated images are surprisingly easy to spot... for now
   • with careful pre- and post-processing and data augmentation, a standard image classifier trained on only one specific CNN generator is able to generalize surprisingly well to unseen architectures, datasets, and training methods
5. [arXiv 2020] DeepFake Detection by Analyzing Convolutional Traces
6. [arXiv 2020] Fighting Deepfake by Exposing the ConvolutionalTraces on Images
7. [arXiv 2020] CNN Detection of GAN-Generated Face Imagesbased on Cross-Band Co-occurrences Analysis
8. [Media Watermarking,Security and Forensics 2019] Detecting GAN generated Fake Images using Co-occurrence Matrices
9. [ECCV 2020] What makes fake images detectable? Understanding properties that generalize
   • code repo
   • proj page
10. [CVPR 2021] Closer Look at Fourier Spectrum Discrepancies for CNN-generated Images Detection
    • code
11. [IJCAI2021] Beyond the Spectrum: Detecting Deepfakes via Re-Synthesis
12. [arXiv 2021] Wavelet-Packet Powered Deepfake Image Detection

(10) Domain Adaptation

1. [CVPR 2020] One-Shot Domain Adaptation For Face Generation
2. [Workshop on Media Forensics 2021] FReTAL: Generalizing Deepfake Detection using Knowledge Distillation and Representation Learning

(11) Metrics Learning

1. [arXiv 2020] Detecting Deepfakes with Metric Learning
2. [arXiv 2020] Deep Detection for Face Manipulation
3. [CVPR 2021] Frequency-aware Discriminative Feature Learning Supervised by Single-Center Loss for Face Forgery Detection

(12) Transformer

1. [arXiv 2021] M2TR: Multi-modal Multi-scale Transformers for Deepfake Detection

(13) Adversarial Attack/Defend

1. [WACV 2021] Adversarial Deepfakes: Evaluating Vulnerability of Deepfake Detectors to Adversarial Examples
2. [IJCNN 2020] Adversarial Perturbations Fool Deepfake Detectors
3. [CVPR 2021] MagDR: Mask-guided Detection and Reconstruction for Defending Deepfakes
4. [arXiv 2021] Deepfake Forensics via An Adversarial Game

(14) 3D Model

1. [CVPR 2021] Face Forgery Detection by 3D Decomposition