A deepfake face detection method that enhances focus on forgery regions

doi:10.11996/JG.j.2095-302X.2025010047

Abstract

Abstract:

The rapid development of deepfake face technology has led to its widespread use in various undesirable ways, making the detection of manipulated facial images and videos an important research topic. Existing convolutional neural networks suffer from overfitting and poor generalization, performing poorly on unknown synthetic face data. To address this limitation, a deepfake face detection method with enhanced focus on forgery regions was proposed. Firstly, an attention mechanism was introduced to process the feature map used for classification, and the learned attention map could highlight the manipulated facial area, thereby improving the generalization capability of the model. Secondly, a forgery regions detection module was connected to the backbone network, reducing the interference of global face information by detecting forgery traces in the multi-scale anchors, further strengthening the model's attention to the local forgery regions. Finally, a consistent representation learning framework was introduced, ensuring that the model pays more attention to the inherent evidence of forgery and avoids overfitting by explicitly constraining the consistency between different representations of the same input. Experiments were conducted on three datasets, including FaceForensics++, Celeb-DF-v2, and DFDC, using EfficientNet-b4 and Xception as the backbone networks, respectively. The results demonstrated that the proposed method achieved good performance in intra-dataset evaluation, and outperformed the original networks and other advanced methods in cross-dataset evaluation.

Key words: deepfake face detection, attention mechanism, forgery regions detection, multi-scale anchors, consistency representation

CLC Number:

ZHANG Wenxiang, WANG Xiali, WANG Xinyi, YANG Zongbao. A deepfake face detection method that enhances focus on forgery regions[J]. Journal of Graphics, 2025, 46(1): 47-58.

Figures/Tables 14

References 36

[1]	THIES J, ZOLLHÖFER M, STAMMINGER M, et al. Face2Face: real-time face capture and reenactment of RGB videos[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2016: 2387-2395.
[2]	THIES J, ZOLLHÖFER M, NIEßNER M. Deferred neural rendering: Image synthesis using neural textures[J]. ACM Transactions on Graphics (TOG), 2019, 38(4): 66.
[3]	YIN J, GAN C, ZHAO K, et al. A novel model for imbalanced data classification[C]// The 34th AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2020: 6680-6687.
[4]	NGUYEN H H, YAMAGISHI J, ECHIZEN I. Use of a capsule network to detect fake images and videos[EB/OL]. (2019-10-29)[2024-01-06]. https://arxiv.org/abs/1910.12467.
[5]	穆大强, 李腾. 基于多模态融合的人脸反欺骗技术[J]. 图学学报, 2020, 41(5): 750-756. DOI
	MU D Q, LI T. Face anti-spoofing technology based on multi-modal fusion[J]. Journal of Graphics, 2020, 41(5): 750-756 (in Chinese).
[6]	DAS S, SEFERBEKOV S, DATTA A, et al. Towards solving the DeepFake problem: an analysis on improving DeepFake detection using dynamic face augmentation[C]// 2021 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2021: 3769-3778.
[7]	WANG C R, DENG W H. Representative forgery mining for fake face detection[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 14918-14927.
[8]	王欣雨, 刘慧, 朱积成, 等. 基于高低频特征分解的深度多模态医学图像融合网络[J]. 图学学报, 2024, 45(1): 65-77. DOI
	WANG X Y, LIU H, ZHU J C, et al. Deep multimodal medical image fusion network based on high-low frequency feature decomposition[J]. Journal of Graphics, 2024, 45(1): 65-77 (in Chinese). DOI
[9]	LIN X, WANG Z J, MA L Z, et al. Salient object detection based on multiscale segmentation and fuzzy broad learning[J]. The Computer Journal, 2022, 65(4): 1006-1019.
[10]	ZHAO T C, XU X, XU M Z, et al. Learning self-consistency for deepfake detection[C]// 2021 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2021: 15003-15013.
[11]	WANG Z D, BAO J M, ZHOU W G, et al. AltFreezing for more general video face forgery detection[C]// 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2023: 4129-4138.
[12]	YAN Z Y, LUO Y H, LYU S W, et al. Transcending forgery specificity with latent space augmentation for generalizable DeepFake detection[C]// 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2024: 8984-8994.
[13]	LE B M, WOO S S. Quality-agnostic deepfake detection with intra-model collaborative learning[C]// 2023 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2023: 22321-22332.
[14]	CHOLLET F. Xception: deep learning with depthwise separable convolutions[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 1800-1807.
[15]	TAN M X, LE Q V. EfficientNet: Rethinking model scaling for convolutional neural networks[EB/OL]. [2024-05-09]. https://dblp.uni-trier.de/db/conf/icml/icml2019.html#TanL19.
[16]	LI Y Z, CHANG M C, LYU S W. In Ictu oculi: Exposing AI created fake videos by detecting eye blinking[C]// 2018 IEEE International Workshop on Information Forensics and Security. New York: IEEE Press, 2018: 1-7.
[17]	PU J M, MANGAOKAR N, WANG B L, et al. Noisescope: Detecting deepfake images in a blind setting[C]// The 36th Annual Computer Security Applications Conference. New York: ACM, 2020: 913-927.
[18]	SHAHZAD S A, HASHMI A, KHAN S, et al. Lip sync matters: a novel multimodal forgery detector[C]// 2022 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference. New York: IEEE Press, 2022: 1885-1892.
[19]	VERMA V, LAMB A, BECKHAM C, et al. Manifold Mixup: better representations by interpolating hidden states[EB/OL]. [2024-05-09]. https://dblp.uni-trier.de/db/conf/icml/icml2019.html#VermaLBNMLB19.
[20]	ZHANG Y C, JIAO R S, LIAO Q C, et al. Uncertainty-guided mutual consistency learning for semi-supervised medical image segmentation[J]. Artificial Intelligence in Medicine, 2023, 138: 102476.
[21]	NI Y S, MENG D P, YU C Q, et al. CORE: consistent representation learning for face forgery detection[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2022: 12-21.
[22]	DANG H, LIU F, STEHOUWER J, et al. On the detection of digital face manipulation[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 5780-5789.
[23]	RÖSSLER A, COZZOLINO D, VERDOLIVA L, et al. FaceForensics++: learning to detect manipulated facial images[C]// 2019 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2019: 1-11.
[24]	LI Y Z, YANG X, SUN P, et al. Celeb-DF: a large-scale challenging dataset for DeepFake forensics[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 3204-3213.
[25]	DOLHANSKY B, BITTON J, PFLAUM B, et al. The DeepFake detection challenge (DFDC) dataset[EB/OL]. (2020-10-28) [2024-03-11]. https://arxiv.org/abs/2006.07397.
[26]	DONG S C, WANG J, JI R H, et al. Implicit identity leakage: The stumbling block to improving deepfake detection generalization[C]// 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2023: 3994-4004.
[27]	LUO Y C, ZHANG Y, YAN J C, et al. Generalizing face forgery detection with high-frequency features[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 16312-16321.
[28]	BAI W M, LIU Y F, ZHANG Z P, et al. AUNet: learning relations between action units for face forgery detection[C]// 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2023: 24709-24719.
[29]	SHIOHARA K, YAMASAKI T. Detecting deepfakes with self-blended images[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2022: 18699-18708.
[30]	ZHAO H Q, WEI T Y, ZHOU W B, et al. Multi-attentional deepfake detection[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 2185-2194.
[31]	SUN K, LIU H, YE Q X, et al. Domain general face forgery detection by learning to weight[C]// The 35th AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2021: 2638-2646.
[32]	HALIASSOS A, VOUGIOUKAS K, PETRIDIS S, et al. Lips don't lie: a generalisable and robust approach to face forgery detection[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 5037-5047.
[33]	ZHENG Y L, BAO J M, CHEN D, et al. Exploring temporal coherence for more general video face forgery detection[C]// 2021 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2021: 15024-15034.
[34]	WANG Y, YU K, CHEN C, et al. Dynamic graph learning with content-guided spatial-frequency relation reasoning for deepfake detection[C]// 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2023: 7278-7287.
[35]	GU Q Q, CHEN S, YAO T P, et al. Exploiting fine-grained face forgery clues via progressive enhancement learning[C]// The 36th AAAI Conference on Artificial Intelligence. Palo Alto: AAAI Press, 2022: 735-743.
[36]	SELVARAJU R R, COGSWELL M, DAS A, et al. Grad-CAM: visual explanations from deep networks via gradient-based localization[C]// 2017 IEEE International Conference on Computer Vision. New York: IEEE Press, 2017: 618-626.

阶段	操作	输入分辨率	输出通道数	层数
1	Conv 3×3	224×224	48	1
2	MBConv1，k3×3	112×112	24	1
3	MBConv1，k3×3	112×112	24	1
4	MBConv6，k3×3	112×112	32	1
5	MBConv6，k3×3	56×56	32	3
6	MBConv6，k5×5	56×56	56	1
7	MBConv6，k5×5	28×28	56	3
8	MBConv6，k3×3	28×28	112	1
9	MBConv6，k3×3	14×14	112	5
10	MBConv6，k5×5	14×14	160	1
11	MBConv6，k5×5	14×14	160	5
12	MBConv6，k5×5	14×14	272	1
13	MBConv6，k5×5	7×7	272	7
14	MBConv6，k3×3	7×7	448	1
15	MBConv6，k3×3	7×7	448	1
16	Conv 1×1 & Pooling & FC	7×7	1792	1

阶段	操作	输入分辨率	输出通道数	层数
1	Conv 3×3	224×224	48	1
2	MBConv1，k3×3	112×112	24	1
3	MBConv1，k3×3	112×112	24	1
4	MBConv6，k3×3	112×112	32	1
5	MBConv6，k3×3	56×56	32	3
6	MBConv6，k5×5	56×56	56	1
7	MBConv6，k5×5	28×28	56	3
8	MBConv6，k3×3	28×28	112	1
9	MBConv6，k3×3	14×14	112	5
10	MBConv6，k5×5	14×14	160	1
11	MBConv6，k5×5	14×14	160	5
12	MBConv6，k5×5	14×14	272	1
13	MBConv6，k5×5	7×7	272	7
14	MBConv6，k3×3	7×7	448	1
15	MBConv6，k3×3	7×7	448	1
16	Conv 1×1 & Pooling & FC	7×7	1792	1

名称	版本
CPU	16核，AMD EPYC 9654
GPU	NVIDIA GeForce RTX 4090
内存	60 GB
操作系统	Ubuntu 22.04.4 LTS
GPU加速库	CUDA 11.8.0，CUDNN 8.9.4
环境	Python 3.10.13，Pytorch 2.1.0

名称	版本
CPU	16核，AMD EPYC 9654
GPU	NVIDIA GeForce RTX 4090
内存	60 GB
操作系统	Ubuntu 22.04.4 LTS
GPU加速库	CUDA 11.8.0，CUDNN 8.9.4
环境	Python 3.10.13，Pytorch 2.1.0

网络	数据增强	测试集AUC/%			AVG
网络	数据增强	FF++	CDF	DFDC	AVG
EfficientNet-b4		98.10	71.77	53.32	74.40
EfficientNet-b4	√	99.73	84.80	70.32	84.95
Xception		99.58	50.73	45.05	65.12
Xception	√	99.72	54.48	45.29	66.50