Deepfake detection method based on multi-feature fusion of frequency domain and spatial domain

doi:10.11996/JG.j.2095-302X.2025010104

Abstract

Abstract:

In today's society, the rapid advancement of facial forgery technology has posed a substantial challenge to social security, especially in the context where deep learning techniques have been widely employed to generate realistic fake videos. These high-quality forged contents not only threaten personal privacy but can also be utilized for illegal activities. Faced with this challenge, traditional forgery detection methods based on single features have become inadequate to meet detection demands. To address this issue, a deepfake detection method based on multi-feature fusion in both frequency and spatial domains was proposed to enhance the detection accuracy and generalization capability for facial forgeries. The frequency domain was dynamically divided into three bands to extract forgery artifacts that cannot be mined in the spatial domain. The spatial domain employed the EfficientNet_b4 network and Transformer architecture to segment image blocks at multiple scales, calculate differences between different blocks, perform detection based on consistency information between upper and lower image blocks, and capture more detailed forgery feature information. Finally, a fusion block using a query-key-value mechanism integrated the methods from the frequency and spatial domains, thereby more comprehensively mining feature information from both domains to enhance the accuracy and transferability of forgery detection. Extensive experimental results confirmed the effectiveness of the proposed method, demonstrating significantly superior performance compared to traditional deepfake detection methods.

Key words: deepfake detection, EfficientNet_b4 network, frequency domain features, spatial domain features, feature fusion

CLC Number:

TP391.41

DONG Jiale, DENG Zhengjie, LI Xiyan, WANG Shiyun. Deepfake detection method based on multi-feature fusion of frequency domain and spatial domain[J]. Journal of Graphics, 2025, 46(1): 104-113.

Figures/Tables 14

References 25

[1]	WANG B, HUANG L Q, HUANG T Q, et al. Two-stream Xception structure based on feature fusion for DeepFake detection[J]. International Journal of Computational Intelligence Systems, 2023, 16(1): 134.
[2]	LIU D C, CHEN T, PENG C L, et al. Attention consistency refined masked frequency forgery representation for generalizing face forgery detection[EB/OL]. [2024-06-20]. https://arxiv.org/abs/2307.11438.
[3]	LUAN T, LIANG G Q, PEI P F. Interpretable DeepFake detection based on frequency spatial transformer[J]. International Journal of Emerging Technologies and Advanced Applications, 2024, 1(2): 19-25.
[4]	COCCOMINI D A, MESSINA N, GENNARO C, et al. Combining EfficientNet and vision transformers for video deepfake detection[C]// The 21st International Conference on Image Analysis and Processing. Cham: Springer, 2022: 219-229.
[5]	ZHU X Y, FEI H Y, ZHANG B, et al. Face forgery detection by 3D decomposition and composition search[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(7): 8342-8357.
[6]	DURALL R, KEUPER M, PFREUNDT F J, et al. Unmasking DeepFakes with simple features[EB/OL]. [2024-06-20]. https://arxiv.org/abs/1911.00686.
[7]	GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets[C]// The 27th International Conference on Neural Information Processing Systems. Cambridge: MIT Press, 2014: 2672-2680.
[8]	KINGMA D P. Auto-encoding variational Bayes[EB/OL]. [2024-06-20]. https://openreview.net/forum?id=33X9fd2-9FyZd.
[9]	WANG S Y, WANG O, ZHANG R, et al. CNN-generated images are surprisingly easy to spot... for now[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 8692-8701.
[10]	BAI W M, LIU Y F, ZHANG Z P, et al. AUNet: learning relations between action units for face forgery detection[C]// 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2023: 24709-24719.
[11]	TAN M X, LE Q. EfficientNet: rethinking model scaling for convolutional neural networks[EB/OL]. [2024-06-20]. https://proceedings.mlr.press/v97/tan19a.html.
[12]	LIANG W Y, WU Y F, WU J S, et al. FAClue: exploring frequency clues by adaptive frequency-attention for Deepfake detection[C]// 2023 42nd Chinese Control Conference. New York: IEEE Press, 2023: 7621-7626.
[13]	AFCHAR D, NOZICK V, YAMAGISHI J, et al. MesoNet: a compact facial video forgery detection network[C]// 2018 IEEE International Workshop on Information Forensics and Security. New York: IEEE Press, 2018: 1-7.
[14]	QIAN Y Y, YIN G J, SHENG L, et al. Thinking in frequency: face forgery detection by mining frequency-aware clues[C]// The 16th European Conference on Computer Vision. Cham: Springer, 2020: 86-103.
[15]	NIRKIN Y, WOLF L, KELLER Y, et al. DeepFake detection based on discrepancies between faces and their context[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(10): 6111-6121.
[16]	LIN K H, HAN W H, LI S D, et al. IR-Capsule: two-stream network for face forgery detection[J]. Cognitive Computation, 2023, 15(1): 13-22.
[17]	JIN X, WU N, JIANG Q, et al. A dual descriptor combined with frequency domain reconstruction learning for face forgery detection in deepfake videos[J]. Forensic Science International: Digital Investigation, 2024, 49: 301747.
[18]	SONG H X, HUANG S Y, DONG Y P, et al. Robustness and generalizability of deepfake detection: a study with diffusion models[EB/OL]. [2024-06-20]. https://arxiv.org/abs/2309.02218.
[19]	翟永杰, 李佳蔚, 陈年昊, 等. 融合改进Transformer的车辆部件检测方法[J]. 图学学报, 2024, 45(5): 930-940. DOI
	ZHAI Y J, LI J W, CHEN N H, et al. The vehicle parts detection method enhanced with Transformer integration[J]. Journal of Graphics, 2024, 45(5): 930-940 (in Chinese). DOI
[20]	ZHOU P, HAN X T, MORARIU V I, et al. Two-stream neural networks for tampered face detection[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops. New York: IEEE Press, 2017: 1831-1839.
[21]	LI Y Z, LYU S W. Exposing DeepFake videos by detecting face warp artifacts[C]// IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. New York: IEEE Press, 2019: 46-52.
[22]	NGUYEN H H, YAMAGISHI J, ECHIZEN I. Capsule-forensics: using capsule networks to detect forged images and videos[C]// ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing. New York: IEEE Press, 2019: 2307-2311.
[23]	MASI I, KILLEKAR A, MASCARENHAS R M, et al. Two-branch recurrent network for isolating deepfakes in videos[C]// The 16th European Conference on Computer Vision. Cham: Springer, 2020: 667-684.
[24]	LI X D, LANG Y N, CHEN Y F, et al. Sharp multiple instance learning for DeepFake video detection[C]// The 28th ACM International Conference on Multimedia. New York: ACM, 2020: 1864-1872.
[25]	李滔, 胡婷, 武丹丹. 结合金字塔结构和注意力机制的单目深度估计[J]. 图学学报, 2024, 45(3): 454-463. DOI
	LI T, HU T, WU D D. Monocular depth estimation combining pyramid structure and attention mechanism[J]. Journal of Graphics, 2024, 45(3): 454-463 (in Chinese). DOI

模型	DF	FS	F2F	NT
文献[13]	93.40	92.93	93.51	79.37
文献[14]	91.65	87.03	90.73	60.57
文献[15]	94.50	84.50	80.30	74.00
文献[16]	97.30	94.20	81.80	79.40
文献[17]	97.80	93.40	88.70	84.20
本文模型	95.88	95.82	95.57	87.00

模型	DF	FS	F2F	NT
文献[13]	93.40	92.93	93.51	79.37
文献[14]	91.65	87.03	90.73	60.57
文献[15]	94.50	84.50	80.30	74.00
文献[16]	97.30	94.20	81.80	79.40
文献[17]	97.80	93.40	88.70	84.20
本文模型	95.88	95.82	95.57	87.00

模型	Insight		Text2img		Inpainting
模型	ACC/%	AUC	ACC/%	AUC	ACC/%	AUC
RECCE^[18]	58.99	63.12	38.14	35.12	51.35	51.52
本文模型	90.21	96.33	96.50	99.32	92.81	97.98

模型	Insight		Text2img		Inpainting
模型	ACC/%	AUC	ACC/%	AUC	ACC/%	AUC
RECCE^[18]	58.99	63.12	38.14	35.12	51.35	51.52
本文模型	90.21	96.33	96.50	99.32	92.81	97.98

划分方式	DF	FS	F2F	NT
能量	95.73	95.26	95.36	86.05
信息熵	95.08	95.27	95.29	85.61
平均	95.34	95.36	95.16	85.27
动态	95.88	95.82	95.57	87.00