Research on image privacy detection based on deep transfer learning

doi:10.11996/JG.j.2095-302X.2023061112

Abstract

Abstract:

Addressing the absence of an early warning mechanism for image privacy leakage in current social media platforms, an optimization scheme for image privacy target detection based on the YOLOv8 model was proposed, thus reducing the risk of privacy leakage when users share images. Building upon the YOLOv8 baseline model, the bottleneck transformer (BoT) module was integrated into the backbone network, capturing global contextual information and modeling long-range dependencies of targets. Concurrently, the bidirectional feature pyramid network (BiFPN) structure was introduced to improve the neck network, facilitating the deep fusion of multi-scale features. On this basis, based on the deep transfer learning method, the YOLOv8 pre-training model was fine-tuned and trained to achieve automatic detection of image privacy. A privacy image dataset was constructed using the LabelImg annotation tool, and the common YOLO series model was compared with the improved YOLOv8 in the transfer learning mode. The results demonstrated that YOLOv8 exhibited strong performance in the baseline model, while the F1 and mAP@.5 values of the improved model proposed in this study reached 0.885 and 0.927, respectively, reflecting a 4.0% and 3.4% increase compared with YOLOv8. This significantly enhanced detection accuracy was well-suited for image privacy detection in various application scenarios.

Key words: image privacy, target detection, YOLOv8, global contextual information, transfer learning

CLC Number:

TP391.4

WANG Da-fu, WANG Jing, SHI Yu-kai, DENG Zhi-wen, JIA Zhi-yong. Research on image privacy detection based on deep transfer learning[J]. Journal of Graphics, 2023, 44(6): 1112-1120.

Figures/Tables 12

References 23

[1]	李凤华, 孙哲, 牛犇, 等. 跨社交网络的隐私图片分享框架[J]. 通信学报, 2019, 40(7): 1-13. DOI
	LI F H, SUN Z, NIU B, et al. Privacy-preserving photo sharing framework cross different social network[J]. Journal on Communications, 2019, 40(7): 1-13 (in Chinese). DOI
[2]	赵景欣, 岳星辉, 冯崇朋, 等. 基于通用数据保护条例的数据隐私安全综述[J]. 计算机研究与发展, 2022, 59(10): 2130-2163.
	ZHAO J X, YUE X H, FENG C P, et al. Survey of data privacy security based on general data protection regulation[J]. Journal of Computer Research and Development, 2022, 59(10): 2130-2163 (in Chinese).
[3]	王树义, 刘赛, 马峥. 基于深度迁移学习的微博图像隐私分类研究[J]. 数据分析与知识发现, 2020, 4(10): 80-92.
	WANG S Y, LIU S, MA Z. Microblog image privacy classification with deep transfer learning[J]. Data Analysis and Knowledge Discovery, 2020, 4(10): 80-92 (in Chinese).
[4]	ZERR S, SIERSDORFER S, HARE J, et al. Privacy-aware image classification and search[C]// The 35th International ACM SIGIR Conference on Research and Development in Information Retrieval. New York: ACM, 2012: 35-44.
[5]	TONGE A, CARAGEA C. Privacy prediction of images shared on social media sites using deep features[EB/OL]. [2023-02-12]. https://arxiv.org/abs/1510.08583.pdf.
[6]	TONGE A, CARAGEA C. Image privacy prediction using deep neural networks[J]. ACM Transactions on the Web, 2020, 14(2): Article No. 7: 1-7: 32.
[7]	郑剑, 冷碧玉. 多类别图像数据分类隐私保护算法[J]. 科学技术与工程, 2020, 20(29): 12007-12013.
	ZHENG J, LENG B Y. Privacy protection algorithm of multi-category image data classification[J]. Science Technology and Engineering, 2020, 20(29): 12007-12013 (in Chinese).
[8]	李小波, 李阳贵, 郭宁, 等. 融合注意力机制的YOLOv5口罩检测算法[J]. 图学学报, 2023, 44(1): 16-25. DOI
	LI X B, LI Y G, GUO N, et al. Mask detection algorithm based on YOLOv5 integrating attention mechanism[J]. Journal of Graphics, 2023, 44(1): 16-25 (in Chinese).
[9]	蒋心璐, 陈天恩, 王聪, 等. 大田环境下的农业害虫图像小目标检测算法[J/OL]. 计算机工程. https://doi.org/10.19678/j.issn.1000-3428.0067030.
	JIANG X L, CHEN T E, WANG C, et al. Small object detection algorithm of agricultural pest image in field environment[J/OL]. Computer Engineering. https://doi.org/10.19678/j.issn.1000-3428.0067030 (in Chinese).
[10]	夏烨, 雷哓晖, 祁雁楠, 等. 基于改进Ghost-YOLOv5s- BiFPN算法检测梨树花序[J]. 智慧农业: 中英文, 2022, 4(3): 108-119.
	XIA Y, LEI X H, QI Y N, et al. Detection of pear inflorescence based on improved ghost-YOLOv5s-BiFPN algorithm[J]. Smart Agriculture: English and Chinese, 2022, 4(3): 108-119 (in Chinese).
[11]	余平平, 林耀海, 赖云锋, 等. 融合BiFPN和YOLOv5s的密集型原木端面检测方法[J]. 林业工程学报, 2023, 8(1): 126-134.
	YU P P, LIN Y H, LAI Y F, et al. Dense log end face detection method using the hybrid of BiFPN and YOLOv5s[J]. Journal of Forestry Engineering, 2023, 8(1): 126-134 (in Chinese).
[12]	杨观赐, 杨静, 苏志东, 等. 改进的YOLO特征提取算法及其在服务机器人隐私情境检测中的应用[J]. 自动化学报, 2018, 44(12): 2238-2249.
	YANG G C, YANG J, SU Z D, et al. An improved YOLO feature extraction algorithm and its application to privacy situation detection of social robots[J]. Acta Automatica Sinica, 2018, 44(12): 2238-2249 (in Chinese).
[13]	毛雨晴, 赵奎. 基于改进YOLOv5的多任务安全人头检测算法[J]. 计算机工程, 2022, 48(8): 136-143. DOI
	MAO Y Q, ZHAO K. Multi-task secure head-detection algorithm based on improved YOLOv5[J]. Computer Engineering, 2022, 48(8): 136-143 (in Chinese). DOI
[14]	WU C L, YANG C K, LIN Y L. Privacy protection and beautification of cornea images[J]. Multimedia Tools and Applications, 2022, 81(22): 32421-32448. DOI
[15]	NEPAL U, ESLAMIAT H. Comparing YOLOv3, YOLOv4 and YOLOv5 for autonomous landing spot detection in faulty UAVs[J]. Sensors, 2022, 22(2): 464. DOI URL
[16]	LearnOpenCV. YOLOv8 ultralytics: state-of-the-art YOLO models[EB/OL]. (2023-01-10) [2023-03-11]. https://learnopencv.com/ultralytics-YOLOv8/.
[17]	WANG C Y, BOCHKOVSKIY A, LIAO H Y M. YOLOv7: trainable bag-of-freebies sets new state-of-the-art for real-time object detectors[C]// 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2023: 7464-7475.
[18]	GE Z, LIU S T, WANG F, et al. YOLOX: exceeding YOLO series in 2021[EB/OL]. [2023-03-23]. https://arxiv.org/abs/2107.08430.pdf.
[19]	FENG C J, ZHONG Y J, GAO Y, et al. TOOD: task-aligned one-stage object detection[C]// 2021 IEEE/CVF International Conference on Computer Vision. New York: IEEE Press, 2022: 3490-3499.
[20]	JU R Y, CAI W M. Fracture detection in pediatric wrist trauma X-ray images using YOLOv8 algorithm[EB/OL]. [2023-05-11]. https://arxiv.org/abs/2304.05071.pdf.
[21]	SRINIVAS A, LIN T Y, PARMAR N, et al. Bottleneck transformers for visual recognition[C]// 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2021: 16514-16524.
[22]	ZHANG J, ZHANG J S, ZHOU K X, et al. An improved YOLOv5-based underwater object-detection Framework[J]. Sensors, 2023, 23(7): 3693-3713. DOI URL
[23]	TAN M X, PANG R M, LE Q V. EfficientDet: scalable and efficient object detection[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2020: 10778-10787.

隐私类别	相关子项
个人基本信息	生日、学/工号、家庭关系、社交关系、职业、教育经历、工作单位、吸烟、酗酒、性取向
生物识别信息	指纹、虹膜、人脸特征、身体裸露、基因信息
健康信息	病历、体检记录、住院记录、生病状况
证件信息	身份证、军官证、护照、驾驶证、户口本、学生证、准考证、车牌
通讯信息	电话号码、电子邮箱、通讯录、通信内容
位置信息	行踪轨迹、精准定位、住宿信息、车票
财产信息	银行账户、交易记录、借贷信息、房产信息、票据

隐私类别	相关子项
个人基本信息	生日、学/工号、家庭关系、社交关系、职业、教育经历、工作单位、吸烟、酗酒、性取向
生物识别信息	指纹、虹膜、人脸特征、身体裸露、基因信息
健康信息	病历、体检记录、住院记录、生病状况
证件信息	身份证、军官证、护照、驾驶证、户口本、学生证、准考证、车牌
通讯信息	电话号码、电子邮箱、通讯录、通信内容
位置信息	行踪轨迹、精准定位、住宿信息、车票
财产信息	银行账户、交易记录、借贷信息、房产信息、票据

隐私类别	标注类别	总计
个人基本信息	Personal	269
生物识别信息	Biometrics	307
健康信息	Health	300
证件信息	Certification	479
通讯信息	Communication	152
位置信息	Location	149
财产信息	Property	281

隐私类别	标注类别	总计
个人基本信息	Personal	269
生物识别信息	Biometrics	307
健康信息	Health	300
证件信息	Certification	479
通讯信息	Communication	152
位置信息	Location	149
财产信息	Property	281

模型	P	R	F1	mAP@.5	FPS
YOLOv3	0.732	0.674	0.709	0.717	47
YOLOv4	0.779	0.697	0.736	0.766	94
YOLOv5	0.875	0.762	0.815	0.850	133
YOLOv7	0.862	0.662	0.749	0.784	112
YOLOX	0.869	0.748	0.803	0.832	78
YOLOv8	0.863	0.834	0.848	0.896	226
本文模型	0.906	0.865	0.885	0.927	203