基于深度相机的虚拟眼镜试戴

doi:10.11996/JG.j.2095-302X.2023050988

图学学报 ›› 2023, Vol. 44 ›› Issue (5): 988-996.DOI: 10.11996/JG.j.2095-302X.2023050988

• 计算机图形学与虚拟现实 • 上一篇下一篇

基于深度相机的虚拟眼镜试戴

王可欣(), 金映含, 张东亮()

浙江大学计算机科学与技术学院，浙江杭州 310000

收稿日期:2023-03-21 接受日期:2023-07-06 出版日期:2023-10-31 发布日期:2023-10-31
通讯作者: 张东亮(1971-)，男，教授，博士。主要研究方向为计算机辅助设计、计算机图形学和交互设计等。E-mail：dzhang@zju.edu.cn
作者简介:王可欣(1998-)，女，硕士研究生。主要研究方向为计算机图形学。E-mail：22021133@zju.edu.cn
基金资助:
国家重点研发计划项目(2022YFB3303100);国家自然科学基金项目(61972340)

Virtual glasses try-on using a depth camera

WANG Ke-xin(), JIN Ying-han, ZHANG Dong-liang()

College of Computer Science and Technology, Zhejiang University, Hangzhou Zhejiang, 310000

Received:2023-03-21 Accepted:2023-07-06 Online:2023-10-31 Published:2023-10-31
Contact: ZHANG Dong-liang (1971-), professor, Ph.D. His main research interests cover computer-aided design, computer graphics and interactive design, etc. E-mail：dzhang@zju.edu.cn
About author:WANG Ke-xin (1998-), master student. Her main research interest covers computer graphics. E-mail：22021133@zju.edu.cn
Supported by:
National Key Research and Development Program of China(2022YFB3303100);National Natural Science Foundation of China(61972340)

摘要/Abstract

摘要：

当前，网上购物极大便利人们的生活，但是在网上购买眼镜时，消费者无法像在实体店中一样试戴眼镜，不能直观地看到眼镜佩戴效果，因而难以做出购买决策。针对这一问题，提出一种基于深度相机的虚拟眼镜试戴方法。该方法由2个阶段组成：第一阶段是重建人脸三维模型，利用深度相机采集得到的多角度人脸点云进行点云配准。点云配准时，首先得到特征点云完成一次粗配准，然后提取点云的重叠部分完成一次精细配准，由配准后的点云得到三维人脸模型；第二阶段是眼镜试戴，首先自动对齐人脸模型和眼镜模型并根据视频流实时估计的人脸姿态参数变换模型，并渲染模型，然后将经过消隐处理后的眼镜模型融合进视频流中，最终实现虚拟眼镜试戴效果。实验结果表明，该算法实时性好，试戴效果逼真，给用户带来良好的试戴体验。

关键词: 深度相机, 点云配准, 人脸重建, 虚拟眼镜, 网购

Abstract:

Online shopping has significantly enhanced people’s lives. However, the challenge for consumers when purchasing glasses online arises from the lack of opportunity to physically try on glasses and directly visualize the results of the try-on, a convenience readily available in physical stores. To tackle this issue, this paper proposed a method for virtual glasses try-ons using depth cameras. The method comprised two phases: ① 3D face model reconstruction: This phase utilized the multi-angle face point cloud collected by the depth camera. It involved a two-step process for point clouds registration. First, a coarse registration was performed using feature point clouds. Next, the overlapping parts of the point clouds were extracted for fine registration, ultimately yielding a 3D face model from the registered point cloud. ② Glasses try-on: In this phase, the face model and the glasses model were automatically aligned. The model was then transformed in response to the real-time facial pose parameters estimated based on the video stream. Afterward, the models were rendered, and the glasses model was fused into the video stream, following the occlusion culling procedure, resulting in a realistic virtual glasses try-on effect. Experimental results validated the algorithm’s excellent real-time performance and its ability to provide users with an immersive try-on experience.

Key words: depth camera, point cloud registration, face reconstruction, virtual glasses, online shopping

中图分类号:

TP391

王可欣, 金映含, 张东亮. 基于深度相机的虚拟眼镜试戴[J]. 图学学报, 2023, 44(5): 988-996.

WANG Ke-xin, JIN Ying-han, ZHANG Dong-liang. Virtual glasses try-on using a depth camera[J]. Journal of Graphics, 2023, 44(5): 988-996.

图/表 16

图1 本文方法流程示意图

Fig. 1 Overview of our method

图2 68个人脸特征点

Fig. 2 68 feature points of a face

图3 对齐步骤中定义的关键点((a)眼镜模型的3个关键点；(b)人脸模型的3个关键点)

Fig. 3 Key points defined during the alignment step ((a) Three key points of a glasses model; (b) Three key points of a face model)

图4 眼镜模型对齐到人脸模型((a)未对齐前的人脸模型和眼镜模型；(b)对齐后的人脸模型和眼镜模型)

Fig. 4 Align the glasses model to the face model ((a) The face model and glasses model before alignment; (b) The aligned face model and glasses model)

图5 图像投影原理

Fig. 5 Image projection principle

图6 实现消隐((a)眼镜单独渲染结果I1；(b)眼镜模型和人脸模型渲染结果I2；(c)消隐结果I3)

Fig. 6 Implementation of occlusion culling ((a) Glasses rendering result I1; (b) Glasses and face rendering result I2; (c) Occlusion culling result I3)

图7 深度相机放置位置

Fig. 7 Depth camera placement position

图8 界面

Fig. 8 Interface

图9 点云采集的3个视角((a)左转30°视角；(b)正面视角；(c)右转30°视角)

Fig. 9 Three views of point cloud collection ((a) Left 30 degrees; (b) Front view; (c) Right 30 degrees)

图10 特征点云配准前后不同视角点云对比图 ((a)特征点云配准前；(b)特征点云配准后)

Fig. 10 Comparison of point clouds from different views before and after feature point cloud registration ((a) Before feature point cloud registration; (b) After feature point cloud registration)

图11 重叠点云配准后的点云结果

Fig. 11 Point clouds results after overlapping point cloud registration

表1 特点云配准后以及重叠部分点云配准后的重叠率(%)

Table 1 Oerlap rates after feature point cloud registration and overlapping part point cloud registration (%)

重叠率	P_L₃₀和P_F配准	P_R₃₀和P_F配准
特征点云配准后	85.13	85.29
重叠部分点云配准后	94.79	91.60

图12 虚拟试戴结果

Fig. 12 Virtual glasses try-on results

图13 不同受试者进行虚拟试戴

Fig. 13 Virtual glasses try-on results of different users

图14 不同方法试戴结果对比((a)本文；(b)文献[4]；(c)文献[5]；(d)文献[9])

Fig. 14 Comparing virtual glasses try-on results using different methods ((a) Ours; (b) Literature [4]; (c) Literature [5]; (d) Literature [9])

表2 虚拟试戴可用性调研结果

Table 2 Virtual try-on usability study results

指标	平均分(1~5)
眼镜佩戴精准度	4.3
3D效果真实感	4.3
易用性	4.8
使用意愿	4.7

参考文献 22

[1]	LIANG J B, LIN M C. Machine learning for digital try-on: challenges and progress[J]. Computational Visual Media, 2021, 7(2): 159-167. DOI
[2]	SONG W S, GONG Y H, WANG Y C. VTONShoes: virtual try-on of shoes in augmented reality on a mobile device[C]// 2022 IEEE International Symposium on Mixed and Augmented Reality. New York: IEEE Press, 2022: 234-242.
[3]	KIPS R, JIANG R, BA S, et al. Real-time virtual-try-on from a single example image through deep inverse graphics and learned differentiable renderers[J]. Computer Graphics Forum, 2022, 41(2): 29-40.
[4]	MARELLI D, BIANCO S, CIOCCA G. A web application for glasses virtual try-on in 3D space[C]// The 23rd IEEE International Symposium on Consumer Technologies. New York: IEEE Press, 2019: 299-303.
[5]	HUANG W Y, HSIEH C H, YEH J S. Vision-based virtual eyeglasses fitting system[C]// 2013 IEEE International Symposium on Consumer Electronics. New York: IEEE Press, 2013: 45-46.
[6]	NISWAR A, KHAN I R, FARBIZ F. Virtual try-on of eyeglasses using 3D model of the head[C]// The 10th International Conference on Virtual Reality Continuum and Its Applications in Industry. New York: ACM, 2011: 435-438.
[7]	HUANG S H, YANG Y I, CHU C H. Human-centric design personalization of 3D glasses frame in markerless augmented reality[J]. Advanced Engineering Informatics, 2012, 26(1): 35-45. DOI URL
[8]	TANG D F, ZHANG J Y, TANG K T, et al. Making 3D eyeglasses try-on practical[C]// 2014 IEEE International Conference on Multimedia and Expo Workshops. New York: IEEE Press, 2014: 1-6.
[9]	章锌栋, 付东翔. 基于三维镜框建模的眼镜虚拟试戴系统[J]. 电子科技, 2022, 35(9): 52-57.
	ZHANG X D, FU D X. Glasses virtual try-on system based on 3D frame modeling[J]. Electronic Science and Technology, 2022, 35(9): 52-57. (in Chinese)
[10]	刘越. 考虑低重叠率的点云地图配准方法研究[D]. 哈尔滨: 哈尔滨工业大学, 2021.
	LIU Y. Research on low-overlap point cloud map registration[D]. Harbin: Harbin Institute of Technology, 2021. (in Chinese)
[11]	FERDANI D, FANINI B, PICCIOLI M C, et al. 3D reconstruction and validation of historical background for immersive VR applications and games: the case study of the Forum of Augustus in Rome[J]. Journal of Cultural Heritage, 2020, 43: 129-143. DOI URL
[12]	KIEMEN A L, BRAXTON A M, GRAHN M P, et al. CODA: quantitative 3D reconstruction of large tissues at cellular resolution[J]. Nature Methods, 2022, 19(11): 1490-1499. DOI
[13]	BLANZ V, VETTER T. A morphable model for the synthesis of 3D faces[C]// The 26th Annual Conference on Computer Graphics and Interactive Techniques. New York: ACM, 1999: 187-194.
[14]	TRAN A T, HASSNER T, MASI I, et al. Regressing robust and discriminative 3D morphable models with a very deep neural network[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 1493-1502.
[15]	TEWARI A, ZOLLHÖFER M, KIM H, et al. Notice of removal: MoFA: model-based deep convolutional face autoencoder for unsupervised monocular reconstruction[C]// 2017 IEEE International Conference on Computer Vision Workshops. New York: IEEE Press, 2018: 1274-1283.
[16]	FENG Y, WU F, SHAO X H, et al. Joint 3D face reconstruction and dense alignment with position map regression network[C]// European Conference on Computer Vision. Cham: Springer, 2018: 557-574.
[17]	ZHANG S, YU H, DONG J Y, et al. Automatic reconstruction of dense 3D face point cloud with a single depth image[C]// 2015 IEEE International Conference on Systems, Man, and Cybernetics. New York: IEEE Press, 2016: 1439-1444.
[18]	WANG C W, PENG C C. 3D face point cloud reconstruction and recognition using depth sensor[J]. Sensors, 2021, 21(8): 2587. DOI URL
[19]	BESL P J, MCKAY N D. A method for registration of 3-D shapes[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1992, 14(2): 239-256. DOI URL
[20]	KAZEMI V, SULLIVAN J. One millisecond face alignment with an ensemble of regression trees[C]// 2014 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2014: 1867-1874.
[21]	KAZHDAN M, BOLITHO M, HOPPE H. Poisson surface reconstruction[C]// The 4th Eurographics Symposium on Geometry Processing. New York: ACM, 2006: 61-70.
[22]	HEARN D, BAKER M P, CARITHERS W. Computer graphics with OpenGL[M]. Upper Saddle Rive: Prentice Hall, 2003: 365-368.

基于深度相机的虚拟眼镜试戴

Virtual glasses try-on using a depth camera

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 16

参考文献 22

相关文章 6

编辑推荐

Metrics

本文评价

[1]	范强1，刘鹏1，杨俊2，周沛希1. 基于3D-Harris 与FPFH 改进的3D-NDT 配准算法[J]. 图学学报, 2020, 41(4): 567-575.
[2]	李健 1，杨静茹 1，何斌 2 . 一种应用于大角度变换点云的配准方法[J]. 图学学报, 2018, 39(6): 1098-1104.
[3]	赵夫群1,2，周明全 2,3，耿国华2. 基于局部特征的点云配准算法[J]. 图学学报, 2018, 39(3): 389-394.
[4]	赵文涛1，陈雄文2，张朝贺3. 车架有限元模型中焊点的快速构建研究与应用[J]. 图学学报, 2018, 39(1): 104-108.
[5]	赵夫群1,2，周明全2,3. 改进的概率迭代最近点配准算法[J]. 图学学报, 2017, 38(1): 15-22.
[6]	荣坚，赵翠莲，范志坚，陈晓波. 基于双视角的真实感三维人脸重建系统[J]. 图学学报, 2012, 33(4): 83-87.