Virtual glasses try-on using a depth camera

doi:10.11996/JG.j.2095-302X.2023050988

Abstract

Abstract:

Online shopping has significantly enhanced people’s lives. However, the challenge for consumers when purchasing glasses online arises from the lack of opportunity to physically try on glasses and directly visualize the results of the try-on, a convenience readily available in physical stores. To tackle this issue, this paper proposed a method for virtual glasses try-ons using depth cameras. The method comprised two phases: ① 3D face model reconstruction: This phase utilized the multi-angle face point cloud collected by the depth camera. It involved a two-step process for point clouds registration. First, a coarse registration was performed using feature point clouds. Next, the overlapping parts of the point clouds were extracted for fine registration, ultimately yielding a 3D face model from the registered point cloud. ② Glasses try-on: In this phase, the face model and the glasses model were automatically aligned. The model was then transformed in response to the real-time facial pose parameters estimated based on the video stream. Afterward, the models were rendered, and the glasses model was fused into the video stream, following the occlusion culling procedure, resulting in a realistic virtual glasses try-on effect. Experimental results validated the algorithm’s excellent real-time performance and its ability to provide users with an immersive try-on experience.

Key words: depth camera, point cloud registration, face reconstruction, virtual glasses, online shopping

CLC Number:

TP391

WANG Ke-xin, JIN Ying-han, ZHANG Dong-liang. Virtual glasses try-on using a depth camera[J]. Journal of Graphics, 2023, 44(5): 988-996.

Figures/Tables 16

Fig. 1 Overview of our method

Fig. 2 68 feature points of a face

Fig. 3 Key points defined during the alignment step ((a) Three key points of a glasses model; (b) Three key points of a face model)

Fig. 4 Align the glasses model to the face model ((a) The face model and glasses model before alignment; (b) The aligned face model and glasses model)

Fig. 5 Image projection principle

Fig. 6 Implementation of occlusion culling ((a) Glasses rendering result I1; (b) Glasses and face rendering result I2; (c) Occlusion culling result I3)

Fig. 7 Depth camera placement position

Fig. 8 Interface

Fig. 9 Three views of point cloud collection ((a) Left 30 degrees; (b) Front view; (c) Right 30 degrees)

Fig. 10 Comparison of point clouds from different views before and after feature point cloud registration ((a) Before feature point cloud registration; (b) After feature point cloud registration)

Fig. 11 Point clouds results after overlapping point cloud registration

Table 1 Oerlap rates after feature point cloud registration and overlapping part point cloud registration (%)

重叠率	P_L₃₀和P_F配准	P_R₃₀和P_F配准
特征点云配准后	85.13	85.29
重叠部分点云配准后	94.79	91.60

Fig. 12 Virtual glasses try-on results

Fig. 13 Virtual glasses try-on results of different users

Fig. 14 Comparing virtual glasses try-on results using different methods ((a) Ours; (b) Literature [4]; (c) Literature [5]; (d) Literature [9])

Table 2 Virtual try-on usability study results

指标	平均分(1~5)
眼镜佩戴精准度	4.3
3D效果真实感	4.3
易用性	4.8
使用意愿	4.7

References 22

[1]	LIANG J B, LIN M C. Machine learning for digital try-on: challenges and progress[J]. Computational Visual Media, 2021, 7(2): 159-167. DOI
[2]	SONG W S, GONG Y H, WANG Y C. VTONShoes: virtual try-on of shoes in augmented reality on a mobile device[C]// 2022 IEEE International Symposium on Mixed and Augmented Reality. New York: IEEE Press, 2022: 234-242.
[3]	KIPS R, JIANG R, BA S, et al. Real-time virtual-try-on from a single example image through deep inverse graphics and learned differentiable renderers[J]. Computer Graphics Forum, 2022, 41(2): 29-40.
[4]	MARELLI D, BIANCO S, CIOCCA G. A web application for glasses virtual try-on in 3D space[C]// The 23rd IEEE International Symposium on Consumer Technologies. New York: IEEE Press, 2019: 299-303.
[5]	HUANG W Y, HSIEH C H, YEH J S. Vision-based virtual eyeglasses fitting system[C]// 2013 IEEE International Symposium on Consumer Electronics. New York: IEEE Press, 2013: 45-46.
[6]	NISWAR A, KHAN I R, FARBIZ F. Virtual try-on of eyeglasses using 3D model of the head[C]// The 10th International Conference on Virtual Reality Continuum and Its Applications in Industry. New York: ACM, 2011: 435-438.
[7]	HUANG S H, YANG Y I, CHU C H. Human-centric design personalization of 3D glasses frame in markerless augmented reality[J]. Advanced Engineering Informatics, 2012, 26(1): 35-45. DOI URL
[8]	TANG D F, ZHANG J Y, TANG K T, et al. Making 3D eyeglasses try-on practical[C]// 2014 IEEE International Conference on Multimedia and Expo Workshops. New York: IEEE Press, 2014: 1-6.
[9]	章锌栋, 付东翔. 基于三维镜框建模的眼镜虚拟试戴系统[J]. 电子科技, 2022, 35(9): 52-57.
	ZHANG X D, FU D X. Glasses virtual try-on system based on 3D frame modeling[J]. Electronic Science and Technology, 2022, 35(9): 52-57. (in Chinese)
[10]	刘越. 考虑低重叠率的点云地图配准方法研究[D]. 哈尔滨: 哈尔滨工业大学, 2021.
	LIU Y. Research on low-overlap point cloud map registration[D]. Harbin: Harbin Institute of Technology, 2021. (in Chinese)
[11]	FERDANI D, FANINI B, PICCIOLI M C, et al. 3D reconstruction and validation of historical background for immersive VR applications and games: the case study of the Forum of Augustus in Rome[J]. Journal of Cultural Heritage, 2020, 43: 129-143. DOI URL
[12]	KIEMEN A L, BRAXTON A M, GRAHN M P, et al. CODA: quantitative 3D reconstruction of large tissues at cellular resolution[J]. Nature Methods, 2022, 19(11): 1490-1499. DOI
[13]	BLANZ V, VETTER T. A morphable model for the synthesis of 3D faces[C]// The 26th Annual Conference on Computer Graphics and Interactive Techniques. New York: ACM, 1999: 187-194.
[14]	TRAN A T, HASSNER T, MASI I, et al. Regressing robust and discriminative 3D morphable models with a very deep neural network[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 1493-1502.
[15]	TEWARI A, ZOLLHÖFER M, KIM H, et al. Notice of removal: MoFA: model-based deep convolutional face autoencoder for unsupervised monocular reconstruction[C]// 2017 IEEE International Conference on Computer Vision Workshops. New York: IEEE Press, 2018: 1274-1283.
[16]	FENG Y, WU F, SHAO X H, et al. Joint 3D face reconstruction and dense alignment with position map regression network[C]// European Conference on Computer Vision. Cham: Springer, 2018: 557-574.
[17]	ZHANG S, YU H, DONG J Y, et al. Automatic reconstruction of dense 3D face point cloud with a single depth image[C]// 2015 IEEE International Conference on Systems, Man, and Cybernetics. New York: IEEE Press, 2016: 1439-1444.
[18]	WANG C W, PENG C C. 3D face point cloud reconstruction and recognition using depth sensor[J]. Sensors, 2021, 21(8): 2587. DOI URL
[19]	BESL P J, MCKAY N D. A method for registration of 3-D shapes[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1992, 14(2): 239-256. DOI URL
[20]	KAZEMI V, SULLIVAN J. One millisecond face alignment with an ensemble of regression trees[C]// 2014 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2014: 1867-1874.
[21]	KAZHDAN M, BOLITHO M, HOPPE H. Poisson surface reconstruction[C]// The 4th Eurographics Symposium on Geometry Processing. New York: ACM, 2006: 61-70.
[22]	HEARN D, BAKER M P, CARITHERS W. Computer graphics with OpenGL[M]. Upper Saddle Rive: Prentice Hall, 2003: 365-368.