基于情绪捕捉实现艺术再创造的虚拟现实体验

doi:10.11996/JG.j.2095-302X.2025040775

图学学报 ›› 2025, Vol. 46 ›› Issue (4): 775-782.DOI: 10.11996/JG.j.2095-302X.2025040775

• 计算机图形学与虚拟现实 • 上一篇下一篇

基于情绪捕捉实现艺术再创造的虚拟现实体验

张雨菲(), 丁玎, 李竹颖

东南大学计算机科学与工程学院，江苏南京 211189

收稿日期:2024-10-05 修回日期:2025-01-16 出版日期:2025-08-30 发布日期:2025-08-11
通讯作者:丁玎(1988-)，男，副教授，博士。主要研究方向为人机交互、交互智能、虚拟现实等。E-mail：dingding-1@seu.edu.cn
第一作者:张雨菲(2002-)，女，硕士研究生。主要研究方向为虚拟现实与人机交互。E-mail：220242397@seu.edu.cn
基金资助:
中国高等教育学会高等教育科学研究规划课题(23MY0201)

A virtual reality experience for artistic re-creation based on emotion capture

ZHANG Yufei(), DING Ding, LI Zhuying

School of Computer Science and Engineering, Southeast University, Nanjing Jiangsu 211189, China

Received:2024-10-05 Revised:2025-01-16 Published:2025-08-30 Online:2025-08-11
First author：ZHANG Yufei (2002-), master student. Her main research interests cover virtual reality and human-computer interaction. E-mail：220242397@seu.edu.cn
Supported by:
China Association of Higher Education’s Higher Education Scientific Research Planning Subjects(23MY0201)

摘要/Abstract

摘要：

随着虚拟现实(VR)产业的蓬勃发展和国民审美需求的提升，艺术作品的展现形式成为VR应用中一个重要的研究课题。基于VR技术的沉浸式艺术因其能为观众提供视觉、听觉等多感官、全方位的体验受到了广泛的喜爱。针对现阶段已有应用存在体验单一、与环境交互有限以及沉浸感不足等问题，设计并开发了VR环境下基于情绪捕捉的艺术再生成体验系统，利用面部识别等技术来捕捉情绪，实现人与艺术作品的互动。体验者通过与作品的交互将其感受反馈给艺术作品，使作品体验变得丰富有趣并富有个性化，并进一步调节体验者的情绪。此外，用户验证实验采用问卷调查的方式对系统可用性、沉浸度和功能与自身的情绪进行评估，实验结果表明用户对系统的满意程度较高，使用系统后，用户的情绪状况得到了一定程度的改善。

关键词: 虚拟现实, 人机交互, 情绪检测, 艺术再创造, 情绪调节

Abstract:

With the vigorous growth of the virtual reality (VR) industry and the rising aesthetic demands among the public, the presentation of artistic works has become a key area of study for VR applications. Immersive art based on VR technology is popular for its ability to provide audiences with a multi-sensory, all-encompassing experience, including visual and auditory stimuli. However, existing applications at this stage were confronted with issues such as monolithic experience, limited interaction with the environment, and an insufficient sense of immersion. In view of the above problems, an emotion-capture-based art re-generation experience system in a virtual reality environment was designed and developed, using technologies such as facial recognition to capture emotions and enable interaction between users and artworks. Users’ feelings were fed back to the artwork by interacting with it, thereby enriching and personalizing the experience of the artwork and further regulating the emotions of the experiencer. In addition, an empirical study was conducted to evaluate the system usability, immersion, functionality, and their own emotions using questionnaires, and the results of the experiment showed that users were satisfied with the system, and demonstrated that users’ emotional states were improved to a certain extent.

Key words: virtual reality, human-computer interaction, emotion detection, artistic reinvention, emotion regulation

中图分类号:

张雨菲, 丁玎, 李竹颖. 基于情绪捕捉实现艺术再创造的虚拟现实体验[J]. 图学学报, 2025, 46(4): 775-782.

ZHANG Yufei, DING Ding, LI Zhuying. A virtual reality experience for artistic re-creation based on emotion capture[J]. Journal of Graphics, 2025, 46(4): 775-782.

图/表 10

图1 虚拟博物馆环境图((a) 放置画作的场馆图；(b) 放置雕塑的场馆图)

Fig. 1 Virtual museum environment ((a) The venue where the paintings are placed; (b) The venue where the sculptures are placed)

图2 蒙娜丽莎画作再创造((a) 蒙娜丽莎初始状态；(b) 蒙娜丽莎对不同情绪的反馈状态：从左往右，从上至下依次是开心、惊讶、悲伤和生气)

Fig. 2 Re-creation of the Mona Lisa painting ((a) Mona Lisa’s initial state; (b) Mona Lisa’s feedback states to different emotions: from left to right, top to bottom are happy, surprised, sad and angry)

图3 三维画作再创造((a) 初始状态；(b) 情绪为开心时画作的状态)

Fig. 3 Re-creation of the 3D painting ((a) Initial state; (b) State of the painting when the mood is happy)

图4 雕塑再创造((a) 初始状态；(b) 不同情绪的画作状态：从左往右，从上至下依次是开心、惊讶、悲伤和生气)

Fig. 4 Sculpture re-creation ((a) Initial state; (b) Painted state with different moods: from left to right, top to bottom are happy, surprised, sad and angry)

图5 CMA分布图

Fig. 5 Graph of CMA distribution

表1 VR体验前后情绪配对t样本检验(M±SD)

Table 1 Paired t-test for mood before and after VR experience (M±SD)

情绪	体验前	体验后	t	df	Sig (2-tailed)
Joy	60.2±14.18	76.8±9.95	-7.105	21	0.000**
Happy	54.5±17.11	70.5±17.92	-5.636	21	0.000**
Calm	48.9±20.06	62.3±19.92	-2.512	21	0.020*
Relax	50.7±18.98	68.4±18.99	-5.098	21	0.000**
Disgust	18.6±16.20	7.3±12.03	5.743	21	0.000**
Sad	27.5±16.39	9.8±10.41	5.266	21	0.000**
Angry	14.8±12.95	7.7±10.43	4.830	21	0.000**
Fear	14.1±12.50	9.1±15.17	1.229	21	0.233
Dizzy	5.6±7.91	21.6±19.42	-4.021	21	0.001**

表2 IPQ子量表单样本t检验结果(检验值=4)

Table 2 Sample t-test results for the IPQ subscale form (test value=4)

子量表	M	SD	t	df	Sig(2-tailed)
SP	5.60	0.714	10.504	21	0.000**
INV	4.44	1.104	1.882	21	0.074
REAL	4.52	1.343	1.826	21	0.082
GP	5.78	1.152	7.218	21	0.000**

表3 情绪识别量表单样本t检验结果(检验值=4)

Table 3 Sample t-test results for the Emotion Recognition scale form (test value=4)

情绪识别	M	SD	t	df	Sig(2-tailed)
准确性	4.43	1.434	1.369	21	0.186
合理性	5.71	1.102	7.129	21	0.000**

表4 VRNQ总分单样本t检验结果(检验值=100)

Table 4 Sample t-test results for VRNQ total score (test value=100)

得分	M	SD	t	df	Sig(2-tailed)
总分	117.14	9.342	8.604	21	0.000**

表5 VRNQ部分总分单样本t检验结果(检验值=25)

Table 5 Sample t-test result for the total score of the VRNQ section (test value=25)

子量表	M	SD	t	df
用户体验	28.41	3.418	4.678	21
系统机制	30.36	3.048	8.254	21
辅助机制	29.55	2.132	10.000	21
引起的症状及影响	28.82	3.390	5.284	21

参考文献 31

[1]	中共中央办公厅, 国务院办公厅. 《关于推进实施国家文化数字化战略的意见》[EB/OL]. (2022-05-22) [2024-05-19]. https://www.gov.cn/xinwen/2022-05/22/content_5691759.htm.
	General Office of the CPC Central Committee, General Office of the State Council. Opinions of the State Council Office on promoting the implementation of digital culture[EB/OL]. (2022-05-22) [2024-05-19]. https://www.gov.cn/xinwen/2022-05/22/content_5691759.htm (in Chinese).
[2]	CECOTTI H. Cultural heritage in fully immersive virtual reality[J]. Virtual Worlds, 2022, 1(1): 82-102.
[3]	EVANS C, HUSTON P, BUNN W, et al. IL DIVINO: Michelangelo’s Sistine ceiling in VR[EB/OL]. (2019-11-18) [2024-05-19]. https://store.steampowered.com/app/1165850/IL_DIVINO_Michelangelos_Sistine_Ceiling_in_VR/.
[4]	Borrowed Light Studios. The night cafe: a VR tribute to Vincent van Gogh[EB/OL]. (2016-06-03) [2024-05-19]. https://store.steampowered.com/app/482390/The_Night_Cafe_A_VR_Tribute_to_Vincent_Van_Gogh/.
[5]	YUAN C, YUN Z,. Tunable a VR reconstruction of “Listening to a Guqin” from emperor Zhao Ji[C]// SIGGRAPH ASIA 2016 VR Showcase. New York: ACM, 2016: 6.
[6]	JIN S, FAN M, WANG Y C, et al. Reconstructing traditional Chinese paintings with immersive virtual reality[C]// 2020 CHI Conference on Human Factors in Computing Systems. New York: ACM, 2020: 1-8.
[7]	Cinétévé Experience, BackLight TAPE. The Scream[EB/OL]. (2019-06-19) [2024-05-19]. https://store.steampowered.com/app/1097120/The_Scream/?curator_clanid=32244664.
[8]	林莹莹, 蔡睿凡, 朱雨真, 等. 基于Leap Motion 的虚拟现实陶艺体验系统[J]. 图学学报, 2020, 41(1): 57-65. DOI
	LIN Y Y, CAI R F, ZHU Y Z, et al. Virtual reality pottery modeling system based on leap motion[J]. Journal of Graphics, 2020, 41(1): 57-65 (in Chinese).
[9]	MEHRABIAN A, RUSSELL J A. An approach to environmental psychology[M/OL]. Cambridge: The MIT Press, 1974 [2024-05-14]. https://psycnet.apa.org/record/1974-22049-000.
[10]	EKMAN P, FRIESEN WV. Constants across cultures in the face and emotion[J]. Journal of personality and social psychology, 1971, 17(2):124. DOI PMID
[11]	WALLER B M, PASQUALINI M S. Analysing facial expression using the facial action coding system (FACS)[M]// MÜLLER C, CIENKI A, FRICKE E, et al. Body- Language-Communication. Berlin: De Gruyter Mouton, 2013: 917-931.
[12]	ZHANG K H, HUANG Y Z, DU Y, et al. Facial expression recognition based on deep evolutional spatial-temporal networks[J]. IEEE Transactions on Image Processing, 2017, 26(9): 4193-4203. DOI PMID
[13]	CHEN S, CHENG H C, SITU J, et al. Mirror hearts: exploring the (Mis-)alignment between AI-recognized and self-reported emotions[C]// 2023 CHI Conference on Human Factors in Computing Systems. New York: ACM, 2023: 206.
[14]	BRUNO P, MELNYK V, VÖLCKNER F. Temperature and emotions: Effects of physical temperature on responses to emotional advertising[J]. International Journal of Research in Marketing, 2017, 34(1): 302-320.
[15]	GOSHVARPOUR A, ABBASI A, GOSHVARPOUR A,. An emotion recognition approach based on wavelet transform and second-order difference plot of ECG[J]. Journal of AI and Data Mining, 2017, 5(2): 211-221.
[16]	GRECO A, LANATA A, CITI L, et al. Skin admittance measurement for emotion recognition: a study over frequency sweep[J]. Electronics, 2016, 5(3): 46.
[17]	HUI T K L, SHERRATT R S. Coverage of emotion recognition for common wearable biosensors[J]. Biosensors, 2018, 8(2): 30.
[18]	TABBAA L, SEARLE R, BAFTI S M, et al. VREED: virtual reality emotion recognition dataset using eye tracking & physiological measures[J]. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2021, 5(4): 178.
[19]	MARÍN-MORALES J, HIGUERA-TRUJILLO J L, GRECO A, et al. Affective computing in virtual reality: emotion recognition from brain and heartbeat dynamics using wearable sensors[J]. Scientific Reports, 2018, 8(1): 13657.
[20]	DUBOVI I. Cognitive and emotional engagement while learning with VR: the perspective of multimodal methodology[J]. Computers & Education, 2022, 183: 104495.
[21]	KOOLE S L. The psychology of emotion regulation: an integrative review[J]. Cognition and Emotion, 2008, 23(1): 4-41.
[22]	KRAMER E. Art therapy in a children’s community: a study of the function of art therapy in the treatment program of Wiltwyck School for Boys[M]. Springfield, IL: Charles C Thomas Publisher, Ltd., 1958: 5-23.
[23]	RAJCIC N, MCCORMACK J. Mirror ritual: an affective interface for emotional self-reflection[C]// 2020 CHI Conference on Human Factors in Computing Systems. New York: ACM, 2020: 1-13.
[24]	WATERWORTH J, WATERWORTH E L. Relaxation island: a virtual tropical paradise[EB/OL]. [2024-06-05]. https://www.diva-portal.org/smash/record.jsf?pid=diva2%3A154544&dswid=7041.
[25]	SEAY A F, GROMALA D, HODGES L, et al. The Meditation Chamber: a bio-interactive therapeutic virtual environment[EB/OL]. [2024-06-05]. http://physiologicalcomputing.net/chi2002/chi_papers/Fleming_Seay_Meditation_chamber.pdf.
[26]	PATIBANDA R, MUELLER F F, LESKOVSEK M, et al. Life tree: understanding the design of breathing exercise games[C]// Annual Symposium on Computer-Human Interaction in Play. New York: ACM, 2017: 19-31.
[27]	JÓŹWIK S, CIEŚLIK B, GAJDA R, et al. Evaluation of the impact of virtual reality-enhanced cardiac rehabilitation on depressive and anxiety symptoms in patients with coronary artery disease: a randomised controlled trial[J]. Journal of Clinical Medicine, 2021, 10(10): 2148.
[28]	STERN R A, ARRUDA J E, HOOPER C R, et al. Visual analogue mood scales to measure internal mood state in neurologically impaired patients: description and initial validity evidence[J]. Aphasiology, 1997, 11(1): 59-71.
[29]	SCHUBERT T W. The sense of presence in virtual environments: a three-component scale measuring spatial presence, involvement, and realness[J]. Zeitschrift für Medienpsychologie, 2003, 15(2): 69-71.
[30]	KOURTESIS P, COLLINA S, DOUMAS L A A, et al. Validation of the virtual reality neuroscience questionnaire: maximum duration of immersive virtual reality sessions without the presence of pertinent adverse symptomatology[J]. Frontiers in Human Neuroscience, 2019, 13: 417. DOI PMID
[31]	RUSSELL J A. A circumplex model of affect[J]. Journal of Personality and Social Psychology, 1980, 39(6): 1161-1178.

基于情绪捕捉实现艺术再创造的虚拟现实体验

A virtual reality experience for artistic re-creation based on emotion capture

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 10

参考文献 31

相关文章 15

编辑推荐

Metrics

本文评价

[1]	杜欣, 任洋甫, 胥森哲, 王巨宏, 郑宇飞, 张松海. 虚拟现实重定向行走中的场景缩放增益研究[J]. 图学学报, 2025, 46(4): 793-806.
[2]	朱晓强, 杨伊菲. 虚拟现实环境下的自由雕刻系统[J]. 图学学报, 2025, 46(2): 345-357.
[3]	杨浩中, 孔笑宇, 辜睿坤, 汪淼. 虚拟现实中的大模型技术研究进展与趋势[J]. 图学学报, 2024, 45(6): 1117-1131.
[4]	栾帅, 吴健, 樊润泽, 王莉莉. 基于观察质量场的虚拟对象协同操作方法[J]. 图学学报, 2024, 45(6): 1338-1348.
[5]	任洋甫, 于歌, 傅月瑶, 胥森哲, 何煜, 王巨宏, 张松海. 虚拟现实中场景和时间对用户空间方向认知的影响[J]. 图学学报, 2024, 45(6): 1349-1363.
[6]	厉向东, 夏涵飞, 单逸飞, 阴凯琳, 耿卫东. 图形用户界面自动生成技术的现状与挑战[J]. 图学学报, 2024, 45(3): 409-421.
[7]	严家豪, 吕健, 侯宇康, 莫心祝. 虚拟现实中眼动交互频率对视觉疲劳影响的研究[J]. 图学学报, 2024, 45(3): 528-538.
[8]	王浩淼, 桑胜举, 段晓东, 张伟华, 陶体伟, 马婷. 虚拟现实环境下的协同式三维建模方法[J]. 图学学报, 2024, 45(1): 169-182.
[9]	韩兆阳, 翁冬冬, 郭署山, 贺文杰, 江海燕, 李冬. 一种基于简易标记点编码的光学跟踪系统[J]. 图学学报, 2023, 44(5): 997-1012.
[10]	谢红霞, 胡毓宁, 张赟, 王亚奇, 杜辉, 秦爱红. 全景图像视频的场景分析与内容处理方法综述[J]. 图学学报, 2023, 44(4): 640-657.
[11]	高越, 韩红雷. 基于可发光设备的观众席画面展示系统[J]. 图学学报, 2023, 44(4): 784-793.
[12]	段艳花, 刘子建, 宁铎. 基于眼动技术的 TMA 界面评估及优化[J]. 图学学报, 2022, 43(4): 745-752.
[13]	马宁, 王亚辉. 智能汽车座舱人机交互任务复杂度分析方法[J]. 图学学报, 2022, 43(2): 356-360.
[14]	王秋惠, 王雅馨. 医院消杀机器人作业安全与交互设计策略[J]. 图学学报, 2022, 43(1): 172-180.
[15]	朱永宁 , 葛婷 , 杜盛瑀 , 楼泽如 , 王建民 . 虚拟现实全景流体绘画系统的可用性研究[J]. 图学学报, 2021, 42(5): 833-840.