基于知识图谱的航空多模态数据组织与知识发现技术研究

doi:10.11996/JG.j.2095-302X.2024020300

图学学报 ›› 2024, Vol. 45 ›› Issue (2): 300-307.DOI: 10.11996/JG.j.2095-302X.2024020300

• 数字化设计与制造专刊 • 上一篇下一篇

基于知识图谱的航空多模态数据组织与知识发现技术研究

何柳(), 安然, 刘姝妍, 李润岐, 陶剑, 曾照洋()

中国航空综合技术研究所，北京 100028

收稿日期:2024-01-05 修回日期:2024-01-26 出版日期:2024-04-30 发布日期:2024-04-29
通讯作者: 曾照洋(1979-)，男，研究员，硕士。主要研究方向为质量工程、可靠性和保障等。E-mail：zengzy003@avic.com
作者简介:何柳(1988-)，男，高级工程师，硕士。主要研究方向为人工智能、计算机视觉和多模态机器学习。E-mail：hel054@avic.com
基金资助:
工信部民机专项科研项目(MJZ2-3N21)

Research on knowledge graph-based aviation multi-modal data organization and discovery method

HE Liu(), AN Ran, LIU Shuyan, LI Runqi, TAO Jian, ZENG Zhaoyang()

China Aero-Polytechnology Establishment, Beijing 100028, China

Received:2024-01-05 Revised:2024-01-26 Online:2024-04-30 Published:2024-04-29
Contact: ZENG Zhaoyang (1979-), researcher, master. His main research interests cover quality engineering, reliability and maintainability, etc. E-mail：zengzy003@avic.com
About author:HE Liu (1988-), senior engineer, master. His main research interests cover artificial intelligence, computer vision and multimodal machine learning. E-mail：hel054@avic.com
Supported by:
Ministry of Industry and Information Technology Foundation(MJZ2-3N21)

摘要/Abstract

摘要：

航空产品在研制使用生命周期过程中产生的数据呈现出多源多模态的特性，面向此类数据进行知识工程建设时，传统基于关键词的文本检索方式已经无法满足科研人员在科研过程中对知识的获取需求。知识图谱作为当前人工智能领域对知识表示的一种方式，可以对知识单元以及数据之间的体系性和关系性进行充分表达和规范化存储，是垂直领域数据组织与知识发现服务的一种有效途径。因此提出以知识图谱作为航空领域知识表达模型，以标准知识单元作为数据载体，对科研人员的业务思维进行建模，同时利用深度神经网络作为多模态数据的特征编码器，构建适合机器理解与计算的特征向量，结合两者的特点构建面向多模态数据的搜索、推荐引擎。在此技术基础上设计系统架构并实现知识发现平台，将多模态数据在知识层面进行组织索引，满足航空科研人员的多模态知识检索需求。

关键词: 数据组织, 知识发现, 知识图谱, 多模态, 跨模态检索

Abstract:

The data generated in the life cycle of aviation products shows the characteristics of multi-source and multi-modal. When constructing knowledge engineering for such data, the traditional text retrieval method based on keywords has become inadequate to meet the needs of researchers for knowledge acquisition in the process of scientific research. As a current method for knowledge representation in the specific domain of artificial intelligence, knowledge graph provides an adequate expression and standardized storage of the systematic and relational aspects between knowledge units and data. It served as an effective means of organizing domain-specific data and facilitating knowledge discovery services. Therefore, to reconstruct the operational thinking of researchers, it was proposed to use the knowledge graph as the knowledge expression model in the aviation domain, employing the standard knowledge unit as data carriers. Furthermore, the deep neural network was utilized as the feature encoder of multi-modal data, generating feature vectors for machine understanding and computation. By combining these two key technologies, a search and recommendation engine was built for multi-modal data. Building upon this technological foundation, a system architecture was designed and a knowledge discovery platform was implemented. This platform organized and indexed multi-modal data at the knowledge level, meeting the multi-modal knowledge retrieval needs of aviation researchers.

Key words: data organization, knowledge discovery, knowledge graph, multi-modal, cross-modal retrieval

中图分类号:

TP391

何柳, 安然, 刘姝妍, 李润岐, 陶剑, 曾照洋. 基于知识图谱的航空多模态数据组织与知识发现技术研究[J]. 图学学报, 2024, 45(2): 300-307.

HE Liu, AN Ran, LIU Shuyan, LI Runqi, TAO Jian, ZENG Zhaoyang. Research on knowledge graph-based aviation multi-modal data organization and discovery method[J]. Journal of Graphics, 2024, 45(2): 300-307.

图/表 7

图1 航空多模态数据组织与知识发现平台总体架构

Fig. 1 General architecture of aeronautical multi-modal data organization and knowledge discovery platform

图2 标准知识单元数据结构

Fig. 2 Data structure of standard knowledge unit

图3 基于知识图谱的搜索/推荐技术架构

Fig. 3 Knowledge graph based search/recommendation technology architecture

图4 多模态知识图谱数据结构

Fig. 4 Data structure of multi-modal knowledge graph

图5 跨模态检索算法流程

Fig. 5 Algorithm of cross-modal retrieval

图6 基于标准知识单元的知识发现

Fig. 6 Knowledge discovery based on standard knowledge units

图7 文本―图像的跨模态搜索

Fig. 7 Text―image cross-modal retrieval

参考文献 14

[1]	刘峤, 李杨, 段宏, 等. 知识图谱构建技术综述[J]. 计算机研究与发展, 2016, 53(3): 582-600.
	LIU Q, LI Y, DUAN H, et al. Knowledge graph construction techniques[J]. Journal of Computer Research and Development, 2016, 53(3): 582-600 (in Chinese).
[2]	TANG J, ZHANG J, YAO L M, et al. ArnetMiner: extraction and mining of academic social networks[C]// The 14th ACM SIGKDD international conference on Knowledge discovery and data mining. New York: ACM, 2008: 990-998.
[3]	BALTRUSAITIS T, AHUJA C, MORENCY L P. Multimodal machine learning: a survey and taxonomy[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41(2): 423-443. DOI URL
[4]	陈烨, 周刚, 卢记仓. 多模态知识图谱构建与应用研究综述[J]. 计算机应用研究, 2021, 38 (12): 3535-3543.
	CHEN Y, ZHOU G, LU J C. Survey on construction and application research for multi-modal knowledge graphs[J]. Application Research of Computers, 2021, 38 (12): 3535-3543 (in Chinese).
[5]	ZHANG C, YANG Z C, HE X D, et al. Multimodal intelligence: representation learning, information fusion, and applications[J]. IEEE Journal of Selected Topics in Signal Processing, 2020, 14(3): 478-493. DOI URL
[6]	付雷杰, 曹岩, 白瑀, 等. 国内垂直领域知识图谱发展现状与展望[J]. 计算机应用研究, 2021, 38 (11): 3201-3214.
	FU L J, CAO Y, BAI Y, et al. Development status and prospect of vertical domain knowledge graph in China[J]. Application Research of Computers, 2021, 38 (11): 3201-3214 (in Chinese).
[7]	ESSAM M, ELDAWLATLY S, ABBAS H. Contextualized word representations for self-attention network[C]// 2018 13th International Conference on Computer Engineering and Systems. New York: IEEE Press, 2018: 116-121.
[8]	邱凌, 张安思, 李少波, 等. 航空制造知识图谱构建研究综述[J]. 计算机应用研究, 2022, 39 (4): 968-977.
	QIU L, ZHANG A S, LI S B, et al. Survey on building knowledge graphs for aerospace manufacturing[J]. Application Research of Computers, 2022, 39 (4): 968-977 (in Chinese).
[9]	YANG X T, RAMESH P, CHITTA R, et al. Deep multimodal representation learning from temporal data[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 5066-5074.
[10]	BENGIO Y, COURVILLE A, VINCENT P. Representation learning: a review and new perspectives[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(8): 1798-1828. DOI PMID
[11]	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2016: 770-778.
[12]	DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[EB/OL]. [2023-12-10]. http://arxiv.org/abs/1810.04805v2.
[13]	彭宇新, 綦金玮, 黄鑫. 多媒体内容理解的研究现状与展望[J]. 计算机研究与发展, 2019, 56(1): 183-208.
	PENG Y X, QI J W, HUANG X. Current research status and prospects on multimedia content understanding[J]. Journal of Computer Research and Development, 2019, 56(1): 183-208 (in Chinese).
[14]	王树徽, 闫旭, 黄庆明. 跨媒体分析与推理技术研究综述[J]. 计算机科学, 2021, 48(3): 79-86. DOI
	WANG S H, YAN X, HUANG Q M. Overview of research on cross-media analysis and reasoning technology[J]. Computer Science, 2021, 48(3): 79-86 (in Chinese).

[1]	王刚锋 , 张寰 , 刘思濛 , 岳萍 , 张栋 . 基于语义工艺知识的驱动桥装配序列规划研究 [J]. 图学学报, 2024, 45(3): 564-574.
[2]	苑朝, 赵亚冬, 张耀, 王嘉璇, 徐大伟, 翟永杰, 朱松松. 基于YOLO轻量化的多模态行人检测算法[J]. 图学学报, 2024, 45(1): 35-46.
[3]	王欣雨, 刘慧, 朱积成, 盛玉瑞, 张彩明. 基于高低频特征分解的深度多模态医学图像融合网络[J]. 图学学报, 2024, 45(1): 65-77.
[4]	陈逸天, 张玮, 谭思危, 朱融晨, 王毅超, 朱闽峰, 陈为. 历史人物群体对比可视化[J]. 图学学报, 2023, 44(6): 1227-1238.
[5]	薛皓玮, 王美丽. 融合生物力学约束与多模态数据的手部重建[J]. 图学学报, 2023, 44(4): 794-800.
[6]	孙亚男, 温玉辉, 舒叶芷, 刘永进. 融合动作特征的多模态情绪识别 [J]. 图学学报, 2022, 43(6): 1159-1169.
[7]	李晓英, 余亚平. 基于多模态感官体验的儿童音画交互设计研究[J]. 图学学报, 2022, 43(4): 736-743.
[8]	邓壮林, 张绍兵, 成苗, 何莲. 多模态硬币图像单应性矩阵预测[J]. 图学学报, 2022, 43(3): 361-369.
[9]	胡俊, 顾晶晶, 王秋红. 基于遥感图像的多模态小目标检测[J]. 图学学报, 2022, 43(2): 197-204.
[10]	汪玉金, 谢诚, 余蓓蓓, 向鸿鑫, 柳青. 属性语义与图谱语义融合增强的零次学习图像识别[J]. 图学学报, 2021, 42(6): 899-907.
[11]	宋建炜 , 邓逸川 , 苏成 , . 基于预训练语言模型的建筑施工安全事故文本的命名实体识别研究[J]. 图学学报, 2021, 42(2): 307-315.
[12]	黄欢 , 孙力娟 , 曹莹 , 郭剑 , 任恒毅 . 基于注意力的短视频多模态情感分析[J]. 图学学报, 2021, 42(1): 8-14.
[13]	穆大强，李腾 . 基于多模态融合的人脸反欺骗技术[J]. 图学学报, 2020, 41(5): 750-756.
[14]	蒋圣南，陈恩庆，郑铭耀，段建康 . 基于 ResNeXt 的人体动作识别[J]. 图学学报, 2020, 41(2): 277-282.
[15]	张俐，李承文. 网格关联的四叉树索引点云排序方法[J]. 图学学报, 2017, 38(3): 320-325.

基于知识图谱的航空多模态数据组织与知识发现技术研究

Research on knowledge graph-based aviation multi-modal data organization and discovery method

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 7

参考文献 14

相关文章 15

编辑推荐

Metrics

本文评价