Research on knowledge graph-based aviation multi-modal data organization and discovery method

doi:10.11996/JG.j.2095-302X.2024020300

Abstract

Abstract:

The data generated in the life cycle of aviation products shows the characteristics of multi-source and multi-modal. When constructing knowledge engineering for such data, the traditional text retrieval method based on keywords has become inadequate to meet the needs of researchers for knowledge acquisition in the process of scientific research. As a current method for knowledge representation in the specific domain of artificial intelligence, knowledge graph provides an adequate expression and standardized storage of the systematic and relational aspects between knowledge units and data. It served as an effective means of organizing domain-specific data and facilitating knowledge discovery services. Therefore, to reconstruct the operational thinking of researchers, it was proposed to use the knowledge graph as the knowledge expression model in the aviation domain, employing the standard knowledge unit as data carriers. Furthermore, the deep neural network was utilized as the feature encoder of multi-modal data, generating feature vectors for machine understanding and computation. By combining these two key technologies, a search and recommendation engine was built for multi-modal data. Building upon this technological foundation, a system architecture was designed and a knowledge discovery platform was implemented. This platform organized and indexed multi-modal data at the knowledge level, meeting the multi-modal knowledge retrieval needs of aviation researchers.

Key words: data organization, knowledge discovery, knowledge graph, multi-modal, cross-modal retrieval

CLC Number:

TP391

HE Liu, AN Ran, LIU Shuyan, LI Runqi, TAO Jian, ZENG Zhaoyang. Research on knowledge graph-based aviation multi-modal data organization and discovery method[J]. Journal of Graphics, 2024, 45(2): 300-307.

Figures/Tables 7

Fig. 1 General architecture of aeronautical multi-modal data organization and knowledge discovery platform

Fig. 2 Data structure of standard knowledge unit

Fig. 3 Knowledge graph based search/recommendation technology architecture

Fig. 4 Data structure of multi-modal knowledge graph

Fig. 5 Algorithm of cross-modal retrieval

Fig. 6 Knowledge discovery based on standard knowledge units

Fig. 7 Text―image cross-modal retrieval

References 14

[1]	刘峤, 李杨, 段宏, 等. 知识图谱构建技术综述[J]. 计算机研究与发展, 2016, 53(3): 582-600.
	LIU Q, LI Y, DUAN H, et al. Knowledge graph construction techniques[J]. Journal of Computer Research and Development, 2016, 53(3): 582-600 (in Chinese).
[2]	TANG J, ZHANG J, YAO L M, et al. ArnetMiner: extraction and mining of academic social networks[C]// The 14th ACM SIGKDD international conference on Knowledge discovery and data mining. New York: ACM, 2008: 990-998.
[3]	BALTRUSAITIS T, AHUJA C, MORENCY L P. Multimodal machine learning: a survey and taxonomy[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41(2): 423-443. DOI URL
[4]	陈烨, 周刚, 卢记仓. 多模态知识图谱构建与应用研究综述[J]. 计算机应用研究, 2021, 38 (12): 3535-3543.
	CHEN Y, ZHOU G, LU J C. Survey on construction and application research for multi-modal knowledge graphs[J]. Application Research of Computers, 2021, 38 (12): 3535-3543 (in Chinese).
[5]	ZHANG C, YANG Z C, HE X D, et al. Multimodal intelligence: representation learning, information fusion, and applications[J]. IEEE Journal of Selected Topics in Signal Processing, 2020, 14(3): 478-493. DOI URL
[6]	付雷杰, 曹岩, 白瑀, 等. 国内垂直领域知识图谱发展现状与展望[J]. 计算机应用研究, 2021, 38 (11): 3201-3214.
	FU L J, CAO Y, BAI Y, et al. Development status and prospect of vertical domain knowledge graph in China[J]. Application Research of Computers, 2021, 38 (11): 3201-3214 (in Chinese).
[7]	ESSAM M, ELDAWLATLY S, ABBAS H. Contextualized word representations for self-attention network[C]// 2018 13th International Conference on Computer Engineering and Systems. New York: IEEE Press, 2018: 116-121.
[8]	邱凌, 张安思, 李少波, 等. 航空制造知识图谱构建研究综述[J]. 计算机应用研究, 2022, 39 (4): 968-977.
	QIU L, ZHANG A S, LI S B, et al. Survey on building knowledge graphs for aerospace manufacturing[J]. Application Research of Computers, 2022, 39 (4): 968-977 (in Chinese).
[9]	YANG X T, RAMESH P, CHITTA R, et al. Deep multimodal representation learning from temporal data[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2017: 5066-5074.
[10]	BENGIO Y, COURVILLE A, VINCENT P. Representation learning: a review and new perspectives[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(8): 1798-1828. DOI PMID
[11]	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE Press, 2016: 770-778.
[12]	DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[EB/OL]. [2023-12-10]. http://arxiv.org/abs/1810.04805v2.
[13]	彭宇新, 綦金玮, 黄鑫. 多媒体内容理解的研究现状与展望[J]. 计算机研究与发展, 2019, 56(1): 183-208.
	PENG Y X, QI J W, HUANG X. Current research status and prospects on multimedia content understanding[J]. Journal of Computer Research and Development, 2019, 56(1): 183-208 (in Chinese).
[14]	王树徽, 闫旭, 黄庆明. 跨媒体分析与推理技术研究综述[J]. 计算机科学, 2021, 48(3): 79-86. DOI
	WANG S H, YAN X, HUANG Q M. Overview of research on cross-media analysis and reasoning technology[J]. Computer Science, 2021, 48(3): 79-86 (in Chinese).