[Object] Hydropower underground engineering encounters significant safety management challenges owing to overlapping construction activities, diverse process stages, and dynamic resource flows. This involves multidisciplinary safety tasks, such as safety hazard identification and rectification, emergency response, and regulatory compliance checks, which require specialized domain knowledge. In this context, safety management knowledge is intricate, such as expert experience, patterns and characteristics, and management codes, and is dispersed across multimodal data formats, including text, tables, and images. Efficient extraction of these multimodal data sources can significantly enhance data utility and support intelligent safety management. However, owing to the diverse nature of data formats, the complexity of the knowledge system, and the various management scenarios, current research struggles with limited knowledge sources, acquisition difficulties, and poor generalization. [Methods] This study proposes a method of constructing a multimodal knowledge graph (KG) for safety management in hydropower underground engineering. (1) A large-scale, high-quality, multisource heterogeneous dataset is built from safety hazard identification and rectification records, regulations, and images. (2) Knowledge modeling employs top-down and bottom-up approaches to define entities, relationships, attributes, and events pertinent to safety management in hydropower underground engineering. (3) The entity and relationship information from text data is obtained using a knowledge extraction method that uses a large language model (LLM) tuned with domain knowledge, enriched by specific examples for each entity type to handle small sample sizes. This approach uses demonstrations to provide the model with prior knowledge. (4) Instance segmentation is used to annotate safety hazard images. The entities identified in the images are then converted into vectors. Image and text data are linked based on semantic similarity. Image data are integrated into the textual KG, enabling the transformation from multimodal data to multimodal knowledge. (5) The multimodal KG is stored in Neo4j, an open-source graph database management system. (6) A scenario-specific knowledge acquisition method addresses the specific needs of safety management scenarios, integrating KG with LLMs to enable retrieval-augmented generation and interpretable knowledge reasoning. [Results] (1) This paper collected more than 120 000 safety hazard records, 30 regulatory documents, and 300 000 images of safety hazards. Leveraging these comprehensive data, this paper constructed a large-scale, high-quality, multisource heterogeneous dataset specifically designed for managing safety in hydropower underground engineering projects. (2) Taking a hydropower underground engineering project as an example, the constructed multimodal KG was applied to intelligent recommendations for safety hazard rectification and compliance checks. (3) The workflow for generating intelligent recommendations for safety hazard rectification measures involved the following steps. After users input safety hazard information, the scene-KG was extracted from the multimodal KG and fed into an LLM to generate appropriate rectification measures. (4) Based on the scene-KG, an inference retrieval method extended neighboring nodes and constructed inference-KG for compliance checks. By integrating inference-KG with an LLM, the system retrieved relevant content from regulatory documents based on user input. [Conclusions] The proposed method effectively extracts and applies domain knowledge from multimodal data in the context of safety management in hydropower underground engineering. It also successfully applies domain knowledge for safety management. The results serve as a reference for transitioning infrastructure construction safety management from a data-driven approach to a knowledge-driven approach.
[1] FAN Q X, DENG Z Y, LIN P, et al. Coordinated deformation control technologies for the high sidewall-bottom transfixion zone of large underground hydro-powerhouses [J]. Journal of Zhejiang University, 2022, 23(7): 543-563.
[2] 樊启祥, 林鹏, 蒋树, 等. 金沙江下游大型水电站岩石力学与工程综述[J]. 清华大学学报(自然科学版), 2020, 60(7): 537-556. FAN Q X, LIN P, JIANG S, et al. Review on the rock mechanics and engineering practice for large hydropower stations along the downstream section of the Jinsha River [J]. Journal of Tsinghua University (Science and Technology), 2020, 60(7): 537-556. (in Chinese)
[3] AN R N, LIN P, LI Z C, et al. Intelligent ventilation-on-demand control system for the construction of underground tunnel complex [J]. Journal of Intelligent Construction, 2024, 2(2): 9180032.
[4] 林鹏, 魏鹏程, 樊启祥, 等. 基于CNN模型的施工现场典型安全隐患数据学习[J]. 清华大学学报(自然科学版), 2019, 59(8): 628-634. LIN P, WEI P C, FAN Q X, et al. CNN model for mining safety hazard data from a construction site [J]. Journal of Tsinghua University (Science and Technology), 2019, 59(8): 628-634. (in Chinese)
[5] XIANG Y F, LIN P, AN R N, et al. Full participation flat closed-loop safety management method for offshore wind power construction sites [J]. Journal of Intelligent Construction, 2023, 1(1): 9180006.
[6] WANG B, WANG Y J. Big data in safety management: An overview [J]. Safety Science, 2021, 143: 105414.
[7] CARABANTES M. Black-box artificial intelligence: An epistemological and critical analysis [J]. AI & Society, 2020, 35(2): 309-317.
[8] LIANG W X, TADESSE G A, HO D, et al. Advances, challenges and opportunities in creating data for trustworthy AI [J]. Nature Machine Intelligence, 2022, 4(8): 669-677.
[9] 郑霞忠, 汪珂, 陈云, 等. 水电工程施工安全隐患文本智能类推研究[J]. 安全与环境学报, 2023, 23(12): 4449-4456. ZHENG X Z, WANG K, CHEN Y, et al. Research on text intelligent analogy of potential safety hazards in hydropower engineering construction [J]. Journal of Safety and Environment, 2023, 23(12): 4449-4456. (in Chinese)
[10] 黄简, 杨程, 冯天波, 等. 面向风电机组运维的知识图谱构建研究与应用[J]. 电力系统保护与控制, 2024, 52(8): 167-177. HUANG J, YANG C, FENG T B, et al. Research and application of knowledge graph construction for wind turbine operation and maintenance [J]. Power System Protection and Control, 2024, 52(8): 167-177. (in Chinese)
[11] 李秋荣, 刘晓晓, 王波, 等. 滑坡地质灾害语料库构建与命名实体识别[J/OL]. 南京信息工程大学学报. (2024-05-29) [2024-07-02]. https://doi.org/10.13878/j.cnki.jnuist.20240429001. LI Q R, LIU X X, WANG B, et al. Corpus construction and named entity recognition for landslide geological hazards [J/OL]. Journal of Nanjing University of Information Science & Technology. (2024-05-29) [2024-07-02]. https://doi.org/10.13878/j.cnki.jnuist.20240429001. (in Chinese)
[12] 刘宇, 李勇. 面向城市可持续发展的城市商圈/街区知识图谱构建方法与应用展望[J]. 地球信息科学学报, 2023, 25(12): 2374-2386. LIU Y, LI Y. Methodology for constructing urban business area/region knowledge graph and the applications in urban sustainable development [J]. Journal of Geo-information Science, 2023, 25(12): 2374-2386. (in Chinese)
[13] HOGAN A, BLOMQVIST E, COCHEZ M, et al. Knowledge graphs [J]. ACM Computing Surveys (Csur), 2021, 54(4): 1-37.
[14] 董兴芝. 面向智能高铁安全保障的知识图谱构建及应用关键技术研究[D]. 北京: 中国铁道科学研究院, 2022. DONG X Z. Research on key technology of knowledge graph construction and its application for intelligent high-speed railway safety assurance [D]. Beijing: China Academy of Railway Sciences, 2022. (in Chinese)
[15] 周义棋, 刘畅, 龙增, 等. 电网应急预案知识图谱构建方法与应用[J]. 中国安全生产科学技术, 2023, 19(1): 5-13. ZHOU Y Q, LIU C, LONG Z, et al. Construction method and application of knowledge graph in emergency plans for power grid [J]. Journal of Safety Science and Technology, 2023, 19(1): 5-13. (in Chinese)
[16] 汪珂. 基于知识图谱的水电工程施工安全隐患治理措施智能决策[D]. 宜昌: 三峡大学, 2023. WANG K. Intelligent analogy of safety hazard management in hydroelectric engineering construction based on knowledge graph [D]. Yichang: China Three Gorges University, 2023. (in Chinese)
[17] LAMPLE G, BALLESTEROS M, SUBRAMANIAN S, et al. Neural architectures for named entity recognition [C]//Proceedings of 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. San Diego, USA: Association for Computational Linguistics, 2016: 260-270.
[18] ALT C, HÜBNER M, HENNIG L. Fine-tuning pre-trained transformer language models to distantly supervised relation extraction [C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Florence, Italy: Association for Computational Linguistics, 2019: 1388-1398.
[19] CHEN X, ZHANG N Y, XIE X, et al. Knowprompt: Knowledge-aware prompt-tuning with synergistic optimization for relation extraction [C]//Proceedings of ACM Web Conference 2022. Lyon, France: ACM, 2022: 2778-2788.
[20] DEVLIN J, CHANG M W, LEE K, et al. BERT: Pre-training of deep bidirectional transformers for language understanding [C]//Proceedings of 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Minneapolis, USA: Association for Computational Linguistics, 2019: 4171-4186.
[21] 刘奕含, 宁念文, 杨东霖, 等. 面向城市交通的动态知识图谱综述： 构建、 表示与应用[J]. 地球信息科学学报, 2024, 26(4): 946-966. LIU Y H, NING N W, YANG D L, et al. Survey of dynamic knowledge graph for urban traffic: Construction, representation and application [J]. Journal of Geo-information Science, 2024, 26(4): 946-966. (in Chinese)
[22] DONG J W, JING X W, LU X, et al. Process knowledge graph modeling techniques and application methods for ship heterogeneous models [J]. Scientific Reports, 2022, 12(1): 2911.
[23] SCHICK T, SCHÜTZE H. True few-shot learning with prompts： A real-world perspective [J]. Transactions of the Association for Computational Linguistics, 2022, 10: 716-731.
[24] HAN X, ZHAO W L, DING N, et al. PTR: Prompt tuning with rules for text classification [J]. AI Open, 2022, 3: 182-192.
[25] DU Z X, QIAN Y J, LIU X, et al. GLM: General language model pretraining with autoregressive blank infilling [C]//Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. Dublin, Ireland: Association for Computational Linguistics, 2022: 320-335.
[26] LEE D H, KADAKIA A, TAN K, et al. Good examples make a faster learner: Simple demonstration-based learning for low-resource NER [C]//Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. Dublin, Ireland: Association for Computational Linguistics, 2022: 2687-2700.
[27] JOULIN A, GRAVE E, BOJANOWSKI P, et al. Bag of tricks for efficient text classification [C]//Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics. Valencia, Spain: Association for Computational Linguistics, 2017: 427-431.
[28] LEWIS P, PEREZ E, PIKTUS A, et al. Retrieval-augmented generation for knowledge-intensive NLP tasks [C]//Proceedings of the 34th International Conference on Neural Information Processing Systems. Vancouver, Canada: Curran Associates Inc., 2020: 793.
[29] WEN Y L, WANG Z F, SUN J M. MindMap: Knowledge graph prompting sparks graph of thoughts in large language models [C]//Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. Bangkok, Thailand: Association for Computational Linguistics, 2024: 10370-10388.
