Prototype-based continuous complex relation extraction network model

Yingli LIU, Jinyu ZHANG, Shaojie WEN, Tao SHEN

Journal of Tsinghua University(Science and Technology) ›› 2025, Vol. 65 ›› Issue (11) : 2245-2258.

PDF(4942 KB)
PDF(4942 KB)
Journal of Tsinghua University(Science and Technology) ›› 2025, Vol. 65 ›› Issue (11) : 2245-2258. DOI: 10.16511/j.cnki.qhdxxb.2025.21.035
Computer Science and Technology

Prototype-based continuous complex relation extraction network model

Author information +
History +

Abstract

Objective: Existing methods for continuous relation extraction (CRE) face remarkable challenges in complex Chinese semantic environments, particularly the problem of catastrophic forgetting when learning new tasks while retaining knowledge from old tasks. Traditional approaches often retrain models on a combination of historical and new data, leading to inefficiencies and resource constraints as data volumes increase. To address these limitations, this study aims to develop a robust CRE model that can efficiently learn new knowledge while preserving historical relationships, even in scenarios with complex sentence structures, overlapping entities, and imbalanced data distributions. The proposed model integrates prototype representations, memory replay strategies, and contrastive learning to enhance feature discrimination and stability in embedding spaces, thereby improving the classification performance across single-domain and cross-domain datasets. Methods: The proposed prototype-based continuous complex relation extraction network (PBCRE-Net) model consists of two primary stages: initial training and memory replay, designed to mitigate catastrophic forgetting and improve adaptability in dynamic learning environments. The initial training includes: 1) Entity-aware feature extraction: Input texts are processed using a pretrained BERT model to generate contextual embeddings with special tokens ([E11], [E12], [E21], and [E22]) that mark entity boundaries. 2) Supervised contrastive learning: A dual-head classifier (classification and contrast) is employed to minimize intraclass distances and maximize interclass distances in the embedding space. This objective is achieved through a combination of cross-entropy loss and contrastive loss. 3) Prototype generation: For each relation category, representative samples are selected via K-means clustering and their prototypes are computed as weighted averages of cluster centroids to capture category-specific features. The memory replay includes: 1) Memory sample selection: Memory modules store exemplars from previous tasks using K-means clustering. Weights are assigned based on cluster distribution to balance sampling during replay. 2) Memory augmentation: To prevent overfitting, synthetic samples are generated by swapping entity pairs or appending unrelated sentences to existing exemplars, thereby expanding the memory pool. 3) Consistency Loss: During replay, an embedding consistency constraint is applied to maintain stability in the embedding space across tasks. 4) Joint optimization: The model is trained on a mixture of new task data, historical memory samples, and augmented samples, combining cross-entropy loss and consistency loss. Results: Experimental evaluations on CMeIE (medical domain) and ASaRED (alloy domain) datasets demonstrate the superiority of PBCRE-Net in complex CRE scenarios. Single-domain Performance: On the CMeIE dataset, PBCRE-Net achieved an average accuracy (ACC) of 84.27% and a macro F1-score (Macro-F1) of 81.93% across 10 incremental tasks. Notably, the proposed model outperformed baseline models, such as EMAR and CRL, by 3%-5% in subsequent tasks (T8—T10), where catastrophic forgetting is highly severe. The model effectively handled triplet overlap (e.g., entity-pair and single-entity overlaps) and class imbalance, an objective accomplished through prototype-based contrastive learning and memory augmentation. Cross-domain adaptability: In cross-domain experiments combining CMeIE and ASaRED, PBCRE-Net maintained an ACC of 74.16% and Macro F1 of 69.34% across 10 tasks, considerably surpassing competing models (e.g., CRECL and DPCRE). The memory replay mechanism and consistency loss ensured stable embedding spaces despite domain shifts, thus reducing catastrophic forgetting in critical relation categories like material composition and alloy property. Robustness to memory constraints: Reducing the memory size from 20 to 5 samples per task decreased the performance of the proposed model by 15%, yet PBCRE-Net outperformed the alternatives under constrained memory conditions. This highlights its efficiency in real-world scenarios with limited storage. Conclusions: This study introduces PBCRE-Net, a novel CRE framework that mitigates catastrophic forgetting through prototype representations, memory replay, and contrastive learning. Key contributions include a supervised contrastive learning strategy to enhance feature discriminability in complex semantics and a memory augmentation mechanism to mitigate overfitting and stabilize embedding spaces. Superior performance in single-domain and cross-domain CRE tasks was validated by extensive experiments. Future work will extend PBCRE-Net to multilingual settings via cross-lingual transfer learning and incorporate physical constraints to improve relation extraction accuracy in scientific domains. In addition, addressing polysemy through semantic alignment techniques will further enhance the applicability of PBCRE-Net.

Key words

continuous learning / relation extraction / prototype representation / complex semantics

Cite this article

Download Citations
Yingli LIU , Jinyu ZHANG , Shaojie WEN , et al. Prototype-based continuous complex relation extraction network model[J]. Journal of Tsinghua University(Science and Technology). 2025, 65(11): 2245-2258 https://doi.org/10.16511/j.cnki.qhdxxb.2025.21.035

References

1
张林峰, 孙伟杰, 李鑫宇, 等. 科学智能(AI4S)全球发展观察与展望[R]. 北京: 科学智能研究院、深势科技、络绎科学, 2023.
ZHANG L F, SUN W J, LI X Y, et al. AI for science global outlook[R]. Beijing: Beijing Academy of AI for Science, DP Technology, Science L, 2023. (in Chinese)
2
KIM H , CHOI H , KANG D J , et al. Materials discovery with extreme properties via reinforcement learning-guided combinatorial chemistry[J]. Chemical Science, 2024, 15 (21): 7908- 7925.
3
QIAO B , ZOU Z Y , HUANG Y , et al. A joint model for entity and relation extraction based on BERT[J]. Neural Computing and Applications, 2022, 34 (5): 3471- 3481.
4
YUAN J Y , ZHANG F Y , QIU Y M , et al. Document-level biomedical relation extraction via hierarchical tree graph and relation segmentation module[J]. Bioinformatics, 2024, 40 (7): btae418.
5
HE J Y , NGUYEN D Q , AKHONDI S A , et al. ChEMU 2020:Natural language processing methods are effective for information extraction from chemical patents[J]. Frontiers in Research Metrics and Analytics, 2021, 6, 654438.
6
LI Y, HE J Y, YOSHIKAWA H, et al. End-to-end chemical reaction extraction from patents[C]//Proceedings of PatentSemTech'22. New York, USA: ACM, 2022: 123-130.
7
MULLICK A , GHOSH A , CHAITANYA G S , et al. MatSciRE: Leveraging pointer networks to automate entity and relation extraction for material science knowledge base construction[J]. Computational Materials Science, 2024, 233, 112659.
8
胡明昊, 王芳, 徐先涛, 等. 国防科技领域两阶段开放信息抽取方法[J]. 清华大学学报(自然科学版), 2023, 63 (9): 1309- 1316.
HU M H , WANG F , XU X T , et al. Two-stage open information extraction method for the defence technology field[J]. Journal of Tsinghua University (Science and Technology), 2023, 63 (9): 1309- 1316.
9
张江石, 李泳暾, 吴静茹, 等. 煤矿事故原因智能分析方法研究与应用[J]. 清华大学学报(自然科学版), 2025, 65 (3): 555- 568.
ZHANG J S , LI Y T , WU J R , et al. Research and application of intelligent methods for analyzing the causes of coal mining accidents[J]. Journal of Tsinghua University (Science and Technology), 2025, 65 (3): 555- 568.
10
LIU Y L , NIU C , WANG Z , et al. Machine learning in materials genome initiative: A review[J]. Journal of Materials Science & Technology, 2020, 57, 113- 122.
11
ZHANG Y , HU W , YAO W , et al. Offline data-driven multiobjective optimization evolutionary algorithm based on generative adversarial network[J]. IEEE Transactions on Evolutionary Computation, 2024, 28 (2): 293- 306.
12
GAO Q , WANG W B , REN J Q , et al. Optimizing strength-ductility in NiCoMn medium entropy alloys with atomic-scale rapid composition design[J]. Journal of Materials Science & Technology, 2025, 215, 71- 85.
13
LIU Y L , CUI Y T , ZHOU H H , et al. Machine learning based methods for materials inverse design: A review[J]. Computers, Materials & Continua, 2025, 82 (2): 1463- 1492.
14
王立平, 张超, 蔡恩磊, 等. 面向自主工业软件的知识提取和知识库构建方法[J]. 清华大学学报(自然科学版), 2022, 62 (5): 978- 986.
WANG L P , ZHANG C , CAI E L , et al. Knowledge extraction and knowledge base construction method from industrial software packages[J]. Journal of Tsinghua University (Science and Technology), 2022, 62 (5): 978- 986.
15
ZENG D J, LIU K, LAI S W, et al. Relation classification via convolutional deep neural network[C]//Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers. Dublin, Ireland: Dublin City University and Association for Computational Linguistics, 2014: 2335-2344.
16
LI F, ZHANG M S, FU G H, et al. A Bi-LSTM-RNN model for relation classification using low-cost sequence features[EB/OL]. (2016-08-27)[2023-01-01]. https://arxiv.org/abs/1608.07720.
17
XU Y, MOU L L, LI G, et al. Classifying relations via long short term memory networks along shortest dependency paths[C]//Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Lisbon, Portugal: Association Linguistics, 2015: 1785-1794.
18
SUN C Z, GONG Y Y, WU Y B, et al. Joint type inference on entities and relations via graph convolutional networks[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Florence, Italy: Association for Computational Linguistics, 2019: 1361-1370.
19
KATIYAR A, CARDIE C. Going out on a limb: Joint extraction of entity mentions and relations without dependency trees[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Vancouver, Canada: Association for Computational Linguistics, 2017: 917-928.
20
SUI D , ZENG X R , CHEN Y B , et al. Joint entity and relation extraction with set prediction networks[J]. IEEE Transactions on Neural Networks and Learning Systems, 2024, 35 (9): 12784- 12795.
21
WANG H, XIONG W H, YU M, et al. Sentence embedding alignment for lifelong relation extraction[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Minneapolis, Minnesota: Association for Computational Linguistics, 2019: 796-806.
22
HAN X, DAI Y, GAO T Y, et al. Continual relation learning via episodic memory activation and reconsolidation[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 2020: 6429-6440.
23
ZHAO K, XU H, YANG J G, et al. Consistent representation learning for continual relation extraction[C]//Findings of the Association for Computational Linguistics: ACL 2022. Dublin, Ireland: Association for Computational Linguistics, 2022: 3402-3411.
24
CUI L, YANG D Q, YU J X, et al. Refining sample embeddings with relation prototypes to enhance continual relation extraction[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, 2021: 232-243.
25
KIRKPATRICK J , PASCANU R , RABINOWITZ N , et al. Overcoming catastrophic forgetting in neural networks[J]. Proceedings of the National Academy of Sciences of the United States of America, 2017, 114 (13): 3521- 3526.
26
ZENKE F, POOLE B, GANGULI S. Continual learning through synaptic intelligence[C]//Proceedings of the 34th International Conference on Machine Learning. Sydney, Australia: PMLR, 2017: 3987-3995.
27
ZHAO B W, XIAO X, GAN G J, et al. Maintaining discrimination and fairness in class incremental learning[C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle, WA, USA: IEEE, 2020: 13205-13214.
28
FERNANDO C, BANARSE D, BLUNDELL C, et al. PathNet: Evolution channels gradient descent in super neural networks[EB/OL]. (2017-02-02)[2023-01-01]. https://arxiv.org/abs/1701.08734.
29
PHAM Q, LIU C H, HOI S C H. DualNet: Continual learning, fast and slow[C]//Proceedings of the 35th International Conference on Neural Information Processing Systems. Curran Associates Inc., 2021: 16131-16144.
30
YAN S P, XIE J W, HE X M. DER: Dynamically expandable representation for class incremental learning[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Nashville, USA: IEEE, 2021: 3013-3022.
31
MAI Z D, LI R W, KIM H, et al. Supervised contrastive replay: Revisiting the nearest class mean classifier in online class-incremental continual learning[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Nashville, USA: IEEE, 2021: 3584-3594.
32
REBUFFI S A, KOLESNIKOV A, SPERL G, et al. iCaRL: Incremental classifier and representation learning[C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, USA: IEEE, 2017: 5533-5542.
33
ZHANG H , LIANG B , YANG M , et al. Prompt-based prototypical framework for continual relation extraction[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2022, 30, 2801- 2813.
34
CHEN X L, HE K M. Exploring simple Siamese representation learning[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Nashville, USA: IEEE, 2021: 15745-15753.
35
JAISWAL A , BABU A R , ZADEH M Z , et al. A survey on contrastive self-supervised learning[J]. Technologies, 2021, 9 (1): 2.
36
WANG F, LIU H P. Understanding the behaviour of contrastive loss[C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Nashville, USA: IEEE, 2021: 2495-2504.
37
VAN DEN OORD A, LI Y Z, VINYALS O. Representation learning with contrastive predictive coding[EB/OL]. (2018-07-10)[2023-01-01]. https://arxiv.org/abs/1807.03748.
38
DENG H J , YANG Y M . Context-enriched contrastive loss: Enhancing presentation of inherent sample connections in contrastive learning framework[J]. IEEE Transactions on Multimedia, 2025, 27, 429- 441.
39
WEI Y B , XU Y , ZHU L , et al. Multi-level cross-modal contrastive learning for review-aware recommendation[J]. Expert Systems with Applications, 2024, 247, 123341.
40
HU C W, YANG D Q, JIN H L, et al. Improving continual relation extraction through prototypical contrastive learning[C]//Proceedings of the 29th International Conference on Computational Linguistics. Gyeongju, Republic of Korea: International Committee on Computational Linguistics, 2022: 1885-1895.
41
DEVLIN J, CHANG M W, LEE K, et al. BERT: Pre training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Minneapolis, Minnesota: Association for Computational Linguistics, 2019: 4171-4186.
42
ZAN H , LIU T , CHEN J , et al. Corpus construction for named-entity and entity relations for paediatric diseases[J]. Journal of Chinese Information Processing, 2020, 34 (2): 34- 56.
43
LIU Y L , WEN S J , YIN J C , et al. ASaRE-Net: Automatic information extraction from Al-Si alloy materials science literature for corpus construction[J]. Journal of Materials Science, 2024, 59 (32): 15330- 15346.
44
HUANG M Y, XIAO M, WANG L D, et al. DP-CRE: Continual relation extraction via decoupled contrastive learning and memory structure preservation[C]//Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). Torino, Italia: ELRA, 2024: 5338-5349.

RIGHTS & PERMISSIONS

All rights reserved. Unauthorized reproduction is prohibited.
PDF(4942 KB)

Accesses

Citation

Detail

Sections
Recommended

/