PromptWE: A fact-checking method based on prompt learning with explanations
ZHANG Xiangran1,2, LI Luyang2
1. Polytechnic Institute, Zhejiang University, Hangzhou 310015, China; 2. School of Information Science and Technology, Beijing Foreign Studies University, Beijing 100089, China
Abstract：[Objective] In the contemporary “We Media” era, the simplification of news production and dissemination has elevated every individual to the status of news producer and disseminator, and a large amount of false information also follows. Despite the increasing and abundant information on the Internet, the regulation of false information is relatively weak. Consequently, fact-checking is becoming more and more important work, while traditional related work tends to simply label predictions without explaining the reason for the label. The generated explanation in a few studies is also relatively primitive which is hard to comprehend. Because Fact-checking demands a substantial amount of common sense, reasoning, and background knowledge about claims. Prompt learning may further utilize common sense and reasoning ability in pre-trained language models. It may also incorporate the relevant information or additional details within the explanation for claims. In all, it is essential to generate high-quality smooth explanations and further leverage generated explanations for improving classification performance through prompt learning. [Methods] To address this multifaceted challenge, we propose the PromptWE model (Prompt With Evidence) that uses the prompt learning paradigm to integrate auto-generated explanations with claims. We not only provide natural language explanations that enhance the explainability of the classification result but also further improve the model performance by combining explanation into prompt learning. The model performs hierarchical evidence distillation on many related new reports for every claim to obtain relevant evidence, then uses the BART-CNN model to summarize these incoherent pieces of evidence into one smooth explanation. Consequently, it integrates the claim and explanation into six self-designed templates for prompt learning. Finally, we ensemble the result from different templates to predict the authenticity of the news. Moreover, we replace the generated explanation with the professional explanation from the dataset to investigate the impact of expert evidence on the prompt learning models. [Results] Our method achieves good results on two fact-checking datasets: Liar-RAW and RAWFC. Its F1 score is 5 % higher than the state-of-the-art model on both datasets at least. We also find that ensemble learning with multiple templates can effectively improve the F1 score of the model. For explanation generation, the model has a higher ROUGE-2 score than the former model. After integrating professional evidence into the prompt templates, the model achieves significant improvement in the classification results on the two datasets, with a maximum improvement of 15 % when compared to the results of the PromptWE model. Also, we find that for multi-class classification task, the model with integrated professional evidence exhibited exhibits significant performance improvement on more challenging categories, such as half-true and mostly true. [Conclusions] Related experiments indicate that incorporating extracted explanations as supplementary background knowledge about claims, along with the common sense and reasoning abilities learned from pre-trained models, into prompt learning templates can further enhance classification performance for claim veracity. Moreover, sequentially employing the methods of hierarchical evidence extraction and text summarization makes explanations more concise, coherent, and comprehensible. Also, the explanation extracted from unrelated evidence is better suited for integration into prompt learning methods. The further improvement in classification performance after incorporating professional evidence underscores that this approach could swiftly identify accurate and informative prompt templates, facilitating subsequent more efficient utilization of general large models like ChatGPT.
张翔然, 李璐旸. PromptWE：一种融合解释的提示学习事实核查模型[J]. 清华大学学报（自然科学版）, 2024, 64(5): 760-769.
ZHANG Xiangran, LI Luyang. PromptWE: A fact-checking method based on prompt learning with explanations. Journal of Tsinghua University(Science and Technology), 2024, 64(5): 760-769.
[1]彭兰. 网络传播概论[M]. 4版. 北京:中国人民大学出版社, 2017.PENG L. Introduction to network communication[M]. 4th ed. Beijing:China Renmin University Press, 2017. (in Chinese) [2]ZHOU X Y, ZAFARANI R. A survey of fake news:Fundamental theories, detection methods, and opportunities[J]. ACM Computing Surveys, 2000, 53(5):109. [3]SHU K, CUI L M, WANG S H, et al. dEFEND:Explainable fake news detection[C]//Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. Anchorage, AK, USA:Association for Computing Machinery, 2019:395-405. [4]YANG Z W, MA J, CHEN H C, et al. A coarse-to-fine cascaded evidence-distillation neural network for explainable fake news detection[C]//Proceedings of the 29th International Conference on Computational Linguistics. Gyeongju, Republic of Korea:International Committee on Computational Linguistics, 2022:2608-2621. [5]POPAT K, MUKHERJEE S, YATES A, et al. DeClarE:Debunking fake news and false claims using evidence-aware deep learning[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Brussels, Belgium:Association for Computational Linguistics, 2018:22-32. [6]WU L W, RAO Y, SUN L, et al. Evidence inference networks for interpretable claim verification[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2021, 35(16):14058-14066. [7]YANG F, PENTYALA S K, MOHSENI S, et al. XFake:Explainable fake news detector with visualizations[C]//The World Wide Web Conference. New York, NY, USA:Association for Computing Machinery, 2019:3600-3604. [8]LU Y J, LI C T. GCAN:Graph-aware Co-attention networks for explainable fake news detection on social media[C/OL]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online:Association for Computational Linguistics, 2020:505-514. [9]NIE Y X, CHEN H N, BANSAL M. Combining fact extraction and verification with neural semantic matching networks[C]//Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence. Honolulu, Hawaii, USA:AAAI Press, 2019, 33(1):6859-6866. [10]MA J, GAO W, JOTY S, et al. Sentence-level evidence embedding for claim verification with hierarchical attention networks[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Florence, Italy:Association for Computational Linguistics, 2019:2561-2571. [11]ATANASOVA P, SIMONSEN J G, LIOMA C, et al. Generating fact checking explanations[C/OL]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online:Association for Computational Linguistics, 2020:7352-7364. [12]KOTONYA N, TONI F. Explainable automated fact-checking for public health claims[C/OL]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Online:Association for Computational Linguistics, 2020:7740-7754. [13]DEVLIN J, CHANG M W, LEE K, et al. BERT:Pre-training of deep bidirectional transformers for language understanding[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies, Volume 1(Long and Short Papers). Minneapolis, Minnesota:Association for Computational Linguistics, 2019:4171-4186. [14]LIU P F, YUAN W Z, FU J L, et al. Pre-train, prompt, and predict:A systematic survey of prompting methods in natural language processing[J]. ACM Computing Surveys, 2023, 55(9):195. [15]HU S D, DING N, WANG H D, et al. Knowledgeable prompt-tuning:Incorporating knowledge into prompt verbalizer for text classification[C]//Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1:Long Papers). Dublin, Ireland:Association for Computational Linguistics, 2022:2225-2240. [16]ZHU Y, ZHOU X K, QIANG J P, et al. Prompt-learning for short text classification[J]. arXiv preprint arXiv:2202.11345, 2022. [17]LIU X C, GAO Y, BAI Y, et al. PSP:Pre-trained soft prompts for few-shot abstractive summarization[C]//Proceedings of the 29th International Conference on Computational Linguistics. Gyeongju, Republic of Korea:International Committee on Computational Linguistics, 2022:6355-6368. [18]LEWIS M, LIU Y H, GOYAL N, et al. BART:Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension[C/OL]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online:Association for Computational Linguistics, 2020:7871-7880. [19]SANH V, DEBUT L, CHAUMOND J, et al. DistilBERT, a distilled version of BERT:smaller, faster, cheaper and lighter[J]. arXiv preprint arXiv:1910.01108, 2019. [20]LIU Y H, OTT M, GOYAL N, et al. RoBERTa:A robustly optimized bert pretraining approach[J]. arXiv preprint arXiv:1907.11692, 2019. [21]CUI L Y, WU Y, LIU J, et al. Template-based named entity recognition using BART[C/OL]//Findings of the Association for Computational Linguistics:ACL-IJCNLP 2021. Online:Association for Computational Linguistics, 2021:1835-1845. [22]DING N, CHEN Y L, HAN X, et al. Prompt-learning for fine-grained entity typing[C]//Findings of the Association for Computational Linguistics:EMNLP 2022. Abu Dhabi, United Arab Emirates:Association for Computational Linguistics, 2022:6888-6901. [23]LI X L, LIANG P. Prefix-tuning:Optimizing continuous prompts for generation[C/OL]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1:Long Papers). Online:Association for Computational Linguistics, 2021:4582-4597. [24]SCHICK T, SCHVTZE H. Exploiting cloze-questions for few-shot text classification and natural language inference[C/OL]//Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics:Main Volume. Online:Association for Computational Linguistics, 2021:255-269. [25]WANG Y X, ZHANG Y H, LI X B, et al. COVID-19 fake news detection using bidirectional encoder representations from transformers based models[J]. arXiv preprint arXiv:2109.14816, 2021. [26]DING N, HU S D, ZHAO W L, et al. OpenPrompt:An open-source framework for prompt-learning[C]//Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics:System Demonstrations. Dublin, Ireland:Association for Computational Linguistics, 2022:105-113. [27]LIN C Y. ROUGE:A package for automatic evaluation of summaries[C/OL]//Text Summarization Branches Out. Barcelona, Spain:Association for Computational Linguistics, 2004:74-81.
2024, Vol. 64 
