Evaluation and mitigation of shortcut learning in multimodal misinformation detection

QIAO Yuanhao, WANG Jiacheng, WANG Li, SUN Ziming, NING Zefei, MA Zhenxuan

Journal of Tsinghua University(Science and Technology) ›› 2026, Vol. 66 ›› Issue (11) : 2311-2324.

PDF(3575 KB)
PDF(3575 KB)
Journal of Tsinghua University(Science and Technology) ›› 2026, Vol. 66 ›› Issue (11) : 2311-2324. DOI: 10.16511/j.cnki.qhdxxb.2026.21.007
COMPUTER SCIENCE AND TECHNOLOGY

Evaluation and mitigation of shortcut learning in multimodal misinformation detection

  • {{article.zuoZhe_EN}}
Author information +
History +

Abstract

[Objective] Multimodal misinformation detection models achieve promising performance by jointly exploiting textual, visual, and cross-modal information. However, a model that achieves high accuracy on conventional test sets has not necessarily learned stable evidence of information veracity. During training, models may rely on noncausal shortcut cues correlated with real/fake labels, such as writing style, source markers, image quality, interface traces, or image-text consistency. In real-world deployment, these correlations may change, substantially degrading detection performance. This problem is further complicated in multimodal settings because shortcuts can originate from text, images, and cross-modal interactions, while generative models can also manipulate these superficial attributes without substantially changing the underlying event content. Therefore, this paper proposes M2S-Bench, a unified framework for evaluating shortcut learning in multimodal misinformation detection. [Methods] M2S-Bench considers two evaluation settings: internal shortcut induction and external shortcut injection. It examines five representative shortcut attributes: text style, watermarks and source markers, image quality, user interface traces, and cross-modal consistency. Internal shortcut induction identifies shortcut attributes naturally present in the original datasets and reorganizes samples so that the shortcut-label correspondence observed during training is reversed during testing. External shortcut injection actively changes the target shortcut attribute while keeping the veracity label unchanged. Text style is modified through large language model rewriting; visual shortcuts are changed through controlled image processing or editing; and cross-modal consistency is manipulated from the image and text sides. A quality-control process combining model-based labeling, rule-based filtering, and human sampling review reduces annotation noise and unintended semantic changes. Changes in BERTScore-F1 and image-text similarity are used to verify semantic preservation and manipulation controllability. Six models are evaluated on Weibo21 and FineFake: BERT+ResNet, CLIP, MMBT, C3N, Qwen2.5-VL-7B, and Qwen2.5-VL-72B. For mitigation, a data-centric dual-view consistency strategy is adopted. Trainable models use two views with identical veracity labels but different shortcut attributes, whereas multimodal large language models use dual-view examples and prompt constraints without updating parameters. A new Dual-view Aug baseline, which uses the same paired data without consistency constraints, isolates the effect of data intervention from that of consistency regularization. [Results] The results show that multimodal misinformation detectors widely depend on noncausal cues. Under internal shortcut induction, user interface traces cause the largest average degradation, decreasing accuracy by 0.332 8 on Weibo21 and 0.412 3 on FineFake. Other shortcut types also produce different decreases, demonstrating shortcut dependence at textual, visual, and cross-modal levels. External shortcut injection causes stronger effects. Text-style injection causes the largest average accuracy reductions: 0.592 4 on Weibo21 and 0.645 0 on FineFake. User interface traces reduce accuracy by 0.327 3 and 0.451 7, respectively, whereas image-quality manipulation causes reductions of 0.237 2 and 0.233 4, respectively. Text-side cross-modal consistency manipulation also has a stronger effect than image-side manipulation. Model-level results show that conventional trainable multimodal models are generally more sensitive to strong shortcuts, whereas the Qwen2.5-VL models are relatively stable under many visual shortcut settings. In mitigation experiments, Dual-view Aug improves average accuracy by 17.77 and 24.24 percentage points on internal and external shortcut tests, respectively. Adding consistency constraints further increases the gains to 19.20 and 24.79 percentage points, respectively, while normal-test accuracy changes by only about +0.94 percentage points on average. [Conclusions] Multimodal misinformation detectors not only inherit text-style shortcuts but also develop shortcut dependence on visual appearance and image-text interactions. M2S-Bench provides a unified way to expose these hidden robustness risks under naturally occurring bias and active content manipulation. The mitigation results show that dual-view data construction is the main source of robustness improvement, while prediction-consistency constraints provide additional stabilization. The proposed framework supports more reliable evaluation and development of multimodal misinformation detection systems for real-world deployment.

Key words

multimodal misinformation detection / shortcut learning / robustness evaluation / consistency learning / multimodal large language models

Cite this article

Download Citations
QIAO Yuanhao, WANG Jiacheng, WANG Li, SUN Ziming, NING Zefei, MA Zhenxuan. Evaluation and mitigation of shortcut learning in multimodal misinformation detection[J]. Journal of Tsinghua University(Science and Technology). 2026, 66(11): 2311-2324 https://doi.org/10.16511/j.cnki.qhdxxb.2026.21.007

References

[1] WU G Y, ZENG Q T, LU L K, et al. Multi-domain feature enhanced adaptive fusion network for multi-modal fake news detection[J]. Multimedia Systems, 2026, 32(3): 164.
[2] GUPTA A K, SINGH M P. Multimodal fake news detection using semantic-visual alignment and attention fusion[J]. SN Computer Science, 2026, 7(4): 305.
[3] GUO H, MA Z H, ZENG Z, et al. Each fake news is fake in its own way: An attribution multi-granularity benchmark for multimodal fake news detection[C]//Proceedings of the Thirty-Ninth AAAI Conference on Artificial Intelligence. Philadelphia, USA: AAAI, 2025: 228-236.
[4] LU W H, TONG Y, YE Z Q. DAMMFND: Domain-aware multimodal multi-view fake news detection[C]//Proceedings of the Thirty-Ninth AAAI Conference on Artificial Intelligence. Philadelphia, USA: AAAI, 2025: 559-567.
[5] LIU Y F, LIU Y K, LI Z L, et al. Modality interactive mixture-of- experts for fake news detection[C]//Proceedings of the ACM on Web Conference 2025. Sydney, Australia: Association for Computing Machinery, 2025: 5139-5150.
[6] 周帅帅, 朱恩昌, 高盛祥, 等. 基于联邦知识蒸馏的跨语言社交媒体事件检测[C]//第二十三届全国计算语言学大会论文集(卷1:主会). 太原: 中国中文信息学会, 2024: 467-480. ZHOU S S, ZHU E C, GAO S X, et al. Cross-lingual social event detection based on federated knowledge distillation[C]// Proceedings of the 23rd Chinese National Conference on Computational Linguistics (Volume 1: Main Conference). Taiyuan, China: Chinese Information Processing Society of China, 2024: 467- 480. (in Chinese)
[7] WANG Y H, WANG L, YANG Y J, et al. Detecting fake news by enhanced text representation with multi-EDU-structure awareness[J]. Expert Systems with Applications, 2022, 206: 117781.
[8] YANG Y J, WANG Y H, WANG L, et al. PostCom2DR: Utilizing information from post and comments to detect rumors[J]. Expert Systems with Applications, 2022, 189: 116071.
[9] CHEN Z M, WANG L, ZHU X F, et al. TSNN: A topic and structure aware neural network for rumor detection[J]. Neurocomputing, 2023, 531: 114-124.
[10] WAN H R, LUO M N, SU Z X, et al. On the risk of evidence pollution for malicious social text detection in the era of LLMs[C]//Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Vienna, Austria: Association for Computational Linguistics, 2025: 9731-9761.
[11] WAN H R, WU J Y, LUO M N, et al. Truth over tricks: Measuring and mitigating shortcut learning in misinformation detection[C]//Proceedings of the Thirty-Ninth Annual Conference on Neural Information Processing Systems. San Diego, USA: Curran Associates, Inc., 2025: 141018-141060.
[12] WAN H R, LUO M N, MA Z H, et al. How do social bots participate in misinformation spread? A comprehensive dataset and analysis[C]//Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Suzhou, China: Association for Computational Linguistics, 2025: 31493- 31516.
[13] WANG Y Q, MA F L, JIN Z W, et al. EANN: Event adversarial neural networks for multi-modal fake news detection[C]// Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. London, UK: Association for Computing Machinery, 2018: 849-857.
[14] WANG H T, ZOU H, ZHOU X G, et al. Learning feasible causal algorithmic recourse: A prior structural knowledge free approach[C]//Proceedings of the ACM on Web Conference 2025. Sydney, Australia: Association for Computing Machinery, 2025: 4507-4518.
[15] CHEN Y X, LI D S, ZHANG P, et al. Cross-modal ambiguity learning for multimodal fake news detection[C]//Proceedings of the ACM Web Conference 2022. Association for Computing Machinery, 2022: 2897-2905.
[16] 肖聪, 刘璟, 王明文, 等. 基于多视角特征融合的多模态虚假新闻检测[J]. 中文信息学报, 2025, 39(4): 126-137. XIAO C, LIU J, WANG M W, et al. Multi-modal fake news detection based on multi-view feature fusion[J]. Journal of Chinese Information Processing, 2025, 39(4): 126-137. (in Chinese)
[17] LI F X, WU J Y, FU T C, et al. Drifting away from truth: GenAI -driven news diversity challenges LVLM-based misinformation detection[C]//Proceedings of the Fortieth AAAI Conference on Artificial Intelligence. Singapore: AAAI, 2026: 588-596.
[18] MA Z H, LUO M N, ZENG Z, et al. Graphing the truth: Harnessing causal insights for advanced multimodal fake news detection[J]. IEEE Transactions on Information Forensics and Security, 2025, 20: 12934-12949.
[19] LU M L, HE Z J, GUO Y H, et al. LLM-driven adversarial example synthesis for emerging topic rumor detection on social media[J]. IEEE Transactions on Knowledge and Data Engineering, 2026, 38(4): 2340-2353.
[20] WU J Y, GUO J F, HOOI B. Fake news in sheep’s clothing: Robust fake news detection against LLM-empowered style attacks[C]//Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. Barcelona, Spain: Association for Computing Machinery, 2024: 3367-3378.
[21] NAN Q, CAO J, ZHU Y C, et al. MDFEND: Multi-domain fake news detection[C]//Proceedings of the 30th ACM International Conference on Information & Knowledge Management. Association for Computing Machinery, 2021: 3343-3347.
[22] ZHOU Z Y, ZHANG X M, ZHANG L T, et al. FineFake: A knowledge-enriched dataset for fine-grained multi-domain fake news detection[J]. Information Fusion, 2026, 132: 104253.
[23] DEVLIN J, CHANG M W, LEE K, et al. BERT: Pre-training of deep bidirectional transformers for language understanding[C]// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1(Long and Short Papers). Minneapolis, USA: Association for Computational Linguistics, 2019: 4171-4186.
[24] HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE, 2016: 770-778.
[25] RADFORD A, KIM J W, HALLACY C, et al. Learning transferable visual models from natural language supervision[C]// Proceedings of the 38th International Conference on Machine Learning. PMLR, 2021: 8748-8763.
[26] KIELA D, BHOOSHAN S, PEREZ E, et al. Supervised multimodal bitransformers for classifying images and text [C]//Workshop on Visually Grounded Interaction and Language at the 33rd Conference on Neural Information Processing Systems. Vancouver, Canada, 2019.
[27] QIAO J, LI X H, GAO C, et al. Improving multimodal fake news detection by leveraging cross-modal content correlation[J]. Information Processing & Management, 2025, 62(5): 104120.
PDF(3575 KB)

Accesses

Citation

Detail

Sections
Recommended

/

〈 〉