识别模型对非因果捷径线索的依赖有助于评估其真实部署可靠性并指导鲁棒检测方法设计,为此提出多模态虚假信息捷径学习测评框架M2S-Bench。本文从内部捷径诱导和外部捷径注入2种测评设置出发,围绕文本风格、水印与来源标记、图像质量、用户界面痕迹和跨模态一致性等5类捷径属性,通过控制训练与测试阶段捷径属性和真假标签的对应关系,检验模型对非因果线索的依赖程度。在Weibo21和FineFake数据集上的实验表明,测试阶段对应关系发生反转后,多数模型性能明显下降,其中外部文本风格注入和内部用户界面痕迹诱导的平均准确率最高降幅分别为64.50和41.23百分点。本文进一步采用双视角一致性训练进行缓解,在基本保持正常测试性能的同时,使模型在内部捷径诱导设置下的准确率平均提升19.20百分点,在外部捷径注入设置下的准确率平均提升24.79百分点。结果表明,多模态检测模型不仅延续文本风格捷径,还会引入视觉端和图文交互捷径;双视角一致性约束能够有效提升模型的鲁棒性。
Abstract
[Objective] Multimodal misinformation detection models achieve promising performance by jointly exploiting textual, visual, and cross-modal information. However, a model that achieves high accuracy on conventional test sets has not necessarily learned stable evidence of information veracity. During training, models may rely on noncausal shortcut cues correlated with real/fake labels, such as writing style, source markers, image quality, interface traces, or image-text consistency. In real-world deployment, these correlations may change, substantially degrading detection performance. This problem is further complicated in multimodal settings because shortcuts can originate from text, images, and cross-modal interactions, while generative models can also manipulate these superficial attributes without substantially changing the underlying event content. Therefore, this paper proposes M2S-Bench, a unified framework for evaluating shortcut learning in multimodal misinformation detection. [Methods] M2S-Bench considers two evaluation settings: internal shortcut induction and external shortcut injection. It examines five representative shortcut attributes: text style, watermarks and source markers, image quality, user interface traces, and cross-modal consistency. Internal shortcut induction identifies shortcut attributes naturally present in the original datasets and reorganizes samples so that the shortcut-label correspondence observed during training is reversed during testing. External shortcut injection actively changes the target shortcut attribute while keeping the veracity label unchanged. Text style is modified through large language model rewriting; visual shortcuts are changed through controlled image processing or editing; and cross-modal consistency is manipulated from the image and text sides. A quality-control process combining model-based labeling, rule-based filtering, and human sampling review reduces annotation noise and unintended semantic changes. Changes in BERTScore-F1 and image-text similarity are used to verify semantic preservation and manipulation controllability. Six models are evaluated on Weibo21 and FineFake: BERT+ResNet, CLIP, MMBT, C3N, Qwen2.5-VL-7B, and Qwen2.5-VL-72B. For mitigation, a data-centric dual-view consistency strategy is adopted. Trainable models use two views with identical veracity labels but different shortcut attributes, whereas multimodal large language models use dual-view examples and prompt constraints without updating parameters. A new Dual-view Aug baseline, which uses the same paired data without consistency constraints, isolates the effect of data intervention from that of consistency regularization. [Results] The results show that multimodal misinformation detectors widely depend on noncausal cues. Under internal shortcut induction, user interface traces cause the largest average degradation, decreasing accuracy by 0.332 8 on Weibo21 and 0.412 3 on FineFake. Other shortcut types also produce different decreases, demonstrating shortcut dependence at textual, visual, and cross-modal levels. External shortcut injection causes stronger effects. Text-style injection causes the largest average accuracy reductions: 0.592 4 on Weibo21 and 0.645 0 on FineFake. User interface traces reduce accuracy by 0.327 3 and 0.451 7, respectively, whereas image-quality manipulation causes reductions of 0.237 2 and 0.233 4, respectively. Text-side cross-modal consistency manipulation also has a stronger effect than image-side manipulation. Model-level results show that conventional trainable multimodal models are generally more sensitive to strong shortcuts, whereas the Qwen2.5-VL models are relatively stable under many visual shortcut settings. In mitigation experiments, Dual-view Aug improves average accuracy by 17.77 and 24.24 percentage points on internal and external shortcut tests, respectively. Adding consistency constraints further increases the gains to 19.20 and 24.79 percentage points, respectively, while normal-test accuracy changes by only about +0.94 percentage points on average. [Conclusions] Multimodal misinformation detectors not only inherit text-style shortcuts but also develop shortcut dependence on visual appearance and image-text interactions. M2S-Bench provides a unified way to expose these hidden robustness risks under naturally occurring bias and active content manipulation. The mitigation results show that dual-view data construction is the main source of robustness improvement, while prediction-consistency constraints provide additional stabilization. The proposed framework supports more reliable evaluation and development of multimodal misinformation detection systems for real-world deployment.
关键词
多模态虚假信息检测 /
捷径学习 /
鲁棒性评测 /
一致性学习 /
多模态大模型
Key words
multimodal misinformation detection /
shortcut learning /
robustness evaluation /
consistency learning /
multimodal large language models
{{custom_sec.title}}
{{custom_sec.title}}
{{custom_sec.content}}
参考文献
[1] WU G Y, ZENG Q T, LU L K, et al. Multi-domain feature enhanced adaptive fusion network for multi-modal fake news detection[J]. Multimedia Systems, 2026, 32(3): 164.
[2] GUPTA A K, SINGH M P. Multimodal fake news detection using semantic-visual alignment and attention fusion[J]. SN Computer Science, 2026, 7(4): 305.
[3] GUO H, MA Z H, ZENG Z, et al. Each fake news is fake in its own way: An attribution multi-granularity benchmark for multimodal fake news detection[C]//Proceedings of the Thirty-Ninth AAAI Conference on Artificial Intelligence. Philadelphia, USA: AAAI, 2025: 228-236.
[4] LU W H, TONG Y, YE Z Q. DAMMFND: Domain-aware multimodal multi-view fake news detection[C]//Proceedings of the Thirty-Ninth AAAI Conference on Artificial Intelligence. Philadelphia, USA: AAAI, 2025: 559-567.
[5] LIU Y F, LIU Y K, LI Z L, et al. Modality interactive mixture-of- experts for fake news detection[C]//Proceedings of the ACM on Web Conference 2025. Sydney, Australia: Association for Computing Machinery, 2025: 5139-5150.
[6] 周帅帅, 朱恩昌, 高盛祥, 等. 基于联邦知识蒸馏的跨语言社交媒体事件检测[C]//第二十三届全国计算语言学大会论文集(卷1:主会). 太原: 中国中文信息学会, 2024: 467-480. ZHOU S S, ZHU E C, GAO S X, et al. Cross-lingual social event detection based on federated knowledge distillation[C]// Proceedings of the 23rd Chinese National Conference on Computational Linguistics (Volume 1: Main Conference). Taiyuan, China: Chinese Information Processing Society of China, 2024: 467- 480. (in Chinese)
[7] WANG Y H, WANG L, YANG Y J, et al. Detecting fake news by enhanced text representation with multi-EDU-structure awareness[J]. Expert Systems with Applications, 2022, 206: 117781.
[8] YANG Y J, WANG Y H, WANG L, et al. PostCom2DR: Utilizing information from post and comments to detect rumors[J]. Expert Systems with Applications, 2022, 189: 116071.
[9] CHEN Z M, WANG L, ZHU X F, et al. TSNN: A topic and structure aware neural network for rumor detection[J]. Neurocomputing, 2023, 531: 114-124.
[10] WAN H R, LUO M N, SU Z X, et al. On the risk of evidence pollution for malicious social text detection in the era of LLMs[C]//Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Vienna, Austria: Association for Computational Linguistics, 2025: 9731-9761.
[11] WAN H R, WU J Y, LUO M N, et al. Truth over tricks: Measuring and mitigating shortcut learning in misinformation detection[C]//Proceedings of the Thirty-Ninth Annual Conference on Neural Information Processing Systems. San Diego, USA: Curran Associates, Inc., 2025: 141018-141060.
[12] WAN H R, LUO M N, MA Z H, et al. How do social bots participate in misinformation spread? A comprehensive dataset and analysis[C]//Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Suzhou, China: Association for Computational Linguistics, 2025: 31493- 31516.
[13] WANG Y Q, MA F L, JIN Z W, et al. EANN: Event adversarial neural networks for multi-modal fake news detection[C]// Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. London, UK: Association for Computing Machinery, 2018: 849-857.
[14] WANG H T, ZOU H, ZHOU X G, et al. Learning feasible causal algorithmic recourse: A prior structural knowledge free approach[C]//Proceedings of the ACM on Web Conference 2025. Sydney, Australia: Association for Computing Machinery, 2025: 4507-4518.
[15] CHEN Y X, LI D S, ZHANG P, et al. Cross-modal ambiguity learning for multimodal fake news detection[C]//Proceedings of the ACM Web Conference 2022. Association for Computing Machinery, 2022: 2897-2905.
[16] 肖聪, 刘璟, 王明文, 等. 基于多视角特征融合的多模态虚假新闻检测[J]. 中文信息学报, 2025, 39(4): 126-137. XIAO C, LIU J, WANG M W, et al. Multi-modal fake news detection based on multi-view feature fusion[J]. Journal of Chinese Information Processing, 2025, 39(4): 126-137. (in Chinese)
[17] LI F X, WU J Y, FU T C, et al. Drifting away from truth: GenAI -driven news diversity challenges LVLM-based misinformation detection[C]//Proceedings of the Fortieth AAAI Conference on Artificial Intelligence. Singapore: AAAI, 2026: 588-596.
[18] MA Z H, LUO M N, ZENG Z, et al. Graphing the truth: Harnessing causal insights for advanced multimodal fake news detection[J]. IEEE Transactions on Information Forensics and Security, 2025, 20: 12934-12949.
[19] LU M L, HE Z J, GUO Y H, et al. LLM-driven adversarial example synthesis for emerging topic rumor detection on social media[J]. IEEE Transactions on Knowledge and Data Engineering, 2026, 38(4): 2340-2353.
[20] WU J Y, GUO J F, HOOI B. Fake news in sheep’s clothing: Robust fake news detection against LLM-empowered style attacks[C]//Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. Barcelona, Spain: Association for Computing Machinery, 2024: 3367-3378.
[21] NAN Q, CAO J, ZHU Y C, et al. MDFEND: Multi-domain fake news detection[C]//Proceedings of the 30th ACM International Conference on Information & Knowledge Management. Association for Computing Machinery, 2021: 3343-3347.
[22] ZHOU Z Y, ZHANG X M, ZHANG L T, et al. FineFake: A knowledge-enriched dataset for fine-grained multi-domain fake news detection[J]. Information Fusion, 2026, 132: 104253.
[23] DEVLIN J, CHANG M W, LEE K, et al. BERT: Pre-training of deep bidirectional transformers for language understanding[C]// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1(Long and Short Papers). Minneapolis, USA: Association for Computational Linguistics, 2019: 4171-4186.
[24] HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE, 2016: 770-778.
[25] RADFORD A, KIM J W, HALLACY C, et al. Learning transferable visual models from natural language supervision[C]// Proceedings of the 38th International Conference on Machine Learning. PMLR, 2021: 8748-8763.
[26] KIELA D, BHOOSHAN S, PEREZ E, et al. Supervised multimodal bitransformers for classifying images and text [C]//Workshop on Visually Grounded Interaction and Language at the 33rd Conference on Neural Information Processing Systems. Vancouver, Canada, 2019.
[27] QIAO J, LI X H, GAO C, et al. Improving multimodal fake news detection by leveraging cross-modal content correlation[J]. Information Processing & Management, 2025, 62(5): 104120.
基金
国家自然科学基金区域创新发展联合基金(U22A20167)