PDF(2941 KB)
Consistent thought tree with retrieval: An optimization framework for reasoning in large language models
Haiyan WU, Chengxiong LU, Zhidong DENG, Chaoqun SUN, Yanhong CHEN, Zhiqiang ZHANG
Journal of Tsinghua University(Science and Technology) ›› 2026, Vol. 66 ›› Issue (10) : 2045-2056.
PDF(2941 KB)
PDF(2941 KB)
Consistent thought tree with retrieval: An optimization framework for reasoning in large language models
Objective: Large language models (LLMs) excel at text generation and complex problem solving but are prone to factual and logical errors in multidimensional reasoning due to internal knowledge limitations. Although retrieval-augmented generation (RAG) mitigates this problem by integrating external knowledge, it introduces knowledge noise, multisource conflicts, and a disconnection between retrieval and reasoning. Structured approaches such as chain-of-thought and tree-of-thought enhance reasoning via explicit intermediate steps, but they rely on internal knowledge without effective fact-checking. Recent methods that fuse RAG with structured reasoning lack coordination in situations involving conflicts between external and internal knowledge, and tree architectures focus locally without ensuring cross-level consistency. To address these issues, this study develops a framework that deeply integrates retrieval with tree-structured reasoning and enforces consistency constraints to guarantee factual accuracy and logical coherence. Methods: This study proposed the Consistent Thought Tree with Retrieval (CTTR) framework, featuring three core designs. First, dynamic retrieval embedding enabled context-aware knowledge acquisition at each tree node for on-demand external knowledge fusion. For each node, a retrieval query was built from the content and context, and the top-k documents were retrieved via cosine similarity. A reading-note mechanism processed the retrieved content through the LLM to ground each branch in accurate facts. Second, knowledge consistency verification resolved conflicts by comparing the semantic consistency among three results: those obtained from internal knowledge, external retrieval, and fused synthesis. Their embeddings were computed, and similarities were ranked; finally, the highest pair was selected as the output. Irrelevant branches were pruned to prevent noise propagation. Third, node consistency evaluation applied majority voting and confidence assessment among nodes of the same level, thereby enhancing reliability via multipath convergence and avoiding single-chain deviations. The algorithm processed each tree layer through retrieval, verification, optimization, and integration to produce the optimal result. Results: Experiments encompassed four tasks using GLM4-9B and DeepSeek-R1-Distill-Llama-8B: hallucination detection (TruthfulQA with 817 questions, HalluQA Chinese with 452 questions), code generation (HumanEval with 164 problems), the 24-Game (100 problems from 4nums), and creative writing (science fiction and historical narrative evaluated by GPT-4 scoring and human review). CTTR achieved 66.50%/57.78% on GLM4-9B and 91.31%/45.56% on DeepSeek-R1-Distill-Llama-8B for hallucination detection, averaging an improvement of 2.89% over the retrieval-augmented thought tree (RATT). Code generation reached 73.17%/87.20% and 75.61%/88.41% pass@1/pass@5, showing an improvement of 3.96%. Further, the 24-Game success rates of 11% and 24% indicated an improvement of 3.50%. Creative writing scores reached 8.96/8.81 and 9.57/8.45, reflecting an improvement of 1.24%. Ablation studies confirmed that removing knowledge selection (KS) or consistent nodes (CN) degraded the performance, with KS contributing the most; this validated the necessity of dual consistency. Resource analysis showed that compared to RATT, CTTR reduced token consumption by 11% and the time required to 247.67 s by pruning redundant retrievals. Poisoning attack tests demonstrated the robustness of CTTR against malicious noise injection. Conclusions: CTTR effectively addresses the need to maintain logical consistency and factual accuracy for complex problems. By embedding retrieval into tree-structured reasoning with dual consistency evaluations, it enhances hallucination detection, code generation, mathematical reasoning, and creative writing. The results confirmed that retrieval–tree fusion strengthens complex reasoning, and that dual consistency mechanisms are task-adaptive. KS is prominent in fact-intensive tasks, while CN is valuable in open-domain generation. CTTR improves accuracy while also optimizing resource efficiency. As a training-free framework deployable without fine-tuning, CTTR reduces application costs. Future work will explore the dynamic mechanisms that adaptively switch paths by task complexity, using direct responses for simple tasks and tree reasoning for complex ones to balance efficiency and accuracy.
large language models / retrieval-augmented generation / tree of thought / knowledge conflict / consistency evaluation
| 1 |
SWARUP A, BHANDARKAR A, WILSON R, et al. From syntax to semantics: Evaluating the impact of linguistic structures on LLM-based information extraction [C]//Proceedings of the 1st Joint Workshop on Large Language Models and Structure Modeling. Vienna, Austria: ACL, 2025: 36–48.
|
| 2 |
|
| 3 |
BROWN T B, MANN B, RYDER N, et al. Language models are few-shot learners [C]//Proceedings of the 34th International Conference on Neural Information Processing Systems. Vancouver, BC, Canada: Curran Associates Inc., 2020: 1877-1901.
|
| 4 |
XU X H, CHEN H, LIN Z J, et al. TaD: A plug and-play task-aware decoding method to better adapt LLMs on downstream tasks [C]//Proceedings of the 33rd International Joint Conference on Artificial Intelligence. Jeju, Korea: IJCAI, 2024: 6587–6596.
|
| 5 |
LIN S, HILTON J, EVANS O. TruthfulQA: Measuring how models mimic human falsehoods [C]//Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Dublin, Ireland: ACL, 2022: 3214–3252.
|
| 6 |
LEWIS P, PEREZ E, PIKTUS A, et al. Retrieval-augmented generation for knowledge-intensive NLP tasks [C]//Proceedings of the 34th International Conference on Neural Information Processing Systems. Vancouver, BC, Canada: Curran Associates Inc., 2020: 9459–9474.
|
| 7 |
JIN Z R, CAO P F, CHEN Y B, et al. Tug-of-war between knowledge: Exploring and resolving knowledge conflicts in retrieval-augmented language models [C]//Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation. Torino, Italy: ELRA and ACL, 2024: 16867–16878.
|
| 8 |
JIN Z R, CAO P F, YUAN H B, et al. Cutting off the head ends the conflict: A mechanism for interpreting and mitigating knowledge conflicts in language models [C]//Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Findings Volume). Bangkok, Thailand: ACL, 2024: 1193–1215.
|
| 9 |
|
| 10 |
WEI J S, WANG X Z, SCHUURMANS D, et al. Chain-of- thought prompting elicits reasoning in large language models [C]//Proceedings of the 36th International Conference on Neural Information Processing Systems. New Orleans, LA, USA: Curran Associates Inc., 2022: 24824–24837.
|
| 11 |
YAO S Y, YU D, ZHAO J, et al. Tree of thoughts: Deliberate problem solving with large language models [C]//Proceedings of the 37th International Conference on Neural Information Processing Systems. New Orleans, LA, USA: Curran Associates Inc., 2023: 11809–11822.
|
| 12 |
JIANG Z X, PENG H Y, FENG S S, et al. LLMs can find mathematical reasoning mistakes by pedagogical chain-of- thought [C]//Proceedings of the 33rd International Joint Conference on Artificial Intelligence. Jeju, Korea: IJCAI, 2024: 3439–3447.
|
| 13 |
BI Z, ZHANG N Y, JIANG Y N, et al. When do program- of-thought works for reasoning? [C]//Proceedings of the 38th AAAI Conference on Artificial Intelligence. Vancouver, Canada: AAAI Press, 2024: 17691–17699.
|
| 14 |
WANG X, LI C, CHANG Y, et al. NegativePrompt: Leveraging psychology for large language models enhancement via negative emotional stimuli [C]//Proceedings of the 33rd International Joint Conference on Artificial Intelligence. Jeju, Korea: IJCAI, 2024: 6504–6512.
|
| 15 |
WANG Z H, LIU A J, LIN H W, et al. Rat: Retrieval augmented thoughts elicit context-aware reasoning and verification in long-horizon generation [C]//Proceedings of the 38th International Conference on Neural Information Processing Systems. Vancouver, BC, Canada: Curran Associates Inc., 2024.
|
| 16 |
ZHANG J H, WANG X T, REN W J Y, et al. RATT: A thought structure for coherent and correct LLM reasoning [C]//Proceedings of the 39th AAAI Conference on Artificial Intelligence and 37th Conference on Innovative Applications of Artificial Intelligence and Fifteenth Symposium on Educational Advances in Artificial Intelligence. Philadelphia, Pennsylvania, USA: AAAI Press, 2025: 26733–26741.
|
| 17 |
KIM G, KIM S, JEON B, et al. Tree of clarifications: Answering ambiguous questions with retrieval-augmented large language models [C]//Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Singapore, Singapore: ACL, 2023: 996–1009.
|
| 18 |
BESTA M, BLACH N, KUBICEK A, et al. Graph of thoughts: Solving elaborate problems with large language models [C]//Proceedings of the 38th AAAI Conference on Artificial Intelligence. Vancouver, Canada: AAAI Press, 2024: 17682-17690.
|
| 19 |
KHANDELWAL A, GUPTA M, AGRAWAL P. CoCoA: Confidence-and context-aware adaptive decoding for resolving knowledge conflicts in large language models [C]//Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Suzhou, China: ACL, 2025: 6835–6855.
|
| 20 |
YU M, MENG F C, ZHOU X Y, et al. A survey on trustworthy LLM agents: Threats and countermeasures [C]//Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining (Volume 2). Toronto, Canada: Association for Computing Machinery, 2025: 6216–6226.
|
| 21 |
ZHANG J H, LIU K P. Thought space explorer: Navigating and expanding thought space for large language model reasoning [C]//Proceedings of the 2024 IEEE International Conference on Big Data. Washington DC, USA: IEEE, 2024: 8259–8261.
|
| 22 |
WANG X Z, WEI J S, SCHUURMANS D, et al. Self-consistency improves chain of thought reasoning in language models [C]//Proceedings of the 11th International Conference on Learning Representations. Kigali, Rwanda: ICLR, 2023.
|
| 23 |
|
| 24 |
GUU K, LEE K, TUNG Z, et al. Retrieval-augmented language model pre-training [C/OL]//Proceedings of the 37th International Conference on Machine Learning. Online: PMLR, 2020: 3929-3938 [2026-08-11]. https://proceedings.mlr.press/v119/guu20a.html.
|
| 25 |
IZACARD G, GRAVE E. Leveraging passage retrieval with generative models for open domain question answering [C/OL]//Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. Online: Association for Computational Linguistics, 2021: 874-880 [2026-08-11]. https://aclanthology.org/2021.eacl-main.74.
|
| 26 |
ZHANG Z H, ZHU A A, YANG L J, et al. Accelerating iterative retrieval-augmented language model serving with speculation [C]//Proceedings of the 41st International Conference on Machine Learning. Vienna, Austria: JMLR, 2024: 60626–60643.
|
| 27 |
SHUSTER K, POFF S, CHEN M Y, et al. Retrieval augmentation reduces hallucination in conversation [C]//Findings of the Association for Computational Linguistics: EMNLP 2021. Punta Cana, Dominican Republic: ACL, 2021: 3784–3803.
|
| 28 |
YANG K Y, SWOPE A M, GU A, et al. LeanDojo: Theorem proving with retrieval-augmented language models [C]//Proceedings of the 37th International Conference on Neural Information Processing Systems. New Orleans, LA, USA: Curran Associates Inc., 2023: 21573–21612.
|
| 29 |
CHU Z, CHEN J C, CHEN Q L, et al. Navigate through enigmatic labyrinth a survey of chain of thought reasoning: Advances, frontiers and future [C]//Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Bangkok, Thailand: ACL, 2024: 1173–1203.
|
| 30 |
|
| 31 |
CHEN X S, ZHOU S H, LIANG K, et al. Skip-thinking: Chunk-wise chain-of-thought distillation enable smaller language models to reason better and faster [C]//Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. Suzhou, China: ACL, 2025: 12142–12157.
|
| 32 |
MO S T, XIN M. Tree of uncertain thoughts reasoning for large language models [C]//ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Seoul, Korea: IEEE, 2024: 12742–12746.
|
| 33 |
JIA Y J, CAI J P, LIU X M. Entropy-Guided tree of thoughts: A dynamic approach to diverse path generation in LLM reasoning [C]//Proceedings of the 2025 3rd International Conference on Big Data and Privacy Computing (BDPC). Fuzhou, China: IEEE, 2025: 132–136.
|
| 34 |
WANG Y, LI P, SUN M S, et al. Self-knowledge guided retrieval augmentation for large language models [C]//Findings of the Association for Computational Linguistics: EMNLP 2023. Singapore, Singapore: ACL, 2023: 10303–10315.
|
| 35 |
WANG F, WAN X C, SUN R X, et al. Astute RAG: Overcoming imperfect retrieval augmentation and knowledge conflicts for large language models [C]//Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Vienna, Austria: ACL, 2025: 30553-30571.
|
| 36 |
GADBAIL S, DESAI M, KARLAPALEM K. Iti-validator: A guardrail framework for validating and correcting LLM-generated itineraries [C]//Proceedings of the 33rd ACM International Conference on Advances in Geographic Information Systems. Minneapolis, MN, USA: Association for Computing Machinery, 2025: 1110–1113.
|
| 37 |
YU W H, ZHANG H M, PAN X M, et al. Chain-of-note: Enhancing robustness in retrieval-augmented language models [C]//Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Miami, Florida, USA: ACL, 2024: 14672–14685.
|
| 38 |
WANG J N, SUN Q S, LI X, et al. Boosting language models reasoning with chain-of-knowledge prompting [C]//Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Bangkok, Thailand: ACL, 2024: 4958–4981.
|
| 39 |
|
| 40 |
LIU J W, XIA C Q, WANG Y Y, et al. Is your code generated by ChatGPT really correct? Rigorous evaluation of large language models for code generation [C]//Proceedings of the 37th International Conference on Neural Information Processing Systems. New Orleans, LA, USA: Curran Associates Inc., 2023, 36: 21558–21572.
|
| 41 |
WANG X Y, CHEN Y Y, YUAN L F, et al. Executable code actions elicit better LLM agents [C]//Proceedings of the 41st International Conference on Machine Learning. Vienna, Austria: JMLR, 2024: 50208–50232.
|
| 42 |
WEN Z H, TIAN Z L, WU W, et al. GROVE: A retrieval- augmented complex story generation framework with a forest of evidence [C]//Findings of the Association for Computational Linguistics: EMNLP 2023. Singapore: ACL, 2023: 3980–3998.
|
| 43 |
DU Z X, QIAN Y J, LIU X, et al. GLM: General language model pretraining with autoregressive blank infilling [C]//Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Toronto, Canada: ACL, 2022: 320–335.
|
| 44 |
|
| 45 |
LI M X, WANG W J, FENG F L, et al. Think twice before trusting: Self-detection for large language models through comprehensive answer reflection [C]//Findings of the Association for Computational Linguistics: EMNLP 2024. Miami, Florida, USA: ACL, 2024: 11858–11875.
|
| 46 |
ZOU W, GENG R P, WANG B H, et al. PoisonedRAG: Knowledge corruption attacks to retrieval-augmented generation of large language models [C]//Proceedings of the 34th USENIX Security Symposium (USENIX Security 25). Seattle, WA, USA: USENIX Association, 2025: 3827–3844.
|
| 47 |
YANG Z L, QI P, ZHANG S Z, et al. HotpotQA: A dataset for diverse, explainable multi-hop question answering [C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Brussels, Belgium: ACL, 2018: 2369-2380.
|
| 48 |
|
/
| 〈 |
|
〉 |