acceptodds
Under review as a conference paper at ICLR 2027

When Answers Agree but Reasoning Disagrees: Conflict-Aware Agentic RAG for Medical Reasoning

Abstract

Large language models (LLMs) show promise in medical question answering, yet extended reasoning can introduce factual errors and conflicting claims that undermine answer reliability. Recent multi-round retrieval-augmented generation (MA-RAG) mitigates these issues through dynamic retrieval using answer-level consistency signals, leading to state-of-the-art performance yet overlooking the conflicts within individual reasoning trajectories. Specifically, it will cost N times tokens to get the retrieval signal during answer sampling and fails when the model consistently produces incorrect answers through disordered reasoning. In this paper, we begin by analysing reasoning trajectories and explore how they can guide retrieval and repair direction. We observe that trajectories can be identified into three patterns: smooth, repeat and conflict. Notably, trajectories containing conflicting claims have substantially lower answer accuracy than repetitive or smooth reasoning, suggesting that these semantic patterns offer a useful signal for external refinement. Motivated by this finding, we propose a Conflict-aware Agentic RAG framework for medical REasoning (CARE), which allows for a more reliable reasoning process with fewer tokens by analyzing reasoning trajectories to guide targeted interventions. It is achieved by preserving smooth reasoning, truncating repetition, and retrieving evidence to resolve factual conflicts, then integrating relevant reasoning and evidence to produce more reliable answers. Experiments on eight medical benchmarks show that CARE improves average accuracy by 2% over the strong adaptive RAG baseline MA-RAG while reducing generated tokens by approximately 77%. On MedThink-Bench, CARE further improves alignment with expert reasoning by 12.81 points across four semantic metrics. These results demonstrate the potential of trajectory-level conflict detection for improving both the reliability and efficiency of medical reasoning. Our code is available at https://anonymous.4open.science/r/CARE-C3F1/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
RAG, Large Language Models, Medical Question Answering

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_LeQUOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CARE, a conflict-aware agentic RAG framework for medical question answering. It classifies reasoning trajectories into Smooth, Repeat, and Conflict patterns and applies different interventions, including early truncation, targeted retrieval, and answer repair. Experiments across eight medical QA benchmarks report improvements in accuracy and reduced generation costs compared with MA-RAG.

Critical strengths
  1. The paper highlights that agreement in final answers may conceal inconsistencies within reasoning trajectories, providing a meaningful direction for improving adaptive retrieval.

  2. CARE is evaluated across eight medical benchmarks and multiple model sizes, showing promising accuracy improvements and lower generation costs relative to MA-RAG.

Critical weaknesses
  1. The paper does not clearly establish whether detected conflicts represent genuine reasoning errors or normal exploration of alternative answers. The case study illustrates this concern, as the detected conflict involves two incorrect options while the original trajectory already reaches the correct answer.

  2. The experiments do not sufficiently isolate conflict-aware routing from the benefits of additional retrieval, answer repair, and early stopping. Furthermore, textual similarity to expert rationales does not establish reasoning reliability.

Questions for authors
  1. How do the authors distinguish genuine factual contradictions from normal hypothesis exploration? Is there independent evidence that the detected patterns identify reasoning errors rather than general uncertainty?

  2. Can the authors demonstrate the independent benefit of conflict-aware routing against simpler verification or repair strategies under matched retrieval and generation budgets?

Reviewer_4ijbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CARE diagnoses a medical reasoning trajectory as smooth, repetitive, or conflicting, then applies selective truncation, retrieval, and answer repair. Experiments show accuracy improvements across eight medical QA benchmarks with substantially fewer generated tokens than MA-RAG.

Critical strengths
  • The focus on within-trajectory signals is well motivated, and the accuracy–generation-cost results are encouraging.
  • Component ablations and selector experiments help assess the contributions of the pipeline.
Critical weaknesses
  • Changes in answer commitments can reflect productive reconsideration as well as factual inconsistency. More direct validation of the graph labels is needed to establish how reliably the proposed signal distinguishes these cases.
  • The reasoning-reliability evidence relies on reference-similarity metrics such as BLEU, ROUGE, and BERTScore. These measure agreement with expert text but do not directly establish factual correctness of intermediate claims; the Repair stage can also change wording. Claim-level assessment would strengthen this central conclusion.
Questions for authors

See weaknesses.

Reviewer_tptSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CARE uses a single reasoning trace to detect repeated or conflicting answers. It truncates repetition, retrieves evidence for conflicts, and uses a Repair agent to generate the final answer. Across eight medical QA benchmarks, it reports small accuracy gains over MA-RAG with about 77% fewer generated tokens (Tables 1 and 4).

Critical strengths

Answer switches within a reasoning trace are a useful signal for identifying unreliable answers. In Table 3, conflict traces start at 38.64% accuracy, compared with 53.47% for repeat and 74.32% for smooth traces. Good efficiency gains over MA-RAG. Generated tokens drop from 6,501 to 1,465 (4B) and 7,181 to 1,654 (8B), with slightly better accuracy (Tables 1 and 4).

Critical weaknesses
  1. I'm not convinced the gains come from conflict-aware retrieval. CARE improves only 1.62 and 0.90 points over the best baselines (Table 1). About 59% of the overall gain comes from smooth and repeat cases, which don't use conflict retrieval (Table 3). The ablations lack a Solver + Repair baseline without routing and a random-retrieval control. This makes it hard to isolate the benefit of conflict detection. Related work on retrieval during reasoning, such as Search-o1 (EMNLP 2025) and DRAGIN (ACL 2024), is also not discussed.
  2. The reasoning alignment claim is not well supported. The 12.81-point gain is against MA-RAG, the weakest baseline on METEOR, BLEU and ROUGE-L (Table 6). For Qwen3-4B, Repair raises METEOR from 23.07 to 30.69 with no accuracy change (Table 2), which may reflect its short JSON output (App. H) rather than better reasoning.
Questions for authors

Can you compare Solver + Repair without routing against CARE, and test additional retrieval on a random 19% of traces? This would help isolate the benefit of conflict-aware retrieval. 2. Can you report paired significance tests and clarify whether k=3 was selected using a held-out validation set (Fig. 7)? 3. Can you report length-matched MedThink results and the benchmark's LLM-judge scores to distinguish improved reasoning from changes in output format?

Reviewer_wps2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper addresses factual errors and high token costs in medical LLM reasoning. It categorizes reasoning trajectories into smooth, repeat, and conflict modes and proposes the CARE agentic RAG framework to apply different intervention strategies for each category. Evaluations across eight medical benchmarks show a 2% accuracy improvement over MA-RAG and around 77% token reduction, alongside enhanced alignment with expert reasoning on MedThink-Bench. The authors claim novelty in using trajectory-internal semantic conflict as a retrieval trigger for medical QA.

Critical strengths

None

Critical weaknesses
  1. The claimed core innovation, diagnosing smooth/repeat/conflict reasoning trajectories to trigger targeted RAG interventions, is not sufficiently differentiated from existing self-critique, reflection, and trace-aware retrieval work. The paper fails to articulate a distinct technical insight that separates its trajectory-graph conflict detection from prior consistency checking and reasoning repair methods, such that the contribution appears incremental and lacks sufficient novelty to justify publication.
  2. The performance improvements lack strong causal evidence linking gains to the proposed trajectory-level conflict signal. Key ablations for the trajectory pattern classifier are missing, and the experimental design cannot rule out alternative explanations for the observed accuracy lift and token reduction, which weakens the empirical support for the central claim of the work.
Questions for authors
  1. Precisely, what new technical mechanism distinguishes your trajectory-graph conflict detection from prior sentence-level contradiction detection and self-reflection modules in RAG systems?
  2. Can you provide ablation results that isolate the impact of your three trajectory-based intervention rules, separate from simpler token-reduction or post-hoc editing baselines?
  3. What is the false positive and false negative rate of your conflict detector, and how do these errors degrade medical answer reliability?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.