acceptodds
Under review as a conference paper at ICLR 2027

CG-OAS: ORDER-CONSISTENCY-GATED ANSWER SELECTION FOR MEDICAL RETRIEVAL-AUGMENTED GENERATION

Abstract

Medical RAG suffers from evidence-order sensitivity — the same set of evidence can produce inconsistent answers simply due to different ordering of the input. While prior work typically treats this as a robustness weakness, we find that this order sensitivity itself can be repurposed as a usable reliability signal. Through evidence-order perturbation (generating variants by permuting the same evidence set), examples fall into two regimes: strict-consistency (all permutations converge to the same answer) and non-strict (disagreement across permutations). The non-strict regime directly pinpoints where the model's evidence comprehension is fragile, providing a principled basis for answer selection. Motivated by this finding, we propose CG-OAS, a consistency-gated answer selection protocol. CG-OAS preserves the predicted answer when all evaluated evidence permutations agree; otherwise, it applies a lightweight, dataset-specific selector trained on non-unanimous validation examples. The selector uses only features derived from hard predictions, requiring neither evidence text nor model probabilities. The protocol returns an answer for every question and operates on outputs from validation-selected LoRA checkpoints without modifying retrieval or requiring additional generator training. On the MedMCQA and MedQA evaluation sets, learned selection improves accuracy over majority voting on non-unanimous questions by 2.90 and 4.98 percentage points, respectively, while leaving unanimous predictions unchanged. Overall, CG-OAS achieves a macro-averaged accuracy of 61.14%, exceeding original-order and majority-vote LoRA RAG by 1.36 and 1.04 percentage points, respectively, with paired bootstrap confidence intervals supporting both macro-averaged gains. These results show that evidence-order perturbations can provide actionable signals for targeted answer selection in medical RAG without updating the generator.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Retrieval-Augmented Generation, Evidence-Order Sensitivity, Answer Selection, Prediction Consistency, Medical Question Answering

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_wzArOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how the order of retrieved passages affects model predictions in medical RAG. The authors propose CG-OAS, a method that generates multiple answers by changing the order of the same retrieved passages. When all predictions agree, the answer is kept. Otherwise, a classifier trained on validation data selects among the predicted answers using features derived from their frequencies and occurrence patterns. The method is evaluated on MedMCQA and MedQA, showing small accuracy improvements over majority voting and the original passage order. However, the results do not show a clear advantage over a similar supervised selector without the consistency gate.

Critical strengths

The paper studies an interesting problem: how changing the order of retrieved passages can affect a model's answers, even when the evidence remains the same. I also found it useful that the authors compare their method with a no-gate version and include some additional tests on the effect of repeated permutations. These analyses help clarify the behavior and limitations of the method.

Critical weaknesses

1. The paper is hard to read and poorly presented.

I had a hard time following this paper and understanding what the authors were actually trying to show. The writing is often verbose, repetitive, and unnecessarily complicated. There is a lot of technical detail, but very little effort to explain the main ideas in a clear and intuitive way.

I have no problem with LLM assistance in writing, but here the text often feels as if it was written to satisfy a checklist or an automated reviewer, rather than to help a human reader understand the work.

The presentation does not help either. Tables 1 and 2, for example, have excessively large headers and awkward formatting, despite reporting fairly simple results. The related work is also rather limited. Only a small number of papers are discussed, and I was left without a clear understanding of how this work fits into the existing literature or what gap it is supposed to fill.

Overall, I think the paper needs to be substantially rewritten. In its current form, even understanding and assessing the contribution requires too much effort from the reader.

2. The contribution is weak, and the experimental results are not convincing.

The method itself seems quite simple: generate multiple answers by changing the order of retrieved passages, then train a classifier to select an answer. I am not convinced there is enough novelty here, especially given the existing work on self-consistency and supervised answer selection.

The experimental evaluation is also quite limited. The authors use only one relatively old 7B model (Qwen2.5-7B-Instruct), with no experiments on more recent or larger models. The baselines are few and mostly basic, with no convincing comparison against other supervised answer selection methods. This makes it difficult to judge how useful the proposed approach really is.

The proposed consistency gate also seems to add very little. The authors report no statistically significant improvement over the no-gate version, so it is unclear what the main proposed mechanism actually contributes.

I am also concerned that the authors selected the classifier family based on test-set accuracy (Section 4). This makes the reported gains difficult to trust. Even setting this aside, the improvement over majority voting is less than one percentage point, and the paper does not convincingly show why this approach is preferable to simpler alternatives.

Overall, I do not think the current results provide enough evidence for a contribution at the level expected for ICLR.

Questions for authors

As questions for the authors:

  • Why was the classifier family selected using test-set accuracy? Can the authors report results where all model choices are made using validation data only? This is important for assessing whether the reported gains are reliable.

  • What does the consistency gate actually add? Given the results against the no-gate baseline, can the authors show a clear benefit from this component, or from the order-based features compared with a simpler supervised voting method?

As future direction, I would suggest that the authors consider including stronger baselines and evaluating their method on more recent models of different sizes. The current evaluation, based on a single 7B model, is quite limited and makes it difficult to understand how well the findings generalize.

More generally, I encourage the authors to consider the points raised in the weaknesses above when revising the paper. Given the extent of these issues, I think it is unlikely that they can be adequately addressed during the rebuttal period or that my overall assessment would change substantially.

Reviewer_1s1sOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CG-OAS (consistency-gated order-aware answer selection), a method for choosing answers in medical RAG, where a model answers questions using passages retrieved from a document collection. The motivation for this paper is that the same retrieved passages can produce different answers depending on the order in which they are shown to the model. While prior work treats this order sensitivity as a robustness problem, the authors instead ask whether the pattern of answers produced across different orders can be used as a signal to pick a better final answer.

In testing, each question's retrieved passages (up to five, from medical textbooks) are shown to the model in 20 different orders, producing 20 answers. If all 20 answers agree, that answer is kept. If they disagree, a small classifier picks one of the answers the model gave, using only simple statistics about the 20 answers (how often each answer appears, how spread out the answers are, whether it was the answer under the original order, etc.). The classifier is trained on about 100 labeled validation questions per dataset and never sees the passages or the model's probabilities. The authors evaluate one model (Qwen2.5-7B with a dataset-specific fine-tuned adapter) on two medical exam datasets, MedMCQA and MedQA. They compare it with using the original order only, majority voting across the 20 answers, and the same classifier applied to every question without the agreement check.

Results show that CG-OAS improves average accuracy by about 1.3 percentage points over the original order and about 1 point over majority voting, with all of the gains coming from questions where the 20 answers disagreed. However, CG-OAS performs about the same as the classifier applied to every question, and falls slightly below it when repeated orders are removed, so the agreement check itself adds no measurable benefit. The authors also note that the classifier type for each dataset was chosen based on test accuracy. These results suggest that the pattern of answers across evidence orders contains useful information for answer selection beyond majority voting, without retraining the generator.

Critical strengths
  1. The paper introduces a simple method that could easily be applied to any existing RAG pipeline without retraining the generator. It uses only the answer letters the model outputs, so it doesn’t need the model’s probabilities and could be applied to models accessed only through an API

  2. The paper shows that disagreement across evidence orders is a useful signal for spotting hard questions, whereas questions where the 20 answers disagree make up about a quarter to a third of each dataset and are answered correctly only about 37–39% of the time, compared with about 68% when all 20 answers agree.

Critical weaknesses
  1. The gains of CG-OAS are small, with the improvement over majority voting being about 1 percentage point and not statistically significant on MedQA alone. This is compounded into an optimistic estimate by several factors: (i) the classifier type was chosen using test accuracy; (ii) the classifier is trained on only about 100 validation questions per dataset, using 25 features, three of which are identical copies; (iii) when repeated orders are removed, the gain shrinks and the chosen classifier types swap between datasets, which suggests the results are unstable.

  2. The consistency gate, which gives the method its name, cannot change any prediction and shows no benefit. What remains is a small trained re-ranker on top of majority voting, which is a standard ensembling technique. Order-permutation ensembling for RAG has also been studied before (Stable-RAG, Mixture-of-Intervention). This makes the contribution of the paper limited.

  3. Several baselines are missing: (i) standard self-consistency with sampling at the same 20-call budget; (ii) confidence-based selection using the model’s answer probabilities, which are easy to get for an open model; (iii) a no-retrieval baseline.

  4. The study uses a single 7B model, with about 60% accuracy. Testing using other models could be beneficial.

Questions for authors
  1. How do the results change if the classifier type is chosen on validation data instead of test accuracy, for example by cross-validation on the full validation set?

  2. What is the intended contribution of the method beyond a trained re-ranker on top of majority voting? Would the paper be better framed around the disagreement signal itself?

Reviewer_mTwfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes CG-OAS, an answer-selection method for medical RAG that generates multiple answers by permuting retrieved evidence. Unanimous predictions are accepted directly, while non-unanimous predictions are passed to a supervised answer selector. The method reports accuracy improvements over majority voting on two medical QA benchmarks.

Critical strengths
  • The idea of using evidence-order sensitivity as a signal for answer selection is interesting.

  • The method is relatively simple and does not require modifying the generator LLM.

Critical weaknesses
  • Very limited evaluation: The evaluation uses only one generator (Qwen2.5-7B) and is restricted to two medical QA benchmarks. This is an extremely narrow evaluation and provides little evidence of generalizability.

  • Small improvements for substantial additional complexity: The accuracy improvement over majority voting is approximately 1 percentage point, shrinking to 0.4 percentage points under permutation deduplication. These gains seem too small to justify training a separate answer selector, particularly given the additional inference cost.

  • Limited practical applicability: The method requires multiple generation calls per query. In many practical applications, models are queried only once, making the proposed approach inapplicable. The paper thus has limited scope.

  • Unclear relevance to larger models: The experiments use only a 7B model. An important unanswered question is whether larger, more capable models exhibit comparable evidence-order sensitivity. If larger models are substantially more permutation-invariant, the proposed method will become unnecessary at scale.

  • Poor writing and presentation: The paper is poorly presented, with excessive attention to implementation details at the expense of higher-level insights. For example, the main text repeatedly reports specific random seeds (e.g., 20260703), which adds little to the scientific discussion.

  • Test-set leakage: Section 4 states that the reported CG-OAS classifier family (among logistic regression, random forest, and gradient boosting) is selected based on the highest test accuracy. This is a serious method flaw as the test set is being used for model selection, invalidating its role as an independent evaluation set. The reported improvements cannot be interpreted as unbiased evidence.

Questions for authors
  • Oracle answer-selection accuracy: What is the pass@20 accuracy, i.e., the fraction of questions for which at least one of the 20 predictions is correct? This would establish the upper bound achievable by a perfect answer selector.

  • Scaling to larger models: How does evidence-order sensitivity vary with model size? Do larger models exhibit similar levels of disagreement across permutations, and does CG-OAS provide improvements for them?

  • Inference cost: How does performance vary with the number of permutations? Is there any meaningful accuracy improvement with substantially fewer than 20 generation calls?

  • Test-set model selection: Why were classifier families selected using test accuracy rather than validation accuracy? Please rerun the evaluation with all model-selection decisions made exclusively on validation data?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.