- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies how the order of retrieved passages affects model predictions in medical RAG. The authors propose CG-OAS, a method that generates multiple answers by changing the order of the same retrieved passages. When all predictions agree, the answer is kept. Otherwise, a classifier trained on validation data selects among the predicted answers using features derived from their frequencies and occurrence patterns. The method is evaluated on MedMCQA and MedQA, showing small accuracy improvements over majority voting and the original passage order. However, the results do not show a clear advantage over a similar supervised selector without the consistency gate.
Critical strengths
The paper studies an interesting problem: how changing the order of retrieved passages can affect a model's answers, even when the evidence remains the same. I also found it useful that the authors compare their method with a no-gate version and include some additional tests on the effect of repeated permutations. These analyses help clarify the behavior and limitations of the method.
Critical weaknesses
1. The paper is hard to read and poorly presented.
I had a hard time following this paper and understanding what the authors were actually trying to show. The writing is often verbose, repetitive, and unnecessarily complicated. There is a lot of technical detail, but very little effort to explain the main ideas in a clear and intuitive way.
I have no problem with LLM assistance in writing, but here the text often feels as if it was written to satisfy a checklist or an automated reviewer, rather than to help a human reader understand the work.
The presentation does not help either. Tables 1 and 2, for example, have excessively large headers and awkward formatting, despite reporting fairly simple results. The related work is also rather limited. Only a small number of papers are discussed, and I was left without a clear understanding of how this work fits into the existing literature or what gap it is supposed to fill.
Overall, I think the paper needs to be substantially rewritten. In its current form, even understanding and assessing the contribution requires too much effort from the reader.
2. The contribution is weak, and the experimental results are not convincing.
The method itself seems quite simple: generate multiple answers by changing the order of retrieved passages, then train a classifier to select an answer. I am not convinced there is enough novelty here, especially given the existing work on self-consistency and supervised answer selection.
The experimental evaluation is also quite limited. The authors use only one relatively old 7B model (Qwen2.5-7B-Instruct), with no experiments on more recent or larger models. The baselines are few and mostly basic, with no convincing comparison against other supervised answer selection methods. This makes it difficult to judge how useful the proposed approach really is.
The proposed consistency gate also seems to add very little. The authors report no statistically significant improvement over the no-gate version, so it is unclear what the main proposed mechanism actually contributes.
I am also concerned that the authors selected the classifier family based on test-set accuracy (Section 4). This makes the reported gains difficult to trust. Even setting this aside, the improvement over majority voting is less than one percentage point, and the paper does not convincingly show why this approach is preferable to simpler alternatives.
Overall, I do not think the current results provide enough evidence for a contribution at the level expected for ICLR.
Questions for authors
As questions for the authors:
-
Why was the classifier family selected using test-set accuracy? Can the authors report results where all model choices are made using validation data only? This is important for assessing whether the reported gains are reliable.
-
What does the consistency gate actually add? Given the results against the no-gate baseline, can the authors show a clear benefit from this component, or from the order-based features compared with a simpler supervised voting method?
As future direction, I would suggest that the authors consider including stronger baselines and evaluating their method on more recent models of different sizes. The current evaluation, based on a single 7B model, is quite limited and makes it difficult to understand how well the findings generalize.
More generally, I encourage the authors to consider the points raised in the weaknesses above when revising the paper. Given the extent of these issues, I think it is unlikely that they can be adequately addressed during the rebuttal period or that my overall assessment would change substantially.