acceptodds
Withdrawn from ICLR 2027

When Semantically Equivalent Contexts Yield Different Answers: Measuring Contextual Uncertainty to Improve Retrieval-Augmented Question Answering

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A fundamental challenge of retrieval-augmented generation (RAG) is that retrieving relevant evidence does not guarantee that an LLM uses it consistently or correctly. Current RAG systems lack a model-agnostic mechanism to identify such failures and use the diagnosis to improve QA performance. We introduce SECURE (SEmantic Context Uncertainty for REtrieval-based QA), a training-free method requiring only text input/output access to the generator. SECURE rephrases one retrieved chunk at a time, then measures the semantic consistency of the resulting answers across these intended meaning-preserving context variants. The resulting score estimates answer risk, while each single-chunk comparison exposes which edit changed the response, without requiring reference answers or model internals. Across four benchmarks, SECURE achieves state-of-the-art performance in uncertainty evaluation, using the same correctness criterion for all methods. We then close the diagnostic-to-improvement loop: SECURE-RAG uses this same signal to trigger adaptive semantic chunking and guide iterative context refinement. On the selected high-uncertainty questions, SECURE-RAG improves reported QA accuracy in 15 of 16 backbone–benchmark combinations. Together, these results show that evidence-rephrasing sensitivity can both rank answer risk and guide targeted RAG refinement in the evaluated settings.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Retrieval-Augmented Generation, Semantic Uncertainty, Open-book Question Answering, Training-free Methods

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XdD5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes SECURE, a training-free, black-box method to quantify and localize uncertainty in retrieval-augmented QA by rephrasing one retrieved chunk at a time, generating greedy answers for these semantically intended-equivalent contexts, and measuring semantic consistency between answers via pairwise entailment to produce a degree-based uncertainty score (DSE).

Critical strengths

The core idea of probing RAG reliability by rephrasing individual evidence chunks and attributing answer sensitivity to specific chunks is original and well-motivated, going beyond prior question/prompt perturbations.

The diagnostic-to-action loop (adaptive chunking + iterative refinement) is a coherent, training-free pipeline that operationalizes the uncertainty signal to improve context quality.

Broad evaluation on four standard QA datasets with multiple backbones; clear demonstration that higher SECURE correlates with lower accuracy across models and thresholds.

Critical weaknesses

The assumption that single-chunk paraphrases preserve all question-relevant semantics is only partially validated: automatic mutual entailment rates (93–98%) are high but not definitive, leaving residual confounding between true sensitivity and rewrite noise.

The DSE metric’s theoretical connection to semantic entropy is only sketched under restrictive “equivalence-class” conditions; its specific log-degree averaging is somewhat heuristic and could use stronger justification or alternatives.

The method depends heavily on LLM-as-judge entailment for short answers; inaccuracies or calibration drift in judging can affect both the SECURE score and chunk labels (certain/necessary/unnecessary).

Missing comparisons to several closely related recent lines: path- or retrieval-perturbation UQ for RAG (e.g., R2C), fidelity/faithfulness-decomposed UQ (e.g., FRANQ), and contrastive/conflict-aware decoding (e.g., DCCD), as well as permutation-sensitivity stabilization (Stable-RAG).

Reliance on different correctness criteria across tasks (fuzzy ROUGE-L > 0.3 for AUROC vs EM for QA) may complicate cross-method interpretation; sensitivity to the judge choice (LLM vs NLI model) for answer comparisons is not fully explored.

Some algorithmic details critical to reproducibility and cost evaluation are deferred to appendices (e.g., iteration budgets, stopping criteria behavior, typical k, number of refilled chunks per round), making it hard to assess practical deployment overhead.

The paper cites uncertainty estimation broadly but omits direct quantitative comparisons or discussion with several recent, targeted RAG reliability works (R2C, FRANQ, Stable-RAG, DCCD), limiting context on where SECURE stands in today’s landscape.

Questions for authors

How sensitive are your results to the choice of judge and rewriter models? Can you report uncertainty AUROC and QA gains when (a) using a strong non-LLM NLI (e.g., DeBERTa-MNLI) for answer comparison, and (b) using a different LLM for rephrasing than for answering?

What fraction of answer flips under rephrasing can be attributed to imperfect rewrites versus true model instability? A small-scale human study on flipped cases would be illuminating.

How does SECURE behave on inherently ambiguous questions (e.g., AmbigNQ items with multiple valid answers)? Does the metric systematically up-rank such items as “uncertain,” and how do your refinement actions handle multi-answer validity?

In iterative refinement, from what candidate pool do you “refill” chunks similar to necessary ones (top-m from the original retrieval index or a larger corpus)? How many are added per round, and what safeguards prevent topic drift?

Reviewer_QaZtOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SECURE, a method for estimating the risk of an incorrect answer in retrieval-augmented question answering. It requires no training. The method rewrites one context chunk at a time and checks whether the responses remain consistent. The paper also proposes SECURE-RAG, which uses this signal to decide when to apply adaptive semantic chunking and how to refine the context over several rounds. The evaluation covers four QA datasets and several generator models. It also examines rewrite quality, the contribution of each component, and computational cost.

Critical strengths

A useful link between finding problems and improving answers. The method checks whether an answer changes when the retrieved evidence is reworded. It also tracks which context edit led to each response. Using this information to refine the context gives the method a practical use beyond assigning an uncertainty score (§§4–5).

Promising results across several datasets and models. The experiments cover four QA benchmarks and models from both the Qwen and Llama families. In Table 2, the full pipeline improves results in 15 of the 16 model–dataset combinations on the selected high-uncertainty subsets. The component ablations and automatic checks of rewrite quality also help readers assess the approach.

Critical weaknesses

The benefit varies across generators. On the selected TriviaQA subset in Table 2, Llama-3 8B improves from 42.60 to 53.91, while Llama-3 70B improves from 73.12 to 74.87. The paper does not fully explain how model capability affects these gains.

The analysis focuses on overall gains and successful examples. Table 2 reports net accuracy changes, and Appendix L shows successful corrections. It remains unclear how often refinement fixes an answer or makes it worse, especially given the performance drops from some components in §6.3.

Questions for authors

Q1. How useful is the method for stronger generators? Could the authors explain why the gains vary across models? If feasible, a small evaluation with another stronger generator would help clarify how broadly the method applies.

Q2. How often does context refinement help or hurt? Could the authors report how many answers change from wrong to correct and from correct to wrong after applying the pipeline?

Q3. Could the authors include some failure cases? Could the authors add one or two examples where the method does not help or makes the answer worse, and briefly discuss the causes?

The paper presents an interesting approach and promising results. I would mainly like to understand when the method helps most and how often refinement makes answers worse. Further analysis of the existing results could address these concerns. I currently lean toward weak acceptance.

Reviewer_M5SoOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes an approach to uncertainty estimation in Retrieval Augmented Question Answering (RAQA), SECURE. The proposed approach rewrites chunks in the set of top-k chunks retrieved for a given question one by one and observes the extent to which there is answer variation after each chunk rewrite. Answer variation is measured based on textual entailment between answers collected. If variation is above a certain threshold, then there is uncertainty. For questions where the answer is deemed to be uncertain, the authors propose to analyze the set of retrieved passages. An initial step is sentence splitting and regrouping; then newly grouped chunks are processed one-by one to identify necessary/unnecessary information. When an uncertain chunk is marked as unnecessary it is discarded, when marked as necessary additional information is retrieved.

There are some methodological and experimental weaknesses (see below); in addition the presentation could be improved.

Critical strengths

The idea of context perturbation as a test for certainty is interesting though not entirely new.

Critical weaknesses
  • Methodology. (a) Meaning preserving chunk rewrites are the core of the proposed approach. However, there is no human evaluation of the extent these are "really" re-writes (or just a comma is modified). Moreover, it is not clear whether there are re-write errors (e.g., hallucinations) and what is done w.r.t. errors (what happens with a judgement based on a re-write error?). (b) The uncertainty detection approach as well as the QA approach are extremely expensive requiring several forward-passes by LLMs and NLI models. (c) What is the intuition behind the semantic chunking into sentences? In this open-book QA tasks, it is common to use chunks that are already small ranging between 100-300 words. This also adds another layer of time-cost, e.g. a sentence splitter needs to be run.

  • Experimental. (a) Important information-based baselines as perplexity and entropy are not reported; neither p(true). (b) In experiments, what is used as a corpus the the set of passages available in the datasets. There are rather curated when compared to using the chunked Wikipedia dump as most of previous work on open-book QA does. (c) The metrics to assess correctness are problematic, these can affect the ranking of UQ approaches. Although no metric is perfect, specif LLM-as-Judge and prompt ones have been found to work better than token overlap [1].

[1] https://arxiv.org/abs/2502.18108

Questions for authors
  • Line 156 and 159, the function π applies to two different inputs.
  • Table 1, enlarge number (better table spacing is possible)
  • Line 225. When talking about the labels assigned by the NLI model it mentions 'entailment' and 'neutral', but what about the third important label these classifiers output: 'contradiction'?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.