acceptodds
Under review as a conference paper at ICLR 2027

Answering by Format: How Evidence Formatting Shapes LLM Abstention in RAG

11 views

Abstract

We study evidence-grounded answering and abstention in retrieval-augmented generation (RAG), where the decision to answer should depend on whether the evidence is sufficient. However, we identify a spurious evidence-rendering gate: changing how the same evidence is presented, without altering its content or order, can switch a large language model (LLM) between answering and abstaining. In controlled experiments with nine LLMs, correlating evidence format with whether training examples call for an answer or abstention induces this gate in eight of them. On insufficient evidence, this training increases the answer-rate gap between the two formats by an average of 0.70. Reversing this association reverses the learned format bias in most models. To assess the performance consequences of this association, we vary its strength and find that accuracy on the answer-or-abstain task declines as the association strengthens. To examine whether such correlations also occur in practice, we audit public training corpora and find associations between formatting cues and answer-versus-abstention labels. Controlled retraining further shows that artificially introducing such correlations can induce the gate. Beyond these controlled experiments, we also observe format-dependent answering in open (ChatQA-1.5-8B; TrustAlign-Qwen2.5-3B/7B) and closed (claude-opus-5) models. In the controlled experiments, removing the association between evidence format and answer-versus-abstention labels improves task accuracy by up to 0.37 relative to format-correlated training. Reliable abstention in RAG therefore requires controlling evidence presentation during both training and evaluation. We release our code at https://anonymous.4open.science/r/paper-artifact-staging-B0B7/

Primary area:
foundation or frontier models, including LLMs
Keywords:
Retrieval-Augmented Generation, Abstention, Spurious Correlations, Shortcut Learning, Format Sensitivity, LLM Reliability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_kmVbOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2(+1)
Read review
Paper summary

The paper reveals a phenomenon: whether a model abstains on a question with insufficient evidence is associated with how the content is organized on the surface, and this association has no clear direction. The paper examines this phenomenon from three angles: the association between format and answer labels in controlled training experiments, an audit of format and labels in public corpora, and mitigation methods at training and inference time. Controlled training creates a strong format–label association, while the off-the-shelf experiments use different models, tasks, prompts, and output classification rules. The effects observed in off-the-shelf models have not been shown to arise from the same training mechanism.

Critical strengths
  1. The scenarios examined are important and real. Evidence-grounded reasoning has practical applications in high-stakes settings, such as law and medicine.

  2. The paper conducts statistical significance analyses, and most experiments report statistically significant results.

Critical weaknesses

Major concerns:

  1. Controlled training creates a strong format–label association, while the off-the-shelf experiments use different models, tasks, prompts, and output classification rules. The effects observed in off-the-shelf models have not been shown to arise from the same training mechanism.

  2. The training and evaluation settings are not fully aligned. The evaluation prompt for the RAG task does not explicitly tell the LLM to abstain when an answer cannot be directly inferred from the provided material. This may cause the observed results to differ from the conclusions claimed in the paper. The external evaluations involving ContractNLI and ChatQA, however, remove this potential bias caused by prompt ambiguity.

  3. The models used are too outdated. The models (or their base models) reported in Tables 1, A5, and A10 that predate Qwen3-Base are all quite sensitive to format. Among the Claude models, which were released later, only claude-opus-5 on 2Wiki dev shows a fairly substantial format effect in Table 1. In the other three settings, the CI either includes zero or the absolute difference is too small. More evidence is needed to show that this phenomenon also occurs in advanced open-source and commercial models.

  4. The paper uses LLM-as-a-judge evaluation to determine whether a model abstains, but it does not report agreement between human and LLM judgments. Since this is the paper’s main metric, the authors should report human–LLM agreement rather than only agreement between LLM judges.

Minor concerns:

  1. The paper’s conclusion is too obvious. Prior NLI work (I consider this paper’s evaluation setting to be a subset of NLI) has already found that LLMs use surface cues instead of semantic inference (arXiv:2401.14440; ACL Anthology: 2024.lrec-main.602). The training results are largely predictable and offer little new insight.

  2. The paper repeatedly uses two em dashes within a single sentence to enclose an aside, a construction GPT often favors. Many sentences are also long and difficult to follow. There are too many metrics, and the paper contains far too much material. I found it very hard to read. Perhaps this is because I am not a native English speaker, but the style feels weird and uncomfortable to me. When I searched the full text for “prompt” to find the prompts used for different datasets, I also found an internal version label, “V3 prompt” (line 2188), which may suggest that AI-generated content was not carefully reviewed.

Questions for authors

Chapters 4, 5, and 6 use different models and benchmarks. What motivated this choice? Why not evaluate and train the same family of LLMs on same benchmarks?

Reviewer_ncq1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper identifies a spurious evidence-rendering gate in retrieval-augmented generation. Changing how the same evidence is presented without changing its content can switch a large language model between answering and abstaining. Format-correlated training induces this gate in eight of nine models and increases the answer-rate gap on insufficient evidence by an average of 0.70. Released RAG models and Claude models also show format-dependent answering. Across a seven-rung dose ladder, sufficiency-oriented accuracy drops at every rung as the training correlation strengthens, while format reliance increases and evidence-state sensitivity decreases. Removing the correlation between evidence format and training labels through marginal balancing or exact crossing recovers sufficiency-oriented accuracy by up to 0.37.

Critical strengths
  • The experimental framework isolates the effect of evidence formatting on answer-versus-abstain decisions by controlling for content, document order, token length, and item count.
  • The analysis links behavioral format reliance to internal margins at the decision position and identifies how training correlation suppresses evidence-state sensitivity.
Critical weaknesses
  • The 0.7 insufficient evidence answer-rate gap comes from recipe A, where abstain training examples use three distractor paragraphs rather than raw renderer and excluding yes/no questions. When recipe B renders gold-removed insufficient subsets using raw renderer, the pooled adapter DiD drops to 0.265 and 0.290.
  • In the TrustAlign experiments, Qwen2.5-Instruct already shows a rendering gate before TrustAlign post training. Removing the training data quote and double space shows little effect on the rendering gate. The corpus intervention only induces a gate by injecting a single-token marker at AUROC 1.000.
Questions for authors
  • Why does the pooled adapter DiD drop between recipe A and B when abstain examples use gold-removed insufficient item subsets using raw renderer instead of three distractor paragraphs and leaving out yes/no questions?
  • Since removing TrustAlign quote and double-space shows little effect on the rendering gate and the corpus intervention only tests a single-token marker at AUROC 1.000, at what AUROC scale does TrustAlign post-training begin to induce a gate?
Reviewer_cqUfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Prose very hard to understand; likely heavily AI generated, but here's what I understood. The authors claim that in a question answering setup that has context like RAG, the decision to answer a question should be based on whether the evidence provided is sufficient. They find something they call a spurious evidence-rendering gate that modulates between answering and abstention. In the abstract they claim that this can happen without modifying the input query. The intro states that they identify this spurious evidence-rendering gate in both small open-source models and frontier claude models, find how the gate is learned, and provide an approach to mitigate the effect of this gate.

Critical strengths
  • Understanding mechanisms that are unknown or less known to the community by investigating perturbations in inputs/outputs or model internals is valuable.
  • Hard to point out any other strengths given the paper is very jargon-filled and heavily AI-generated.
Critical weaknesses
  1. Fundamental problem is that the paper is heavily AI generated, making it impossible for me to really follow. Terms like 'gate-breadth fleet of nine checkpoints' and 'core intervention set ... carries the dose, balancing' are littered throughout.

  2. The related work is poor. There are hundreds of works looking at prompt formatting for language models, for queries etc. Information retrieval obviously predates computers; IR with computers have been talked about and theorized since world war 2, so there's plenty of work one should cite that predates 2024.

Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.