acceptodds
Under review as a conference paper at ICLR 2027

Regime Boundary Alignment for Evidence-Gated Question Answering

Abstract

Retrieval-augmented language models are expected to answer from the retrieved evidence, but in practice they often keep answering when that evidence is missing. We trace this behavior to the training signal: answer-focused fine-tuning assigns no target to unsupported contexts, so it cannot distinguish a reader that abstains from one that guesses, and unsupported answering stays near 100% even as supported accuracy improves. We introduce Regime Boundary Alignment (RBA), which trains a single reader on matched variants of the same question and gold answer. The reader is trained to produce the gold answer when the context supports it, including when conflicting evidence is also present, and to abstain when the correct support is removed; inference is ordinary decoding, with no verifier, threshold, or regime label. On three multi-hop QA datasets across three seeds, RBA reduces the unsupported-answer rate by more than sixty percentage points relative to conflict-focused training while matching its supported accuracy. On a held-out TriviaQA retrieval-miss slice, the same reader reduces unsupported answering from 100% to below 1% while also improving supported accuracy. These results indicate that evidence-gated answering must be learned on both sides of the support boundary.

Primary area:
foundation or frontier models, including LLMs
Keywords:
retrieval-augmented generation, selective answering, evidence grounding, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Ed6xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

RBA trains models on matched evidence conditions to answer supported questions and abstain when evidence is missing.

Critical strengths

The focus on training data is intuitive and useful.

Critical weaknesses
  1. Too few recent RAG methods are compared, making the contribution difficult to judge.

  2. The novelty remains unclear without stronger comparisons and discussion of conflicting evidence.

  3. The Introduction is difficult to follow, and unclear captions make the figures and tables harder to understand.

Questions for authors

How does RBA compare with recent RAG methods on missing and conflicting evidence?

Reviewer_RRyLOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Regime Boundary Alignment (RBA), a joint training objective that teaches LLMs to abstain when retrieved evidence is insufficient while preserving their question-answering ability on supported contexts. For each question, RBA constructs a matched triplet of contexts: C+C^+ containing only benchmark-annotated supporting documents, C±C^\pm containing both supporting documents and distractors, and C0C^0 containing only distractors. The model is then trained to generate the gold answer under C+C^+ and C±C^\pm, and to abstain under C0C^0. Experiments on four QA benchmarks show that RBA substantially reduces the unsupported-answer rate (UAR) while maintaining comparable accuracy on supported examples. Further experiments show that this behaviour generalizes across different model families and retrieval algorithms, but fails to generalize effectively to other forms of insufficient evidence, particularly partial evidence in multi-hop QA.

Critical strengths

The studied problem is important for reliable LLMs and agents. Determining whether the available evidence is sufficient to answer a question is useful not only for avoiding unsupported answers, but also for deciding when to retrieve more information or ask clarifying questions.

Critical weaknesses
  1. Limited novelty. The main method is essentially a simple data augmentation strategy combined with weighted SFT. Some important ablations are missing to justify the proposed design, such as whether the clean context C+C^+ is necessary, and whether matched triplets provide any advantage over standard SFT with independently constructed answerable and unanswerable samples.

  2. The authors do not clearly specify how many documents are included in each context, or how many distractors are retrieved to construct C±C^\pm. In addition, the evaluation setup is unclear. For example, does Supported Gold evaluate on C+C^+ (support only) or C±C^\pm (support with distractors)? Moreover, constructing C0C^{0} by removing supporting documents from C±C^\pm may introduce systematic differences in document numbers and context length between answerable and unanswerable examples. Without controlling for these differences, it is unclear whether the model learns evidence sufficiency or simply relies on these superficial patterns. These missing details make the experiments difficult to reproduce and the results difficult to interpret.

  3. Limited generalization. The main training and evaluation settings are in-domain, and the near-perfect UAR may simply reflect overfitting to the specific document construction patterns. Although the method transfers to dense retrieval misses, it fails to generalize effectively to more realistic insufficient-evidence scenarios, particularly partial evidence in multi-hop QA, where UAR remains at 83.07%. In addition, the authors do not report the correctness of answers generated on unsupported contexts, making it difficult to understand the model's actual behaviour. More detailed analyses of answer correctness and confidence calibration, especially under distribution shifts, would help better establish whether the model learns a meaningful evidence-sufficiency boundary.

Questions for authors

Please see the weaknesses above for my main questions and concerns.

Reviewer_hFrROpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes Regime Boundary Alignment (RBA), a supervised fine-tuning approach for evidence-grounded question answering. Each training question is paired with supporting documents, supporting documents plus distractors, and distractors without supporting documents. The model learns to answer in the first two conditions and abstain in the third. The experiments report substantially reduced unsupported answering on retrieval-miss evaluations while approximately preserving supported-answer accuracy.

Critical strengths

The paper addresses a relevant problem. The method is straightforward to implement and requires no additional inference-time components.

Critical weaknesses
  1. The contribution beyond conventional answerability training is insufficiently established. The proposed objective is a weighted supervised likelihood loss over answer and abstention targets. The distinctive contribution therefore appears to be the matched context construction. However, the experiments do not isolate its advantage over conventional fine-tuning with comparable answerable and question-relevant unanswerable examples. Conflict-SFT receives no abstention supervision, so the main comparison primarily demonstrates the benefit of adding that supervision. 


  2. The DTA comparison does not adequately establish superiority over existing abstention methods. DTA is a relevant comparator because it explicitly trains answering and abstention through preference optimization. However, its answering policy differs from RBA’s: DTA permits answering from parametric knowledge when retrieved evidence is insufficient, whereas RBA requires abstention whenever the supplied evidence lacks support. Consequently, this evaluation can penalize DTA for behavior it was deliberately trained to produce. The manuscript does not sufficiently explain its adaptation to the evidence-only policy or its reported 100% unsupported answering and 33.74% supported-answer accuracy. Without a comparison under a common answering policy and comparable training conditions, these results are difficult to interpret.


  3. The evaluation construction is insufficiently documented, and the results suggest limited generalization. The main test set contains held-out benchmark questions, each with supported and BM25 no-support views. The relationship between these views and the support-removal construction described in Section 3.2 is unclear: the manuscript does not explicitly establish whether the test no-support contexts are naturally occurring retrieval misses or are formed by removing annotated support. This distinction matters for assessing practical relevance and possible shared construction cues between training and evaluation.

Table 2 shows a relatively high unsupported answer rate: 83.07% on partial evidence and 99.30% on semantic negation. This suggests that the learned behavior is specific to retrieval misses rather than a general ability to determine evidence sufficiency. Possible overfitting to construction-specific regularities is a concern.

  1. Additional answerability baselines and independent evaluation would strengthen the contribution. A conventional answer/abstention fine-tuning baseline using comparably relevant negative contexts would directly test the value of matched construction. A context-sufficiency classifier coupled to the same reader would provide another useful practical comparison. Evaluation on SQuAD 2.0, or another independently constructed unanswerable-QA benchmark, would help assess whether the learned behavior extends beyond the current evaluation setup. Although SQuAD 2.0 differs from multi-hop RAG, it directly tests recognition that a passage does not support an answer.

  2. The prompting baseline and unusually low unadapted performance of the QWEN model require better justification. The unadapted Qwen model achieves only 2.85% macro supported-answer exact match. Exact prompts, chat templates, and answer-processing rules are absent. The prompting comparison is also insufficiently documented. A validation-selected prompt, potentially using an optimizer such as GEPA, would provide a more informative comparison.

  3. The experimental section is difficult to follow and lacks reproducibility details. The manuscript does not clearly specify the per-dataset training mixture, source splits, distractor provenance and construction, filtering criteria, or selection of the 147 hard-replay examples. The supplied PDF contains no appendix resolving these omissions. While the earlier sections are better organized, the experimental description makes the scope and interpretation of the results difficult to assess.

Questions for authors

Please provide any answers that can clarify the critical weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.