acceptodds
Under review as a conference paper at ICLR 2027

Why Errors Survive Selective Grounded Correction

Abstract

Large language models (LLMs) can make mistakes even when given the sources needed to answer. A support score is the fraction of output claims fully supported by those sources. But does a higher score mean more mistakes were fixed? Across legal, document, and open-domain question answering, we study selective correction, which edits only claims a checker marks as errors. Holding the first edit fixed, we compare removing claims that still fail with editing them again. We find that an answer can say less and score higher, or fix a mistake and keep the same score. Editing again restores support to 37 claims that removal would discard, yet changes average support by only to percentage points. Weaker wording can also raise scores: adding "It appears that" before checking and removal raises support by 2.0–2.5 points over removal alone in our legal experiments. Human annotators find that all 52 sampled hedges narrow the assertion and that some credited repairs edit already supported claims. In 17 of 18 traced configurations, most surviving errors are not sent back for another edit, and failed repairs dominate the other. We call for reporting what each step repairs, preserves, and removes, with its token cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
grounded correction, claim verification, llm evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zLSmOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper discusses the issue of errors in RAG to survive a first round of corrections. It compares the options of rejecting claims that are found to be incorrect, editing them, or revisiting claims that had been ignored in the initial response. A set of experiments is reported on a combination of existing datasets and a variety of models open and closed models. The key finding is that evaluation should not report only the accuracy fn the answer predicted, but also how the coverage. This would give a better assessment of the impact of the edits done on the answers.

Critical strengths

I appreciate the experiments with a wide range of models and datasets.

Critical weaknesses
  • The main weakness of the paper is the lack of novelty: The fact that one should measure both precision (the proportion of what is returned is accurate) and recall (the proportion of what should be returned is correct) in the output of LLMs (and broadly any NLP system) is a well-established practice. There is always a trade-off between returning more information and returning more accurate information in the output, which is why it is advisable to use precision-recall curves and the area under them. The FactScore paper (https://arxiv.org/pdf/2305.14251) makes this clear in its limitations section and it makes the same recommendation, to report both precision and recall

  • While I appreciate the long list of references in the related work section, it would be better to mention fewer and elaborate on how they relate to the work in this paper.

Questions for authors
  • it is unclear why the application context makes a difference to this; what is special about being after the first edit? What is different from just taking the output of an LLM?

  • Could you give an algorithm for the process described in Figure 2? Intuition from a figure is nice for intuition, but an algorithm offer greater precision

Reviewer_Gu4cOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies whether a higher support score actually indicates that LLM edits have corrected more mistakes. Across three question-answering domains, the paper investigates selective correction. By holding the first edit fixed, the authors compare removing unsupported claims with editing them again. Their main finding is that the same support score can hide whether the model has performed a useful repair. The paper calls for more detailed reporting of what each editing step actually changes.

Critical strengths
  • The motivation and problem definition are relevant, addressing a critical limitation of support scores.
  • The selective workflow allows for an interesting decomposition of changes in the support score.
  • The paper evaluates a range of domains and models and presents interesting findings on why the score stalls.
Critical weaknesses
  • First, the appendix is about 70 pages long and thus rather extensive. The results section contains numerous references to appendix tables. While the effort is commendable, it is infeasible for a reviewer to verify every claim against all these tables.

    • In addition, the paper is extremely dense, which affects its readability.
  • The limitation of precision as a metric, namely that it can remain high despite missing relevant cases, is well established. The support score suffers from the same problem, as it essentially functions as a precision-style metric. I therefore do not consider the finding that "an answer can say less and score higher, or fix a mistake and keep the same score" to be the main novelty of the paper. Instead, I find the experimental results explaining why the score stalls more valuable. Overall, I consider the main contribution somewhat weaker than the paper suggests.

Questions for authors

see above

Reviewer_PrdpOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies selective grounded correction. A checker flags claims that their cited source does not support, an editor rewrites the flagged claims, the rewrites are re-checked, and claims that still fail are removed. The question is what a further step adds after one edit. The authors hold the first edit fixed and compare stopping, which only filters, with retrying, which edits the re-flagged rewrites again. The comparison covers legal, document and open-domain QA with several generators and editors. They find that another edit barely moves the support score while saving some claims that stopping would delete. Filtering alone raises the score. A mechanical hedge added to accepted claims raises the score on legal tasks. Most surviving errors were never flagged in the first place. Human audits confirm that hedges narrow what a claim asserts and that part of the judge-credited repairs were already supported before editing. The paper concludes that a support score cannot distinguish repair from removal or hedging and calls for reporting what each step fixes together with its cost.

Critical strengths
  1. The paper makes one point that the grounded-generation community needs to hear. A support score rewards deletion and hedging as much as repair, so a pipeline can look better while saying less. The shared-prefix ladder is the right way to show this, since it separates filtering from a second rewrite. The before/after human audit is the right instrument for separating a correction from a narrower assertion.

  2. The reporting is unusually honest. Every contrast comes with an interval, negative and null results are kept, there is a second judge, a held-out confirmation set and a pre-registered policy test, and the paper states repeatedly what its design cannot identify.

Critical weaknesses
  1. The headline finding follows from the pipeline design and from the strictness of the judge. The second check inspects only rewrites, so an error that the first check missed can never enter the next edit list. That most surviving errors lie outside that list is a property of the procedure the authors chose. A fresh full-answer check at round two is untested.

  2. The effect sizes are too small to carry the conclusions. Retry rescues a few dozen claims out of about ten thousand, filtering moves support by under one point in most configurations, and the hedge result holds on legal only. Table 17 shows no correctness interval excluding zero in any configuration, so no downstream measure changes. The tension between factual precision and how much an answer says is also an established topic, and the paper does not connect to that work.

Questions for authors
  1. What does the 17 of 18 result become when the second round checks every claim with a fresh call?

  2. How should the reader interpret "error" when most strict errors are partial-support verdicts? Would the authors restate the abstract in terms of missing qualifications?

  3. Table 12 shows that humans judge fewer than half of the judge-credited legal hedges as fully supported after editing. How much of the hedge gain comes from the judge accepting hedged claims that humans reject?

  4. Is there any downstream endpoint on which stopping and retrying differ?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.