acceptodds
Under review as a conference paper at ICLR 2027

ClaimSift: Selective Answer Revision in Large Language Models under Mixed Feedback

Abstract

Large language models (LLMs) can revise answers using feedback. With mixed correct and incorrect feedback proposals, however, successful revision requires judging the proposals, updating dependent items, and preserving initially correct unrelated items. Final Accuracy measures the revised answer but obscures these decisions. Moreover, complete revisions with the same revision reward can cause different damage to initially correct items. This raises the question of how to learn selective revision while accounting for this damage. We introduce ClaimSift, a ProofWriter- and FOLIO-based benchmark for evaluating complete revisions, and propose Preservation-Aware Revision Optimization (PARO). PARO combines revision reward with explicit preservation cost, while a shared penalty weight adjusts to observed risk relative to fixed risk budgets. Experiments on ProofWriter show that PARO achieves the highest Overall Success among evaluated Qwen3-8B trained revision methods, exceeding TD-DPO and DPO by 9.75 and 9.59 percentage points, respectively, and improving Content Preservation over DPO by 3.42 points.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, selective answer revision, mixed feedback, content preservation, reinforcement learning, benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_odpgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose ClaimSift, a benchmark built on ProofWriter and FOLIO for evaluating selective revision under mixed correct and incorrect feedback, and PARO, a constrained RL method that combines revision reward with explicit preservation cost and an adaptive Lagrangian penalty weight calibrated to a reference policy's damage rate. ClaimSift decomposes revision evaluation into five dimensions — Proposal Judgment, Target Correction, Proposal Resistance, Dependency Propagation, and Content Preservation — and defines Overall Success as their conjunction on each case. Experiments on Qwen3-8B show PARO achieves 54.47% Overall Success on ProofWriter, exceeding DPO and TD-DPO by ~9.6–9.8 points, with Content Preservation improving 3.42 points over DPO. Additional analysis reveals that the adaptive penalty's advantage over a development-tuned fixed penalty is small (+1.02 on ProofWriter, +2.72 on FOLIO) and not statistically established as an independent effect.

Critical strengths
  1. Careful evaluation decomposition. The five-metric framework (TC, PR, DP, CP, OS) with the Section 3.2 six-model diagnostic (Table 15) cleanly demonstrates that no single component predicts Overall Success — Llama-3.1-8B reaches 88.67% TC but only 2.60% OS, Qwen3-14B reaches 97.09% CP but only 3.49% TC — providing genuine diagnostic value that Final Accuracy alone cannot offer.

  2. Thorough benchmark construction. The paired-variant design with blocking/restoration dependency validation (Appendix B.1), four feedback scenarios per problem, and explicit admission criteria create a benchmark where every ground-truth label and dependency annotation is semantically verified rather than heuristically assigned.

Critical weaknesses
  1. Benchmark domain is too narrow to support general claims about "selective revision in LLMs." Both ProofWriter and FOLIO use binary TRUE/FALSE labels over clean logical structures with perfect ground truth and mechanically verifiable dependencies. Real revision tasks involve natural-language answers with partial correctness, fuzzy dependencies, and no oracle decomposition into target/dependent/unrelated items. The method requires ground-truth labels and dependency annotations during training (Eq. 4–5), making it inapplicable to open-domain settings without substantial additional machinery. The title and abstract frame the contribution as general, but the evidence covers only formal logic with binary labels.

  2. The adaptive penalty provides marginal and statistically unconfirmed gains over a tuned fixed penalty. On ProofWriter, PARO achieves 54.47% vs. Tuned Fixed 53.45% (+1.02 points); on FOLIO, 55.66% vs. 52.94% (+2.72 points). The paired analysis (Table 30) shows PARO gains 8.37% of cases over Fixed 1.0 on ProofWriter but also loses 7.34%, with a net of just 1.02 ± 1.89 — the SD exceeds the mean. The paper itself acknowledges this (Section 6.4). Without this, PARO's contribution reduces to "adding a preservation cost to RL improves revision," which is unsurprising given the objective explicitly rewards not damaging correct items.

  3. No ablation of reward component weights or controller hyperparameters. The reward weights (αs=0.5, αc=0.25, αj=0.25), reference coefficient (τ=0.01), and controller settings (λ0=1.0, η=0.1, λmax=10) are fixed throughout (Table 6) with no sensitivity analysis. Given that the gap between PARO and Fixed 1.0 is ~1–3 points, these design choices could easily be load-bearing. A different (αs, αc, αj) split or a different η could shift the balance between correction and preservation enough to close or widen the gap, but this is never tested.

Questions for authors

Have you considered evaluating on a non-logical, non-binary revision task to test generalization?

PARO requires ground-truth labels and dependency annotations during training. How would the method generalize to domains without oracle item-level correctness?

2 comments
AuthorsOpenReview ↗

Response to Reviewer odpg on Task Scope and PARO

Thank you for identifying the value of the component scores and asking how far the evaluation and the adaptive mechanism support our claims. These are distinct questions, and we address each below.

W1/Q1. Non-logical and non-binary revision. ClaimSift's original design covers structured binary answer revision with supplied evidence and roles annotated for scoring. The paired ProofWriter variants keep unrelated labels fixed while changing the target and dependent labels. This lets the evaluation distinguish correction, dependent changes, and retention within one answer. The saved-response diagnostic shows why that matters: Llama-3.1-8B-Instruct reaches 88.67% Target Correction and 2.60% Overall Success, while Qwen3-14B reaches 97.09% Content Preservation and 3.49% Target Correction (original Figure 2 and Table 15). FOLIO uses an in-domain binary protocol. The revised abstract and Section 4 state this task scope. The submission reports no free-form or non-logical revision evaluation. Such an evaluation would need a rubric for partial item correctness and for which dependent edits are required.

W1/Q2. Item-level supervision. Training uses correct values to score repairs and damage, and dependency annotations to score required updates (Eqs. 4–5 and Appendix B). At inference, the model receives evidence, queries, the initial answer, and proposals. The labels and dependency annotations stay with the scorer. In an unannotated domain, a verifier could provide the training signals if it can reliably distinguish new damage from remaining errors and identify required dependent updates. This would need validation against independent annotations before using the verifier's outputs for training.

W2. Preservation cost and adaptive weighting. Consider two revisions that repair the same item, judge both proposals correctly, and miss the same dependent update. With a positive preservation weight and equal revision reward and reference log-ratio, the revision that also damages an initially correct item receives a lower return (original Eqs. 4–6). This distinction applies even when neither revision passes Overall Success.

The original comparison with Development-tuned Fixed 1.0 reports mean Target Correction gains of 7.46 pp on ProofWriter and 7.55 pp on FOLIO (Tables 4–5). Under the reported pairing, the summaries imply gains in at least four of five repetitions per domain (revised Appendix F.8). Reported mean damage across all initially correct items falls from 12.40% to 11.19% and from 15.88% to 14.44%, respectively (original Tables 23–24). The original paired Overall Success differences are +1.02 ± 1.89 pp and +2.72 ± 4.81 pp (mean ± SD, Table 30). These summaries give clearer evidence for Target Correction than for consistent Overall Success gains.

W2. Dense correctness reward. With the same initial answer, two candidates can finish with the same number of correct items. For example, one can repair one more item and damage one more. A dense final correctness reward gives the two candidates the same score. With the other return terms fixed, unequal effective weights for repair and damage can distinguish them. When the effective weights are equal, the combined repair and damage terms and a matched dense correctness reward rank candidates identically (original Appendix H.3). For Eqs. 4–5, matching requires accounting for the normalization denominators (revised Appendix H.3). Comparing trained policies would require selecting the dense coefficient on Development and holding the other reward terms, reference, and policy update fixed.

W3. Sensitivity. Mean Overall Success peaks at 1.0 among the tested fixed weights in both domains.

Table R1. Overall Success under fixed preservation weights (%, original Tables 4–5).

Dataset0.51.02.0
ProofWriter51.9553.4552.16
FOLIO48.7752.9451.00

On ProofWriter, increasing the weight from 0.5 to 2.0 raises Content Preservation from 96.64% to 97.01% and lowers Target Correction from 79.33% to 75.20%. This sweep measures sensitivity to the fixed preservation weight. The reward mix, reference coefficient, and adaptive controller settings are held fixed (original Table 6). Sensitivity to those settings remains an open question.

Thanks for your response, and I will keep my rating.

Reviewer_MccGOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces ClaimSift, a benchmark for evaluating LLM answer revision under mixed feedback, covering error correction, feedback judgment, dependency propagation, and content preservation. It also proposes PARO, which penalizes damage to initially correct answers with an adaptive weight. Experiments on ProofWriter and FOLIO show improved overall revision success over existing baselines.

Critical strengths

The paper addresses the practically important problem of selective answer revision in LLMs under mixed correct and incorrect feedback.

Critical weaknesses
  1. The paper is difficult to follow due to unclear explanations, excessive terminology, and poorly organized presentation. The authors should prioritize improving its readability and clarity.
  2. The data construction process lacks sufficient detail. It is unclear how much of the dataset was generated automatically or with human involvement, how many annotators were involved (if any), and how the quality of the constructed data was verified.
  3. Several design choices lack sufficient justification, including the fixed structure of benchmark instances, the selection of feedback scenarios, and the definitions of evaluation metrics. The authors should better motivate these choices and explain their implications for the evaluation.
  4. The core technical contribution of PARO is not sufficiently isolated. Most improvements appear to come from the preservation penalty, while the independent benefit of the adaptive weighting mechanism remains unclear. Stronger comparisons with standard RL methods using the same reward are needed.
Questions for authors

See critical weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer MccG on ClaimSift and the Preservation Controls

Thank you for pointing out where the presentation made the task and the ablations difficult to interpret. Your questions help us separate how the cases are built, what the metrics measure, and which training choices the controls isolate.

W1. Readability. We now use Figure 1 to follow one revision: accept the gate correction, update the terminal that depends on it, reject the incorrect seat proposal, and retain the departure time. Section 3 follows the same order from model input to proposal judgments, answer items, and scoring. In original Table 13, two revisions have identical final values and pass the same answer checks. Changing ACCEPT to REJECT makes Overall Success fail without changing Final Accuracy. This is why we report judgments alongside answer scores. We use Content Preservation for the evaluation of initially correct unrelated items and preservation cost for the training term that counts damage across all initially correct items (Eq. 5).

W2. Construction and validation. During discussion, one external colleague independently checked 96 answer labels in 16 ProofWriter variants from eight source families without seeing the reference labels. All 96 judgments matched the reference labels (revised Appendix B.1).

Original Table 14 lists 1,964 ProofWriter and 1,994 FOLIO candidate variants, with two variants per source family. The variants share queries and unrelated labels while reversing target and dependent labels. Original Appendix B.1 describes how to check that blocking annotated support removes dependent conclusions, restoring it recovers them, and unrelated labels stay fixed.

Table R1. Dependency checks on the first-order example (original Table 9).

InterventionTarget and dependent labelsUnrelated labels
Support blockedUUUFFF
Support restoredTTTFFF
Blocked, alternative support remainsTTTFFF
Unrelated premise removedTTTUUU

T, F, and U denote entailment, entailment of the negation, and an unresolved query. The last two rows fail necessity and invariance, respectively. ProofWriter uses closed-world semantics. FOLIO admission requires entailment of each query or its negation.

The original AI Use Statement reports assistance with synthetic dataset generation. The original submission does not quantify human participation in dataset construction.

W3. Fixed structure, feedback scenarios, and scoring. One target, two dependents, and three unrelated items let the same case test necessary edits and retention. The single-proposal scenarios vary the target proposal's correctness. The paired two-proposal scenarios hold the case and first proposal fixed and vary whether the proposal on the initially correct unrelated item is correct. For example, a correct target proposal on an already correct item should be accepted and kept. An incorrect proposal that repeats an initial error should be rejected and the target changed (original Table 8). Section 4 explains the two controlled contrasts and refers to Table 8.

Overall Success requires the applicable answer checks and correct, consistent judgments. Component scores locate the failure. In original Appendix B.4, the position rule scores four of six judgments in the original order and two of six after reversal. On dual-proposal cases, accepting the first and following initial-answer agreement for the second gives both labels by construction. These results quantify the available cues. Assessing whether a trained model uses them requires its responses to reordered proposals.

W4. Preservation cost, adaptation, and the RL update. In the original ablation, adding a fixed preservation cost improves Content Preservation and Overall Success while reducing Target Correction. Both arms use the same revision reward and the same REINFORCE update with a leave-one-out baseline (original Eqs. 7–8). The added cost changes the complete return.

Table R2. w/o Preservation → Fixed 1.0 (%, original Tables 4–5).

DatasetTarget CorrectionContent PreservationOverall Success
ProofWriter92.40 → 77.4789.70 → 96.1739.54 → 53.45
FOLIO86.15 → 71.5087.56 → 94.1038.33 → 52.94

This contrast isolates the extra damage signal. The adaptive comparison changes the weight schedule (revised Sections 6.3–6.4). Under the reported pairing, the Target Correction summaries imply gains in at least four of five repetitions per domain (revised Appendix F.8). The reported mean Overall Success gains over Fixed 1.0 are +1.02 and +2.72 pp, with uncertainty across repetitions (original Table 30). A matched RL comparison would keep the full return, initialization, and sampling budget fixed and vary the policy update.

Thank you for the detailed response and clarifications, and apologies for my delayed reply. I appreciate the efforts to clarify the benchmark design and improve the presentation. However, my concerns about dataset validation and the independent contribution of adaptive weighting remain insufficiently addressed. I will therefore maintain my current rating.

Reviewer_zaVzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ClaimSift, a benchmark for evaluating selective answer revision under mixed correct and incorrect feedback. The authors propose PARO, a preservation-aware optimization method that combines revision reward with an adaptive penalty for damaging correct items.

Critical strengths
  1. The paper addresses a meaningful limitation in existing revision evaluations by considering whether models can correct errors while preserving already correct information. The joint evaluation of feedback judgment, dependency propagation, and content preservation is useful and goes beyond measuring final answer accuracy.

  2. The experiments provide a reasonably systematic comparison of preservation-aware training methods. The results suggest that explicitly penalizing damage to initially correct items can improve the balance between correction and preservation.

Critical weaknesses
  1. The contribution of the proposed adaptive optimization method is not sufficiently established. Although PARO improves Overall Success over DPO and TD-DPO by approximately 10 percentage points, much of this gain is already achieved by a fixed preservation penalty. Compared with the tuned fixed penalty, PARO improves Overall Success by only 1.02 points on ProofWriter and 2.72 points on FOLIO. The authors also acknowledge that the independent benefit of adaptation is not statistically established. This makes it difficult to determine whether the adaptive penalty mechanism provides a meaningful improvement over simpler preservation-aware optimization.

  2. The benchmark is relatively narrow compared with realistic mixed-feedback revision. Each problem follows a fixed structure of one target, two dependent items, and three unrelated items, with feedback generated from predefined scenarios. While this provides a controlled evaluation, it is unclear whether the findings generalize to more complex dependencies, multiple conflicting corrections, or natural-language answers. Furthermore, the Keep All baseline achieves higher aggregate Overall Success than PARO on both datasets, suggesting that the metric is sensitive to the proportion of cases requiring correction. This weakens the interpretation of Overall Success as the primary measure of selective revision ability.

Questions for authors
  1. Can the authors provide stronger evidence that the adaptive penalty offers a consistent advantage over a tuned fixed penalty, beyond the relatively small gains reported in Tables 4 and 5?

  2. How would PARO perform with more varied dependency structures, larger answer sets, and less structured feedback?

2 comments
AuthorsOpenReview ↗

Response to Reviewer zaVz on Preservation and Evaluation Scope

Thank you for distinguishing the effect of a preservation cost from the effect of adapting its weight, and for asking how the test mix affects Overall Success. We address these questions in order.

W1/Q1. What does adaptive weighting add? Compared with Development-tuned Fixed 1.0, PARO improves mean Target Correction by 7.46 percentage points on ProofWriter and 7.55 on FOLIO (original Tables 4–5). The two methods share the revision reward, preservation cost, and policy update, and differ in how they set the weight. Under the reported pairing, the summaries imply higher Target Correction in at least four of five repetitions per domain (revised Appendix F.8). Reported mean damage across all initially correct items falls from 12.40% to 11.19% on ProofWriter and from 15.88% to 14.44% on FOLIO (original Tables 23–24).

Table R1. Paired Overall Success relative to Fixed 1.0 (original Table 30). Gain and Loss give the percentages of cases that succeed with PARO alone and Fixed 1.0 alone, respectively.

DatasetGainLossNet, mean ± SD (pp)
ProofWriter8.37%7.34%+1.02 ± 1.89
FOLIO10.16%7.44%+2.72 ± 4.81

From the five-repetition summaries in original Table 30, the approximate 95% paired t intervals are [−1.33, +3.37] pp for ProofWriter and [−3.25, +8.69] pp for FOLIO. Reported mean Overall Success favors PARO in both domains, and both intervals include zero.

W2/Q2. What does the fixed task structure cover? The fixed roles make it possible to vary whether revision is necessary while checking which related items must change and which unrelated items must stay correct. The six-model diagnostic illustrates the value: Llama-3.1-8B-Instruct reaches 88.67% Target Correction and 2.60% Overall Success, while Qwen3-14B reaches 97.09% Content Preservation and 3.49% Target Correction (original Figure 2 and Table 15). Each profile uses that model's admission and decoding settings. For larger answer sets or more complex dependencies, the same checks can be applied once the required edits are annotated. Less structured feedback would also require identifying which item and value each proposal refers to. The reported model comparison covers the original structured setting. Original Appendix H.1 tests input cues in a balanced design using rules.

W2. Why does Keep All lead in aggregate? The two decisions are whether to accept a proposal and whether to update the initial answer.

Table R2. Required judgments and actions (original Table 8).

Initial targetCorrect target proposalIncorrect target proposal
IncorrectAccept and updateReject and update
CorrectAccept and keepReject and keep

Keep All cannot make the required update when the target starts incorrect. Accepting a proposal that repeats that error fails Proposal Judgment, and rejecting it while retaining the proposed value fails Judgment–Revision Consistency. When the target starts correct, Keep All preserves the correct values and gives the required judgments, attaining the bound in original Eq. 16.

Table R3. Overall Success by initial target correctness, pooling counts across the four feedback scenarios and five runs (original Tables 31–32).

DatasetInitially incorrect: PARO / Keep AllInitially correct: PARO / Keep All
ProofWriter56.60% / 0.00%53.42% / 100.00%
FOLIO55.44% / 0.00%55.76% / 100.00%

Initially correct targets account for 203/303 ProofWriter and 218/309 FOLIO problems (original Tables 12 and 14). With that mix, Keep All leads 67.00% to 54.47% on ProofWriter and 70.55% to 55.66% on FOLIO (original Table 2). Holding subgroup scores fixed, the ranking changes at 54.86% and 55.62% initially correct targets, respectively (original Appendix F.3). Table R3 shows what the aggregate masks: PARO completes many revisions that need a correction.

Thank you to the authors for the detailed response and additional analysis. However, my main concern remains that PARO's advantage over a tuned fixed penalty is relatively small and not statistically established in terms of Overall Success. The evaluation also remains limited to a fixed, structured revision setting, without further evidence of generalization to more complex tasks.

Overall, while the rebuttal helps clarify the contributions, I still have some reservations about the significance of the benchmark,. I will therefore maintain my score of 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.