acceptodds
Desk rejected at ICLR 2027

When Revision Backfires: Consequence-Aware Verification for Multi-Role LLM Reasoning

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Verifier-guided reasoning is usually drawn as a monotone pipeline: a solver proposes, a verifier checks, a reviser repairs. That picture hides an asymmetry. Sending an answer back for revision is an action, not a prediction: replacing a correct answer with a wrong revision turns a point into zero, whereas keeping a wrong one merely preserves the status quo. Across 18 long multi-role math runs, unnecessary revision, not verdict error in general, is the failure most strongly tied to lost end-to-end value, negative within every run. We recast verification as consequence-aware control and introduce Keeper, a preserve-first policy family that routes only what is worth rewriting and restores the solver answer when evidence for replacing it is weak. On a held-out controller panel, Keeper improves three outcomes by 12.3–14.9 points and turns 17 of 19 repair opportunities into useful actions while degrading none. With the first-pass solver and reviser fixed, it beats a shared checker in all six comparisons while spending 13.5% fewer response tokens than always revising. A same-scale 7B preserve-first student, frozen and preregistered, improves all 12 checkpoints trained after it was frozen and 29 of 35 workflows of a problem-disjoint split with six unseen task families, blocking 99% of harmful revisions; a same-data variant improves every one of those 35 workflows over its gate. Score verifiers by the consequences of the routes they trigger, not by verdicts in isolation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
verifier-guided reasoning, multi-role LLM systems, selective revision, cost-sensitive control, self-correction, LLM reasoning, LoRA

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ZvX8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper identified the problem of monotone pipeline of using solver, verifier and revisor. The revisor may wrongly modify the correct answer, leading to performance degradation. To mitigate this issue, this work proposed KEEPER, a preserve-first policy family that routes only what is worth rewriting and restores the solver answer when evidence for replacing it is weak. The experiment shows that the framework can improve the performance while generating fewer tokens.

Critical strengths

The motivation is clear and problem is clearly formulated and explained.

The experiment shows that the proposed method works better than other pipelines while saving tokens.

Critical weaknesses
  1. The experiment only evaluate on Qwen2.5- 7B-Instruct. It is necessary to evaluate on more models to show it generalizability, such as different model family and different sizes.

  2. The content is not well organized. For example, Section 3 lacks focus and is difficult to follow. Section 4 should show the main results and much of its content should be moved to Section 5.

  3. Too many statements are likely written by AI and hard to interpret. It is OK to polish the paper with AI but it is important to ensure the readability after polishing. For example, line 209-211, too much information is compressed in the sentences and is not explained clearly.

Questions for authors

Is it possible to train the three roles (solver, verifier and revisor) in a single model?

In Figure 2, revision candidates are not always available, right? If so, the figure should be revised to reflect this.

In Table 1, what is r? Please define it explicitly.

4 comments
AuthorsOpenReview ↗

Response to Reviewer ZvX8 (Part 1/4)

We thank Reviewer ZvX8 for a direct and constructive review. The original submission evaluated only Qwen2.5-7B-Instruct on math (W1), mixed protocol, main results, and boundary analysis across Sections 3-5 (W2), compressed too many clauses into dense sentences such as lines 209-211 (W3), and left several architectural and statistical details implicit (Q1-Q3). In response, we trained and evaluated cross-scale (Qwen2.5-3B) and cross-family (Llama-3.1-8B) 5-fold suites across both mathematics and competitive Python programming (APPS) and restructured and rewrote the manuscript. Below is a summary of revisions followed by point-by-point responses.

Summary of Revisions

  1. W1 (generalize beyond Qwen2.5-7B across model scales, model families, and non-math domains) (PDF §4.3 p. 7, App. L-N, Tables 30-31, 34-35, Figs. 5b, 6):
    • 30 matched 5-fold SVAMP adapters (Qwen2.5-3B, Qwen2.5-7B, Llama-3.1-8B): Consequence beats Correctness in 45/45 individual fold-reviser cells (5/5 folds in every row; +1.99 to +2.37 pp on Qwen2.5-3B and +5.39 to +5.41 pp on Llama-3.1-8B, ensemble +5.75 pp).
    • 50 strong-reviser OOF competition math adapters (Qwen2.5-7B and Llama-3.1-8B): At default τ=0\tau = 0 and 8.21% revision rate, pre-revision KEEPER beats Always Revise (83.89%) on both Qwen2.5-7B (85.56%, 0 harms) and Llama-3.1-8B (85.11%, 1 harm), and K=32K = 32 zero-gradient calibration reaches 82.06% / 82.04% across 7B / 8B.
    • 82 adapters on APPS competitive Python programming (n = 500 problems, 54 workflows: 31 Qwen2.5-7B + 23 Llama-3.1-8B): Consequence beats Correctness across Qwen2.5-3B (+1.41 pp strict), Qwen2.5-7B (+0.65 pp zero-shot, +0.84 pp 5-fold OOF), and Llama-3.1-8B (+1.16 pp zero-shot, +0.95 pp 5-fold OOF pre, +0.73 pp post, p≤0.0080p \le 0.0080), preventing 100% of broken-program overwrites.
  2. W2 (reorganize Sections 3, 4, and 5) (PDF §2-§5 pp. 2-9): Restructured the manuscript so §2 (Verification as Control) defines the problem and runtime contract, §3 (Evaluation Protocol) states the experimental protocol cleanly, §4 (Results) presents the main empirical findings, and §5 (Analysis and Deployment Implications) collects regime boundaries and deployment monitoring.
  3. W3 (rewrite compressed prose, including lines 209-211) (PDF §2.3 pp. 3-4, §3 p. 5): Unpacked multi-clause sentences throughout into clear single-claim sentences, including original lines 209-211 (now three short sentences in §3, lines 219-222) and the preserve-rule definition (§2.3, lines 153-161, Eq. 5).
  4. Q1 (training solver, verifier, and reviser in a single model) (PDF §2.4 p. 4, §4.5 Table 5 p. 9, Table 33): Explicitly documented how one frozen base model hosts all three roles via either a single shared LoRA or lightweight per-role LoRAs switched per turn.
  5. Q2 (Figure 2 when revision candidates are not generated) (PDF §2.3, Table 1, Fig. 2 p. 4): Replaced Figure 2 with a two-panel diagram where Mode A returns asa_s immediately on KEEP with no revision candidate generated.
  6. Q3 (define rr in original Table 1, now Table 2) (PDF §4.1 p. 6, Table 2): Defined rr in prose and caption as the sample Pearson correlation across the 4,186 RL snapshots (r=−0.259r = -0.259, negative in 18/18 runs).
AuthorsOpenReview ↗

Response to Reviewer ZvX8 (Part 2/4)

W1: Multi-Backbone (Qwen2.5-3B, Qwen2.5-7B, Llama-3.1-8B) and Non-Math Code (APPS) Generalization

W1: "The experiment only evaluate on Qwen2.5-7B-Instruct. It is necessary to evaluate on more models to show it generalizability, such as different model family and different sizes."

We expanded evaluation across both model scale (Qwen2.5-3B-Instruct) and model family (Llama-3.1-8B-Instruct) in three multi-fold suites spanning math and executable Python programming (Appendices L-N, Tables 30-31, 34-35, Figures 5b and 6):

1. 30-adapter 5-fold strict parity on SVAMP (n = 887, 735 source groups; Table 30). Trained on identical 17,655-row problem-disjoint splits with verified-identical initial LoRA tensors, RNG seeds, and 1,104 steps/fold (zero external teacher calls at test time), Consequence beats Correctness in every fold of all nine backbone ×\times reviser pairs (45/45 cells):

Student backbone (10 adapters)ReviserGateCorrectnessConsequence5-Fold Δ\Delta (t95t_{95} CI)Ensemble Cons / Corr (Δ\Delta)
Qwen2.5-3B-InstructUntrained82.3086.70 ±\pm 0.4888.68 ±\pm 0.07+1.99 ([+1.37, +2.60])88.61 / 86.58 (+2.03)
Qwen2.5-3B-InstructShared82.3086.31 ±\pm 0.6288.68 ±\pm 0.07+2.37 ([+1.59, +3.15])88.61 / 86.13 (+2.48)
Qwen2.5-3B-InstructDual82.4186.52 ±\pm 0.5488.68 ±\pm 0.07+2.17 ([+1.48, +2.86])88.61 / 86.36 (+2.25)
Qwen2.5-7B-InstructUntrained82.3087.49 ±\pm 0.2388.73 ±\pm 0.08+1.24 ([+0.94, +1.54])88.73 / 87.26 (+1.47; vote +1.58)
Qwen2.5-7B-InstructShared82.3087.42 ±\pm 0.2788.73 ±\pm 0.08+1.31 ([+0.94, +1.68])88.73 / 87.15 (+1.58; vote +1.69)
Qwen2.5-7B-InstructDual82.4187.30 ±\pm 0.2688.73 ±\pm 0.08+1.42 ([+1.07, +1.78])88.73 / 87.03 (+1.69; vote +1.80)
Llama-3.1-8B-InstructUntrained82.3083.20 ±\pm 0.6288.61 ±\pm 0.00+5.41 ([+4.64, +6.17])88.61 / 82.86 (+5.75)
Llama-3.1-8B-InstructShared82.3083.22 ±\pm 0.5288.61 ±\pm 0.00+5.39 ([+4.74, +6.03])88.61 / 82.86 (+5.75)
Llama-3.1-8B-InstructDual82.4183.20 ±\pm 0.4888.61 ±\pm 0.00+5.41 ([+4.81, +6.00])88.61 / 82.86 (+5.75)

2. Cross-family strong-reviser OOF competition math (50 adapters; Table 34). On competition math (n = 658; Never 78.42%, Gate 77.96%, Always 83.89%), 5-fold OOF pre-revision KEEPER beats Always Revise at default τ=0\tau = 0 and 8.21% revision rate on both Qwen2.5-7B (85.56%, +7.60 pp vs. Gate, 0 harms) and Llama-3.1-8B (85.11%, +7.14 pp vs. Gate, 1 harm); under K=32K = 32 zero-gradient calibration, cross-domain 3B / 7B / 8B pre-revision KEEPER reaches 81.76% / 82.06% / 82.04% (all above matched Correctness).

3. Non-math domain generalization to competitive code on APPS (82 adapters across 3B / 7B / 8B, n = 500, 54 workflows; Table 35, Figure 6b-c). Evaluating 31 Qwen2.5-7B and 23 Llama-3.1-8B code workflows graded by official Python unit tests, naive Checker Gate drops strict pass rate from 11.70% to 9.49% (-2.21 pp, p<10−4p < 10^{-4}; on Llama-8B Shared, 9.55% to 5.85%). Zero-shot KEEPER beats Correctness across Qwen2.5-3B (+1.41 pp strict), Qwen2.5-7B (+0.65 pp strict, p=0.0298p = 0.0298; +2.13 pp test-avg), and Llama-3.1-8B (+1.16 pp strict, p=0.0002p = 0.0002; +2.74 pp test-avg), and 5-fold OOF APPS training on Llama-3.1-8B beats Correctness by +0.95 pp strict pre-revision (p=0.0002p = 0.0002) and +0.73 pp post-revision (p=0.0080p = 0.0080, reaching 7.80% nested-calibrated on the 23 Llama-8B workflows vs. 7.62% Never Revise).

AuthorsOpenReview ↗

Response to Reviewer ZvX8 (Part 3/4)

W2 and W3: Reorganizing Sections 2-5 and Rewriting Compressed Prose (Including Lines 209-211)

W2: "The content is not well organized. For example, Section 3 lacks focus and is difficult to follow. Section 4 should show the main results and much of its content should be moved to Section 5." W3: "Too many statements are likely written by AI and hard to interpret... For example, line 209-211, too much information is compressed in the sentences and is not explained clearly."

We reorganized Sections 2-5 and rewrote dense passages as shorter, single-claim sentences.

  1. Section architecture (§2-§5).
    • Section 2 (Verification as Control, pp. 2-4) presents the binary consequence identity (Eq. 4, Figure 1), a side-by-side runtime contract (new Table 1, p. 4), a two-panel workflow diagram (revised Figure 2, p. 4), and single-backbone multi-role LoRA adaptation (§2.4, p. 4).
    • Section 3 (Evaluation Protocol, pp. 5-6) is now a set of short, self-contained protocol blocks: evidence tiers, the held-out role-action controller, controlled comparisons, baselines and metrics, and leakage controls and reproducibility.
    • Section 4 (Results, pp. 6-9) presents the core findings: the 18-run correlation (Table 2), the 114-case intervention (Table 3), the preserve-rule diagnostic and preserve-first student transfer across 3B/7B/8B, strong-reviser arbitration, and APPS code (Table 4, Figure 3), the six fixed-solver pipelines (Figure 4), and baselines and cost (Table 5).
    • Section 5 (Analysis and Deployment Implications, pp. 8-9) collects the regime analysis (When Preserve-First Control Helps) and operational guidance (Monitoring, Errors, and Scope).
  2. Unpacking compressed prose (§2.3, §3, and throughout).
    • Long-run analysis description (original lines 209-211; revised §3, p. 5, lines 219-222). The original passage combined the dataset, run filter, snapshot count, and unfiltered check into internal shorthand. In plain terms, we logged 75 multi-role math RL runs on a local DeepMath-103K subset, totaling 6,570 training-step snapshots where false rejection correlates negatively with system gain (r=−0.151r = -0.151). Keeping the runs started directly from the base model without SFT warm-start that logged at least 194 steps gives the 18 long runs (4,186 step snapshots, r=−0.259r = -0.259) analyzed in §4.1. Section 3 (p. 5, lines 219-222) now states these steps in three plain single-claim sentences without internal terms.
    • Decision rule and Sections 2-5 (§2.3, pp. 3-4, lines 143-161, Eq. 5). Section 2.3 now separates online pre-revision routing (which decides before generating any revision) from the 458-item offline diagnostic, and defines both the semantic judge score qselect∈[0,1]q_{\mathrm{select}} \in [0, 1] and integer candidate agreement (support≥1\mathrm{support} \ge 1) before stating the fallback threshold (qselect<0.55∧support<2q_{\mathrm{select}} < 0.55 \wedge \mathrm{support} < 2). Similar unpacking into single-claim sentences is applied across Sections 2-5 and Limitations.
AuthorsOpenReview ↗

Response to Reviewer ZvX8 (Part 4/4)

Q1: Training All Three Roles (Solver, Verifier, Reviser) in a Single Model

Q1: "Is it possible to train the three roles (solver, verifier and revisor) in a single model?"

Yes. Our multi-role RL experiments across both math and APPS code already train and serve all three roles on a single frozen base model, and we state this explicitly in Section 2.4 (Role Adaptation with LoRA, p. 4) and Section 4.5 / Table 5 (p. 9):

  • One shared LoRA (Shared adapter). One base model and one shared LoRA parameter set serve solver, verifier, and reviser through role prompts. This minimizes memory, but gradient interference across roles causes severe verifier false rejection (FR = 39.49% on our 1,472-item math panel, Table 5; on APPS Llama-8B Shared, naive gating drops strict pass rate from 9.55% to 5.85%, Table 35). KEEPER repairs the math Shared workflow from 26.92% to 46.78% (+19.86 pp, cutting FR to 1.22%) and restores APPS Llama-8B Shared to 9.55% (zero broken programs).
  • One base model with modular role LoRAs (Dual / Role-Isolated / KEEPER). A single base model (Qwen2.5-7B or Llama-3.1-8B, loaded once in GPU memory) hosts lightweight per-role LoRA adapters switched per turn. This keeps the single-model memory footprint while allowing the verifier (KEEPER) to be trained for consequence-aware preservation without destabilizing the solver or reviser (§2.4, §4.4-§4.5, Tables 33 and 35).

Q2: Revising Figure 2 to Reflect That Revision Candidates Are Not Always Available

Q2: "In Figure 2, revision candidates are not always available, right? If so, the figure should be revised to reflect this."

Yes. In our primary deployment (online pre-revision routing), KEEPER decides before the reviser is called; when it outputs KEEP, no revision candidate is generated. Under lazy short-circuit serving, neither the checker nor the reviser is called, cutting measured post-solver H200 latency by 86.5% relative to the Gate (Table 1, Appendix K, Figure 5c). We replaced Figure 2 (p. 4) with a two-panel diagram:

  • Panel (a), online pre-revision routing (primary deployment). KEEPER acts before the reviser; the KEEP branch returns asa_s immediately with no candidate generated, and only REVISE invokes the reviser.
  • Panel (b), offline post-generation selection (n = 458 diagnostic only). This panel shows the offline diagnostic, in which an existing candidate pool is filtered by (qselect,support)(q_{\mathrm{select}}, \mathrm{support}) with fallback to asa_s.

Q3: Definition of rr in Original Table 1 (Now Table 2)

Q3: "In Table 1 [now Table 2], what is r? Please define it explicitly."

We now define rr in both Section 4.1 (p. 6) and the Table 2 caption. It is the sample Pearson correlation coefficient computed across all 4,186 logged RL training-step snapshots tt (from 18 multi-role runs) between the snapshot's false-rejection share, πFR,t=Pr⁡(Vt=0,St=1)\pi_{\mathrm{FR},t} = \Pr(V_t = 0, S_t = 1) over all rollout items at step tt, and the snapshot's system gain SGt=Yˉt−Sˉt\mathrm{SG}_t = \bar{Y}_t - \bar{S}_t (final workflow accuracy minus first-pass solver accuracy at step tt). Across all 4,186 snapshots, r=−0.259r = -0.259 (run-clustered 95% CI [-0.366, -0.184]; run-demeaned r=−0.303r = -0.303; first-differenced r=−0.423r = -0.423), and the correlation is negative within each of the 18 individual runs.

Reviewer_WJAYOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines why asking a model to revise an answer can reduce final accuracy and introduces KEEPER, a preserve-first approach. It differentiates between routing before revision generation and selecting among generated candidates. The research also evaluates 7B students trained with various supervision targets. Evidence includes an 18-run training audit, a 114-case controller intervention, a 458-item selection diagnostic, six pipeline comparisons, matched-revision-count random replay, and tests on new problems. The findings indicate that preventing destructive revisions is beneficial, though the capacity to retain useful repairs varies among implementations.

Critical strengths
  1. The decomposition of original and revised answer correctness offers a straightforward approach to linking errors by verifiers to the impacts on the final answer. The gains over random replay, with candidates, feedback, and family-level revision counts held constant, provide meaningful evidence for the benefits of selection.
  2. The paper reports stronger holdouts and clustered analyses, complemented by a strong never-revise baseline, some Dual regressions, and instances where students trained on correctness labels outperform those trained on consequence labels. A particularly useful comparison in G.3 maintains the data and training setup while altering only the labels. This helps assess how much depends on preserve-first control and how much depends on the supervision target.
Critical weaknesses
  1. Across different variants, the improvements over retaining the original answer are mixed. In Table 22, the frozen consequence student achieves only a 0.37 percentage point gain over never revise, and retains just 28/124 beneficial revisions. For the correctness-trained student, performance is better in G.3, indicating this limitation applies specifically to the particular consequence-trained variant rather than the preserve-first control as a whole.
  2. The five role views and label construction for the 32,000-row dataset require clearer definitions to ensure reproducibility of the 114-case experiment. While the appendices detail the students' decision rules, their runtime inputs and invocation points still need a unified mapping to the two interfaces described in Section 2.3. Without this, it is challenging to ascertain which variants actually avoid revision-generation costs.
Questions for authors
  1. What are the five counterfactual role views, how is each constructed, what information does it reveal, and what extra inference cost is involved? Do the 32,000 rows represent unique workflow states or multiple records from shared states? How are the target view-action labels created, including whether all candidate outcomes are observed, how ties and unavailable actions are managed, and how target selection slots are chosen? While the reported separation checks are useful, they do not fully enable reconstruction of the supervision process.
  2. Which student is intended for deployment as the KEEPER policy, or is KEEPER a family of preserve-first controllers? Appendices detail the decision rules of the semantic-verdict, correctness-trained, and consequence-trained students, and Appendix F describes a retrospective evaluation. Please map each variant’s runtime inputs and invocation points to pre-revision routing or post-generation selection in Section 2.3. Does its veto prevent revision generation or replace an already generated answer?
  3. G.3 reports improvements across all 35 new-problem workflows for the correctness-trained student. Could the authors provide per-workflow paired results and uncertainties, compared to the consequence student and never revise?
  4. A minor reporting question: could the authors specify the correctness references, denominators, and aggregations for FR in each experiment? Section 2.2 and Table 3 use a full-sample event rate (9/458 and 5/458 in Table 3), while Appendix Table 8 explicitly conditions on teacher-correct items. Please clarify the statistic in Table 1 and whether Appendices E–G pool item-level events or average rates across seeds/checkpoints. Different definitions might be appropriate, but they should be clearly distinguished.
7 comments
AuthorsOpenReview ↗

Response to Reviewer WJAY (Part 1/6)

We thank Reviewer WJAY for an engaged and constructive review, and for recognizing the consequence decomposition, the fixed-candidate randomization replay, the clustered holdouts, and the matched-data label ablation. Every weakness and question led to new multi-fold GPU experiments or explicit manuscript specifications. Below is an overview of revisions followed by point-by-point responses.

Summary of Revisions

  1. W1 (+0.37 pp in Table 23; Consequence vs. Correctness parity, strong-reviser OOF routing, and APPS code) (PDF §4.3 p. 7, App. K-N, Tables 28, 30-32, 34-35, Figs. 5-6): We trained 30 matched 5-fold adapters (3B / 7B / 8B; identical splits, initial weights, and step counts) on the 17,655-row panel, 20 GSM8K strong-reviser arbitrator adapters, 50 5-fold OOF competition math strong-reviser adapters, and 82 zero-shot and 5-fold OOF competitive code adapters on APPS (n = 500, 54 workflows). At default τ=0\tau = 0, Consequence wins 45/45 fold-reviser cells on SVAMP (n = 887; +1.24 to +5.41 pp) and leads at a matched 5% LLR budget on all three backbones (88.84%-89.29%). On competition math (n = 658) with strong Qwen2.5-Math-7B revisions, 5-fold OOF pre-revision KEEPER reaches 85.56% (Qwen-7B; +47 repairs, 0/33 harms) and 85.11% (Llama-8B) at an 8.21% revision rate, exceeding Always Revise (83.89%), and on APPS code it beats Correctness across both zero-shot (+0.65 to +1.16 pp strict) and 5-fold OOF (+0.51 to +0.95 pp strict) suites while preventing 100% of broken-program overwrites.
  2. W2 and Q1 (five counterfactual role views, 32,000 rows, and view-action targets) (PDF §3 p. 5, §4.2 p. 6, App. M pp. 24-25): Documented the five deterministic counterfactual role views (5 + 1 forward passes), the 32,000-row curriculum provenance (8 training states tested on 114 disjoint held-out states across 49 episodes, zero state/prompt/feature overlap), and the lexicographic view-action label construction.
  3. W2 and Q2 (deployment policy vs. family; Mode A vs. Mode B) (PDF §2.3 pp. 3-4, Table 1, Fig. 2, App. K, Fig. 5c): Added Table 1 and two-panel Figure 2 mapping every controller variant to Mode A (online pre-revision routing) or Mode B (offline post-generation selection), plus measured lazy short-circuit H200 serving (54.1 ms/item post-solver, -86.5% vs. Gate).
  4. Q3 (paired results and uncertainties across all 35 unseen-problem workflows) (PDF App. M, Table 33 p. 24): Added the full 35-workflow (6,440-evaluation) paired breakdown on the 184-item held-out split: pooled Correctness (11.29%) and Consequence (11.27%) match within -0.02 pp [-0.45, +0.43] (0/35 workflows at pHolm<0.05p_{\mathrm{Holm}} < 0.05), while Fixed KEEPER reaches 11.54% (+4.77 pp vs. Gate, +0.37 pp [+0.16, +0.62] vs. Never Revise, 403/407 harms blocked).
  5. Q4 (FR references, denominators, and aggregations) (PDF §4.1 p. 6, App. M p. 25): Added a complete FR ledger confirming that workflow tables uniformly report the joint panel share FR=Pr⁡(V=0,S=1)\mathrm{FR} = \Pr(V=0, S=1) over all NN items against task ground truth, with the sole teacher-conditional exception labeled in Appendix D (Table 9).
AuthorsOpenReview ↗

Response to Reviewer WJAY (Part 2/6)

W1: Why Fixed Consequence Gained +0.37 pp in Table 23, Plus 5-Fold Parity, Strong-Reviser OOF, and Code Results

"In Table 22 [now Table 23], the frozen consequence student achieves only a 0.37 percentage point gain over never revise, and retains just 28/124 beneficial revisions. For the correctness-trained student, performance is better in G.3..."

Two factors caused that conservative behavior on the 184-item held-out split: (i) in our earlier one-stage training, class rebalancing gave the consequence student fewer optimizer updates than the correctness student, and (ii) the 184-item split has very low solver accuracy (8.15%-14.83%), making a threshold τ=0\tau = 0 learned under higher-solver regimes conservative. To isolate supervision quality from step count, base-rate shift, and domain, we ran four controlled multi-fold suites (Appendices K-N, Tables 28, 30-32, 34-35, Figures 5-6):

  1. Strict 5-fold parity on SVAMP under identical splits, initial weights, RNG seeds, and step counts (30 adapters across 3B / 7B / 8B, n = 887; Tables 28 and 30). With identical 17,655-row problem-disjoint splits, identical initial LoRA tensors, and 1,104 steps/fold, Consequence beats Correctness in all 45/45 fold-reviser pairs: Qwen2.5-7B 5-fold vote reaches 88.84% vs. 87.26% / 87.15% / 87.03% (+1.58 to +1.80 pp, all Bonferroni 95% CIs > 0; 2/2 beneficial revisions passed vs. 20-22 harms admitted by Correctness), Qwen2.5-3B gains +1.99 to +2.37 pp, and Llama-3.1-8B gains +5.39 to +5.41 pp (+5.75 pp ensemble).
  2. Continuous LLR iso-revision-rate curves and K = 32 calibration (Tables 31 and 34, Figures 5b and 6a). Ranked by teacher-forced LLR ℓ(x)\ell(x), 5-fold Consequence beats Correctness at an identical 5% gated budget on all three backbones on SVAMP (88.84%-89.29%) and at 5%-10% budgets on the 184-item split (+0.54 to +2.17 pp). With zero weight updates, K=32K = 32 scalar threshold calibration lifts fixed 5-fold KEEPER by +1.66 to +2.82 pp on the 184-item split and from 78.42% to 82.06% (7B) and 82.04% (8B) on competition math (vs. 81.45% and 81.32% for Correctness).
  3. High-repair OOF routing with strong Qwen2.5-Math-7B (70 adapters across GSM8K and competition math; Tables 32 and 34). When trained out of fold directly on strong-reviser trajectories of competition math (n = 658; Never 78.42%, Gate 77.96%, Always 83.89% with +69 repairs / -33 harms), pre-revision KEEPER is not overly conservative: at default τ=0\tau = 0 and only 8.21% revision rate (54 calls), Qwen2.5-7B reaches 85.56% (+47/69 repairs retained, 0/33 harms, +1.67 pp above Always Revise) and Llama-3.1-8B reaches 85.11% (+45 repairs, 1/33 harm), whereas Correctness needs 2.3×2.3\times more revisions (18.5%-18.8%) and admits 5-9 harms. On post-revision (x,as,ar)(x, a_s, a_r), Consequence raises decisive AUROC from 0.3935 to 0.6289 on competition math (81.61% vs. 79.94%) and reaches 92.27% on OOF GSM8K (n = 1,877; unified multi-regime arbitrator: 92.01% GSM8K, 81.00% math OOF / 84.04% nested-calibrated, 88.73% SVAMP).
  4. Non-math generalization to APPS competitive programming (82 adapters, n = 500, 54 workflows; Table 35, Figure 6b-c). Naive Checker Gate breaks 17.93 passing programs/workflow (11.70% Never to 9.49% Gate, -2.21 pp, p<10−4p < 10^{-4}). Both zero-shot math-trained KEEPER (+0.65 pp strict on 7B, p=0.0298p = 0.0298; +1.16 pp on 8B, p=0.0002p = 0.0002; 7.77% standalone on Llama-8B vs. 7.62% Never) and 5-fold OOF APPS KEEPER (+0.84 pp 7B pre, p=0.0030p = 0.0030; +0.95 pp 8B pre, p=0.0002p = 0.0002; +0.73 pp 8B post, p=0.0080p = 0.0080) beat matched Correctness while blocking 100% of broken-program overwrites.
AuthorsOpenReview ↗

Response to Reviewer WJAY (Part 3/6)

W2 and Q1: Construction of the Five Counterfactual Role Views, Inference Cost, 32,000-Row Provenance, and Target Labels

Q1: "What are the five counterfactual role views, how is each constructed, what information does it reveal, and what extra inference cost is involved? Do the 32,000 rows represent unique workflow states or multiple records from shared states? How are the target view-action labels created, including whether all candidate outcomes are observed, how ties and unavailable actions are managed, and how target selection slots are chosen?"

The complete specification appears in Section 3 and revised Appendix M (pp. 24-25).

  1. Five deterministic counterfactual role views and runtime cost. Given a workflow state with up to 10 candidate slots (solver, checker, and redo/reviser entries, each carrying candidate_summary text and a role tag), we construct:
    • VIEW_ORIGINAL: the unmodified state;
    • VIEW_SOLVER_ABLATED, VIEW_REDO_ABLATED, VIEW_CHECKER_ABLATED: the same state with the named role's candidate_summary redacted (measuring reliance on that role's content);
    • VIEW_ROLE_SWAPPED: identical candidate texts with role tags deterministically permuted across slots (measuring role-tag bias vs. content).
    • Inference cost and information flow. A base two-stage gate/selector runs once on each view (5 forward passes) and emits (pred_gate, pred_selector, pred_action, gate_margin, selector_margin). The compact rank-64 LoRA listwise selector receives only these five summary tuples (one short pass, no ground-truth or outcome keys), for 6 passes in total in this auxiliary diagnostic.
  2. 32,000-row provenance and leakage check (Appendix M). The 32,000 training rows are oversampled listwise curriculum records generated from 8 unique training states (182 unique prompt hashes, 9 unique five-view feature vectors). They are tested on 114 disjoint held-out states from 49 episodes, with zero overlap by row ID, state ID, prompt hash, or feature vector. Because the source states are few, we present the 114-case study as an auxiliary counterfactual diagnostic (§4.2); all primary student claims rely on our 17,655-row problem-disjoint 5-fold splits (zero duplicate prefixes).
  3. Target view-action labels, ties, unavailable actions, and slot indexing. Offline label construction observes all logged candidate outcomes and redo transitions in the recorded episode. Within a view, if a visible candidate in slots 0-9 is correct, the target action is ACTION_SELECT_SLOT_k for the lowest-index correct visible slot k∈{0,…,9}k \in \{0,\dots,9\}; if none is correct and a logged redo repairs the state, the target is ACTION_REDO; if the first-pass answer should terminate unchanged, ACTION_STOP; and if redaction removes every valid selectable target and redo is unavailable, ACTION_SELECT_UNKNOWN. Across the five views, the target view maximizes the lexicographic tuple (terminal_success, exact_action, useful_action, selector_margin, gate_margin), with ties broken by canonical view order starting from VIEW_ORIGINAL.
Reviewer_wT7zOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a practical problem in multi-role LLM systems: identifying an incorrect answer does not necessarily mean that revising it will improve the final result. The authors propose KEEPER, which considers the consequences of revision and preserves the original answer when there is insufficient reason to replace it.

Critical strengths

I like the distinction between finding an error and deciding whether another attempt will help. The second decision depends on what the reviser can actually do, which is easy to overlook when evaluating the verifier on its own.

The idea is straightforward and useful for existing agent workflows. It gives the system a way to avoid unnecessary changes while still allowing repairs.

The comparisons with fixed candidates and matched revision counts are helpful. They provide evidence that the controller is choosing which answers to revise, rather than gaining only by revising fewer answers.

Critical weaknesses

The controller seems quite cautious. On the evaluation with new problems, it is only 0.37 percentage points better than keeping the first answer, and it also blocks many useful corrections. I would like to understand whether it still helps when the reviser is already reasonably good, rather than mainly protecting against a reviser that often makes things worse.

The benefit of learning revision consequences is not yet clear. In Table 21, the student trained on answer correctness performs better than the consequence-trained student on the Dual and Shared panels. This raises the question of when collecting labels for revision outcomes is worth the additional effort.

It is unclear how easily the controller can be reused in another agent setup. Its decisions depend on the reviser’s ability, so changing the reviser may require further training or calibration. The extra cost of the controller, including voting in some experiments, also needs to be included when assessing the practical benefit.

Questions for authors

Can you test revisers of different strengths with the same solver and questions? Please compare against never revising, always revising, and the original gate, and report both prevented harmful revisions and lost useful corrections.

Can you extend the comparison in Table 21 to new problems and compare the correctness-trained and consequence-trained students at similar revision rates? This would help separate better selection from simply being more cautious.

What happens if the reviser is changed while the controller is kept frozen? How much calibration or further training is needed to recover its performance?

Can you report total inference tokens, model calls, and latency for each version, including the controller and any voting?

6 comments
AuthorsOpenReview ↗

Response to Reviewer wT7z (Part 1/5)

We thank Reviewer wT7z for a clear and practical review, and for drawing the core distinction between detecting an error and knowing whether revision will improve the answer. Your four questions (Q1-Q4) address real-world deployment directly. We ran eight multi-fold GPU training, calibration, code-execution, and serving suites (182 LoRA adapters plus N = 256 H200 latency benchmarks) to answer them thoroughly.

Summary of New Results

  1. W1 and Q1 (reviser-strength sweep, strong-reviser OOF routing, and APPS code) (PDF §4.3 p. 7, App. K, M-N, Tables 29, 31-32, 34-35, Figs. 5a, 6): Holding the solver and questions fixed, we swept revisers from destructive (Shared/Dual on math; 54 code workflows on APPS, n = 500) to moderate (184-item held-out split) to strong Qwen2.5-Math-7B on competition math (n = 658; Always = 83.89%) and GSM8K (n = 1,877 OOF; Always = 91.26%). Training 50 new 5-fold OOF adapters on competition math reaches 85.56% on Qwen2.5-7B (+47 repairs, 0/33 harms) and 85.11% on Llama-3.1-8B (+45 / -1) at default τ=0\tau = 0 and only 8.21% revision rate (beating Always Revise at 12×12\times fewer reviser calls), while 20 GSM8K post-revision adapters reach 92.27% (89.18% on SVAMP) and one unified balanced arbitrator achieves 92.01% GSM8K, 81.00% competition math (84.04% nested-calibrated), and 88.73% SVAMP. On APPS competitive programming (82 adapters), KEEPER recovers full solver pass rates (+2.21 pp vs. Gate, +0.65 to +1.16 pp strict over Correctness) and exceeds Never Revise on Llama-8B (7.65% gated / 7.77% standalone vs. 7.62%).
  2. W2 and Q2 (Consequence vs. Correctness on new problems and at matched revision rates) (PDF §4.3, App. K-M, Tables 28, 30-31, Fig. 5b): Across 30 matched 5-fold adapters (3B / 7B / 8B; identical splits, initial weights, and step counts), Consequence wins 45/45 fold-reviser pairs on SVAMP at τ=0\tau = 0, and at an identical 5% LLR budget it beats Correctness on all three backbones (88.84%-89.29%). This separates calibration at the natural zero-utility boundary from superior item ranking at selective budgets.
  3. W3 and Q3 (fixed-controller reuse under reviser shift; K = 16, 32, 64 calibration cost) (PDF App. K, M-N, Tables 29, 31, 34, Fig. 6a): Moderate shifts need zero weight updates. Under extreme base-rate or reviser shifts, calibrating a single scalar threshold with zero gradient updates on K=32K = 32 labeled items (1,000 MC draws) recovers topological ordering (n = 256; 5.47% to 8.22%, matching Gate 8.21%), lifts the 184-item split by +1.66 to +2.82 pp, and raises fixed cross-domain pre-revision KEEPER on competition math from 78.42% to 82.06% (7B) and 82.04% (8B) (K=64K = 64: 82.81% / 82.79%).
  4. W3 and Q4 (total tokens, model calls, voting cost, and H200 latency) (PDF §2.3 Table 1 p. 4, App. K, M, Fig. 5c): Lazy short-circuit KEEPER(x, a_s) (887/887 boolean parity with eager execution) adds 54.1 ms/item post-solver (+8.0 tok/item), cutting measured H200 post-solver latency by 86.5% vs. Checker Gate (401.7 ms/item) and 89.3% vs. Always Revise (506.0 ms/item).
AuthorsOpenReview ↗

Response to Reviewer wT7z (Part 2/5)

W1 and Q1: Fixed-Solver Reviser-Strength Sweep, 5-Fold Strong-Reviser OOF Routing, and Competitive Code

Q1: "Can you test revisers of different strengths with the same solver and questions? Please compare against never revising, always revising, and the original gate, and report both prevented harmful revisions and lost useful corrections."

Holding the first-pass solver and questions fixed, we evaluated revisers from net-destructive (math Shared/Dual; APPS code) to moderately net-positive (184-item held-out split) to strongly net-positive (Qwen2.5-Math-7B on competition math [n = 658] and GSM8K [n = 1,877 OOF]; Appendices K, M-N, Tables 29, 31-32, 34-35, Figures 5a and 6):

Benchmark (Never)ReviserAlways (+B/-H)Gate (+B/-H)Pre-Revision KEEPERPost-Revision KEEPER
Held-out (8.15%)Shared8.15% (+9/-9)7.61% (+5/-6)10.33% (+4/-0; blk 9, miss 5)n/a
Held-out (8.15%)Untrained13.59% (+17/-7)9.78% (+8/-5)11.96% (+7/-0; blk 7, miss 10)n/a
Held-out (8.15%)Dual14.67% (+17/-5)11.41% (+8/-2)12.50% (+8/-0; blk 5, miss 9)n/a
SVAMP (88.61%)Math-7B81.29% (+15/-80)81.06% (+10/-77)88.61% (+0/-0; blk 80, miss 15)89.18% (+8/-3; blk 77, miss 7)
Comp. Math (78.42%)Shared/Dual66.26 / 67.33%68.84 / 69.45%78.42% (blk 96/85, miss 16/12)n/a
Comp. Math (78.42%)Math-7B (K=32)83.89% (+69/-33)77.96% (+28/-31)82.06% (7B K=32; 82.81% K=64)79.64% (81.60% at K=32)
Comp. Math (78.42%)Math-7B (OOF)83.89% (+69/-33)77.96% (+28/-31)85.56% (+47/-0; blk 33, miss 22)81.61% (+22/-1; blk 32, miss 47)
GSM8K OOF (91.05%)Math-7B (OOF)91.26% (+54/-50)90.30% (+28/-42)91.05% (91.74% calibrated)92.27% (+35/-12; blk 38, miss 19)
APPS Code (11.70%)54 workflows9.36% (+6.9/-18.6)9.49% (+6.9/-17.9)11.70% (0 broken; 7.77% Llama-8B)11.32% (7.80% Llama-8B cal.)

(Notation: +B/-H = useful corrections kept / harmful revisions admitted; blk = harms prevented, miss = oracle repairs missed.)

Three deployment takeaways:

  1. In-domain 5-fold OOF routing beats Always Revise at 8.21% revision cost (Table 34). When trained out of fold on competition math (50 adapters, matched initial LoRA weights per fold), pre-revision KEEPER at τ=0\tau = 0 reaches 85.56% on Qwen2.5-7B (+7.60 pp [+5.02, +10.33] vs. Gate, +1.67 pp above Always Revise, keeping 47/69 repairs with 0/33 harms) and 85.11% on Llama-3.1-8B (+45/-1, +1.22 pp above Always Revise) at only 54 reviser calls (8.21%).
  2. One unified controller handles opposing reviser regimes without prior knowledge (Tables 32, 34). The same Qwen2.5-Math-7B reviser adds +5.47 pp on competition math yet loses -7.32 pp on SVAMP (81.29% vs. 88.61%). A single unified balanced GSM8K + competition math 5-fold arbitrator at fixed τ=0\tau = 0 reaches 92.01% on OOF GSM8K, 81.00% on OOF competition math (84.04% nested-calibrated), and 88.73% zero-shot on SVAMP (AUROC 0.8800).
  3. Executable code verification on APPS (Table 35). Across 54 Python code workflows (n = 500), Checker Gate breaks 17.93 passing programs/workflow (-2.21 pp, p<10−4p < 10^{-4}); KEEPER prevents 100% of broken overwrites and beats Correctness by +0.65 to +1.16 pp strict (p≤0.0298p \le 0.0298).
AuthorsOpenReview ↗

Response to Reviewer wT7z (Part 3/5)

W2 and Q2: Matched-Label Consequence vs. Correctness Across 30 5-Fold Adapters and at Identical Revision Rates

Q2: "Can you extend the comparison in Table 21 [now Table 22] to new problems and compare the correctness-trained and consequence-trained students at similar revision rates? This would help separate better selection from simply being more cautious."

We trained 30 matched 5-fold LoRA adapters (Qwen2.5-3B, Qwen2.5-7B, Llama-3.1-8B ×\times 5 folds ×\times 2 labelings; identical 17,655-row problem-disjoint splits, initial weights, RNG seeds, and 1,104 steps per fold) and evaluated the continuous score ℓ(x)=log⁡pθ(REVISE∣x)−log⁡pθ(KEEP∣x)\ell(x) = \log p_\theta(\texttt{REVISE}\mid x) - \log p_\theta(\texttt{KEEP}\mid x) at identical revision budgets (3%-30%) on three unseen benchmarks: SVAMP (n = 887), the 184-item held-out split, and competition math (n = 658) (Appendices K-M, Tables 28, 30-31, Figure 5b):

Benchmark (N / A / G)Backbone (mode)ReviserDef. τ=0\tau{=}0 (Cons / Corr)3% rate5% rate10% rate15% rate30% rate
SVAMP (88.6/80.6/82.3)Qwen-7B (gated)Shared88.73 / 87.15 (88.84 vote)89.18/88.9589.29/88.9588.50/88.1687.37/86.8184.33/83.88
SVAMP (88.6/81.9/82.4)Qwen-7B (gated)Dual88.73 / 87.03 (88.84 vote)89.06/88.8489.18/88.9588.28/88.0587.26/86.8184.55/83.77
SVAMP (88.6/80.6/82.3)Qwen-3B (gated)Shared88.61 / 86.1389.18/88.5089.06/88.6187.94/88.0586.70/86.7083.65/82.98
SVAMP (88.6/81.9/82.4)Llama-8B (gated)Dual88.61 / 82.8688.95/89.2988.84/88.6187.03/88.0585.79/86.7083.43/84.33
Held-out (8.2/14.7/11.4)Qwen-7B (standalone)Dual8.15 / 16.3010.33/9.2411.96/10.3312.50/11.4113.04/13.5914.13/15.76
Held-out (8.2/14.7/11.4)Qwen-3B (standalone)Dual8.15 / 16.309.78/8.7011.41/9.2411.96/11.4113.59/13.5915.22/16.30
Comp. Math (78.4/83.9/78.0)Llama-8B (standalone)Math-7B78.42 / 77.5178.12/78.1278.42/77.9679.18/77.6680.09/78.8881.16/79.18

(Table 31 in the revised PDF reports all ten backbone ×\times benchmark rows, including Llama-8B on the 184-item split and Qwen-3B/7B on competition math.)

Separating selection from caution:

  • Calibration at the default threshold. On SVAMP (and on APPS competitive code in Table 35), where first-pass solutions face net-destructive revisers, Consequence's zero-utility boundary preserves the solver output unless revision has positive expected value, matching or beating Never Revise on every backbone (88.61%-88.73% on SVAMP; 11.70% vs. 10.29%-11.06% strict on APPS), whereas Correctness over-rejects correct solutions and drops 1.1 to 5.4 pp below Never Revise on SVAMP and breaks 9.20-13.50 passing programs/workflow on APPS.
  • Better ranking at matched selective budgets. At an identical 5% budget on SVAMP, Consequence beats Correctness on all three backbones (88.84%-89.29%, above both Never Revise and rate-matched random routing; Figure 5b), leads at 5%-10% budgets across all three backbones on the 184-item split (+0.54 to +2.17 pp), and leads by +1.21 to +1.98 pp at 10%-30% budgets on competition math (and by +1.03 to +1.15 pp under K=32K = 32 calibrated post-revision OOF arbitration in Table 34).

Comments on the paper

2 comments
AuthorsOpenReview ↗

Author Response to All Reviewers

We thank Reviewers WJAY, wT7z, and ZvX8 for careful, constructive reviews. We addressed their four shared themes with nine multi-fold GPU training, calibration, code-execution, and serving suites totaling 182 LoRA adapters across Qwen2.5-3B-Instruct, Qwen2.5-7B-Instruct, and Llama-3.1-8B-Instruct, plus an N = 256 H200 latency benchmark (9 main pages + Appendices A-N, Tables 1 and 28-35, Figures 2 and 5-6).

Overview of New Evidence and Manuscript Revisions

  1. Strict 3-backbone 5-fold parity and iso-rate ranking (WJAY W1/Q3, wT7z W2/Q2, ZvX8 W1; PDF §4.3 p. 7, App. K-M, Tables 28, 30-31, 33, Fig. 5b): Across 30 matched-label 5-fold adapters (3B/7B/8B; identical 17,655-row problem-disjoint splits, initial LoRA weights, RNG seeds, and 1,104 steps/fold), Consequence beats Correctness in 45/45 fold-reviser cells on SVAMP (n = 887; +1.24 to +5.41 pp 5-fold mean) and at a matched 5% LLR budget on all three backbones (88.84%-89.29%). Across all 35 unseen-problem workflows (184-item held-out split, 6,440 evaluations), Consequence (11.27%) and Correctness (11.29%) match within -0.02 pp (0/35 at pHolm<0.05p_{\mathrm{Holm}} < 0.05), while Fixed KEEPER reaches 11.54% (+4.77 pp vs. Gate, +0.37 pp [+0.16, +0.62] vs. Never Revise, 403/407 harms blocked).
  2. Strong-reviser (Qwen2.5-Math-7B) OOF routing, unified arbitration, and K = 32 calibration (wT7z W1/Q1/W3/Q3, WJAY W1; PDF §4.3, App. K, M-N, Tables 29, 32, 34, Figs. 5a, 6a): Training 50 new 5-fold OOF adapters on competition math (n = 658; Never 78.42%, Gate 77.96%, Always 83.89%), pre-revision KEEPER at default τ=0\tau = 0 reaches 85.56% on Qwen2.5-7B (+47 repairs, 0/33 harms) and 85.11% on Llama-3.1-8B (+45 / -1) at an 8.21% revision rate (12×12\times fewer calls than Always Revise). On GSM8K (n = 1,877 OOF), 20 post-revision (x,as,ar)(x, a_s, a_r) adapters reach 92.27% (89.18% zero-shot on SVAMP), and one unified balanced multi-regime arbitrator reaches 92.01% GSM8K, 81.00% competition math (84.04% nested-calibratedibrated), and 88.73% SVAMP at fixed τ=0\tau = 0. With zero weight updates, K=32K = 32 scalar calibration lifts cross-domain pre-revision KEEPER on competition math from 78.42% to 82.06% (7B) and 82.04% (8B) and recovers topological ordering (n = 256; 5.47% to 8.22%, matching Gate 8.21%).
  3. Non-math generalization to competitive Python programming on APPS (ZvX8 W1, WJAY W1, wT7z W1; PDF §4.3, App. N, Table 35, Fig. 6b-c): Across 54 multi-role workflows on APPS (n = 500 test problems, 27,000 official unit-test evaluations; 31 Qwen-7B + 23 Llama-8B), naive Checker Gate drops strict pass rate from 11.70% to 9.49% (-2.21 pp [-2.99, -1.48], p<10−4p < 10^{-4}), breaking 17.93 passing programs/workflow (968 total). Across 42 zero-shot math-trained adapters and 40 newly trained 5-fold OOF APPS adapters, KEEPER prevents 100% of broken overwrites, beats Correctness by +0.65 to +1.16 pp strict (p≤0.0298p \le 0.0298) and +2.13 to +2.76 pp test-case average (p<10−4p < 10^{-4}), and 2-stage zero-shot KEEPER reaches 7.65% gated / 7.77% standalone on the 23 Llama-8B workflows (above Never 7.62%, p=0.0004p = 0.0004).
  4. Runtime contract, lazy H200 serving, and prose restructure (WJAY W2/Q1-Q2/Q4, wT7z W3/Q4, ZvX8 W2-W3/Q1-Q3; PDF §2-§5, Table 1, Fig. 2, Fig. 5c): Table 1 and two-panel Figure 2 separate Mode A (online pre-revision routing) from Mode B (offline post-generation selection). Lazy short-circuit KEEPER(x, a_s) skips both checker and reviser on KEEP (887/887 boolean parity) and cuts measured H200 post-solver latency by 86.5% vs. Gate (54.1 vs. 401.7 ms/item). Sections 2-5 use single-claim sentences and explicitly define multi-LoRA role sharing (§2.4), Pearson r=−0.259r = -0.259 (§4.1), the five role views, and FR=Pr⁡(V=0,S=1)\mathrm{FR} = \Pr(V=0, S=1).
Suite (PDF Table)ScopeNever / Gate / AlwaysKEEPER (τ=0\tau=0)vs. Gate / Correctness
SVAMP 5-fold (Tab. 28, 30)n=887 (3B/7B/8B)88.61 / 82.41 / 81.85%88.61-88.84%+6.43 / +1.24 to +5.75 pp
GSM8K 5-fold Post (Tab. 32)n=1,877 OOF (7B)91.05 / 90.30 / 91.26%92.27%+1.97 / +0.58 pp
Comp. Math OOF (Tab. 34)n=658 OOF (7B/8B)78.42 / 77.96 / 83.89%85.56% / 85.11%+7.60 pp (8.2% rev, 0 harm)
APPS Code (Tab. 35)n=500, 54 workflows11.70 / 9.49 / 9.36%11.70% (7.77% Llama-8B)+2.21 / +0.65 to +1.16 pp
Lazy H200 (Tab. 29, Fig. 5c)N=256 SVAMP0.0 / 401.7 / 506.0 ms54.1 ms/item-86.5% post-solver latency

Point-by-point analyses and tables follow in our individual replies.

Dear Reviewer wT7z and ZvX8,

Please take a moment to review and respond to the authors' rebuttal to facilitate further discussion, as the author-reviewer discussion period ends on November 18 at 11:59 PM AoE.

Best,
AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.