- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper identified the problem of monotone pipeline of using solver, verifier and revisor. The revisor may wrongly modify the correct answer, leading to performance degradation. To mitigate this issue, this work proposed KEEPER, a preserve-first policy family that routes only what is worth rewriting and restores the solver answer when evidence for replacing it is weak. The experiment shows that the framework can improve the performance while generating fewer tokens.
Critical strengths
The motivation is clear and problem is clearly formulated and explained.
The experiment shows that the proposed method works better than other pipelines while saving tokens.
Critical weaknesses
-
The experiment only evaluate on Qwen2.5- 7B-Instruct. It is necessary to evaluate on more models to show it generalizability, such as different model family and different sizes.
-
The content is not well organized. For example, Section 3 lacks focus and is difficult to follow. Section 4 should show the main results and much of its content should be moved to Section 5.
-
Too many statements are likely written by AI and hard to interpret. It is OK to polish the paper with AI but it is important to ensure the readability after polishing. For example, line 209-211, too much information is compressed in the sentences and is not explained clearly.
Questions for authors
Is it possible to train the three roles (solver, verifier and revisor) in a single model?
In Figure 2, revision candidates are not always available, right? If so, the figure should be revised to reflect this.
In Table 1, what is r? Please define it explicitly.
4 comments
Response to Reviewer ZvX8 (Part 2/4)
W1: Multi-Backbone (Qwen2.5-3B, Qwen2.5-7B, Llama-3.1-8B) and Non-Math Code (APPS) Generalization
W1: "The experiment only evaluate on Qwen2.5-7B-Instruct. It is necessary to evaluate on more models to show it generalizability, such as different model family and different sizes."
We expanded evaluation across both model scale (Qwen2.5-3B-Instruct) and model family (Llama-3.1-8B-Instruct) in three multi-fold suites spanning math and executable Python programming (Appendices L-N, Tables 30-31, 34-35, Figures 5b and 6):
1. 30-adapter 5-fold strict parity on SVAMP (n = 887, 735 source groups; Table 30). Trained on identical 17,655-row problem-disjoint splits with verified-identical initial LoRA tensors, RNG seeds, and 1,104 steps/fold (zero external teacher calls at test time), Consequence beats Correctness in every fold of all nine backbone reviser pairs (45/45 cells):
| Student backbone (10 adapters) | Reviser | Gate | Correctness | Consequence | 5-Fold ( CI) | Ensemble Cons / Corr () |
|---|---|---|---|---|---|---|
| Qwen2.5-3B-Instruct | Untrained | 82.30 | 86.70 0.48 | 88.68 0.07 | +1.99 ([+1.37, +2.60]) | 88.61 / 86.58 (+2.03) |
| Qwen2.5-3B-Instruct | Shared | 82.30 | 86.31 0.62 | 88.68 0.07 | +2.37 ([+1.59, +3.15]) | 88.61 / 86.13 (+2.48) |
| Qwen2.5-3B-Instruct | Dual | 82.41 | 86.52 0.54 | 88.68 0.07 | +2.17 ([+1.48, +2.86]) | 88.61 / 86.36 (+2.25) |
| Qwen2.5-7B-Instruct | Untrained | 82.30 | 87.49 0.23 | 88.73 0.08 | +1.24 ([+0.94, +1.54]) | 88.73 / 87.26 (+1.47; vote +1.58) |
| Qwen2.5-7B-Instruct | Shared | 82.30 | 87.42 0.27 | 88.73 0.08 | +1.31 ([+0.94, +1.68]) | 88.73 / 87.15 (+1.58; vote +1.69) |
| Qwen2.5-7B-Instruct | Dual | 82.41 | 87.30 0.26 | 88.73 0.08 | +1.42 ([+1.07, +1.78]) | 88.73 / 87.03 (+1.69; vote +1.80) |
| Llama-3.1-8B-Instruct | Untrained | 82.30 | 83.20 0.62 | 88.61 0.00 | +5.41 ([+4.64, +6.17]) | 88.61 / 82.86 (+5.75) |
| Llama-3.1-8B-Instruct | Shared | 82.30 | 83.22 0.52 | 88.61 0.00 | +5.39 ([+4.74, +6.03]) | 88.61 / 82.86 (+5.75) |
| Llama-3.1-8B-Instruct | Dual | 82.41 | 83.20 0.48 | 88.61 0.00 | +5.41 ([+4.81, +6.00]) | 88.61 / 82.86 (+5.75) |
2. Cross-family strong-reviser OOF competition math (50 adapters; Table 34). On competition math (n = 658; Never 78.42%, Gate 77.96%, Always 83.89%), 5-fold OOF pre-revision KEEPER beats Always Revise at default and 8.21% revision rate on both Qwen2.5-7B (85.56%, +7.60 pp vs. Gate, 0 harms) and Llama-3.1-8B (85.11%, +7.14 pp vs. Gate, 1 harm); under zero-gradient calibration, cross-domain 3B / 7B / 8B pre-revision KEEPER reaches 81.76% / 82.06% / 82.04% (all above matched Correctness).
3. Non-math domain generalization to competitive code on APPS (82 adapters across 3B / 7B / 8B, n = 500, 54 workflows; Table 35, Figure 6b-c). Evaluating 31 Qwen2.5-7B and 23 Llama-3.1-8B code workflows graded by official Python unit tests, naive Checker Gate drops strict pass rate from 11.70% to 9.49% (-2.21 pp, ; on Llama-8B Shared, 9.55% to 5.85%). Zero-shot KEEPER beats Correctness across Qwen2.5-3B (+1.41 pp strict), Qwen2.5-7B (+0.65 pp strict, ; +2.13 pp test-avg), and Llama-3.1-8B (+1.16 pp strict, ; +2.74 pp test-avg), and 5-fold OOF APPS training on Llama-3.1-8B beats Correctness by +0.95 pp strict pre-revision () and +0.73 pp post-revision (, reaching 7.80% nested-calibrated on the 23 Llama-8B workflows vs. 7.62% Never Revise).
Response to Reviewer ZvX8 (Part 3/4)
W2 and W3: Reorganizing Sections 2-5 and Rewriting Compressed Prose (Including Lines 209-211)
W2: "The content is not well organized. For example, Section 3 lacks focus and is difficult to follow. Section 4 should show the main results and much of its content should be moved to Section 5." W3: "Too many statements are likely written by AI and hard to interpret... For example, line 209-211, too much information is compressed in the sentences and is not explained clearly."
We reorganized Sections 2-5 and rewrote dense passages as shorter, single-claim sentences.
- Section architecture (§2-§5).
- Section 2 (Verification as Control, pp. 2-4) presents the binary consequence identity (Eq. 4, Figure 1), a side-by-side runtime contract (new Table 1, p. 4), a two-panel workflow diagram (revised Figure 2, p. 4), and single-backbone multi-role LoRA adaptation (§2.4, p. 4).
- Section 3 (Evaluation Protocol, pp. 5-6) is now a set of short, self-contained protocol blocks: evidence tiers, the held-out role-action controller, controlled comparisons, baselines and metrics, and leakage controls and reproducibility.
- Section 4 (Results, pp. 6-9) presents the core findings: the 18-run correlation (Table 2), the 114-case intervention (Table 3), the preserve-rule diagnostic and preserve-first student transfer across 3B/7B/8B, strong-reviser arbitration, and APPS code (Table 4, Figure 3), the six fixed-solver pipelines (Figure 4), and baselines and cost (Table 5).
- Section 5 (Analysis and Deployment Implications, pp. 8-9) collects the regime analysis (When Preserve-First Control Helps) and operational guidance (Monitoring, Errors, and Scope).
- Unpacking compressed prose (§2.3, §3, and throughout).
- Long-run analysis description (original lines 209-211; revised §3, p. 5, lines 219-222). The original passage combined the dataset, run filter, snapshot count, and unfiltered check into internal shorthand. In plain terms, we logged 75 multi-role math RL runs on a local DeepMath-103K subset, totaling 6,570 training-step snapshots where false rejection correlates negatively with system gain (). Keeping the runs started directly from the base model without SFT warm-start that logged at least 194 steps gives the 18 long runs (4,186 step snapshots, ) analyzed in §4.1. Section 3 (p. 5, lines 219-222) now states these steps in three plain single-claim sentences without internal terms.
- Decision rule and Sections 2-5 (§2.3, pp. 3-4, lines 143-161, Eq. 5). Section 2.3 now separates online pre-revision routing (which decides before generating any revision) from the 458-item offline diagnostic, and defines both the semantic judge score and integer candidate agreement () before stating the fallback threshold (). Similar unpacking into single-claim sentences is applied across Sections 2-5 and Limitations.
Response to Reviewer ZvX8 (Part 4/4)
Q1: Training All Three Roles (Solver, Verifier, Reviser) in a Single Model
Q1: "Is it possible to train the three roles (solver, verifier and revisor) in a single model?"
Yes. Our multi-role RL experiments across both math and APPS code already train and serve all three roles on a single frozen base model, and we state this explicitly in Section 2.4 (Role Adaptation with LoRA, p. 4) and Section 4.5 / Table 5 (p. 9):
- One shared LoRA (Shared adapter). One base model and one shared LoRA parameter set serve solver, verifier, and reviser through role prompts. This minimizes memory, but gradient interference across roles causes severe verifier false rejection (FR = 39.49% on our 1,472-item math panel, Table 5; on APPS Llama-8B Shared, naive gating drops strict pass rate from 9.55% to 5.85%, Table 35). KEEPER repairs the math Shared workflow from 26.92% to 46.78% (+19.86 pp, cutting FR to 1.22%) and restores APPS Llama-8B Shared to 9.55% (zero broken programs).
- One base model with modular role LoRAs (Dual / Role-Isolated / KEEPER). A single base model (Qwen2.5-7B or Llama-3.1-8B, loaded once in GPU memory) hosts lightweight per-role LoRA adapters switched per turn. This keeps the single-model memory footprint while allowing the verifier (KEEPER) to be trained for consequence-aware preservation without destabilizing the solver or reviser (§2.4, §4.4-§4.5, Tables 33 and 35).
Q2: Revising Figure 2 to Reflect That Revision Candidates Are Not Always Available
Q2: "In Figure 2, revision candidates are not always available, right? If so, the figure should be revised to reflect this."
Yes. In our primary deployment (online pre-revision routing), KEEPER decides before the reviser is called; when it outputs KEEP, no revision candidate is generated. Under lazy short-circuit serving, neither the checker nor the reviser is called, cutting measured post-solver H200 latency by 86.5% relative to the Gate (Table 1, Appendix K, Figure 5c). We replaced Figure 2 (p. 4) with a two-panel diagram:
- Panel (a), online pre-revision routing (primary deployment). KEEPER acts before the reviser; the KEEP branch returns immediately with no candidate generated, and only REVISE invokes the reviser.
- Panel (b), offline post-generation selection (n = 458 diagnostic only). This panel shows the offline diagnostic, in which an existing candidate pool is filtered by with fallback to .
Q3: Definition of in Original Table 1 (Now Table 2)
Q3: "In Table 1 [now Table 2], what is r? Please define it explicitly."
We now define in both Section 4.1 (p. 6) and the Table 2 caption. It is the sample Pearson correlation coefficient computed across all 4,186 logged RL training-step snapshots (from 18 multi-role runs) between the snapshot's false-rejection share, over all rollout items at step , and the snapshot's system gain (final workflow accuracy minus first-pass solver accuracy at step ). Across all 4,186 snapshots, (run-clustered 95% CI [-0.366, -0.184]; run-demeaned ; first-differenced ), and the correlation is negative within each of the 18 individual runs.
Response to Reviewer ZvX8 (Part 1/4)
We thank Reviewer ZvX8 for a direct and constructive review. The original submission evaluated only Qwen2.5-7B-Instruct on math (W1), mixed protocol, main results, and boundary analysis across Sections 3-5 (W2), compressed too many clauses into dense sentences such as lines 209-211 (W3), and left several architectural and statistical details implicit (Q1-Q3). In response, we trained and evaluated cross-scale (Qwen2.5-3B) and cross-family (Llama-3.1-8B) 5-fold suites across both mathematics and competitive Python programming (APPS) and restructured and rewrote the manuscript. Below is a summary of revisions followed by point-by-point responses.
Summary of Revisions