acceptodds
Under review as a conference paper at ICLR 2027

Learning from Local Repairs: Edit-Certified Advantage Redistribution for Rubric-Based Reinforcement Learning

Abstract

Rubric-based reinforcement learning provides richer feedback than binary rewards, but still broadcasts one response-level advantage across all generated tokens. On-policy self-distillation (OPSD) instead derives token-level guidance from distribution shifts induced by privileged context. However, privileged context can also alter wording and style, making distribution shifts an ambiguous basis for token-level credit assignment. To address this ambiguity, we introduce Edit-Certified Advantage Redistribution, which uses successful local revisions to obtain response-specific evidence for token-level credit. The failed criteria guide a reviser toward a local repair that preserves unrelated content. A revision is certified when it remains local, yields a correct answer, improves the weighted rubric score, and preserves previously satisfied criteria. Aligned edits and their neighborhoods define edit-certified repair support on the original response. ECAR uses this support to redistribute the original advantage while preserving its response-level mean and each token advantage's sign. Responses without such support retain the standard sequence-level RL update. ECAR achieves higher average performance than reinforcement learning and self-distillation baselines across five mathematical reasoning benchmarks. Ablations further establish the importance of both the revision certificate and edit-based localization. These results highlight the value of certified local repairs for token-level credit assignment.

Primary area:
foundation or frontier models, including LLMs
Keywords:
reinforcement learning, self-distillation, credit assignment, large language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Up2FOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Rubric based RL assigns a response advantage uniformly across tokens, while probability changes under privileged context may reflect style rather than errors. ECAR uses successful local revisions to identify repair support. Revisions must produce a correct answer, improve rubric scores, preserve satisfied criteria, and remain local. Edit locations and sampled token probability drops guide advantage redistribution, preserving the response mean and token advantage signs. Experiments across three Qwen3 sizes show gains over ROPD, supported by ablations, a common training endpoint, and three independent runs for the main 4B comparison.

Critical strengths

ECAR uses verified local improvements to guide token weighting. Ablations examine certification, edit locations, tokenwise weighting, and the update formulation. Gains across three model sizes and repeated ECAR versus ROPD runs support its empirical usefulness.

Critical weaknesses
  1. Uniform compensation outside repair support is insufficiently justified. Unchanged tokens are not necessarily correct or deserving of higher advantages. Preserving the mean token advantage also does not preserve the original policy gradient, since token gradients differ. No ablation isolates whether this compensation improves learning.

  2. Certification cost and coverage remain unclear. The paper describes extra revision generation, verification, and scoring, but does not quantify their computational and time overhead or report eligibility and certification rates throughout training.

Questions for authors
  1. How does performance change when adjustments within repair support remain identical but advantages outside it are left unchanged?

  2. How often do accepted revisions include unnecessary changes or leave reasoning errors unresolved?

  3. How do revision eligibility, certification success, and support size change across training and model sizes?

  4. What additional computation and time does ECAR require, and do its gains over ROPD persist under comparable computational resources?

2 comments
AuthorsOpenReview ↗

compensation ablation, repair quality, coverage, and cost

Thank you for the questions. We report the requested ablation and new analyses below.

Outside-support compensation

We kept adjustments inside repair support unchanged and restored the original ROPD advantages outside it. Test Avg. is 59.44 versus 59.77 for full ECAR, a difference of 0.33 points (Table 3).

Across 40 effective routed trajectories reconstructed from the first 150 steps, outside-support adjustment averages 2.22% of the absolute original advantage per token, consistent with the small performance change.

Compensation balances the changes inside support to preserve mean advantage. It does not label unchanged tokens as correct or preserve the policy gradient, which also depends on token-specific gradients. The ablation shows that local repair guidance remains effective when outside advantages are left unchanged.

Quality of accepted revisions

Certification checks for a correct final answer, a higher rubric score, repair of a failed criterion, and preservation of previously satisfied criteria. If the judge finds that an edit breaks a previously satisfied criterion, we reject the revision even when its total score improves. The minimal-edit prompt and locality checks also constrain the extent of changes.

We examined 47 accepted revisions using the same rules. Mean rubric score rises from 0.7145 to 0.9103. Of these, 29 satisfy all criteria and 18 (38.30%) retain unmet criteria. Those 18 still improve from 0.5648 to 0.7658, and all accepted revisions preserve previously satisfied criteria under the judge's assessment.

The checks establish judged improvement while filtering detected damage. ECAR uses the edits to guide the original response locally, with its ROPD advantage retaining the update direction. Local corrections can thus help even when other criteria remain unresolved.

Coverage across training and model sizes

Appendix E.5 reports candidates with nonzero advantage and one to three failed criteria. In the tables, candidate fraction and coverage use all valid scored responses as the denominator. Success is the accepted fraction among candidates. Token support counts edits and their neighborhoods over all valid response tokens.

4B stepsCandidate fractionSuccessCoverageToken support
1-5041.405%17.53%7.257%1.718%
51-10017.671%17.15%3.031%0.792%
101-15014.498%17.46%2.531%0.551%
151-2009.052%17.84%1.615%0.364%

Counts are pooled by interval; success rates use the rounded percentages. Success stays near 17-18%, so declining coverage mainly reflects fewer candidates.

To examine model scale, each initial model generated eight responses to the same 64 questions, without training:

ModelCandidates / validCertified / candidatesSuccessCoverageToken support
1.7B257 / 51214 / 2575.45%2.734%0.610%
4B285 / 51250 / 28517.54%9.765%1.913%
8B263 / 512107 / 26340.68%20.898%4.899%

Across scales, candidate fractions are similar (50.20%, 55.66%, 51.37%), but success rises from 5.45% to 17.54% and 40.68%. This supports larger models' stronger ability to repair responses while meeting preservation and locality requirements. ECAR improves over ROPD at all three scales, including 1.7B with lower initial coverage.

Cost and performance

On the same eight H200 GPUs with the same API settings, ROPD takes 355.18 seconds per update and ECAR 474.64 seconds (+33.6%), including API waiting and retries. Generating and checking revisions takes 101.05 seconds. The actor stage, including conditional probability evaluation, increases from 61.31 to 73.56 seconds.

At 200 updates, three-run Test Avg. is 60.03 +/- 0.39 for ECAR versus 56.64 +/- 1.45 for ROPD (+3.39 points; sample standard deviations). To account for the extra per-update time, we compare 150 ECAR updates with 200 ROPD updates: 20.68 versus 19.73 hours (+4.8%). ECAR reaches 59.48 +/- 1.41, exceeding ROPD by 2.84 points, with gains in every seed pair (Appendix D).

We reuse rubrics and batch verification. The teacher shares policy parameters and supplies its signal without backpropagation. Inference requires no revision or verification.

Thank you for the additional experiments and clarifications. The ablations strengthen the evidence that certified local repairs provide useful guidance, while suggesting that outside compensation provides only a small additional gain. The coverage and training time comparisons also address my remaining practical concerns. I will maintain my score.

Reviewer_JuoeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

ECAR uses certified local revisions to identify repair-related support on original rollouts, then redistributes rubric-derived token advantages while preserving their response-level mean and sign.

Critical strengths

The mechanism is clearly specified, with explicit certification, edit alignment, bounded redistribution, and fallback to ROPD. Table 1 reports 59.77 versus 57.23 Test Avg for ROPD. The question-paired bootstrap 95% confidence interval for the difference is [0.23, 4.95]. Table 8 additionally reports three independent step-200 runs: 60.03 ± 0.39 versus 56.64 ± 1.45, where ± denotes sample standard deviation. Table 3 provides targeted controls for certification, support location, and tokenwise weighting. Its same-support local-JSD comparison—58.35 versus ECAR’s 59.77—supports redistribution over that auxiliary distillation objective.

Critical weaknesses

The accuracy–resource tradeoff remains unquantified. Appendix D describes additional revision generation, verification, and context evaluation, but does not report measured total training time or API cost. The fixed-step comparison therefore establishes improvement at equal update counts, not equal computational expenditure.

ECAR adds an online ground-truth correctness gate relative to ROPD’s rubric reward. Both methods already share ground-truth-filtered CleanRef data, so the concern is the gate’s incremental supervision. The local-JSD control addresses one alternative use of certified evidence, but direct learning from the same certified revisions remains untested.

Some comparator results are sensitive to training stability. CriPO obtains 4.17 AIME24 accuracy at 1.7B versus 10.42 initially, and its 4B Test Avg reaches 14.15 at step 200. Appendix E.3 supplies relevant diagnostics, and CriPO performs better at 8B. Nevertheless, the sensitivity of the smaller-scale comparisons to method-specific tuning or stabilization remains unclear.

The certified-edit mechanism is distinguishable from the acknowledged ROPD, HeRL, RLSD, and CriPO precedents [1–4], and targeted ablations support its internal design. However, the improvement over ROPD involves additional revision generation, verification, and online ground-truth gating whose resource implications are not quantified. Despite the positive accuracy evidence, I remain uncertain whether the incremental benefit over simpler uses of the same certified evidence is sufficient to recommend acceptance.

[1] Fang et al. Rubric-based On-policy Distillation. 2026.

[2] Zhang et al. Experience is the Best Teacher: Motivating Effective Exploration in Reinforcement Learning for LLMs. 2026.

[3] Yang et al. Self-Distilled RLVR. 2026.

[4] Xia et al. Enhancing Rubric-based RL via Self-Distillation. Initial version, July 2026.

Questions for authors

What are the measured training and verification costs relative to ROPD? How does ECAR compare with directly learning from the same certified revisions? Would targeted stabilization materially improve the smaller-scale CriPO comparisons?

5 comments
AuthorsOpenReview ↗

ECAR: resource tradeoff, revision SFT, and CriPO stability

Thank you for the detailed assessment. We address the resource tradeoff, direct learning from revisions, and CriPO stability below.

Training and verification costs

Appendix D compares ROPD and ECAR on the same eight H200 GPUs with the same API settings. Their measured update times are 355.18 and 474.64 seconds (+33.6%), including API waiting and retries. Generating and checking revisions takes 101.05 seconds per update. The actor stage, including conditional probability evaluation, increases from 61.31 to 73.56 seconds.

At 200 updates, three-run Test Avg. is 56.64 +/- 1.45 for ROPD and 60.03 +/- 0.39 for ECAR (+3.39 points; sample standard deviations). To account for ECAR's higher per-update cost, we also compare 150 ECAR updates with 200 ROPD updates. These take 20.68 and 19.73 hours (+4.8%), respectively. ECAR achieves 59.48 +/- 1.41, a gain of 2.84 points, with improvements in all three seed pairs.

We reuse rubrics and batch verification by question. The conditional teacher shares policy parameters and runs without backpropagation, supplying credit adjustments within the existing policy update. Inference requires no revision or verification calls.

Is learning directly from revisions sufficient?

Certification identifies an improved, acceptable revision. The next question is how to learn from it: imitate the complete revised answer, or use its edits to guide updates on the original response.

We tested the first option with online ROPD+SFT. It uses the same eligibility and acceptance rules, then treats each complete accepted revision as an SFT target for the original problem. With objective LROPD+0.03LSFTL_{\mathrm{ROPD}}+0.03L_{\mathrm{SFT}}, this additional run obtains 36.67 Test Avg. and 81.46 IFEval. Its mathematical score is below the main-table GRPO baseline's 49.59.

To understand the difference between these learning signals, we froze the online ROPD+SFT checkpoint and sampled eight responses for each of 128 training problems. There were 47 accepted revisions. We replaced each original response with its revision and recomputed its advantage, keeping the rubric and the other seven responses fixed. This gives 35 positive, one zero, and 11 negative advantages (23.40%).

Thus, improving a response does not necessarily bring it above its group's average reward. Whole-response SFT nevertheless encourages imitation of that entire answer. ECAR uses the successful repair to identify where the original response needs adjustment. Within those positions, it compares original-token probabilities under ordinary and revision contexts, using the same preceding tokens. The probability drop determines the local strength, while the original ROPD advantage retains the update direction. This is why certification is coupled to token localization and credit assignment. The SFT results and diagnostic are in Appendix E.4.

CriPO stability

The 4B records show early learning followed by deterioration. Rubric reward rises from 0.519 in the first ten steps to 0.706 over steps 31-50. Between steps 1-50 and 101-150, entropy rises from 0.46 to 3.23 and truncation from 10.7% to 36.5%. Answers to the same questions change from coherent derivations to verbose, repetitive, fragmented generation. Test Avg. falls from 43.94 at step 50 to 14.15 at step 200.

Our runs finished on September 2-3, 2026, using the then-public method with a 50-step warmup followed by a constant distillation coefficient. CriPO v4, released September 27 after our runs and the September 25 submission deadline, adds adaptive strength, late-stage decay, and restrictions on long-response corrections. The original experiments primarily study medicine and science; ours use mathematical reasoning.

These later controls address the strength and extent of self-distillation, making them relevant to the deterioration we observed. ECAR controls its guidance through repair outcomes: an additional adjustment requires an accepted repair, applies to the identified edit support, and is bounded by the original advantage. A larger distribution shift under a corrective prompt therefore does not by itself produce a stronger update. The revision must first demonstrate improvement, and the adjustment remains bounded.

We will identify the evaluated CriPO version explicitly and plan to add comparisons using its updated stabilization designs in a subsequent version.

Thank you for the additional evidence. The approximately matched-time comparison (+2.84 points, improving all three seed pairs) meaningfully addresses my resource concern. Please reconcile the reported 20.68 hours with 474.64 seconds/update × 150 updates, which implies 19.78 hours. My main remaining concern is the revision-SFT comparison. Please specify its objective, SFT coefficient, loss normalization, training budget, and any available sensitivity or replication results. Also clarify whether the methods consume identical certified revisions or independently generate them under the same acceptance rules. The 11/47 negative-advantage diagnostic does not establish why SFT underperforms: an accepted revision can fall below its group’s average reward without being an unsuitable SFT target. Sampling from the trained SFT checkpoint does not establish that this disagreement caused the degradation. The CriPO version explanation is helpful; please identify the evaluated version and qualify the claims accordingly. I would like clarification of the SFT comparison before deciding whether to revise my rating.

AuthorsOpenReview ↗

Training time, revision SFT, and the CriPO version

Thank you for the follow-up. We have clarified the timing calculation in Appendix D, expanded the revision-SFT setup and analysis in Appendix E.4, and specified the evaluated CriPO version in the baseline configurations.

Training time. The 20.68 hours are the sum of the measured times for ECAR’s first 150 updates, which averaged 496.36 s/update. The figure of 474.64 s/update is the average over all 200 updates. It is lower because the final 50 updates averaged 409.46 s/update. Thus, multiplying 474.64 by 150 gives 19.78 hours, but the measured time at step 150 is 20.68 hours. At comparable training times (20.68 versus 19.73 hours), ECAR at step 150 improves Test Avg. over ROPD at step 200 by 2.84 points on average, with gains in all three seed pairs.

Revision-SFT setup. We trained Qwen3-4B for 150 updates, with 32 problems and eight rollouts per problem per update and a constant learning rate of 10−610^{-6}. The objective was LROPD+0.03LSFTL_{\mathrm{ROPD}} + 0.03L_{\mathrm{SFT}}. The SFT term predicts the full certified revision, including EOS, given the original problem. For each PPO minibatch, we average the cross-entropy over all accepted target tokens across GPUs. Prompts, padding, and rejected responses do not contribute to the SFT loss. We evaluated a single fixed coefficient of 0.03. Test Avg. rose from 32.29 at step 50 to 35.60 at step 100 and 36.67 at step 150. AIME24 validation selected step 150.

ROPD+SFT and ECAR each generate rollouts and revise them using their own current policy. Both apply the same acceptance criteria, but each uses its own certified revisions.

What the training logs show. We agree that a revision can contain a useful correction even when its reward is below the group average. Appendix E.4 now reports the revisions that actually contributed to the SFT loss in updates 101–150 of the reported run. Of the 934 certified targets, 181 (19.38%) had negative group-relative advantage when substituted for their original responses.

The distinction is between learning from an edit and imitating the whole answer. Full-response SFT encourages the model to reproduce the useful correction together with the reasoning left unchanged, even when the revised answer still scores below its peers. ReST (Appendix A.7) found that the reward threshold used to select self-generated targets affected self-training performance. STaR (Section 5) reported that imitating flawed reasoning could hurt generalization even when the final answers were correct. Our logs show that this mismatch occurred during training; the count itself does not measure its effect on test accuracy. ECAR uses the edit to locate where to adjust credit on the original rollout while preserving the direction of the original advantage.

CriPO version. We reimplemented CriPO v3 on the ROPD codebase and evaluated it in our mathematical reasoning setting. We now specify this version in the appendix. Our experiments preceded the September 27 release of v4, which introduced additional stabilization changes. The decline we report was observed in our v3-based implementation.

Reviewer_Z71XOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes ECAR, which locates tokens need to modify via successful local repairs and reassign advantage for rubric-based RL. This method is intuitive and outperforms baselines across multiple benchmarks.

Critical strengths
  1. This is an important research topic. Credict assignment is a challenging task for rubric-based RL.

  2. The method is reasonable and simple. It is easy to implment and combine with existing RL method.

  3. The proposed method outperforms ROPD on three Qwen3 models across multiple benchmarks.

Critical weaknesses
  1. It lacks experiments on thinkng models. All the experiments in this paper are on non-thinking mode while math reasoning is one important application scenario of thinking models. For long reasoning trajectories, a small local mistake may lead to a lot of later content needed to be revised, which makes revision difficult to satisfy locality constraints.

  2. It lacks the statistical information of the real effective training covering ratio. The revision must satisfy constraints and then ECAR can conduct advantage redistribution. The paper should report how many rollout accepts ECAR updating.

  3. The paper does not the computation costs. ECAR need to conduct extra generation, answer revision, and probability estimation. It uses much higher computation compared to baselines.

  4. The presentation is poor. Excessive defensive writing and unnecessary formalism significantly undermine the paper’s readability.

Questions for authors

refer to weakness

5 comments
AuthorsOpenReview ↗

ECAR: training cost, repair coverage, and long reasoning trajectories

Thank you for the feedback. We added cost and coverage measurements and revised the presentation.

Training cost and performance

Appendix D compares training time on the same eight H200 GPUs with the same API settings. ROPD takes 355.18 seconds per update and ECAR 474.64 seconds (+33.6%), including API waiting and retries.

At 200 updates, Test Avg. across three runs is 56.64 +/- 1.45 for ROPD and 60.03 +/- 0.39 for ECAR, a gain of 3.39 percentage points. The deviations are sample standard deviations.

To compare similar training times, we also evaluate ECAR at 150 updates against ROPD at 200. Their measured times are 20.68 and 19.73 hours, respectively, a difference of 4.8%. ECAR reaches 59.48 +/- 1.41, exceeding ROPD by 2.84 points, with gains in all three seed pairs.

Generating and checking revisions takes 101.05 seconds per update. The actor stage, including conditional probability evaluation, increases from 61.31 to 73.56 seconds. We reuse rubrics and batch verification by question. The teacher shares policy parameters and runs without backpropagation. Its signal enters the existing policy update. Inference uses the trained policy directly.

Long reasoning trajectories and locality

Our non-thinking-mode responses still contain multi-step mathematical derivations. ECAR allows a repair to change several separated parts of a response.

If an early mistake affects later calculations, those calculations must also be corrected. We align the original and accepted revised responses to identify the actual edits and their neighborhoods. This can include several downstream corrections while leaving valid intermediate text unchanged. When a repair requires extensive rewriting and fails the locality check, the response retains its original ROPD update.

Appendix E.5 measures how often these local repairs are available across training and model sizes.

Effective training coverage

Responses first enter revision as candidates, then undergo certification. Candidate fraction and certification coverage use all valid scored rollouts as their denominator. Certification success is the accepted fraction among candidates. Token-support coverage counts edited positions and their neighborhoods over all valid response tokens.

4B stepsCandidate fractionCertification successCertification coverageToken-support coverage
1-5041.405%17.53%7.257%1.718%
51-10017.671%17.15%3.031%0.792%
101-15014.498%17.46%2.531%0.551%
151-2009.052%17.84%1.615%0.364%

Counts are pooled within each interval. Conditional success rates are approximated from the reported percentages.

The candidate fraction falls from 41.405% to 9.052%, but candidates continue to pass certification at about 17-18%. The declining coverage therefore mainly reflects fewer responses entering revision. Responses without an accepted repair still receive the ROPD update. Together with the certification and random-support ablations, this supports selective guidance on repair-related positions.

We also sampled eight responses to the same 64 questions from each initial model, without training:

ModelCandidate fractionSuccessCoverageToken support
1.7B50.20%5.45%2.734%0.610%
4B55.66%17.54%9.765%1.913%
8B51.37%40.68%20.898%4.899%

Candidate fractions are similar, but larger models certify more of their candidates. This supports their stronger ability to repair responses while preserving satisfied criteria and meeting locality requirements. ECAR improves over ROPD at all three scales, including 1.7B with lower initial coverage.

Presentation

We removed repeated explanations and descriptions of what the method is not. Acceptance conditions now appear in prose, with formal definitions and special cases in the appendix. Each key equation is preceded by its purpose, making the procedure easier to follow.

Thank you for the detailed rebuttal. I appreciate the additional experiments on training costs and effective coverage, which largely address my second and third concerns.

However, my main concern remains unresolved. All experiments are still conducted in non-thinking mode. The authors' explanation of handling downstream edits and falling back to ROPD does not demonstrate that ECAR remains effective for long reasoning trajectories, where early errors may require extensive revisions that violate the locality constraints. The reported low certification coverage further raises concerns about the applicability of the method.

While the rebuttal improves the paper, I do not find it sufficient to resolve my primary concern. I therefore maintain my original rating.

AuthorsOpenReview ↗

Thinking-mode training: performance and certified local repairs

Thank you for clarifying the remaining concern. We have now trained both ROPD and ECAR on Qwen3-4B with thinking enabled, retaining the training data and core optimization parameters. The maximum generation length is 32K tokens, with a 40K total context limit.

Performance with thinking enabled

ECAR reaches 63.64 Test Avg., compared with 62.57 for ROPD, a gain of 1.07 percentage points. It improves on all four mathematical test benchmarks:

MethodAIME24 (val)AIME25HMMT-FHMMT-NAMC23Test Avg.IFEval
Base68.9660.2141.0449.5886.7559.3982.46
ROPD71.8864.1743.3352.7190.0662.5781.99
ECAR73.3465.2143.6454.1691.5663.6482.17

These results directly evaluate ECAR during thinking-mode training, including its ability to obtain useful guidance from repairs of reasoning trajectories.

Repairs and coverage on long trajectories

ROPD's rubric reward is computed from the final answer outside the thinking segment. ECAR uses Qwen to revise the full response, including thinking content that needs correction. After correctness and locality checks, the LLM judge examines the full revised trajectory for repaired failed criteria, preservation of satisfied criteria, and reasoning consistency. Actual edits and their neighborhoods define support across both the thinking and final-answer segments.

The resulting coverage is:

StageCandidate fractionSuccess among candidatesCertification coverageToken support
Initialization (512 responses)54.883%31.843%17.48%0.88577%
Steps 1-2551.563%28.864%14.88%0.64697%
Steps 25-5052.734%27.704%14.61%0.50751%
Steps 50-7545.508%26.009%11.84%0.35561%
Steps 75-10040.195%25.248%10.15%0.33824%

Candidate fraction and certification coverage use all valid scored responses as the denominator. Success is conditional on entering revision, and token support is measured over all valid response tokens.

During training, about 25-29% of candidate revisions pass certification, covering 10-15% of valid responses. Even in the last interval, roughly one quarter of candidates yield accepted repairs. Thus, long reasoning trajectories continue to provide usable local corrections under the locality constraints.

The certificate requires a correct final answer, improvement on failed criteria, and preservation of satisfied criteria. An accepted repair can still leave other rubric criteria unmet. This allows a verified local improvement to guide learning without requiring every reasoning issue in a trajectory to be repaired at once.

The longer thinking trajectories make local edits and their neighborhoods a smaller fraction of the response, with token-support coverage of 0.33824-0.64697% during training. This is consistent with the smaller improvement over ROPD: 1.07 points here versus 2.54 in the non-thinking main experiment. Non-thinking training therefore shows the larger incremental benefit in our experiments, while the new results establish that ECAR also improves performance in thinking mode.

We have added the settings, benchmark results, and coverage analysis to Appendix E.6, "Thinking-Mode Training."

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary: ECAR’s Gains at Comparable Training Time, in Thinking Mode, and over Revision-SFT

Dear Area Chair,

Thank you for considering our paper. ECAR asks a simple question: when a model successfully repairs a mathematical solution, what should it learn from that repair? Our approach uses the revision to locate the original tokens involved in the correction. It then adjusts how strongly those tokens are updated while retaining the direction of the original response-level advantage. This lets the model learn from a useful correction without treating the entire revised solution as an imitation target.

The original submission showed gains over ROPD across three Qwen3 sizes (1.7B, 4B, and 8B). At 4B, ECAR improved the four-benchmark math test average from 57.23 to 59.77. At a fixed 200 updates, the gain was 3.39 points across three independent training seeds. The discussion added evidence on three questions that the original experiments left open.

Does the gain persist at comparable training time? ECAR costs 33.6% more per update, so we compared 150 ECAR updates with 200 ROPD updates on the same eight H200 GPUs, including verification time and API waits. These runs took 20.68 and 19.73 hours, respectively. ECAR scored 59.48 versus 56.64 for ROPD, a 2.84-point gain, with improvements in all three seed pairs. This comparison addresses the cost of obtaining the repairs, beyond the original equal-update comparison.

Do local repairs still help on long thinking trajectories? We trained both methods with thinking enabled and allowed revisions to correct the reasoning trace as well as the final answer. During training, 10–15% of valid responses yielded a certified repair, even though the edits and their neighborhoods covered less than 0.7% of response tokens. At the reviewer’s request, we then reevaluated both step-50 checkpoints using Qwen3’s published thinking-mode sampling settings. Across three independent evaluation seeds, Test Avg. was 63.90 ± 0.26 for ECAR, 62.81 ± 0.22 for ROPD, and 60.93 ± 0.06 for Base. ECAR outscored ROPD in each repeat.

What matters about how a repair is used? Two new comparisons help separate the effects. First, keeping the adjustments inside the repair region unchanged while removing compensation outside it gave 59.44, close to full ECAR’s 59.77. The local adjustment therefore retains most of the observed gain on its own. Second, we tested online revision-SFT: each method generated its own revisions under the same acceptance rules, but SFT learned from the entire accepted answer. Reducing the SFT weight from 0.03 to 0.01 improved its step-100 score from 35.60 to 45.65; ECAR reached 56.48 at the same step. The lower weight also helped at step 50, where ECAR remained ahead. Thus, the SFT coefficient mattered substantially, but the advantage of learning from the location of a repair persisted at both checkpoints tested.

The discussion also clarified why coverage changes: during non-thinking 4B training, fewer responses became eligible for revision over time, while the fraction of candidates passing certification stayed near 17–18%. We report these measurements alongside the cost breakdown and thinking-mode results in Appendices D and E. The main ECAR–ROPD comparison is replicated across training seeds; the remaining question raised by Reviewer Juoe is how much the individual component and revision-SFT comparisons vary across seeds. We plan to report those paired-seed comparisons in a follow-up revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.