- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Rubric based RL assigns a response advantage uniformly across tokens, while probability changes under privileged context may reflect style rather than errors. ECAR uses successful local revisions to identify repair support. Revisions must produce a correct answer, improve rubric scores, preserve satisfied criteria, and remain local. Edit locations and sampled token probability drops guide advantage redistribution, preserving the response mean and token advantage signs. Experiments across three Qwen3 sizes show gains over ROPD, supported by ablations, a common training endpoint, and three independent runs for the main 4B comparison.
Critical strengths
ECAR uses verified local improvements to guide token weighting. Ablations examine certification, edit locations, tokenwise weighting, and the update formulation. Gains across three model sizes and repeated ECAR versus ROPD runs support its empirical usefulness.
Critical weaknesses
-
Uniform compensation outside repair support is insufficiently justified. Unchanged tokens are not necessarily correct or deserving of higher advantages. Preserving the mean token advantage also does not preserve the original policy gradient, since token gradients differ. No ablation isolates whether this compensation improves learning.
-
Certification cost and coverage remain unclear. The paper describes extra revision generation, verification, and scoring, but does not quantify their computational and time overhead or report eligibility and certification rates throughout training.
Questions for authors
-
How does performance change when adjustments within repair support remain identical but advantages outside it are left unchanged?
-
How often do accepted revisions include unnecessary changes or leave reasoning errors unresolved?
-
How do revision eligibility, certification success, and support size change across training and model sizes?
-
What additional computation and time does ECAR require, and do its gains over ROPD persist under comparable computational resources?
2 comments
Thank you for the additional experiments and clarifications. The ablations strengthen the evidence that certified local repairs provide useful guidance, while suggesting that outside compensation provides only a small additional gain. The coverage and training time comparisons also address my remaining practical concerns. I will maintain my score.
compensation ablation, repair quality, coverage, and cost
Thank you for the questions. We report the requested ablation and new analyses below.
Outside-support compensation
We kept adjustments inside repair support unchanged and restored the original ROPD advantages outside it. Test Avg. is 59.44 versus 59.77 for full ECAR, a difference of 0.33 points (Table 3).
Across 40 effective routed trajectories reconstructed from the first 150 steps, outside-support adjustment averages 2.22% of the absolute original advantage per token, consistent with the small performance change.
Compensation balances the changes inside support to preserve mean advantage. It does not label unchanged tokens as correct or preserve the policy gradient, which also depends on token-specific gradients. The ablation shows that local repair guidance remains effective when outside advantages are left unchanged.
Quality of accepted revisions
Certification checks for a correct final answer, a higher rubric score, repair of a failed criterion, and preservation of previously satisfied criteria. If the judge finds that an edit breaks a previously satisfied criterion, we reject the revision even when its total score improves. The minimal-edit prompt and locality checks also constrain the extent of changes.
We examined 47 accepted revisions using the same rules. Mean rubric score rises from 0.7145 to 0.9103. Of these, 29 satisfy all criteria and 18 (38.30%) retain unmet criteria. Those 18 still improve from 0.5648 to 0.7658, and all accepted revisions preserve previously satisfied criteria under the judge's assessment.
The checks establish judged improvement while filtering detected damage. ECAR uses the edits to guide the original response locally, with its ROPD advantage retaining the update direction. Local corrections can thus help even when other criteria remain unresolved.
Coverage across training and model sizes
Appendix E.5 reports candidates with nonzero advantage and one to three failed criteria. In the tables, candidate fraction and coverage use all valid scored responses as the denominator. Success is the accepted fraction among candidates. Token support counts edits and their neighborhoods over all valid response tokens.
Counts are pooled by interval; success rates use the rounded percentages. Success stays near 17-18%, so declining coverage mainly reflects fewer candidates.
To examine model scale, each initial model generated eight responses to the same 64 questions, without training:
Across scales, candidate fractions are similar (50.20%, 55.66%, 51.37%), but success rises from 5.45% to 17.54% and 40.68%. This supports larger models' stronger ability to repair responses while meeting preservation and locality requirements. ECAR improves over ROPD at all three scales, including 1.7B with lower initial coverage.
Cost and performance
On the same eight H200 GPUs with the same API settings, ROPD takes 355.18 seconds per update and ECAR 474.64 seconds (+33.6%), including API waiting and retries. Generating and checking revisions takes 101.05 seconds. The actor stage, including conditional probability evaluation, increases from 61.31 to 73.56 seconds.
At 200 updates, three-run Test Avg. is 60.03 +/- 0.39 for ECAR versus 56.64 +/- 1.45 for ROPD (+3.39 points; sample standard deviations). To account for the extra per-update time, we compare 150 ECAR updates with 200 ROPD updates: 20.68 versus 19.73 hours (+4.8%). ECAR reaches 59.48 +/- 1.41, exceeding ROPD by 2.84 points, with gains in every seed pair (Appendix D).
We reuse rubrics and batch verification. The teacher shares policy parameters and supplies its signal without backpropagation. Inference requires no revision or verification.