acceptodds
Under review as a conference paper at ICLR 2027

VERPO: VERIFIED EVIDENCE REGULARIZED POLICY OPTIMIZATION

Abstract

Verifiable rewards improve language models through reliable task-level feedback, but methods based on Group Relative Policy Optimization (GRPO) apply a sequence-level advantage uniformly across all tokens. This coarse credit assignment reinforces or penalizes entire responses without identifying which local decisions to preserve, reinforce, or revise. Conversely, evidence-conditioned self-distillation provides denser token-level supervision, yet teacher imitation can transfer stylistic artifacts and miscalibrated confidence that destabilize training when misaligned with task success. We introduce VERPO, which converts evidence-conditioned guidance into reward-aligned token-level credit assignment while retaining the outcome objective. VERPO decomposes teacher guidance into an evidence-free reference term and signed, evidence-induced corrections at each token. A stopped controller combines selective acceptance, token-wise localization, and cost-aware scaling by balancing alignment with the local GRPO update direction against Fisher movement cost. Furthermore, we introduce Fisher Evidence Contrast (FEC), which attenuates nuisance shifts along an estimated evidence-presence direction through a regularized projection. Across five scientific reasoning and tool-use tasks, VERPO prevents optimization collapse and consistently achieves the highest multi-task average across model backbones, yielding marked improvements particularly on smaller models over strong baselines. Qualitative diagnostics confirm that token acceptance selectively targets reasoning bottlenecks consistent with local reward alignment and Fisher movement cost.

Primary area:
reinforcement learning
Keywords:
large language models, reinforcement learning with verifiable rewards, on-policy self-distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_jJ2VOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper utilizes a teacher model that has seen the correct answer to provide word-by-word guidance, and then filters these guidances according to reward direction and update cost to improve the effectiveness and stability of large model inference and tool call training.

Critical strengths

It proposes a complete solution to the specific problem that "the reward for the final answer cannot accurately guide the intermediate steps" and provides multi-model experiments, component ablation, and partial computing power matching verification.

Critical weaknesses

Some lead margins are very small, and the statistical reliability is insufficient. On Qwen3-4B, the average score across five tasks is only 0.31 percentage points higher than the strongest baseline; the main table does not provide the mean and confidence intervals for multiple independent training iterations. Such a gap may stem from random fluctuations and requires validation with multiple random seeds.

Using "highest score during training" makes the model selection process unclear. The main table reports the best checkpoint, but the appendix only explicitly lists the training and test sets, without clearly explaining how the independent validation set is used to select checkpoints. Here, we need to rule out the risk that repeatedly viewing the same evaluation set and taking the highest score might lead to overly optimistic reporting results; currently, we cannot directly conclude that test leakage has occurred.

Word-by-word weights do not equal the true causal contribution of each word. It estimates whether the teacher's suggestion aligns with the current reward update direction, but does not prove that "modifying this word will improve the final accuracy." The authors themselves acknowledge that this is only a local proxy; to prove accurate allocation of credit, experiments involving the deletion, replacement, or intervention of high-weight segments are needed.

Overall, I think this paper is valuable after the revision. Looking forward to authors' rebuttal.

Questions for authors

See weaknesses.

6 comments
AuthorsOpenReview ↗

Seed Variability, Checkpoint Selection, and Intervention Evidence

Thank you for your valuable comments.

W1. Variability of the 0.31-point Qwen3-4B lead

We added Qwen3-4B runs with seeds 43 and 44 alongside historical seed 42. All five tasks now have three completed seeds, for 15 runs in total. Table 16 and Appendix F.6, Q9 report every seed's result, including the completed Biology seed 44 at 61.62% Best and 56.76% Final. We summarize Best within steps 1–200 using the sample SD and a 95% Student-t confidence interval with two degrees of freedom. Scores are percentages.

TaskSeed 42 BestSeed 43 BestSeed 44 BestMean Best ± SD95% confidence interval
Biology62.0059.7561.6261.12 ± 1.2058.13 to 64.12
Chemistry69.1770.9270.7670.28 ± 0.9767.88 to 72.69
Materials79.7280.7879.1879.89 ± 0.8177.87 to 81.92
Physics70.5569.0669.9369.85 ± 0.7567.99 to 71.71
Tool Use61.4060.3865.6962.49 ± 2.8255.49 to 69.49

Variability is largest on Tool Use. The intervals estimate mean Best under this protocol, assuming independent, approximately normal seed-level scores. With three seeds, individual runs can noticeably affect the interval. All main-text comparisons use Best-checkpoint avg@16 for VERPO-AM, VERPO-LW, and every baseline. The Qwen3-4B comparison with SRPO Lietal,˙2026Li et al\., 2026 below reproduces Table 3. Both five-task averages use Best, and ΔBest gives VERPO-LW minus SRPO in percentage points.

TaskSRPO BestVERPO-LW BestΔBest
Biology60.1262.00+1.88
Chemistry72.4769.17−3.30
Materials78.7879.72+0.94
Physics67.8970.55+2.66
Tool Use62.0461.40−0.64
Five-task average68.2668.57+0.31

The original averages of 68.57% and 68.26% give the 0.31-point lead. Our repeats cover VERPO-LW, so establishing the lead's significance needs matched baseline repeats. We report it as an observed difference, with its statistical significance unresolved.

Locations in the revised manuscript

  • Section 4.2, “Main Results,” and Table 3 on page 7 report the original Qwen3-4B five-task comparison and the 0.31-point average margin.

  • Appendix F.6, Q9, “How much do results vary across training seeds?”, and Table 16 on page 32 report the available seed-level scores, sample standard deviations, and 95% confidence intervals.

  • Appendix F.6, Q8, “How large is the Best–Final gap?”, on page 32 explains the Best and fixed-step Final reporting used in the same table.

W2. Best selection and evaluation splits

We select the highest evaluation score during training as Best for every method in Table 3. Historical runs used repeated evaluations on the released test split for checkpoint selection, which the runtime labels “validation.” We added step-200 Final scores to show performance retained at the end of training. For avg@16, we average correctness over 16 sampled responses per evaluation prompt and evaluate every five training steps. Appendix H gives the dataset splits and decoding settings. Table 16 pairs Best within steps 1–200 with Final at step 200 for the available Qwen3-4B seeds. The historical seed-42 results below show the gap for each task. Scores are percentages and gaps are percentage points. Appendix F.6, Q8 discusses these differences.

TaskBestFinal at step 200Best − Final
Biology62.0059.132.87
Chemistry69.1765.803.37
Materials79.7279.520.20
Physics70.5569.141.41
Tool Use61.4057.813.59

The Best–Final gaps show how much peak performance is retained. Selecting peaks on the test split can make the reported scores optimistic, including the small average lead. The seed intervals describe this same selection protocol. Independent generalization estimates would need separate selection and held-out evaluation splits.

Locations in the revised manuscript

  • Section 4.2, “Main Results,” and the caption of Table 3 on page 7 identify the main-table metric as Best-checkpoint avg@16.

  • Appendix F.6, Q8, “How large is the Best–Final gap?”, and Table 16 on page 32 provide the paired Best and fixed step-200 Final values. The table caption restricts the reported seed-level Best to steps 1–200.

  • Appendix H.1, “Datasets and Data Processing,” pages 34–35, and Table 18 on page 35 list the train/test splits. Appendix H.3, “Shared Training Setup,” and Table 19 on page 35 give the evaluation sampling and decoding settings.

W3. The token-removal intervention

We resumed Llama-3.2-1B Materials from a shared step-100 checkpoint at 20.41% accuracy and trained to step 200. We removed evidence-loss contributions from either the highest-weight 20% of eligible positive-weight tokens per response or an equal number chosen at random. Other weights and the original normalization stayed fixed. All three branches share the same checkpoint and continuation window. Accuracy is reported in percent.

Continuation branchShared step-100 accuracyBestFinal at step 200
Full VERPO-LW continuation20.4152.3950.48
Remove highest-weight 20%20.4130.1720.24
Equal-count random removal20.4149.1347.99

Targeted removal gives 18.96 points lower Best and 27.75 points lower Final than random removal. Full continuation performs best on both metrics. Figure 11 and Appendix F.6, Q11 report the trajectories and results. The high-weight corrections matter more for later learning in this backbone, task, and shared trajectory. Equal token counts can remove different total weight, so both selection and correction magnitude contribute. Assigning causal credit to individual tokens needs further controls.

Locations in the revised manuscript

  • Section 3.3, “Token-Wise ZPD Controller,” pages 4–5, defines the local acceptance proxy. Section 4.3, “Ablation Study,” reports “Correction-Removal Intervention” and Table 5c,dc,d on page 9.

  • Appendix F.6, Q11, “Do high-weight tokens matter more for learning?”, starts on page 32 and describes the matched-count deletion experiment. Figure 11 on page 33 reports all three continuation branches.

  • Appendix F.6, Q12, “What could explain the intervention outcome?”, on page 33 discusses the interpretation of the intervention and the influence of correction magnitude.

Hi, we sincerely thank you very much for these constructive comments and evaluation of our manuscript. We would like to kindly ask you to take a look at our responses and reevaluate our work based on our clarifications. Please let us know whether our response addresses your concerns or whether there is any further detail we can provide to help address these concerns.

Thank you again for dedicating your time to reviewing our paper.

Reviewer_QXDZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to combine GRPO, which learns from verifiable rewards, with evidence-conditioned self-distillation, in which a self-teacher that sees privileged evidence provides token-level guidance. It proposes VERPO, which splits the teacher signal into an evidence-free reference term and a signed evidence-induced correction. A closed-form token-wise controller then accepts and scales each correction according to its alignment with the local GRPO update direction and a Fisher movement cost. The paper also introduces Fisher Evidence Contrast (FEC), which removes the component of the correction caused by the mere presence of evidence. Experiments on four SciKnowEval science tasks and ToolAlpaca with Qwen3-4B, Qwen3-8B, and Llama-3.2-1B report the best average accuracy on each backbone, together with ablations and token-level diagnostics.

Critical strengths

S1. A sensible way to filter teacher guidance. Imitating an evidence-conditioned teacher can transfer style and confidence shifts that hurt training. The paper addresses this real problem by separating evidence-free reference restoration from signed corrections, and by accepting a correction only when it agrees with the GRPO direction. The controller has a simple closed form and adds no learned parameters.

S2. Stable training where baselines fail. On Llama-3.2-1B, several baselines collapse or reward hack, while VERPO trains stably across all five tasks. The uniform-weight ablation also shows that adding evidence corrections without selection easily destabilizes small models, which supports the motivation for selective acceptance.

Critical weaknesses

W1. The contribution of the token-wise controller is not isolated. The controller is the paper's main new component, yet the only direct comparison with a constant weight is Table 4c, on a single task (Chemistry). Even there the result is mixed. A constant weight of 0.25 reaches 71.99 on Qwen3-4B, above VERPO-LW's 69.17 in Table 3, but it is lower on the other two backbones. FEC is compared with CTR only on Llama-3.2-1B, and the overall margin on Qwen3-4B is 0.31 points. The current results do not show whether the gains come from the controller and FEC, or simply from adding a down-weighted evidence loss to GRPO.

W2. The diagnostics do not show that the controller targets reasoning bottlenecks. The abstract states that the diagnostics "confirm that token acceptance selectively targets reasoning bottlenecks." Section 4.4, however, only shows where the weights fall. Most of the weight lands in the intermediate reasoning region, which the paper itself partly attributes to that region being longer. The token rankings are also described as characterizing "acceptance rather than causal error contributions." No experiment checks whether the accepted tokens are the ones that decide correctness, for example by removing corrections on high-weight tokens versus random tokens. At present, this claim rests on how the controller is designed, not on experimental evidence.

Questions for authors
  1. Please compare VERPO-LW with a constant evidence weight on all five tasks and all three backbones.

  2. Were the baselines tuned for Llama-3.2-1B? Several of them collapse on this backbone, where VERPO shows its largest gain.

  3. Do high-weight tokens matter more for correctness than other tokens? A simple test would be to remove corrections on high-weight tokens versus random tokens.

5 comments
AuthorsOpenReview ↗

Extended Comparisons, Baseline Tuning, and Token Interventions

Thank you for your valuable comments.

W1 and Q1. Uniform weighting and the FEC comparison

We extended the uniform-weight comparison to all 15 task–backbone settings in Table 13 and Appendix F.6, Q1. We replaced adaptive weights with a constant 0.25 while keeping the evidence direction and reference component. Scores are Best percentages, and ΔBest gives VERPO-LW minus Uniform in percentage points.

ModelTaskVERPO-LW BestUniform BestΔBest
Llama-3.2-1BBiology60.1519.88+40.27
Llama-3.2-1BChemistry66.5753.48+13.09
Llama-3.2-1BMaterials52.2617.95+34.31
Llama-3.2-1BPhysics54.2252.34+1.88
Llama-3.2-1BTool Use49.6347.70+1.93
Qwen3-4BBiology62.0058.13+3.87
Qwen3-4BChemistry69.1771.99−2.82
Qwen3-4BMaterials79.7277.86+1.86
Qwen3-4BPhysics70.5568.05+2.50
Qwen3-4BTool Use61.4061.30+0.10
Qwen3-8BBiology62.3753.38+8.99
Qwen3-8BChemistry80.2673.30+6.96
Qwen3-8BMaterials77.6278.39-0.77
Qwen3-8BPhysics68.8562.10+6.75
Qwen3-8BTool Use68.1564.15+4.00

Adaptive weighting wins in 13 of 15 settings. Uniform weighting leads on Qwen3-4B Chemistry and Qwen3-8B Materials. The gain is broad across this suite but depends on the task and backbone. This control tests adaptive selection and scaling together. We tested the controller components on Chemistry with matched initial correction strength. Positive-benefit filtering helps Llama but lowers Qwen3-4B Best. Removing Fisher cost lowers Best by 4.67 and 6.88 points, respectively. These results show the cost term helping both backbones, while sign filtering alone has mixed effects. Table 14 and Appendix F.6, Q3–Q4 give the settings. We also compared FEC with CTR Panetal,˙2026Pan et al\., 2026 on Qwen3-4B, keeping the controller and LW path. FEC gives higher Best on four tasks. CTR leads on Physics at 73.12% versus 70.55%. FEC's benefit therefore extends to this larger backbone, with a task-specific exception. Table 17 and Appendix F.6, Q10 report the results. Qwen3-4B avg@16 scores are percentages. ΔBest is FEC minus CTR in percentage points.

TaskFEC BestCTR BestΔBestFEC FinalCTR Final
Biology62.0053.75+8.2559.1352.12
Chemistry69.1765.77+3.4065.8063.76
Materials79.7279.52+0.2079.5279.52
Physics70.5573.12−2.5769.1472.50
Tool Use61.4057.16+4.2457.8155.60

We added GRPO+OPSD Shaoetal,˙2024;Zhaoetal,˙2026aShao et al\., 2024; Zhao et al\., 2026a on Chemistry. VERPO-LW has higher Best on both backbones. The baseline uses a frozen Teacher and different reference-solution construction, so this tests the full recipe. Table 14 and Appendix F.6, Q6 give the comparison. The full Chemistry comparison is below. Scores are avg@16 percentages, and RH denotes reported reward hacking. The signed-weight run shows reported reward hacking on Llama-3.2-1B and reaches 63.09% Best and 57.94% Final on Qwen3-4B. Appendix F.6, Q5 and Table 14 report these results.

Chemistry methodLlama BestLlama FinalQwen3-4B BestQwen3-4B Final
Uniform weight 0.2553.4841.6171.9961.99
Positive-benefit filter64.7062.4462.8662.56
Fisher cost removed61.9057.6262.2958.60
GRPO+OPSD54.3546.0467.0266.61
Signed-weightRHRH63.0957.94
VERPO-LW66.5766.5769.1765.80

The repeated runs measure VERPO-LW's variability. Testing the small 0.31-point Qwen3-4B lead requires matched baseline repeats, so we report it as an observed difference.

Locations in the revised manuscript

  • Section 4.3, “Ablation Study,” pages 8–9, contains “Uniform versus Adaptive Weighting” and “Chemistry Comparisons,” with Table 5a,ba,b on page 9. The original Llama direction ablation is in Table 4aa on page 8.

  • Appendix F.6, Q1, “Does adaptive weighting outperform a constant weight?”, and Table 13 on page 30 report all 15 task–backbone settings.

  • Appendix F.6, Q3, “Does positive-benefit filtering improve learning?”, Q4, “Does the Fisher cost term improve learning?”, Q5, “What happens without positive-benefit clipping?”, and Q6, “Can reinforcement and self-distillation match VERPO-LW?”, appear on page 31 and use Table 14 for the controller and direct-distillation comparisons.

  • Appendix F.6, Q10, “Does FEC help on a larger backbone?”, and Table 17 on page 32 provide the Qwen3-4B FEC/CTR comparison. The 0.31-point lead is reported in Section 4.2 and Table 3 on page 7, while Appendix F.6, Q9 and Table 16 on page 32 report the available seed variability.

W2 and Q3. The token-removal intervention

We resumed Llama-3.2-1B Materials from the shared step-100 checkpoint at 20.41% accuracy and trained to step 200. We removed the evidence-loss contributions of either the highest-weight 20% of eligible positive-weight tokens per response or an equal number chosen at random. Other weights and the original loss normalization stayed fixed.

Continuation branchBestFinal at step 200
Full VERPO-LW continuation52.39%50.48%
Remove highest-weight 20%30.17%20.24%
Equal-count random removal49.13%47.99%

Targeted removal gives 18.96 points lower Best and 27.75 points lower Final than random removal. Figure 11 and Appendix F.6, Q11 show the curves and results. High-weight corrections matter more for learning in this run. The gap combines token selection and removed weight, so a causal bottleneck claim needs further controls and replication. The revised abstract says, “Qualitative diagnostics show selective acceptance within intermediate reasoning, consistent with local reward alignment and Fisher movement cost.”

Locations in the revised manuscript

  • Section 4.3, “Ablation Study,” contains “Correction-Removal Intervention” and Table 5c,dc,d on page 9. Section 4.4, “Further Analysis,” discusses the original token-weight diagnostics under “Token-Level FEC Weight Analysis” with Figure 6 on page 10.

  • Appendix F.6, Q11, “Do high-weight tokens matter more for learning?”, starts on page 32 and reports the shared-checkpoint deletion protocol. Figure 11 on page 33 supplies the training curves and Best/Final results.

  • Appendix F.6, Q12, “What could explain the intervention outcome?”, on page 33 discusses the proposed mechanism and the role of removed correction magnitude.

Q2. Retuning the Llama baselines

We retuned SDPO Hu¨botteretal,˙2026Hübotter et al\., 2026 on Physics and RLCSD Panetal,˙2026Pan et al\., 2026 on Chemistry using Llama-3.2-1B. We varied learning rate and importance-sampling clipping for SDPO, and learning rate and modulation strength for RLCSD. All grid scores below are avg@16 percentages..

Baseline / taskLearning rateIS clip / modulationBestFinal
SDPO / Physics1e-05220.560.00
SDPO / Physics1e-05143.3613.20
SDPO / Physics5e-06237.500.00
SDPO / Physics5e-06148.2845.23
RLCSD / Chemistry5e-060.521.1321.13
RLCSD / Chemistry5e-060.2520.2614.16
RLCSD / Chemistry2.5e-060.510.565.62
RLCSD / Chemistry2.5e-060.2524.0713.57

The best completed SDPO setting improves Physics Best from the original pre-collapse peak of 40.15% to 48.28%, with 45.23% Final. VERPO-LW reaches 54.22% Best. The best newly completed RLCSD setting reaches 24.07% Best and 13.57% Final on Chemistry, compared with 66.57% Best for VERPO-LW. Tuning narrows the Physics gap, while the Chemistry gap stays large in the tested grid. This evidence covers these two baseline–task pairs. Table 15 and Appendix F.6, Q7 give the grids.

Locations in the revised manuscript

  • Section 4.2, “Main Results,” and Table 3 on page 7 contain the original Llama baseline scores. Section 4.4, “Further Analysis,” discusses the original Physics training dynamics with Figure 4 on page 9.

  • Appendix F.6, Q7, “Can retuning close the baseline gap?”, on page 31 describes the retuning comparisons. Table 15 on page 32 reports the complete SDPO/Physics and RLCSD/Chemistry grids.

Thank you very much for your constructive comments and review of our manuscript. We respectfully invite you to consider our responses and reassess our work in light of the clarifications provided. We would be grateful to know whether our replies resolve your concerns or whether further details would be useful. Once again, thank you for taking the time to review our paper.

Reviewer_3t1qOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes VERPO, which combines outcome-based reinforcement learning with evidence-conditioned self-distillation. It separates evidence-free reference regularization from signed teacher corrections and weights each correction using its alignment with the sampled GRPO direction and its estimated Fisher movement cost. Fisher Evidence Contrast additionally projects out a direction associated with evidence presence. Experiments cover four scientific reasoning tasks and one tool-use task across three model backbones.

Critical strengths
  1. A coherent approach to controlling teacher guidance. Separating reference regularization from signed corrections allows the method to reject evidence without disabling reference preservation. This directly addresses the practical concern that privileged teacher guidance can introduce changes unrelated to task success. FEC also provides a concrete refinement of correct-versus-wrong teacher contrast.

  2. Promising empirical gains, particularly on the smallest model. VERPO-LW improves the Llama-3.2-1B five-task average from 47.51 with GRPO to 56.57. The approximate compute-matched comparison retains a 2.35-point advantage. FEC improves over the unprojected contrast on all five tasks in this setting. These results provide meaningful evidence that the overall training recipe is useful.

Critical weaknesses
  1. The acceptance rule does not clearly address the credit-assignment problem used to motivate it.

    The paper motivates token-level guidance by noting that a sequence-level reward cannot distinguish useful from harmful decisions within a response. However, Eq. (6) defines the acceptance score as

    hi,t=[A^i(eyi,t−pi,t)⊤ui,t]+.h_{i,t}=\left[\widehat{A}i(e{y_{i,t}}-p_{i,t})^{\top}u_{i,t}\right]_{+}.

    Here, A^i\widehat{A}i is the sequence-level advantage, eyi,te{y_{i,t}} is the sampled-token one-hot vector, pi,tp_{i,t} is the actor distribution, and ui,tu_{i,t} is the evidence correction direction.

    Although the teacher provides token-specific information, acceptance still depends on agreement with reinforcing or suppressing the sampled token according to the sequence-level outcome. For a positive-advantage response, a correction that favors a better alternative over the sampled token can therefore be rejected. Besides, when every response in a group receives the same reward, all evidence corrections are suppressed even if the teacher provides useful guidance.

    The paper correctly acknowledges that this score measures alignment rather than verified improvement. The remaining question is why this alignment is an effective criterion for selecting local corrections. The reported weight and alignment diagnostics describe the gate’s behavior without establishing this.

  2. The need for Fisher-based cost scaling is not well established by the experiments.

    Equations (6)–(7) define the movement cost and controller weight as

    ci,t=ui,t⊤F(pi,t)ui,t,F(p)=diag⁡(p)−pp⊤.c_{i,t}=u_{i,t}^{\top}F(p_{i,t})u_{i,t},\qquad F(p)=\operatorname{diag}(p)-pp^{\top}.

    wi,t∗=sg⁡[hi,thi,t+αcostci,t+ϵcost].w_{i,t}^{*}=\operatorname{sg}\left[\frac{h_{i,t}}{h_{i,t}+\alpha_{\mathrm{cost}}c_{i,t}+\epsilon_{\mathrm{cost}}}\right].

    The cost estimates how much a correction would locally change the actor’s distribution. This provides a plausible reason to damp large updates, but does not establish why this estimate is preferable to simply reducing or clipping the correction strength. Table 1 identifies cost-aware scaling as the additional capability over RLCSD, making its empirical justification central to assessing the contribution.

    The FEC-versus-CTR comparison supports the projection step, but does not isolate the cost term. The constant-weight experiments change both token selection and correction strength. They show that strong corrections can destabilize training, while a small constant can work well. This does not establish that constant weighting is generally better, but makes a comparison against simpler damping particularly relevant.

    The motivation also needs clarification for VERPO-AM, where the cost is calculated for ui,tu_{i,t}, but the applied update instead rescales the GRPO direction.

Questions for authors
  1. Does filtering corrections by their agreement with the sampled GRPO direction improve learning compared with applying those corrections without this filter, at comparable overall correction strength?

  2. Can you isolate the contribution of the Fisher cost by setting αcost=0\alpha_{\mathrm{cost}}=0, while retaining the same evidence direction and reference regularization?

  3. Why is ci,t=ui,t⊤F(pi,t)ui,tc_{i,t}=u_{i,t}^{\top}F(p_{i,t})u_{i,t} an appropriate cost for VERPO-AM, where the update rescales the GRPO direction rather than applying ui,tu_{i,t}? What connects this estimated cost to the update actually performed?

  4. What exactly supplies the positive and negative evidence in each task? The main text describes ground-truth and wrong answers, while the appendix also mentions selected sibling responses. Please specify how these are chosen, what happens when suitable sibling responses are unavailable, and whether the compared distillation methods use the same positive evidence.

6 comments
AuthorsOpenReview ↗

Controller Ablations and Evidence Selection

Thank you for your valuable comments.

W1 and Q1. Alignment filtering at matched initial correction strength

We compared uniform weight 0.25 with weight 0.25 applied only to tokens with positive raw alignment. With the Actor and Teacher frozen, we matched the sum of squared evidence-gradient norms over four training batches by adjusting the evidence-loss coefficient, then kept it fixed during training. This matches initial correction strength, which can evolve differently across runs. Chemistry avg@16 accuracy is reported in percent. Best is the comparison metric.

ModelMethodBestFinal at step 200
Llama-3.2-1BUniform weight 0.2553.4841.61
Llama-3.2-1BPositive-benefit filter64.7062.44
Llama-3.2-1BFull VERPO-LW66.5766.57
Qwen3-4BUniform weight 0.2571.9961.99
Qwen3-4BPositive-benefit filter62.8662.56
Qwen3-4BFull VERPO-LW69.1765.80

Filtering raises Chemistry Best by 11.22 points on Llama-3.2-1B but lowers it on Qwen3-4B. The full controller reaches 66.57% and 69.17%, respectively. Sign filtering alone therefore helps only one of these two backbones under this calibration. Table 14 and Appendix F.6, Q3 give the comparison. We also removed the highest-weight 20% of corrections after the shared step-100 checkpoint on Llama Materials. Best falls to 30.17%, compared with 49.13% for equal-count random removal and 52.39% for full continuation. The high-weight corrections matter more in this run, with both token selection and removed weight contributing to the gap. Figure 11 and Appendix F.6, Q11 report the intervention. The Materials continuation branches share the same step-100 checkpoint. Scores below are percentages.

Continuation branchShared step-100 accuracyBestFinal at step 200
Full VERPO-LW continuation20.4152.3950.48
Remove highest-weight 20%20.4130.1720.24
Equal-count random removal20.4149.1347.99

The controller uses the current sampled GRPO direction Shaoetal,˙2024Shao et al\., 2024 to accept corrections, with prefix-dependent Actor probabilities and Teacher directions giving token-specific decisions. This can reject a useful alternative with negative alignment. Equal group rewards give zero advantages and zero evidence acceptance, while reference restoration stays active.

Locations in the revised manuscript

  • Section 3.3, “Token-Wise ZPD Controller,” pages 4–5, defines the local alignment proxy and acceptance rule. Section 4.3, “Ablation Study,” discusses the calibrated filter under “Chemistry Comparisons” on page 8, with the results in Table 5bb on page 9.

  • Appendix F.6, Q3, “Does positive-benefit filtering improve learning?”, and Table 14 on page 31 provide the strength calibration and Chemistry comparison.

  • The intervention cited in this answer appears in Section 4.3 under “Correction-Removal Intervention” and Table 5c,dc,d on page 9. Appendix F.6, Q11, “Do high-weight tokens matter more for learning?”, starts on page 32, with Figure 11 on page 33.

W2 and Q2. The role of Fisher cost

We set the cost coefficient to zero, kept the FEC direction, reference term, positive-benefit clipping, and denominator stabilizer, and matched initial correction strength using the frozen-batch procedure above. The calibrated coefficient stayed fixed during training. The cost-removal comparison reports Chemistry avg@16 accuracy in percent.

ModelControllerBestFinal at step 200
Llama-3.2-1BFull controller66.5766.57
Llama-3.2-1BFisher cost removed61.9057.62
Qwen3-4BFull controller69.1765.80
Qwen3-4BFisher cost removed62.2958.60

Removing Fisher cost lowers Best by 4.67 points on Llama-3.2-1B and 6.88 points on Qwen3-4B. Final also falls on both backbones. The cost term helps under this calibration, with a comparison against separately tuned clipping rules still open. Table 14 and Appendix F.6, Q4 give the results.

Locations in the revised manuscript

  • Section 3.3, “Token-Wise ZPD Controller,” pages 4–5, defines the Fisher cost and its role in the acceptance rule. Section 4.3 discusses the cost-removal result under “Chemistry Comparisons” on page 8, with Table 5bb on page 9.

  • Appendix F.6, Q4, “Does the Fisher cost term improve learning?”, and Table 14 on page 31 give the cost-off setting, retained components, initial-strength calibration, and measured results.

W2 and Q3. Fisher cost in VERPO-AM

We explored VERPO-AM as a heuristic inspired by RLCSD Panetal,˙2026Pan et al\., 2026 and RLSD Yangetal,˙2026aYang et al\., 2026a. We use the stopped evidence-acceptance weight to scale the sequence advantage at each token. The multiplier preserves the advantage's sign and equals one when acceptance is zero. Table 3 shows higher five-task Best averages for LW than AM on all three backbones. The averages below are percentages. The cost-removal experiment tests the cost term within LW.

ModelVERPO-LW average BestVERPO-AM average Best
Llama-3.2-1B56.5755.19
Qwen3-4B68.5766.71
Qwen3-8B71.4470.58

Fisher cost describes local movement along the evidence direction used by VERPO-LW. In AM, this cost informs modulation through the acceptance weight. An analysis of the actual AM update would need the rescaled outcome direction and its parameter-space interactions. The new cost-removal evidence applies to LW. Section 3 defines both updates.

Locations in the revised manuscript

  • Section 3.5, “Two Actor-Update Paths,” page 6, defines the AM multiplier in Equation 1010 and the joint objective in Equation 1111. Section 3.3, pages 4–5, defines the evidence-direction cost used to obtain the acceptance signal.

  • Section 4.2, “Main Results,” and Table 3 on page 7 provide the LW and AM averages reproduced above. Appendix F.1, “Joint Loss Weighting and Advantage Modulation,” page 26, gives the joint-path comparison in Table 7 and path-annealing results in Figure 7.

  • Appendix F.6, Q4, “Does the Fisher cost term improve learning?”, and Table 14 on page 31 provide the related LW cost-removal experiment. The AM interpretation in this answer follows the update definition in Section 3.5.

Q4. Evidence selection and matching

We select evidence from sibling responses to the same prompt. Correctness is checked against the answer option for Biology, Chemistry, Materials, and Physics, and against the required actions and arguments for Tool Use. “GT” means a verified-correct sibling, with ground truth supplying the correctness label. The supplementary code implements this protocol explicitly. It selects nonempty, format-valid siblings from the same prompt, excludes the target response, and uses a fixed rollout order. The positive and negative Teacher branches condition on the selected siblings and replay the same target response. Fix requires a positive sibling, while CTR/FEC require both positive and negative siblings. When required evidence is unavailable, the code masks out its contribution. The outcome and reference terms stay active, and zero acceptance gives an AM multiplier of one. The table below compares the evidence definitions used by VERPO and RLCSD Panetal,˙2026Pan et al\., 2026.

TermVERPO selection logicRLCSD selection logic
Positive evidence / Correct evidence / GTA verifier-correct sibling response selected from the same prompt's rollout groupSame
Negative evidence / Incorrect evidence / Wrong answerAn incorrect sibling response selected from the same prompt's rollout groupSame
q⁺Teacher distribution conditioned on the correct sibling as evidenceSame
q⁻Teacher distribution conditioned on the incorrect sibling as evidenceSame
q⁰Teacher distribution under the original prompt without added evidenceThe contrast q⁺ − q⁻ is constructed directly from the two evidence-conditioned branches

VERPO and RLCSD use the same definitions of correct and incorrect sibling evidence. VERPO also evaluates q⁰ for its direction construction.

Locations in the revised manuscript

  • Section 3.4, “Fix, Contrastive, and Fisher Evidence Contrast,” page 5, clarifies the verified-correct and incorrect sibling contexts under “Evidence Contexts” and defines q⁰, q⁺, and q⁻. “Endpoint Comparisons” specifies the RLCSD contrast q⁺ − q⁻.

  • Section 3.5, “Two Actor-Update Paths,” page 6, explains how evidence eligibility enters the masked weight and why zero acceptance gives an AM multiplier of one. Algorithm 1 on the same page specifies the masked update.

  • Appendix F.6, Q10, “Does FEC help on a larger backbone?”, page 32 and Table 17, reports the related FEC-versus-CTR comparison with the LW path and controller retained. The supplementary code documents the sibling-selection and missing-evidence rules.

We are very grateful for your constructive feedback and assessment of our manuscript. We would appreciate it if you could examine our replies and reconsider our work based on the clarifications we have provided. Please advise whether our response satisfactorily addresses your concerns or whether additional information would be helpful. Thank you again for dedicating your time to reviewing our paper.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author Rebuttal by Authors

We sincerely thank all reviewers for their time and constructive feedback.

We appreciate QXDZ’s confirmation that our response addressed most of their concerns and thank all reviewers for helping us improve the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.