- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper utilizes a teacher model that has seen the correct answer to provide word-by-word guidance, and then filters these guidances according to reward direction and update cost to improve the effectiveness and stability of large model inference and tool call training.
Critical strengths
It proposes a complete solution to the specific problem that "the reward for the final answer cannot accurately guide the intermediate steps" and provides multi-model experiments, component ablation, and partial computing power matching verification.
Critical weaknesses
Some lead margins are very small, and the statistical reliability is insufficient. On Qwen3-4B, the average score across five tasks is only 0.31 percentage points higher than the strongest baseline; the main table does not provide the mean and confidence intervals for multiple independent training iterations. Such a gap may stem from random fluctuations and requires validation with multiple random seeds.
Using "highest score during training" makes the model selection process unclear. The main table reports the best checkpoint, but the appendix only explicitly lists the training and test sets, without clearly explaining how the independent validation set is used to select checkpoints. Here, we need to rule out the risk that repeatedly viewing the same evaluation set and taking the highest score might lead to overly optimistic reporting results; currently, we cannot directly conclude that test leakage has occurred.
Word-by-word weights do not equal the true causal contribution of each word. It estimates whether the teacher's suggestion aligns with the current reward update direction, but does not prove that "modifying this word will improve the final accuracy." The authors themselves acknowledge that this is only a local proxy; to prove accurate allocation of credit, experiments involving the deletion, replacement, or intervention of high-weight segments are needed.
Overall, I think this paper is valuable after the revision. Looking forward to authors' rebuttal.
Questions for authors
See weaknesses.
6 comments
Continue
W3. The token-removal intervention
We resumed Llama-3.2-1B Materials from a shared step-100 checkpoint at 20.41% accuracy and trained to step 200. We removed evidence-loss contributions from either the highest-weight 20% of eligible positive-weight tokens per response or an equal number chosen at random. Other weights and the original normalization stayed fixed. All three branches share the same checkpoint and continuation window. Accuracy is reported in percent.
| Continuation branch | Shared step-100 accuracy | Best | Final at step 200 |
|---|---|---|---|
| Full VERPO-LW continuation | 20.41 | 52.39 | 50.48 |
| Remove highest-weight 20% | 20.41 | 30.17 | 20.24 |
| Equal-count random removal | 20.41 | 49.13 | 47.99 |
Targeted removal gives 18.96 points lower Best and 27.75 points lower Final than random removal. Full continuation performs best on both metrics. Figure 11 and Appendix F.6, Q11 report the trajectories and results. The high-weight corrections matter more for later learning in this backbone, task, and shared trajectory. Equal token counts can remove different total weight, so both selection and correction magnitude contribute. Assigning causal credit to individual tokens needs further controls.
Locations in the revised manuscript
-
Section 3.3, “Token-Wise ZPD Controller,” pages 4–5, defines the local acceptance proxy. Section 4.3, “Ablation Study,” reports “Correction-Removal Intervention” and Table 5 on page 9.
-
Appendix F.6, Q11, “Do high-weight tokens matter more for learning?”, starts on page 32 and describes the matched-count deletion experiment. Figure 11 on page 33 reports all three continuation branches.
-
Appendix F.6, Q12, “What could explain the intervention outcome?”, on page 33 discusses the interpretation of the intervention and the influence of correction magnitude.
Hi, we sincerely thank you very much for these constructive comments and evaluation of our manuscript. We would like to kindly ask you to take a look at our responses and reevaluate our work based on our clarifications. Please let us know whether our response addresses your concerns or whether there is any further detail we can provide to help address these concerns.
Thank you again for dedicating your time to reviewing our paper.
Seed Variability, Checkpoint Selection, and Intervention Evidence
Thank you for your valuable comments.
W1. Variability of the 0.31-point Qwen3-4B lead
We added Qwen3-4B runs with seeds 43 and 44 alongside historical seed 42. All five tasks now have three completed seeds, for 15 runs in total. Table 16 and Appendix F.6, Q9 report every seed's result, including the completed Biology seed 44 at 61.62% Best and 56.76% Final. We summarize Best within steps 1–200 using the sample SD and a 95% Student-t confidence interval with two degrees of freedom. Scores are percentages.
Variability is largest on Tool Use. The intervals estimate mean Best under this protocol, assuming independent, approximately normal seed-level scores. With three seeds, individual runs can noticeably affect the interval. All main-text comparisons use Best-checkpoint avg@16 for VERPO-AM, VERPO-LW, and every baseline. The Qwen3-4B comparison with SRPO Lietal,˙2026 below reproduces Table 3. Both five-task averages use Best, and ΔBest gives VERPO-LW minus SRPO in percentage points.
The original averages of 68.57% and 68.26% give the 0.31-point lead. Our repeats cover VERPO-LW, so establishing the lead's significance needs matched baseline repeats. We report it as an observed difference, with its statistical significance unresolved.
Locations in the revised manuscript
Section 4.2, “Main Results,” and Table 3 on page 7 report the original Qwen3-4B five-task comparison and the 0.31-point average margin.
Appendix F.6, Q9, “How much do results vary across training seeds?”, and Table 16 on page 32 report the available seed-level scores, sample standard deviations, and 95% confidence intervals.
Appendix F.6, Q8, “How large is the Best–Final gap?”, on page 32 explains the Best and fixed-step Final reporting used in the same table.
W2. Best selection and evaluation splits
We select the highest evaluation score during training as Best for every method in Table 3. Historical runs used repeated evaluations on the released test split for checkpoint selection, which the runtime labels “validation.” We added step-200 Final scores to show performance retained at the end of training. For avg@16, we average correctness over 16 sampled responses per evaluation prompt and evaluate every five training steps. Appendix H gives the dataset splits and decoding settings. Table 16 pairs Best within steps 1–200 with Final at step 200 for the available Qwen3-4B seeds. The historical seed-42 results below show the gap for each task. Scores are percentages and gaps are percentage points. Appendix F.6, Q8 discusses these differences.
The Best–Final gaps show how much peak performance is retained. Selecting peaks on the test split can make the reported scores optimistic, including the small average lead. The seed intervals describe this same selection protocol. Independent generalization estimates would need separate selection and held-out evaluation splits.
Locations in the revised manuscript
Section 4.2, “Main Results,” and the caption of Table 3 on page 7 identify the main-table metric as Best-checkpoint avg@16.
Appendix F.6, Q8, “How large is the Best–Final gap?”, and Table 16 on page 32 provide the paired Best and fixed step-200 Final values. The table caption restricts the reported seed-level Best to steps 1–200.
Appendix H.1, “Datasets and Data Processing,” pages 34–35, and Table 18 on page 35 list the train/test splits. Appendix H.3, “Shared Training Setup,” and Table 19 on page 35 give the evaluation sampling and decoding settings.