- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a cross-tokenizer OPD framework named CAST, which regroups aligned tokens into six-token credit chunks and redistributes each chunk's summed teacher-student gap with weights proportional to student entropy and teacher smoothness. With a QWen3-8B teacher, CAST improves the six-benchmark average for both Mistral and Llama students.
Critical strengths
-
This paper is studying a timely and interesting problem since most of the OPD works are constrained by same-tokenizers
-
The paper is well written with a clear motivation. The idea of credit distribution makes sense and is demonstrated by substantial experiments
Critical weaknesses
-
The credit redistribution rule is interesting but it tends to discard token-level teacher signal if there are many one-to-one aligned tokens (79.7% in Mistral and 95.5% in Llama). Currently CAST regroups all aligned tokens into a fixed group of K (K=6), then sum the advantages and redistributes that sum with weights proportional to student entropy times teacher smoothness. However, in this case, the penalty for a one-to-one aligned token could be distributed to other tokens. A variant that keeps per-token advantages for one-to-one tokens and applies CAST only within multi-token alignment chunks would resolve this directly.
-
The external baselines are weak. On Mistral, MinED, DPCA, GRPO, and ALM are all below the SFT initialization. It would be great to add widely-used cross-tokenizer baselines such as GOLD (in TRL) and SimCT. Also, the configurations are not described for the baselines such as the number of rollouts, anchoring, their alignment algorithm (whether it is the same one as CAST), and any other hyperparameter tuning.
-
The benefit of the smoothness term appears to be weak. Relative to uncertainty-only credit, adding smoothness changes essentially only LiveCodeBench (17.0 → 27.7); the other five benchmarks move by −0.20 points on average (Table 4). Used alone, smoothness scores below uniform credit (43.35 against 45.07). Also, the paper never explains why abrupt variation in teacher scores should reduce credit. Such variation plausibly marks the tokens the teacher considers errors, which are the tokens on-policy distillation most needs to correct. It would be great for the authors to add more explanations here.
-
Key hyperparameters are not ablated, and the paper does not say how they were chosen: the credit-chunk length K, the smoothness floor, the anchoring weight (global anchoring is compared only at , the value used for selective anchoring) and the clipping threshold .
Questions for authors
Please refer to the weaknesses
3 comments
Additional ablation of the anchoring penalty cap
dmax-followup
We have now completed this anchoring penalty cap experiments and added the results to Appendix J.4, Table 30. For each setting, we select the evaluated checkpoint with the highest Avg-6. The selected results are:
| Avg-6 | |
|---|---|
| 3 | 45.22 |
| 10 (default) | 47.22 |
| 30 | 44.10 |
The default cap of 10 achieves the highest Avg-6 among the tested values, exceeding caps of 3 and 30 by 2.00 and 3.12 points, respectively. Table 30 also reports the individual scores on all six benchmarks. These results complement the clipping-activity diagnostics in Table 29 and support retaining the default cap.
Please let us know if further experiments or clarifications would help address this concern.
Thank you again for your constructive review. We would appreciate your feedback on the hybrid credit experiment (Appendix F), the added GOLD and SimCT baselines (Table 1), and the hyperparameter studies, including the newly completed clipping-threshold ablation (Appendix J, Table 30).
As the discussion period draws to a close, could you let us know whether these results address your concerns, or if any specific issue remains? We would be glad to provide further clarification.
Response: token-level credit, expanded baselines, and parameter sensitivity
Thank you for your careful review. Your comments gave us a useful perspective on preserving local teacher signals, broadening the baseline comparisons, and examining parameter sensitivity. We address each concern below; references refer to the revised paper.
1. Preserving one-to-one teacher signals. Section 3.2 now states explicitly that redistribution can change individual advantages, including their signs, while preserving the chunk total (Eq. (1)). One-to-one correspondence establishes comparable scores; CAST additionally aggregates neighboring feedback before allocating it across positions.
Appendix F tests a hybrid that retains raw advantages at original 1:1 positions and applies CAST to K=6 groups formed from the remaining aligned positions. Tables 15–16 report its performance and generation diagnostics. At step 150, none of 2,000 MATH-500 responses contained
\boxed, and 43.25% exceeded 0.5 on the defined tail-repetition metric. Over steps 226–250, 99.16% of responses reached the length cap. Thus, this finer-grained feedback variant exhibited both formatting and generation deterioration. With Qwen3-8B and Qwen3-1.7B sharing a tokenizer, CAST reaches 48.68 Avg-6 versus 46.42 for uniform and 45.48 for shuffled credit (Table 3). Shuffling preserves the weights but changes their positions, supporting the value of credit placement even with exact token correspondence.2. External baselines and configurations. We added GOLD from TRL and the authors' released SimCT implementation to Table 1: their Avg-6 scores are 8.62/4.62 on Mistral and 28.91/30.64 on Llama. We had tried SimCT before submission but omitted it after poor results and slow training; the new run with the released code took approximately three days and gave similar results.
Appendix G and Table 17 now specify rollout counts, anchoring, alignment, losses, and optimization settings. GRPO uses eight rollouts, DPCA one, and the other listed methods two. External baselines use no separate reference anchor; CAST and matched uniform credit anchor unaligned positions with λ=0.2. Appendix G describes MinED's edit-distance mapping, ULD's sorted-distribution comparison, ALM's text-chunk likelihood matching, DPCA's decoded-text alignment, GOLD's byte-offset alignment, and SimCT's greedy spans. External methods use a fixed recipe without separate searches. CAST's original checks varied seeds and only two learning rates, 10−6 and 5×10−7, for the multi-seed comparison with ULD; the new sensitivity experiments are reported separately in Appendix J.
3. Smoothness: benefit and rationale. We now state that its gain over uncertainty-only credit is concentrated on LCB (17.0 to 27.7), while the other five scores change by −0.20 points on average (Section 4.3). Smoothness alone scores 43.35 versus uniform's 45.07; combined with uncertainty it reaches 47.22 (Table 4). Its role is to moderate the uncertainty prior.
Section 3.2 distinguishes adjacent teacher-score variation (Eq. (3)) from the same-position teacher–student gap (Eq. (1)). Variation alone does not identify the student's error or correction direction; all aligned gaps still contribute to the chunk total. Reversing smoothness to favor abrupt changes yields 43.93 Avg-6 versus CAST's 47.22 (Table 13). Over steps 51–250, reverse reduces effective credit support from 2.7822 to 2.2714, lowers entropy from 0.4222 to 0.3162, raises repetition from 43.15% to 47.75%, and increases mean gradient norm 5.03-fold (Table 14). Effective support is Neff=1/∑iwi2 (Eq. (13)). Figure 5(c) also shows lower, steadier late-training norms for CAST than uncertainty-only credit. These observations support moderating concentrated credit.
4. Hyperparameter choices and sensitivity. Appendix J reports seed-44 runs selected by highest Avg-6. Our chunk design was informed by DPCA, with K=6 as default; the initial floor 0.1 and cap 10 followed suggestions from Opus 4.8, also disclosed in the AI-use statement.
Please let us know during the discussion if further experiments or clarifications are needed; we would be happy to address them.