acceptodds
Under review as a conference paper at ICLR 2027

CAST: Credit Alignment for Cross-Tokenizer On-Policy Distillation

Abstract

Cross-tokenizer on-policy distillation (OPD) requires aligning teacher and student sequences before their scores can be compared. Decoded-text alignment provides such correspondence at the chunk level, but leaves a distinct problem unresolved: credit alignment. A chunk-level signal specifies how much a span should change, but not how that signal should be distributed across student tokens, while tokens without teacher correspondence remain unconstrained. We propose CAST (Chunk-aware credit Assignment with Selective Token Anchoring), a shift-or-stay framework that addresses both issues. For aligned chunks, CAST redistributes each chunk's advantage across student tokens using predictive uncertainty and local teacher-signal smoothness, emphasizing learning opportunities supported by locally consistent teacher feedback while preserving the chunk's total advantage. The resulting weights exactly solve a KL-regularized credit-allocation problem. For unaligned positions, CAST selectively anchors the student to its initial policy through a KL penalty, limiting drift where teacher supervision is unavailable. Because credit assignment is separated from chunk construction, CAST also applies to punctuation-defined or fixed-length chunks under shared tokenization. Across six mathematics, science, and code benchmarks, CAST improves the average over the strongest evaluated baseline by and points for cross-tokenizer transfer from Qwen3-8B to Mistral-7B and Llama-3.1-8B, respectively, and by points for shared-tokenizer transfer to Qwen3-1.7B.

Primary area:
foundation or frontier models, including LLMs
Keywords:
cross-tokenizer, online policy distillation, credit assignment, selective token anchoring

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_H6U8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a cross-tokenizer OPD framework named CAST, which regroups aligned tokens into six-token credit chunks and redistributes each chunk's summed teacher-student gap with weights proportional to student entropy and teacher smoothness. With a QWen3-8B teacher, CAST improves the six-benchmark average for both Mistral and Llama students.

Critical strengths
  1. This paper is studying a timely and interesting problem since most of the OPD works are constrained by same-tokenizers

  2. The paper is well written with a clear motivation. The idea of credit distribution makes sense and is demonstrated by substantial experiments

Critical weaknesses
  1. The credit redistribution rule is interesting but it tends to discard token-level teacher signal if there are many one-to-one aligned tokens (79.7% in Mistral and 95.5% in Llama). Currently CAST regroups all aligned tokens into a fixed group of K (K=6), then sum the advantages and redistributes that sum with weights proportional to student entropy times teacher smoothness. However, in this case, the penalty for a one-to-one aligned token could be distributed to other tokens. A variant that keeps per-token advantages for one-to-one tokens and applies CAST only within multi-token alignment chunks would resolve this directly.

  2. The external baselines are weak. On Mistral, MinED, DPCA, GRPO, and ALM are all below the SFT initialization. It would be great to add widely-used cross-tokenizer baselines such as GOLD (in TRL) and SimCT. Also, the configurations are not described for the baselines such as the number of rollouts, anchoring, their alignment algorithm (whether it is the same one as CAST), and any other hyperparameter tuning.

  3. The benefit of the smoothness term appears to be weak. Relative to uncertainty-only credit, adding smoothness changes essentially only LiveCodeBench (17.0 → 27.7); the other five benchmarks move by −0.20 points on average (Table 4). Used alone, smoothness scores below uniform credit (43.35 against 45.07). Also, the paper never explains why abrupt variation in teacher scores should reduce credit. Such variation plausibly marks the tokens the teacher considers errors, which are the tokens on-policy distillation most needs to correct. It would be great for the authors to add more explanations here.

  4. Key hyperparameters are not ablated, and the paper does not say how they were chosen: the credit-chunk length K, the smoothness floor, the anchoring weight λKL\lambda_{KL} (global anchoring is compared only at λKL=0.2\lambda_{KL}= 0.2, the value used for selective anchoring) and the clipping threshold dmaxd_{max}.

Questions for authors

Please refer to the weaknesses

3 comments
AuthorsOpenReview ↗

Response: token-level credit, expanded baselines, and parameter sensitivity

Thank you for your careful review. Your comments gave us a useful perspective on preserving local teacher signals, broadening the baseline comparisons, and examining parameter sensitivity. We address each concern below; references refer to the revised paper.

1. Preserving one-to-one teacher signals. Section 3.2 now states explicitly that redistribution can change individual advantages, including their signs, while preserving the chunk total (Eq. (1)). One-to-one correspondence establishes comparable scores; CAST additionally aggregates neighboring feedback before allocating it across positions.

Appendix F tests a hybrid that retains raw advantages at original 1:1 positions and applies CAST to K=6K=6 groups formed from the remaining aligned positions. Tables 15–16 report its performance and generation diagnostics. At step 150, none of 2,000 MATH-500 responses contained \boxed, and 43.25% exceeded 0.5 on the defined tail-repetition metric. Over steps 226–250, 99.16% of responses reached the length cap. Thus, this finer-grained feedback variant exhibited both formatting and generation deterioration. With Qwen3-8B and Qwen3-1.7B sharing a tokenizer, CAST reaches 48.68 Avg-6 versus 46.42 for uniform and 45.48 for shuffled credit (Table 3). Shuffling preserves the weights but changes their positions, supporting the value of credit placement even with exact token correspondence.

2. External baselines and configurations. We added GOLD from TRL and the authors' released SimCT implementation to Table 1: their Avg-6 scores are 8.62/4.62 on Mistral and 28.91/30.64 on Llama. We had tried SimCT before submission but omitted it after poor results and slow training; the new run with the released code took approximately three days and gave similar results.

Appendix G and Table 17 now specify rollout counts, anchoring, alignment, losses, and optimization settings. GRPO uses eight rollouts, DPCA one, and the other listed methods two. External baselines use no separate reference anchor; CAST and matched uniform credit anchor unaligned positions with λ=0.2\lambda=0.2. Appendix G describes MinED's edit-distance mapping, ULD's sorted-distribution comparison, ALM's text-chunk likelihood matching, DPCA's decoded-text alignment, GOLD's byte-offset alignment, and SimCT's greedy spans. External methods use a fixed recipe without separate searches. CAST's original checks varied seeds and only two learning rates, 10−610^{-6} and 5×10−75\times10^{-7}, for the multi-seed comparison with ULD; the new sensitivity experiments are reported separately in Appendix J.

3. Smoothness: benefit and rationale. We now state that its gain over uncertainty-only credit is concentrated on LCB (17.0 to 27.7), while the other five scores change by −0.20 points on average (Section 4.3). Smoothness alone scores 43.35 versus uniform's 45.07; combined with uncertainty it reaches 47.22 (Table 4). Its role is to moderate the uncertainty prior.

Section 3.2 distinguishes adjacent teacher-score variation (Eq. (3)) from the same-position teacher–student gap (Eq. (1)). Variation alone does not identify the student's error or correction direction; all aligned gaps still contribute to the chunk total. Reversing smoothness to favor abrupt changes yields 43.93 Avg-6 versus CAST's 47.22 (Table 13). Over steps 51–250, reverse reduces effective credit support from 2.7822 to 2.2714, lowers entropy from 0.4222 to 0.3162, raises repetition from 43.15% to 47.75%, and increases mean gradient norm 5.03-fold (Table 14). Effective support is Neff=1/∑iwi2N_{\mathrm{eff}}=1/\sum_i w_i^2 (Eq. (13)). Figure 5(c) also shows lower, steadier late-training norms for CAST than uncertainty-only credit. These observations support moderating concentrated credit.

4. Hyperparameter choices and sensitivity. Appendix J reports seed-44 runs selected by highest Avg-6. Our chunk design was informed by DPCA, with K=6K=6 as default; the initial floor 0.1 and cap 10 followed suggestions from Opus 4.8, also disclosed in the AI-use statement.

  • Table 27: K=3,6,12K=3,6,12 gives Avg-6 of 46.76, 47.22, 20.07; floors 0.05, 0.1, 0.2 give 47.14, 47.22, 45.21.
  • Table 28: selective weights 0.05, 0.2, 0.4 give 47.23, 47.22, 44.70; global weights 0.05, 0.1, 0.2 give 47.16, 45.94, 44.30. The original comparison used 0.2; lowering global anchoring to 0.05 largely closes the gap, leaving 0.07 points at that shared weight.
  • Table 29: dmax⁡=10d_{\max}=10 caps the penalty in Eq. (9). Across seven runs and 1,750 updates, 0.0330%–0.2283% of unaligned tokens are clipped, reducing total penalty by 4.80%–16.94%. Given the low clipping frequency and limited revision time, we did not run a dedicated cap sweep.

Please let us know during the discussion if further experiments or clarifications are needed; we would be happy to address them.

AuthorsOpenReview ↗

Additional ablation of the anchoring penalty cap

dmax-followup

We have now completed this anchoring penalty cap experiments and added the results to Appendix J.4, Table 30. For each setting, we select the evaluated checkpoint with the highest Avg-6. The selected results are:

dmax⁡d_{\max}Avg-6
345.22
10 (default)47.22
3044.10

The default cap of 10 achieves the highest Avg-6 among the tested values, exceeding caps of 3 and 30 by 2.00 and 3.12 points, respectively. Table 30 also reports the individual scores on all six benchmarks. These results complement the clipping-activity diagnostics in Table 29 and support retaining the default cap.

Please let us know if further experiments or clarifications would help address this concern.

Thank you again for your constructive review. We would appreciate your feedback on the hybrid credit experiment (Appendix F), the added GOLD and SimCT baselines (Table 1), and the hyperparameter studies, including the newly completed clipping-threshold ablation (Appendix J, Table 30).

As the discussion period draws to a close, could you let us know whether these results address your concerns, or if any specific issue remains? We would be glad to provide further clarification.

Reviewer_5zAHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CAST, a token-level credit assignment method for on-policy distillation under heterogeneous and shared tokenization. For aligned positions, the method redistributes each chunk’s teacher–student log-probability gap across student tokens, with weights based on the product of student predictive entropy and inverse local variation of teacher scores. For unaligned positions, a KL penalty term anchors the student to its initial policy. Extensive experimental results show improvements over the compared baselines.

Critical strengths
  1. The paper addressed cross-tokenizer distillation, which is a well-known and important problem in the space.
  2. The experimental evaluation is comprehensive.
Critical weaknesses
  1. The proposed method appears ad hoc, as its key design choices are not sufficiently motivated or justified.
  2. The presentation is not sufficiently clear. This can be addressed by expanding the preliminary section to include more formulation setup for on-policy distillation before diving into advantage redistribution.
Questions for authors
  1. In the proposed method, less credit is given if the teacher score varies abruptly. What's the rationale of that?
  2. Why are Eqs. (4) and (5) constructed in the proposed way instead of simpler alternatives?
  3. In Eq. (6), why does minimizing the proposed objective indicate that the weights form a good reallocation distribution?
6 comments
AuthorsOpenReview ↗

Clarifying CAST's formulation, allocation rationale, and empirical support

Thank you for your careful review. Your questions gave us a useful perspective on presenting CAST: explaining the OPD setup first, then connecting each allocation choice to its mathematical properties and experimental evidence. We have revised Sections 3.1-3.3 and expanded the supporting analysis.

1. Motivation and empirical support (W1). CAST separates two decisions: allocating a chunk's feedback across aligned positions while preserving its total advantage (Section 3.2, Eq. (1)), and penalizing departures from the initial policy at unaligned positions (Section 3.3, Eqs. (8)-(9)). Student entropy supplies an allocation prior; smoothness moderates it using local teacher-score variation. With alignment, selective anchoring, and two rollouts fixed, Table 4 gives 47.22 Avg-6 for CAST, versus 45.07 for uniform credit and 44.17 for shuffled credit. Shuffling preserves the weights but changes their token assignments. The same ordering holds under shared tokenization: 48.68, 46.42, and 45.48, respectively (Table 3). These controls support the importance of both the allocation and its placement.

2. OPD preliminaries (W2). Section 3.1 now defines the fixed teacher, prompt distribution, rollout-policy snapshot, sampled response, and prefix. It introduces the sampled-token advantage, likelihood ratio, and stop-gradient update surrogate, together with the conditional reverse-KL relation at a fixed prefix. It then explains why heterogeneous tokenization requires alignment, distinguishing alignment chunks from credit chunks. This setup leads into the chunk total in Eq. (1), CAST weights in Eq. (5), and clipped update in Eq. (10); Appendix A.1, Eq. (12), specifies teacher-score projection.

3. Why reduce credit at abrupt teacher-score changes? (Q1). Section 3.2 now distinguishes variation across neighboring teacher scores from the teacher-student gap at a given position. A large variation alone does not specify the correction needed at that position. All aligned gaps still enter the chunk sum in Eq. (1); inverse variation adjusts where that total is allocated. When high uncertainty and large variation coincide, weighting both positively can concentrate credit further.

We tested this direction explicitly: CAST (reverse smoothness) replaces sis_i by 1/si1/s_i, giving wi∝ui(δi+ϵg)w_i\propto u_i(\delta_i+\epsilon_g). With the same training configuration and only this weighting direction changed, CAST scores 47.22 Avg-6 versus 43.93 for the reverse variant (Appendix E, Table 13). Over steps 51-250, the reverse variant has fewer effective credit tokens (2.2714 vs. 2.7822), lower predictive entropy (0.3162 vs. 0.4222 nats), more repeated 4-grams (47.75% vs. 43.15%), and a larger mean gradient norm (2.77983 vs. 0.55284; Table 14). Equation (13) defines effective support as Neff=1/∑iwi2N_{\mathrm{eff}}=1/\sum_i w_i^2. Figure 5(c) also shows lower, visually steadier late-training gradient norms for CAST than uncertainty-only weighting (Appendix B.4). These observations support smoothness as a moderator of concentrated credit.

4. Why Eqs. (4)-(5)? (Q2). Section 3.2 now explains that si=1/(δi+ϵg)s_i=1/(\delta_i+\epsilon_g) is positive and decreases with local variation; the chunk mean adapts the scale, and the 0.1 floor avoids a singular inverse in flat chunks. Equation (7) bounds the ratio of smoothness factors by Kg+1K_g+1. Multiplication makes smoothness a relative adjustment to uncertainty: the normalized product is invariant to common positive rescaling of either factor within a chunk, whereas a raw sum depends on their relative scales. Table 4 evaluates simpler allocation rules: uniform, uncertainty-only, and smoothness-only score 45.07, 45.60, and 43.35, versus 47.22 for CAST. Section 4.3 states that the gain over uncertainty-only is concentrated on LiveCodeBench.

5. Why the objective in Eq. (6)? (Q3). We now explain its two preferences before Proposition 1. With piu=ui/∑jujp_i^u=u_i/\sum_j u_j and ρi=1+δi/ϵg\rho_i=1+\delta_i/\epsilon_g, KL(w∥pu)\mathrm{KL}(w\|p^u) penalizes departures from the uncertainty prior, while ∑iwilog⁡ρi\sum_i w_i\log\rho_i penalizes allocating mass to larger local variation. Their joint minimizer is wi∗∝piu/ρiw_i^*\propto p_i^u/\rho_i, exactly Eq. (5); Appendix A.2 gives the proof. Thus Eq. (6) provides a precise allocation criterion. Its empirical usefulness is assessed by Table 4 and the shared-rollout gradient diagnostics in Figure 4 (Appendix B.5), where CAST improves gradient SNR and training/held-out direction agreement.

Please let us know if any further experiments or clarifications would help address your concerns; we would be glad to follow up during the discussion.

Thank you again for your careful review. As the discussion period draws to a close, we would appreciate your feedback on the expanded OPD setup and allocation rationale in Sections 3.1–3.2, together with the reverse-smoothness experiment and training diagnostics in Appendix E (Tables 13–14). These additions directly address your questions about the formulation, Eqs. (4)–(6), and the role of teacher-score variation.

Could you let us know whether these revisions address your concerns, or whether any specific issue still requires clarification? We would be glad to respond during the remaining discussion time.

We thank the author for the additional results and clarifications. However, I'm still not clear why larger local variation calls for smaller mass. The authors show that the opposite direction is not ideal, but it doesn't automatically justify the current choice. I hope to get an intuition here. To be more specific, I accept the other design based on student uncertainty, and that's because it bears a clear intuition: we want to find "where the student doesn't understand" to enforce more teacher supervison. Is there such a straightforward rationale for the design based on smoothness/variation as well?

Reviewer_iWMtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CAST addresses how to distribute chunk-level teacher feedback across student tokens in cross-tokenizer OPD, using student uncertainty and local teacher-signal smoothness to assign credit while preserving each chunk’s total advantage. It also uses a selective KL penalty to keep unmatched positions close to the initial student policy, with experiments on six math, science, and code benchmarks showing improvements over the strongest evaluated baselines in both cross-tokenizer and shared-tokenizer settings.

Critical strengths

CAST beats uniform and shuffled credit with the other training settings held fixed: 47.22 vs. 45.07 and 44.17 on Mistral (Table 4). This supports the benefit of assigning credit to the right tokens.

Adding smoothness raises LCB from 17.0 to 27.7 compared with uncertainty-only weighting. The three-seed results show the same benefit, which helps explain why uncertainty alone isn't enough.

The weighting also helps with shared tokenization: Qwen3-1.7B reaches 48.68 vs. 46.42 with uniform credit (Table 3). This suggests the idea is useful beyond handling tokenizer mismatch.

Critical weaknesses

The paper picks checkpoints, and sometimes seeds, based on the six test benchmarks, raising concerns about overfitting to these benchmarks. Selecting models on a separate validation set would help show whether the gains generalize.

Anchoring adds 3.22 average points with two rollouts, but only 0.07 with one (Table 4). Why it helps so much more with two rollouts is still unclear.

App. A.1 doesn't specify when an alignment match is considered too large or unreliable, and the clipping threshold d_max in Eq. (9) is also unspecified. These choices affect which tokens get anchored and how strongly, so they matter for reproducing the method.

Questions for authors

Do the gains over uniform credit and ULD hold when checkpoints are chosen on a separate validation set?

Which unmatched tokens benefit most from anchoring—EOS, formatting tokens, or ordinary text?

Since CAST changes how the learning signal is distributed across tokens, a broader KD discussion would help clarify what is new about this choice. For example, work on loss design, probability-mass allocation, and hard/soft supervision [1–4] provides useful context for explaining what CAST adds and how these approaches might complement each other.

[1] DistiLLM: Towards Streamlined Distillation for Large Language Models. ICML, 2024.

[2] DistiLLM-2: A Contrastive Approach Boosts the Distillation of LLMs. ICML, 2025.

[3] ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via α-β-Divergence. ICML, 2025.

[4] The Bridge-Garden Dilemma in LLM Distillation: Why Mixing Hard and Soft Labels Works. ICML, 2026.

3 comments
AuthorsOpenReview ↗

Response to iWMt: validation, anchoring diagnostics, and KD context

Thank you for the careful review and constructive suggestions. Your comments brought a useful perspective on separating model selection from evaluation, examining where anchoring acts, and situating CAST within the broader KD literature. We have revised the paper and added the following analyses.

1. Checkpoint selection and gains over uniform credit/ULD (W1, Q1). Appendix H now reports six cross-benchmark validation rounds. In each round, one benchmark selects the checkpoint and the other five determine Avg-5; the validation score is excluded. All trained configurations use seed 44 and available checkpoints from steps 150–250, with no seed selection. Tables 19–24 report every rotation, selected step, and benchmark score; Appendix C retains the original selection protocol.

Table 18 shows that CAST exceeds both matched uniform credit and ULD in 5/6 rotations on Mistral, and ULD in 6/6 on Llama. Against uniform credit, the five positive differences are +0.58 to +3.48 points; GPQA selection gives −0.59. Against ULD, Mistral differences are −0.87 to +3.42, with AIME26 the exception; Llama gains are +1.20 to +2.27. Thus, gains persist under most validation choices when the selection benchmark is excluded from the reported average.

2. Why anchoring helps more with two rollouts (W2). Appendix I.2 adds a coverage diagnostic motivated by the interaction in Table 4: anchoring adds 3.22 Avg-6 points at n=2n=2 and 0.07 at n=1n=1. A second trajectory from the same prompt can visit additional contexts without aligned teacher supervision, where Eq. (9) penalizes departures from the initial policy.

We sampled 16 responses for each of 12 problems across mathematics, science, and code. Table 26 compares single trajectories with distinct pairs from one fixed CAST checkpoint. The probability of encountering an unmatched position rises from 4.17% to 8.06% for two complete trajectories. With equal total token budgets, it is 3.75% versus 1.04%; the extra length of the second complete trajectory therefore contributes to the coverage increase. These diagnostics support state coverage as a possible contributor to the rollout–anchoring interaction.

3. Alignment acceptance and clipping (W3). Appendix A.1 now specifies the full rules. A match requires identical, nonempty decoded strings after Unicode NFC normalization and removal of leading whitespace; the student span must not end with the Unicode replacement character (U+FFFD, “�”). Each side must contain at most six tokens under its own tokenizer. Larger matched spans are marked unaligned; there is no additional similarity/confidence threshold. The appendix also specifies unmatched suffixes, teacher truncation, EOS handling, and the distinction between this acceptance limit and credit-regrouping length KK.

Section 3.3, Eq. (9), and Table 9 now state dmax⁡=10d_{\max}=10. Appendix A.1 gives the numerical sequence: clip the log-probability difference to [−20,20][-20,20], compute ez−z−1e^z-z-1, cap each token's penalty at 10, then average over unaligned response positions and multiply by λKL=0.2\lambda_{\mathrm{KL}}=0.2.

4. Which unmatched tokens benefit (Q2). Appendix I.1, Table 25, measures reference-policy KL at identical CAST-generated prefixes. Ordinary content shows the largest reduction: 0.28740 without anchoring to 0.24788 with CAST, a 13.75% decrease, alongside higher predictive entropy (1.5492 to 1.6141 nats). Formatting KL increases and aggregate KL does not decrease. No unmatched EOS occurs in this sample; Appendix A.1 explains how the first terminal token can receive a teacher EOS score.

5. Broader KD context (Q3). Section 2 now cites and discusses all four suggested works: DistiLLM's skew-KL objectives and response reuse, DistiLLM-2's contrastive objective, ABKD's probability-mass allocation, and Bridge–Garden's hard/soft supervision mixtures. CAST allocates a fixed chunk advantage across token positions while preserving its sum (Section 3.2, Eq. (1)); ABKD reallocates mass over output classes, while Bridge–Garden changes supervision mixtures. Section 2 also explains how an alternative objective or supervision mixture could supply chunk advantages for CAST to distribute, with a shared comparison space needed for full-distribution losses across tokenizers.

Please let us know if further experiments or clarifications would help address your concerns; we would welcome the opportunity to respond during the discussion.

Thank you for the detailed responses and revisions. They address most of my concerns, and I will maintain my positive score.

Thank you for reviewing our responses and revised manuscript. We appreciate your constructive feedback and continued positive assessment. Your comments helped us strengthen the checkpoint-selection analysis, anchoring diagnostics, and methodological presentation. Please let us know if any further clarification would be helpful.

Comments on the paper

2 comments
AuthorsOpenReview ↗

summary of revisions and additional experiments

We thank the Program Chairs, Senior Area Chairs, and Area Chairs for their time and effort in coordinating the review process, and all three reviewers for their careful reading and constructive feedback. The reviewers' comments brought new perspectives on separating model selection from evaluation, understanding token-level credit and anchoring, and testing the method against additional baselines and parameter choices. We have revised the manuscript and prepared point-by-point responses. Below we summarize the main changes; all references refer to the revised paper.

1. Checkpoint selection using a separate benchmark. Appendix H rotates each of the six benchmarks into the validation role: checkpoints are selected using that benchmark alone, and Avg-5 is computed on the other five. All trained configurations use seed 44 and available checkpoints from steps 150–250. On Mistral, CAST exceeds matched uniform credit in five of six rotations and ULD in five of six; on Llama, it exceeds ULD in all six. Table 18 summarizes these comparisons; Tables 19–24 provide all selected checkpoints and individual scores.

2. Clearer formulation and design rationale. Section 3.1 now introduces the OPD sampling setup, token advantage, likelihood ratio, reverse-KL relation, and update surrogate before presenting CAST. Section 3.2 explains the inverse-variation factor and normalized product in Eqs. (4)–(5), and the allocation preferences encoded by Eq. (6); Appendix A.2 provides the proof. Appendix A.1 specifies alignment acceptance, unmatched-token handling, and numerical clipping, including dmax⁡=10d_{\max}=10. Section 2 expands the KD discussion to include DistiLLM, DistiLLM-2, ABKD, and Bridge–Garden, clarifying how their objectives and supervision choices relate to CAST's allocation across token positions.

3. Additional tests of credit allocation and smoothness. Appendix E compares CAST with a reverse-smoothness variant that favors abrupt teacher-score changes. CAST reaches 47.22 Avg-6 versus 43.93 for the reverse variant (Table 13). Over steps 51–250, reversing smoothness produces more concentrated credit, lower predictive entropy, more repetition, and a 5.03-fold larger mean gradient norm (Table 14). These diagnostics support smoothness moderating the uncertainty-based allocation. Appendix F also evaluates a hybrid that retains raw advantages at original one-to-one positions and redistributes credit over groups formed from the remaining aligned tokens. Tables 15–16 document its performance, formatting changes, and generation deterioration. These experiments complement the shared-tokenizer uniform/shuffled controls in Table 3.

4. Expanded external baselines and reproducibility details. Table 1 now includes GOLD from TRL and the released SimCT implementation for both students. Appendix G and Table 17 document rollout counts, reference anchoring, alignment and loss formulations, optimization settings, and the tuning scope for the comparisons.

5. Anchoring diagnostics. Appendix I examines reference-policy drift by unmatched-token category and exposure to unmatched positions across trajectories. Table 25 shows the largest reference-KL reduction for ordinary content, at 13.75%. Table 26 compares one trajectory with two distinct trajectories from the same fixed CAST checkpoint and includes an equal-token-budget control. These analyses provide a more detailed account of where anchoring acts and a possible contribution of trajectory coverage to its interaction with rollout count.

6. Hyperparameter sensitivity. Appendix J adds sweeps over credit-chunk length, smoothness floor, and selective/global anchoring weights. Table 27 reports Avg-6 of 46.76, 47.22, and 20.07 for K=3,6,12K=3,6,12. Table 28 shows that lowering global anchoring to λ=0.05\lambda=0.05 largely closes the gap with selective anchoring: 47.16 versus 47.23 at the same weight. Table 29 separately reports clipping activity at the default dmax⁡=10d_{\max}=10 over seven runs and 1,750 updates. The appendix also states how the initial parameter values were chosen.

We appreciate the perspectives that helped improve the paper. Please let us know during the discussion if further experiments or clarifications would help address any remaining questions; we would be happy to follow up.

AuthorsOpenReview ↗

Final summary of revisions and acknowledgments

Dear Area Chair, Senior Area Chairs, Program Chairs, and Reviewers,

Thank you for your time, careful evaluation, and constructive discussion of CAST. Your feedback helped us strengthen the experimental evidence, clarify the method, and better explain its contribution. Below, we summarize the discussion outcomes and the revisions in our updated manuscript.

  1. Reviewer feedback and discussion outcomes
  • Reviewer iWMt: maintained a positive recommendation. The reviewer highlighted the benefits of assigning credit to appropriate tokens and the gains under shared tokenization. After our responses and revisions, they stated that these “address most of my concerns” and confirmed: “I will maintain my positive score.” Our additions cover checkpoint selection, anchoring diagnostics, alignment and clipping details, and broader KD context.
  • Reviewer 5zAH: expressed a more favorable assessment. The reviewer recognized the importance of cross-tokenizer distillation and the comprehensive evaluation. Following the discussion of smoothness, they reported being “more inclined toward acceptance than before” and stated, “I would not oppose its acceptance.” They maintained their current score, with remaining concerns about presentation and novelty. The revised manuscript expands the OPD preliminaries, clarifies CAST's contribution, and includes the mathematical examples discussed in our follow-up.
  • Reviewer H6U8: additional experiments and clarifications provided. In the initial review, the reviewer described the problem as timely and interesting and recognized the motivation and experimental support for credit distribution. In response to their four main concerns, we evaluated a variant preserving one-to-one advantages, added GOLD and SimCT, expanded the smoothness analysis, and completed sensitivity experiments for the requested hyperparameters.
  1. Key revisions in the updated manuscript
  • Formulation, presentation, and positioning. Section 3.1 now introduces the OPD sampling setup, token advantage, reverse-KL relation, and update surrogate before redistribution. Sections 3.2–3.3 explain the allocation rule and selective anchoring; Appendix A specifies alignment acceptance and numerical clipping. Section 2 expands the discussion of DistiLLM, DistiLLM-2, ABKD, and Bridge–Garden. The abstract and introduction clarify the distinction between establishing token correspondence and allocating a chunk's feedback.
  • Checkpoint selection separated from the reported evaluation. Appendix H, Tables 18–24, reports six rotations: one benchmark selects the checkpoint, while the other five determine Avg-5. CAST exceeds both matched uniform credit and ULD in five of six rotations on Mistral, and ULD in all six on Llama.
  • Smoothness rationale and diagnostics. Appendix E, Tables 13–14, compares CAST with reverse smoothness: Avg-6 is 47.22 versus 43.93, alongside broader credit support, higher predictive entropy, less repetition, and a smaller mean gradient norm for CAST. Appendix K, Tables 31–33, adds mathematical examples explaining how student uncertainty can arise from several valid next steps. A teacher may prefer a different wording or order of steps, giving a valid entry low probability followed by a high-probability continuation. In the two primary examples, CAST reduces the entry's share of negative feedback from 17.41% to 9.12% and from 71.42% to 49.42%.
  • Additional comparisons and baseline details. Appendix F, Tables 15–16, reports the one-to-one-preserving hybrid and its performance and generation diagnostics. Table 1 adds GOLD and SimCT for both students. Appendix G, Table 17, documents rollout counts, alignment, anchoring, losses, and optimization settings. These additions complement the shared-tokenizer controls in Table 3, where CAST reaches 48.68 Avg-6 versus 46.42 for uniform and 45.48 for shuffled credit.
  • Anchoring and parameter sensitivity. Appendix I, Tables 25–26, adds unmatched-token category and rollout-coverage analyses. Appendix J, Tables 27–30, reports sensitivity to chunk length, smoothness floor, selective/global anchoring weights, and the clipping cap. The cap comparison gives Avg-6 of 45.22, 47.22, and 44.10 for dmax⁡=3,10,30d_{\max}=3,10,30, respectively. We thank the reviewers for the perspectives that helped improve this work, and the AC, SACs, and PCs for coordinating and assessing the discussion. These revisions clarify CAST's central contribution: assigning a chunk's feedback to individual tokens while preserving its total advantage, and anchoring positions without aligned teacher supervision. The new experiments provide broader support for this design. We remain available to clarify any remaining questions during the subsequent review process, and appreciate your continued consideration.

Yours sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.