- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the role of logit standardization in concrete score distillation for autoregressive language models. It argues that, for a fixed candidate window and a shift-invariant score objective, subtracting the logit mean is inert and the operational effect of standardization is a token-wise temperature. Based on this observation, the paper proposes Token-Adaptive Concrete Score Distillation, or TA-CSD, which predicts a student-side temperature from detached teacher and student uncertainty features. Experiments across multiple same-tokenizer teacher-student pairs evaluate calibration, proper scoring rules, token accuracy, and downstream reasoning. Additional cross-tokenizer experiments examine the effect of fixed standardization regimes across several divergence formulations.
Critical strengths
- The paper provides a clear analysis of logit standardization in concrete score distillation. In particular, the authors distinguish the roles of centering and scaling in the shift-invariant score objective and explain that what actually affects the optimization process is token-dependent scaling. This analysis helps to explain why different standardization strategies may exhibit different effects under different objectives and training settings.
- The experiments do not rely solely on a single calibration metric but report calibration, sharpness, token accuracy, and reasoning performance simultaneously. The paper also includes multi-seed results, paired comparisons, matched-learning-rate control, and experiments on the second model family, allowing readers to observe more comprehensively the benefits and costs brought by the method.
Critical weaknesses
- The paper proposes a bilevel calibration objective, but the TA-CSD used in the main experiment employs a jointly trained gate based on detached features. The relationship between the optimization goals and the actual training process of the two has not been fully explored, which may make it unclear to readers whether the bilevel formulation is the main part of the method, a theoretical explanation, or a future extension direction.
- The paper uses effective-learning-rate to explain the impact of scaling on the training dynamics, but this metric is mainly based on the proportion of logit gradients. For training processes using adaptive optimizers such as AdamW, the logit-gradient ratio is not necessarily equivalent to the actual parameter update magnitude. Therefore, related conclusions should be further explained in conjunction with parameter-update statistics or normalized update norms.
- Current experiments have shown that the method's effectiveness varies with model settings and tokenizer configurations, but the direct verification of TA-CSD in cross-tokenizer or more significant distribution variation scenarios is still limited. Supplementing these settings can help illustrate whether the learned token-specific scale has stable transferability, rather than relying mainly on specific model combinations.
Questions for authors
- Can the author more clearly distinguish the positioning of the jointly trained gate and the bilevel calibration objective? For example, what is the actual optimization target in the main experiment, and is the bilevel formulation used for diagnosis, theoretical explanation, or future extension?
- Could the authors evaluate TA-CSD itself, rather than only fixed standardization variants, on at least one cross-tokenizer setting? Reporting calibration, Brier score, token accuracy, and downstream reasoning together would clarify whether the learned gate transfers beyond same-tokenizer distillation.
2 comments
To author
Thank you for your clarification. Most of my concerns have been addressed. I will maintain the 3 score.
Response to Reviewer QfcA
We thank the reviewer for their careful reading and the positive assessment. All changes are colored in blue in the revised PDF, so below we point to them rather than repeat them. "New" marks results produced for this response.
W1 / Q1. Is the bilevel objective the method, a theoretical explanation, or a future extension? (new) The submission was unclear here. The revision fixes this in the abstract, §3.4, Tables 3–4 and Appendix G.
W2. The logit-gradient ratio is not the parameter update under AdamW. (new) We agree, and measuring the update changed our interpretation (§5.1, new Appendix B, Table 6):
W3 / Q2. TA-CSD on a cross-tokenizer pair. (new) We ran it once, and the result is negative (App. C, Table 8).