- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
TCAD uses relative, length-normalized teacher likelihood to weight student-generated trajectories during distillation. It estimates forward KL, reverse KL and JSD on a compact union of teacher and student token supports, with a residual tail accounting for omitted probability mass. An analysis describes the compact objective’s tail approximation and separates loss-value unbiasedness from gradient correctness. Ablations examine trajectory weighting and sample counts. Experiments use within-family teacher/student pairs from Qwen dense and DeepSeek MoE, with three independent training seeds and checkpoints fixed at 500 updates. For FKL, the four-domain macro average rises from 52.23 to 53.92 and from 37.08 to 38.68 in those two families.
Critical strengths
S1. The fixed-support ablation isolates the trajectory weights: with the compact support unchanged, removing them lowers the Qwen macro average from 53.92 to 52.47. This tells us more than the full-method comparison alone, because it ties a specific part of the gain to weighting in this recipe.
S2. The analysis makes clear what the compact estimate guarantees. Compact KL is a lower bound with an explicit conditional-tail gap, and the paper distinguishes an unbiased loss value from an unbiased parameter gradient. The exact-enumeration and sample-count controls are also useful: closer local gradients do not consistently translate into better final scores.
Critical weaknesses
W1. The RKL and JSD results need a more explicit account of the training update. The paper already distinguishes value unbiasedness from gradient correctness, and the derivation includes the student-proposal term. However, the actual surrogate losses and detach choices are not specified in the algorithm. Since the proposal depends on the student for these objectives, those details determine which parameter gradient is applied. Teacher-sampled FKL at fixed support avoids this specific dependence.
W2. The significance claimed for the main results cannot be checked from the reported summaries. The paper states three independent training seeds, greedy decoding and benchmark-level paired tests, and the macro means can be recalculated, but the per-seed benchmark vectors and their aggregation into the tests are not supplied. The item counts of 1,773 for HumanEval, 1,192 for GPQA and 480 per AIME year also do not say which variants are included or whether the units are unique questions or repetitions.
Questions for authors
Q1. Could you state the surrogate losses actually used for RKL and JSD, including how the student-proposal gradient is handled and which terms are detached? On a small fixed support, how closely do the sample-average gradients match the exact-enumeration gradients for these two objectives?
Q2. What do the benchmark item counts include, that is, which variants were evaluated and how many items are unique questions versus repetitions? Could you also share the per-seed benchmark scores and the exact vectors and aggregation that entered the paired tests?
7 comments
Response to Reviewer ZuPL (Part 2)
To Q1: Can you quantify the difference between sample-averaged RKL/JSD gradients and exact enumeration gradients?
For a fixed compact support, exact enumeration computes the exact gradient over all compact events, whereas Monte Carlo approximates it by averaging gradient contributions from samples drawn with replacement from the student distribution.
For RKL, define the per-sample gradient contribution as:
\left( \log\frac{q(a)}{p(a)}+1 \right) \nabla_\theta\log q(a).$$ The Monte Carlo estimator is: $$\widehat g_{\mathrm{RKL}} = \frac1M\sum_{m=1}^{M}h_R(a_m), \qquad a_m\sim q.$$ Consequently: $$\mathbb E[\widehat g_{\mathrm{RKL}}] = \sum_{a\in\mathcal A_K}q(a)h_R(a) = g_{\mathrm{RKL}}^{\mathrm{Exact}}.$$ Similarly, for JSD, define: $$h_J(a)= \frac12 \log\frac{q(a)}{m(a)} \nabla_\theta\log q(a).$$ The Monte Carlo estimator is: $$\widehat g_{\mathrm{JSD}} = \frac1M\sum_{m=1}^{M}h_J(b_m), \qquad b_m\sim q.$$ Its expectation is: $$\mathbb E[\widehat g_{\mathrm{JSD}}] = g_{\mathrm{JSD}}^{\mathrm{Exact}}.$$ Thus, both Monte Carlo gradient estimators are unbiased with respect to their corresponding exact compact divergence gradients. With finite sampling, individual estimates may still differ from the exact gradient due to sampling variance: $$\operatorname{Cov}(\widehat g) = \frac{1}{M}\operatorname{Cov}(h).$$ Therefore, the estimator variance decreases proportionally to $1/M$, while its standard deviation decreases proportionally to $1/\sqrt M$. The finite-sample discrepancy arises from Monte Carlo sampling variance, rather than from omitting the gradient contribution induced by the student-dependent proposal. The gradient cosine similarity reported in Appendix D.3 evaluates local gradient alignment specifically for FKL. Since RKL and JSD use different proposal distributions and score-function estimators, this diagnostic cannot be interpreted as an empirical gradient error or cosine similarity measurement for RKL/JSD. Our theoretical guarantee is gradient unbiasedness under a fixed generation state and compact support, rather than exact agreement between every finite-sample estimate and its exact enumeration counterpart. -------------------------------------------------------------------------------------------------------------------- ### **To Q2: Clarification of benchmark item counts, three training seeds, and the inputs to paired tests.** For each method, we average benchmark scores across three independent training seeds. The three seeds assess training randomness, whereas paired t-tests are conducted over the same 15 benchmarks, rather than treating the 45 seed-benchmark combinations as independent observations. The evaluation protocols for AIME, GPQA, and HumanEval are detailed in our response to W2 above. Briefly, AIME uses 16 independent generations per question at $T=0.7$, GPQA uses the unweighted average of the Extended, Main, and Diamond subset accuracies, and HumanEval uses greedy decoding over its 164 unique programming problems. The reported values for HumanEval and GPQA represent evaluation records rather than unique questions. Specifically, we evaluate the Extended, Main, and Diamond subsets separately and compute the final GPQA score as the unweighted arithmetic mean of their individual accuracies. The three subsets contain 546, 448, and 198 evaluation records, respectively, totaling 1,192 records across the subsets rather than 1,192 unique questions. The six two-sided paired t-test results reported above are based on unrounded benchmark-level mean scores, with paired comparisons across the same 15 benchmarks. All six comparisons satisfy $p<0.01$. We will revise Appendix C.1 and Table 6 to clarify the benchmark-specific decoding configurations, unique question counts, and evaluation records, as well as the statistical aggregation procedure. -------------------------------------------------------------------------------------------------------------------- We sincerely appreciate the reviewer's constructive feedback and the opportunity to clarify these aspects. We hope that our responses adequately address the concerns regarding the theoretical properties and practical implementation of our Monte Carlo gradient estimators, as well as the evaluation protocol. **We hope that these clarifications resolve the main concerns and encourage the reviewer to consider a more positive assessment of our work.**Thank you for your detailed explanation of the RKL/JSD surrogate model and the separation operation, which addresses my concerns at the formulation level. However, two questions I raised remain missing:
- an empirical RKL/JSD comparison of sample-averaged gradients versus precisely enumerated gradients;
- benchmark scores for each seed and unrounded paired test vectors.
The p-values for FKL cosine diagnostics and reporting answer different questions. Could you provide these checks and inputs?
Ablation experiments showed that teacher confidence weighting played a beneficial role in this formulation (accuracy of Qwen3-4B increased from 52.47% to 53.92%), but this is not a direct comparison with standard OPD.
I still consider this a practical supplement to the established refinement objectives, and therefore these clarifications improve the technical description, but also leave me with reservations about its suitability. Furthermore, I would like the authors to submit an updated PDF to acknowledge your revisions.
Response to Reviewer ZuPL (Part 1)
Thank you for your careful assessment and constructive feedback. Below, we provide point-to-point responses to your questions and the weaknesses you identified.
To W1: Clarification of the RKL/JSD training updates, surrogate objectives, and detach operations.
We clarify the actual gradient computation for RKL and JSD, distinguishing Monte Carlo value estimation from parameter-gradient estimation.
For a fixed generation state and compact support, let:
q(a)=\bar p_\theta(a\mid s_t),\qquad m(a)=\frac{p(a)+q(a)}{2}.$$ Here, $p$ is independent of $\theta$, while $q$ depends on $\theta$. All derivations assume fixed support without differentiating through discrete Top-K selection. We denote stop-gradient by $\operatorname{sg}[\cdot]$. **1. Reverse KL (RKL)** The RKL objective and Monte Carlo value estimator are: $$D_{\mathrm{RKL}}(q\|p) = \sum_{a\in\mathcal A_K}q(a)\log\frac{q(a)}{p(a)}, \qquad a_m\sim\operatorname{sg}[q],$$ $$\widehat D_{\mathrm{RKL}} = \frac{1}{M}\sum_{m=1}^{M} \log\frac{q(a_m)}{p(a_m)}.$$ Accounting for the student-dependent proposal, the full gradient and corresponding surrogate are: $$\nabla_\theta D_{\mathrm{RKL}} = \mathbb E_{a\sim q} \left[ \left(\log\frac{q(a)}{p(a)}+1\right) \nabla_\theta\log q(a) \right],$$ $$\widetilde L_{\mathrm{RKL}} = \frac{1}{M}\sum_{m=1}^{M} \operatorname{sg}\left[ \log\frac{q(a_m)}{p(a_m)}+1 \right]\log q(a_m).$$ **2. Jensen–Shannon Divergence (JSD)** The JSD objective and Monte Carlo value estimator are: $$D_{\mathrm{JSD}}(p,q) = \frac12D_{\mathrm{KL}}(p\|m) + \frac12D_{\mathrm{KL}}(q\|m),$$ $$\widehat D_{\mathrm{JSD}} = \frac{1}{2M}\sum_{m=1}^{M} \left[ \log\frac{p(a_m)}{m(a_m)} + \log\frac{q(b_m)}{m(b_m)} \right], \qquad a_m\sim p,\quad b_m\sim q.$$ Since $2m=p+q$, the mixture-derivative terms cancel, yielding: $$\nabla_\theta D_{\mathrm{JSD}} = \frac12\mathbb E_{a\sim q} \left[ \log\frac{q(a)}{m(a)} \nabla_\theta\log q(a) \right],$$ $$\widetilde L_{\mathrm{JSD}} = \frac{1}{2M}\sum_{m=1}^{M} \operatorname{sg}\left[ \log\frac{q(b_m)}{m(b_m)} \right]\log q(b_m).$$ **3. Actual Implementation** Both objectives combine the original Monte Carlo value with the corresponding surrogate: $$L_{\mathrm{RKL}} = \operatorname{sg}[\widehat D_{\mathrm{RKL}}] + \widetilde L_{\mathrm{RKL}} - \operatorname{sg}[\widetilde L_{\mathrm{RKL}}],$$ $$L_{\mathrm{JSD}} = \operatorname{sg}[\widehat D_{\mathrm{JSD}}] + \widetilde L_{\mathrm{JSD}} - \operatorname{sg}[\widetilde L_{\mathrm{JSD}}].$$ This preserves the original divergence value in the forward pass and uses the surrogate gradient during backpropagation. Our implementation follows this formulation: sampled indices are non-differentiable, the student proposal and score-function coefficients are detached, and gradients flow through student log-probabilities. The teacher remains frozen, while the student-side tail-probability gradient is retained. Under a fixed generation state and compact support, these estimators are unbiased with respect to the corresponding exact compact-divergence gradients. This guarantee does not extend to discrete support selection or on-policy rollout sampling. --- ### **To W2: Statistical significance, three training seeds, and benchmark-level paired tests.** For each method, benchmark scores are averaged across three independently trained checkpoints (macro-average standard deviation below 0.5 percentage points). Two-sided paired t-tests are then conducted across the 15 benchmark-level mean scores, rather than treating the 45 seed-benchmark combinations as independent observations. For evaluation, AIME uses 16 generations per question ($T=0.7$), with individual responses scored and averaged; GPQA uses the unweighted mean of Extended, Main, and Diamond accuracies; and HumanEval uses greedy decoding over 164 unique problems. The original counts of 1,773 for HumanEval and 1,192 for GPQA represent evaluation records rather than unique questions. We will clarify these distinctions in Table 6. Based on the unrounded benchmark-level mean scores, all six two-sided paired t-tests are significant at $p<0.01$: | Model | Objective | Mean Improvement | Paired t-test p-value | |---|---|---:|---:| | Qwen | FKL | +1.69 | $6.85\times10^{-8}$ | | Qwen | RKL | +2.93 | $1.67\times10^{-5}$ | | Qwen | JSD | +1.69 | $4.93\times10^{-8}$ | | DeepSeek | FKL | +1.59 | $4.96\times10^{-5}$ | | DeepSeek | RKL | +1.49 | $8.89\times10^{-3}$ | | DeepSeek | JSD | +1.54 | $1.24\times10^{-4}$ | All p-values use unrounded benchmark-level mean scores and paired comparisons across the same 15 benchmarks. Finally, Appendix C.1 originally described all benchmarks as using deterministic greedy decoding, which does not reflect AIME's actual sampling protocol. We will revise it in the camera-ready version to specify AIME's setting and distinguish it from the other benchmark configurations.