- Rating
- 2
- Confidence
- 1
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper separates fixed-prefix propriety, autoregressive composition, and the fidelity of practical OPD updates to an occupancy-weighted objective. It maps coordinate-local rewards to potentials, characterizes the logarithmic composition boundary within separable Bregman divergences, and derives omitted future credit, descent criteria, and circulation tests. Finite trees, projected Qwen3.5 measurements, and matched training examine these distinctions.
Critical strengths
- The paper makes a consequential distinction between a proper conditional objective, composition into the same path-space divergence, and descent under student occupancy. The logarithmic and power constructions demonstrate both nonimplications. Its value extends beyond restating nonconservative distillation updates: it connects bounded rewards and support restriction to a declared objective while acknowledging earlier results.
- The validation separates exact identities from neural observations and includes unfavorable outcomes. In particular, corrected log improves ARC accuracy relative to local log but hurts HellaSwag accuracy. Paired schedules, seed-level intervals, and explicit limits on parameter projections make this a useful investigation rather than an unconditional recommendation to restore future credit.
Critical weaknesses
- The neural fidelity statistic appears to differ from the population quantity that certifies descent. Equation (16) uses expected vectors, whereas Appendix H.2 defines Equation (84) for each prompt trajectory and H.4 then averages within seed. Averaging these ratios need not preserve the sign of the ratio formed from mean vectors. Likewise, averaging per-trajectory wedge norms cannot certify a nonzero expected wedge. Thus the reported negative intervals alone do not establish population ascent or nonintegrability, even within the measured subspace. This affects the central interpretation of Figure 2, beyond the projection limitation already acknowledged.
- The sampling measure is insufficiently specified for the claimed on-policy geometry. Sampling uses temperature 0.8 and top-p 0.95, but the score and divergences are written using qθ. Without identifying these distributions, the score-function correction need not target the stated objective. Furthermore, H.5 reuses initialization prompt trajectories at trained endpoints, while Observation 7 attributes changes to current occupancy. Reusing old response tokens requires an explicit distribution correction; holding samples fixed within a central difference is a separate, valid variance-reduction step.
Questions for authors
- Could the authors provide aggregation pseudocode and results that average the direct field, future-credit vector, and signed wedge matrix before computing fidelity or norms, with trajectory-count convergence and seed-level intervals? Do the reported ascent cases persist?
- Could the authors clarify whether qθ in rollout sampling, log probabilities, divergences, and directional scores includes temperature and nucleus truncation, and explain any correction needed to make these estimators target Equations (6) and (14)?
- Could the authors clarify whether endpoint responses are regenerated at each final adapter or reused from initialization, and provide the occupancy correction if reused?
3 comments
Thanks for the detailed rebuttal and the new experiments. The additional analysis clears up my concern regarding averaging per-trajectory fidelity ratios, and your clarification on the endpoint evaluation makes sense—using fixed trajectories for direct comparison while evaluating the objective on fresh rollouts for each final model is much clearer now.
Just one quick question remaining: were the fidelity estimates in Tables 3 and 4 computed using the score of the actual rollout distribution (including temperature and truncation), as defined in Appendix H.8?
Since the theoretical side of on-policy distillation isn't my main area of expertise, I'd suggest the AC weigh the assessments of the higher-confidence reviewers more heavily on this aspect.
Thank you for your constructive follow-up. We appreciate that the vector-aggregation results and endpoint clarification have addressed your first and third concerns.
To answer your remaining question directly, Tables 3 and 4 use the untempered student score defined in Eq. 78. The underlying directional measurements are held fixed across all three aggregation methods, allowing us to isolate the effect of aggregating vectors before computing fidelity ratios.
For the actual generation law, Appendix H.8 derives the corresponding future-credit score, ∇log q̃_θ (Eq. 89), accounting for temperature and truncation. The matched-law estimator retains the same conditional divergence but replaces the future-credit score with that of the actual rollout distribution.
Thus, Tables 3 and 4 establish the results of the vector-aggregation comparison, while Appendix H.8 specifies the estimator for the generation-law objective. Theorem 4.5 and Propositions 4.8–4.9 establish the theoretical separation and descent criteria independently of these empirical aggregation choices.
We appreciate your careful consideration.
Thank you for the precise questions on population fidelity, the rollout distribution, and endpoint evaluation. We address them below.
Q1 (Q1 & Weakness 1): Could the authors provide aggregation pseudocode and results that average the direct field, future-credit vector, and signed wedge matrix before computing fidelity or norms, with trajectory-count convergence and seed-level intervals? Do the reported ascent cases persist?
We recomputed fidelity by aggregating the projected vectors before forming ratios. We report three estimates: the manuscript's per-trajectory ratio, the ratio formed from within-seed mean vectors, and a cross-fitted ratio of seed-averaged inner products over pairs of distinct trajectories. Cross-fitting removes the within-trajectory product bias from both the numerator and the denominator. All intervals use the seed as the bootstrap unit.
Table 3 (p. 24) reports the three estimates. After vector aggregation, the ARC and HellaSwag estimates remain negative. The cross-fitted intervals exclude zero for ARC at horizon 64 and at every reported HellaSwag horizon, while GSM8K remains at the zero boundary. Table 4 (p. 24) subsamples 2, 4, and 8 trajectories per seed and shows the same pattern. On HellaSwag, the mean-vector interval excludes zero at every count.
For circulation, we first aggregate the signed wedge matrices within each seed and its fixed direction basis, and only then compute norms. Cross-fitted trajectory pairs estimate the squared norm of the expected wedge. Its intervals exclude zero on HellaSwag with eight directions at horizons 8 and 16.
Manuscript revision: App. H.7 (pp. 23–24) now gives the step-by-step aggregation procedure, Table 3, and Table 4.
Q2 (Q2 & Weakness 2): Could the authors clarify whether qθ in rollout sampling, log probabilities, divergences, and directional scores includes temperature and nucleus truncation, and explain any correction needed to make these estimators target Equations (6) and (14)?
Rollouts follow the generation law q̃_θ, with temperature 0.8, top-k 50, and top-p 0.95. The conditional divergences use the untempered student distribution q_θ.
Under this sampling law, the occupancy-weighted objective is
L_B^q̃(θ) = E_{Y∼q̃_θ} Σ_t B(q_θ(·|H_t), p(·|H_t)).
The direct field is obtained by differentiating the conditional divergence at fixed prefixes (Eq. 79). The corresponding future-credit term uses the rollout score s̃_t = ∇log q̃_θ(Y_t|H_t). Where the retained token set N_t is fixed, this score is
s̃_t = (1/τ)(∇z_{Y_t} − Σ_{k∈N_t} q̃_θ(k|H_t)∇z_k).
This gives the matched-law decomposition for the generation distribution.
Manuscript revision: App. H.8 now states the rollout law, the conditional divergence, the objective, and the matched score (Eqs. 88–89, p. 24). App. I lists the full sampling settings (p. 25).
Q3 (Q3 & Weakness 2): Could the authors clarify whether endpoint responses are regenerated at each final adapter or reused from initialization, and provide the occupancy correction if reused?
The endpoint diagnostic in App. H.5 evaluates every trained adapter on the common initialization trajectories. Holding the reference responses and parameter directions fixed allows the local fields to be compared directly across trained states.
Current-occupancy objectives are evaluated separately, on fresh responses sampled from each final adapter (App. H.6).
For current-occupancy estimates from reused responses, the required sequence likelihood ratio is q̃_{θM}(y|x)/q̃_{θ0}(y|x), provided that the reference law covers the target law. Fresh rollouts sample from the trained policy directly.
Manuscript revision: App. H.5 now distinguishes the fixed-reference endpoint diagnostic from current-occupancy evaluation and states the likelihood ratio (p. 23). Observation 7 (p. 9) and App. K.19 (p. 58) describe the endpoint change on the common reference trajectories, and App. K.19 points to the fresh-sequence evaluation for current occupancy.