- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies recoverability in on-policy distillation. It argues that students with the same weights may have different future training dynamics due to different hidden states, such as optimizer states. The paper provides theoretical and experimental results to support its claims.
Critical strengths
- The paper investigates an interesting question about OPD.
- The paper provides theoretical analysis and controlled experiments on LLMs.
Critical weaknesses
- Limited theoretical novelty.
The main theoretical results largely formalize general properties of stateful optimization or follow directly from their definitions. In particular, the finding that identical weights with different optimizer states can lead to different future outcomes is an expected consequence of momentum-based optimization.
- Weak connection to OPD-specific mechanisms.
The paper aims to study recoverability in OPD, but much of the analysis concerns general properties of stateful optimization rather than mechanisms specific to OPD. Theorem 3 provides a more direct connection to on-policy dynamics, but the result is relatively simple and relies on strong assumptions about competing updates. It is unclear whether these assumptions hold in actual OPD settings.
- Limited practical significance of the experimental findings.
The experiments mainly confirm expected results. The paper fails to identify a new OPD-specific mechanism or provide actionable insights into OPD, such as predicting which states are more recoverable without conducting additional evaluations.
Questions for authors
-
Which conclusions in the paper would not hold outside OPD?
-
Why is the competing update a reasonable assumption in the OPD setting, and what does it correspond to in practice?
3 comments
Thank you for the detailed rebuttal and clarifications. However, my main concerns remain unresolved.
The theoretical contributions are still limited, as most results are general properties of stateful optimization or straightforward consequences of the definitions. The OPD-specific analysis relies on strong assumptions whose relevance to practical OPD training remains unclear.
Moreover, I do not find the empirical results particularly compelling. The experiments largely confirm expected phenomena, without revealing a new OPD-specific mechanism or providing actionable insights for improving OPD. Overall, I find the paper lacking in both theoretical novelty and empirical significance.
I therefore maintain my original rating.
Thank you for the follow-up. We clarify the OPD-specific mechanism, why the competing update is meaningful, and what the results can be used for.
1. What goes beyond general optimizer memory?
In OPD, the student's own policy decides when corrective teacher feedback becomes available. Theorem 3 makes this explicit: an H-step recovery opportunity occurs with probability p^H. For 0 < λ < λ_max, this student-dependent access, combined with a competing update, creates a bistable recovery barrier.
Replacing p^H with a fixed exposure rate c removes the barrier. The mean-field drift becomes ṗ = p(1−p)[c(q−p)−λp], which has a single stable interior root for every λ (new Remark after Theorem 3; derivation in App. A.5). The mechanism is therefore the feedback loop between student behavior, access to correction, and learning, not optimizer memory alone.
2. Why is the competing update a reasonable assumption?
The competing update has a concrete loss-based interpretation. A zero-target binary cross-entropy term contributes −λp to the logit update (App. A.5, Eqs. 18–20). It models opposing training pressure, such as updates on other prompts through shared parameters or regularization toward a reference model. Such pressure need not wait for the recovery-relevant state to be visited, whereas corrective supervision is gated by the student's own p^H visitation. This asymmetry produces the bistable regime of Theorem 3, and λ = 0 recovers the smooth regime as a special case.
3. What can the results be used for?
Checkpoint decisions need more than current outputs. Equalizing the full state removes recovery discrepancies in all 60 control cells, while matching seven observable families leaves them in all 30 cells (App. B.6, Table 7). Under ordinary nonzero-rate training, optimizer cross-swaps move recovery more than residual weight differences (Tables 1–2).
Optimizer state also improves recovery prediction. For the same checkpoints, adding the optimizer state to the exact weights lowers the squared error of predicting recovery on held-out future tapes by 50–91% at B = 1, in all three settings.
These results support three practical rules: keep optimizer state when continuing OPD from a checkpoint, test whether a monitor is sufficient before using it for recovery decisions, and spend direct queries where natural access is rare and teacher feedback is reliable (Theorem 4).
We sincerely thank you for finding the question interesting and for the concrete questions. We address them below.
Q1 (Weakness 1): The main theoretical results largely formalize general properties of stateful optimization or follow directly from their definitions.
That optimizer state shapes future updates is our premise, not our result. The paper answers which summary identifies a finite-budget OPD recovery event, and how large the error is without it. Theorem 1 gives the exact criterion and a sharp minimax bound (half the fiber diameter, attained; App. A.3). Theorem 2 shows that this ambiguity is reachable inside one OPD process, and §5 measures it in a full Transformer and in Qwen3.5 distillation on GSM8K.
Q2 (Question 1 & Weakness 2): Which conclusions in the paper would not hold outside OPD?
In OPD, the student's own policy decides where teacher supervision becomes available. Compared with a fixed, exogenous data stream, this creates three distinctions.
(1) Theorem 3. On-policy, an opportunity needs a length-H student path and occurs with probability p^H. Together with a competing update (0 < λ < λ_max), this creates the bistable barrier. With a fixed data rate c, the drift bracket is c(q−p)−λp, which has one stable root and no barrier.
(2) Theorem 4. In OPD, natural access h is set by the student's own policy, so access and learning form a feedback loop. Direct intervention removes this student-dependent bottleneck. With a fixed stream, access no longer depends on the student, and this loop disappears.
(3) Theorem 2. On-policy, the hidden state changes subsequent updates and therefore the future student-generated data (Eq. 14), so the difference compounds through the data. The natural-continuation separation is in Eq. 16.
Theorem 1 is general by design: it is the measurement tool.
Manuscript revision: We have added this fixed-rate contrast as a Remark after Theorem 3 (p. 6), with the derivation in App. A.5 (p. 17).
Q3 (Question 2): Why is the competing update a reasonable assumption in the OPD setting, and what does it correspond to in practice?
It captures an asymmetry between supervision and opposing pressure. In Theorem 3, it is an update that lowers the recovery branch whether or not the recovery state is visited (§3.5). App. A.5 derives its −λp term exactly from binary cross-entropy toward the zero target. Teacher supervision on the missing behavior waits for the student to visit it, while the opposing pressure does not. In practical OPD, this pressure can represent interference from training on other prompts through shared parameters, or regularization toward a reference model. λ = 0 recovers the smooth regime as a special case.
Q4 (Weakness 3): The paper fails to identify a new OPD-specific mechanism or provide actionable insights into OPD, such as predicting which states are more recoverable without conducting additional evaluations.
When the fiber diameter is positive, no observable-only predictor is correct for every state: its worst-case error is at least half the diameter (Theorem 1). §5 shows this ambiguity is present in practice, and this tells us what to track. Equalizing the full state removes the recovery discrepancy in all 60 cells, while matching on seven observable families leaves a gap in all 30 cells (App. B.6, Table 7). Under ordinary training, the optimizer factor carries more short-budget recovery variation than residual weights (Tables 1–2). The theory also gives actionable rules: look near the regime boundary (Theorem 3), and send direct queries where natural access is rare and the teacher is reliable on those states (Theorem 4; Table 6 reports complementary intervention outcomes).