acceptodds
Under review as a conference paper at ICLR 2027

Same Weights, Different Futures: State Identifiability in On-Policy Distillation

Abstract

On-policy distillation changes not only a student's predictions but also the states on which it receives future supervision. Thus, two students that currently appear identical may still have different probabilities of recovering a missing behavior within a finite query budget. Specifically, we study which representation of training state determines this recoverability. To formalize this, we define budgeted recoverability on a transition-sufficient state and show that an observable summary is sufficient exactly when all full states sharing that summary have the same recovery probability. The resulting observation-fiber diameter gives a sharp minimax lower bound on prediction error from the summary alone. We then construct two positive-probability histories of one finite OPD process that have identical observations but different recovery laws, and show that Adam can preserve such differences in optimizer state even when weights are identical. We further characterize scalar recovery regimes and decompose the value of finite-query intervention into state access and the signed reliability of teacher feedback. Empirically, common-future Transformer experiments realize the exact-weight separation, while optimizer cross-swaps test the same mechanism under ordinary training in a procedural task and Qwen3.5 reasoning on GSM8K. Prospective validation indicates that recovery-gap magnitude transfers more reliably than state ordering. Together, these results show that recoverability is determined by the state consumed by future transitions, not by current weights or predictions alone.

Primary area:
learning theory
Keywords:
on-policy distillation, state identifiability, recoverability, optimizer state

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_r5sJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies recoverability in on-policy distillation. It argues that students with the same weights may have different future training dynamics due to different hidden states, such as optimizer states. The paper provides theoretical and experimental results to support its claims.

Critical strengths
  1. The paper investigates an interesting question about OPD.
  2. The paper provides theoretical analysis and controlled experiments on LLMs.
Critical weaknesses
  1. Limited theoretical novelty.

The main theoretical results largely formalize general properties of stateful optimization or follow directly from their definitions. In particular, the finding that identical weights with different optimizer states can lead to different future outcomes is an expected consequence of momentum-based optimization.

  1. Weak connection to OPD-specific mechanisms.

The paper aims to study recoverability in OPD, but much of the analysis concerns general properties of stateful optimization rather than mechanisms specific to OPD. Theorem 3 provides a more direct connection to on-policy dynamics, but the result is relatively simple and relies on strong assumptions about competing updates. It is unclear whether these assumptions hold in actual OPD settings.

  1. Limited practical significance of the experimental findings.

The experiments mainly confirm expected results. The paper fails to identify a new OPD-specific mechanism or provide actionable insights into OPD, such as predicting which states are more recoverable without conducting additional evaluations.

Questions for authors
  1. Which conclusions in the paper would not hold outside OPD?

  2. Why is the competing update a reasonable assumption in the OPD setting, and what does it correspond to in practice?

3 comments

We sincerely thank you for finding the question interesting and for the concrete questions. We address them below.


Q1 (Weakness 1): The main theoretical results largely formalize general properties of stateful optimization or follow directly from their definitions.

That optimizer state shapes future updates is our premise, not our result. The paper answers which summary identifies a finite-budget OPD recovery event, and how large the error is without it. Theorem 1 gives the exact criterion and a sharp minimax bound (half the fiber diameter, attained; App. A.3). Theorem 2 shows that this ambiguity is reachable inside one OPD process, and §5 measures it in a full Transformer and in Qwen3.5 distillation on GSM8K.


Q2 (Question 1 & Weakness 2): Which conclusions in the paper would not hold outside OPD?

In OPD, the student's own policy decides where teacher supervision becomes available. Compared with a fixed, exogenous data stream, this creates three distinctions.

(1) Theorem 3. On-policy, an opportunity needs a length-H student path and occurs with probability p^H. Together with a competing update (0 < λ < λ_max), this creates the bistable barrier. With a fixed data rate c, the drift bracket is c(q−p)−λp, which has one stable root and no barrier.

(2) Theorem 4. In OPD, natural access h is set by the student's own policy, so access and learning form a feedback loop. Direct intervention removes this student-dependent bottleneck. With a fixed stream, access no longer depends on the student, and this loop disappears.

(3) Theorem 2. On-policy, the hidden state changes subsequent updates and therefore the future student-generated data (Eq. 14), so the difference compounds through the data. The natural-continuation separation is in Eq. 16.

Theorem 1 is general by design: it is the measurement tool.

Manuscript revision: We have added this fixed-rate contrast as a Remark after Theorem 3 (p. 6), with the derivation in App. A.5 (p. 17).


Q3 (Question 2): Why is the competing update a reasonable assumption in the OPD setting, and what does it correspond to in practice?

It captures an asymmetry between supervision and opposing pressure. In Theorem 3, it is an update that lowers the recovery branch whether or not the recovery state is visited (§3.5). App. A.5 derives its −λp term exactly from binary cross-entropy toward the zero target. Teacher supervision on the missing behavior waits for the student to visit it, while the opposing pressure does not. In practical OPD, this pressure can represent interference from training on other prompts through shared parameters, or regularization toward a reference model. λ = 0 recovers the smooth regime as a special case.


Q4 (Weakness 3): The paper fails to identify a new OPD-specific mechanism or provide actionable insights into OPD, such as predicting which states are more recoverable without conducting additional evaluations.

When the fiber diameter is positive, no observable-only predictor is correct for every state: its worst-case error is at least half the diameter (Theorem 1). §5 shows this ambiguity is present in practice, and this tells us what to track. Equalizing the full state removes the recovery discrepancy in all 60 cells, while matching on seven observable families leaves a gap in all 30 cells (App. B.6, Table 7). Under ordinary training, the optimizer factor carries more short-budget recovery variation than residual weights (Tables 1–2). The theory also gives actionable rules: look near the regime boundary (Theorem 3), and send direct queries where natural access is rare and the teacher is reliable on those states (Theorem 4; Table 6 reports complementary intervention outcomes).

Thank you for the detailed rebuttal and clarifications. However, my main concerns remain unresolved.

The theoretical contributions are still limited, as most results are general properties of stateful optimization or straightforward consequences of the definitions. The OPD-specific analysis relies on strong assumptions whose relevance to practical OPD training remains unclear.

Moreover, I do not find the empirical results particularly compelling. The experiments largely confirm expected phenomena, without revealing a new OPD-specific mechanism or providing actionable insights for improving OPD. Overall, I find the paper lacking in both theoretical novelty and empirical significance.

I therefore maintain my original rating.

Thank you for the follow-up. We clarify the OPD-specific mechanism, why the competing update is meaningful, and what the results can be used for.


1. What goes beyond general optimizer memory?

In OPD, the student's own policy decides when corrective teacher feedback becomes available. Theorem 3 makes this explicit: an H-step recovery opportunity occurs with probability p^H. For 0 < λ < λ_max, this student-dependent access, combined with a competing update, creates a bistable recovery barrier.

Replacing p^H with a fixed exposure rate c removes the barrier. The mean-field drift becomes ṗ = p(1−p)[c(q−p)−λp], which has a single stable interior root for every λ (new Remark after Theorem 3; derivation in App. A.5). The mechanism is therefore the feedback loop between student behavior, access to correction, and learning, not optimizer memory alone.


2. Why is the competing update a reasonable assumption?

The competing update has a concrete loss-based interpretation. A zero-target binary cross-entropy term contributes −λp to the logit update (App. A.5, Eqs. 18–20). It models opposing training pressure, such as updates on other prompts through shared parameters or regularization toward a reference model. Such pressure need not wait for the recovery-relevant state to be visited, whereas corrective supervision is gated by the student's own p^H visitation. This asymmetry produces the bistable regime of Theorem 3, and λ = 0 recovers the smooth regime as a special case.


3. What can the results be used for?

Checkpoint decisions need more than current outputs. Equalizing the full state removes recovery discrepancies in all 60 control cells, while matching seven observable families leaves them in all 30 cells (App. B.6, Table 7). Under ordinary nonzero-rate training, optimizer cross-swaps move recovery more than residual weight differences (Tables 1–2).

Optimizer state also improves recovery prediction. For the same checkpoints, adding the optimizer state to the exact weights lowers the squared error of predicting recovery on held-out future tapes by 50–91% at B = 1, in all three settings.

These results support three practical rules: keep optimizer state when continuing OPD from a checkpoint, test whether a monitor is sufficient before using it for recovery decisions, and spend direct queries where natural access is rare and teacher feedback is reliable (Theorem 4).

Reviewer_XFh3OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

During on-policy training, student predictions do not fully describe the internal state of the model, and runs with identical predictions can diverge into different states throughout training. This paper identifies a transition-sufficient training state of a student network during on-policy distillation. They define an observation-fiber diameter and use it to establish a sharp minimax lower bound on prediction error from observable summaries alone. They also formalize how natural visitation, direct state intervention, and teacher reliability jointly determine query value, and characterize scalar recovery regimes.

Critical strengths

There is clear mathematical and empirical rigor in proving finite-budget recoverability and observation fiber sufficiency, reachable-history non-identifiability and intervention value decomposition.

Critical weaknesses
  • The core premise feels obvious and doesn't seem to need the mathematical or empirical rigour that the paper delivers. Specifically, Observation 1: Identical current weights can lead to different recovery probabilities. Once we establish that training dynamics are governed by hidden factors like optimizer momentum, that explanation feels sufficient for the central claim without needing heavy math or contrived setups.

  • The paper spends a lot of effort discussing how observable summaries aren't sufficient and don't form Markovian fully identifiable states, but spends little effort explaining why it helps to know a fully described state or why practitioners should actually track it.

  • The takeaway that intervention value turns negative when r < 1/2 does not seem actionable. Not only does it seem intuitively obvious that training on an inaccurate teacher will degrade performance but also knowing this theoretical boundary in practice doesn’t seem useful because the teacher's real-time accuracy is not known.

Questions for authors
  • You state that metrics like probe logits, teacher divergence, entropy, support overlap, occupancy, and task success are insufficient training states. When is this actually problematic in practice? For instance, can you indicate empirically how monitoring these metrics during Generalized Knowledge Distillation (GKD) can falsely signal training stability or progress compared to your framework?

  • Section 4.1 is highly contrived to construct a scenario with identical weights but different Adam momentum. Given that the theoretical framework is already rigorous, it is unclear why such an unlikely scenario warrants empirical validation.

  • While Section 4.2 attempts to test this under nonzero learning rates, it remains unclear what real-world training failure mode this is meant to emulate or diagnose.

  • Ultimately, what is the core practical takeaway of all this analysis? What is the interesting takeaway for the field and is it practically relevant?

3 comments

We sincerely thank you for recognizing the rigor of the analysis. We answer the practical questions below.


Q1 (Weakness 1): The core premise feels obvious and doesn't seem to need the mathematical or empirical rigour that the paper delivers.

"Momentum is hidden state" says that something is missing. It does not say whether a given monitor is enough, how wrong it can be, or when the difference decides recovery. The paper answers all three: an exact sufficiency test and its error price (Theorem 1), reachability within one OPD process (Theorem 2), and the regimes where hidden state matters (Theorem 3).


Q2 (Question 1): You state that metrics like probe logits, teacher divergence, entropy, support overlap, occupancy, and task success are insufficient training states. When is this actually problematic in practice? For instance, can you indicate empirically how monitoring these metrics during GKD can falsely signal training stability or progress compared to your framework?

Two runs can look the same on every such monitor and still have different chances of recovering a missing behavior.

(1) Matched monitors, different futures. Appendix B.1 matches histories, outcome-blind and within fixed resolutions, on exactly the metrics you list: logits on fixed probes, teacher divergence, entropy, support overlap, branch mass, occupancy, and task success. These are the quantities a GKD-style run typically monitors. All 30 matched cells still show a positive recovery gap, while full-state controls show zero gap in all 60 cells (App. B.6, Table 7).

(2) Exact agreement, different futures. In §4.1 the weights are byte-identical, so every metric computed from the current student agrees exactly, and recovery still differs (Observation 1).

A monitor-based decision (stop, roll back, or select a checkpoint) then treats different futures as the same.


Q3 (Question 2): Section 4.1 is highly contrived to construct a scenario with identical weights but different Adam momentum. It is unclear why such an unlikely scenario warrants empirical validation.

The exact-weight construction isolates optimizer memory under controlled conditions. Corollary 3.1 shows that different minibatch orders produce distinct Adam moments at identical weights. Zero LR only makes the weights exactly equal, so the difference can come only from non-weight state. The experiment shows that the theoretical fiber is realized by a full Transformer with real on-policy sampling, and that resetting Adam removes the gap exactly (Observation 2). Section 4.2 then shows that the same optimizer effect persists under ordinary nonzero-rate training (Tables 1–2).


Q4 (Question 3): While Section 4.2 attempts to test this under nonzero learning rates, it remains unclear what real-world training failure mode this is meant to emulate or diagnose.

Two routine operations: (i) resuming or branching OPD runs from weights-only checkpoints; (ii) choosing between checkpoints that look the same on monitors. The cross-swaps show that the optimizer state, which these monitors do not see, moves recovery more than residual weight differences at short budgets (Tables 1–2), including in Qwen3.5 distillation on GSM8K.

Q5 (Weaknesses 2–3 & Question 4): Why should practitioners actually track the full state? The takeaway that intervention value turns negative when r < 1/2 does not seem actionable, because the teacher's real-time accuracy is not known. What is the core practical takeaway?

The takeaways are concrete:

  • Preserve the full state. Store and transfer optimizer state with OPD checkpoints whenever recovery decisions depend on them.
  • Audit monitors. Compare matched histories under common continuations, then use optimizer cross-swaps to identify the hidden state that drives recovery.
  • Target access. Spend direct queries where natural access is rare. The expected signed-evidence gain scales with (1−h), shrinking as natural visitation grows and vanishing at h = 1.
  • Gate by local reliability. Gate direct queries by teacher reliability on the queried states. This r is local, not the teacher's overall accuracy, and it can be checked on a small verified probe set.

Why r < 1/2 matters. The key point is not that a bad teacher hurts. It is that forcing access multiplies the teacher's local error: when r < 1/2, the negative expected signed evidence grows with (1−h), exactly where the student would not have gone alone (Theorem 4). Table 6 is consistent with this: under the same query and compute budget, directional targets and teacher guidance raise recovery, while rollout exposure and the hazard variant lower it. More exposure is not automatically more useful supervision.

Core takeaway. In OPD, a student's future depends on the state its next transitions read, not on its weights or predictions. The paper gives the test for when a summary is enough, the error when it is not, and the regimes where this decides recovery.

Thank you for the responses. I will maintain my score.

Reviewer_z7x5OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper tries to determine which representation of training state determines whether a student in on-policy distillation can recover a missing behavior given a budget. It shows that using an observational summary to determine this is sufficient when the rate of recovery is constant across states with the same observation. When it is not constant, the spread of the recovery rate bounds the error of any summary based predictor.

Critical strengths

None.

Critical weaknesses

Weakness 1: the finding is mostly known, and the theory adds little to it. The optimizer state affects training results, and that is why many model-training libraries save the full optimizer state in their checkpoints. For experiments with the same model weights, the difference in recovery rate may be due to the noise within the optimizer’s state. The paper does not make clear what the implications of the results are beyond just the variance across different training runs, and the lack of implementation details, such as sampling settings, makes it difficult to determine the cause of the variance in recovery rate for non-zero learning rate experiments. The theory is also not that significant. For example, Theorem 1 holds by definition. If two states with the same observation can lead to different recovery rates, then that observation, by definition, is not sufficient to predict the recovery rate. Theorem 4 relies on terms such as h and r, which are not measured.

Weakness 2: the paper needs more clarity. Perhaps the biggest weakness is the lack of clarity. Many important terms, such as “opportunity”, “recovery-relevant branch”, "direct intervention", "progress statistic", and "hazard variant", were not properly defined. These are examples of those terms, but there were many more in the paper. Without any proper definition of these terms, the paper is very difficult to follow.

Questions for authors

OPD is typically run for a lot more than 16 steps, or a budget of B=16 in the case of the paper. Is there evidence that the optimizer state's effect on the recovery rate persists for a larger budget?

8 comments

We sincerely thank you for your time and feedback. We address each concern below.


Q1 (Weakness 1): The optimizer state affects training results, and that is why many model-training libraries save the full optimizer state in their checkpoints.

Saving optimizer state for resuming is engineering practice; our results answer a different question. It does not tell us whether the signals used to monitor OPD (teacher divergence, entropy, logits, occupancy, task success) identify future recovery, or how large the error is when they do not. Theorem 1 answers both, and §5 measures the answer. This matters because distilled students are often shared or re-initialized from weights only.


Q2 (Weakness 1): For experiments with the same model weights, the difference in recovery rate may be due to the noise within the optimizer's state, and the lack of implementation details, such as sampling settings, makes it difficult to determine the cause of the variance.

The gap is not run-to-run noise. The design removes noise in four ways (§4.1, Table 4).

(1) Common future tapes. All hidden states share 64 tapes, which fix prompts, sampling uniforms, and dropout. Generation stays on-policy, because each student maps the same uniforms through its own distribution.

(2) Replication unit. The base seed is the replication unit, and intervals resample base seeds.

(3) Calibration. A permutation null subtracts the finite-sample diameter, and a split-sample estimate selects on half of the tapes and tests on the other half. Both stay positive at B = 1.

(4) Carrier removal. Resetting the Adam moments, or using stateless SGD, makes recoveries agree exactly (Observation 2).

The exact disappearance under optimizer equalization, together with permutation calibration and split-sample validation, identifies optimizer memory as the source of the recovery separation. At nonzero learning rates, cross-swaps attribute the gap to the optimizer factor rather than to residual weights (Tables 1–2).

Manuscript revision: We have collected all model, sampling, and optimizer settings into one table (Appendix D, Table 8, p. 27), including Qwen3.5 sampling (temperature 0.8, top-p 0.95, up to 128 new tokens) and AdamW settings for every experiment.


Q3 (Weakness 1): Theorem 1 holds by definition. Theorem 4 relies on terms such as h and r, which are not measured.

(1) Theorem 1. The fiber criterion fixes the right object. The content is the sharp minimax bound: half the fiber diameter is a lower bound, and it is attained (App. A.3). It gives the exact minimum worst-case error for prediction from that summary alone. In §4.1, the subset diameter across hidden Adam states lower-bounds this fiber diameter (Eq. 31); Table 4 estimates it on common future tapes with permutation calibration and split-sample validation.

(2) Theorem 4. Both terms are measurable quantities. h is the probability that a natural query reaches the recovery-relevant state; it can be estimated from on-policy rollouts. r is teacher reliability on the queried states (Table 3); it can be estimated on a verified probe set. Table 6 is consistent with this separation of access from feedback: under equal resources, directional targets and teacher guidance improve recovery, while rollout exposure and the hazard variant reduce it.

Q4 (Weakness 2): Many important terms, such as "opportunity", "recovery-relevant branch", "direct intervention", "progress statistic", and "hazard variant", were not properly defined.

We now define every term at first use and collect all of them in a glossary.

  • Opportunity: an event in which the student's own length-H path reaches the state where the teacher can supervise the missing behavior; it occurs with probability p^H (§3.5).
  • Recovery-relevant branch: the student branch whose probability p must rise for recovery (§3.5).
  • Direct intervention: a query that presents that state explicitly, setting I_i = 1 instead of waiting for natural visitation (§3.6).
  • Progress statistic: the scalar that the update moves toward recovery, such as the branch logit θ; it is monotone in the signed evidence sum (Theorem 4).
  • Hazard variant: an equal-resource control that changes the per-query chance of reaching that state while keeping the recovery event fixed (Table 6).

Manuscript revision: We have added a glossary of operational terms to Table 3 (p. 14), with a pointer in §3.1 (p. 4). The progress statistic is now defined in Theorem 4 (p. 6), and the hazard variant in Observation 6 (p. 9).


Q5 (Question): OPD is typically run for a lot more than 16 steps, or a budget of B=16 in the case of the paper. Is there evidence that the optimizer state's effect on the recovery rate persists for a larger budget?

B counts the updates spent on recovering one missing behavior after the observation point; it is not the length of OPD training. A long run contains many such windows. The effect holds across the budget:

(1) Qwen3.5/GSM8K. The optimizer effect on reference-answer probability stays above the weights effect at B = 8 (Table 2).

(2) Exact weight fiber. The probability-area functional integrates recovery over all 16 steps and remains separated (Table 4).

As B grows, cumulative binary recovery approaches its ceiling, so the question becomes how fast recovery happens. The area functional measures exactly this. Theorem 3 separately characterizes the long-run mean-field regimes, including bistability and collapse.

Reviewer_z7x5OpenReview ↗

Response to rebuttal

Thanks for the rebuttal.

My concerns were still not fully addressed.

  • The point about noise was noise within the optimizer. The training data itself has noise, and so the optimizer state will have noise. The rebuttal did not address this.
  • My point about the budget B was that if OPD was run for a large number of steps, and so after thousands of steps, it might be the case that recovery will happen eventually. The results show recovery happens for B=16, which is significantly lower than the number of steps used for OPD.
  • The point about undefined terms remains. Terms such as "natural query", "replication unit", and "split sample procedure" remain undefined, and these terms are used in the rebuttals.

Overall, I will keep my original score.

Message to AC: This is my first time writing a review, so take this into account. However, I still believe the paper, and to some extent the rebuttal, still has a lot of clarity issues. The dense jargon is really difficult to comprehend, in my opinion.

Reviewer_Dx97OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

It investigates whether the current observable state of a student model is sufficient to determine its future recoverability under on-policy distillation. A finite-budget recoverability is formalised using a transition sufficient training state and characterize its identifiability through observation fibers, deriving a minimax lower bound on prediction error. They construct training histories with identical observations but different recovery probabilities. They analyze simplified recovery dynamics and teacher-feedback reliability. The experiments demonstrate that identical weights with different Adam optimizer states can yield different recovery outcomes. Validation is on controlled DistilGPT2 experiments, optimizer-state cross-swaps under nonzero learning rates. Qwen3.5 distillation on GSM8K.

Critical strengths

I find strengths in the exact weight construction, common future random tapes, optimizer resets, and the factorial cross-swaps because they provide systematic experimental methodology for isolating optimizer-memory effects on recovery.

Critical weaknesses
  • The central observation that identical weights or predictions do not determine future learning behavior when hidden optimizer states differ follows directly from stateful optimization dynamics.

  • The relationship to existing work on observable-state sufficiency, training memory, and optimizer-state interventions is insufficiently developed. [1]

  • The experiments demonstrate that optimizer memory can influence recovery, but do not establish a quantitative relationship between optimizer-state properties and recovery outcomes or provide a method for predicting which states are more recoverable.

-The exact-weight experiment uses an artificial zero-learning-rate construction, while the natural-history experiments rely on approximate matching and exhibit uncertain signed effects. The prospective results indicate that recovery-gap magnitude transfers more consistently than state ordering. The experiments establish the existence of recovery differences without demonstrating that the proposed framework can reliably predict or improve recovery in practical on-policy distillation.

  • On Theorem 3, it presents a mathematically correct stability analysis of a simplified scalar system, with limited theoretical significance. The bistability arises from the interaction between on-policy sampling and a competing loss term. The assumption that useful sequences occur with probability (p^H) restricts the generality of the analysis. The theorem characterizes deterministic mean-field equilibria and leaves their relationship to finite-budget recoverability unclear. The model also excludes optimizer memory, leaving the observed differences in recovery probabilities between models with identical observable states unexplained. The connection between the theoretical regimes and the neural experiments remains insufficiently established.

  • On Theorem 2 it is a valid counterexample but also its theoretical significance is limited. It illustrates an established consequence of stateful optimization through a simplified OPD construction. The zero-versus-one recovery separation depends on a specifically chosen recovery threshold. It does not provide a characterization of how frequently such states arise or how substantial their recovery differences are in realistic training settings.

[1]Vasileios Sevetlidis and George Pavlidis, Process-Tensor Tomography of {SGD}: Measuring Non-Markovian Memory via Back-Flow of Distinguishability, AISTATS, 2026

Questions for authors

Q1. (a) What is the main theoretical novelty beyond established results on state sufficiency and optimizer memory? (b) which results are specific to on-policy distillation rather than general stateful learning processes?

Q2. (a) Given that recovery-state ordering does not transfer reliably across seeds, how useful is the proposed framework for predicting recoverability? (b) Can optimizer-state information improve predictions over observable model statistics alone?

Q3. Beyond this constructed example, can you characterize when naturally occurring OPD histories with similar observable states exhibit substantial differences in recoverability?

2 comments

We sincerely thank you for the detailed review and for recognizing the exact-weight construction, resets, and cross-swaps. We address your concerns below.


Q1 (Q1a & Weaknesses 1–2): What is the main theoretical novelty beyond established results on state sufficiency and optimizer memory? The relationship to existing work on observable-state sufficiency, training memory, and optimizer-state interventions is insufficiently developed.

That Adam has memory is our starting point, not our result. The paper answers which summary identifies a finite-budget OPD recovery event, and what minimum worst-case error remains when it does not.

(1) Exact criterion and sharp bound. Theorem 1 gives the exact criterion (constancy on observation fibers) and a sharp minimax bound: half the fiber diameter, which is attained (App. A.3).

(2) Relation to prior work. Bisimulation and state abstraction ask for policy-uniform equivalence; our criterion targets the identifiability of one recovery event (App. A.2). Sevetlidis & Pavlidis (AISTATS 2026) measure how much non-Markovian memory the SGD process carries; we ask whether a summary identifies an OPD recovery event.

Manuscript revision: We have added this comparison to Related Work (§2, p. 3).


Q2 (Q1b & Weakness 5): Which results are specific to on-policy distillation rather than general stateful learning processes? The assumption that useful sequences occur with probability p^H restricts the generality of Theorem 3, and its relationship to finite-budget recoverability is unclear.

In OPD, the student's own policy decides which states receive teacher feedback. Three results depend on this.

(1) Theorem 3. An opportunity needs a length-H student path, so it occurs with probability p^H. Together with a competing update (0 < λ < λ_max), this creates the bistable barrier. With a fixed data rate c, the drift bracket becomes c(q−p)−λp, which has a single stable root, so no barrier appears.

(2) Theorem 4. In OPD, natural access h is set by the student's own policy. Direct intervention removes this student-dependent bottleneck, and teacher reliability determines the sign of the expected signed-evidence gain.

(3) Theorem 2. The hidden state changes the next update and, through the updated policy, which data are sampled afterward (Eq. 14).

Theorem 1 is general by design: it is the measurement tool.

Scope of Theorem 3. p^H is the natural model of an H-step behavior that the student must generate itself. The mean-field regimes give the drift: they tell which states can reach the recovery basin at all, and finite-budget recovery means reaching it within B queries. In the bistable regime, an unstable root separates two basins, and finite stochastic paths can cross it (§3.5, App. A.5). Theorem 3 isolates the on-policy visitation mechanism; Theorem 2, Corollary 3.1, and §5 show directly that hidden optimizer state changes finite-budget recovery.

Manuscript revision: We have added the fixed-rate contrast as a Remark after Theorem 3 (p. 6), with the derivation in App. A.5 (p. 17).


Q3 (Weakness 6): The zero-versus-one recovery separation in Theorem 2 depends on a specifically chosen recovery threshold, and it does not characterize how frequently such states arise.

The separation is not an artifact of forced access. Under natural on-policy continuation, the same two reachable histories still have different recovery probabilities (Eq. 16), giving a positive minimax lower bound. Corollary 3.1 shows how easily such states arise: exchanging unequal gradients across positions yields a distinct Adam state at the same weights. Table 4 realizes this in a Transformer, with eight distinct Adam states at one weight hash for every base seed.

Q4 (Q2a): Given that recovery-state ordering does not transfer reliably across seeds, how useful is the proposed framework for predicting recoverability?

The theory predicts this split between magnitude and ordering. Within-fiber ambiguity is a property of the summary. The paired recovery gap G measures recovery differences among matched histories (App. A.8), and its magnitude transfers to unseen seeds at a comparable scale (Table 5). Ordering inside a fiber depends on which hidden carrier is active, so it is a separate prediction target (App. C.1).


Q5 (Q2b & Weakness 3): Can optimizer-state information improve predictions over observable model statistics alone?

Yes. Adding the optimizer state lowers held-out prediction error in all three settings.

(1) Theory. Refining a summary can only shrink the fiber diameter (Eq. 13), and the full state removes it (Proposition 1).

(2) Direct test on the existing runs. The target is each state's recovery rate on held-out tapes. Per base seed, we split the common tapes in half, 200 times. The weights-only predictor knows the exact weights and predicts each state by the mean of all states with the same weights; the second predictor also conditions on the optimizer state. Both are fit on one half and scored by squared error on the other. With no optimizer effect, the richer predictor would lose, since it averages fewer samples.

SettingError reduction, B = 1Error reduction, recovery area
Exact weight fiber (8 seeds)91%26%
Procedural, nonzero LR (8 seeds)86%11%
Qwen3.5/GSM8K (7 seeds)50%57%

(3) Full-state controls agree. Equalizing the full state removes the gap in all 60 cells, while matching on seven observable families leaves it in all 30 (Table 7, App. B.6).


Q6 (Q3): Beyond this constructed example, can you characterize when naturally occurring OPD histories with similar observable states exhibit substantial differences in recoverability?

The theory gives two mechanisms, and the nonzero-LR experiments test the second one directly.

(1) Sensitive regimes. Student-dependent visitation creates sensitive regimes near the recovery boundary (Theorem 3).

(2) Omitted carriers. An omitted carrier that changes the next transition can raise the coupling failure ε, so similar observations can lead to different futures (App. A.7). Observations 3–4 test this under ordinary training: pairs selected only on the current function still separate through the optimizer factor (Tables 1–2).


Q7 (Weakness 4): The exact-weight experiment uses an artificial zero-learning-rate construction, while the natural-history experiments rely on approximate matching and exhibit uncertain signed effects.

Zero LR is an identification device, and §4.2 tests the same carrier at ordinary learning rates. Zero LR makes every function of the weights identical, so the difference can only come from non-weight state. The fiber diameter measures the size of the ambiguity. The sign of an individual difference depends on the hidden state, which a summary alone cannot determine.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.