acceptodds
Under review as a conference paper at ICLR 2027

Rethinking On-Policy Distillation of Large Language Models: One Training Example

Abstract

On-policy distillation (OPD) trains on student rollouts with dense teacher supervision, but the role of training data remains unclear. We find that a single query supports hundreds of improving updates and recovers most of full-data OPD's gain across domains and model families. We explain this result through the states visited during training and the rate at which the student aligns with the teacher. We measure state coverage: the fraction of the states full-data OPD visits that a query set's rollouts reach. One query reaches 71.5%, mostly within 100 steps. Adding semantically distinct queries raises coverage and validation accuracy together, until 16 queries reach 98.9% and match full-data training. Yet alignment slows at a similar pace whether OPD trains on one query or the whole dataset, and even a fixed set of states takes hundreds of steps to absorb. OPD is therefore data-overfed but algorithm-starved. Rollouts quickly expose broad supervision early, while the student absorbs it increasingly slowly. As a further stress test, content-light templates and off-domain WildChat queries also approach the real-query baseline. These findings suggest that training queries matter through the teacher-supervised states their rollouts induce, while learning progress depends on how efficiently the student absorbs that supervision.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, on-policy distillation, LLM post-training

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_hAxZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how many training queries are needed for on-policy distillation. It reports that repeatedly training on one query recovers a substantial fraction of the improvement obtained with the full dataset, while 16 semantically diverse queries can match full-data performance under the tested budget. The experiments cover mathematical reasoning, code generation, instruction following, and tool use, with additional results for multiple teachers and content-light inputs. The paper offers two explanations. First, rollouts from a small query set visit many of the teacher-hidden-state clusters reached by full-data training. Second, teacher–student alignment slows at a similar rate across query-set sizes. The authors interpret these observations as evidence that OPD receives abundant supervision but absorbs it inefficiently.

Critical strengths

The central empirical observation is interesting. Substantial improvement from a single query, including one the student never solves, is worth documenting. The extension to content-light templates makes the result more informative than a demonstration based only on a carefully chosen problem.

The experiments cover several domains and model families. The multi-teacher results and out-of-domain evaluations provide useful evidence that the phenomenon is not confined to one mathematical benchmark.

The appendix includes meaningful controls. These examine cluster count, reference construction, state-extraction positions, query diversity at a fixed query count, frozen trajectories, and learning rate. The paper also explicitly distinguishes rollout-subsampling uncertainty from variation across trained models.

Critical weaknesses
  1. The coverage metric does not establish how much useful supervision is shared. Touching 71.5% of the K-means clusters leaves open whether states within a cluster have similar teacher targets or training gradients. Appendix B.1 checks robustness to clustering choices, but not this connection. Coverage and accuracy could simply rise together as rollouts become more varied.

  2. The absorption-rate analysis mixes learning with changes in the sampling distribution. Equations (5)–(6) evaluate each checkpoint on its own rollouts, so a smaller distance may partly reflect visiting states where teacher and student already agree. Similar normalized curves do not establish that OPD is "algorithm-starved." A common, fixed probe set is needed to separate these effects; prolonged learning on frozen trajectories does not resolve this issue.

  3. The main comparisons lack independent repetitions across training seeds and query sets. Mathematics uses three selected queries, and the other domains use one query each. The 30 repetitions in Appendix B.1 resample existing rollouts. They do not show whether independently chosen 16-query sets consistently match full-data training, particularly when the reported scores differ by only 0.1 points.

  4. The data-sufficiency claim depends on the training budget and model pairing. One-shot recovery of full-data gains falls from 87% at step 300 to 72% at step 1000. The teacher–student pairs are also within-family and same-size; the instruction-following teacher is obtained by post-training the student itself. It remains unclear whether so few queries suffice with a larger teacher or a less closely related pair.

  5. The training loss needs clearer specification. Equations (1)–(2) define advantages, but leave the optimized surrogate and treatment of stored trajectories unclear. There is also a gap between the headline experiment and its explanation: Section 3 uses top-16 supervision, whereas Sections 4–5 use sampled-token supervision. The proposed mechanism should be checked under the setting used for the main result.

Questions for authors

Do states assigned to the same coverage cluster have similar teacher distributions or training gradients? Can coverage predict held-out improvement across independently selected query sets?

Does the decline in absorption rate persist when all checkpoints and query settings are evaluated on the same fixed states and token distribution? How much of the measured change comes from parameter updates versus changes in visited states?

What is the distribution of one-shot performance across randomly selected queries and independent training seeds? Do independently sampled 16-query sets consistently match full-data training?

Does the 16-shot result persist beyond 300 steps and with independently tuned full-data baselines? What happens with a larger teacher or a less closely related teacher–student pair?

Please provide the exact training loss and off-policy update rule, repeat the main mechanistic comparisons under top-16 supervision, and resolve the evaluation-protocol inconsistencies noted above.

7 comments

Response to W1、Q1

W1: The coverage metric does not establish how much useful supervision is shared. Touching 71.5% of the K-means clusters leaves open whether states within a cluster have similar teacher targets or training gradients. Appendix B.1 checks robustness to clustering choices, but not this connection. Coverage and accuracy could simply rise together as rollouts become more varied.

Q1: Do states assigned to the same coverage cluster have similar teacher distributions or training gradients? Can coverage predict held-out improvement across independently selected query sets?

[Global Response 1] tests both the semantic interpretation and the alternative rollout-diversity explanation. Same-cluster states have closer teacher targets, with mean JS divergence 0.4683 versus 0.6869; gradient directions are more aligned on average, but the intervals include zero. Coverage and accuracy also rise together across response diversity, semantic query diversity, and the 1/4/16 ladder, while changing query order has little effect. This joint pattern supports coverage as an empirical correlate of data efficiency beyond simple rollout variation. We will continue causal validation on independently selected query sets.

Response to W2、Q2

W2: The absorption-rate analysis mixes learning with changes in the sampling distribution. Equations (5)–(6) evaluate each checkpoint on its own rollouts, so a smaller distance may partly reflect visiting states where teacher and student already agree. Similar normalized curves do not establish that OPD is "algorithm-starved." A common, fixed probe set is needed to separate these effects; prolonged learning on frozen trajectories does not resolve this issue.

Q2: Does the decline in absorption rate persist when all checkpoints and query settings are evaluated on the same fixed states and token distribution? How much of the measured change comes from parameter updates versus changes in visited states?

[Global Response 2] isolates state visitation by evaluating every checkpoint on the same 1,024 prefixes with full-vocabulary JS divergence and reverse KL. Both divergences decrease, and the per-update absorption rate is lower over steps 250–300 than over 50–100 in every tested run. The slowdown therefore remains when all checkpoints see the same evaluation states, which separates parameter-driven slowdown from changes in visited states. This fixed-prefix control also complements the separate fixed-training-trajectory experiment: one holds evaluation states fixed, while the other holds training trajectories fixed.

Response to W3 and Q3

W3: The main comparisons lack independent repetitions across training seeds and query sets. Mathematics uses three selected queries, and the other domains use one query each. The 30 repetitions in Appendix B.1 resample existing rollouts. They do not show whether independently chosen 16-query sets consistently match full-data training, particularly when the reported scores differ by only 0.1 points.

Q3: What is the distribution of one-shot performance across randomly selected queries and independent training seeds? Do independently sampled 16-query sets consistently match full-data training?

Building upon Our Original Analysis

  • Training query difficulty. Figure 4 shows that one-shot OPD improves held-out mathematics accuracy with each of the easy, medium, and hard training queries, including the hard query that the student never solves during training.

  • Comparison with full-data OPD. At step 300, the selected 16-query set reaches 70.9% average accuracy versus 70.8% for full-data OPD in mathematics (§4.2). In three-domain MOPD, the selected sets with 16 queries per domain reach 52.9% versus 52.8% for full-data MOPD (Appendix D). Whether independently sampled 16-query sets consistently achieve comparable results remains open.

  • Domains, model families, and sizes. Across our experiments, one-shot OPD improves performance in four task domains and three model families, with student sizes ranging from 1.5B to 7B parameters.

One-shot performance across queries (Appendix A.4, Table 6)

We report step-300 results for three distinct training queries in each non-mathematical domain.

Domain and metricInitial studentTeacherQuery 1Query 2Query 3
Code generation (LCB v6, avg@3)19.4330.1022.8623.2427.24
Instruction following (Multi-IF third-turn, 8-language mean)20.8428.5825.0725.9225.39
Agentic tool use (BFCL v3, avg@8)70.77578.94876.02173.60875.091

One-shot OPD improves on the initial student in all nine runs. Across the three queries tested per domain, teacher–student gap recovery ranges from 32.1% to 73.2% in code generation, 54.7% to 65.6% in instruction following, and 34.7% to 64.2% in agentic tool use. Estimating the one-shot performance distribution across randomly sampled queries and independent training seeds would require substantial additional compute and training time.

Response to W4 and Q4

W4: The data-sufficiency claim depends on the training budget and model pairing. One-shot recovery of full-data gains falls from 87% at step 300 to 72% at step 1000. The teacher–student pairs are also within-family and same-size; the instruction-following teacher is obtained by post-training the student itself. It remains unclear whether so few queries suffice with a larger teacher or a less closely related pair.

Q4: Does the 16-shot result persist beyond 300 steps and with independently tuned full-data baselines? What happens with a larger teacher or a less closely related teacher–student pair?

Building upon Our Original Analysis

  • Comparison under a shared training recipe. At step 300, the selected 16-query set reaches 70.9% average accuracy versus 70.8% for full-data OPD in mathematics (§4.2). In three-domain MOPD, 16 queries per domain reach 52.9% versus 52.8% for full-data MOPD (Appendix D). We hold the training recipe fixed across query budgets so that the comparison isolates the choice of training queries.

  • One-shot performance through 1,000 steps. In mathematics, one-shot accuracy stays near its step-300 level, moving from 68.5% to 68.4%, while full-data accuracy rises from 69.8% to 72.1%. One-shot OPD thus recovers 87% of the full-data gain at step 300 and 72% at step 1,000 (§3.2).

  • Choice of teacher–student pairs. We match the teacher and student in size and model family to study query budgets in a setting where OPD already works. For code generation and instruction following, the teachers are obtained by post-training the same 1.5B model used to initialize the student. Changing the teacher's size or family can affect distillation stability independently of the query budget. For a larger or less closely related teacher, we would first establish a working full-data OPD baseline for that pair and then vary the number of training queries.

New Experiment: Extending 16-Shot OPD to 1,000 Steps

We extend 16-shot OPD to 1,000 steps in code generation and instruction following and report the scores every 100 steps.

Training stepLiveCodeBench (avg@3)Multi-IF English (turn 3 overall)
019.0523.22
10025.9035.51
20026.4834.55
30026.4835.61
40026.6734.21
50027.8134.92
60028.1934.71
70029.3332.49
80028.1934.12
90029.7134.31
1,00029.9032.99

At step 1,000, 16-shot OPD retains substantial gains over the initial student in both domains. Code improves from 26.48 at step 300 to 29.90 at step 1,000. Instruction following declines from 35.61 to 32.99 over the same interval and finishes 9.77 points above its initial score.

Reviewer_56jqOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This study examines the prompt requirements of on-policy distillation. Across three model families and four domains, one query recovers significant portion of the full-data improvement, while 16 diverse queries achieve comparable performance in the tested settings. The authors explain these findings through hidden-state coverage and declining alignment rates, supported by experiments with fixed trajectories, minimal prompts, and multiple teachers.

Critical strengths
  1. Substantial gains from very few prompts challenge common assumptions about distillation data requirements. Results across several model families and tasks make this observation broadly relevant.

  2. Experiments varying query diversity, trajectory reuse, and input content help establish when the phenomenon occurs and provide useful guidance for prompt selection.

Critical weaknesses
  1. Cluster occupancy is insufficiently validated as an explanation for generalization. Sharing a teacher-hidden-state cluster does not necessarily imply comparable targets or transferable gradients. The robustness checks support the consistency of the measurement, but leave its connection to effective supervision unresolved.

  2. The claimed optimization bottleneck rests on ambiguous diagnostics. Changes in the measured teacher–student distance also reflect shifts in sampled states and tokens. Additionally, signed overlap advantage permits cancellation: distributions (0.9, 0.1) and (0.1, 0.9) have identical entropy, complete support overlap, and zero overlap advantage despite substantial disagreement. These issues weaken the interpretation of the observed trends as slow supervision absorption.

Questions for authors
  1. Regarding Clarification, what evidence shows that hidden-state cluster coverage captures transferable supervision, such as similarity in teacher targets or training gradients?

  2. Targeted validation: Do the alignment and slowdown findings persist when checkpoints are compared on a shared, fixed state set using KL or JS divergence, thereby avoiding visitation shifts and signed cancellation?

2 comments

Response to W1、Q1

W1: Cluster occupancy is insufficiently validated as an explanation for generalization. Sharing a teacher-hidden-state cluster does not necessarily imply comparable targets or transferable gradients. The robustness checks support the consistency of the measurement, but leave its connection to effective supervision unresolved.

Q1: Regarding Clarification, what evidence shows that hidden-state cluster coverage captures transferable supervision, such as similarity in teacher targets or training gradients?

[Global Response 1] provides the direct test requested here. Coverage itself measures state-support breadth. Within the same reference clusters, teacher next-token distributions are consistently closer, with mean JS divergence 0.4683 versus 0.6869, while the gradient-cosine differences are +0.0191 for expected gradients and +0.0431 for replayed-token gradients, with both 95% intervals including zero. The BGE-M3 check preserves the query-count ordering. We therefore separate three statements: clusters are associated with similar teacher targets, coverage measures state-support breadth, and transferable gradients are not established.

Response to W2、Q2

W2: The claimed optimization bottleneck rests on ambiguous diagnostics. Changes in the measured teacher–student distance also reflect shifts in sampled states and tokens. Additionally, signed overlap advantage permits cancellation: distributions 09˙,01˙0\.9, 0\.1 and 01˙,09˙0\.1, 0\.9 have identical entropy, complete support overlap, and zero overlap advantage despite substantial disagreement. These issues weaken the interpretation of the observed trends as slow supervision absorption.

Q2: Targeted validation: Do the alignment and slowdown findings persist when checkpoints are compared on a shared, fixed state set using KL or JS divergence, thereby avoiding visitation shifts and signed cancellation?

Thanks for separating two issues in the absorption-rate analysis: trajectory-dependent evaluation and signed cancellation in the overlap-token advantage.

The first concern is material. The trajectory-level dtd_t in Eq. 55 is evaluated on checkpoint-specific rollouts, so both visited states and sampled tokens can change during training. [Global Response 2] addresses this by evaluating every checkpoint on the same 1,024 prefixes with equal averaging weights.

The signed-cancellation example is mathematically correct, but it concerns the overlap-token advantage in Eq. 44, which is a diagnostic. The absorption rate is not defined from that signed quantity. It is obtained from

$d _t=\frac{1}{| \mathcal{T} _t|}\sum _{i \in\mathcal{T} _t}\left|\log\pi _T(y _i\mid s _i)-\log\pi _{\theta_t}(y _i\mid s _i)\right| $

so individual token differences are averaged by magnitude and cannot cancel through opposite signs. The new fixed-probe evaluation goes further by comparing complete next-token distributions using nonnegative JS divergence and reverse KL, without sampling a particular token.

Appendix C.3, Table 13, and Figure 19 show that both divergences decrease in all four runs, and the absorption rate over steps 250–300 is lower than over 50–100. For full-data OPD, JS decreases from 0.01254 to 0.00168 while its interval rate decreases from 1.99% to 0.20%; reverse KL decreases from 0.05856 to 0.00859 while its rate decreases from 2.00% to 0.10%.

These results show that the slowdown remains when evaluation states, weights, and token sampling are held fixed. This separates evaluation-state variation from the observed slowdown, while the fixed-trajectory experiment separately addresses whether new training states are required. The two controls are complementary; neither assigns a percentage decomposition of the original trajectory-level change.

AuthorsOpenReview ↗

Gentle Reminder to Reviewer 56jq

Dear Reviewer 56jq,

Thank you again for your review. In response to your comments, we have revised the manuscript and marked all changes in red. Since the discussion period ends soon, here is a short summary of how we addressed your main concerns.

  • State coverage and supervision. [Global Response 1] adds matched-state measurements and an encoder control (Appendix B.3, Table 11, Figure 17; Appendix B.1, Table 7, Figure 13). Same-cluster states have closer teacher next-token distributions (mean JS 0.4683 vs. 0.6869), while mean OPD-gradient cosine is higher within clusters (+0.019 for expected gradients and +0.043 for replayed tokens). The BGE-M3 control preserves the query-count ordering. These results associate cluster membership with teacher-target similarity while defining coverage as relative state-support breadth.

  • Fixed-probe slowdown. [Global Response 2] evaluates every checkpoint on the same 1,024 prefixes using full-vocabulary JS divergence and reverse KL (Appendix C.3, Table 13, Figure 19). Both divergences decrease in every tested run, with a smaller fractional reduction per update over steps 250–300 than over steps 50–100. The signed overlap-token advantage is only a diagnostic; Eq. (5) uses absolute token gaps, and the fixed-probe evaluation compares complete distributions without sampling a particular next token. The slowdown therefore persists without evaluation-state shifts or signed cancellation.

We would be grateful to know whether these additions address your concerns, and we are happy to clarify anything further.

Reviewer_NFn9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies a striking phenomenon: on-policy distillation (OPD) trained on a single query continues to improve the student for hundreds of optimization steps and recovers most of the gain from full-data OPD. The authors ask why this happens and decompose the answer into two perspectives. From the data side, a query matters through the states its rollouts reach; one query already covers 71.5% of the state clusters visited by full-data OPD, and 16 semantically diverse queries cover 98.9% and match full-data training. From the algorithm side, the absorption rate—the fraction of the remaining teacher–student gap closed per update—declines similarly whether the student trains on 1, 4, 16, or all 17k queries. The paper further shows that content-light templates and off-domain WildChat prompts can drive OPD nearly as well as curated problems, and that one-shot OPD extracts substantially more signal from the same query than one-shot RLVR. The central claim is that OPD is “data-overfed but algorithm-starved”: supervision is abundant even from one query, but the student absorbs it slowly.

Critical strengths
  1. The one-shot result is counterintuitive and clearly presented. Figure 1 and Figure 3 show that one query recovers 87% of full-data OPD’s gain at step 300 and continues to improve beyond that. The effect holds across four domains and three model families. This is the kind of controlled experiment that changes how practitioners think about data efficiency in OPD.

  2. Separating the question into “why does one query supply enough signal?” (data) and “why does the student take so long to absorb it?” (algorithm) is conceptually clarifying. The state-coverage analysis and the absorption-rate analysis are complementary rather than redundant, and each is supported by targeted ablations.

  3. The construction is sensible: states are represented by teacher final-layer hidden vectors, clustered with PCA and K-means, and coverage is the fraction of clusters a setting touches. The paper goes beyond a single number with robustness checks over K \in {50,100,200,500}, alternative cluster constructions, different state positions, and a nearest-reference distance diagnostic (Table 7). The finding that one query covers 71.5% and that most coverage appears early is robust and informative.

  4. Holding the query count at 16 while drawing from one semantic cluster versus 16 clusters (Figure 14) separates diversity from count. The result—70.9 vs 69.9 validation accuracy, and 98.9% vs 76.8% state coverage—supports the claim that diverse queries expand coverage rather than merely adding more data. The query-scheduling control in Figure 15 further shows that ordering a fixed query set matters little.

  5. The content-light and WildChat results shift attention from “collecting more problems” to “selecting inputs by the states they induce.” The multi-teacher extension (Appendix D) shows that 16 diverse queries per domain match full-data MOPD, which is a practically useful result. The comparison with one-shot RLVR (Figure 9) is also informative: OPD gains more than twice the validation accuracy over 1000 steps, and the bottleneck is slow absorption rather than outcome variation.

Critical weaknesses
  1. The paper convincingly shows that absorption slows similarly across data settings, but it does not explain why. The fixed-state and learning-rate ablations rule out two natural candidates (new-state arrival and step size), but the mechanistic cause remains unclear. Is it due to increasing curvature of the loss landscape, diminishing gradient signal as the student approaches the teacher, gradient noise, or the top-k/sampled-token objective’s implicit weighting? Without a mechanistic account, the “algorithm-starved” framing is descriptive rather than explanatory. The paper would be stronger with at least a diagnostic experiment—e.g., measuring gradient norms, Hessian eigenvalues, or the fraction of tokens where the teacher and student top-k sets overlap—to narrow down the cause.

  2. The metric requires a full-data OPD run to build the reference clusters, which limits its practical use for selecting queries in a new domain. The paper acknowledges this in future work, but the current results do not show whether coverage can be estimated without a reference run—e.g., by clustering states from a small pilot run or by using intrinsic diversity metrics. A practical selection procedure would strengthen the contribution.

  3. The out-of-domain evaluation (Table 4) shows no degradation on code and instruction following, which is reassuring. But the student trains for hundreds of steps on a single query with 64 rollouts per step and a KL coefficient of 0. Could there be subtler overfitting—e.g., to the query’s surface form, to specific reasoning patterns, or to the teacher’s token-level idiosyncrasies—that does not show up on the chosen benchmarks? A probe on paraphrases of the training query, or on a held-out set of similar problems, would help.

  4. The paper claims that “an input is useful largely because it starts the student reasoning,” and shows that empty templates and WildChat prompts work nearly as well as curated problems. But the base template still contains a <think> token, and the paper notes that immediately closing thinking yields short meta-level replies unsuitable for training. This suggests that the reasoning trigger—not just any input—is doing important work. The claim would be sharper if the paper separated “content-light” from “reasoning-triggering” and tested an input with no <think> token or a non-reasoning prompt.

  5. Figure 9 compares one-shot OPD and one-shot RLVR on a single medium query with one student. The result—OPD gains more than twice the validation accuracy over 1000 steps—is striking, but it is not clear how sensitive it is to the query, the RLVR hyperparameters, or the model pair. The paper says RLVR uses 8 groups of 8 with GRPO outcome rewards and a KL penalty of 0.001; were these tuned for the one-shot setting? A broader comparison across queries and model pairs would make the “different bottlenecks” claim more robust.

  6. Appendix D shows that 16 diverse queries per domain match full-data MOPD on three domains with one student. This is a promising result, but it is a single configuration. How sensitive is it to the number of domains, the routing strategy, and the diversity of queries within each domain? The paper notes that 16-shot MOPD reaches 101% of full-data MOPD’s gain, but the per-domain proportions (93%, 136%, 109%) suggest that the match is not uniform. A brief discussion of this variance would help.

  7. The paper is primarily empirical, which is fine, but the “algorithm-starved” claim would benefit from a simple theoretical model. For example, under a quadratic approximation of the loss, the absorption rate might be related to the condition number of the Hessian or to the step size relative to the local curvature. The learning-rate ablation hints at this, but the paper does not formalize it. A short theoretical section—even a toy analysis—would sharpen the contribution.

  8. The abstract and introduction say that one query “recovers most of full-data OPD’s gain” and that “16 semantically diverse queries cover 98.9% and match full-data training.” The main math results support this, but the code, instruction-following, and agentic results (Figure 12) recover 73%, 66%, and 64% of the teacher–student gap, respectively. These are meaningful gains but not “most” in the same sense. The paper should qualify the claim by domain.

Questions for authors

What causes the absorption rate to decline? You rule out new-state arrival and step size. Can you measure gradient norms, loss curvature, or the fraction of tokens where the teacher and student top-k sets differ over training? A diagnostic that distinguishes “the signal becomes harder to fit” from “the optimizer takes smaller effective steps” would substantially strengthen the algorithm perspective.

Can state coverage be estimated without a full-data reference run? The metric currently requires full-data OPD to build clusters. Could you approximate coverage by clustering states from a small pilot run, by using intrinsic diversity (e.g., embedding dispersion), or by selecting queries greedily to maximize coverage of a running state buffer? A practical procedure would make the data-side recommendation actionable.

Is one-shot OPD overfitting to the query’s surface form? Out-of-domain benchmarks show no degradation, but have you tested paraphrases of the training query or a held-out set of similar problems? For example, if you train on t22 (the medium query), does performance on a paraphrased version of t22 improve as much as on MATH-500? This would distinguish genuine reasoning improvement from query-specific memorization.

How much of the content-light result depends on the <think> token? The base template ends with <think>, and you note that immediately closing thinking yields short meta-level replies. What happens with a template that has no <think> token or with a non-reasoning prompt? Is the <think> token the key ingredient that induces useful states, or would any prompt that elicits long rollouts work?

How sensitive is the RLVR comparison to hyperparameters and query choice? You use one medium query and one student pair. Did you tune the RLVR learning rate, KL coefficient, and group size for the one-shot setting? Would the OPD advantage persist on an easy query (where RLVR has no outcome variation) or a hard query (where RLVR never succeeds)? A small sweep would make the “different bottlenecks” claim more convincing.

How does the state-coverage metric scale with model size and context length? The current experiments use 1.5B–7B students and relatively short contexts. For larger models or long-context tasks, the state space is much larger and clustering may be less stable. Do you expect the same coverage–diversity relationship to hold, or would the number of required queries grow?

What is the relationship between the top-k and sampled-token objectives in the one-shot setting? The math runs use the top-k objective with k=16, while other domains use the sampled-token objective. Does the one-shot phenomenon depend on this choice? Would the absorption rate decline differently under a different divergence (e.g., reverse KL, Jensen–Shannon) or a different top-k value?

How robust is the multi-teacher result to domain count and routing? Appendix D uses three domains and one student. Would 16 queries per domain still match full-data MOPD with five or ten domains? Does the routing strategy (domain-specific teachers) affect the required query diversity? A brief sensitivity analysis would help.

Can you formalize the “algorithm-starved” claim? A simple model—e.g., a quadratic loss with a teacher–student gap and a fixed step size—might predict how the absorption rate scales with the condition number or with the distance to the teacher. Even a toy analysis would clarify whether the slowdown is fundamental to OPD or an artifact of the current optimization setup.

What is the practical recipe for selecting queries? You suggest selecting inputs by the states they induce. Given a new domain, how many pilot queries would a practitioner need to run to estimate coverage, and how should they choose the 16 diverse queries? A step-by-step procedure or a small algorithm box would make the recommendation easier to adopt.

7 comments

Response to W1, Q1, W7, and Q9

Concern. The mechanism behind the slowdown remains unclear.

Requested evidence. Gradient norms, curvature diagnostics, or teacher–student top-kk overlap.

Requested formalization. A toy analysis, such as a quadratic loss model relating absorption to curvature and step size.

We thank the reviewer for asking what causes the absorption rate to decline and for requesting both diagnostic evidence and a formal analysis.

Building upon Our Original Analysis:The slowdown is consistent with a shrinking and less directionally coherent residual signal.

Across the tested query counts, the absorption rate declines while the student continues to approach the teacher. The fixed-training-trajectory run continues improving for about 200 updates, showing that new training states are not required for continued progress (Appendix C.5). The 0.5×0.5\times, 1×1\times, and 2×2\times learning-rate curves largely collapse when time is rescaled by the learning rate, showing that the tested learning rates change the pace along a shared alignment curve (Appendix C.2). These controls rule out the lack of new states and a simple learning-rate scaling as sufficient explanations for the slowdown.

New Experiment:Fixed-probe control, supervision diagnostics, and quadratic analysis (Appendix C.3, Table 13, Figure 19; Appendix C.6, Table 15; Appendix C.7).

Fixed-probe control (Appendix C.3, Table 13, and Figure 19). Every checkpoint is evaluated on the same 1,024 prefixes, generated by the initial student before the first OPD update. JS divergence and reverse KL both compare complete next-token distributions, and their fractional reductions per update are smaller over steps 250–300 than over 50–100. The slowdown therefore persists when the evaluation states and averaging weights are fixed.

Residual gaps and gradient diagnostics (Appendix C.6 and Table 15). The diagnostics for each run are computed from that run's own training batch and compared within the run.

MeasurementOne-shot16-shotFull data
Gradient norm0.210 → 0.03350.289 → 0.06190.346 → 0.0831
Mean absolute teacher–student log-probability gap0.0674 → 0.02630.107 → 0.03910.118 → 0.0496
Student top-16 tokens also in the teacher top-1690.0% → 94.9%88.4% → 94.9%88.3% → 94.7%

Gradient norms and absolute teacher–student gaps decline together, with within-run per-step correlations of 0.991, 0.972, and 0.966. The increasing top-16 overlap shows stronger agreement on candidate membership.

Gradient-direction diagnostic (Appendix C.6). In a separate single-prompt diagnostic, we sample 16 rollouts capped at 1,024 generated tokens per checkpoint. Mean pairwise OPD-gradient cosine in the last four transformer layers decreases from 0.457 at step 0 to 0.130 at step 100 and 0.016 at step 300, over 120 pairs per checkpoint. The remaining discrepancy therefore becomes not only smaller but also less coherent in this diagnostic.

Quadratic illustration (Appendix C.7). Under constant-step gradient descent on a positive-semidefinite quadratic, mode ii has contraction factor ri=(1−αλi)2r _i=(1-\alpha\lambda _i)^2. Let pt,ip _{t,i} denote its share of the loss at step tt. The fractional loss reduction v~t=1−Ft+1/Ft\widetilde v _t=1-F _{t+1}/F _t satisfies

$\widetilde v _{t+1}-\widetilde v _t = -\frac{\operatorname{Var} _{p_t}(r)}{\mathbb{E} _{p _t}[r]} \le 0 $

when Ft+1>0F_{t+1}>0. For 0<α≤1/λmax⁡0<\alpha\le1/\lambda_{\max}, faster-decaying high-curvature modes account for a shrinking share of the residual loss. The model shows how fractional progress can slow at a constant step size.

Together, these results support an optimization-side mechanism in which the remaining discrepancy becomes smaller and less directionally coherent. The relative contributions of curvature, gradient noise, and Adam's realized update size remain unresolved.

Response to W2, Q2, and Q10

NFn9 W2: State coverage requires a full-data OPD reference, and its practical use for query selection without that reference is not established.

NFn9 Q2: Can coverage be estimated from a small pilot run, intrinsic diversity, or greedy state coverage instead?

NFn9 Q10: What concrete procedure determines how many pilot queries to run and how to select 16 diverse queries in a new domain?

We thank the reviewer for asking how state coverage can be estimated without a full-data OPD reference and how to select diverse queries in a new domain. We separate the implemented BGE-M3 selector, a smaller-reference feasibility check, and a proposed state-aware selector.

Building upon Our Original Analysis:State coverage uses a full-data reference partition.

  • A state is the location where OPD computes supervision; the signal is the local teacher–student gap.
  • The metric is the fraction of full-data reference clusters touched by a setting's states.
  • The BGE-M3 selector chooses one query from each of 16 semantic clusters without a full-data OPD reference.
  • A pilot state count is not a pilot query count; sixteen is a tested budget, not a universal threshold.

The coverage analysis uses full-data OPD as a reference because it asks how much of that state space a smaller query set reaches. The implemented selector instead operates on the semantic diversity of the available query pool.

Implemented BGE-M3 selector (Appendix D and Figure 22). Embed the available query pool with BGE-M3, cluster it into 16 semantic groups within each domain, select one representative per group, and train under the same compute and evaluation budget as the baseline. This requires query embeddings, without a full-data OPD run or pilot training. In the three-domain MOPD experiment, 16-shot training per domain reaches 52.9 five-benchmark average accuracy versus 52.8 for full data at 300 steps.

New Experiment:Smaller-reference feasibility and state-aware selection (Appendix C.8 and Table 16).

Smaller-reference feasibility check (Appendix C.8 and Table 16). We subsample 1,000 states from the existing full-data reference, stratified across 500 training steps, refit PCA-50 and K=200K=200, and repeat the sample and clustering ten times.

Reference1-shot4-shot16-shotFull data held out
Original: 12,000 states71.5%79.8%98.9%100.0%
Subsampled: 1,000 states70.9 ± 2.0%82.1 ± 1.8%95.8 ± 1.1%97.4 ± 1.2%

Values are means ± one standard deviation. All ten draws preserve the 1 < 4 < 16 ordering, showing that a smaller state reference can retain the selection ordering. The states still come from an existing full-data run, so this check does not validate an independently chosen pilot or determine the required number of pilot queries.

A retrospective check over the already selected 16-shot pool reaches 87.5% pilot coverage with four greedy queries versus 77.5% on average for random 4-shot subsets. Selection and coverage measurement use the same pilot coordinates, so this is a sanity check rather than an independent or end-to-end result. Intrinsic dispersion is also not an established substitute: mean squared radii increase across settings, while effective ranks are nonmonotonic (73.7/106.5/85.4/97.1).

Proposed state-aware selector (Appendix C.8).

  1. Form a candidate pool covering diverse semantic groups, with more candidates than the target selection budget.
  2. Generate equal-budget current-student rollouts per query and sample prefixes at matched relative response positions. Existing RL-teacher queries or trajectories can initialize a buffer, but require refreshing because they follow a different policy.
  3. Fit the representation, PCA, and reference clusters on a pilot split; retain separate queries and rollouts to check stability.
  4. Greedily select the query adding the most unoccupied pilot clusters, and record marginal coverage as the selected set grows.
  5. Expand the pilot if held-out estimates are unstable, refresh shortlisted rollouts under the evolving student, and validate the selected set through matched-compute training.

Response to W3 and Q3

NFn9 W3: The out-of-domain evaluation (Table 4) shows no degradation on code and instruction following, which is reassuring. But the student trains for hundreds of steps on a single query with 64 rollouts per step and a KL coefficient of 0. Could there be subtler overfitting—e.g., to the query's surface form, to specific reasoning patterns, or to the teacher's token-level idiosyncrasies—that does not show up on the chosen benchmarks? A probe on paraphrases of the training query, or on a held-out set of similar problems, would help.

NFn9 Q3: Is one-shot OPD overfitting to the query's surface form? Out-of-domain benchmarks show no degradation, but have you tested paraphrases of the training query or a held-out set of similar problems? For example, if you train on t22 (the medium query), does performance on a paraphrased version of t22 improve as much as on MATH-500? This would distinguish genuine reasoning improvement from query-specific memorization.

We thank the reviewer for asking whether prolonged training on a single query can lead to overfitting to its wording, specific reasoning patterns, or the teacher's token preferences. We examine this concern using held-out benchmarks, controlled changes to training queries and input formats, and an audit of training-query rollouts through step 1,000.

Building upon Our Original Analysis

  • Held-out code and instruction-following performance. LiveCodeBench evaluates code generation, and Multi-IF evaluates multi-turn, multilingual instruction following [1, 2]. The Qwen3 Technical Report includes LiveCodeBench v5 and Multi-IF [3]. After one-shot OPD on the mathematical query t22, LiveCodeBench v6 rises from 21.96 at initialization to 23.55 at step 300 and 24.75 at step 1,000. Multi-IF moves from 21.67 to 23.32 and 23.27 at the same checkpoints (Appendix A.2, Table 4). From step 300 to step 1,000, the code score continues to improve and the instruction-following score remains essentially stable.

  • Training-query difficulty and input format. Before OPD, the student solves the easy, medium, and hard training questions in 8/8, 4/8, and 0/8 sampled rollouts, respectively (Appendix A.1). All three yield comparable gains on held-out mathematics benchmarks. During training, the student solves the easy question in nearly every rollout and never solves the hard question (§3.2, Figure 4). An empty user turn followed by a reasoning cue, a content-light template with a domain system prompt, and off-domain WildChat prompts also recover much of the gain from training on real mathematics problems. Explicit domain content changes final accuracy by about one point in this comparison (§6.1, Figure 8). These experiments show substantial gains from inputs that omit both the wording and the answer of t22.

  • Query diversity and state coverage. Rollouts from one training query reach 71.5% of the full-data OPD reference state clusters by step 300 (§4.1). With the query count fixed at 16 and the same training budget, queries drawn from distinct semantic clusters reach 70.9% validation accuracy and 98.9% state coverage, compared with 69.9% accuracy and approximately 76.8% coverage for queries drawn from one cluster (Appendix B.2, Figure 15). Under this fixed budget, higher held-out accuracy accompanies broader coverage of the states induced by the training queries.

New Analysis: Long-Horizon Rollout Audit (Appendix A.2, Table 5)

[4] report that, in one of their one-shot RLVR runs, reasoning on the training query mixed correct calculations with unintelligible multilingual text even as test performance remained strong. We inspect all 64 saved rollouts on the original t22 query at each of steps 50, 300, and 1,000. The number of correct responses rises from 55 to 56 to 60, and mean generation length stays in a narrow range at 5,681, 5,909, and 5,797 tokens, respectively. At step 1,000, 63 of 64 responses give a final answer in \boxed{}. The audited solutions remain readable and generally preserve mathematical notation. A few contain isolated Chinese phrases within otherwise coherent reasoning.

The revised Appendix A.2 describes the results in terms of performance on the measured benchmarks and rollout quality at the audited checkpoints. Whether paraphrases of t22 or matched held-out neighboring problems show gains comparable to those on MATH-500 remains open.

References

[1] LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code.

[2] Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Following.

[3] Qwen3 Technical Report.

[4] Reinforcement Learning for Reasoning in Large Language Models with One Training Example.

Comments on the paper

4 comments
AuthorsOpenReview ↗

[Global Response 1] State Coverage and Transferable Supervision

Relevant reviews: 56jq W1 Q1; hAxZ W1 Q1.

We thank both reviewers for asking whether states in the same coverage cluster have similar teacher targets or training gradients, and whether coverage and accuracy may both depend on rollout diversity. We have clarified the state coverage definition and the role of a state, added matched-state measurements of teacher targets and OPD gradients and stated which claims the present evidence does not support.

Building upon Our Original Analysis:Coverage measures the breadth of state-level supervision.

OPD computes a local teacher–student gap at each visited state. More distinct states create more opportunities for local supervision, while states in the same cluster can still produce different signals because the gap depends on the current student and optimized surrogate. Using full-data OPD as a fixed reference, coverage is the fraction of its reference clusters reached by another setting. One-shot OPD reaches 71.5%, showing broad state-support overlap.

New Experiment:Matched teacher targets, OPD gradients, and representation controls.

Coverage records which state regions are reached by a query set. To test whether states in the same covered region provide similar supervision, we compare their teacher targets and OPD gradient directions. We have revised §4.1 to state this interpretation and added direct matched-state measurements in Appendix B.3, Table 11 and Figure 17.

Matched targets and gradients AppendixB3˙,Table11andFigure17Appendix B\.3, Table 11 and Figure 17. For each state from the one-shot run on the medium query, we select one same-cluster and one different-cluster state from held-out full-data rollouts. All three states come from different queries and match on relative response position and 50-step training block; the two held-out states also match on response length. Teacher-target JS divergence compares the full-vocabulary teacher next-token distributions using natural logs; zero means identical distributions, and the final-token-matched row uses the subset with a common prefix-end token. OPD-gradient cosine compares the step-300 full-parameter gradients at a one-shot state and its matched state, computed either in expectation over the student’s next-token distribution or for the replayed token. Positive same-minus-different values indicate greater within-cluster similarity.

MeasureTripletsSame clusterDifferent clusterSame − different [95% CI]
Teacher next-token JS divergence2780.46830.6869−0.2186
−02˙341,−02˙038−0\.2341, −0\.2038

| |Teacher JS with matched final token|105|0.4051|0.6225|−0.2174

−02˙564,−01˙752−0\.2564, −0\.1752

| |Expected OPD-gradient cosine|36|0.0186|−0.0006|+0.0191

−00˙303,+00˙731−0\.0303, +0\.0731

| |Replayed-token OPD-gradient cosine|36|0.0426|−0.0005|+0.0431

−00˙173,+01˙133−0\.0173, +0\.1133

|

Same-cluster teacher distributions are closer in 79.9% of triplets, and this association remains under final-token matching. The mean gradient cosines are also higher within clusters, by 0.019 in expectation and 0.043 for the replayed token, although both intervals include zero. Thus, coverage-cluster membership is consistently associated with teacher-target similarity, while its relationship with gradient direction is suggestive but not statistically resolved.

Encoder sensitivity(Appendix B.1, Table 7, and Figure 13). We suspect that our use of teacher hidden states may have prompted the concern that a coverage cluster inherits teacher-target semantics. The teacher hidden state is one embedding choice for computing state coverage, and another encoder can replace it. Appendix B.1 compares teacher hidden states and BGE-M3 using exactly the same 34,384 state records, final 64 prefix tokens, eight relative response positions, reference/held-out split, raw-vector convention, PCA-50, K=200K=200, and three clustering seeds.

EncoderOne-shot4-shot16-shotFull-data held out
Teacher hidden state78.7%84.3%99.0%100.0%
BGE-M380.0%97.5%99.8%99.8%

Both encoders preserve the ordering by query count and near-full coverage at 16-shot. The higher 4-shot BGE value reflects more complementary query regions in that representation. Absolute coverage changes with the embedding and clustering partition. Because this controlled protocol differs from the original 71.5% analysis, we do not mix their absolute levels; the check supports the relative ordering across encoders, not a representation-independent amount of useful supervision.

Across the tested query sets, coverage and accuracy rise together for response diversity, semantic query diversity, and the 1/4/16 ladder, while changing query order has little effect. This supports coverage as an empirical correlate of data efficiency in the tested configurations. We will continue to investigate whether and how state coverage causally shapes parameter updates and their transfer across states.

AuthorsOpenReview ↗

[Global Response 2] Alignment Slows on a Common Fixed Probe

Relevant reviews: hAxZ W2 Q2; 56jq W2 Q2.

We thank the reviewers for noting that each checkpoint uses its own rollouts and that the signed overlap-token advantage can cancel across tokens. Because the original measurements evaluate each checkpoint on its own rollouts, changes in dtd_t can reflect both parameter updates and shifts in visited states. The fixed-training-state ablation addresses training-state reuse rather than evaluation on a common fixed probe. We therefore add a common fixed probe set with full-vocabulary JS divergence and reverse KL, and report the equivalent per-update absorption rates over checkpoint intervals.

New Experiment:Common fixed probe set with JS divergence and reverse KL(Appendix C.3, Table 13, and Figure 19).

Both alignment and its slowdown persist on a common fixed probe set. Appendix C.3, Table 13 and Figure 19, compares every checkpoint using the same 1,024 prefixes: 64 positions from an initial-student continuation for each of 16 MATH-500 validation questions. Prefixes and averaging weights remain fixed. At each prefix, we compare the complete next-token distributions using JS and reverse KL, KL(πθt∥πT)\mathrm{KL}(\pi _{\theta_t}\|\pi _T), avoiding dependence on one sampled token.

For mean divergence DtD _t, the equivalent fractional reduction per update over steps aa–bb is

va:b=1−(Db/Da)1/(b−a).v _{a:b}=1-(D _b/D _a)^{1/(b-a)}.

Training settingJS divergence, step 50 → 300JS absorption rate, early → lateReverse KL, step 50 → 300KL absorption rate, early → late
4-shot sampled-token OPD0.01361 → 0.003991.37% → 0.17%0.06403 → 0.019211.38% → 0.15%
16-shot sampled-token OPD0.01288 → 0.002821.68% → 0.16%0.06065 → 0.013651.68% → 0.14%
Full-data sampled-token OPD0.01254 → 0.001681.99% → 0.20%0.05856 → 0.008592.00% → 0.10%
One-shot top-16 OPD0.01457 → 0.007150.74% → 0.09%0.06918 → 0.033710.74% → 0.09%

Early and late denote steps 50–100 and 250–300; rates are percentages per update. All distances are in nats. Early sampled-token one-shot checkpoints are unavailable, so the one-shot row is explicitly the separate top-16 run. Confidence intervals in Figure 19 resample the 16 questions jointly at both interval endpoints.

All four runs reduce both JS divergence and reverse KL. For both divergences, the equivalent per-update absorption rate is lower over steps 250–300 than over 50–100. Full-data OPD reaches the lowest divergence at step 300 and also slows. The slowdown therefore persists when the evaluation prefixes are fixed and the signed overlap-token advantage is not used.

This control complements Appendix C.5, Figure 21: the two experiments address different sources of variation. The ablation in Appendix C.5 freezes training trajectories and shows continued learning for about 200 updates, whereas the new probe freezes evaluation prefixes for normally trained checkpoints. Taken together, they provide complementary evidence about continued learning and fixed-prefix slowdown, but neither test gives a numerical decomposition of the original on-policy distance change into parameter updates and changes in visited states.

AuthorsOpenReview ↗

[Global Response 3] Training Objectives and Mechanism Consistency

Relevant reviews: hAxZ W5 Q5; NFn9 Q7.

We thank the reviewers for requesting the exact training loss and off-policy update rule, and for asking whether the mechanism persists under top-16 supervision. We have specified the sampled-token and top-16 surrogates and added a matched objective comparison.

Building upon Our Original Analysis:Sampled-token and top-16 OPD share the same teacher–student log-ratio and PPO pipeline.

Objective choices across experiments. The mathematical phenomenon experiment in Section 3 uses top-16 only because it exposes more candidate-level dynamic metrics. In the matched medium-query (t22) comparison, top-16 and sampled-token OPD produce comparable one-shot gains and both have more than 95% top-16 overlap. We therefore use sampled-token OPD for the subsequent mechanism analyses to reduce token-level computation. All Section 4 experiments use sampled-token OPD consistently, so their mechanism comparisons share the same objective; Sections 4–5 also use sampled-token OPD consistently within each mechanism comparison.

Exact update. Section 2.2 defines the two advantages, and Appendix A.1, Eq. 77, specifies their optimized surrogates. Let πold\pi_{\mathrm{old}} denote the student immediately before an update. For token ww at state sis_i, define

\

r_\{iw\}(\theta)= \frac\{\pi_\theta(w\mid s_i)\} \{\pi_\{\mathrm\{old\}\}(w\mid s_i)\}, \qquad \ell(r,a)= -\min\\\{ra,\operatorname\{clip\}(r,0\.8,1\.2)a\\\}\. \

For sampled-token OPD, the detached advantage is

\

a_i= \log\pi_T(y_i\mid s_i)\- \log\pi_\{\mathrm\{old\}\}(y_i\mid s_i)\. \

The scalar implementation also applies dual clipping with coefficient 3 when ai<0a_i<0. The loss is averaged over non-padding response tokens.

For top-16 OPD, let ViV_i contain the 16 most probable tokens under the pre-update student, and let qiq_i be the corresponding probabilities renormalized over ViV_i. The detached candidate advantage is

\

a_\{iw\}=q_i(w)\left[ \log\pi_T(w\mid s_i)\- \log\pi_\{\mathrm\{old\}\}(w\mid s_i) \right]\. \

The loss is the response-token mean of

\

\sum_\{w\in V_i\}\ell(r_\{iw\},a_\{iw\})\. \

Clipping is applied per candidate before summation, and this path does not use the sampled-token path's additional dual clip. Both objectives use one PPO epoch, one full rollout minibatch, and no additional KL reward or actor KL loss.

Stored trajectories. Appendix C.5 reuses the initial student's 64 token sequences at every update. Student probabilities and student-dependent advantages are recomputed from the current checkpoint, while the teacher distribution at each stored prefix remains fixed. No correction is applied to the initial behavior-policy state distribution, so this is a controlled replay experiment rather than an unbiased estimate of fresh-rollout on-policy OPD.

New Experiment:Matched top-16 and sampled-token comparison(Appendix C.4, Table 14, and Figure 20).

We added a sampled-token run on the medium query with the same teacher, initial student, optimizer, and schedule as top-16 OPD, and recorded the same top-16 diagnostics solely for comparison; they do not enter the sampled-token training loss. Appendix C.4, Table 14, and Figure 20 report this matched comparison.

StepObjectiveAvg.Top-16 Overlap Ratio
300Top-1668.470.9522
300Sampled token67.330.9505
1000Top-1668.410.9713
1000Sampled token70.130.9707

Avg. is mean accuracy across MATH-500, AIME 2025, and AMC 2023. Both objectives produce substantial one-shot gains with over 95% top-16 overlap. Their accuracy differs by 1.14 points at step 300 and 1.72 at step 1000, supporting qualitative consistency rather than statistical equivalence.

Appendix C.3 shows that both JS divergence and reverse KL decrease with a lower late absorption rate. These are evaluation diagnostics, not alternative training losses.

AuthorsOpenReview ↗

To AC: Rebuttal Summary

Dear AC, SAC, and PC,

Thank you for coordinating the review and discussion. We have responded to every weakness and question raised by the three reviewers. All three reviews identify the one-shot phenomenon and the cross-domain evidence as informative; the remaining disagreement concerns the strength of our interpretation of the observed slowdown rather than the existence of the phenomenon.

Three recurring issues are addressed in [Global Response 1] through [Global Response 3]. [Global Response 1] shows that one query reaches 71.5% of the full-data state clusters while 16 diverse queries reach 98.9%. Same-cluster states have closer teacher targets (mean JS divergence 0.4683 versus 0.6869), although the gradient-alignment intervals include zero; we therefore claim target similarity rather than established gradient transfer. [Global Response 2] evaluates all checkpoints on the same 1,024 prefixes. JS divergence and reverse KL continue to decrease, while their fractional reduction per update is lower at steps 250–300 than at steps 50–100. [Global Response 3] specifies both training objectives and their clipping and replay rules; a matched comparison shows substantial one-shot gains and over 95% top-16 overlap under both objectives.

Reviewer NFn9 (rating 2; no response so far)

We added gradient-norm, teacher–student-gap, top-16-containment, and gradient-direction analyses, together with a fixed-step quadratic illustration, to examine the slowdown. We also added a BGE-M3 selector that requires no full-data OPD run; in the three-domain MOPD experiment, 16 selected queries per domain reach 52.9 accuracy versus 52.8 for full data at step 300. Held-out evaluations and a rollout audit found no measured degradation through step 1,000, while we now state explicitly that operation without a reasoning cue remains untested. The OPD/RLVR comparison, MOPD setting, and recovery percentages are qualified in the revision. We believe these additions address W1–W8 and Q1–Q10.

Reviewer 56jq (rating 1; no response so far)

We believe W1–W2 and Q1–Q2 are addressed by [Global Response 1] and [Global Response 2]. The matched-state results support closer teacher targets within clusters, while we explicitly state that gradient transfer is not established. The fixed-probe evaluation shows that the slowdown persists with fixed states, complete next-token distributions, and nonnegative divergences, without relying on the signed overlap-token diagnostic.

Reviewer hAxZ (rating 2; responded; rating remains 2)

We believe W1–W5 are addressed by [Global Response 1] through [Global Response 3], together with the consolidated query-variation results and 1,000-step runs. The reviewer agrees that the fixed-probe results confirm slower learning, but argues that they do not show sufficient task-useful supervision because remaining errors may require different data. We agree that the fixed-probe result alone does not answer that task-level question. We have narrowed the claim to the tested settings and clarified the methodological limits in the revised manuscript. Our response explicitly separates local supervision, parameter updates, and task usefulness, and states that the experiments do not causally determine whether residual benchmark errors are optimization-limited or would benefit from additional data.

Two reviewers have not responded to the final revisions. The responding reviewer maintained rating 2, with the remaining disagreement concerning the strength of the headline interpretation. We have narrowed the claim to the tested settings and clarified several methodological limits in the revised manuscript, while retaining our conclusion about the observed data coverage and absorption dynamics. We remain glad to run or clarify any further condition during the discussion period. We respectfully ask the AC to consider the possibility of AI-assisted review.

Sincerely,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.