acceptodds
Under review as a conference paper at ICLR 2027

Function-Space Temporal Self-Distillation

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) improves reasoning in language models, but scalar outcomes provide only coarse supervision over response trajectories. Self-distillation offers denser token-level guidance by comparing an unassisted policy with a feedback-informed self-teacher; however, because the teacher evolves alongside the student, its targets can become unstable, motivating additional regularization. We introduce Function-Space Temporal Self-Distillation (FTSD), a single-policy framework that combines current privileged adaptation with function-space temporal consolidation. FTSD stores sparse historical next-token distributions as fixed temporal targets, providing temporal separation without maintaining a persistent teacher model. It further incorporates privileged disagreement correction to suppress locally overvalued actions, reliability-adaptive grounding to strengthen reference supervision when self-generated evidence repeatedly fails, and quality- and coverage-aware temporal memory to retain verified behaviors and improve privileged demonstrations. Across reasoning and tool-use benchmarks, FTSD achieves the highest average accuracy under a 5-hour training budget, outperforming the strongest baseline by approximately 4 percentage points on average while reducing peak memory usage from about 70 GB to 56 GB on Olmo3-7B-Instruct.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning with Verifiable Rewards, Self-Distillation, Function-Space Distillation, Temporal Self-Distillation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Vpc4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FTSD, a self-distillation method that does not require maintaining an additional persistent teacher model. The same model serves as both student and teacher: the student sees only the original problem and the current response prefix, while the teacher additionally sees a verified solution demonstration and provides token-level supervision.

To prevent the teacher targets from continually changing as the student is updated, the method stores selected historical teacher output probabilities as fixed targets for subsequent training. FTSD also suppresses tokens that the student favors excessively relative to the teacher and increases the weight of the reference-solution cross-entropy loss when the model repeatedly fails to solve a problem. Historical memories are additionally selected and updated based on solution quality and coverage.

The authors evaluate the method on scientific reasoning and tool-use tasks. Under the same wall-clock training budget, FTSD achieves better average performance, reduces the GPU memory overhead associated with persistent model copies, and shows better retention on benchmarks assessing capabilities outside the training task.

Critical strengths
  1. Providing stable supervision by storing historical outputs has clear practical value.

    FTSD stores sparse historical teacher output distributions as fixed targets, preventing these targets from changing immediately with every parameter update. This provides historical supervision without requiring a persistent copy of the teacher parameters. The paper reports a reduction in peak GPU memory from approximately 70 GB to 56 GB on Olmo3-7B-Instruct. This is a valuable improvement for resource-constrained training.

  2. The full method achieves strong results and is evaluated from multiple perspectives.

    Across five tasks and two base models, FTSD outperforms the compared methods in average performance within a five-hour budget. The authors report not only performance on the new tasks but also retention of existing capabilities, and compare settings that remove individual losses, use different sources of teacher demonstrations, or employ EMA teachers, parameter snapshots, and standard CE replay. These experiments suggest that the components of the full method have complementary effects and provide a foundation for further analysis.

Critical weaknesses
  1. It remains unclear what additional information the same-model teacher provides after seeing a successful solution.

    I understand that the current teacher prioritizes another verified successful response and explicitly excludes the response that the student is currently learning from. Therefore, current teacher supervision should not simply be characterized as copying the same answer. However, historical targets are constructed differently: according to Equation (11) and the STORE procedure in the appendix, when storing a successful trajectory, the method can use that trajectory itself as the demonstration and compute teacher distributions along its own prefixes.

    In this setting, the teacher may become more confident because the demonstration activates reasoning knowledge already present in the model, or because the continuation already appears in the context, making the solution easier to reproduce. Both mechanisms may benefit training, but they lead to different interpretations of the method. Moreover, verification of a complete solution does not imply that every local judgment made by the teacher along an arbitrary prefix is more reliable; this is particularly relevant to the interpretation of suppressing tokens that the student overvalues.

    In addition, reference solutions enter training through multiple pathways, including fallback teacher demonstrations, memory initialization and historical targets, and reliability-adaptive CE. The fact that the method remains effective after removing CE in Table 2 does not rule out the influence of reference information through other pathways. The paper already compares demonstration sources and CE replay, but it does not sufficiently disentangle the gains from demonstration content, reference information, and fixed historical targets.

  2. Retention on other tasks is strong, but the results cannot yet be directly attributed to temporal memory.

    Figure 5 and Table 3 in the appendix do show that FTSD performs well on several benchmarks of existing capabilities. My concern is not the absence of forgetting evaluations, but why these improvements should be attributed to the temporal loss.

    The temporal loss primarily constrains problems and retained trajectories from the current training task; it does not directly replay the tasks used to evaluate existing capabilities. Furthermore, the stored historical targets come from a teacher that has seen additional demonstrations, rather than representing the pretrained model’s behavior on the original inputs. Preserving historical outputs on training problems therefore does not automatically amount to protecting existing capabilities on other tasks.

    Removing the temporal loss in Table 2 reduces new-task performance, but this does not establish its independent contribution to retention on other tasks. The full method also includes reference supervision, demonstration selection, and other training constraints. The current results do not sufficiently distinguish whether temporal memory mainly preserves behaviors learned early on the new task or actually reduces damage to general capabilities.

Questions for authors
  1. Does the teacher’s effectiveness mainly come from the reasoning process, the final answer, or textual cues in the demonstration?

    With other conditions held constant, could you compare demonstrations containing the full solution, only the final answer, and a paraphrased solution that preserves the reasoning content? This would help identify which information is primarily responsible for the teacher’s improvement.

    For the historical STORE procedure, I would also appreciate a comparison between using the retained trajectory itself as the demonstration and using another independently generated successful trajectory. When the demonstration is the very trajectory being stored, what evidence shows that the historical distribution provides guidance beyond repeated learning from that solution? For example, can it still provide useful corrections under different wording or erroneous prefixes, rather than merely predicting the next token of the original trajectory more accurately?

  2. Could you directly evaluate the contribution of the temporal loss to retaining capabilities outside the training task?

    While keeping the other components unchanged, could you also report existing-capability benchmark results for settings without the temporal loss, with standard CE replay, and with fixed historical-distribution replay? These comparisons should control for the training budget and, where possible, compare checkpoints with similar new-task performance.

    Such experiments would help answer whether more stable outputs on training problems actually correspond to less forgetting on other tasks. Does temporal memory primarily preserve solutions learned early on the new task, or does it also preserve behaviors useful for other capabilities? If the latter explanation holds, I would appreciate more direct evidence.

  3. What additional benefit does temporal memory provide when access to reference-solution information is held constant?

    I would like the authors to separately control the three pathways through which reference solutions enter training: fallback teacher demonstrations, memory initialization and the associated historical targets, and adaptive CE. This does not require changing the verifier used to judge correctness; rather, it requires controlling how reference-solution content enters the supervision.

    With these pathways held constant, how much additional benefit do fixed historical distributions provide over standard reference supervision, CE replay on the same trajectories, and teacher targets recomputed at each update?

    Figure 7 in the appendix already reports usage counts for different demonstration sources and changes in the losses, which is helpful. However, usage frequency or loss magnitude does not directly measure a component’s contribution to final performance. Could you further use controlled experiments that enable the relevant pathways only early or late in training to establish the roles of the three reference-solution pathways during cold start and subsequent training?

4 comments
AuthorsOpenReview ↗

Author Rebuttal (Part 1/3)

Dear Reviewer Vpc4,

We thank the reviewer for the detailed feedback and recognition of FTSD's practical value, memory efficiency, and experimental results. The key concerns raised by the reviewer concern privileged supervision, temporal consolidation, and reference grounding. We address these concerns below and will incorporate the corresponding clarifications and experimental findings into the revised manuscript.

W.1 Source of Privileged Information and Historical Target Construction & Q.1 Demonstration Content and STORE Procedure

Response

FTSD uses a single policy under two conditioning contexts: current supervision uses a distinct verified sibling (Eqs. 3–4), while historical STORE may self-condition (Eq. 11). This preserves temporally fixed privileged predictions without guaranteeing every local judgment, so Lneg\mathcal{L}_{\mathrm{neg}} uses margin-based correction. On Olmo3-7B-Instruct (Biology, Tool Use), we examine:

  • Demonstration content: Full solution, final answer only, and paraphrase (same reasoning, new wording).
  • Historical STORE: Self-conditioning versus a distinct verified trajectory (fallback for 38% / 22% of Biology / Tool Use problems).
  • Reference contribution: Removing LCE\mathcal{L}_{\mathrm{CE}} alone versus all reference-based supervision.
  • Unseen solutions and errors: Transfer to another verified solution and correction of errors.

Table: Demonstration content and historical target ablation on Olmo3-7B-Instruct (best avg@16 within 5 h, %).

CaseSettingBiology (%)Tool Use (%)Avg. (%)
Current teacherFull solution (ours)62.366.664.5
Current teacherFinal answer only58.963.561.2
Current teacherParaphrased verified solution61.865.963.9
Historical STORESelf-conditioned (ours)62.366.664.5
Historical STOREIndependent verified trajectory62.065.863.9
Reference pathwaysw/o LCE\mathcal{L}_{\mathrm{CE}}59.164.361.7
Reference pathwaysw/o reference-based supervision58.461.760.1

The token-level analysis uses a fixed checkpoint and STORE's prompt format on problems with ≥2 verified successes (yAy_A, yBy_B) and ≥1 failed rollout, removing the fallback. The teacher conditioned on yAy_A is scored on prefixes of the unseen yBy_B; an unrelated demonstration controls for added context, and conditioning on yBy_B is the copy reference.

Table: Token-level prediction of an unseen verified solution yBy_B.

PredictorContextBio NLL ↓Tool NLL ↓Bio Top-1 (%) ↑Tool Top-1 (%) ↑
Studentxx0.760.9181.876.4
Unrelated-problem demonstrationx⊕dotherx \oplus d_{\mathrm{other}}0.790.9580.775.9
Independent teacherx⊕yAx \oplus y_A0.670.7985.180.2
Self-conditioned (copy reference)x⊕yBx \oplus y_B0.150.2196.795.2

Table: Teacher behavior at the first erroneous decision of failed rollouts (wrong option; wrong tool call/argument) (%).

MetricBiology (%)Tool Use (%)
Teacher lowers pp (erroneous token) relative to student86.074.0
Teacher raises pp (verified alternative) relative to student82.067.0
  • Demonstration information: Answer only loses 3.3 points, paraphrasing 0.6, so solution-level reasoning, not surface wording, provides the privileged information.
  • Historical STORE: An independent trajectory reaches 63.9% vs. 64.5%, so temporal supervision need not show the retained trajectory (fallback retained here).
  • Reference influence: 61.7% without CE and 60.1% without all reference supervision confirm multiple reference pathways, as the reviewer notes (see Q.3).
  • Beyond copying: On the unseen yBy_B, the independent teacher lowers NLL from 0.76 to 0.67 (Biology) and 0.91 to 0.79 (Tool Use) and raises top-1 by 3.3 / 3.8 points, while the unrelated demonstration slightly degrades both. Closing 14.8% / 17.1% of the gap to the copy reference, privileged predictions transfer solution-level information to differently worded solutions, beyond added context.
  • Error correction: The teacher lowers the erroneous token in 86.0% / 74.0% of cases and raises the verified alternative in 82.0% / 67.0%, so privileged disagreement is directionally correct at genuine errors; the margin γ\gamma in N\mathcal{N} limits suppression to large disagreements.

Though near-copy at top-1, self-conditioned targets are not trajectory replay: cached distributions beat CE replay of the same trajectories (Table 2, Case 3: 64.5% vs. 58.5%), so the retained soft mass matters. Thus, privileged predictions provide transferable, solution-level guidance and correct erroneous decisions, not merely encoding the retained solution. We will extend this to perturbed-prefix recovery in the revision.

AuthorsOpenReview ↗

Author Rebuttal (Part 2/3)

W.2 Attribution of Knowledge Retention to Temporal Consolidation & Q.2 Effect of Temporal Loss on Existing Capabilities

Response

We agree that preserving historical predictions on training problems does not necessarily guarantee retention on unrelated tasks. While Ltemp\mathcal{L}_{\mathrm{temp}} preserves previously acquired privileged predictions through fixed historical targets (Eq. 13), FTSD does not explicitly replay examples from the existing-capability benchmarks.

To isolate this contribution, we extend the ablations on Olmo3-7B-Instruct across Biology and Tool Use, comparing function-space caching, w/o Ltemp\mathcal{L}_{\mathrm{temp}}, and CE replay of y~p\tilde{y}_p, while keeping the remaining components and training configurations unchanged. We evaluate retention across the same five prior-capability benchmarks in Table 3 at the 5-hour checkpoint and at approximately matched new-task accuracy. For matching, we select checkpoints within 0.5 percentage points of the weakest variant's new-task accuracy, separately for each training task.

Table: Effect of temporal consolidation on PRIOR-capability retention for Olmo3-7B-Instruct (mean accuracy across five benchmarks, %). Matched columns control for new-task accuracy.

SettingBiology (5h)Tool Use (5h)Biology (Matched)Tool Use (Matched)
Function-space cache (ours)60.959.561.060.2
w/o Ltemp\mathcal{L}_{\mathrm{temp}}59.057.759.458.6
CE replay of y~p\tilde{y}_p59.358.059.158.9
SDPO (on-policy, Table 3)59.658.7——

The results provide three observations:

  • Temporal contribution: At 5 hours, FTSD improves prior-capability retention over removing Ltemp\mathcal{L}_{\mathrm{temp}} by 1.9 points on Biology and 1.8 on Tool Use.
  • Beyond trajectory replay: Fixed historical distributions outperform CE trajectory replay by 1.6 and 1.5 points, respectively, suggesting that preserving privileged predictive information provides benefits beyond replaying retained solutions.
  • Matched new-task performance: FTSD maintains higher prior-capability retention than both alternatives even at comparable new-task accuracy, with improvements of at least 1.3 points across the two benchmarks.

These results provide direct evidence that temporal consolidation improves retention of capabilities outside the adaptation task, rather than merely improving new-task performance. While the mechanism is consistent with historical supervision constraining policy updates, the experiments do not establish that parameter drift is the sole explanation or guarantee retention across arbitrary domains. We will incorporate these controlled comparisons and clarify this distinction in the final revised manuscript.

Q.3 Reference Supervision Pathways and the Contribution of Temporal Memory

Response

We appreciate the reviewer's suggestion to disentangle reference supervision from temporal consolidation and examine their contributions throughout training. FTSD preferentially retains verified self-generated trajectories, while reference information enters through fallback demonstrations, memory initialization, and reliability-adaptive CE. We agree that the usage statistics in Figure 7 do not establish their independent contributions.

We first compare fixed historical distributions against recomputed privileged targets, alongside removing temporal consolidation and CE replay of the same retained trajectories. The fixed and recomputed variants preserve the remaining objectives, reference pathways, memory selection, and replay configurations, isolating the effect of temporal separation.

Table: Effect of historical supervision mechanisms on Olmo3-7B-Instruct (best avg@16 within 5 h, %).

SettingBiology (%)Tool Use (%)Avg. (%)
Function-space cache (ours)62.366.664.5
w/o Ltemp\mathcal{L}_{\mathrm{temp}}59.960.360.1
CE replay of y~p\tilde{y}_p58.758.358.5
Recomputed privileged targets60.262.461.3

Fixed targets outperform recomputed targets by 3.2 points (64.5% versus 61.3%), demonstrating the benefit of temporal separation beyond reference access and trajectory reuse. Removing temporal consolidation yields 60.1%, while trajectory-level CE replay achieves 58.5%, further supporting the value of historical privileged distributions. Unlike the immediate-refresh ablation (H=1H=1, Table 4, Case 10), which retains periodic staggered refresh, the recomputed variant updates privileged targets at every optimization step.

AuthorsOpenReview ↗

Author Rebuttal (Part 3/3)

Q.3 Reference Supervision Pathways and the Contribution of Temporal Memory (continued)

We further isolate the three reference pathways while preserving the verifier and remaining training configuration.

Table: Ablation of reference supervision pathways on Olmo3-7B-Instruct (best avg@16 within 5 h, %).

SettingBiology (%)Tool Use (%)Avg. (%)
Full objective (ours)62.366.664.5
w/o reference fallback demonstrations61.164.262.7
w/o reference memory initialization61.465.063.2
w/o LCE\mathcal{L}_{\mathrm{CE}}59.164.361.7

Each pathway contributes to final performance, with removing LCE\mathcal{L}_{\mathrm{CE}} producing the largest reduction (2.8 points), followed by reference fallback (1.8 points) and memory initialization (1.3 points).

To examine their temporal roles, we additionally restrict fallback demonstrations and CE supervision to the early (first 2.5 h) or late (remaining 2.5 h) phase. Reference memory initialization is separately tested through delayed seeding. We report final-checkpoint accuracy to ensure consistent comparisons among phase-specific variants.

Table: Phase-specific activation of reference pathways on Olmo3-7B-Instruct (FINAL-checkpoint avg@16 accuracy at 5 h, %).

SettingBiology (%)Tool Use (%)Avg. (%)
Full objective (ours)61.865.963.9
Reference fallback (early only)61.265.263.2
Reference fallback (late only)61.064.462.7
Reference memory initialization (delayed)61.164.963.0
LCE\mathcal{L}_{\mathrm{CE}} (early only)61.065.863.4
LCE\mathcal{L}_{\mathrm{CE}} (late only)58.964.161.5

The phase-specific results show that:

  • Early reference supervision is more beneficial: Early-only fallback and CE achieve 63.2% and 63.4%, respectively, compared with 62.7% and 61.5% when restricted to the late phase.
  • Memory initialization timing matters: Delaying reference seeding reduces final accuracy from 63.9% to 63.0%, supporting its role in establishing useful historical supervision early in training.
  • Temporal consolidation provides a distinct benefit: Fixed historical targets outperform recomputed targets even when reference pathways are held constant, indicating that the gains cannot be attributed solely to reference supervision.

Together, these findings support the complementary roles of early reference grounding and temporally fixed privileged supervision. We will incorporate these controlled comparisons and clarify their relationship to the training dynamics in Figure 7 in the final revised manuscript.

Happy to have further discussion!

AuthorsOpenReview ↗

A quick Follow-up

Dear, Reviewer,

Just a quick followup if there are any concern with the response.

Hope we addressed all the concern raised by the reviewer and it will help to raise the reviewers score.

We are happy to discuss if reviewer has any further concern.

Thanks

Reviewer_5gZaOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Function-Space Temporal Self-Distillation (FTSD), a framework designed to address the moving-target problem and limited reliability of self-teachers in reinforcement learning via self-distillation. FTSD stores sparse historical token distributions as fixed temporal targets, avoiding the need for a persistent teacher model. It further incorporates privileged disagreement correction, reliability-adaptive reference grounding, and quality- and coverage-aware temporal memory. Experiments on two LLM backbones across scientific reasoning and tool-use benchmarks demonstrate improved accuracy, faster convergence, reduced memory overhead, and better preservation of prior capabilities compared with GRPO and SDPO.

Critical strengths

-Storing historical teacher predictions in function space instead of maintaining a separate parameter-space teacher is a practical approach to mitigating moving-target instability while reducing memory overhead. The experimental results and ablation studies provide evidence supporting its effectiveness.

  • FTSD achieves approximately 4 percentage points higher average accuracy than SDPO across two LLM backbones and five benchmarks. The ablations further investigate the contributions of individual components, including temporal consolidation, reference grounding, and memory retention strategies, providing useful evidence for the proposed design.
Critical weaknesses
  • Although the ablation studies demonstrate the contributions of temporal consolidation and reliability-adaptive reference grounding within FTSD, it remains unclear how much of the improvement over SDPO comes from temporal consolidation versus the additional ground-truth CE supervision. Without comparing against SDPO equipped with the same reference grounding mechanism, it is difficult to isolate the benefit of the proposed temporal self-distillation approach.
  • The forgetting experiments primarily evaluate the retention of pretrained capabilities after adaptation to a single task, rather than sequential learning across multiple distinct tasks. Moreover, temporal memory mainly preserves historical predictions associated with the current training data, raising concerns about its ability to protect previously acquired knowledge that is absent from subsequent training stages. The impact of substantial task distribution shifts on teacher reliability and potential negative transfer also remains unexplored.
Questions for authors

See weaknesses above.

5 comments
AuthorsOpenReview ↗

Author Rebuttal (Part 1/2)

Dear Reviewer 5gZa,

We appreciate your encouraging assessment of FTSD, particularly its practical approach to mitigating moving-target instability, improving performance, and reducing memory overhead. Your comments raise important questions regarding the independent contribution of temporal consolidation and its effectiveness under sequential task adaptation. We address these concerns below and will incorporate the additional experimental findings and clarifications into the revised manuscript.

W.1 Isolating Temporal Consolidation from Reference Grounding

Response

We appreciate the reviewer's suggestion to equip SDPO with identical reference grounding to isolate its contribution from temporal consolidation. Table 2 (Case 1) already shows that FTSD without LCE\mathcal{L}_{\mathrm{CE}} achieves 61.7% average accuracy, while removing Ltemp\mathcal{L}_{\mathrm{temp}} reduces performance from 64.5% to 60.1%. To further disentangle these effects, we additionally evaluate SDPO with the same reliability-adaptive reference grounding (Eqs. 14–16), preserving its original objective and matching training configurations and wall-clock budgets.

Table: Effect of reliability-adaptive reference grounding on Olmo3-7B-Instruct (best avg@16 within 5 h, %).

SettingBiology (%)Tool Use (%)Avg. (%)
SDPO (on-policy)52.862.157.5
SDPO (on-policy) + LCE\mathcal{L}_{\mathrm{CE}}56.964.460.7
FTSD w/o Ltemp\mathcal{L}_{\mathrm{temp}}59.960.360.1
FTSD w/o LCE\mathcal{L}_{\mathrm{CE}}59.164.361.7
FTSD (ours)62.366.664.5

Adding reference grounding improves SDPO from 57.5% to 60.7%, confirming its importance. However, FTSD achieves a further 3.8-point improvement (64.5%) while remaining competitive without LCE\mathcal{L}_{\mathrm{CE}} (61.7%). Moreover, removing Ltemp\mathcal{L}_{\mathrm{temp}} reduces FTSD by 4.4 points, demonstrating the substantial contribution of temporal consolidation. Together, these comparisons indicate that reference grounding alone does not explain FTSD's improvements, while temporal consolidation provides a complementary benefit. We will incorporate this baseline and clarification into the final revised manuscript.

AuthorsOpenReview ↗

Author Rebuttal (Part 2/2)

W.2 Sequential Learning, Knowledge Retention, and Distribution Shifts

Response

We thank the reviewer for highlighting the distinction between single-task retention and sequential adaptation. Our existing analysis (Section 3.3, Appendix A.2) evaluates pretrained capability retention after individual task adaptation. Table 3 shows that after Tool Use training, FTSD retains 59.45% compared with 58.73% for SDPO and 54.79% for GRPO. To address sequential learning directly, we additionally evaluate Olmo3-7B-Instruct on Biology and Tool Use in both task orders, using matched per-stage training budgets. Stage 2 begins from the Stage-1 checkpoint, with historical FTSD memory carried forward. No new Stage-1 examples are sampled, although their retained trajectories and privileged targets remain available through temporal replay (Eq. 13). We evaluate both task-specific accuracy and retention across the five prior-capability benchmarks in Table 3.

Table: Sequential adaptation on Olmo3-7B-Instruct. Task accuracies use avg@16 (%); prior-capability scores average the five benchmarks in Table 3. All Stage-2 metrics are evaluated on the same checkpoint.

Task SequenceSettingStage 1 Task Acc.Stage 1 Prior-Cap.Stage 2 Task Acc.Stage 2 Task-1 Acc.Stage 2 Prior-Cap.
-Olmo3-7B-Instruct-61.2--61.2
Biology → Tool UseSDPO (on-policy)52.859.661.448.356.8
Biology → Tool UseFTSD (on-policy)62.360.965.959.159.3
Biology → Tool UseFTSD w/o Stage-1 memory62.360.964.157.558.1
Tool Use → BiologySDPO (on-policy)62.158.752.257.257.1
Tool Use → BiologyFTSD (on-policy)66.659.561.764.859.0
Tool Use → BiologyFTSD w/o Stage-1 memory66.659.559.861.457.9

The results provide three observations:

  • Pretrained capability retention: FTSD retains 59.3% and 59.0% after sequential adaptation, compared with 56.8% and 57.1% for SDPO in the respective task orders.
  • First-task retention: Biology accuracy decreases by 3.2 points for FTSD versus 4.5 for SDPO; Tool Use decreases by 1.8 versus 4.9 points. FTSD consequently retains higher first-task accuracy in both sequences.
  • Contribution of historical memory: Clearing Stage-1 memory reduces final prior-capability retention from 59.3% to 58.1% and from 59.0% to 57.9%. It also reduces first-task retention, supporting the contribution of carried-over historical targets beyond the remaining FTSD objectives.

For this sequential evaluation, we disable target refresh during Stage 2 to examine the contribution of fixed historical supervision; this differs from the standard staggered-refresh configuration. The consistent benefit of retaining Stage-1 memory in both task orders suggests that harmful transfer does not dominate in these settings, although these results do not directly measure historical teacher reliability. We will include this evaluation and clarify its scope in the final revised manuscript.

Happy to have further discussion!

Thank you for your comments. Can you upload the revised version of the manuscript so that I can see the updates? Or is this not possible at this stage?

Reviewer_HNLDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes FTSD to mitigate the moving target problem in SDPO. Instead of maintaining a persistent EMA teacher model, FTSD stores sparse historical next token distributions as fixed temporal targets. It further introduces privileged disagreement correction, reliability adaptive reference grounding, and quality and coverage aware temporal memory. Under a 5 hour wall clock training budget across scientific reasoning and tool use benchmarks, FTSD achieves ~4% average improvement over strong baselines (GRPO, SDPO), reduces peak memory consumption from 70 GB to 56 GB, and alleviates catastrophic forgetting.

Critical strengths
  1. Clear motivation addressing key pain points of existing on policy self distillation

In SDPO, both student and teacher originate from the same policy parameters. The teacher drifts continuously during optimization and creates a moving target problem; the EMA teacher also introduces non trivial memory overhead. Instead of storing full model weight snapshots, this work stores sparse next token probability distributions in function space as fixed temporal targets. The idea deviates from conventional target network / EMA paradigms and bears novelty.

  1. A holistic multi component framework rather than minor loss tweaks

The proposed pipeline consists of four core building blocks. A cyclic scheduling term \rho_s balances current step adaptation and historical consolidation. The whole design is self consistent.

  1. The paper conducts thorough experiments to demonstrate the effectiveness.

Evaluations are conducted on two backbones (Qwen3 8B, Olmo3 7B Instruct) and five benchmarks (Chemistry, Physics, Biology, Materials, ToolAlpaca), under realistic wall clock budgets (1 h / 5 h), instead of fixed optimization steps. The authors also conduct thorough ablation studies on the components and hyperparameters. I really appreciate the persuasive memory analysis which removes the EMA teacher cuts peak memory from 70 GB to 56 GB for the 7B model. Extrapolation to 6T parameter models demonstrates large scale practical benefits.

  1. Interesting empirical observations

FTSD produces shorter effective generated tokens with lower per step wall clock cost and faster convergence. It mitigates catastrophic forgetting during continued fine tuning, yielding both performance gain and memory reduction.

Critical weaknesses
  1. Ambiguous conceptual boundary for the core contribution

The notion of function space temporal self distillation is central to the paper but under explained in the main text. Readers may confuse it with prior continual learning work on function space regularization (e.g., Dark Experience Replay). The paper stores sparse top k token distributions instead of full output functions. One may question whether this is more accurately described as compressed historical logit replay. The distinction should be clarified in introduction and related work sections.

  1. Missing critical baseline comparisons.

A baseline with full historical distribution replay is absent. Case 3 in the appendix includes “CE replay of y_p”, which operates on trajectories only. There is no ablation that stores full historical teacher distributions without top k sparsification. It remains unclear whether performance gains stem from (i) fixed temporal targets per se, (ii) the sparse top k compression, or both. We cannot tell whether sparsity is required for performance or merely a memory saving trick.

  1. Strong component coupling with incomplete mechanistic explanations

Ablations show removing either L_temp or L_CE leads to noticeable performance drops, while L_distill alone performs poorly. Full performance requires all components to co activate. The paper mentions that aggressive local correction via L_neg works only under temporal stabilization, yet provides little mechanistic insight on why standalone L_neg fails.

Questions for authors

Q1: The paper claims function space temporal self distillation but only stores sparse top k token distributions instead of complete output functions. What performance impacts are introduced by the sparsity truncation? How would a full distribution variant perform?

Q2: Why does the privileged disagreement correction loss L_neg yield limited benefits in isolation and only becomes useful together with temporal consolidation? Please provide mechanistic reasoning.

Q3: Performance gains are substantially smaller on the tool use benchmark compared to scientific reasoning tasks. Could you explain this observation?

Q4: All primary results are obtained under a single random seed. Can you provide multi seed statistics to rule out the possibility that improvements originate from random fluctuation?

3 comments
AuthorsOpenReview ↗

Author Rebuttal (Part 1/2)

Dear Reviewer HNLD,

We sincerely thank the reviewer for their thoughtful and constructive feedback, and for recognizing the novelty of our function-space temporal targets, the coherence of the overall framework, and the thoroughness of our experimental evaluation and memory-efficiency analysis. We greatly appreciate the reviewer's insightful suggestions and address each concern point by point below. We will incorporate the corresponding clarifications and additional experimental results into the revised manuscript.

W.1 Ambiguous Conceptual Boundary for the Core Contribution

Response

We thank the reviewer for the careful framing of this boundary and the suggestion to contrast FTSD with prior function-space regularization such as Dark Experience Replay (DER). We agree that FTSD's storage can be viewed as privileged, compressed logit replay, and we will make this explicit. The key distinction lies not in storing historical outputs but in three properties:

  • The targets are privileged, meaning demonstration-conditioned rather than the policy's own unconditioned predictions.
  • These targets are temporally fixed across optimization steps and matched against the unconditioned student, capturing the influence of verified evidence while addressing the moving-target problem.
  • Quality- and coverage-aware retention with staggered refresh selectively preserves useful historical behaviors while allowing continued adaptation.

Additionally, top-KK sparsification retains the remaining probability mass as a tail category rather than renormalizing, providing efficient compression. We will expand the Introduction and Related Work to situate FTSD against DER and prior sparse-logit replay methods.

W.2 Missing Critical Baseline Comparisons & Q.1 Impact of Sparsification

Response

We are thankful to the reviewer's suggestion to add a full-distribution replay baseline, which provides a valuable way to separate fixed temporal targets from sparsification. The primary motivation of FTSD is to preserve temporally fixed privileged predictions without maintaining a persistent parameter-space teacher, while top-KK sparsification provides a compact representation of these historical targets.

Importantly, our formulation retains the remaining probability mass through an aggregated tail category rather than discarding it entirely. As demonstrated in Table 2 (Case 1), removing Ltemp\mathcal{L}_{\mathrm{temp}} reduces average accuracy from 64.5% to 60.1%, supporting the importance of temporal consolidation. We clarify that the existing CE replay baseline operates on retained trajectories rather than historical teacher distributions. To further isolate the effect of sparsification, we include an additional controlled comparison with full historical teacher-distribution replay, keeping all other training configurations, objectives, and memory selection mechanisms unchanged.

Table: Ablation of historical distribution sparsification on Olmo3-7B-Instruct.

SettingBiology (%)Tool Use (%)Avg. (%)
Function-space cache (ours)62.366.664.5
Function-space cache (full distribution)62.867.064.9
w/o Ltemp\mathcal{L}_{\mathrm{temp}}59.960.360.1

Full-distribution caching achieves slightly higher accuracy than our sparse formulation (64.9% vs. 64.5%), indicating that top-KK compression introduces only a modest performance trade-off while substantially reducing memory requirements. Both variants considerably outperform the setting without Ltemp\mathcal{L}_{\mathrm{temp}} (60.1%), suggesting that the primary gains arise from preserving fixed privileged targets rather than sparsification itself. We do not claim that sparsification independently improves accuracy; its principal benefit is memory efficiency. As reported in Appendix A.5 (Fig. 8), our top-Ktemp=32K_{\mathrm{temp}}=32 temporal cache reaches 942 mean entries and a 4.8 MB CPU memory footprint across five tasks, storing sparse teacher distributions with pooled tail mass alongside trajectory information. Full-distribution caching instead requires storing vocabulary-wide probabilities at each retained token position, substantially increasing memory requirements. Retaining the tail mass rather than renormalizing also avoids discarding residual probability mass. We will incorporate this comparison into Table 2 (Case 3), with the corresponding memory analysis in Appendix A.5 and conceptual clarification in Related Work.

AuthorsOpenReview ↗

Author Rebuttal (Part 2/2)

W.3 Component Coupling and Mechanistic Explanation & Q.2 Role of Disagreement Correction

Response

We thank the reviewer for suggesting a clearer mechanistic explanation. FTSD employs a single shared policy as both student and demonstration-conditioned self-teacher, so both prediction views evolve during optimization. The objectives serve complementary roles:

  • Lneg\mathcal{L}_{\mathrm{neg}}: Suppresses student-overvalued tokens (Eqs. 7–9), but does not determine how the suppressed probability mass should be redistributed.
  • Ltemp\mathcal{L}_{\mathrm{temp}}: Provides fixed historical privileged targets, stabilizing corrections against the evolving self-teacher.
  • LCE\mathcal{L}_{\mathrm{CE}}: Supplies reliable positive supervision when self-generated evidence is insufficient.

Without these stabilizing signals, local disagreement correction may produce inconsistent updates. Table 2 (Case 1) supports this explanation: Ldistill\mathcal{L}_{\mathrm{distill}} alone achieves 56.0%, while retaining Lneg\mathcal{L}_{\mathrm{neg}} without Ltemp\mathcal{L}_{\mathrm{temp}} and LCE\mathcal{L}_{\mathrm{CE}} yields 54.1%, with a larger degradation on Tool Use (57.2% versus 53.8%). Conversely, Lneg\mathcal{L}_{\mathrm{neg}} improves the stabilized full objective from 63.8% to 64.5%, whereas excessive correction (λneg=1.0\lambda_{\mathrm{neg}}=1.0) reduces accuracy to 60.8% (Table 4, Case 9). These results support its role as a targeted refinement rather than independent supervision. We will clarify this interaction in the revised manuscript.

Q.3 Smaller Performance Gains on Tool Use

Response

We appreciate the reviewer's observation. We emphasize that this is a highly challenging task, where most existing baseline methods struggle to achieve significant gains; therefore, it has limited room for performance improvement. Our observations are as follows:

  • Competitive performance: Table 1 shows FTSD matching the strongest baseline on Qwen3-8B (68.5%) and improving over the strongest Olmo3-7B-Instruct baseline (65.0% to 66.6%), including a 4.5-point gain over SDPO (62.1%).
  • Training dynamics: Figure 3 shows a slower initial improvement on Tool Use, consistent with the cold-start behavior observed on science tasks. By the 5-hour budget, FTSD matches or exceeds the strongest baselines on both models.
  • Temporal contribution: Table 2 (Case 1) shows that removing Ltemp\mathcal{L}_{\mathrm{temp}} reduces Tool Use accuracy by 6.3 points (66.6% to 60.3%), compared with 2.4 points on Biology (62.3% to 59.9%), indicating a particularly strong contribution from temporal consolidation on Tool Use.

Thus, even modest improvements over competitive baselines are meaningful given the limited room for further performance gains on this challenging dataset. We believe these improvements highlight the effectiveness and contribution of our proposed temporal supervision. We welcome further discussion and would be happy to address any additional suggestions or concerns raised by the reviewer.

Q.4 Multi-Seed Training and Statistical Reliability

Response

We thank the reviewer for suggesting multi-seed evaluation to assess statistical reliability. Table 1 reports single-seed improvements of 9.5 percentage points on Biology and 4.5 percentage points on Tool Use over SDPO. To verify their consistency, we conduct 3-seed training of both FTSD and SDPO on Olmo3-7B-Instruct under matched training conditions, reporting mean and standard deviation of avg@16 accuracy at the 5-hour budget.

Table: 3-seed comparison of FTSD and SDPO on Olmo3-7B-Instruct (avg@16 accuracy, mean ± std, %).

DatasetMethod5h (%)
BiologySDPO52.1 ± 1.4
BiologyFTSD62.4 ± 0.7
Tool UseSDPO62.0 ± 0.8
Tool UseFTSD66.3 ± 1.2

Across three seeds, FTSD consistently achieves higher mean accuracy than SDPO, with improvements of 10.3 percentage points on Biology and 4.3 percentage points on Tool Use. The relatively small standard deviations indicate stable performance across runs, supporting the reproducibility of the reported gains rather than their dependence on a particular random seed. We will incorporate these multi-seed results into the final revised manuscript.

Happy to have further discussion!

AuthorsOpenReview ↗

A quick follow-up

Dear, Reviewer,

Just a quick followup if there are any concern with the response.

Hope we addressed all the concern raised by the reviewer and it will help to raise the reviewers score.

We are happy to discuss if reviewer has any further concern.

Thanks

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.