acceptodds
Under review as a conference paper at ICLR 2027

Know Thyself, Teach Thyself: Internal Information Flow for Selective Self-Distillation

Abstract

Self-distillation turns knowledge distillation into a closed learning loop and offers a path toward recursive self-improvement. Without an external teacher, however, the model must determine both what information can improve its supervision and which induced changes should be learned. Existing methods typically improve teacher-generated data or select training examples in isolation, leaving the information transferred between these stages unmeasured. We introduce InFlow, a retrieval-guided on-policy self-distillation framework that models this process as potential-to-realized information flow. InFlow first retrieves potentially informative sources using certainty-calibrated hidden-state trajectories, then measures their realized effect through the Jensen–Shannon divergence between the teacher's initial and retrieval-conditioned answer beliefs. Examples with larger belief shifts are selected for on-policy distillation. Our analysis formalizes the information optimized by retrieval and selection and relates the answer-level shift to the teacher–student distillation gap. Across four open-weight language models and three knowledge domains, InFlow achieves the strongest cross-model average among the compared selection methods, with ablations supporting both stages of the framework.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, self-distillation, on-policy distillation, data selection, internal representations, information theory

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_7TEhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work investigates self-distillation of language models without annotations or a strong external teacher. It studies how the model selects retrieved examples to augment the teacher and which augmented supervision signals are suitable for training. The authors propose INFLOW, which consists of three stages. First, it retrieves examples based on the similarity of cross-layer hidden state trajectories and answer confidence. Second, it measures belief shift using the Jensen–Shannon divergence between answer distributions obtained with and without retrieval, and selects targets with the largest shifts. Third, it performs on-policy reverse-KL distillation using an EMA teacher equipped with retrieved context. Experiments show that INFLOW slightly outperforms baseline methods on average across models and achieves consistent improvements on mathematical reasoning tasks.

Critical strengths
  1. The perspective of post-retrieval selection is practically valuable. Instead of selecting data purely based on pre-generation similarity or confidence, INFLOW measures whether retrieved context actually alters the teacher’s answer distribution. This approach is more direct than relying only on pre-retrieval proxy signals and may inspire future work on data selectors.
  2. It delivers consistent gains on mathematical tasks. INFLOW achieves notable improvements over the original model on mathematical benchmarks across four backbones. The Fix/Break analysis in Appendix C.2 and case studies in Appendix F cover scenarios including correct-to-correct, wrong-to-correct, correct-to-wrong and wrong-to-wrong transitions. These results clearly illustrate the effective range and failure modes of the method.
Critical weaknesses
  1. Pi0P_i^0 uses one sample without retrieval while PiSP_i^S uses five retrieved samples. As a result, JS divergence may capture both retrieval effects and ordinary sampling variance.
  2. The ablation study is not sufficiently complete. The paper lacks comparisons between final-layer cosine similarity and trajectory similarity, as well as between trajectory similarity alone and trajectory similarity combined with certainty. These comparisons are needed to demonstrate the importance of trajectory information and whether certainty brings additional benefits.
  3. There is no comparison between JS-based selection and binary hard-flip selection. It remains unvalidated whether continuous JS values carry more information than simple binary answer flips.
Questions for authors
  1. From which token position of the causal LM are the logits of the answer distribution extracted? Are they taken at the answer token position or the token immediately preceding the answer token?
  2. In Appendix F, answer probabilities frequently approach 1 regardless of answer correctness. Does this observation imply that the measured belief shift mostly reflects format consistency or hard answer flips rather than the model’s genuine uncertainty about the problem?
  3. How do the authors handle tied rankings when many samples yield a JS value exactly equal to 0 or log2log 2? Could the full distribution of JS scores be reported?
  4. Would the main findings still hold if both the non-retrieval and retrieval conditions use five paired random samples? This control experiment is critical to rule out the influence of ordinary decoding randomness.
  5. Could the authors provide direct comparisons of final-layer cosine similarity, trajectory similarity, and trajectory plus certainty under the same JS selector setting? Also please compare JS selection against binary answer-flip selection. These results will determine whether cross-layer trajectories, certainty and continuous JS values each contribute extra performance gains.
3 comments

Thank you for the suggestions on the controlled ablations.

W1 and Q4. Sampling variation. We have added 1/1 and 5/5 controls, where the two numbers denote samples before/after retrieval. Retrieved contexts, decoding settings, and training settings are fixed. The table reports Avg@3 ± SD (%) on Ministral-3-8B. The complete line chart is shown in the revised Figure 3. These runs test sample-count sensitivity; the paired-sampling control requested in Q4 have not yet been verified due to time constraints.

Variant100 updates200 updates300 updates400 updates500 updates
1/1 JS61.00 ± 0.1661.27 ± 0.8461.40 ± 0.8661.87 ± 0.6661.07 ± 0.81
1/5 JS (INFLOW)61.81 ± 1.4161.87 ± 0.1961.67 ± 0.7563.47 ± 0.1862.34 ± 0.62
5/5 JS61.20 ± 1.3161.93 ± 1.0962.80 ± 0.0063.40 ± 0.6563.00 ± 1.02

W2 and Q5. Trajectory and certainty. We have added final-layer retrieval with and without certainty under the same JS selector. The four variants below directly test representation choice and the contribution of certainty. They share the candidate pool and training settings. The complete line chart is shown in the revised Figure 3.

Variant100 updates200 updates300 updates400 updates500 updates
Trajectory + certainty61.81 ± 1.4161.87 ± 0.1961.67 ± 0.7563.47 ± 0.1862.34 ± 0.62
Trajectory without certainty61.27 ± 0.0963.07 ± 0.6162.74 ± 0.5061.74 ± 0.7362.20 ± 1.13
Final layer + certainty61.73 ± 0.6862.73 ± 0.1962.93 ± 0.5063.20 ± 1.2362.80 ± 0.85
Final layer without certainty61.73 ± 0.6263.13 ± 1.9562.53 ± 0.5762.73 ± 0.2562.47 ± 0.19

W3 and Q5. JS versus hard flips. We have added hard-flip selection using 1[arg⁡max⁡P0≠arg⁡max⁡PS]\mathbf 1[\arg\max P^0\ne\arg\max P^S], with random tie-breaking independent of JS. Retrieved contexts and training are identical. The comparison tests whether continuous JS ranking adds value beyond detecting a change in the most likely answer. More complete ablation results are provided in the revised Figure 3, Sec. 4.3, and Appendix C.

Variant100 updates200 updates300 updates400 updates500 updates
1/5 JS (INFLOW)61.81 ± 1.4161.87 ± 0.1961.67 ± 0.7563.47 ± 0.1862.34 ± 0.62
1/5 hard flip61.93 ± 0.5262.33 ± 0.2561.60 ± 1.0262.07 ± 0.3862.93 ± 0.74
Random 20%60.27 ± 0.8261.47 ± 1.4262.27 ± 0.4663.20 ± 0.5962.00 ± 0.59

Q1. Logit position. Answer probabilities use logits at t−1t-1, immediately before the answer token at tt; retrieval representations are extracted at tt. We have clarified both positions and normalization over legal option tokens in Secs. 3.1 and 3.3.

Q2. Near-unit probabilities and hard flips. The generated result prefix may already state the chosen option, making its answer letter nearly deterministic even when wrong. These probabilities therefore measure prefix-conditioned prediction, not calibrated reasoning confidence. When the distributions approach point masses, JS does approach a hard-flip distinction. We have clarified this and added the direct hard-flip comparison above in Sec. 4.3.

Q3. Ties and score distribution. We rank unrounded JS scores and break exact ties by random selection, as now specified in Appendix C.2. The requested score distribution is reported by decile in Appendix C.3.

Other modifications. Although you did not raise this point, we also redrew Figure 1 to make it more intuitive because the original relied too heavily on text to explain each step. If you have time to look at the revised diagram, we would welcome any further questions or ambiguities you notice.

Reviewer_7TEhOpenReview ↗

reply to author

The author addressed most of my questions, so I’ve decided to keep my rating.

Thank you for your valuable suggestions and recognition. Wishing you continued success in your research!

Reviewer_37Y3OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses a potential limitation in LLM performance caused by over-consistency of language representation. InFlow retrieves similar examples from an unlabeled dataset using hidden-state embeddings and perturbs the teacher's predictions by providing the retrieved examples as additional context. By selecting training samples based on the difference between the original and retrieval-conditioned predictions, the method aims to enhance reasoning performance through self-distillation. The paper reports the improvements on reasoning benchmarks, ablation studies, and theoretical analyses.

Critical strengths
  1. The problem the paper aims to solve is clear.
  2. The proposed methodology is straightforward and mathematically describable.
  3. The main experiment results cover a wide range of domain, baselines, and LLMs.
  4. Ablation studies section includes helpful informations for understanding InFlow's behavior.
Critical weaknesses
  1. Weak justification for realized information. The paper defines realized information using the JS divergence between the teacher's predictions before and after retrieval, and directly treats it as a measure of informative supervision. However, a large difference can indicate either improvement or degradation, since JS divergence measures the magnitude of a belief shift rather than its correctness. Although the authors acknowledge this limitation and provide an analysis in Appendix C.2, acknowledging it does not resolve the fundamental issue with the proposed selection criterion. In particular, the analysis on Llama-3.1-8B shows that the selected examples can contain more harmful than corrective belief shifts. Further justification is needed to explain why maximizing JS divergence is an appropriate objective for selecting useful supervision.
  2. Weak ablation study. I agree that JS divergence is a good design choice and even it is intuitive. However, the paper reports the comparison with random selection. Technically, random baseline is a lowest boudary in ablation study. More baselines should be compared as long as the paper provides ablation study on example selection algorithm.
  3. Over-claimed performance gains. Table 1 provides performance comparisons across domains, LLMs, and existing algorithms. Although the paper emphasizes the strongest cross-model average, InFlow outperforms the strongest competing method on only two of the four backbones, with marginal gains. The paper should further analyze the backbone-dependent behavior to explain when and why the proposed selection criterion is effective or narrow the claim.
  4. Limited benefits in efficiency. Although InFlow improves performance over the original models, the gains over competing methods are marginal. Considering the additional complexity of retrieval and example selection, it is unclear whether these improvements justify using InFlow for on-policy distillation.
Questions for authors
  1. What explains the inconsistent performance gains across different LLM backbones? In particular, is the effectiveness of InFlow related to the quality of retrieved examples or the reliability of the teacher's confidence estimates?
  2. Could the authors provide a breakdown of the computational costs of retrieval, belief-shift estimation, and distillation, and compare the methods under a fixed wall-clock budget?
5 comments

Thank you for the comments on the selection criterion and computational cost.

W1. Justification for realized information. We apologize that our use of “realized” may have caused confusion. Proposition 1 shows that, under the stated assumptions and fixed budgets, INFLOW maximizes the potential information supplied by retrieval and the realized intervention information retained for distillation. Here, “realized” refers to information expressed in the teacher’s answer distribution after retrieval augmentation. Because the setting is annotation-free self-distillation, this additional information is not guaranteed to be beneficial: it can help correct existing errors or reinforce them, as characterized by the theoretical analysis in Sec. 5.2 and the statistics in Appendix C.3. We have clarified this terminology in Secs. 3.3 and 5.1.

W2. Selection baselines beyond Random. We have added hard-flip selection, scored by 1[arg⁡max⁡P0≠arg⁡max⁡PS]\mathbf 1[\arg\max P^0\ne\arg\max P^S], with random tie-breaking independent of JS. It shares the retrieved contexts, 1/5 beliefs, 20% selection ratio, and training settings with INFLOW. This directly compares continuous JS ranking with binary answer changes. The table reports Avg@3 ± SD (%) on Ministral-3-8B; full curves appear in the revised Figure 3.

Variant100 updates200 updates300 updates400 updates500 updates
1/5 JS (INFLOW)61.81 ± 1.4161.87 ± 0.1961.67 ± 0.7563.47 ± 0.1862.34 ± 0.62
1/5 hard flip61.93 ± 0.5262.33 ± 0.2561.60 ± 1.0262.07 ± 0.3862.93 ± 0.74
Random 20%60.27 ± 0.8261.47 ± 1.4262.27 ± 0.4663.20 ± 0.5962.00 ± 0.59

W3 and Q1. Backbone-dependent performance. Our original claim concerned the highest Overall accuracy averaged across the four backbones on multiple-choice tasks, rather than the best performance on every backbone. We have revised the wording to remove this ambiguity. INFLOW improves all four unadapted checkpoints, while the strongest method varies by model and domain. The open-ended results below report Overall EM ±\pm SD (%) at 250 updates: INFLOW achieves the best scores among the compared methods on Ministral and Llama, while NEURON leads on both Qwen backbones. Per-task results are in Table 2.

MethodQwen3-4BQwen3.5-9BMinistral-3-8BLlama-3.1-8B
Original50.46 ± 0.5274.07 ± 0.2568.33 ± 0.6270.33 ± 0.66
Full51.00 ± 0.8674.80 ± 0.1670.53 ± 0.9371.33 ± 0.77
Redundancy51.60 ± 0.7574.73 ± 0.4170.00 ± 0.5971.40 ± 1.04
Consistency50.53 ± 0.3474.00 ± 0.5968.40 ± 0.7170.47 ± 0.62
NEURON53.07 ± 0.7575.33 ± 0.6670.93 ± 0.6870.93 ± 1.20
Mutual48.60 ± 0.5974.40 ± 0.4369.73 ± 0.5069.27 ± 0.34
INFLOW51.73 ± 0.7774.87 ± 0.3371.33 ± 0.7771.53 ± 0.57

W4 and Q2. Computational cost and fixed-time comparison. We have added the following cost breakdown for Ministral-3-8B on an RTX 4090 at 250 updates. Values are in minutes except selection, which is in seconds. INFLOW scoring includes predictive entropy and semantic clustering. Shared preparation is charged to each method using it. Total adaptation includes loading/setup and excludes evaluation. Fixed-wall-clock accuracy results are still pending due to time constraints.

MethodInitial prep.Features / retrievalTeacher generationScoringSelection (s)DistillationLoading / setupTotal adaptationEvaluation
Full2.864.3523.0317.530.05424.430.4772.677.46
Redundancy2.864.3523.0317.530.15822.480.4570.717.53
Consistency0.000.0019.884.250.01717.670.4542.257.76
NEURON2.8618.8121.3715.770.02019.270.4578.527.25
Mutual2.860.8522.4017.120.02023.550.5067.287.97
INFLOW2.860.8522.3721.320.02624.570.4572.427.45

Other modifications. Although you did not raise this point, we also redrew Figure 1 to make it more intuitive because the original relied too heavily on text to explain each step. If you have time to look at the revised diagram, we would welcome any further questions or ambiguities you notice.

Reviewer_37Y3OpenReview ↗

Response to the authors

Thank you for the authors' response.

I acknowledge that the additional hard-flip baseline helps address my concern about the ablation study.

However, my two remaining concerns have not been sufficiently resolved.

  1. As the authors have acknowledged, a larger belief shift does not necessarily indicate more beneficial supervision. Although I find the theoretical analysis interesting, I remain unconvinced that maximizing JS divergence is sufficiently justified as a criterion for selecting useful training examples. I would like the authors to provide a mathematical justification, even under simplifying assumptions, for why maximizing JS divergence is expected to improve the quality of supervision.
  2. A fixed wall-clock budget is a standard experimental control when evaluating computational efficiency. Since the authors have not provided such a comparison, I remain unconvinced that the performance gains justify the additional computational cost.

Therefore, I maintain my rating for now.

I would appreciate further clarification on why larger belief shifts should lead to more effective supervision, and whether the reported gains remain meaningful under comparable computational budgets.

Thank you very much for your constructive feedback and for acknowledging that our hard-slip control successfully addresses the ablation concern. We also greatly appreciate your suggestion to establish a more direct mathematical connection between the Jensen-Shannon divergence and supervision utility.


1. Why does maximizing JS support supervision?

As briefly explained in the previous rebuttal, within a framework where questions lack existing reliable annotations or hints and require autonomous exploration and self-improvement via related retrieval, a method that guarantees consistently beneficial and accurate supervision simply does not exist. Our central motivation is to introduce, filter, and maximize internal information flow toward signals the model considers credible in annotation-free settings. It intuitively drives the largest belief shift in a credible direction under a fixed budget. While Section 5.1 establishes this information-flow maximization principle, we can further formalize this intuition by connecting it to supervision utility under a mild posterior assumption.

Let pp and qq denote the initial and retrieval-conditioned answer distributions, respectively. If qq represents the correct-answer posterior given the contextual evidence, the expected reduction in logarithmic prediction loss satisfies:

2\mathrm{JS}(p\|q)+2\mathrm{KL}\left(\frac{p+q}{2}\middle\|p\right) \geq 2\mathrm{JS}(p\|q).$$ > **Key Takeaway:** At a fixed target budget, top-JS selection maximizes a lower bound on the expected supervision gain under this assumption, providing a rigorous justification beyond measuring belief change alone. This argument also gracefully extends to imperfect teachers and distillation. Let $g$ be the correct-answer posterior and $r$ be the post-distillation student distribution. Then: $$\begin{aligned} G_{\mathrm{student}} &:=\mathbb{E}_{Y\sim g}\left[\log\frac{r(Y)}{p(Y)}\right]\\ &\geq 2\mathrm{JS}(p\|q) -\mathrm{KL}(q\|r) -2L\mathrm{TV}(g,q), \end{aligned}$$ where $L=\max_a|\log(r(a)/p(a))|$. This bound clearly identifies the necessary conditions for useful supervision: with a uniform bound on these errors, top-JS selection also maximizes the corresponding aggregate utility lower bound. This elegantly explains both our original motivation for JS and why harmful distribution shifts remain possible. --- ### 2. Fixed Wall-Clock Budget Comparison Thank you for raising this important point. To directly address your concern regarding computational efficiency, we have conducted a controlled fixed wall-clock budget experiment on Ministral-3-8B under identical hardware settings. **Experimental Protocol:** * **Unified Time Budget:** We set the total end-to-end adaptation time of INFLOW at 250 updates (`72.42 min`) as the strict wall-clock budget. This total time accounts for **all pipeline stages**, including data preparation, retrieval, initial and teacher response generation, scoring, and distillation updates (excluding evaluation time). * **Stopping Criterion:** For all baselines, training is terminated immediately at the update step where their cumulative end-to-end wall-clock time exceeds INFLOW's budget. *(Note: For **Full**, reaching 250 updates takes `72.67 min`—virtually identical to INFLOW's `72.42 min`—so we directly report its existing 250-update checkpoint).* | Method | Stopping Update | HotpotQA EM±SD | CoQA EM±SD | SQuAD EM±SD | Overall EM±SD | | :--- | :---: | :---: | :---: | :---: | :---: | | **Full** | 250 | 76.00 ± 0.82 | 62.89 ± 0.83 | 70.89 ± 1.66 | 70.53 ± 0.93 | | **Redundancy** | 270 | 75.50 ± 0.82 | 62.22 ± 0.90 | 70.67 ± 1.85 | 70.07 ± 0.58 | | **Consistency** | 619 | 74.33 ± 0.91 | 59.78 ± 0.55 | 69.33 ± 2.38 | 68.47 ± 0.80 | | **NEURON** | 231 | 75.17 ± 0.76 | 63.56 ± 1.87 | **72.00 ± 1.29** | 70.73 ± 0.69 | | **Mutual** | 273 | 75.67 ± 1.10 | 61.33 ± 0.68 | 70.67 ± 2.09 | 69.87 ± 0.54 | | **INFLOW (Ours)** | **250** | **76.83 ± 0.85** | **64.44 ± 2.06** | 70.89 ± 1.57 | **71.33 ± 0.77** | --- We sincerely appreciate your continued engagement and recognition of our work. We hope this additional analysis addresses your remaining concerns, and we remain happy to provide any further clarification if needed.
Reviewer_ABTnOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes INFLOW, a retrieval-guided self-distillation method that combines demonstration retrieval, training data selection, and on-policy self-distillation. For each question, demonstrations are retrieved from the unlabeled question pool based on the similarity of layerwise hidden-state trajectories and the certainty of model-generated answers. Training samples are then selected according to the Jensen–Shannon divergence between the initial and retrieval-conditioned answer distributions. The selected samples are subsequently used for on-policy self-distillation from a retrieval-conditioned EMA teacher. Thorough experimental evaluation and theoretical analyses are provided to support the proposed method.

Critical strengths
  1. The proposed method is generally clear to follow.
  2. The method is supported by theoretical analyses.
  3. The method doesn't involve too much computational overhead compared to simialr methods.
Critical weaknesses
  1. The method is only applied to multiple-choice tasks. It's not sure how it can be generalized to open-ended generation and how well it performs in that case.
  2. In equation 10, the objective takes an expectation with respect to the student model distribution pθp_{\theta}, which itself depends on the parameters being optimized. Therefore, differentiating the stated objective generally requires accounting for the dependence of the sampling distribution on θ\theta, e.g., through a score-function/policy-gradient term. However, the proposed Algorithm 1 doesn't seem to account for that. This discrepancy necessitates either changing equation 10 or changing the optimization algorithm.
Questions for authors
  1. In equation 10, the teacher's distribution conditions on the retrieved set based on teacher's generation but the prefix of student's generated text. Does this mismatch make sense? Would it be better to condition on a new retrieved set based on student's generation?
  2. In Table 5, "full" and "redundancy" baselines do retrieval only base on the final-layer answer representations (while the proposed method uses the trajectory over all layers), and the "consistency" baseline is not even retrieval-based. The former makes the comparison seem unfair and the latter is comparing between methods with totally different principles. Since the proposed method has multiple components, why not fixing one and comparing the other against methods with a similar principle?
4 comments

Thank you for the careful reading and constructive questions.

W1. Open-ended generation. We have added HotpotQA for multi-hop natural language logical reasoning, CoQA for conversational multi-turn QA, and SQuAD for reading comprehension analysis, using 200/150/150 questions in each 500-question split. HotpotQA uses disjoint training and held-out samples from train-medium. We judge correctness via exact keyword matching against the golden answers from the datasets. All trainable methods use 250 updates. All task columns report EM ± SD (%), with SD computed across three evaluation rollouts (Sec. 4.2 and Appendix C). Due to time constraints, we omitted ablation studies on the update count for open-ended questions, yet we present a fine-grained breakdown of the time cost across all modules of our approach.

The open-ended extension keeps one initial answer and five retrieval-conditioned answers. The initial answer supplies the retrieval representation and certainty exp⁡(−Hˉ0)\exp(-\bar H_0). Here Hˉ0\bar H_0 is length-normalized next-token predictive entropy. For INFLOW alone, an LLM jointly assigns the six answers to semantic-equivalence classes. JS is computed in this shared semantic space. We also update Appendix A.2 from its sequence-likelihood weighting to this predictive-entropy weighting and specify the joint clustering prompt.

Qwen3-4B

MethodHotpotQACoQASQuADOverall
Original56.83 ± 0.2436.22 ± 1.3756.22 ± 1.1350.46 ± 0.52
Full58.50 ± 0.4136.00 ± 1.4456.00 ± 1.0951.00 ± 0.86
Redundancy58.83 ± 0.4736.44 ± 1.9157.11 ± 0.3151.60 ± 0.75
Consistency57.33 ± 0.2435.56 ± 0.8356.44 ± 0.6350.53 ± 0.34
NEURON58.17 ± 0.8540.22 ± 1.1359.11 ± 0.3153.07 ± 0.75
Mutual55.17 ± 0.4735.33 ± 1.9653.11 ± 0.3148.60 ± 0.59
INFLOW59.17 ± 0.2436.22 ± 1.7557.33 ± 1.0951.73 ± 0.77

Qwen3.5-9B

MethodHotpotQACoQASQuADOverall
Original80.67 ± 1.0361.11 ± 1.3778.22 ± 0.8374.07 ± 0.25
Full81.50 ± 0.7162.00 ± 1.0978.67 ± 0.0074.80 ± 0.16
Redundancy81.00 ± 0.7162.44 ± 1.3778.67 ± 0.9474.73 ± 0.41
Consistency80.50 ± 0.7160.67 ± 1.8978.67 ± 0.5474.00 ± 0.59
NEURON81.17 ± 0.2463.56 ± 0.3179.33 ± 1.9175.33 ± 0.66
Mutual81.17 ± 1.0360.89 ± 1.5778.89 ± 1.1374.40 ± 0.43
INFLOW81.33 ± 0.8562.67 ± 1.9178.44 ± 0.8374.87 ± 0.33

Ministral-3-8B

MethodHotpotQACoQASQuADOverall
Original74.00 ± 0.7160.22 ± 0.6368.89 ± 2.4568.33 ± 0.62
Full76.00 ± 0.8262.89 ± 0.8370.89 ± 1.6670.53 ± 0.93
Redundancy75.33 ± 0.8562.44 ± 0.8370.44 ± 1.9170.00 ± 0.59
Consistency74.17 ± 0.8560.00 ± 0.5469.11 ± 2.4568.40 ± 0.71
NEURON75.50 ± 0.7164.00 ± 1.9671.78 ± 1.2670.93 ± 0.68
Mutual76.00 ± 1.0860.89 ± 0.6370.22 ± 2.2069.73 ± 0.50
INFLOW76.83 ± 0.8564.44 ± 2.0670.89 ± 1.5771.33 ± 0.77

Llama-3.1-8B

MethodHotpotQACoQASQuADOverall
Original70.83 ± 1.0358.89 ± 0.3181.11 ± 1.7570.33 ± 0.66
Full72.50 ± 1.0860.67 ± 0.0080.44 ± 1.3771.33 ± 0.77
Redundancy72.67 ± 1.7859.11 ± 0.8382.00 ± 1.9671.40 ± 1.04
Consistency71.33 ± 0.8558.67 ± 1.6381.11 ± 1.7570.47 ± 0.62
NEURON70.67 ± 1.4359.11 ± 1.2683.11 ± 1.6670.93 ± 1.20
Mutual71.67 ± 1.1858.44 ± 0.3176.89 ± 0.3169.27 ± 0.34
INFLOW73.50 ± 0.4159.56 ± 0.8380.89 ± 1.7571.53 ± 0.57

W2. The gradient in Eq. (10) and Algorithm 1. Thank you for pointing this out. Algorithm 1 did not specify the differentiation steps precisely enough. In the implementation, we sample a fresh student trajectory and compute the token-level KL loss in Eq. (10) on its prefixes, treating the sampled trajectory as fixed during backpropagation. We have revised Algorithm 1 and made this stop-gradient convention explicit alongside Eq. (10) to clarify the optimization procedure.

Q1. Fixed retrieval context and student-generated prefixes. The retrieved set supplies question-level evidence. Both models are evaluated at the same student prefix, so the teacher guides states the student actually visits. Keeping the evidence fixed preserves the retrieval intervention used to select that target. Re-retrieving at each rollout would introduce an adaptive context policy with a different selection signal and additional cost. We will make the distinction between question-level evidence and token-level supervision explicit in Sec. 3.4. (Sec. 3.4 and Algorithm 1)

Q2. Baseline fairness and component attribution. We have adapted the baselines to the same self-distillation framework while preserving their core principles. Full and Redundancy use representation-based retrieval, whereas Consistency selects by answer agreement without retrieval. These differences preserve the methods’ intended designs. All trainable methods share the sampling settings, update count, batch size, LoRA configuration, and loss; Full retains all targets, while the selectors retain 20%. To analyze the difference between last-layer-based representation similarity (Full, Redundancy) and our trajectory-based measurement, we have added final-layer retrieval ablations with and without certainty under the same JS selector. More complete ablation results are provided in the revised Figure 3.

Variant100 updates200 updates300 updates400 updates500 updates
INFLOW: trajectory + certainty61.81 ± 1.4161.87 ± 0.1961.67 ± 0.7563.47 ± 0.1862.34 ± 0.62
Trajectory without certainty61.27 ± 0.0963.07 ± 0.6162.74 ± 0.5061.74 ± 0.7362.20 ± 1.13
Final layer + certainty, 1/5 JS61.73 ± 0.6862.73 ± 0.1962.93 ± 0.5063.20 ± 1.2362.80 ± 0.85
Final layer without certainty, 1/5 JS61.73 ± 0.6263.13 ± 1.9562.53 ± 0.5762.73 ± 0.2562.47 ± 0.19

Other modifications. Although you did not raise this point, we also redrew Figure 1 to make it more intuitive because the original relied too heavily on text to explain each step. If you have time to look at the revised diagram, we would welcome any further questions or ambiguities you notice.

Thank the authors for their reply, which has addressed most of my concerns. I have raised my score to 3.

Your suggestion to incorporate experiments on open-ended tasks has greatly strengthened the applicability and coverage of our paper. We extend our heartfelt thanks and warmest wishes to you!

Comments on the paper

1 comment
Area_Chair_LnpgOpenReview ↗

Reviewers: please respond to the author rebuttal

Dear reviewers,

Thank you to those who have already engaged with the authors' responses. The author-reviewer discussion closes on November 18 (AoE).

If you have not yet replied, please read the authors' response to your review and post a reply here before then: whether your concerns have been addressed, what remains open, and whether you are updating your score. If your assessment is unchanged, please say so explicitly. Engaging with the other reviews and the ongoing discussion is also welcome.

Thank you, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.