- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work investigates self-distillation of language models without annotations or a strong external teacher. It studies how the model selects retrieved examples to augment the teacher and which augmented supervision signals are suitable for training. The authors propose INFLOW, which consists of three stages. First, it retrieves examples based on the similarity of cross-layer hidden state trajectories and answer confidence. Second, it measures belief shift using the Jensen–Shannon divergence between answer distributions obtained with and without retrieval, and selects targets with the largest shifts. Third, it performs on-policy reverse-KL distillation using an EMA teacher equipped with retrieved context. Experiments show that INFLOW slightly outperforms baseline methods on average across models and achieves consistent improvements on mathematical reasoning tasks.
Critical strengths
- The perspective of post-retrieval selection is practically valuable. Instead of selecting data purely based on pre-generation similarity or confidence, INFLOW measures whether retrieved context actually alters the teacher’s answer distribution. This approach is more direct than relying only on pre-retrieval proxy signals and may inspire future work on data selectors.
- It delivers consistent gains on mathematical tasks. INFLOW achieves notable improvements over the original model on mathematical benchmarks across four backbones. The Fix/Break analysis in Appendix C.2 and case studies in Appendix F cover scenarios including correct-to-correct, wrong-to-correct, correct-to-wrong and wrong-to-wrong transitions. These results clearly illustrate the effective range and failure modes of the method.
Critical weaknesses
- uses one sample without retrieval while uses five retrieved samples. As a result, JS divergence may capture both retrieval effects and ordinary sampling variance.
- The ablation study is not sufficiently complete. The paper lacks comparisons between final-layer cosine similarity and trajectory similarity, as well as between trajectory similarity alone and trajectory similarity combined with certainty. These comparisons are needed to demonstrate the importance of trajectory information and whether certainty brings additional benefits.
- There is no comparison between JS-based selection and binary hard-flip selection. It remains unvalidated whether continuous JS values carry more information than simple binary answer flips.
Questions for authors
- From which token position of the causal LM are the logits of the answer distribution extracted? Are they taken at the answer token position or the token immediately preceding the answer token?
- In Appendix F, answer probabilities frequently approach 1 regardless of answer correctness. Does this observation imply that the measured belief shift mostly reflects format consistency or hard answer flips rather than the model’s genuine uncertainty about the problem?
- How do the authors handle tied rankings when many samples yield a JS value exactly equal to 0 or ? Could the full distribution of JS scores be reported?
- Would the main findings still hold if both the non-retrieval and retrieval conditions use five paired random samples? This control experiment is critical to rule out the influence of ordinary decoding randomness.
- Could the authors provide direct comparisons of final-layer cosine similarity, trajectory similarity, and trajectory plus certainty under the same JS selector setting? Also please compare JS selection against binary answer-flip selection. These results will determine whether cross-layer trajectories, certainty and continuous JS values each contribute extra performance gains.
3 comments
reply to author
The author addressed most of my questions, so I’ve decided to keep my rating.
Thank you for your valuable suggestions and recognition. Wishing you continued success in your research!
Thank you for the suggestions on the controlled ablations.
W1 and Q4. Sampling variation. We have added 1/1 and 5/5 controls, where the two numbers denote samples before/after retrieval. Retrieved contexts, decoding settings, and training settings are fixed. The table reports Avg@3 ± SD (%) on Ministral-3-8B. The complete line chart is shown in the revised Figure 3. These runs test sample-count sensitivity; the paired-sampling control requested in Q4 have not yet been verified due to time constraints.
W2 and Q5. Trajectory and certainty. We have added final-layer retrieval with and without certainty under the same JS selector. The four variants below directly test representation choice and the contribution of certainty. They share the candidate pool and training settings. The complete line chart is shown in the revised Figure 3.
W3 and Q5. JS versus hard flips. We have added hard-flip selection using 1[argmaxP0=argmaxPS], with random tie-breaking independent of JS. Retrieved contexts and training are identical. The comparison tests whether continuous JS ranking adds value beyond detecting a change in the most likely answer. More complete ablation results are provided in the revised Figure 3, Sec. 4.3, and Appendix C.
Q1. Logit position. Answer probabilities use logits at t−1, immediately before the answer token at t; retrieval representations are extracted at t. We have clarified both positions and normalization over legal option tokens in Secs. 3.1 and 3.3.
Q2. Near-unit probabilities and hard flips. The generated
resultprefix may already state the chosen option, making its answer letter nearly deterministic even when wrong. These probabilities therefore measure prefix-conditioned prediction, not calibrated reasoning confidence. When the distributions approach point masses, JS does approach a hard-flip distinction. We have clarified this and added the direct hard-flip comparison above in Sec. 4.3.Q3. Ties and score distribution. We rank unrounded JS scores and break exact ties by random selection, as now specified in Appendix C.2. The requested score distribution is reported by decile in Appendix C.3.
Other modifications. Although you did not raise this point, we also redrew Figure 1 to make it more intuitive because the original relied too heavily on text to explain each step. If you have time to look at the revised diagram, we would welcome any further questions or ambiguities you notice.