- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies self-distillation from naturally occurring user follow-up messages. It observes that conditioning on a follow-up can induce substantial response-dependent score shifts even when the message is empty or unrelated to the current interaction. The proposed method, Contrastive Self-Distillation (CoSD), replaces the usual feedback-versus-no-feedback comparison with a contrast between token log-probabilities under the actual follow-up and those under several length-matched reference messages. Shared user history can be included in the conditioning context. The paper provides an additive cancellation analysis and evaluates the method on WildChat and WildFeedback with several model backbones, alongside controlled on-policy and off-policy personalization experiments. The reported results show improvements over SDPO in many settings, with additional evidence from template perturbations, random-feedback controls, seed replication, judge comparisons, and baseline hyperparameter sweeps.
Critical strengths
-
The paper identifies a concrete and practically consequential failure mode in feedback-conditioned self-distillation. The preliminary measurements show that empty templates and unrelated messages induce substantial, reproducible score shifts that vary across responses. The split-reference analysis in Figures 1–2, together with the random-feedback control in Table 10, provides a convincing empirical motivation for correcting this background. This is useful for learning from uncurated interaction logs, where follow-ups need not contain relevant feedback.
-
The proposed correction is simple to implement and supported by a substantial experimental package. Beyond the main comparisons across corpora and backbones, the appendix includes template sensitivity tests, three-seed replication, multiple judges, baseline hyperparameter sweeps, regularization controls, and computational cost accounting. The template-robustness results are particularly compelling: on Qwen3-4B, alternative templates cause severe SDPO degradation while CoSD remains comparatively stable. These results support the practical utility of the method and make the contribution more substantial than a small benchmark improvement alone.
Critical weaknesses
- The incremental contribution relative to the closest contrastive self-distillation work is not sufficiently established. CREDIT (Shen et al., “From Generic Correlation to Input-Specific Credit in On-Policy Self Distillation,” arXiv:2605.11613) also corrects a nonspecific component of self-distillation through a contrastive baseline. CoSD differs in both its contrast axis and its objective, so I do not regard the methods as equivalent. However, the manuscript neither discusses this relationship nor provides a controlled comparison that isolates the benefit of contrasting feedback rather than inputs. Given the centrality of this design choice, this omission leaves the methodological advance insufficiently demonstrated.
- The mechanistic interpretation is stronger than what the cancellation analysis establishes. Equation (8) correctly cancels shared additive score components. However, Appendix A.2 acknowledges that the compatibility/normalizer decomposition is not identifiable and that one can choose a representation with Z = 1. The identifiable object is therefore the reference-dependent background , rather than a uniquely identified normalizer nuisance. Moreover, the zero-update result for irrelevant messages requires their conditional distribution to match the reference distribution, as Appendix A.1 explicitly notes. Table 10 tests this matched case, but does not establish robustness to irrelevant follow-ups drawn from a different distribution. The paper would be stronger if its main claims consistently reflected these limits and if a controlled diagnostic tested whether the retained token weights track the requested correction beyond the single example in Figure 3.
Questions for authors
- Closest prior work and contribution boundary: What does the feedback-axis contrast remove that an input-axis contrast would retain in your setting? A controlled comparison with the same data, history access, token loss, reference count, and training budget would help isolate this contribution. If comparing directly with CREDIT, please account for its different objective rather than treating a modified input-contrastive control as the original method.
- Scope of the theoretical interpretation: Can the main theoretical claim be formulated directly in terms of the identifiable reference background , consistently with Appendix A.2? What evidence shows that removing this background improves the direction and localization of feedback-related token updates? A small controlled test with known requested changes and irrelevant follow-ups outside the reference distribution would help resolve this concern.
- Contribution of user history: Table 8 shows that removing history reduces CoSD's AlpacaEval LC from 44.56 to 39.34 on Qwen3-4B × WildFeedback. What is the performance of SDPO when it receives exactly the same history in this setting? The corresponding matched-history comparison would clarify how much of the main-table gap comes from the contrastive objective and how much comes from additional conditioning information.
4 comments
Response to Reviewer uBmx (Part 2/3)
W2 / Q2: Framing the theory around the identifiable ; out-of-distribution irrelevant messages and token-level diagnostics
On framing around . Section 4.1 uses the user utterance model to provide intuition for why a message-independent yet response-dependent term arises. Our main theoretical conclusions themselves do not depend on this convention: the cancellation in Eq. 8 holds for any message-independent additive term without relying on the utterance model, and the component retained by CoSD in Eq. 9 is invariant to zero-point shifts of (Appendix A.2 in the paper). The quantity (written in the paper) is precisely what is measured in Section 3.2. In the revision, we keep the intuitive presentation in the main text while explicitly stating in Section 4.1 that the identifiable object is . Regarding the zero-update conclusion for irrelevant messages, its distributional premise (that irrelevant messages are drawn from the same distribution as the references) is already stated in Section 4.4 and Appendix A.1 of the paper.
Token-level attribution experiments as empirical verification. To directly verify that canceling this background improves token-level credit assignment and to test all the diagnostics suggested by the reviewer, we constructed two datasets with known "target spans to be edited" and performed forward-only scoring with base weights on Qwen3-4B and Qwen3-8B (Appendix L in the revision):
- Controlled set (500 examples): real responses paired with templated feedback across three categories: removing the conclusion paragraph (200), avoiding bullet lists (200), and corrupting a number in the response paired with feedback quoting the wrong number (100);
- Real set (914 examples): real follow-up messages requesting explicit error corrections or formatting changes selected from the training set (filtered by keywords such as remove, wrong, format, etc.), where GPT-5-mini generates a minimal edit satisfying the feedback and token-level diff identifies the edited span.
We compare the token-level advantage assigned by each method ( for CoSD), i.e., the actual signal distilled during training, using two statistics:
- Unedited-token |A|: the mean absolute advantage on unedited tokens not targeted by the feedback (lower is better);
- Gap: the mean advantage on unedited tokens minus the mean advantage on the edited span (values above 0 indicate that the span to be edited is suppressed relative to the rest of the response; higher is better).
The results on Qwen3-4B are shown below (formatted as SDPO / CoSD; Qwen3-8B shows the same trends and full results are included in Appendix L of the revision):
| Feedback Type | Unedited-token A ↓ | Gap ↑ |
|---|---|---|
| Remove conclusion | 0.311 / 0.156 | −0.016 / 0.031 |
| No bullet list | 0.419 / 0.321 | −0.113 / 0.012 |
| Wrong number (quoted) | 0.364 / 0.284 | −5.29 / −5.86 |
| Real set | 0.366 / 0.312 | 0.161 / 0.135 |
These experiments directly answer each of the reviewer's token-level diagnostic questions:
- Direction: On conclusion removal and list removal, SDPO assigns a higher mean advantage to the span that should be removed than to the rest of the response (negative Gap), whereas removing the background with CoSD corrects the update direction to positive values.
- Unedited-token |A|: On tokens not targeted by the feedback, CoSD achieves a lower mean than SDPO across all categories and both models (e.g., dropping from 0.311 to 0.156 on conclusion removal and from 0.366 to 0.312 on the real set on Qwen3-4B).
- Out-of-distribution irrelevant messages: Replacing the feedback with first-turn user messages from other conversations (which initiate new tasks and differ in distribution from the follow-up messages in the reference pool, where the ideal advantage should be close to 0), the per-response residual mean on Qwen3-4B is 0.13–0.17 for CoSD versus 0.27–0.32 for SDPO, substantially smaller for CoSD, though non-zero, consistent with the distributional premise noted above.
- Localization: Both methods struggle to pinpoint the exact tokens to be edited from a single follow-up message, and our paper does not make such a claim (Fig. 3 illustrates direction and reduced shift on unedited tokens). When the feedback directly quotes the erroneous text, both methods assign positive advantage to the quoted tokens due to a copying effect, a shared limitation of feedback-conditioned self-distillation discussed in W1 above.
Response to Reviewer uBmx (Part 3/3)
Q3: Contribution of user history
We agree that controlling for user history is important, and we would like to kindly note that this history-aligned comparison was reported in our original submission (Table 3, main text): on Qwen3-4B × WildChat, giving SDPO the exact same user history as CoSD ("History only" in Table 3) only moves LC from 42.88 to 43.18, well below CoSD's 45.41.
To fully address the reviewer's question on Qwen3-4B × WildFeedback, we ran the same experiment using the identical "History only" implementation as in Table 3. All numbers below are AlpacaEval LC from the paper (Tables 1, 3, and 8), except for the one newly added cell:
| AlpacaEval LC (Qwen3-4B) | SDPO | SDPO + History | CoSD w/o History | CoSD |
|---|---|---|---|---|
| WildChat | 42.88 | 43.18 | 45.07 | 45.41 |
| WildFeedback | 33.85 | 38.14 (New) | 39.34 | 44.56 |
The gain from the contrastive objective holds regardless of whether history is included, and while both methods benefit from user history, history provides an even larger boost to CoSD on WildFeedback. The gap between CoSD and SDPO in the main table is primarily driven by the contrastive objective (Appendix O.2 in the revision).
We thank Reviewer uBmx again for the constructive feedback. Please let us know if you have any further questions.
Dear Reviewer uBmx,
We hope you are doing well, and we sincerely appreciate your careful reading and insightful feedback on our paper. As the discussion window moves forward, we wanted to briefly follow up regarding our response and manuscript revisions.
To address the points in your review, we incorporated a discussion of CREDIT and ran a budget- and objective-matched comparison between the feedback and input axes (Section 2.1, Appendix M), explicitly stated as the identifiable object in Section 4.1 alongside token-level diagnostics across controlled, real-edit, and out-of-distribution irrelevant-message sets (Appendix L), and completed the 2×2 history-aligned ablation on WildFeedback (Appendix O.2).
If these additions address the points raised in your review, we would greatly appreciate it if you would consider increasing your score, and we remain glad to discuss any remaining thoughts while the discussion phase is open. Thank you very much again for your time!
Response to Reviewer uBmx (Part 1/3)
We sincerely thank Reviewer uBmx for the careful reading and constructive suggestions. We address each point below and have updated the manuscript accordingly.
W1 / Q1: Relationship to CREDIT and controlled comparison between the feedback and input axes
We thank the reviewer for pointing this out and have added a discussion of CREDIT in the Related Work section and a controlled comparison in Appendix M. The two methods contrast along different axes: CREDIT keeps the response and feedback fixed while replacing the input prompt with other prompts, whereas CoSD keeps the input and response fixed while replacing only the feedback message. Regarding "what the feedback axis removes that the input axis retains," the answer is the message-independent background that depends on the specific (input, response) pair. CREDIT's correction retains the full SDPO log-ratio and can only remove the portion of the background averaged across inputs.
To directly answer the reviewer's question, under the exact same data, history access, token-level loss, number of references (k=3), and training budget, we implemented an input-axis counterpart within our objective: At=Vt(o+;H,x)−31∑jVt(o+;H,xj′), where xj′ are length-matched dialogue contexts from other interactions.
The reason is that the background depends on the specific (input, response) pair. The input axis subtracts the score change of the same response paired with other inputs; the background in those terms belongs to different (input, response) pairs and fails to cancel the background of the current pair, instead injecting extra variance.
We also identified a scenario where the input axis has an advantage: when the feedback directly quotes the erroneous text, tokens appearing in the feedback have their probabilities boosted in the response (a copying effect), causing both SDPO and CoSD to assign positive advantage to the wrong number. Since this copying effect persists under any input prompt, it is subtracted out along the input axis. Among the wrong-number tokens, the fraction receiving a negative advantage is 4% for CoSD versus 46% for the input axis. This shows that the two axes remove different components. Their contributions are therefore complementary (or orthogonal), and combining the two axes is a promising direction for future work.