acceptodds
Under review as a conference paper at ICLR 2027

Contrastive Self-Distillation from User Feedback

Abstract

Large language models deployed in production generate a large volume of real interactions with users. Users rarely provide explicit scores for responses, but they often correct errors, clarify requirements, or express dissatisfaction through follow-up messages, making subsequent user messages a cheap yet rich learning signal. A class of self-distillation methods, such as SDPO, uses the next user message as hindsight evidence. After reading it, the model re-scores the preceding response, and the resulting token-level score change is distilled back into the model. Although distilling this score change can yield strong empirical gains, we show that the change itself contains a systematic confound: even empty templates or unrelated messages induce sizable, response-dependent score shifts. Directly distilling the full score change therefore writes this component into the update as well, which is unrelated to the current feedback. Theoretically, we show that this component is independent of the message and enters the score additively, so it cancels when the scores under the real message are contrasted with those under randomly sampled reference messages, without having to estimate it. Building on this result, we propose CoSD (Contrastive Self-Distillation), which distills how much more the real message shifts the model's scores than a few reference messages sampled from other interactions, requiring no external reward model, preference annotations, or additional generation. Furthermore, CoSD allows cross-session user history to inform how the current message is interpreted. Since message-independent effects of the shared context cancel in the contrast, all branches can condition on the user's history without distilling the user-dependent normalization it introduces. Extensive experiments across multiple backbone models, two real-world interaction corpora, and both on- and off-policy training show that CoSD improves over the initial model and outperforms SDPO.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Self-Distillation, User Feedback, Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_uBmxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies self-distillation from naturally occurring user follow-up messages. It observes that conditioning on a follow-up can induce substantial response-dependent score shifts even when the message is empty or unrelated to the current interaction. The proposed method, Contrastive Self-Distillation (CoSD), replaces the usual feedback-versus-no-feedback comparison with a contrast between token log-probabilities under the actual follow-up and those under several length-matched reference messages. Shared user history can be included in the conditioning context. The paper provides an additive cancellation analysis and evaluates the method on WildChat and WildFeedback with several model backbones, alongside controlled on-policy and off-policy personalization experiments. The reported results show improvements over SDPO in many settings, with additional evidence from template perturbations, random-feedback controls, seed replication, judge comparisons, and baseline hyperparameter sweeps.

Critical strengths
  1. The paper identifies a concrete and practically consequential failure mode in feedback-conditioned self-distillation. The preliminary measurements show that empty templates and unrelated messages induce substantial, reproducible score shifts that vary across responses. The split-reference analysis in Figures 1–2, together with the random-feedback control in Table 10, provides a convincing empirical motivation for correcting this background. This is useful for learning from uncurated interaction logs, where follow-ups need not contain relevant feedback.

  2. The proposed correction is simple to implement and supported by a substantial experimental package. Beyond the main comparisons across corpora and backbones, the appendix includes template sensitivity tests, three-seed replication, multiple judges, baseline hyperparameter sweeps, regularization controls, and computational cost accounting. The template-robustness results are particularly compelling: on Qwen3-4B, alternative templates cause severe SDPO degradation while CoSD remains comparatively stable. These results support the practical utility of the method and make the contribution more substantial than a small benchmark improvement alone.

Critical weaknesses
  1. The incremental contribution relative to the closest contrastive self-distillation work is not sufficiently established. CREDIT (Shen et al., “From Generic Correlation to Input-Specific Credit in On-Policy Self Distillation,” arXiv:2605.11613) also corrects a nonspecific component of self-distillation through a contrastive baseline. CoSD differs in both its contrast axis and its objective, so I do not regard the methods as equivalent. However, the manuscript neither discusses this relationship nor provides a controlled comparison that isolates the benefit of contrasting feedback rather than inputs. Given the centrality of this design choice, this omission leaves the methodological advance insufficiently demonstrated.
  2. The mechanistic interpretation is stronger than what the cancellation analysis establishes. Equation (8) correctly cancels shared additive score components. However, Appendix A.2 acknowledges that the compatibility/normalizer decomposition is not identifiable and that one can choose a representation with Z = 1. The identifiable object is therefore the reference-dependent background bQb_Q, rather than a uniquely identified normalizer nuisance. Moreover, the zero-update result for irrelevant messages requires their conditional distribution to match the reference distribution, as Appendix A.1 explicitly notes. Table 10 tests this matched case, but does not establish robustness to irrelevant follow-ups drawn from a different distribution. The paper would be stronger if its main claims consistently reflected these limits and if a controlled diagnostic tested whether the retained token weights track the requested correction beyond the single example in Figure 3.
Questions for authors
  1. Closest prior work and contribution boundary: What does the feedback-axis contrast remove that an input-axis contrast would retain in your setting? A controlled comparison with the same data, history access, token loss, reference count, and training budget would help isolate this contribution. If comparing directly with CREDIT, please account for its different objective rather than treating a modified input-contrastive control as the original method.
  2. Scope of the theoretical interpretation: Can the main theoretical claim be formulated directly in terms of the identifiable reference background bQb_Q, consistently with Appendix A.2? What evidence shows that removing this background improves the direction and localization of feedback-related token updates? A small controlled test with known requested changes and irrelevant follow-ups outside the reference distribution would help resolve this concern.
  3. Contribution of user history: Table 8 shows that removing history reduces CoSD's AlpacaEval LC from 44.56 to 39.34 on Qwen3-4B × WildFeedback. What is the performance of SDPO when it receives exactly the same history in this setting? The corresponding matched-history comparison would clarify how much of the main-table gap comes from the contrastive objective and how much comes from additional conditioning information.
4 comments
AuthorsOpenReview ↗

Response to Reviewer uBmx (Part 1/3)

We sincerely thank Reviewer uBmx for the careful reading and constructive suggestions. We address each point below and have updated the manuscript accordingly.

W1 / Q1: Relationship to CREDIT and controlled comparison between the feedback and input axes

We thank the reviewer for pointing this out and have added a discussion of CREDIT in the Related Work section and a controlled comparison in Appendix M. The two methods contrast along different axes: CREDIT keeps the response and feedback fixed while replacing the input prompt with other prompts, whereas CoSD keeps the input and response fixed while replacing only the feedback message. Regarding "what the feedback axis removes that the input axis retains," the answer is the message-independent background that depends on the specific (input, response) pair. CREDIT's correction retains the full SDPO log-ratio and can only remove the portion of the background averaged across inputs.

To directly answer the reviewer's question, under the exact same data, history access, token-level loss, number of references (k=3k=3), and training budget, we implemented an input-axis counterpart within our objective: At=Vt(o+;H,x)−13∑jVt(o+;H,xj′)A_t=V_t(o^+;H,x)-\frac13\sum_j V_t(o^+;H,x'_j), where xj′x'_j are length-matched dialogue contexts from other interactions.

  • Training comparison (Qwen3-4B × WildChat): In head-to-head evaluation against CoSD, the input-axis variant achieves win rates of 42.66% on AlpacaEval and 42.4% on Arena, with both 95% CIs excluding 50%. The absolute benchmark metrics are summarized below:
Qwen3-4B × WildChatLCAHACIF-sMMLU-Pro
CoSD, feedback axis45.417.1715.7676.5261.59
Input-axis counterpart43.616.6710.2875.9759.28
  • Offline diagnostics (Qwen3-4B, on the controlled and real-edit evaluation sets detailed in W2 below): the input axis leaves behind substantially more feedback-irrelevant signal:
    • Irrelevant messages: When the feedback is replaced with an unrelated message, the ideal advantage should be close to 0. Across categories, the residual mean ∣A∣|A| is 0.30–0.42 for the input axis, even larger than SDPO (0.27–0.32), versus 0.13–0.17 for CoSD.
    • Unedited-token |A|: On tokens not targeted by the feedback, shift is 0.30–0.76 for the input axis, compared to 0.16–0.32 for CoSD.

The reason is that the background depends on the specific (input, response) pair. The input axis subtracts the score change of the same response paired with other inputs; the background in those terms belongs to different (input, response) pairs and fails to cancel the background of the current pair, instead injecting extra variance.

We also identified a scenario where the input axis has an advantage: when the feedback directly quotes the erroneous text, tokens appearing in the feedback have their probabilities boosted in the response (a copying effect), causing both SDPO and CoSD to assign positive advantage to the wrong number. Since this copying effect persists under any input prompt, it is subtracted out along the input axis. Among the wrong-number tokens, the fraction receiving a negative advantage is 4% for CoSD versus 46% for the input axis. This shows that the two axes remove different components. Their contributions are therefore complementary (or orthogonal), and combining the two axes is a promising direction for future work.

AuthorsOpenReview ↗

Response to Reviewer uBmx (Part 2/3)

W2 / Q2: Framing the theory around the identifiable bQb^Q; out-of-distribution irrelevant messages and token-level diagnostics

On framing around bQb^Q. Section 4.1 uses the user utterance model to provide intuition for why a message-independent yet response-dependent term arises. Our main theoretical conclusions themselves do not depend on this convention: the cancellation in Eq. 8 holds for any message-independent additive term without relying on the utterance model, and the component retained by CoSD in Eq. 9 is invariant to zero-point shifts of rr (Appendix A.2 in the paper). The quantity bQ(c,y)=Eo−∼Q[V(o−)]b^Q(c,y)=\mathbb{E}_{o^-\sim Q}[V(o^-)] (written bQb_Q in the paper) is precisely what is measured in Section 3.2. In the revision, we keep the intuitive presentation in the main text while explicitly stating in Section 4.1 that the identifiable object is bQb_Q. Regarding the zero-update conclusion for irrelevant messages, its distributional premise (that irrelevant messages are drawn from the same distribution as the references) is already stated in Section 4.4 and Appendix A.1 of the paper.

Token-level attribution experiments as empirical verification. To directly verify that canceling this background improves token-level credit assignment and to test all the diagnostics suggested by the reviewer, we constructed two datasets with known "target spans to be edited" and performed forward-only scoring with base weights on Qwen3-4B and Qwen3-8B (Appendix L in the revision):

  • Controlled set (500 examples): real responses paired with templated feedback across three categories: removing the conclusion paragraph (200), avoiding bullet lists (200), and corrupting a number in the response paired with feedback quoting the wrong number (100);
  • Real set (914 examples): real follow-up messages requesting explicit error corrections or formatting changes selected from the training set (filtered by keywords such as remove, wrong, format, etc.), where GPT-5-mini generates a minimal edit satisfying the feedback and token-level diff identifies the edited span.

We compare the token-level advantage AA assigned by each method (k=3k=3 for CoSD), i.e., the actual signal distilled during training, using two statistics:

  • Unedited-token |A|: the mean absolute advantage mean ∣A∣\mathrm{mean}\,|A| on unedited tokens not targeted by the feedback (lower is better);
  • Gap: the mean advantage on unedited tokens minus the mean advantage on the edited span (values above 0 indicate that the span to be edited is suppressed relative to the rest of the response; higher is better).

The results on Qwen3-4B are shown below (formatted as SDPO / CoSD; Qwen3-8B shows the same trends and full results are included in Appendix L of the revision):

Feedback TypeUnedited-token A ↓Gap ↑
Remove conclusion0.311 / 0.156−0.016 / 0.031
No bullet list0.419 / 0.321−0.113 / 0.012
Wrong number (quoted)0.364 / 0.284−5.29 / −5.86
Real set0.366 / 0.3120.161 / 0.135

These experiments directly answer each of the reviewer's token-level diagnostic questions:

  • Direction: On conclusion removal and list removal, SDPO assigns a higher mean advantage to the span that should be removed than to the rest of the response (negative Gap), whereas removing the background with CoSD corrects the update direction to positive values.
  • Unedited-token |A|: On tokens not targeted by the feedback, CoSD achieves a lower mean ∣A∣|A| than SDPO across all categories and both models (e.g., dropping from 0.311 to 0.156 on conclusion removal and from 0.366 to 0.312 on the real set on Qwen3-4B).
  • Out-of-distribution irrelevant messages: Replacing the feedback with first-turn user messages from other conversations (which initiate new tasks and differ in distribution from the follow-up messages in the reference pool, where the ideal advantage should be close to 0), the per-response residual mean ∣A∣|A| on Qwen3-4B is 0.13–0.17 for CoSD versus 0.27–0.32 for SDPO, substantially smaller for CoSD, though non-zero, consistent with the distributional premise noted above.
  • Localization: Both methods struggle to pinpoint the exact tokens to be edited from a single follow-up message, and our paper does not make such a claim (Fig. 3 illustrates direction and reduced shift on unedited tokens). When the feedback directly quotes the erroneous text, both methods assign positive advantage to the quoted tokens due to a copying effect, a shared limitation of feedback-conditioned self-distillation discussed in W1 above.
AuthorsOpenReview ↗

Response to Reviewer uBmx (Part 3/3)

Q3: Contribution of user history

We agree that controlling for user history is important, and we would like to kindly note that this history-aligned comparison was reported in our original submission (Table 3, main text): on Qwen3-4B × WildChat, giving SDPO the exact same user history as CoSD ("History only" in Table 3) only moves LC from 42.88 to 43.18, well below CoSD's 45.41.

To fully address the reviewer's question on Qwen3-4B × WildFeedback, we ran the same experiment using the identical "History only" implementation as in Table 3. All numbers below are AlpacaEval LC from the paper (Tables 1, 3, and 8), except for the one newly added cell:

AlpacaEval LC (Qwen3-4B)SDPOSDPO + HistoryCoSD w/o HistoryCoSD
WildChat42.8843.1845.0745.41
WildFeedback33.8538.14 (New)39.3444.56

The gain from the contrastive objective holds regardless of whether history is included, and while both methods benefit from user history, history provides an even larger boost to CoSD on WildFeedback. The gap between CoSD and SDPO in the main table is primarily driven by the contrastive objective (Appendix O.2 in the revision).

We thank Reviewer uBmx again for the constructive feedback. Please let us know if you have any further questions.

Dear Reviewer uBmx,

We hope you are doing well, and we sincerely appreciate your careful reading and insightful feedback on our paper. As the discussion window moves forward, we wanted to briefly follow up regarding our response and manuscript revisions.

To address the points in your review, we incorporated a discussion of CREDIT and ran a budget- and objective-matched comparison between the feedback and input axes (Section 2.1, Appendix M), explicitly stated bQb_Q as the identifiable object in Section 4.1 alongside token-level diagnostics across controlled, real-edit, and out-of-distribution irrelevant-message sets (Appendix L), and completed the 2×2 history-aligned ablation on WildFeedback (Appendix O.2).

If these additions address the points raised in your review, we would greatly appreciate it if you would consider increasing your score, and we remain glad to discuss any remaining thoughts while the discussion phase is open. Thank you very much again for your time!

Reviewer_YengOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to learn from follow-up messages in real-world user interactions. The authors identify a message-independent component in the token-level score changes used by SDPO. Even empty templates or unrelated messages can produce substantial, reproducible shifts, which may introduce unwanted signals into training.

To address this, the paper proposes Contrastive Self-Distillation (CoSD), which compares scores under real feedback with those under length-matched reference messages from other interactions. The shared additive component cancels in this comparison. The authors also provide a theoretical analysis and extend the method to incorporate cross-session user history.

Experiments on WildChat and WildFeedback, along with separate on-policy and off-policy experiments using simulated users, show consistent improvements over SDPO. Additional ablations examine reference sampling, template sensitivity, and robustness to unrelated feedback.

Critical strengths
  1. The paper identifies a meaningful issue in SDPO: feedback-conditioned score changes contain a substantial component unrelated to the actual message. The empirical analysis supports this observation, and the proposed contrastive objective offers a simple way to remove the shared background component.

  2. The experiments cover several model families, two real-world interaction datasets, and both on-policy and off-policy training. The random-feedback controls are particularly convincing. The template sensitivity experiments also show that CoSD avoids some of the severe instabilities observed with SDPO. Additional ablations and robustness checks further strengthen the evaluation.

Critical weaknesses
  1. Sensitivity to the reference message distribution.

CoSD removes message-independent additive terms regardless of the reference distribution. However, the remaining training signal still depends on the choice of reference messages. As discussed in Appendix A.1, irrelevant follow-up messages can produce nonzero expected updates when their distribution differs from the reference distribution.

The current implementation mainly uses length matching. While the paper studies reference count and semantic relevance, it provides limited evidence on robustness to changes in feedback type or task domain. It remains unclear how well the method works when the reference pool differs substantially from the interactions being used for training.

  1. Limited direct evaluation of token-level credit assignment.

The benchmark results and random-feedback controls support the overall effectiveness of CoSD, but they do not directly establish whether the method assigns more accurate training signals to the parts of a response targeted by user feedback.

Figure 3 provides a useful example, but a systematic evaluation is still missing. For explicit error corrections or formatting requests, it would be useful to measure whether CoSD better identifies the relevant tokens while avoiding unnecessary changes to unaffected content. This would provide more direct evidence for the proposed mechanism.

Questions for authors
  1. How sensitive is CoSD to the reference message distribution? For example, would the method remain effective if feedback from a technical domain were contrasted against length-matched references from general conversations? A controlled comparison across different reference pools would help clarify this.

  2. Can the authors provide quantitative evidence that CoSD improves token-level credit assignment? For tasks with explicit correction or formatting feedback, comparing the advantage assigned to edited and unedited spans against SDPO would be particularly useful.

5 comments
AuthorsOpenReview ↗

Response to Reviewer Yeng (Part 1/2)

We sincerely thank Reviewer Yeng for the thoughtful review and constructive questions. We address each point below and have incorporated these results into the revised manuscript.

W1 / Q1: Sensitivity to the reference message distribution

We thank the reviewer for raising this question. In the paper, Appendix F.2 already shows that filtering out reference messages semantically similar to o+o^+ has no measurable effect. To further address the reviewer's concern, we used GPT-5-mini to classify training conversations by their first user prompt into technical (coding, math, STEM) and general domains, sampled 600 examples from each, and computed CoSD's token-level advantages under five reference pools: same-domain, cross-domain (the exact scenario raised by the reviewer), WildFeedback (cross-corpus), the default pool, and a disjoint draw from the default pool (serving as a resampling noise ceiling). Taking the default pool as the reference, at k=25k=25 the token-level Pearson correlation across the other pools is 0.981–0.987 compared to the noise ceiling of 0.988–0.989, with a per-sample constant offset of at most 0.026. Thus, changing the reference pool introduces no more variation than resampling within the same pool.

We also directly compared downstream performance after full training on Qwen3-4B. Specifically, we replaced all 3 reference messages for every WildChat training example with messages drawn exclusively from the other domain, keeping all other settings identical. In head-to-head evaluation against the original CoSD model, the cross-domain model achieves win rates of 50.50% [47.76, 53.23] on AlpacaEval and 51.07% [48.13, 54.21] on Arena, with both 95% CIs containing 50%. The absolute benchmark metrics are also nearly identical (Appendix N in the revision):

Qwen3-4B × WildChatLCAHACIF-sMMLU-Pro
CoSD, default pool45.417.1715.7676.5261.59
CoSD, cross-domain pool44.717.6215.3577.4561.62
AuthorsOpenReview ↗

Response to Reviewer Yeng (Part 2/2)

W2 / Q2: Direct evidence of token-level credit assignment

We thank the reviewer for this suggestion, which we agree is particularly important. We constructed two datasets with known "target spans to be edited" and performed forward-only scoring with base weights on Qwen3-4B and Qwen3-8B:

  • Controlled set (500 examples): real responses paired with templated feedback, including removing the conclusion paragraph, avoiding bullet lists, and corrupting a number in the response paired with feedback quoting the wrong number;
  • Real set (914 examples): real follow-up messages requesting explicit error corrections or formatting changes selected from the training set (filtered by keywords such as remove, wrong, format, etc.), where GPT-5-mini generates a minimal edit satisfying the feedback and token-level diff identifies the edited span.

We compare the token-level advantage AA assigned by each method, i.e., the actual signal distilled during training, using two statistics:

  • Unedited-token |A|: the mean absolute advantage mean ∣A∣\mathrm{mean}\,|A| on unedited tokens not targeted by the feedback (lower is better);
  • Gap: the mean advantage on unedited tokens minus the mean advantage on the edited span (values above 0 indicate that the span to be edited is suppressed relative to the rest of the response; higher is better).

The results on Qwen3-4B are shown below (formatted as SDPO / CoSD; Qwen3-8B shows the same trends and full results are included in Appendix L of the revision):

Feedback TypeUnedited-token A ↓Gap ↑
Remove conclusion0.311 / 0.156−0.016 / 0.031
No bullet list0.419 / 0.321−0.113 / 0.012
Wrong number (quoted)0.364 / 0.284−5.29 / −5.86
Real set0.366 / 0.3120.161 / 0.135

Our main findings are as follows:

  1. Lower shift on unedited tokens. On tokens not targeted by the feedback, CoSD yields a lower mean ∣A∣|A| than SDPO across all categories. For instance, on Qwen3-4B, shift drops from 0.311 to 0.156 on conclusion removal, and from 0.366 to 0.312 on the real set.
  2. Correct update direction on structural edits. On conclusion removal and list removal, SDPO assigns a higher mean advantage to the span that should be removed than to the rest of the response (negative Gap), whereas CoSD pulls the Gap to 0 or positive values.
  3. Smaller residual under irrelevant messages. When the feedback is replaced with the first-turn user message from another conversation, the ideal advantage AA should be close to 0. On Qwen3-4B, the per-response mean ∣A∣|A| is 0.13–0.17 for CoSD, compared to 0.27–0.32 for SDPO.

We also observe a shared limitation of self-distillation methods: both methods struggle to pinpoint the exact tokens to be edited from a single follow-up message. When the feedback directly quotes the erroneous text, both methods assign positive advantage to the quoted tokens due to a copying effect, a phenomenon common to feedback-conditioned self-distillation. Detailed experimental results have been added to Appendix L.

We thank Reviewer Yeng again for the insightful feedback, and we are happy to answer any further questions.

Dear Reviewer Yeng,

Thank you again for your constructive comments and helpful suggestions. With the discussion deadline approaching, we wanted to politely follow up to see if you have had a chance to look over our response and updated manuscript.

Following your suggestions, we evaluated CoSD's sensitivity to the reference distribution through both five-pool token-level correlation analyses and full cross-domain reference training on Qwen3-4B (Appendix N), and constructed controlled and real-edit benchmarks on Qwen3-4B and Qwen3-8B to directly verify token-level credit assignment and out-of-distribution residuals (Appendix L).

We understand you have a heavy reviewing workload during this period. If these new experiments have resolved your questions, we would be very grateful if you could consider raising your score, and we remain happy to clarify anything else while the discussion window is open.

Reviewer_6o2rOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to learn from users' follow-up messages in chat logs. Methods like SDPO let the model re-score its previous response after reading the next user message, and distill the token-level score change back into the model. The authors find that this change contains a systematic confound: even empty templates or unrelated messages induce sizable, response-dependent score shifts. They show in theory that this part does not depend on the message and is added to the score, so it cancels when the real message is compared with random reference messages. They propose CoSD (Contrastive Self-Distillation). It uses the score change under the real message minus the average change under k length-matched messages from other interactions. Experiments on WildChat and WildFeedback with four backbones show CoSD improves over the base model and SDPO.

Critical strengths
  1. Simple and straightforward idea. The fix is easy to understand and easy to use. Instead of comparing the score with and without the user message, CoSD compares the real message with a few random messages from other interactions, so anything that does not depend on the message cancels out. It keeps SDPO's loss, needs no reward model, no preference labels and no extra generation, and only adds a few forward passes. The same trick also gives a clean way to use the user's history during training.
  2. Clear diagnosis of the problem. Before proposing the method, the paper shows the problem with measurements. Even an empty template or unrelated messages already move the scores a lot, and this shift is stable for each response. The paper also tests simpler fixes, like subtracting a global mean or using the empty template as the baseline, and shows they do not help. This makes the motivation easy to follow.
Critical weaknesses
  1. Gap between the theory and the experiments. The theory treats the teacher as an ideal Bayesian model that reads the user message as a reaction to the response (Eq. 3). But in practice, the message is put before the response with the instruction use this to guide your answer, so the model may read it more like an instruction. These two readings can give opposite signs. For example, under the Bayesian view, "remove the conclusion" is most likely after a response that has a conclusion, so the word "Conclusion" should get a higher score, not the lower score CoSD gives it in Fig. 3. Also, the cancellation holds by construction, and the theory does not explain why SDPO collapses or why CoSD helps. So the theory feels more like a motivation than an explanation of the results.

  2. Novelty and missing related work. Splitting SDPO's signal into a "fit" term minus a log-normalizer is very close to Proposition 3.1 in [1], which the paper cites for other things but not for this. The idea of contrasting with reference contexts is also similar to calibration methods for LLMs (e.g., subtracting the score of a content-free input), which are not discussed. I think the main contribution is applying this idea to user feedback, and the paper should say this more clearly.

[1] Buening et al. (2026). Aligning Language Models from User Interactions.

Questions for authors
  1. In the teacher branch, why is the user's next message put before the response, while in time it comes after?
  2. Why does SDPO drop below the base model in several settings, while prior work reports gains in a similar setup?
  3. How are users linked across sessions in WildChat and WildFeedback, and how many training examples actually have a non-empty user history?
6 comments
AuthorsOpenReview ↗

Response to Reviewer 6o2r (Part 1/3)

We sincerely thank Reviewer 6o2r for the constructive comments and helpful suggestions. We address each point below and have updated the manuscript accordingly.

W1: Gap between the theory and the experiments

We respectfully note that this stems from a notational shorthand. Following the original SDPO paper (Buening et al., 2026, Appendix A), most equations in Section 4.1 abbreviate the teacher's conditioning argument as oo. In fact, the teacher conditions on the templated message T(o)T(o), as defined from Eq. 1 in our main text: st(o):=log⁡πˉθ(yt∣c,T(o),y<t)s_t(o):=\log\bar\pi_\theta(y_t\mid c,T(o),y_{<t}). While the two notations make no difference for understanding the overall framework, the shorthand can indeed cause confusion on the specific detail raised by the reviewer.

Specifically, T(o)T(o) instructs the model to "use this [future user message] to guide your answer." The corresponding ideal teacher therefore infers the user's intent from T(o)T(o) and answers accordingly, rather than treating oo as a post-hoc observation generated after yy to retrodict yy. Under this reading, "remove the conclusion" lowers the probability of "Conclusion," consistent with Fig. 3. This is also how the SDPO authors themselves formulate the teacher. For example, Hübotter et al. (2026, Appendix C) cast the feedback-conditioned teacher as the optimal policy in maximum-entropy RL and use its log-probability as an implicit reward. In the revision, we have unified the notation in Section 4.1 to T(o)T(o). Moreover, the derivation and conclusions of CoSD do not depend on this detail.

On "the cancellation holds by construction, and the theory does not explain why SDPO collapses or why CoSD helps." Our paper starts by empirically identifying the presence of a background signal, and then constructs a contrastive objective that removes it by cancellation. Exact cancellation by construction, without having to estimate the background, is precisely the design goal of this fix. And fixes that require estimating the background (the empty-template baseline and batch-mean centering) fail in practice (Table 3 and Table 10 in the paper). The theory characterizes which part of the SDPO signal is independent of the feedback, while whether distilling this part is harmful is an empirical question answered by controlled experiments that isolate it:

  • Distilling only the template's own background (empty message) drops LC to 3.65; distilling only the background induced by random messages drops LC from 32.75 to 23.87; whereas CoSD under random messages stays at 32.54, matching the theoretical prediction of a zero expected update (Table 10 in the paper).
  • Because the background changes with template wording while the cancellation is unaffected, SDPO is sensitive to the template whereas CoSD is not (Table 4 in the paper).

Furthermore, Appendix J of the paper already provides a partial explanation of why SDPO collapses, analyzing Gemma3-12B where SDPO degrades sharply. Repeating the measurement protocol of Section 3.2 reveals that the background on Gemma3-12B is even stronger than on Qwen3-4B and harder to remove with a constant: the token RMS of the reference background reaches 0.70 of the real-message shift (vs. 0.64 on Qwen3-4B), and a global constant removes only 6.5% of its mean square (vs. 22% on Qwen3-4B). Moreover, the background peaks at the end-of-turn token: conditioning on any message raises its log-probability by 3.56 nats on average, and the shift under the real follow-up is nearly the same (3.72), carrying no information about the message content. Yet zeroing out the SDPO advantage at this single position does not recover SDPO (LC remains at 22.05). Combined with batch-mean centering (Table 10), neither removing a global constant nor masking the single largest position suffices to prevent SDPO's collapse; what remains is a background that varies across both responses and token positions. In contrast, CoSD removes it token by token without needing to locate it. Our experiments show that fixes relying on estimating or localizing the background do not generalize across models, whereas cancellation by construction is agnostic to the specific form of the background.

AuthorsOpenReview ↗

Response to Reviewer 6o2r (Part 2/3)

W2: Novelty and missing related work

About Proposition 3.1. Respectfully, this appears to be a misunderstanding. The problem our paper addresses is a systematic confound in the feedback-conditioned self-distillation signal of the prior SDPO methods we cite (Hübotter et al., 2026; Buening et al., 2026), both of which are discussed in detail in our Introduction and Related Work. It is therefore natural to take their decomposition as the starting point of our analysis, as explicitly noted in Section 4.1 (Page 4, "Following Hübotter et al. (2026), we write..."). Building on this starting point, our work: (1) empirically identifies and measures a sizable, response-varying, and reproducible background component in the SDPO signal that is independent of the actual feedback, and demonstrates through controlled experiments that distilling it harms alignment; (2) proposes CoSD, which cancels this component token by token by contrasting the real follow-up against sampled reference messages without having to estimate it, while also enabling safe incorporation of cross-session user history; and (3) validates CoSD across multiple backbones, two real-world interaction corpora, and both on- and off-policy training. In short, we never claim this decomposition as our contribution, so it is unrelated to the novelty of our work. To make this relationship even clearer, in the revision (Section 4.1) we additionally cite the user-interaction extension by Buening et al. (2026, Appendix A) alongside Hübotter et al. (2026, Appendix C).

Calibration methods. Calibration methods adjust label priors for classification at inference time, whereas we correct token-level credit assignment during training. Both the problem setting and the stage are different. If the reviewer could point us to specific papers they have in mind, we would be very happy to discuss them. Moreover, the specific idea mentioned by the reviewer, i.e., "subtracting the score of a content-free input", has already been tested in our ablation studies and does not work:

  • Subtracting a content-free baseline corresponds to our empty-template baseline, which achieves an LC of 39.86, below SDPO's 42.88 (Table 3 in the paper);
  • Batch calibration corresponds to batch-mean centering, which achieves an LC of 41.81 (Table 10 in the paper).

The reason is shown in Fig. 2: after subtracting the content-free score, the residual remains of the same magnitude and is highly reproducible across two disjoint sets of reference messages (r=0.88/0.91r = 0.88 / 0.91), while a global mean removes only a small fraction of the background. Therefore, one must average over the distribution of real messages on a per-sample basis, which is precisely what CoSD does.

AuthorsOpenReview ↗

Response to Reviewer 6o2r (Part 3/3)

Q1: Why is the user's next message put before the response?

This is the original design of SDPO. The teacher needs to assign a conditional score to every token of yy, and in a causal language model, conditioning information must appear before the tokens being scored. If the message were placed after yy, the scores on yy would not change at all. The template also explicitly notes that this is a "future user message."

Q2: Why is SDPO below Base in several settings, while prior work reported improvements?

The difference mainly comes from the template. The original SDPO paper presents only a single hindsight template (Table 1 of their paper). When run as-is, the offline training code on the default branch of the official repository also uses the same default template for all models, matching the wording in their paper. A second template exists in the repository as commented-out code, which appears to have been designed for models such as Qwen3-8B. For reproducibility and a fair comparison, we used the default template uniformly across all backbones and both methods. In addition, our paper already includes template-sensitivity experiments (Table 4 in the paper) showing that SDPO is brittle to template choice: on Qwen3-4B, SDPO drops from 42.88 to 12.03 and 6.61 across templates while CoSD stays at 44.17–45.41.

To fully resolve the reviewer's question, we trained SDPO and CoSD on Qwen3-8B on WildFeedback using the commented-out template from the repository, which largely recovers the SDPO performance reported in prior work:

Qwen3-8B × WildFeedbackLCIFEval-strict
Base44.4781.52
SDPO, default template35.1381.15
SDPO, commented-out template48.7679.85
CoSD, default template52.1881.89
CoSD, commented-out template53.5483.73

With this template, SDPO improves over Base while IFEval remains largely unchanged. However, even under this template, CoSD still outperforms SDPO and remains stable across both templates. This demonstrates that SDPO relies heavily on per-model template selection, whereas CoSD works reliably across backbones with a single unified template (Appendix O.1 in the revision).

Q3: How are users linked across sessions, and what fraction of training examples have non-empty user history?

Users are linked by the hashed IP provided in WildChat, excluding the current conversation. While WildChat provides the hashed IP field directly, WildFeedback does not; we therefore match WildFeedback conversations back to WildChat using the exact opening text (the first user message and first assistant reply, plus the second user message when needed), and refrain from assigning a user whenever any ambiguity remains. In the training sets, 75.4% of examples in WildChat have non-empty history (with a median of 5 other sessions), and 44.7% in WildFeedback have non-empty history (median of 4). We have added these details to Appendix C in the revision.

We thank Reviewer 6o2r again for the time and thoughtful review. Please let us know if any further clarification would be helpful.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Clarifications and Revisions

Dear Area Chairs and Reviewers,

Thank you very much for your time and constructive reviews, and we thank Reviewer Yeng and Reviewer 6o2r for their encouraging follow-up comments noting that our responses and new experiments addressed their questions and concerns. For ease of reference, we briefly summarize our clarifications of the original submission alongside our new experiments and blue-highlighted revisions below:

  1. Theory, Prior Work & Existing Controls (Reviewer 6o2r, Reviewer uBmx): Clarified why causal LMs must place T(o)T(o) before yy, that the Boltzmann decomposition is cited from prior SDPO work as our starting point rather than claimed as our contribution, and that bQ(c,y)b_Q(c,y) and the irrelevant-message condition were already established in Sec. 3.2, 4.4 & App. A. We also pointed out that content-free / batch-calibration baselines and SDPO's collapse were already analyzed via Fig. 2, Tables 3, 4, 10 & App. J (where empty-template, batch-centering, and EOS-masking fixes all fail). In Sec. 4.1, we unified notation to T(o)T(o), cited Buening et al. (2026), and stated bQ(c,y)b_Q(c,y) explicitly.
  2. Input-Axis Comparison & CREDIT Discussion (Reviewer uBmx): Added a discussion of CREDIT in Sec. 2.1 and trained an objective- and budget-matched input-axis counterpart (k=3k=3) on Qwen3-4B × WildChat (App. M, Table 20). CoSD outperforms the input-axis variant in head-to-head evaluations (42.66% / 42.40% for the input axis on AlpacaEval / Arena) and across all five benchmarks (45.41 vs. 43.61 LC) with much smaller irrelevant-message residuals, while the two axes are complementary on quoted errors.
  3. Token-Level Credit Assignment Diagnostics (Reviewer Yeng, Reviewer uBmx): Clarified that Fig. 3 illustrates update direction and unedited-token shifts rather than exact span localization. Across controlled and real user-requested edits on Qwen3-4B and Qwen3-8B (App. L, Tables 18–19), CoSD consistently lowers shifts on unedited tokens across all categories, corrects the update direction on structural edits from negative to zero/positive Gap, and cuts residuals nearly in half under out-of-distribution irrelevant prompts.
  4. Reference Pool, Templates & User History (Reviewer 6o2r, Reviewer Yeng, Reviewer uBmx): Complementing the semantic-filtering (App. F.2), template-sensitivity (Table 4), and WildChat history-aligned (Table 3) results already in our submission, we added: (i) five-pool correlations (r>0.98r > 0.98) and cross-domain reference training matching the default pool (App. N, Table 21); (ii) Qwen3-8B results under the official repository's commented-out template, recovering SDPO's gain over Base while CoSD remains stronger across both templates (App. O.1, Table 22); and (iii) cross-session user-linking statistics (App. C) plus SDPO + History on WildFeedback (38.14 vs. 44.56 LC; App. O.2, Table 23).

Thank you again for helping us strengthen our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.