- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates label collapse in diffusion language model in-context learning. The authors observe that diffusion language model can produce query-independent labels even with class-balanced demonstrations. It proposes demo-anchored calibration (DAC), a simple method that estimates the condition-level bias from per-demonstration readings and subtracts it from the query log-odds. The authors evaluate label collapse and calibration effectiveness across six diffusion language models and two classification datasets
Critical strengths
[S1] The paper reports an important finding: diffusion language models can produce nearly the same label regardless of the query input, even with class-balanced demonstrations.
[S2] The method is simple and easy to understand. DAC estimates the condition-level bias using the paired midpoint of positive- and negative-demonstration mean log-odds and subtracts this estimate from the query log-odds.
Critical weaknesses
[W1] The justification for DAC’s bias estimate is incomplete. The paired midpoint cancels the parroting component only when its positive- and negative-group contributions are equal and opposite. The paper does not sufficiently justify or validate this assumption. It remains unclear when the estimate reliably recovers the condition-level bias.
[W2] Existing studies often use methods such as Top-k [1] and ConE [2] to select in-context demonstrations. It remains unclear whether DAC is compatible with these demonstration selection methods.
[1] What makes good in-context examples for GPT-3?
[2] Revisiting Demonstration Selection Strategies in In-Context Learning.
Questions for authors
[Q1] The evaluation only consider simple classification. It remains unclear whether these findings generalize to code generation, mathematical reasoning, and agent tasks.
[Q2] The experiments focus on ICL with direct label prediction. Does label collapse also occur when ICL is combined with chain-of-thought? It remains unclear whether DAC can be applied effectively in this setting.
[Q3] Is DAC still effective with an imbalanced demonstration set, such as one containing very few negative demonstrations?
Rebuttal Summary
We thank the AC for handling our submission and for coordinating the thoughtful reviews. Below we briefly summarize the strengths identified by the reviewers and how the rebuttal addressed their key concerns.
Reviewers found the discovery of label collapse in diffusion language models important, and the study broad and systematic across model families, word pairs, and demonstration settings. They highlight the clarity of the paper, the practical single-forward diagnostic, and the simplicity of the demo-anchored mitigation.
Most concerns focused on the consistency of the reported numbers, the robustness and scope of the evaluation, and the evidence behind the causal claims. During the discussion period we:
Some directions raised by reviewers, such as extending the diagnostic and repair beyond classification-style tasks, are acknowledged as important and left as future work. Overall, we believe the new experiments and clarifications substantially strengthen the paper and resolve the main technical and empirical concerns.
We hope the AC will consider the full discussion when making the final decision. We sincerely appreciate the AC's time, effort, and service to the community.