acceptodds
Under review as a conference paper at ICLR 2027

Label Collapse in Diffusion Language Model In-Context Learning

Abstract

In-context learning (ICL) applies large language models to new tasks by prepending a few labeled demonstrations to a query. Prior work has studied how demonstration ordering affects ICL, but the interaction between label composition, arrangement patterns, and model biases has not been systematically examined. We define label collapse as the failure mode in which the model generates query-independent labels, and show that it is architecturally universal: five models spanning autoregressive (Llama-3, Qwen3.5) and discrete diffusion (LLaDA, iLLaDA, LLaDA-Instruct) families all exhibit label collapse even under the class-balanced alternating arrangement, with some label word pairs collapsing in virtually every tested condition and others flipping their collapse direction with arrangement. The phenomenon replicates in a second diffusion lineage (Dream), where iterative decoding never reliably enters the label space but the single pass still tracks its priors. Critically, this collapse is concealed by standard accuracy metrics: models can achieve near-chance accuracy while generating query-independent labels, so the failure can persist undetected in deployment. We measure separation, the log-odds of the two label words at the query position (the quantity that label-bias calibration operates on), as a complementary diagnostic that exposes collapse hidden beneath surface accuracy, together with a measurement protocol that guards label extraction against tokenizer and formatting artifacts: a single forward pass identifies behavioral collapse with 92.5% precision, and batch calibration on as few as four queries, simulated offline, restores every collapsed condition to balance. For autoregressive models, collapse persists where the majority-label-bias account does not apply, and arrangement patterns can reverse the collapse direction; for diffusion models, we provide the first systematic evaluation of ICL failure modes and identify open questions about how iterative denoising interacts with label biases.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
In-Context Learning, Diffusion Language Model

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_uvGNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates label collapse in diffusion language model in-context learning. The authors observe that diffusion language model can produce query-independent labels even with class-balanced demonstrations. It proposes demo-anchored calibration (DAC), a simple method that estimates the condition-level bias from per-demonstration readings and subtracts it from the query log-odds. The authors evaluate label collapse and calibration effectiveness across six diffusion language models and two classification datasets

Critical strengths

[S1] The paper reports an important finding: diffusion language models can produce nearly the same label regardless of the query input, even with class-balanced demonstrations.

[S2] The method is simple and easy to understand. DAC estimates the condition-level bias using the paired midpoint of positive- and negative-demonstration mean log-odds and subtracts this estimate from the query log-odds.

Critical weaknesses

[W1] The justification for DAC’s bias estimate is incomplete. The paired midpoint cancels the parroting component only when its positive- and negative-group contributions are equal and opposite. The paper does not sufficiently justify or validate this assumption. It remains unclear when the estimate reliably recovers the condition-level bias.

[W2] Existing studies often use methods such as Top-k [1] and ConE [2] to select in-context demonstrations. It remains unclear whether DAC is compatible with these demonstration selection methods.

[1] What makes good in-context examples for GPT-3?

[2] Revisiting Demonstration Selection Strategies in In-Context Learning.

Questions for authors

[Q1] The evaluation only consider simple classification. It remains unclear whether these findings generalize to code generation, mathematical reasoning, and agent tasks.

[Q2] The experiments focus on ICL with direct label prediction. Does label collapse also occur when ICL is combined with chain-of-thought? It remains unclear whether DAC can be applied effectively in this setting.

[Q3] Is DAC still effective with an imbalanced demonstration set, such as one containing very few negative demonstrations?

Reviewer_UFTGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies label collapse in diffusion language models during in-context classification, where models repeatedly predict one label regardless of the input. The authors evaluate this behavior across several models, label pairs, and demonstration settings, and compare single-pass predictions with iterative generation. They also propose demo-anchored calibration, which estimates label bias from the demonstrations and adjusts the model's label scores.

Critical strengths
  1. The paper provides a broad evaluation of label collapse across six diffusion language models, 20 label pairs, and different demonstration settings.

  2. The single-pass diagnostic is useful. It predicts generation collapse with 93.7% precision and avoids the cost of iterative decoding.

  3. DAC is simple and shows promising results in correcting label imbalance. The authors also document parsing issues and unsuccessful interventions.

Critical weaknesses
  1. Some key results are inconsistent. Figure 1 reports 94% accuracy for LLaDA-Base with positive/negative labels, while Table 1 reports complete collapse under apparently the same setting. This raises concerns about the reliability of the evaluation.

  2. Dream accounts for 28 of the 67 collapsed conditions, but nearly all its generated outputs are invalid. The single-pass readout and generation pipeline need further validation.

  3. The experimental setup does not sufficiently control for label mapping and demonstration order. Since the first-listed label always represents the positive class, some reported label-word biases may instead reflect ordering effects.

  4. DAC's reported improvements measure label balance rather than classification accuracy. Its effectiveness during actual generation is not demonstrated, and the causal interpretation relies mainly on correlation.

  5. The novelty of DAC is unclear without comparison to LOOC [1], which also uses in-context demonstrations for calibration. Similar collapse behavior is already known in autoregressive models, including the paper's own baselines.

[1] Beyond Performance: Quantifying and Mitigating Label Bias in LLMs (Reif & Schwartz, NAACL 2024)

Questions for authors
  1. Can the authors explain the discrepancy between Figure 1 and Table 1 and verify the Dream evaluation pipeline, particularly its token indexing and high invalid-output rate?

  2. Do the main results hold when label mappings and demonstration order are reversed, and across different demonstration samples and prompt templates?

  3. Does DAC improve classification accuracy and actual generation, rather than only label balance? How does it compare with LOOC and small-batch calibration?

Reviewer_piQcOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper investigates label collapse in the in-context learning (ICL) of diffusion language models (dLLMs), where models produce nearly query-independent label predictions. The authors systematically evaluate six dLLMs from three model families (LLaDA, iLLaDA, and Dream), covering 20 label-word pairs, different numbers of demonstrations, and four arrangements.

The authors propose measuring separation as a diagnostic method, specifically the log-odds of two labels. Based on this diagnostic method, they introduce Demo-Anchored Calibration (DAC), which estimates a label-bias offset and subtracts it from the query separation scores. The authors report that DAC restores balanced prediction rates in 53 out of 67 collapsed conditions at the K=8 snapshot.

Critical strengths

The paper provides a systematic empirical characterization of label collapse across multiple dLLM families, label-word pairs, demonstration counts, and arrangements.

Critical weaknesses
  • While the paper provides an extensive empirical characterization of label collapse in dLLMs, its conceptual and methodological novelty appears limited. As the authors acknowledge in the Related Work section, separation (the log-odds difference between candidate labels) and calibration methods that estimate and subtract label-bias offsets have already been extensively studied (e.g., Zhou et al., 2024). Although DAC introduces a demonstration-anchored estimator, it follows the same underlying calibration framework, with the main distinction being the choice of reference point. The paper does not sufficiently clarify what substantial methodological advance DAC offers over existing calibration approaches.
  • Despite the extensive number of experimental configurations, the evaluation is restricted to six dLLMs from three model families, all at approximately the 7B–8B scale, two English datasets, and a fixed binary classification template. The use of 20 label-word pairs provides useful coverage of label-token sensitivity, but does not establish whether the findings generalize to substantially different task formats, multi-class classification, larger or smaller models, or more diverse linguistic settings.
  • The paper presents a large number of experimental findings but offers limited synthesis of their scientific implications, making it difficult to follow the central narrative and identify the key contributions. For example, the mechanistic analysis in Section 4 provides limited explanatory insight. The statement that "in all six models the bias is present at or before the final projection layer" is essentially trivial, as the final predictions are necessarily determined by the hidden representations and output projection. Furthermore, the Logit Lens observations do not establish where the bias causally originates, yet the conclusions are presented as mechanistic findings. Some claims are also overly general or insufficiently qualified. For instance, the paper states that "hard collapse is arrangement-indifferent," while also reporting that Dream-Instruct leaves the collapsed band on circle/square in four large-K non-alternating conditions. Although these exceptions do not invalidate the overall trend, they should be more clearly reflected in the conclusions.
Questions for authors

The paper reports that DAC repairs 53 out of 67 collapsed conditions, but this result is based on restoring the positive-label rate rather than classification accuracy. Since DAC only shifts the decision threshold without changing the ranking of queries, a more balanced prediction distribution does not necessarily imply improved classification performance. Could the authors report the accuracy before and after DAC, including how many of the 53 repaired conditions achieve meaningful accuracy improvements? Additionally, how does DAC compare with existing calibration baselines in terms of accuracy recovery on the same evaluation conditions?

Reviewer_Y4bUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates label collapse in diffusion language models where the models generate query-independent labels. While the mechanism behind autoregressive LMs has been widely studied, a study in a similar direction has not been published for diffusion LMs, which is a reasonable point. The authors demonstrate that this phenomenon is quite widespread across different families of diffusion LMs, even under class-balanced alternating demonstrations. They also propose a method to efficiently predict the occurrence of such a phenomenon in a single pass, by measuring the log-odds of two label words at the query position. They also propose a method (demo-anchored calibration) to mitigate the problem by removing a bias (obtinaed through demonsrtations of each label) from the query.

Critical strengths
  • The paper is clearly written and easy to follow.
  • The analysis is thorough, and the proposed mitigation method is simple, well motivated by the empirical findings, and effective across different diffusion LM families.
Critical weaknesses
  • The experimental coverage, as the authors mentioned, is narrow in some dimensions: one language, two datasets, one template, and a single model scale (7–9B).
  • Since both the phenomenon and the DAC remedy are studied only on two binary tasks, and the DAC estimator is defined as the midpoint of two label means, it remains unclear whether either transfers to multi-class or more realistic classification settings.
  • Although the appendix shows that Llama-3 collapses about as often as LLaDA and Dream, the paper does not analyze what this similarity implies for its framing of label collapse as a diffusion LM failure mode, nor what distinguishes the two model classes mechanistically.
Questions for authors
  • How could the proposed method be extended to tasks beyond binary classification?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We thank the AC for handling our submission and for coordinating the thoughtful reviews. Below we briefly summarize the strengths identified by the reviewers and how the rebuttal addressed their key concerns.

Reviewers found the discovery of label collapse in diffusion language models important, and the study broad and systematic across model families, word pairs, and demonstration settings. They highlight the clarity of the paper, the practical single-forward diagnostic, and the simplicity of the demo-anchored mitigation.

Most concerns focused on the consistency of the reported numbers, the robustness and scope of the evaluation, and the evidence behind the causal claims. During the discussion period we:

  • Resolved the inconsistency between Figure 1 and Table 1: we reproduced both results and confirmed the underlying data are authentic; the discrepancy came from how the two numbers were measured, and we will revise the ambiguous figure as suggested.
  • Verified robustness across demonstration samplings, prompt templates, label arrangements (including the mirrored arrangement), and label orders.
  • Extended the evaluation with a four-class text classification experiment on five models in both the generation and single-forward settings, showing the phenomenon is not confined to binary templates.
  • Completed the accuracy evaluation: the estimate computed only from the demonstrations recovers classification accuracy on the collapsed conditions where prior calibration methods largely do not.
  • Strengthened the causal account: the offset estimate proposed in the paper, computed from the demonstrations alone, predicts how strongly each condition collapses, and counteracting this same estimate at decoding time reverses the collapse, indicating that it captures the cause of the phenomenon.
  • Disclosed a boundary openly: the repair does not extend to the four-class setting, and further experiments indicate this is structural rather than a sample-size effect. We report this result and discuss what it implies for this class of corrections.

Some directions raised by reviewers, such as extending the diagnostic and repair beyond classification-style tasks, are acknowledged as important and left as future work. Overall, we believe the new experiments and clarifications substantially strengthen the paper and resolve the main technical and empirical concerns.

We hope the AC will consider the full discussion when making the final decision. We sincerely appreciate the AC's time, effort, and service to the community.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.