Label Collapse in Diffusion Language Model In-Context Learning

In-context learning (ICL) applies large language models to new tasks by prepending a few labeled demonstrations to a query. Prior work has studied how demonstration ordering affects ICL, but the interaction between label composition, arrangement patterns, and model biases has not been systematically examined. We define label collapse as the failure mode in which the model generates query-independent labels, and show that it is architecturally universal: five models spanning autoregressive (Llama-3, Qwen3.5) and discrete diffusion (LLaDA, iLLaDA, LLaDA-Instruct) families all exhibit label collapse even under the class-balanced alternating arrangement, with some label word pairs collapsing in virtually every tested condition and others flipping their collapse direction with arrangement. The phenomenon replicates in a second diffusion lineage (Dream), where iterative decoding never reliably enters the label space but the single pass still tracks its priors. Critically, this collapse is concealed by standard accuracy metrics: models can achieve near-chance accuracy while generating query-independent labels, so the failure can persist undetected in deployment. We measure separation, the log-odds of the two label words at the query position (the quantity that label-bias calibration operates on), as a complementary diagnostic that exposes collapse hidden beneath surface accuracy, together with a measurement protocol that guards label extraction against tokenizer and formatting artifacts: a single forward pass identifies behavioral collapse with 92.5% precision, and batch calibration on as few as four queries, simulated offline, restores every collapsed condition to balance. For autoregressive models, collapse persists where the majority-label-bias account does not apply, and arrangement patterns can reverse the collapse direction; for diffusion models, we provide the first systematic evaluation of ICL failure modes and identify open questions about how iterative denoising interacts with label biases.