- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Diffusion In-Context Learner (DICL) for in-context semi-supervised learning (IC-SSL), leveraging a small set of labeled examples and a larger set of unlabeled examples. DICL uses a bidirectional Transformer trained by randomly masking a subset of unknown labels and minimizing the cross-entropy loss on masked positions. During inference, it progressively commits high-confidence pseudo-labels, incorporates them into the context for subsequent predictions, and allows the inference budget to be controlled through the number of denoising steps. The authors evaluate DICL on four synthetic settings: GMM, SubGMM, MIMO 2×2, and MIMO 4×4, comparing it with ICL, DEFINED, OICSSL, and classical statistical methods. The results show that DICL achieves lower prediction error in most settings. However, the independent contribution of the discrete diffusion objective and the effectiveness of the method on real-world data remain insufficiently validated.
Critical strengths
-
A simple iterative inference mechanism. DICL progressively commits high-confidence labels using a single frozen model, without updating model parameters during inference. The number of inference steps also provides a controllable computational budget.
-
Evaluation across multiple controlled settings. The authors evaluate DICL on GMM, SubGMM, and two MIMO detection settings, comparing it with several in-context learning approaches and classical statistical methods. The results demonstrate the potential benefits of exploiting unlabeled examples.
Critical weaknesses
-
The independent contribution of discrete diffusion remains unclear, and a key baseline is missing. DICL is trained using random label masking and cross-entropy reconstruction, while its inference procedure relies on confidence-based ordering and iterative pseudo-label feedback. The current experiments do not distinguish whether the performance gains arise from the diffusion-based training objective or from the combination of masked prediction and iterative self-training. In particular, the paper lacks a direct comparison against an ordinary bidirectional prediction model using the same confidence-based ordering and multi-step pseudo-label feedback. Consequently, the core methodological contribution has not been sufficiently established.
-
The experimental scope and robustness to pseudo-label errors are insufficiently validated. The experiments are limited to GMM, SubGMM, and simulated MIMO tasks, which may not adequately support the broader claims regarding a general IC-SSL framework. Furthermore, all visible labels during training are ground-truth labels, whereas labels progressively introduced during inference may contain errors. The paper does not sufficiently investigate how performance changes under different pseudo-label error rates or whether errors propagate across iterations and degrade subsequent predictions.
Questions for authors
-
Could the authors include a baseline that combines an ordinary bidirectional prediction model with confidence-based ordering and multi-step pseudo-label feedback, while controlling for model architecture, training data, and inference budget as closely as possible? How large is the performance gap between this baseline and DICL?
-
Could the authors systematically vary the error rate among visible pseudo-labels and report the prediction error at different inference steps? This would help establish DICL’s sensitivity to incorrect pseudo-labels and determine whether errors accumulate during iterative inference.
-
Could the authors validate DICL on at least one real-world dataset, or more explicitly narrow the claims regarding a general IC-SSL framework to match the scope of the current synthetic experiments?
3 comments
W2 & Q3. Real-world data
Thank you for your suggestion. We run in-context SSL experiments on frozen image embeddings with real-world data: We encode the 60k CIFAR-100 images with a frozen DINOv2-S/14 (CLS output), L2-normalize the embeddings and PCA-whiten them to on the training classes. The 100 classes are split at random into 64 training, 16 validation and 20 test classes, so every test task uses classes that no network has seen. A task draws classes, a class prior and a random assignment of label names. The first five examples cover the classes and all others are i.i.d. from , drawn without replacement; and . All learned methods use the 4-layer Transformers and are trained with training tasks. Each training context receives a random orthogonal map of the features and a random permutation of the label names.
| n_l | n_u | LogReg | QDA | EM-K=4 | LabelProp | ICL | DEFINED | OICSSL | DICL-N=1 | DICL-N=2 | DICL-N=4 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 5 | 16 | 25.41 | 25.65 | 22.41 | 23.59 | 24.31 | 24.18 | 20.25 | 20.38 | 20.23 | 20.25 |
| 5 | 32 | 25.44 | 25.69 | 20.86 | 22.69 | 24.37 | 23.81 | 18.36 | 18.62 | 18.21 | 18.23 |
| 5 | 64 | 25.45 | 25.69 | 19.50 | 21.99 | 24.37 | 23.20 | 16.58 | 17.07 | 16.44 | 16.41 |
| 10 | 16 | 18.35 | 17.37 | 15.59 | 17.80 | 16.71 | 16.38 | 14.32 | 14.21 | 14.22 | 14.26 |
| 10 | 32 | 18.34 | 17.39 | 14.58 | 17.18 | 16.74 | 16.07 | 13.23 | 13.13 | 13.06 | 13.06 |
| 10 | 64 | 18.32 | 17.34 | 13.50 | 16.63 | 16.70 | 15.58 | 12.18 | 12.11 | 12.00 | 11.98 |
We can observe that DICL outperform all classical methods (including semi-supervised EM and label propagation tuned on the validation set), as well as labeled-only ICL, DEFINED and OICSSL, on real embeddings of unseen classes, demonstrating the effectiveness of DICL on tasks with real-world data.
We hope that our response has addressed your concerns. Please let us know if you have any further questions.
Dear Reviewer 9HMh,
Thank you again for your time and valuable feedback.
We hope our rebuttal has adequately addressed your concerns. As the rebuttal period is approaching its deadline (November 18, Anywhere on Earth), we would greatly appreciate it if you could let us know whether you have any remaining questions or concerns. We would be happy to provide further clarification.
W1 & Q1. Independent contribution of the diffusion objective
Thank you for your question. We ran Algorithm 1 (R=0, same schedule, same budget T⋆) on the OICSSL checkpoint. OICSSL has the same architecture, training contexts, optimizer grid and selection rule as DICL and differs only in its objective.
Table R1. Long-block error (%), geometric mean over nu≥8.
What this shows:
W2 & Q2. Robustness to pseudo-label errors and error propagation
Thank you for your question. We agree that all visible labels in training are ground truth, whereas pseudo-labels at inference can be wrong. Actually, we have already evaluated the performance of DICL under different inference error rates in the original paper (Tables 4 and 5). Specifically, in our wireless communication experiments, varying the signal-to-noise ratio (SNR) from 5 to 25 dB effectively corresponds to evaluating the model under different inference error rates. As shown in Tables 4 and 5, DICL achieves the best performance across most of the considered SNR levels (and thus error rates), demonstrating its robustness to incorrect pseudo-labels. Furthermore, we conduct following additional experiments to explicitly investigate the model's sensitivity to incorrect commitments.
(a) Systematic sweep of the error rate of visible pseudo-labels. We reveal half of the query labels with error rate ε for testing. Wrong labels are drawn either uniformly or as the model's own runner-up class, which mimics its real mistakes. We report the error on the remaining queries after one pass (DICL):
We can notice that revealed pseudo-labels keep helping (error below the "none revealed" column) up to an error rate of 10–20% on SubGMM and 20–50% on GMM. The degradation is smooth in ε, with no sudden collapse.
(b) Prediction error at each inference step. Below is the error of the labels committed at each step of Algorithm 1 (SubGMM (8,32), N=8), against the one-shot error of the same positions:
We can notice that confidence ordering commits nearly error-free labels first, below the break-even rates in (a). Labels committed later are more accurate than the one-shot predictions at the same positions, so conditioning on earlier commitments helps.