- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
AccentMatch combines offline LLM-generated pseudo-labels with a student model’s predictions for semi-supervised text classification. Its main contribution is an instance-specific weighting mechanism based on an exponential moving average of prediction margins: the student receives more weight as its predictions become stable. The method builds on SoftMatch and is evaluated on six datasets with 20 and 50 labels per class. The paper reports improvements across most settings, supported by extensive ablations, multiple teacher and student models, and additional ten-run significance tests.
Critical strengths
A practical approach to controlling external supervision. The method addresses a meaningful problem: an LLM can provide useful early supervision but should not permanently constrain a student that becomes more accurate. Instance-specific weighting is a sensible solution, and Tables 2 and 4 show benefits over equal mixing, agreement filtering, fixed weights, and abrupt teacher removal.
Broad empirical support. The evaluation covers six datasets, several teachers, two student backbones, and extensions to other SSL methods. Appendix H reports significant improvements over the stronger of SoftMatch and VerifyMatch in 10 of 12 settings. This breadth makes the findings more convincing than a result limited to one benchmark or model.
Critical weaknesses
The experimental protocol is not sufficiently specified. RCV1 and GoEmotions support multiple labels per example, whereas the method uses a single argmax class and Appendix M explicitly requests exactly one label. The paper does not explain whether examples were filtered, labels were collapsed, or the method was adapted for multi-label prediction. This matters for interpreting the reported F1 scores and baseline comparisons. The exact Qwen checkpoint, dataset sizes and splits, and validation/model-selection procedure are also missing.
The justification for the particular gating rule remains incomplete. Proposition 3.1 establishes optimality for an oracle hard gate in a randomized prediction setting, assuming the teacher–student error crossover is already known. It does not establish optimality for sigmoid-weighted training targets. Moreover, the binary ablation switches at APM = 0, while the reported empirical crossovers occur around S = 0.6–0.7. Thus, that comparison does not cleanly isolate the benefit of smooth mixing from the choice of switching threshold. The experiments support adaptive fusion, but the specific gate is less well justified.
Questions for authors
-
How are multiple labels handled for RCV1 and GoEmotions? Please specify preprocessing, treatment of Neutral, resulting dataset sizes, and whether every baseline uses the same formulation.
-
Which exact Qwen checkpoint and training/model-selection settings were used? How are APM values initialized, updated across classes, and used during the ten-epoch warmup?
-
Does continuous gating still outperform a hard gate whose threshold is selected on validation data? This would help distinguish the benefit of smooth weighting from a poorly located switching threshold.
4 comments
W2 & Q3:
Thank you for highlighting this distinction. We agree that Proposition 3.1 provides a conditional motivation for competence-based selection; it does not establish the optimality of the proposed sigmoid fusion. We will revise the theoretical discussion to make this scope explicit.
Per your suggestion, we evaluated the binary hard-switching variant across a full grid of switching thresholds, where the model uses only the LLM pseudo-label when the stability score is below the threshold and switches completely to the student's pseudo-label once it reaches or exceeds the threshold. As shown in the table below, tuning the switching threshold improves over extreme values (0.1 or 0.9) as well as the unshifted APM = 0 baseline (tau = 0.5), with the best hard-gate performance occurring at 0.4 and 0.6 across settings (specifically, 0.6 achieves 61.3 on RCV1-20, 28.5 on GoEmotions-20, and 28.4 on GoEmotions-50, while 0.4 reaches 60.9 on RCV1-50).
Crucially, however, even the best-performing hard threshold in every setting consistently underperforms continuous AccentMatch across all datasets and label regimes (e.g., 61.3 vs. 63.2 on RCV1-20 and 28.5 vs. 30.2 on GoEmotions-20). These results show that continuous AccentMatch outperforms any fixed hard gate across the entire threshold grid, including one selected via validation tuning. We will include these results in the paper.
| Method / Hard Threshold (tau) | RCV1 (20) | RCV1 (50) | GoEmotions (20) | GoEmotions (50) |
|---|---|---|---|---|
| Hard Gate (tau = 0.1) | 59.5 | 59.9 | 26.8 | 27.2 |
| Hard Gate (tau = 0.3) | 60.1 | 60.2 | 27.1 | 27.6 |
| Hard Gate (tau = 0.4) | 60.8 | 60.9 | 27.9 | 28.3 |
| Hard Gate (tau = 0.5, APM = 0) | 59.8 | 59.7 | 27.4 | 28.1 |
| Hard Gate (tau = 0.6) | 61.3 | 60.7 | 28.5 | 28.4 |
| Hard Gate (tau = 0.7) | 61.0 | 60.5 | 28.3 | 28.1 |
| Hard Gate (tau = 0.8) | 60.4 | 60.3 | 27.8 | 28.0 |
| Hard Gate (tau = 0.9) | 59.8 | 60.0 | 27.3 | 27.5 |
| AccentMatch (Continuous Sigmoid) | 63.2 | 61.5 | 30.2 | 30.6 |
Thank you for the clarifications and additional experiments. They address my main concerns, and I will raise my score accordingly.
Thank you very much for your response and for raising your score. We appreciate your time, effort, and constructive feedback, which have helped improve the quality of the paper.
We thank the reviewer for the valuable and constructive feedback and suggestions that help improve the quality of our paper. We address the concerns below and we will incorporate them in our paper.
Thank you for pointing out these omissions. We clarify the exact preprocessing protocols, split sizes, and model checkpoints below, and will incorporate all of these details into Section 4.2 and Appendix A.
Both RCV1 and GoEmotions are evaluated strictly as single-label multiclass tasks:
GoEmotions: We preserve the official train and test splits and remove examples with multiple labels within each split. We also keep the neutral class, resulting in 28 classes: 27 emotions plus neutral.
RCV1: While the full RCV1 hierarchy contains 103 topic codes (referenced in Section 5 and Table 6), we restrict target annotations to the 82 leaf topic codes and keep only documents assigned to exactly one leaf code, discarding documents with multiple or zero leaf codes. We construct a custom split from the official 781,265-document test partition, assigning the first 200,000 documents chronologically (in ascending document-ID order) to training and the remaining 581,265 documents to testing, before applying this filtering. This yields 113,336 single-leaf training documents and 355,306 single-leaf test documents. We will update the class-count references in the text from 103 to 82 to reflect the leaf-code target space. Thank you again for raising this point.
For GoEmotions, IMDB, TREC-6, we use the standard benchmark train and test partitions. For Amazon Review and Yelp, we utilize the standard SSL text classification benchmark partition introduced in USB (Wang et al., 2022). RCV1 is discussed above.
Across all six datasets, we randomly sample k in {20, 50} labeled training examples per class without replacement. If a tail class in RCV1 or GoEmotions contains fewer than k examples after filtering, we include all available examples for that class in the labeled set L, which makes |L| slightly smaller than k * C. All remaining training examples form the unlabeled set U.
Baseline Parity: All baselines (supervised, LLM, and SSL) were run from scratch on these exact splits using identical evaluation metrics (macro-F1 for RCV1 and GoEmotions; accuracy for the remaining four datasets).
Reference: Yidong Wang et al., "USB: A Unified Semi-Supervised Learning Benchmark for Classification", NeurIPS 2022 Datasets and Benchmarks Track.
We provide the complete statistics across all six benchmarks below. We will also create a GitHub page for the project and make available our code and the exact splits for all datasets used in the paper for reproducibility purposes.
Our main teacher is Qwen3-8B, specifically the Qwen/Qwen3-8B checkpoint, described in Yang et al. (2025). We utilize nucleus sampling (temperature = 0.7, top_p = 0.8) and a maximum generation length of 256 new tokens.
Reference: Yang et al. (2025). Qwen3 Technical Report. arXiv:2505.09388.
For each unlabeled example, we maintain an APM value for every class, initializing each EMA with that class's first observed pseudo-margin. At each subsequent update, we compute the pseudo-margins and update the APMs for all classes (not just for the currently predicted class), using beta = 0.997.
The first ten epochs accumulate APM statistics without using them for adaptive teacher/student weighting, using an equal mixture (S = 0.5) of the teacher and student predictions while modulating the unsupervised loss with the confidence weight lambda(.) from Eq. 1. After warmup, we retrieve the accumulated APM of the student's current predicted class and apply the sigmoid to obtain its fusion weight S. All class-specific APMs continue to be updated throughout training, preserving their histories when the predicted class changes. We will clarify these details in the revised manuscript.