- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes EfficientMatch for efficient semi-supervised learning. It combines confidence-based consistency regularization with a Mixup loss filtered by the mixing partner’s confidence mask, and measures the cost of reaching target accuracies through iterations, FLOPs, and wall-clock time.
Critical strengths
- The focus on reducing the computational cost of semi-supervised learning is valuable.
- The evaluation protocol combines complementary efficiency metrics coherently.
- The experiments show favorable FLOP and wall-clock costs at the selected accuracy targets.
Critical weaknesses
- The method mainly integrates existing techniques, and its methodological novelty appears limited.
- The evidence that unreliable pseudo-labels cause MixMatch’s low-label instability remains limited.
- The Mixup loss considers the confidence of only one of the two mixed samples.
- The efficiency gains are affected by different µ values across methods. Although some matched-µ results are provided, broader controlled comparisons are missing.
- Asymptotic accuracy and the choice of target accuracies are insufficiently explained or justified.
- Comparisons on larger datasets and with more representative methods are missing. Method names in Figure 1 are inconsistent with the text, and the column widths in Table 1 are unbalanced.
Questions for authors
- Can the authors better justify the methodological novelty and provide stronger evidence for MixMatch’s low-label instability?
- Can the authors consider the confidence of both mixed samples and provide broader comparisons with matched µ?
- Can the authors clarify what asymptotic accuracy means, explain why training cannot simply stop after convergence, and specify how the target accuracies are selected?
- Can the authors add comparisons on larger datasets, including ImageNet, and with more representative methods?
2 comments
Q3. Asymptotic accuracy, stopping after convergence, choice of targets.
- Asymptotic accuracy is now defined in Section 4: the accuracy at the end of the standard 2^20-iteration cosine schedule, more than 12 hours per run on our hardware.
- Why not stop after convergence. Stopping early is possible, and our protocol does exactly that: it stops when a fixed target is reached. Stopping "after convergence" needs a rule to detect it. But the end of the 2^20 schedule is a convention, not a true asymptote, and accuracy keeps rising without a clear plateau. On our full-horizon run, the best accuracy gains about 2 points per doubling of the iteration count from 2^17 to 2^20 (85.5%, 87.4%, 89.2%, 91.6%), and the learning rate is still above 75% of its initial value at 2^19. Declaring convergence at a plateau therefore depends on the scale at which progress is judged, and is itself a parameter that must be chosen. A fixed target makes that choice explicit and identical across methods. The revision adds this to Section 4.
- Targets. They were set after pilot runs, per configuration, as an accuracy reached by most methods within the 2-hour budget, chosen so that each configuration exposes one behavior: the fragility of adaptive thresholding (SVHN/250), MixMatch's reversal (CIFAR-10/250 against CIFAR-10/4000), and an early plateau against slower methods (CIFAR-100/2500). On CIFAR-100, 55% was tried first and dropped as too long. Because the targets are a choice, we added a threshold-free comparison: at fixed budgets, EfficientMatch ranks first or second everywhere; in the window that ends when EfficientMatch reaches its target, MixMatch ranks first in 11 of 12 cases and EfficientMatch second in 11 of 12.
Q4. Larger datasets, including ImageNet; more representative methods. We do not address this in the revision. Every experiment runs on a single GPU, and a three-seed ImageNet study under the same protocol does not fit in the discussion period. Among the five compared methods, RegMixMatch (AAAI 2025) is a recent Mixup-based method, the closest to ours. The paper also tests a FreeMatch-style adaptive threshold as a variant of EfficientMatch (Section 6). The Limitations section states the resulting scope (CIFAR and SVHN, WideResNets), and our claims are restricted to it.
Figure 1 and Table 1. Fixed. The panel titles of Figure 1 now use the method names of the text (FixMatch, MixMatch, EfficientMatch instead of fixmatch, mixmatch, efficientmatch), and the four columns of Table 1 have equal widths.
We thank the reviewer for his constructive review. Changes in the revised PDF are shown in blue.
Q1. Novelty, and evidence for MixMatch's low-label instability. Novelty. We agree that EfficientMatch integrates existing techniques: FixMatch's consistency loss and mask, and Mixup. We do not claim a new component. The revised introduction states two contributions: (i) an evaluation protocol that reports the cost of reaching a fixed pre-asymptotic accuracy in iterations, FLOPs and wall-clock time, and (ii) EfficientMatch, FixMatch's consistency loss with an added Mixup channel, together with an ablation of where a confidence filter should be placed in that channel. That ablation (Q2) shows that the filter is not the source of the speed-up. A further control (general comment) shows that part of the speed-up comes from the additional augmented views, and that Mixup does better than an extra strong view at the same cost. After revision, the paper contributes the protocol, a method that reaches the targets first in all 12 cases, and controlled ablations that show which parts of the method matter and correct our initial explanation.
Evidence on MixMatch. We agree that our evidence is correlational. The revision softens every causal statement: the section title no longer calls the reversal "a validation of our diagnosis", the paragraph now states that the reversal does not by itself explain MixMatch's failure, and FixMatch's failure on CIFAR-100 is no longer attributed to "the same reason". The closest controlled test we ran is unfiltered Mixup in EfficientMatch's own pipeline: it is the fastest variant in all 12 cases (Q2). So unfiltered Mixup is not harmful in itself, and our hypothesis that MixMatch's low-label failure comes from its unfiltered mix is not supported. Other differences remain, such as MixMatch's sharpened soft targets, its lack of a thresholded consistency loss, and its ramp-up. We also measured MixMatch's pseudo-label accuracy: it is lower than test accuracy by 4.7 to 5.3 points at 250 labels and by 3.7 to 4.7 points at 4000 labels (20k to 100k iterations, 3 seeds), a small difference compared with the difference in outcome. The revision presents the diagnosis as an open hypothesis and lists the alternative causes.
Q2. Confidence of both mixed samples; broader matched-μ comparisons. Both samples. You are right that only the partner's mask is used, and the revision now states this in Section 3. This was not our intent. We ran the comparison, changing only the mask of L_mix (same μ = 3, pool and FLOPs).
bothuses the product of the two masks, as you suggest. Corrected minutes to 80% on CIFAR-10/250:Requiring both samples to be reliable is 1.3× to 1.7× slower than the submitted mask. Unfiltered Mixup is faster than the submitted mask in all 12 cases (4 configurations × 3 seeds, 1.03× to 1.52×). The paper is revised accordingly.
Matched μ. The revision adds the converse comparison, using EfficientMatch runs at μ = 7 and μ = 1 already in our μ ablation (no new training).
This covers CIFAR-10 only, the dataset where we ran EfficientMatch at several values of μ.