acceptodds
Under review as a conference paper at ICLR 2027

EfficientMatch: Faster Convergence in Semi-Supervised Learning

Abstract

Semi-supervised learning (SSL) for image classification is most often evaluated under a now-standard protocol consisting of a fixed budget of iterations and a shared architecture. While this benchmark has facilitated comparisons, it has also made asymptotic accuracy the primary design criterion, often at the cost of substantial computation. A complementary question remains comparatively unexplored: which method reaches a useful performance level at the lowest computational cost? We address this gap through two contributions. First, we introduce a unified evaluation protocol that measures the computational budget required to reach a predefined pre-asymptotic accuracy threshold. This budget is assessed jointly in terms of iterations, FLOPs, and wall-clock time, capturing complementary aspects of efficiency and revealing trade-offs that are obscured when considering a single metric. Second, we propose EfficientMatch, an SSL method specifically designed for this efficiency-oriented regime. Its design is motivated by a simple observation: Mixup can accelerate learning when pseudo-labels are reliable, but may hinder convergence otherwise. To address this issue, EfficientMatch exploits pseudo-label confidence to selectively filter which pseudo-labeled samples contribute to the Mixup loss, improving convergence without introducing additional thresholds. Experiments on CIFAR-10, CIFAR-100, and SVHN show that EfficientMatch reaches high target performance thresholds faster than MixMatch, FixMatch, FlexMatch, and RegMixMatch when measured in both FLOPs and wall-clock time, while maintaining a low per-iteration computational cost.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
semi-supervised learning, pseudo-labeling, training efficiency, mixup, confidence thresholding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_d4ZCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes EfficientMatch for efficient semi-supervised learning. It combines confidence-based consistency regularization with a Mixup loss filtered by the mixing partner’s confidence mask, and measures the cost of reaching target accuracies through iterations, FLOPs, and wall-clock time.

Critical strengths
  • The focus on reducing the computational cost of semi-supervised learning is valuable.
  • The evaluation protocol combines complementary efficiency metrics coherently.
  • The experiments show favorable FLOP and wall-clock costs at the selected accuracy targets.
Critical weaknesses
  • The method mainly integrates existing techniques, and its methodological novelty appears limited.
  • The evidence that unreliable pseudo-labels cause MixMatch’s low-label instability remains limited.
  • The Mixup loss considers the confidence of only one of the two mixed samples.
  • The efficiency gains are affected by different µ values across methods. Although some matched-µ results are provided, broader controlled comparisons are missing.
  • Asymptotic accuracy and the choice of target accuracies are insufficiently explained or justified.
  • Comparisons on larger datasets and with more representative methods are missing. Method names in Figure 1 are inconsistent with the text, and the column widths in Table 1 are unbalanced.
Questions for authors
  • Can the authors better justify the methodological novelty and provide stronger evidence for MixMatch’s low-label instability?
  • Can the authors consider the confidence of both mixed samples and provide broader comparisons with matched µ?
  • Can the authors clarify what asymptotic accuracy means, explain why training cannot simply stop after convergence, and specify how the target accuracies are selected?
  • Can the authors add comparisons on larger datasets, including ImageNet, and with more representative methods?
2 comments

We thank the reviewer for his constructive review. Changes in the revised PDF are shown in blue.

Q1. Novelty, and evidence for MixMatch's low-label instability. Novelty. We agree that EfficientMatch integrates existing techniques: FixMatch's consistency loss and mask, and Mixup. We do not claim a new component. The revised introduction states two contributions: (i) an evaluation protocol that reports the cost of reaching a fixed pre-asymptotic accuracy in iterations, FLOPs and wall-clock time, and (ii) EfficientMatch, FixMatch's consistency loss with an added Mixup channel, together with an ablation of where a confidence filter should be placed in that channel. That ablation (Q2) shows that the filter is not the source of the speed-up. A further control (general comment) shows that part of the speed-up comes from the additional augmented views, and that Mixup does better than an extra strong view at the same cost. After revision, the paper contributes the protocol, a method that reaches the targets first in all 12 cases, and controlled ablations that show which parts of the method matter and correct our initial explanation.

Evidence on MixMatch. We agree that our evidence is correlational. The revision softens every causal statement: the section title no longer calls the reversal "a validation of our diagnosis", the paragraph now states that the reversal does not by itself explain MixMatch's failure, and FixMatch's failure on CIFAR-100 is no longer attributed to "the same reason". The closest controlled test we ran is unfiltered Mixup in EfficientMatch's own pipeline: it is the fastest variant in all 12 cases (Q2). So unfiltered Mixup is not harmful in itself, and our hypothesis that MixMatch's low-label failure comes from its unfiltered mix is not supported. Other differences remain, such as MixMatch's sharpened soft targets, its lack of a thresholded consistency loss, and its ramp-up. We also measured MixMatch's pseudo-label accuracy: it is lower than test accuracy by 4.7 to 5.3 points at 250 labels and by 3.7 to 4.7 points at 4000 labels (20k to 100k iterations, 3 seeds), a small difference compared with the difference in outcome. The revision presents the diagnosis as an open hypothesis and lists the alternative causes.

Q2. Confidence of both mixed samples; broader matched-μ comparisons. Both samples. You are right that only the partner's mask is used, and the revision now states this in Section 3. This was not our intent. We ran the comparison, changing only the mask of L_mix (same μ = 3, pool and FLOPs). both uses the product of the two masks, as you suggest. Corrected minutes to 80% on CIFAR-10/250:

Mask of L_mix231203082701
none (unfiltered Mixup)24.414.312.2
partner (submitted)29.620.317.9
anchor41.628.521.6
premix42.830.027.9
both45.933.423.7

Requiring both samples to be reliable is 1.3× to 1.7× slower than the submitted mask. Unfiltered Mixup is faster than the submitted mask in all 12 cases (4 configurations × 3 seeds, 1.03× to 1.52×). The paper is revised accordingly.

Matched μ. The revision adds the converse comparison, using EfficientMatch runs at μ = 7 and μ = 1 already in our μ ablation (no new training).

  • At μ = 7, the value of FixMatch, FlexMatch and RegMixMatch, EfficientMatch reaches the target faster and with fewer FLOPs than all three on every seed of CIFAR-10/250 and CIFAR-10/4000 (time ratios 1.2× to 3.2×, FLOP ratios 1.2× to 2.8×).
  • At μ = 1 on CIFAR-10/4000, it does so against our version of MixMatch on every seed; against MixMatch with the published L_x, it is slower on one of the three seeds.

This covers CIFAR-10 only, the dataset where we ran EfficientMatch at several values of μ.

Q3. Asymptotic accuracy, stopping after convergence, choice of targets.

  • Asymptotic accuracy is now defined in Section 4: the accuracy at the end of the standard 2^20-iteration cosine schedule, more than 12 hours per run on our hardware.
  • Why not stop after convergence. Stopping early is possible, and our protocol does exactly that: it stops when a fixed target is reached. Stopping "after convergence" needs a rule to detect it. But the end of the 2^20 schedule is a convention, not a true asymptote, and accuracy keeps rising without a clear plateau. On our full-horizon run, the best accuracy gains about 2 points per doubling of the iteration count from 2^17 to 2^20 (85.5%, 87.4%, 89.2%, 91.6%), and the learning rate is still above 75% of its initial value at 2^19. Declaring convergence at a plateau therefore depends on the scale at which progress is judged, and is itself a parameter that must be chosen. A fixed target makes that choice explicit and identical across methods. The revision adds this to Section 4.
  • Targets. They were set after pilot runs, per configuration, as an accuracy reached by most methods within the 2-hour budget, chosen so that each configuration exposes one behavior: the fragility of adaptive thresholding (SVHN/250), MixMatch's reversal (CIFAR-10/250 against CIFAR-10/4000), and an early plateau against slower methods (CIFAR-100/2500). On CIFAR-100, 55% was tried first and dropped as too long. Because the targets are a choice, we added a threshold-free comparison: at fixed budgets, EfficientMatch ranks first or second everywhere; in the window that ends when EfficientMatch reaches its target, MixMatch ranks first in 11 of 12 cases and EfficientMatch second in 11 of 12.

Q4. Larger datasets, including ImageNet; more representative methods. We do not address this in the revision. Every experiment runs on a single GPU, and a three-seed ImageNet study under the same protocol does not fit in the discussion period. Among the five compared methods, RegMixMatch (AAAI 2025) is a recent Mixup-based method, the closest to ours. The paper also tests a FreeMatch-style adaptive threshold as a variant of EfficientMatch (Section 6). The Limitations section states the resulting scope (CIFAR and SVHN, WideResNets), and our claims are restricted to it.

Figure 1 and Table 1. Fixed. The panel titles of Figure 1 now use the method names of the text (FixMatch, MixMatch, EfficientMatch instead of fixmatch, mixmatch, efficientmatch), and the four columns of Table 1 have equal widths.

Reviewer_qoisOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an evaluation protocol that measures the iterations, FLOPs and wall-clock time SSL methods need to reach a fixed pre-asymptotic accuracy.

Critical strengths

The method is simple, adds no new threshold, and is the cheapest in time and FLOPs on every reported seed and configuration.

Critical weaknesses

Novelty is limited, since combining FixMatch-style confidence masking with Mixup is close to RegMixMatch and earlier mix-based SSL methods; the difference comes down to which mask is applied and where. The central claim that filtering by the partner’s mask after mixing is what matters is never directly tested: there’s no ablation against no filtering, anchor-mask filtering, or filtering before mixing, even though the text says such variants are evaluated. The protocol is fragile: the target thresholds are chosen by the authors and the ranking flips when they move, results depend on a 2-hour cap and one GPU and compile stack, and baselines keep their default settings while EfficientMatch’s unlabeled ratio was tuned for FLOPs. The scope is small, with only CIFAR and SVHN on WideResNets and the 40-label regime excluded, and the single-seed full run plateaus around 91.6% on CIFAR-10/250, well below FixMatch’s usual numbers, so the speedup may cost final accuracy.

Questions for authors

Could you report a threshold-free measure, such as the area under the accuracy-versus-time or accuracy-versus-FLOPs curve, to show the conclusions don’t hinge on the chosen targets?

4 comments

We thank the reviewer. Several of these concerns were correct, so we measured them. Changes in the revised PDF are shown in blue.

Threshold-free measure (your question). We added it in a new appendix, "Threshold-free Comparison". Each method is scored by its mean test accuracy over a budget [0, H], i.e. the area under its accuracy curve divided by H, along corrected wall-clock time and along FLOPs, on all runs (3 seeds × 4 configurations).

A run stopped at its target keeps its last accuracy. At fixed budgets (15 to 120 minutes, and 2 to 200 PFLOPs), EfficientMatch ranks first or second on average in every configuration and at every budget, the only method to do so; MixMatch drops to fifth on CIFAR-10/250 at 120 minutes. But keeping the last accuracy penalizes the methods that stop first, and EfficientMatch stops first in all 12 cases. We therefore also take H as the first moment any method reaches its target, so that no curve is truncated. In all 12 cases this is the moment EfficientMatch reaches its target. In that window:

  • MixMatch ranks first in 11 of 12 cases along time and in all 12 along FLOPs. It rises fastest early in training.
  • EfficientMatch ranks second in 11 of 12 cases on each axis. In the remaining case it ranks first along time and third along FLOPs.
  • FixMatch, FlexMatch and RegMixMatch rank below both, except in these two cases.

So the conclusions do depend on the targets, as you suspected. EfficientMatch's advantage is in reaching high targets first, not in accuracy at every budget, and the revision now says so from the abstract on. To see what happens with no target at all, we ran MixMatch and FlexMatch for the full 2 hours on CIFAR-10/250 (one seed, 42), and compared them with our full-horizon runs of EfficientMatch, RegMixMatch and FixMatch over the same 2 hours. MixMatch is ahead during the first 10 minutes (67.0% against 59.3% for EfficientMatch at 5 minutes). From about 10 minutes to the end, EfficientMatch is ahead: at 2 hours its best accuracy is 86.5%, against 84.6% for RegMixMatch, 82.1% for FixMatch, 81.9% for FlexMatch and 78.2% for MixMatch, and its mean accuracy over the 2 hours is 81.4%, against 76.1% to 76.5% for the others. This is a single seed.

The central claim is never tested. You are right, and the sentence claiming such variants were evaluated was wrong; we removed it. We have now run the ablation, with only the mask of L_mix changing (same μ, pool and FLOPs). On CIFAR-10/250 (target 80%), corrected minutes to the target:

Mask of L_mix231203082701
none (unfiltered Mixup)24.414.312.2
partner (submitted)29.620.317.9
anchor41.628.521.6
premix42.830.027.9
both45.933.423.7

The central claim does not hold. Unfiltered Mixup is faster than the submitted mask in all 12 cases (4 configurations × 3 seeds, 1.03× to 1.52×), and stricter masks are slower. The revision says so in the abstract, the contributions and the conclusion.

Where the speed-up comes from. To separate mixing from the extra pass, we replaced the mixed inputs of the third pass by a fresh strong view of the same samples, with their own targets, at the same cost per iteration. Iterations to 80% on CIFAR-10/250:

Third pass231203082701
none (FixMatch, μ = 3)202.5k197.0k110.0k
extra strong view, confidence mask104.5k57.0k43.5k
extra strong view, no mask54.0k36.0k35.0k
unfiltered Mixup34.5k20.0k17.0k

The last three rows have the same cost per iteration; a FixMatch iteration at μ = 3 costs 0.56× as much. Each step helps on every seed: the extra pass alone (1.9× to 3.5× fewer iterations), removing its filter (a further 1.2× to 1.9×), and mixing (a further 1.6× to 2.1×). In FLOPs, the extra pass alone saves 1.1× to 1.9× over FixMatch at μ = 3, the next two steps keep their ratios, and unfiltered Mixup needs 3.3× to 5.5× fewer FLOPs than FixMatch at μ = 3. This shows that the confidence filter is not the cause of the speed-up, contrary to our claim, and that part of the speed-up comes from training on additional augmented views. Mixing does better than an extra strong view at the same cost, which suggests that it is a more useful augmentation in this setting, but our experiments do not show why. The revision presents the cause of the speed-up as an open question.

Novelty relative to RegMixMatch. We agree that the ingredients exist. Section 3 ("Differentiation") lists the mechanistic differences with RegMixMatch: one reused threshold instead of a separate, stricter admission threshold; pixel-space Mixup instead of ResizeMix; an anchor-dominated mix (λ ≥ 0.5) instead of α = 1. The ablation above shows that the mask is not what makes EfficientMatch fast, so we no longer present the mask's placement as the source of its efficiency. What the paper contributes is an evaluation protocol for the cost of reaching a fixed accuracy, and the measurements made with it. EfficientMatch reaches the target first in all 12 cases, and at the default μ of each method, its iteration costs 2.6× fewer FLOPs than RegMixMatch's (740 against 1892 GFLOPs).

Targets chosen by the authors. They were not fixed in advance. The revision states this: we set them after pilot runs, per configuration, as an accuracy reached by most methods within the budget, chosen to expose a specific behavior in each configuration. The threshold-free appendix shows how much the ranking depends on that choice.

2-hour cap, one GPU and compile stack. These are now listed in the Limitations section. Runs that do not reach the target are marked † with their maximum accuracy. Wall-clock times depend on our stack, so we also report FLOPs, which do not.

Baselines at default settings, μ tuned for EfficientMatch. We agree that this asymmetry may favor EfficientMatch, and the Limitations section now states it. Two matched-μ comparisons address it on CIFAR-10:

  • at μ = 3: FixMatch needs 2.4× to 3.8× more FLOPs (corrected from 5.1× to 7.9×; see the general comment), and RegMixMatch fails on 2 of 3 seeds;
  • at μ = 7: EfficientMatch runs already in our μ ablation (no new training) are faster and use fewer FLOPs than FixMatch, FlexMatch and RegMixMatch on every seed of CIFAR-10/250 and CIFAR-10/4000. The same holds against our version of MixMatch at μ = 1 on CIFAR-10/4000, but not on every seed against MixMatch with the published L_x (see our answer to Reviewer 4QzD).

Scope. We agree that it is limited: CIFAR and SVHN, WideResNets, no 40-label regime (pilot runs were unstable across seeds). The revision states that we make no claim beyond these configurations.

The 91.6% plateau. It is in fact not a plateau: the run is still rising at the end of the schedule (mean accuracy 90.1% over iterations 700k to 800k, 91.3% over the last 48k), and the appendix now says so. We agree that 91.6% is below the 94.93% reported for FixMatch on CIFAR-10/250. We have now run FixMatch for the full horizon in our framework, on the same seed: it reaches 88.4% (88.3% at the end), 3.2 points below EfficientMatch and 6.5 points below its published result. So in our framework, none of the three methods reaches the published accuracies, and EfficientMatch remains the most accurate. Results also depend on the labeled split: on CIFAR-10/250, MixMatch reaches 76.5%, 77.7% or 79.5% depending on the seed (appendix on inter-seed variability), and seed 42 is a single split. The gap may come from this split, from our pipeline and its optimizations, or from both. The published standard deviation of FixMatch (±0.65 over 5 splits) suggests that the split alone does not explain 6.5 points. The run is a single seed, and its RegMixMatch counterpart stopped at 40% of its horizon. We removed the Discussion paragraph that drew conclusions from it, and no result in the paper is a claim about final accuracy.

Thank you for the detailed and candid response. The new experiments address most of my concerns. The revision now claims no more than the evidence supports. The Limitations section covers the targets, the compute setup and the μ asymmetry, and the scope is stated clearly. I appreciate this, and I am inclined to raise my score.

Thank you for your reply. We report one result obtained since our response, on the regime that our response said we had excluded.

CIFAR-10 with 40 labels. We have now run this regime, with the paper's hyperparameters, 2 hours of training and no accuracy target:

  • EfficientMatch and FixMatch on three seeds;
  • the other methods on one seed (2312).

Best test accuracy (%) reached by each time (raw wall-clock):

MethodSeed30 min60 min90 min120 min
EfficientMatch231257.971.676.679.3
FixMatch231247.865.470.671.8
EfficientMatch030872.378.880.581.5
FixMatch030862.575.878.780.2
EfficientMatch270165.871.874.476.4
FixMatch270157.570.773.575.5
EfficientMatch, unfiltered Mixup231260.265.374.181.9
FlexMatch231238.745.557.573.0
MixMatch231254.458.458.458.4
RegMixMatch231237.049.052.452.4
  • EfficientMatch is ahead of FixMatch at every time on all three seeds. At 120 minutes, the gap is 7.5, 1.3 and 0.9 points.
  • The results depend strongly on the seed: EfficientMatch ends between 76.4% and 81.5%.
  • On seed 2312, MixMatch and RegMixMatch stop improving early. The accuracy of RegMixMatch then falls to 34.4% at the end of the run.
  • No run has converged after 2 hours. These numbers are therefore not comparable to the published full-schedule results (13.81 ± 3.37% error for FixMatch).

So the 40-label regime works within our budget, but the results vary across seeds.

Changes to the revision. We will add this experiment to the appendix. The Limitations section will no longer say that the 40-label regime is excluded. It will state that this regime depends strongly on the seed.

Reviewer_4QzDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

This paper studies the training cost required to reach a specified accuracy in semi-supervised image classification, focusing on efficiency before asymptotic convergence. It jointly evaluates iteration count, FLOPs, and wall-clock time, and proposes EfficientMatch. The method adds a Mixup loss to a FixMatch-style supervised and consistency objective, retaining each mixed-example loss according to the mixing partner’s existing pseudo-label confidence mask, without introducing a second confidence threshold. Experiments cover four dataset/label-budget configurations on CIFAR-10, SVHN, and CIFAR-100. Tables 2–3 report lower time and FLOPs for EfficientMatch than the included baselines at the selected accuracy targets. Additional experiments examine the unlabeled batch ratio, Mixup weight, target representation, and adaptive thresholding. The paper acknowledges that rankings depend on the accuracy target and does not claim universal superiority in asymptotic accuracy.

Critical strengths
  1. The paper focuses on the cost of reaching a target accuracy, which is a useful perspective for semi-supervised learning. EfficientMatch shows promising improvements in runtime and FLOPs across the evaluated settings.
  2. Reporting iterations, FLOPs, and wall-clock time provides a more complete picture than using a single metric. The ablations and discussion of different accuracy targets also help clarify when the proposed method is beneficial.
Critical weaknesses
  1. Equation (5) masks only the mixing partner, allowing low-confidence anchors to dominate the mixed target since λ≥0.5\lambda \geq 0.5. Direct comparisons with unfiltered Mixup, both-sample masking, and pre-mixing filtering are needed to isolate the benefit of the proposed strategy.
  2. Tables 2 and 10 report nearly identical per-step FLOPs for FixMatch with μ=7\mu=7 and μ=3\mu=3, raising questions about the reported efficiency gains. Fast FixMatch is also missing from the target-accuracy comparisons, despite being a closely related method.
Questions for authors

The FLOP-counting script assumes static_shapes=True, while the main experiments appear to use static_shapes=False, making the reported FLOPs potentially inconsistent with actual training. Could the authors clarify the configurations used in Tables 2–3 and recompute FLOPs using actual batch sizes if dynamic batching was enabled, given that the logged mask_ratio does not appear to track Mixup selection?

4 comments

We thank the reviewer for a precise review. The FLOPs inconsistency you found was a real error, and it led us to re-check every number in the paper (see the general comment). Changes in the revised PDF are shown in blue.

Q1. Why does FixMatch's cost stay near 850 GFLOPs per step at μ = 3? What is counted? This was an error in our cost table. It had no entry for FixMatch at μ = 3 and fell back to the μ = 7 cost (850.10 GFLOPs per iteration). The measured cost at μ = 3 is 411.33 GFLOPs per iteration. The runs themselves were at μ = 3, since the flag sets the unlabeled loader's batch size, so the steps and times of the matched-μ table (Table 12 in the revision, Table 10 in the submission) are unchanged. Corrected FLOPs (CIFAR-10/250, target 80%):

SeedFixMatch μ=3, PFLOPs (submitted → corrected)EfficientMatchRatio
2312172.1 → 83.331.12.7×
0308167.5 → 81.021.13.8×
270193.5 → 45.218.52.4×

The text now reads "2.4× to 3.8×" instead of "5.1× to 7.9×". At μ = 3, FixMatch also remains slower in corrected time on every seed (93.6, 91.3 and 51.1 min, against 29.6, 20.3 and 17.9).

What is counted. FLOPs are measured with PyTorch's FlopCounterMode, separately for each method and each μ, on inputs with that configuration's actual batch shapes. Each measurement replays the exact sequence of model calls of the training script: the no-gradient forward pass used for pseudo-labeling, and every forward and backward pass of the loss. The count covers convolutions and matrix products. It excludes the optimizer step, the EMA update and data augmentation, and FlopCounterMode does not count elementwise operations. The submission did not describe any of this. The revision adds a paragraph and a table of per-iteration costs for every method and every μ (appendix "Implementation Details").

Q2. Is keeping the low-confidence anchor intentional, and can you compare filtering positions? Your reading of Eq. (3)–(5) is correct. When m_i = 0 and m_i' = 1, the mixed example stays active, and at least half of its target is the anchor's hard pseudo-label. The mechanism therefore does not exclude low-confidence pseudo-labels from mixed supervision, and the revision states this case explicitly in Section 3.

This case was not intended. Our design goal was that only labeled samples and admitted pseudo-labels supervise the mix. Gating on the partner only guarantees this for the minority component of each mix, not for the dominant anchor. Two variants of the ablation below exclude every unreliable pseudo-label, as we intended: both (anchor and partner must be reliable) and premix (only reliable samples are mixed).

We have now run the comparison you ask for, at fixed μ = 3. Only the mask weighting L_mix changes, with the same pool size, normalization and FLOPs per iteration: none (unfiltered Mixup), partner (submitted), anchor, both, and premix. Corrected minutes to 80% on CIFAR-10/250 (ratio to partner):

Mask of L_mix231203082701
none (unfiltered Mixup)24.4 (0.82×)14.3 (0.70×)12.2 (0.68×)
partner (submitted)29.620.317.9
anchor41.628.521.6
premix42.830.027.9
both45.933.423.7

On every seed, unfiltered Mixup is fastest, followed by the submitted mask, and the three stricter masks are slower. The two variants that implement our original intent are 1.3× to 1.7× slower than the submitted mask. Unfiltered Mixup is faster than the submitted mask on all four configurations and all seeds (1.03× to 1.52×, see the general comment). Filtering the mixed signal slows convergence: the speed-up comes from the Mixup channel, not from the confidence mask. The abstract, the second contribution and the conclusion now say so, and the ablation is a new appendix ("Mask Ablation"). The main tables now also report the unfiltered version, which we recommend; the other experiments use the submitted version.

Q3. Does Appendix A.4 (A.6 in the revision) implement the full Fast FixMatch? Can you report cost to the same targets? No. The submitted implementation covered only the Curriculum Batch Size, and the 3000-step diagnostic compared equal iteration counts, not equal accuracy. The revision says so and no longer presents it as an assessment of Fast FixMatch.

We have now implemented the full method of Chen et al.: the Curriculum Batch Size over the whole 2^20-iteration schedule, strong augmentation of labeled data (RandAugment, where the paper uses AutoAugment), FlexMatch's curriculum pseudo-labeling with the convex mapping, and the unlabeled loss weight scaled by the ratio of unlabeled to labeled batch size. Its batch size changes at every iteration, so it runs without compilation, and wall-clock time is not comparable with the other methods. We therefore compare along FLOPs, which the script measures from the actual batch size of each iteration.

On CIFAR-10/250 (seed 0308), EfficientMatch reaches 80% after 21.1 PFLOPs. With the same budget (196k iterations), full Fast FixMatch reaches 68.3%. It reaches 65% after 13.3 PFLOPs, against 5.2 for EfficientMatch and 8.5 for FixMatch. We stopped the run at that budget, so we do not report its cost to 80%. This is one seed and our own implementation. We do not reproduce the 3× saving over FixMatch reported by Chen et al., and we cannot exclude that the difference comes from our implementation. The revision reports this run in the appendix and keeps Fast FixMatch out of the main comparison.

Q4. "Original implementation" of MixMatch versus the shared optimizer of Table 14 (Table 23 in the revision). The phrase "using the original implementation" was wrong, and we removed it. All five methods are implemented in one training framework with a shared data pipeline, optimizer (SGD with Nesterov momentum, lr 0.03, cosine decay) and schedule. Each method keeps its own losses, μ and specific settings. For MixMatch, the hyperparameter table now lists:

  • kept from the original paper: sharpening temperature T = 0.5, two weak views for label guessing, α = 0.75, and λ_u = 100 ramped up linearly over 16,000 iterations. The ramp-up ends before any reported target crossing (between 18.5k and 213.5k iterations).
  • changed: SGD instead of Adam, and L_x computed against the labeled sample's own label instead of its mixed target.

To check the second change, we ran MixMatch with L_x against the mixed target, as published, on CIFAR-10/250 (seed 2312). It behaves like our version: neither reaches 80% within the 2-hour budget (best accuracy 78.1% against 78.4%). On CIFAR-10/4000, however, the published L_x is faster on every seed (1.13× to 1.50× in corrected time: 40.3, 52.6 and 31.0 minutes instead of 55.5, 59.6 and 46.3). EfficientMatch stays first on every seed (28.2, 29.3 and 29.8 minutes), but by only 1.04× on seed 2701, and MixMatch moves ahead of RegMixMatch on two seeds. The revision reports these runs in a new appendix. The other tables keep our version, so that all analyses use the same MixMatch.

Table 10 (Table 12 in the revision) adds the whole Mixup branch. We agree: compared with FixMatch at μ = 3, EfficientMatch adds the whole Mixup channel, so this comparison cannot show the effect of the partner mask, and the revision says so. The mask ablation of Q2 is the test that isolates the mask. The revision also adds the converse matched-μ comparison, using EfficientMatch runs at μ = 7 and μ = 1 that were already in the μ ablation (no new run). At μ = 7, EfficientMatch reaches the target faster and with fewer FLOPs than FixMatch, FlexMatch and RegMixMatch on every seed of CIFAR-10/250 and CIFAR-10/4000 (1.2× to 3.2× in time, 1.2× to 2.8× in FLOPs). At μ = 1 on CIFAR-10/4000, it does so against our version of MixMatch (1.2× to 1.4× in time, 1.4× to 1.8× in FLOPs). Against MixMatch with the published L_x, it is faster on two seeds (1.02× and 1.24×) but slower on the third (39.6 against 31.0 minutes).

Following your question on FLOP counting, we checked one more case and found that our RegMixMatch FLOPs were overestimated. We report the correction here.

The issue. Our count replays each training script on full batches. RegMixMatch's second forward pass, however, mixes only the samples of its confident pool, whose size changes at every iteration. All its runs use this default path, except the matched-μ comparison, which uses a fixed-shape variant. Its FLOPs in Tables 2 and 3 were therefore upper bounds. The other methods apply their masks in the loss, after a full forward pass, so their counts were exact.

The correction. We reran RegMixMatch on every configuration and seed, and logged the number of mixed samples at every iteration. On average, the second pass covers 18% of the full mixed batch on SVHN/250, 21 to 23% on CIFAR-10/250, 35 to 38% on CIFAR-10/4000 and 5% on CIFAR-100/2500. The FLOPs to reach the target are now the number of iterations of the main run multiplied by the mean cost per iteration of the rerun. In PFLOPs, seeds 2312 / 0308 / 2701:

ConfigurationRegMixMatch, submittedRegMixMatch, exactEfficientMatch
SVHN/25010.4 / 8.5 / 11.47.4 / 6.1 / 8.15.2 / 4.8 / 5.9
CIFAR-10/25066.2 / 48.2 / 39.748.4 / 35.1 / 28.831.1 / 21.1 / 18.5
CIFAR-10/400047.3 / 51.1 / 51.136.9 / 40.0 / 39.630.4 / 30.4 / 32.2
CIFAR-100/2500130.7 / 112.0 / 149.387.5 / 74.9 / 99.889.0 / 77.4 / 93.5

Consequence. The paper said that EfficientMatch has the lowest FLOP cost in all twelve cases. This is no longer true. On CIFAR-100/2500, seeds 2312 and 0308, RegMixMatch needs 1.7% and 3.2% fewer FLOPs. EfficientMatch remains the fastest in wall-clock time in all twelve cases, and EfficientMatch without the mask on L_mix has the lowest FLOP cost in all twelve cases.

Changes in the revision.

  • Tables 2 and 3, and the sentence in the discussion ("ten of the twelve cases").
  • The threshold-free scores along FLOPs. RegMixMatch now ranks first on CIFAR-100/2500 at 200 PFLOPs, 0.1 point ahead of EfficientMatch. The text no longer says that RegMixMatch never ranks first.
  • The FLOP-counting paragraph of the appendix, which describes this computation.
  • The accuracy-versus-FLOPs panel of Figure 2.

The full-horizon run (seed 42) did not log the mixed batch size, so the FLOP savings of its milestone table are now marked as upper bounds.

Thank you for the corrections and additional experiments. One inconsistency remains: for CIFAR-10/4000, seed 2312, Table 14 reports 47.3 PFLOPs for RegMixMatch, while Table 3 reports 36.9—below EfficientMatch’s 39.2 at μ = 7.

Comments on the paper

1 comment

We thank the reviewers for their careful reading. Their comments led us to re-check every number and claim in the paper against our result files. We found errors and fixed them in the revised PDF, where changes are shown in blue.

Errors we acknowledge and have corrected

  1. FLOPs of FixMatch at μ = 3 (4QzD). Our cost table reused the μ = 7 cost (850.10 GFLOPs per iteration). The measured cost at μ = 3 is 411.33. Steps and times were correct. EfficientMatch is 2.4× to 3.8× less FLOP-costly than FixMatch at matched μ, not 5.1× to 7.9×.
  2. A claimed ablation that was not in the paper (qois). Section 3 said that "variants of this design are evaluated in the ablation study". They were not. The sentence is removed, and we have now run this ablation (below).
  3. MixMatch "original implementation" (4QzD). All methods share one training framework, with SGD for MixMatch rather than Adam, and a supervised loss against the sample's own label. Both deviations are now listed in the hyperparameter table (Table 23 in the revision, Table 14 in the submission).
  4. MixMatch at 4000 labels. The introduction said MixMatch "becomes the fastest method" on CIFAR-10/4000; EfficientMatch is fastest on all seeds. An appendix sentence also called MixMatch "competitive or dominant" on CIFAR-100/2500, where it never reaches the target.
  5. Threshold sensitivity on CIFAR-10/4000. Section 6 said the ranking "shows little further change" when the target is lowered. In fact MixMatch becomes fastest on all three seeds.
  6. Evaluation period. It is 500 iterations, not 512; no number changes.

What we added

  • An ablation of the mixing mask (4QzD, qois, d4ZC). It contradicts a claim of the submission. Unfiltered Mixup reaches the target faster than the submitted mask in all 12 cases (4 configurations × 3 seeds), by 1.03× to 1.52× in corrected time, and stricter masks are slower. EfficientMatch's speed-up comes from its Mixup channel, not from its confidence filter. The abstract, contributions and conclusion now say so. The main tables now also report the unfiltered version, which we recommend; the other experiments use the submitted version.
  • A control for the Mixup channel. Replacing the mixed inputs by an extra strong view of the same samples, at the same cost, keeps part of the gain, and Mixup is faster still (1.6× to 2.1× fewer iterations on CIFAR-10/250). Part of the speed-up therefore comes from training on additional augmented views. Why mixing does better than a strong view is an open question, and we do not claim to know the full cause of the speed-up.
  • A threshold-free comparison (qois): mean accuracy over a time or FLOP budget. At fixed budgets, EfficientMatch ranks first or second in every configuration, the only method to do so. In the window that ends when EfficientMatch reaches its target, before any run is stopped, MixMatch ranks first in 11 of 12 cases and EfficientMatch second in 11 of 12. Our conclusions therefore depend on the targets.
  • Full Fast FixMatch (4QzD), compared at equal FLOPs: with the budget EfficientMatch needs to reach 80% on CIFAR-10/250, it reaches 68.3% (one seed).
  • Matched-μ comparisons in both directions (d4ZC, qois), from runs already in the μ ablation.
  • How targets were chosen, why training does not simply stop after convergence, an extended Limitations section, and a description of how FLOPs are counted.

What we do not address, and why

  • ImageNet and other architectures: every result uses a single GPU, and a three-seed study at that scale does not fit in the discussion period.
  • Asymptotic accuracy: no result is a claim about it. The single-seed long run stays in the appendix, labeled as such.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.