acceptodds
Under review as a conference paper at ICLR 2027

Why Weak-to-Strong Consistency Regularization Enables FixMatch to Outperform Pseudo-Labeling: A Feature-Learning Perspective

Abstract

FixMatch, a representative semi-supervised learning (SSL) method, has achieved strong performance and become a widely adopted baseline for subsequent SSL research. Its success is commonly attributed to the combination of pseudo-labeling and weak-to-strong consistency regularization. However, the interaction between these two components remains theoretically underexplored. To address this gap, we explain FixMatch’s advantage over Pseudo-Labeling in theory by connecting feature learning with pseudo-label selection during training. With limited labeled data, a model initially relies on features that fit the labeled samples but generalize poorly, producing conffdent yet incorrect pseudo-labels. Pseudo-Labeling trains on the same weakly augmented images used to select these targets, leaving small prediction errors that can limit further feature learning. Incorrect targets can also reinforce poorly generalizing features and delay the selection of additional correct samples. FixMatch instead trains on strongly augmented versions of the selected images. Strong augmentation can weaken the features supporting current predictions and help the model learn shared features that improve predictions across more samples. More samples can then enter training with correct pseudo-labels, providing supervision for further feature learning. Under the conditions of our theory, this feedback can give FixMatch better representations, higher pseudo-label precision, and lower classiffcation error than Pseudo-Labeling. However, weak-tostrong consistency regularization can also strengthen parameter updates that push the model toward predicting incorrect pseudo-labels. Experiments on CIFAR-10 and CIFAR-100 support key parts of this explanation: strong views increase learning signals toward both correct and incorrect pseudo-labels, FixMatch learns more discriminative features, and replacing the selected pseudo-labeled dataset changes subsequent feature learning and pseudo-label selection.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
semi-supervised learning, FixMatch, pseudo-labeling, consistency regularization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hz8GOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates why weak-to-strong consistency regularization improves FixMatch over same-view Pseudo-Labeling through the interaction between feature learning and pseudo-label selection. It shows that strong views can increase training signals toward both correct and incorrect pseudo-labels and derives a local sufficient condition for beneficial contributions to dominate. In a constructed trainable neural network, it analyzes how shared-feature learning enables new correct selections, while erroneous supervision reinforces spurious features and delays further correct selections. Under explicit conditions, FixMatch achieves higher pseudo-label precision, lower classification error, and better representations as measured by optimal linear-classifier risk over a common finite training interval. Experiments on CIFAR-10 and CIFAR-100 use paired residual measurements, controlled replacement of selected pseudo-labeled datasets, and linear probes to support components of this mechanism, while also documenting training deterioration and amplified signals toward incorrect targets.

Critical strengths
  1. A substantive dynamic account of feature learning and pseudo-label selection. The paper follows both learners’ evolving parameters and selections in a trainable nonlinear network. It proves that newly admitted erroneous supervision can strictly delay subsequent correct selections and establishes a common-time representation advantage that persists under optimal linear readouts. It connects error reinforcement, access to new supervision, and representation quality within an explicit training process.

  2. Controlled experiments that support distinct components of the explanation. Fixed-checkpoint comparisons isolate how strong views amplify residuals for both correct and incorrect targets. Paired continuations from identical starting states show that replacing the complete selected supervision changes subsequent probe performance and selection on a common candidate pool. These interventions provide meaningful empirical support for the proposed links. The explicit reporting of adverse outcomes also helps establish the explanation’s scope.

Critical weaknesses
  1. Substantial training deterioration limits the representativeness of the empirical evidence. One CIFAR-10 FixMatch run declines from 86.52% to 36.95% EMA accuracy, while two CIFAR-100 Pseudo-Labeling runs nearly collapse to single-class prediction. The paired interventions remain informative at these checkpoints, but it is unclear whether their conclusions hold under stable, effective training. Fixed-checkpoint BN diagnostics do not resolve this trajectory-level concern. This weakens confidence that the observed effects explain FixMatch’s practical success.

  2. The connection between the constructed mechanism and real-network behavior remains incomplete. The theory relies on structured feature pathways, specified deletion probabilities, and initialization conditions whose broader robustness is insufficiently characterized. Empirically, replacing complete selected datasets identifies their overall effect, but does not isolate how much subsequent selection changes through the feature extractor versus the classifier. Frozen-supervision continuations also leave the repeated feedback loop untested. Consequently, the experiments support components of the explanation without establishing the practical importance of the specific error-reinforcement and selection-delay mechanism proved theoretically.

Questions for authors
  1. Clarification—training deterioration. What evidence helps explain the severe late-stage deterioration and single-class collapse? In particular, do you have results from stable training runs or an alternative training configuration showing that the reported representation and selection effects persist? This would materially affect my confidence in the experiments’ representativeness.

  2. Clarification—theoretical applicability. Beyond the verified instance in Appendix B.10, what ranges of initial error mass and distance from the ReLU activation boundary can the sufficient inequalities certify? Quantifying these ranges would help assess how strongly the result depends on the illustrated near-boundary, high-precision regime.

  3. Clarification—mechanism identification. Given the limitations acknowledged in Appendix C.7, is there empirical evidence that specifically links subsequent selection improvements to feature-extractor changes, or links newly admitted errors to delayed correct selections? The dataset-replacement intervention establishes an overall supervision effect, but these additional links would substantially strengthen the connection to the theoretical mechanism.

3 comments
AuthorsOpenReview ↗

Response to Reviewer hz8G (1/3): Stable training and repeated experiments

We respond to your comments in three parts.

We thank the reviewer for recognizing the value of our analysis of how feature learning and pseudo-label selection interact during training. We also appreciate your questions about training reliability, the role of the feature extractor, and the scope of the theoretical result. To address the training concern, we repeated the baseline experiments using a fresh official USB FixMatch implementation and recovered stable performance. For controlled comparisons, we derive the same-view Pseudo-Labeling control from this implementation by changing only the training view, keeping the remaining protocol unchanged.

Using these checkpoints, we repeated the original residual and full selected-dataset replacement tests, then added fixed-classifier and fixed-image controls to examine the mechanism. The repeated tests recover the learning-pressure and supervision effects; the additional controls examine the roles of the feature extractor and assigned labels. We also verify the theoretical inequalities over continuous parameter ranges. Below, we distinguish these findings from the specific selection-delay prediction tested by the online intervention.

Q1: Do the reported effects persist under stable, effective training? (Weakness 1 and Question 1)

A1. We recovered stable native FixMatch performance and repeated the original core experiments from its checkpoints. We downloaded the official USB implementation again, retained its FixMatch training code and reference hyperparameters, and trained WRN-28-2 for 1,024 epochs using three seeds.

DatasetFinal EMA accuracy, mean ± sample SDLargest best-to-final decline
CIFAR-10, 40 labels93.38 ± 0.99%0.34 percentage points
CIFAR-100, 400 labels44.50 ± 3.03%1.98 percentage points

Repeating the original core experiments. At native FixMatch epoch-65 checkpoints, we repeated the paired residual test from Section 4.1 on both datasets and all three seeds, adding an independent weak view. Strong views increased mean residuals for both correct and incorrect pseudo-labels.

For same-weak, independent-weak, and strong views, respectively, mean residuals were 0.0045, 0.0234, and 0.1502 on CIFAR-10, and 0.0103, 0.0688, and 0.3275 on CIFAR-100. The raw, non-EMA model evaluates identical selected images and fixed targets. We average four strong augmentations per image, then average across images and equally across seeds.

We repeated the full selected-dataset replacement design from Section 4.2 on both datasets and all three seeds. Each native FixMatch epoch-65 checkpoint provides two branches with identical model and optimizer states. They continue for 50 epochs with the same labeled supervision and strong-augmentation procedures, receiving either FixMatch's complete selected set or the same-view control's. Selected identities, original pseudo-labels, and selection masks remain frozen; both branches retain the native unlabeled loader and loss normalization. Endpoint probe accuracies are reported as mean ± sample SD over three seeds:

DatasetFixMatch-source supervisionSame-view-source supervision
CIFAR-1083.61 ± 0.98%56.79 ± 1.61%
CIFAR-10034.63 ± 0.62%24.62 ± 4.49%

Probe accuracy favored FixMatch-source supervision in all six paired comparisons. Mean counts of correct high-confidence predictions on common pools initially rejected by both sources were 333.7 vs 31.0 on CIFAR-10 and 887.0 vs 110.0 on CIFAR-100, averaged equally over three seeds. These offline counts use confidence ≥0.95; the candidates receive no pseudo-label loss. Thus, the original finding that selected supervision affects later representations also holds when starting from native FixMatch checkpoints.

The probe fits a ridge linear classifier to frozen features using all 50,000 training labels and evaluates on 10,000 test images without updating the SSL model. These continuations use FixMatch as the recipient and a ridge probe, rather than repeating both original recipient types and the SGD probe.

The new full-training baseline curves measure EMA test accuracy; the original full-training probe and precision curves were not repeated. Q2 adds controls to examine how the supervision effect arises. The cause of the original deterioration has not been isolated.

Part 2 explains the protocol comparison and additional mechanism controls.

AuthorsOpenReview ↗

Response to Reviewer hz8G (2/3): Protocol and feature-extractor controls

Q1 continued: Protocol comparison

Why we retain a controlled Pseudo-Labeling comparison. We construct the same-view control by replacing FixMatch’s strong training image with the exact weak image used to generate its pseudo-label. All other settings remain unchanged. Its final EMA accuracies are 17.33 ± 2.72% on CIFAR-10 and 4.27 ± 0.41% on CIFAR-100.

Standalone USB Pseudo-Labeling follows a different protocol: it uses labeled/unlabeled batches of 64/64, increases the unlabeled-loss weight over the first 40% of training, and uses separate batch-normalization forward passes. Our control retains FixMatch’s 64/448 batches, full unlabeled-loss weight, and concatenated forward pass. This allows us to isolate the training-view change.

We also completed seed-0 runs of standalone USB Pseudo-Labeling. Its best-to-final EMA accuracies were 22.26%→20.87% on CIFAR-10 and 12.47%→11.30% on CIFAR-100, and both runs continued to predict all classes. We therefore treat the weak-view control’s low performance as a result under the shared FixMatch protocol, rather than a general property of Pseudo-Labeling.

Q2: Do the experiments identify feature-extractor effects and error-induced selection delay? (Weakness 2 and Question 3)

A2. The full-set replacement tests above establish the overall supervision effect. We added two controls to examine its source: fixing the classifier tests the feature-extractor route, while fixing image identities and counts isolates the effect of assigned labels. The online intervention separately tests the specific selection-delay prediction.

A. Supervision changes predictions with the classifier fixed

For each dataset, we clone the native FixMatch seed-0 model and optimizer at epoch 65 into two branches. Both continue strong-view training for 50 epochs with identical labeled supervision, unlabeled sampling, and loss normalization. Only the frozen pseudo-labels and selection masks differ, supplied by FixMatch or its same-view control.

We keep the raw, non-EMA classifier fixed. Saved checkpoints verify that its weights, biases, and optimizer states remain unchanged.

DatasetCommon candidate poolCorrect high-confidence predictions: FixMatch / same-view sourceProbe accuracy: FixMatch / same-view source
CIFAR-102,831583 / 3784.39% / 57.02%
CIFAR-10025,7861,014 / 16131.33% / 26.53%

The candidates are initially rejected by both supervision sources. Counts are offline correct predictions at confidence ≥0.95, evaluated with the same fixed classifier. These images receive no pseudo-label loss, although they remain in the loader and can affect batch normalization.

Because the classifier does not change, the prediction differences arise through the feature extractor, including its batch-normalization state. This directly addresses whether the supervision effect can operate through the representation instead of requiring classifier adaptation.

Using the same ridge probe defined above, starting accuracies are 84.34% and 32.70% for CIFAR-10 and CIFAR-100, respectively. The higher endpoint scores mainly show better preservation of representations during continuation.

B. Assigned labels affect representations when images and counts are fixed

We also restart paired copies of the same native FixMatch checkpoints for 50 epochs of strong-view training with identical labeled supervision, keeping image identities, selection masks, dataset size, and target-class counts identical. Ground-truth labels and cross-class swaps produce fixed targets with approximately 70% or 90% accuracy.

DatasetProbe accuracy with 70%-correct targetsProbe accuracy with 90%-correct targets
CIFAR-1085.99%88.64%
CIFAR-10047.81%49.95%

The higher-quality targets produce better endpoint representations on both datasets. Since the images, counts, and starting states are identical, these differences identify an effect of the supplied labels on subsequent feature learning.

Together, the two interventions support the links from supervision to representations and from representations to predictions beyond the selected set. They use seed 0 on each dataset and hold supervision fixed, so they test these links without reproducing repeated rounds of selection and learning.

Part 3 reports the selection-delay test and theoretical range verification.

AuthorsOpenReview ↗

Response to Reviewer hz8G (3/3): Selection delay and theoretical scope

Q2 continued: Selection delay

C. Directly testing the selection-delay prediction

We tested whether newly admitted erroneous supervision delays other correct admissions. From each same-view epoch-65 checkpoint, a two-epoch continuation identifies newly admitted wrong targets. Paired branches then restart from the same checkpoint and optimizer state for 20 epochs: one trains normally, while the other masks an identified image’s loss when it is selected with its recorded wrong target. Inputs and loss denominators remain unchanged, and pseudo-labels are recomputed online.

Across three seeds per dataset, we measure first correct admission among initially correct but low-confidence candidates, excluding directly masked images. Candidates never correctly admitted receive a waiting time of 20 epochs.

On CIFAR-100, masking erroneous supervision increases the equal-seed mean waiting time from 2.723 to 2.974 epochs, with longer waits in all three seeds. CIFAR-10 shows mixed results. This intervention therefore does not consistently confirm the specific selection-delay prediction. We distinguish this finding from the separate evidence above that supplied supervision affects representations and subsequent predictions.

Q3: How far does the theorem extend beyond the original parameter instance? (Weakness 2 and Question 2)

A3. We verified the sufficient inequalities over continuous parameter ranges within a specified family. All 1,590 parameter blocks pass the original certificate’s 36 numerical checks and the common-time condition using 60-digit outward-rounded interval arithmetic. These checks cover every parameter value within the stated blocks.

Let ν\nu denote the initially selected erroneous mass relative to the full unlabeled population. Let Δh=h−ε\Delta_h=h-\varepsilon denote the initial semantic-strength gap, where h=log⁡(19)/2h=\log(19)/2 and ε\varepsilon is the initial nonroot semantic strength.

VariationCertified range
Initial erroneous mass, with Δh=0.001\Delta_h=0.001ν∈[0.0004,0.0065]\nu\in[0.0004,0.0065]
Initial strength gap, with ν=0.001\nu=0.001Δh∈[0.000995,0.001075]\Delta_h\in[0.000995,0.001075]
Joint variationν∈[0.0005,0.005]\nu\in[0.0005,0.005], Δh∈[0.001,0.00102]\Delta_h\in[0.001,0.00102]

The parameters vary together through ε=h−Δh\varepsilon=h-\Delta_h, root strength δ=h−Δh/2\delta=h-\Delta_h/2, gate-crossing strength b=h+Δhb=h+\Delta_h, evaluation-group mass 0.249−ν0.249-\nu, and spurious-input scale ρ=(log⁡(19)+δ−0.0001)/5\rho=(\log(19)+\delta-0.0001)/5. Other independent parameters retain their submitted Appendix B.10 values; the network structure and augmentation probabilities remain unchanged.

All settings share the gradient-flow interval [0.4796,0.4799][0.4796,0.4799]. In the joint range, the nonroot filter’s Euclidean distance to the ReLU boundary on the unit evaluation patch lies in [2.450,2.500]×10−4[2.450,2.500]\times10^{-4}. This distance differs from the feature-strength gap Δh\Delta_h.

The verification shows that the result holds over a continuous family rather than only the illustrated instance. It also quantifies the scope you asked about: the family retains high initial pseudo-label precision, initialization close to the activation boundary, and a short common comparison interval.

Within this scope, the dynamic result explains how learned shared features enable new selections and how those selections supply further learning signals. Freezing the initial selected identities and targets blocks learning along subsequent semantic feature pathways in the constructed model. The representation advantage also persists under optimal affine linear readouts. These results identify the role of evolving supervision beyond a comparison of the final trained classifiers.

Changes to the final paper

If our paper is accepted, we will incorporate the results reported in this response as follows:

  • Section 4.3 and Appendix C.1: add the three-seed native FixMatch accuracy and stability results, together with a protocol comparison distinguishing the same-view control from standalone USB Pseudo-Labeling.

  • Section 4.1: add the three-seed residual comparison on native FixMatch checkpoints, including independent weak views and results for correct and incorrect targets.

  • Section 4.2 and Appendices C.4–C.6: add the three-seed full selected-dataset replacement results and the additional seed-0 fixed-classifier and fixed-image controls, specifying recipient models, frozen targets and masks, probe methods, classifier-state checks, batch normalization, and offline candidate counts.

  • Section 3.4 and Appendix B.10: add the certified parameter ranges, linked parameter settings, common-time interval, and distances to the ReLU boundary.

  • Appendix C.7: report the online-masking results by seed, define the candidate pool and waiting-time calculation, and distinguish the conditional selection-delay result from its current empirical support.

Reviewer_87PJOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper tried to present a feature-learning explanation for FixMatch’s advantage over same-view pseudo-labeling. Strong augmentation can increase the training signal from selected pseudo-labels and, under the paper’s theoretical conditions, help learn shared features that lead to more correct selections. Experiments on CIFAR-10 and CIFAR-100 examine prediction residuals, learned representations, and the effect of replacing selected pseudo-labeled datasets.

Critical strengths

The paper is clearly written and easy to follow.

Critical weaknesses
  1. My main concern is the motivation. Prior jobs like UDA, FixMatch, and subsequent feature-learning analyses, have already established important benefits of strong augmentation. This paper adds a conditional account of evolving pseudo-label selection, but its experiments do not isolate that mechanism or show what it explains beyond prior accounts.

  2. Experiements need more verficiation on large-scale datasets, not toy cifar. Besides, CIFAR-10 FixMatch EMA accuracy is 67.91 ± 28.02% across three seeds, while CIFAR-100 Pseudo-Labeling nearly collapses; training-mode and offline selections can also differ sharply.

  3. The theorem is kind of conditional. Its structured binary model assumes strong views can delete a spurious feature while retaining semantic information, and the verified advantage holds over a short chosen interval.

Questions for authors

many paper highlight the significances of strong augs, which yields large loss, thus improving the performance.

3 comments
AuthorsOpenReview ↗

Part 1/3: Training reliability and dataset scale

We respond to your comments in three parts.

We thank the reviewer for the thoughtful feedback on the contribution, experimental reliability, and theoretical scope. We established stable native FixMatch baselines, repeated the core residual and full-dataset replacement tests using their checkpoints, and added controls to examine how supervision affects representations and later selections. These experiments support key links in our dynamic explanation beyond an increase in the immediate learning signal.

Q1: Training reliability and dataset scale (Weakness 2)

Stable results from native FixMatch. We kept the official USB FixMatch training code unchanged and trained WRN-28-2 for 1,024 epochs, using 40 labels on CIFAR-10 and 400 labels on CIFAR-100.

DatasetFinal EMA accuracy, mean ± sample SD over three seeds
CIFAR-1093.38 ± 0.99%
CIFAR-10044.50 ± 3.03%

The largest best-to-final declines were 0.34 and 1.98 percentage points, respectively.

Rechecking the original core experiments. We repeated the paired residual comparison from Section 4.1 at epoch 65 on both datasets and all three seeds, adding an independent weak view. Strong views increased mean residuals for both correct and incorrect pseudo-labels. Part 2 reports the three-view results.

We also repeated the full selected-dataset replacement design from Section 4.2 on both datasets and all three seeds. From each native FixMatch epoch-65 checkpoint, two branches share the model and optimizer states and continue for 50 epochs with identical labeled supervision and strong-augmentation procedures. One receives FixMatch's complete selected set and the other the same-view control's; all selected identities and original pseudo-labels are retained and frozen. Both branches retain the native unlabeled loader and loss normalization. Endpoint probe accuracies (mean ± sample SD over three seeds) are:

DatasetFixMatch-source supervisionSame-view-source supervision
CIFAR-1083.61 ± 0.98%56.79 ± 1.61%
CIFAR-10034.63 ± 0.62%24.62 ± 4.49%

Averaged across three seeds, correct high-confidence predictions on the common pools initially rejected by both sources numbered 333.7 vs 31.0 on CIFAR-10 and 887.0 vs 110.0 on CIFAR-100. Probe accuracy favored FixMatch-source supervision in all three seeds on both datasets. Thus, the original finding that selected supervision affects later representations also appears when starting from native FixMatch checkpoints.

These continuations use FixMatch as the recipient and a ridge probe, rather than repeating both original recipient types and the SGD probe. The ridge probe fits a linear classifier to frozen features using all 50,000 training labels and evaluates on 10,000 test images without updating the SSL model. The new full-training baseline curves measure EMA test accuracy; we did not repeat the original full-training probe and precision curves. Part 2 adds controls that separate label quality from image composition and test the feature-extractor route.

Fair comparisons built on the native implementation. The same-view Pseudo-Labeling control replaces FixMatch's strong training image with the exact weak image used to generate its pseudo-label. The independent-weak control uses a separately sampled weak image. Both retain FixMatch's 64/448 labeled/unlabeled batch sizes, full unlabeled-loss weight, and concatenated batch-normalization forward pass. Standalone USB Pseudo-Labeling uses 64/64 batches, increases the unlabeled-loss weight over the first 40% of training, and handles batch normalization separately. Our controls isolate the training-view choice within the shared recipe. Their lower accuracy should not be generalized to all Pseudo-Labeling configurations.

Offline selections use fixed evaluation views and saved BN statistics; actual online admissions are recorded during training. We distinguish these measurements throughout.

Dataset choice. We appreciate the suggestion to include larger datasets. Our contribution is primarily theoretical. With the same architecture and four labeled samples per class, CIFAR-10 provides a relatively simpler classification task and CIFAR-100 a more demanding one. They allow controlled tests of the proposed links under different classification demands. Larger-scale evaluation would help assess how broadly the findings extend.

Part 2 explains the dynamic contribution and the additional mechanism controls.

AuthorsOpenReview ↗

Part 2/3: Contribution and mechanism experiments

Continued from Part 1/3.

Q2: Contribution beyond existing strong-augmentation accounts (Weakness 1 and Question)

Our contribution concerns how learning from selected samples changes future selections and subsequent learning. We address the theoretical contribution, mechanism identification, and the larger-loss explanation below.

A. Contribution beyond prior explanations (Weakness 1)

UDA (Xie et al., 2020) links label-preserving augmentation to sample connectivity and label propagation. FixMatch (Sohn et al., 2020) empirically demonstrates the benefit of weak-to-strong training. Li et al. (2025), whose analysis motivates SA-FixMatch, explain how augmentation helps recover missed semantic features, primarily through comparisons between FixMatch and supervised learning.

In contrast, our work studies how feature learning and pseudo-label selection influence one another over time. We compare FixMatch and Pseudo-Labeling from the same initial model and selected samples. Our local analysis identifies conditions under which useful feature updates outweigh incorrect-target updates. Section 3.4 then analyzes how learned features can admit further correct samples that support further learning. Within this framework, freezing the initial selected sample identities and targets blocks learning through later feature pathways, while newly admitted incorrect supervision can delay correct selections. Under the conditions discussed in Part 3, these dynamics yield both representation and classification advantages over a common finite interval.

B. Experimental evidence for the proposed mechanism (Weakness 1)

The full-dataset replacements in Part 1 recheck the original supervision effect. To examine what drives it, we added two controls at native FixMatch epoch 65, seed 0 on each dataset. Paired branches share model and optimizer states and continue for 50 epochs with identical labeled supervision and strong-augmentation procedures.

Fixed images with different label quality. We keep image identities, dataset size, selection masks, and target-class counts fixed. Ground-truth labels and cross-class label swaps produce fixed targets with approximately 70% or 90% accuracy. At the endpoint, the 90% group has higher probe accuracy on CIFAR-10 (88.64% vs 85.99%) and CIFAR-100 (49.95% vs 47.81%). Both controls use the ridge probe defined in Part 1. This isolates the labels’ effect on later representations from differences in images, dataset size, and starting state.

Different supervision with a fixed classifier. We fix the raw classification head and train the feature extractor using frozen supervision from FixMatch or the same-view control. On common candidates initially rejected by both sources, FixMatch-sourced supervision yields 583 vs 37 correct high-confidence predictions on CIFAR-10 and 1,014 vs 161 on CIFAR-100. These are offline predictions at confidence at least 0.95. The candidates receive no pseudo-label loss, although their images remain in the loader. With the classifier fixed, these differences arise through the feature extractor, including its batch-normalization state.

These interventions link supplied supervision to later representations and predictions, without reproducing the full repeated selection-and-learning process. Additional tests that mask erroneous supervision did not consistently confirm the predicted selection delay.

C. Why a larger learning signal is insufficient (Question)

We test whether a different training view and a larger immediate signal suffice. At native FixMatch epoch-65 checkpoints, we compare three views on identical selected images and fixed pseudo-labels. The target residual is r=1−p(y^∣view)r = 1 - p(\hat{y} \mid \mathrm{view}), where y^\hat{y} is the weak-view pseudo-label and pp is the predicted class probability. Raw-model means across three seeds are:

DatasetSame weak viewIndependent weak viewStrong view
CIFAR-100.00450.02340.1502
CIFAR-1000.01030.06880.3275

Strong-view measurements average four augmentations per image. Independent weak views increase the residual, yet in separate 1,024-epoch runs under the Part 1 protocol, they reduce mean final EMA accuracy from 17.33% to 10.01% on CIFAR-10 and from 4.27% to 1.56% on CIFAR-100. A larger immediate residual therefore does not ensure better learning. Our theory examines how perturbing currently relied-on features while retaining class-relevant information changes feature updates and later supervision.

AuthorsOpenReview ↗

Part 3/3: Theoretical conditions and scope

Continued from Part 2/3.

Q3: Theoretical conditions and time interval (Weakness 3)

We thank the reviewer for asking us to clarify the assumptions and scope of the theoretical result. The distinction between what we assume and what we derive is important here.

What we assume. We specify a binary data distribution containing shared semantic features and a spurious feature, together with the network initialization and augmentation probabilities. In the verified setting, augmentation removes the spurious input with probability 0.6 and retains each semantic input with probability 0.8. Removal and preservation are therefore probabilistic, although their effects on the different features remain explicitly structured.

What we derive. Starting from the same model and initial selected samples, each learner's changing predictions determine which additional samples enter training. The analysis establishes the feedback described in Part 2 and shows that the representation advantage persists when each representation is evaluated using its optimal affine linear classifier. These results follow from the parameter dynamics and confidence-based selection rules, connecting the assumed augmentation effects to later changes in supervision and representation.

What the additional range verification establishes. We checked the original sufficient inequalities using outward-rounded interval arithmetic over 1,590 parameter blocks, covering both individual-parameter and joint variations. This verifies continuous ranges, rather than only isolated numerical settings. One jointly certified range is:

ParameterCertified range
Initial erroneous population mass ν\nu[0.0005, 0.005]
Initial semantic-strength gap Δh\Delta_h[0.001, 0.00102]

Here, ν\nu is the mass of initially selected erroneous samples relative to the whole population. The gap Δh=h−ε\Delta_h = h - \varepsilon measures how far the initial non-root semantic strength ε\varepsilon lies below the reference strength hh. Related initial strengths, population masses, and the spurious-input scale vary together; the network structure and augmentation probabilities remain fixed.

Across this joint range, the result holds on the common gradient-flow interval [0.4796, 0.4799]. This interval follows from bounds on the learners' feature-learning and selection times. The verification extends the original example to a continuous family of settings, while retaining high initial pseudo-label precision, initialization near the ReLU activation boundary, and a short comparison interval.

If our paper is accepted, we will make the following changes to the final version:

  • Sections 4.1–4.3 and Appendix C.1: add the three-seed native FixMatch and weak-view results, the repeated residual and full-dataset replacement tests, the additional mechanism controls, and a protocol table specifying training views, batch sizes, loss weights, BN settings, and probe methods.

  • Section 3.4: distinguish the assumptions from the derived conclusions and clarify how changing selections and incorrect supervision affect the feature-learning dynamics.

  • Appendix B.10: add the certified parameter ranges, linked parameter settings, and common-time bounds.

Reviewer_t2zqOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
1
Read review
Paper summary

The paper analyzes why FixMatch outperforms traditional pseudo-labeling by looking into how feature learning and pseudo-label selection interact over training. The main intuition is that during pseudo-labeling the model is trained on almost an identical image it was already confident about, leaving little additional to learn, whereas in FixMatch the strong augmentations can hide some image features that were used by the model to yield confident predictions. Hence, the model has to rely on different features to learn that are shared across images from that same class. These shared features make more unlabeled images confidently correct, so they enter training and provide further supervision. However, when the pseudo-label is wrong, this larger error signal also pushes the model harder toward the wrong class. The paper formalizes this trade-off and derives the conditions under which useful signal outweighs harmful one. The paper investigates this interaction on CIFAR-10 with 40 labels and CIFAR-100 with 400 labels through feature probes, dataset-swapping experiments, and controlled changes to pseudo-label quality.

Critical strengths
  • The topic is interesting and very relevant. Most SSL approaches are built on FixMatch, so understanding why weak-to-strong training helps has a lot of value.
  • The experiments are thorough. The dataset-swapping experiments are particularly insightful: continuing from identical model and optimizer states separates the effect of the supplied supervision from that of the starting model.
  • The paper does loss analysis and examines both correct and incorrect pseudo-labels and reports individual runs and failure cases.
  • Overall, the paper is very well written and to-the-point.
Critical weaknesses
  • Main concern is the low FixMatch results compared to prior work. Original FixMatch reports 13.81 ± 3.37% error on CIFAR-10 with 40 labels using RandAugment and WRN-28-2, whereas this paper reports 67.91 ± 28.02% accuracy. Other implementations yield similar results [1].
  • While the dataset-swapping design controls for the starting model, the swapped datasets differ in training samples, dataset size, and label quality. Al.

[1] Wang et al., 2022, USB: A Unified Semi-supervised Learning Benchmark for Classification

Overall I found this paper very interesting. I would be willing to raise my score if the authors resolve the discrepancies with prior work.

Questions for authors

What would happen if pseudo-labeling would use a different weak augmentation for training instead of the same augmentation used to generate the pseudo-label?

3 comments
AuthorsOpenReview ↗

Response to Reviewer t2zq (1/3): Training reliability

We respond to your comments in three parts.

We thank the reviewer for highlighting these weaknesses in our experiments. We focused on matching training conditions but did not adequately address the discrepancy between our FixMatch results and established baselines. We have therefore established stable baselines with a fresh official USB FixMatch implementation, repeated the core residual and full selected-dataset replacement experiments, and added a fixed-image label-quality control. For a controlled comparison, we construct Pseudo-Labeling by replacing FixMatch's strong training view with the weak view used to generate the pseudo-label, keeping the remaining settings unchanged. This control differs from standalone USB Pseudo-Labeling in training details explained below. The new experiments recover stable FixMatch performance and support key predictions about learning pressure and how selected supervision affects subsequent feature learning.

Your question about independent weak views also helps sharpen our explanation: increasing the loss alone is insufficient to explain FixMatch's advantage. Our theory emphasizes how strong augmentation can disrupt the features supporting current confident predictions, encouraging the model to learn other class-relevant features shared across the expanded training set. These features can improve predictions beyond the initially selected samples and provide better supervision for further learning.

To address your concern that dataset swapping changes several factors at once, we also compare different pseudo-label qualities while keeping image identities, dataset size, selection masks, and target-class counts fixed. This comparison tests whether the assigned labels affect subsequent feature learning without changing which images enter training or how many are selected.

Q1: Do new runs recover representative FixMatch performance? (Weakness 1)

A1. Yes. Final EMA test accuracy is 93.38 ± 0.99% on CIFAR-10 and 44.50 ± 3.03% on CIFAR-100 (three seeds; mean ± sample SD). Best-to-final declines are at most 0.34 and 1.98 percentage points, respectively.

We downloaded official USB again, kept its FixMatch training code unchanged, and checked the source files and labeled splits. Runs use WRN-28-2 with 40 labels on CIFAR-10 and 400 labels on CIFAR-100 for 1,024 epochs. These results address the concern that the original CIFAR-10 result of 67.91 ± 28.02% was not representative. From these new native FixMatch checkpoints, we repeated the original residual comparison (Q2) and full selected-dataset replacement (Q3), then added the label-quality control in Q3.

The new full-training baseline curves measure EMA test accuracy; the original full-training probe and precision curves were not repeated.

Part 2 addresses your question about independent weak views.

AuthorsOpenReview ↗

Response to Reviewer t2zq (2/3): Independent weak views

Q2: What happens when Pseudo-Labeling trains on an independent weak view? (Question)

A2. It does not recover FixMatch's performance under the shared protocol. We separately train three variants from initialization using the fresh USB FixMatch code. The same-weak control replaces only the strong training image with the exact weak selection image; independent weak uses a separately sampled weak image; FixMatch uses a strong image. All select pseudo-labels from weak-view predictions at confidence ≥0.95. Architecture, labeled splits, optimizer, loss weights, batch sizes, BN policy, and training budget are identical. Final EMA test accuracies across three seeds are:

DatasetSame weakIndependent weakStrong (FixMatch)
CIFAR-1017.33 ± 2.72%10.01 ± 0.03%93.38 ± 0.99%
CIFAR-1004.27 ± 0.41%1.56 ± 0.79%44.50 ± 3.03%

The residual comparison sharpens our feature-learning explanation. We repeated the paired residual test from Section 4.1 at native FixMatch epoch-65 checkpoints on both datasets and all three seeds, adding an independent weak view. We use identical selected images and fixed pseudo-labels with the raw, non-EMA models. Mean target residuals, averaged equally over three seeds, are:

DatasetSame weakIndependent weakStrong
CIFAR-100.00450.02340.1502
CIFAR-1000.01030.06880.3275

For strong views, we average four augmentations per image before averaging across images and seeds. Strong views increase mean residuals for both correct and incorrect pseudo-labels. Independent weak views also increase residuals, but by less. In the separate full training runs, independent weak views reduce final EMA accuracy from 17.33% to 10.01% on CIFAR-10 and from 4.27% to 1.56% on CIFAR-100 relative to the same-weak control.

These results sharpen our explanation: increasing the residual by changing the weak view is insufficient to explain FixMatch's benefit. Our theory identifies a more specific mechanism. Strong augmentation can disrupt the learned features supporting confident predictions while retaining other class-relevant information, encouraging the model to learn features shared across the expanded training set and useful on a broader range of samples. The substantially larger strong-view residual is consistent with this disruption and quantifies the restored learning pressure. The theory explains how that pressure can support more generalizable feature learning.

Why we retain FixMatch's protocol. The control holds the remaining training settings fixed to isolate the training-view change. Standalone USB Pseudo-Labeling uses 64/64 labeled/unlabeled batches, gradually increases the unlabeled-loss weight over the first 40% of training, and uses separate BN forward passes that retain only labeled-batch updates to running statistics. Our controls retain FixMatch's 64/448 batches, full unlabeled-loss weight, and concatenated BN. Comparing the standalone implementations would change these settings too.

As a separate protocol reference, native USB Pseudo-Labeling seed-0 best-to-final EMA accuracies are 22.26% to 20.87% on CIFAR-10 and 12.47% to 11.30% on CIFAR-100, versus 24.31% to 18.00% and 11.11% to 4.74% for the same-weak control. The view-control results therefore concern Pseudo-Labeling under FixMatch's protocol.

Part 3 addresses the dataset-swapping concern and reports the additional label-quality control.

AuthorsOpenReview ↗

Response to Reviewer t2zq (3/3): Supervision controls

Q3: The swapped datasets differ in samples, size, and label quality. (Weakness 2)

A3. The full-set replacement tests the overall effect of the supervision selected by each learner. We repeated this design on the new native checkpoints, then added a fixed-image comparison to separate label quality from sample composition and quantity.

Repeating the original full selected-dataset replacement

We repeated the full selected-dataset replacement design from Section 4.2 on both datasets and all three seeds. Each native FixMatch epoch-65 checkpoint provides two branches with identical model and optimizer states. They continue for 50 epochs with the same labeled supervision and strong-augmentation procedures, receiving either FixMatch's complete selected set or the same-view control's. Selected identities, original pseudo-labels, and selection masks remain frozen; both branches retain the native unlabeled loader and loss normalization. Endpoint probe accuracies are reported as mean ± sample SD over three seeds:

DatasetFixMatch-source supervisionSame-view-source supervision
CIFAR-1083.61 ± 0.98%56.79 ± 1.61%
CIFAR-10034.63 ± 0.62%24.62 ± 4.49%

Probe accuracy favored FixMatch-source supervision in all six paired comparisons. Mean counts of correct high-confidence predictions on common pools initially rejected by both sources were 333.7 vs 31.0 on CIFAR-10 and 887.0 vs 110.0 on CIFAR-100, averaged equally over three seeds. These offline counts use confidence ≥0.95; the candidates receive no pseudo-label loss. Thus, the original finding that selected supervision affects later representations also holds when starting from native FixMatch checkpoints.

The probe fits a ridge linear classifier to frozen features using all 50,000 training labels and evaluates on 10,000 test images without updating the SSL model. These continuations use FixMatch as the recipient and a ridge probe, rather than repeating both original recipient types and the SGD probe.

Additional control with fixed images and label counts

For each dataset, we clone the native FixMatch seed-0 epoch-65 model and optimizer into two branches. Image identities, selected-set size, selection masks, and target-class counts remain identical. Using ground-truth labels and cross-class label swaps that preserve target-class counts, we construct pseudo-label sets with approximately 70% and 90% accuracy. Both branches continue for 50 epochs with the same labeled supervision and strong-view training, keeping targets and masks fixed. Each arrow below compares the endpoints of the 70%- and 90%-correct branches.

DatasetTest accuracyProbe accuracyCorrect high-confidence candidates
CIFAR-1074.60% → 86.66%85.99% → 88.64%2,272 → 3,804
CIFAR-10035.92% → 45.37%47.81% → 49.95%835 → 2,358

Test accuracy uses the non-EMA model, and probe accuracy uses the same ridge procedure defined above. Candidate counts are offline correct predictions at confidence ≥0.95 on common initially unselected images that receive no pseudo-label loss.

Higher target quality improves all three endpoint measures on both datasets despite identical images, counts, and starting states. This directly supports our view that the labels supplied by selected supervision affect subsequent feature learning and predictions beyond the selected set.

If our paper is accepted, we will make the following changes to the final version:

  • Section 4.3: add a table of three-seed final EMA accuracies for native FixMatch and both weak-view controls, together with FixMatch's best-to-final declines.

  • Appendix C.1: add a protocol table specifying training views, batch sizes, loss ramp-up, and BN for the original experiments, new view controls, and standalone USB Pseudo-Labeling.

  • Section 4.1: add the three-seed paired residual comparisons on native FixMatch checkpoints, including same-weak, independent-weak, and strong views on correct and incorrect targets.

  • Section 4.2: add the three-seed full selected-dataset replacement results and the additional seed-0 fixed-image label-quality comparison, specifying recipient models, frozen supervision, probe methods, controlled variables, and endpoint measurements.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.