- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates why weak-to-strong consistency regularization improves FixMatch over same-view Pseudo-Labeling through the interaction between feature learning and pseudo-label selection. It shows that strong views can increase training signals toward both correct and incorrect pseudo-labels and derives a local sufficient condition for beneficial contributions to dominate. In a constructed trainable neural network, it analyzes how shared-feature learning enables new correct selections, while erroneous supervision reinforces spurious features and delays further correct selections. Under explicit conditions, FixMatch achieves higher pseudo-label precision, lower classification error, and better representations as measured by optimal linear-classifier risk over a common finite training interval. Experiments on CIFAR-10 and CIFAR-100 use paired residual measurements, controlled replacement of selected pseudo-labeled datasets, and linear probes to support components of this mechanism, while also documenting training deterioration and amplified signals toward incorrect targets.
Critical strengths
-
A substantive dynamic account of feature learning and pseudo-label selection. The paper follows both learners’ evolving parameters and selections in a trainable nonlinear network. It proves that newly admitted erroneous supervision can strictly delay subsequent correct selections and establishes a common-time representation advantage that persists under optimal linear readouts. It connects error reinforcement, access to new supervision, and representation quality within an explicit training process.
-
Controlled experiments that support distinct components of the explanation. Fixed-checkpoint comparisons isolate how strong views amplify residuals for both correct and incorrect targets. Paired continuations from identical starting states show that replacing the complete selected supervision changes subsequent probe performance and selection on a common candidate pool. These interventions provide meaningful empirical support for the proposed links. The explicit reporting of adverse outcomes also helps establish the explanation’s scope.
Critical weaknesses
-
Substantial training deterioration limits the representativeness of the empirical evidence. One CIFAR-10 FixMatch run declines from 86.52% to 36.95% EMA accuracy, while two CIFAR-100 Pseudo-Labeling runs nearly collapse to single-class prediction. The paired interventions remain informative at these checkpoints, but it is unclear whether their conclusions hold under stable, effective training. Fixed-checkpoint BN diagnostics do not resolve this trajectory-level concern. This weakens confidence that the observed effects explain FixMatch’s practical success.
-
The connection between the constructed mechanism and real-network behavior remains incomplete. The theory relies on structured feature pathways, specified deletion probabilities, and initialization conditions whose broader robustness is insufficiently characterized. Empirically, replacing complete selected datasets identifies their overall effect, but does not isolate how much subsequent selection changes through the feature extractor versus the classifier. Frozen-supervision continuations also leave the repeated feedback loop untested. Consequently, the experiments support components of the explanation without establishing the practical importance of the specific error-reinforcement and selection-delay mechanism proved theoretically.
Questions for authors
-
Clarification—training deterioration. What evidence helps explain the severe late-stage deterioration and single-class collapse? In particular, do you have results from stable training runs or an alternative training configuration showing that the reported representation and selection effects persist? This would materially affect my confidence in the experiments’ representativeness.
-
Clarification—theoretical applicability. Beyond the verified instance in Appendix B.10, what ranges of initial error mass and distance from the ReLU activation boundary can the sufficient inequalities certify? Quantifying these ranges would help assess how strongly the result depends on the illustrated near-boundary, high-precision regime.
-
Clarification—mechanism identification. Given the limitations acknowledged in Appendix C.7, is there empirical evidence that specifically links subsequent selection improvements to feature-extractor changes, or links newly admitted errors to delayed correct selections? The dataset-replacement intervention establishes an overall supervision effect, but these additional links would substantially strengthen the connection to the theoretical mechanism.
3 comments
Response to Reviewer hz8G (2/3): Protocol and feature-extractor controls
Q1 continued: Protocol comparison
Why we retain a controlled Pseudo-Labeling comparison. We construct the same-view control by replacing FixMatch’s strong training image with the exact weak image used to generate its pseudo-label. All other settings remain unchanged. Its final EMA accuracies are 17.33 ± 2.72% on CIFAR-10 and 4.27 ± 0.41% on CIFAR-100.
Standalone USB Pseudo-Labeling follows a different protocol: it uses labeled/unlabeled batches of 64/64, increases the unlabeled-loss weight over the first 40% of training, and uses separate batch-normalization forward passes. Our control retains FixMatch’s 64/448 batches, full unlabeled-loss weight, and concatenated forward pass. This allows us to isolate the training-view change.
We also completed seed-0 runs of standalone USB Pseudo-Labeling. Its best-to-final EMA accuracies were 22.26%→20.87% on CIFAR-10 and 12.47%→11.30% on CIFAR-100, and both runs continued to predict all classes. We therefore treat the weak-view control’s low performance as a result under the shared FixMatch protocol, rather than a general property of Pseudo-Labeling.
Q2: Do the experiments identify feature-extractor effects and error-induced selection delay? (Weakness 2 and Question 3)
A2. The full-set replacement tests above establish the overall supervision effect. We added two controls to examine its source: fixing the classifier tests the feature-extractor route, while fixing image identities and counts isolates the effect of assigned labels. The online intervention separately tests the specific selection-delay prediction.
A. Supervision changes predictions with the classifier fixed
For each dataset, we clone the native FixMatch seed-0 model and optimizer at epoch 65 into two branches. Both continue strong-view training for 50 epochs with identical labeled supervision, unlabeled sampling, and loss normalization. Only the frozen pseudo-labels and selection masks differ, supplied by FixMatch or its same-view control.
We keep the raw, non-EMA classifier fixed. Saved checkpoints verify that its weights, biases, and optimizer states remain unchanged.
| Dataset | Common candidate pool | Correct high-confidence predictions: FixMatch / same-view source | Probe accuracy: FixMatch / same-view source |
|---|---|---|---|
| CIFAR-10 | 2,831 | 583 / 37 | 84.39% / 57.02% |
| CIFAR-100 | 25,786 | 1,014 / 161 | 31.33% / 26.53% |
The candidates are initially rejected by both supervision sources. Counts are offline correct predictions at confidence ≥0.95, evaluated with the same fixed classifier. These images receive no pseudo-label loss, although they remain in the loader and can affect batch normalization.
Because the classifier does not change, the prediction differences arise through the feature extractor, including its batch-normalization state. This directly addresses whether the supervision effect can operate through the representation instead of requiring classifier adaptation.
Using the same ridge probe defined above, starting accuracies are 84.34% and 32.70% for CIFAR-10 and CIFAR-100, respectively. The higher endpoint scores mainly show better preservation of representations during continuation.
B. Assigned labels affect representations when images and counts are fixed
We also restart paired copies of the same native FixMatch checkpoints for 50 epochs of strong-view training with identical labeled supervision, keeping image identities, selection masks, dataset size, and target-class counts identical. Ground-truth labels and cross-class swaps produce fixed targets with approximately 70% or 90% accuracy.
| Dataset | Probe accuracy with 70%-correct targets | Probe accuracy with 90%-correct targets |
|---|---|---|
| CIFAR-10 | 85.99% | 88.64% |
| CIFAR-100 | 47.81% | 49.95% |
The higher-quality targets produce better endpoint representations on both datasets. Since the images, counts, and starting states are identical, these differences identify an effect of the supplied labels on subsequent feature learning.
Together, the two interventions support the links from supervision to representations and from representations to predictions beyond the selected set. They use seed 0 on each dataset and hold supervision fixed, so they test these links without reproducing repeated rounds of selection and learning.
Part 3 reports the selection-delay test and theoretical range verification.
Response to Reviewer hz8G (3/3): Selection delay and theoretical scope
Q2 continued: Selection delay
C. Directly testing the selection-delay prediction
We tested whether newly admitted erroneous supervision delays other correct admissions. From each same-view epoch-65 checkpoint, a two-epoch continuation identifies newly admitted wrong targets. Paired branches then restart from the same checkpoint and optimizer state for 20 epochs: one trains normally, while the other masks an identified image’s loss when it is selected with its recorded wrong target. Inputs and loss denominators remain unchanged, and pseudo-labels are recomputed online.
Across three seeds per dataset, we measure first correct admission among initially correct but low-confidence candidates, excluding directly masked images. Candidates never correctly admitted receive a waiting time of 20 epochs.
On CIFAR-100, masking erroneous supervision increases the equal-seed mean waiting time from 2.723 to 2.974 epochs, with longer waits in all three seeds. CIFAR-10 shows mixed results. This intervention therefore does not consistently confirm the specific selection-delay prediction. We distinguish this finding from the separate evidence above that supplied supervision affects representations and subsequent predictions.
Q3: How far does the theorem extend beyond the original parameter instance? (Weakness 2 and Question 2)
A3. We verified the sufficient inequalities over continuous parameter ranges within a specified family. All 1,590 parameter blocks pass the original certificate’s 36 numerical checks and the common-time condition using 60-digit outward-rounded interval arithmetic. These checks cover every parameter value within the stated blocks.
Let denote the initially selected erroneous mass relative to the full unlabeled population. Let denote the initial semantic-strength gap, where and is the initial nonroot semantic strength.
| Variation | Certified range |
|---|---|
| Initial erroneous mass, with | |
| Initial strength gap, with | |
| Joint variation | , |
The parameters vary together through , root strength , gate-crossing strength , evaluation-group mass , and spurious-input scale . Other independent parameters retain their submitted Appendix B.10 values; the network structure and augmentation probabilities remain unchanged.
All settings share the gradient-flow interval . In the joint range, the nonroot filter’s Euclidean distance to the ReLU boundary on the unit evaluation patch lies in . This distance differs from the feature-strength gap .
The verification shows that the result holds over a continuous family rather than only the illustrated instance. It also quantifies the scope you asked about: the family retains high initial pseudo-label precision, initialization close to the activation boundary, and a short common comparison interval.
Within this scope, the dynamic result explains how learned shared features enable new selections and how those selections supply further learning signals. Freezing the initial selected identities and targets blocks learning along subsequent semantic feature pathways in the constructed model. The representation advantage also persists under optimal affine linear readouts. These results identify the role of evolving supervision beyond a comparison of the final trained classifiers.
Changes to the final paper
If our paper is accepted, we will incorporate the results reported in this response as follows:
-
Section 4.3 and Appendix C.1: add the three-seed native FixMatch accuracy and stability results, together with a protocol comparison distinguishing the same-view control from standalone USB Pseudo-Labeling.
-
Section 4.1: add the three-seed residual comparison on native FixMatch checkpoints, including independent weak views and results for correct and incorrect targets.
-
Section 4.2 and Appendices C.4–C.6: add the three-seed full selected-dataset replacement results and the additional seed-0 fixed-classifier and fixed-image controls, specifying recipient models, frozen targets and masks, probe methods, classifier-state checks, batch normalization, and offline candidate counts.
-
Section 3.4 and Appendix B.10: add the certified parameter ranges, linked parameter settings, common-time interval, and distances to the ReLU boundary.
-
Appendix C.7: report the online-masking results by seed, define the candidate pool and waiting-time calculation, and distinguish the conditional selection-delay result from its current empirical support.
Response to Reviewer hz8G (1/3): Stable training and repeated experiments
We respond to your comments in three parts.
We thank the reviewer for recognizing the value of our analysis of how feature learning and pseudo-label selection interact during training. We also appreciate your questions about training reliability, the role of the feature extractor, and the scope of the theoretical result. To address the training concern, we repeated the baseline experiments using a fresh official USB FixMatch implementation and recovered stable performance. For controlled comparisons, we derive the same-view Pseudo-Labeling control from this implementation by changing only the training view, keeping the remaining protocol unchanged.
Using these checkpoints, we repeated the original residual and full selected-dataset replacement tests, then added fixed-classifier and fixed-image controls to examine the mechanism. The repeated tests recover the learning-pressure and supervision effects; the additional controls examine the roles of the feature extractor and assigned labels. We also verify the theoretical inequalities over continuous parameter ranges. Below, we distinguish these findings from the specific selection-delay prediction tested by the online intervention.
Q1: Do the reported effects persist under stable, effective training? (Weakness 1 and Question 1)
A1. We recovered stable native FixMatch performance and repeated the original core experiments from its checkpoints. We downloaded the official USB implementation again, retained its FixMatch training code and reference hyperparameters, and trained WRN-28-2 for 1,024 epochs using three seeds.
Repeating the original core experiments. At native FixMatch epoch-65 checkpoints, we repeated the paired residual test from Section 4.1 on both datasets and all three seeds, adding an independent weak view. Strong views increased mean residuals for both correct and incorrect pseudo-labels.
For same-weak, independent-weak, and strong views, respectively, mean residuals were 0.0045, 0.0234, and 0.1502 on CIFAR-10, and 0.0103, 0.0688, and 0.3275 on CIFAR-100. The raw, non-EMA model evaluates identical selected images and fixed targets. We average four strong augmentations per image, then average across images and equally across seeds.
We repeated the full selected-dataset replacement design from Section 4.2 on both datasets and all three seeds. Each native FixMatch epoch-65 checkpoint provides two branches with identical model and optimizer states. They continue for 50 epochs with the same labeled supervision and strong-augmentation procedures, receiving either FixMatch's complete selected set or the same-view control's. Selected identities, original pseudo-labels, and selection masks remain frozen; both branches retain the native unlabeled loader and loss normalization. Endpoint probe accuracies are reported as mean ± sample SD over three seeds:
Probe accuracy favored FixMatch-source supervision in all six paired comparisons. Mean counts of correct high-confidence predictions on common pools initially rejected by both sources were 333.7 vs 31.0 on CIFAR-10 and 887.0 vs 110.0 on CIFAR-100, averaged equally over three seeds. These offline counts use confidence ≥0.95; the candidates receive no pseudo-label loss. Thus, the original finding that selected supervision affects later representations also holds when starting from native FixMatch checkpoints.
The probe fits a ridge linear classifier to frozen features using all 50,000 training labels and evaluates on 10,000 test images without updating the SSL model. These continuations use FixMatch as the recipient and a ridge probe, rather than repeating both original recipient types and the SGD probe.
The new full-training baseline curves measure EMA test accuracy; the original full-training probe and precision curves were not repeated. Q2 adds controls to examine how the supervision effect arises. The cause of the original deterioration has not been isolated.
Part 2 explains the protocol comparison and additional mechanism controls.