acceptodds
Under review as a conference paper at ICLR 2027

What Is Averaging Worth in Self-Training? Overlapping Correlation Bounds across Steps, Networks, and Views

Abstract

Most self-training methods use ensemble averaging across training steps (e.g., exponential moving average (EMA) teachers), networks, or input data transformation views to aid semi-supervised learning. However, little is known about how ensemble size and member weighting affect these averages and thus self-training efficacy. Here, we use a prediction error decomposition to show that variance reduction by training step, network, and view averaging overlaps and is bounded by measurable error correlation. We further identify the private and shared variance of each averaging type by inverting measured correlation, and show gains in effective sample size are sublinearly additive across averaging types. To connect variance reduction to increased task performance, we isolate the contributions of effective sample size gains from pseudo-label quality improvements. By decoupling these components, we show that averaging over larger ensembles approaches an upper bound in shared variance reduction and thus provides quickly diminishing task performance returns, suggesting that targeted reductions in private variance could more efficiently improve self-training. Together with extensive experimentation on Pascal VOC and Cityscapes, we apply our findings to optimize the ensemble averaging configurations of state-of-the-art self-training methods, which improves task performance on the ADE20K, COCO, Pancreas-CT, and Left Atrium datasets. This work establishes error correlation as a common budget that unifies step, network, and view averaging in self-training, and will guide the design of averaging configurations in future methods.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Semi-supervised learning, Self-training, Ensemble averaging, Error correlation, Effective sample size

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WWsMOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies the ensembling/averaging methods used in self-training (averaging over training steps, networks, or augmented views) and suggests that all three remove the same kind of error, so their benefits are not independent and are limited by how correlated the members' errors are.

Critical strengths

The area targeted by the paper is interesting. A fine grained decomposition of ensembling that accounts for common (non-independent) methods such EMA and models their specific case is a topic I would like to see studied. Also the specifics of semi-supervised learning could create and interesting problem solving relatively to ensembling in the general classification setup.

Critical weaknesses
  • Overall I found the paper extremely difficult to read, understand, and follow. This is a subset of the issues I had: (1) the actual task or problem setting is never defined. This seems very unusual to me. For example, a paper working in standard regression would typically start with something like "we have an input X∈RdX \in R^d, a target Y∈RY \in R, a function F:Rd→RF:R^d \to R etc etc". But we don't seem to have this here so its hard to understand the problem setting. (2) In preliminaries terms like "probe set" and "pixelwise error field" are just dropped without explaining what they are. I'm not familiar with these terms, and they seem to assume they mean something to the reader. (3) some notation is used in preliminaries without being defined. For example XX in Definition 3.1. This is essential to everything that follows and is not defined. The Δ\Delta symbol. The subscript notation seems an abuse of notation that i don't get. for example e(k)e_{(k)} is not clear to me. What do the parentheses mean exactly? (4) The intro of the paper is not reader friendly. Lots of sentences that seem plausible at a skim, but on carefully inspection I don't clearly understand after reading and re-reading multiple times. For example, I don't know what this sentence means: "In this paper, we show that the three average types reduce variance in the same prediction error component in a manner bounded by a measurable error correlation ρ, such that the performance gains produced by each type overlap." I get the same feeling throughout the whole intro section. Also "identify the private and shared variance of each average type", am i expected to know what private and shared variance are?
  • I spent time trying to understand the theory but struggled to connect it to existing ideas clearly. For example, there seems to be an idea of a shared variance component which remains even as you increase ensemble size and reduce variance in this way. I'm not clear how this is different to the bias component of a standard bias-variance decomposition of the ensemble setting. I know such decompositions already exist for ensembles (see e.g. [1] which is cited but not properly reconciled). I get the impression this is very close to being just the bias component. Also, as far as I can tell the main theorem of the paper 4.5 is effectively just expanding on an assumption the consequences that directly follow from it. So its more of an empirical claim than a theorem and this is not tested empirically.

[1] Wood, Danny, et al. "A unified theory of diversity in ensemble learning." Journal of machine learning research 24.359 (2023): 1-49.

Questions for authors

None beyond a request to respond to the issues described in the weakness section.

Reviewer_NN5LOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates the benefits and limitations of ensemble averaging across training steps (EMA), independently or jointly trained networks, and augmented input views in semi-supervised learning. The authors develop a unified variance decomposition that characterizes how these averaging strategies reduce overlapping components of prediction error. Using error correlations, they derive bounds on the effective ensemble size and show how the variance components of different averaging strategies can be identified.

The paper further distinguishes two mechanisms through which averaging influences self-training performance: reducing prediction variance (readout channel) and improving pseudo-label quality (target channel). The authors propose an empirical model connecting these contributions to downstream performance and validate their predictions through controlled experiments on Pascal VOC and Cityscapes, with additional classification experiments on CIFAR-100. Finally, they apply their analysis to UniMatch V2 and DualFete, using the estimated variance components to select averaging strategies and incorporate less-correlated models, reporting performance improvements across ADE20K, COCO, Pancreas-CT, and Left Atrium.

Critical strengths

S1. The paper extends classical ensemble variance analysis to jointly characterize averaging across training steps, networks, and input views. In particular, Theorem 4.5 provides a principled way to identify private and shared error variance components from measured correlations, explaining why the gains from different averaging strategies cannot simply be added. The framework produces testable predictions, including the increased correlation between EMA teachers relative to their students (Corollary 4.6), which is supported by the experiments in Section 5.1. This provides a useful perspective for analyzing averaging strategies that are typically studied independently.

S2. Beyond validating the variance bounds, the authors investigate how effective ensemble size and pseudo-label quality contribute to downstream performance. Their experiments cover multiple datasets and model families, including applications to UniMatch V2 and DualFete. The resulting performance improvements across four segmentation benchmarks provide evidence that the proposed analysis can inform ensemble configurations, rather than serving only as a theoretical characterization.

Critical weaknesses

W1. The theoretical framework relies on restrictive assumptions whose validity in realistic self-training settings is insufficiently established.

  • W1.1. Restrictive covariance assumptions: The main results depend on equal-variance ensemble members, specific correlation structures, and mutually uncorrelated error components (Assumptions 4.1 and 4.2). It remains unclear how robust the inferred private and shared variance components are when these assumptions are violated.

  • W1.2. Approximation of weight-space EMA: The analysis treats weight-space EMA as averaging prediction errors through a first-order linearization (Assumption 4.1(ii)). The accuracy of this approximation is not sufficiently evaluated along nonlinear neural network training trajectories.

  • W1.3. These limitations are significant because identifying private and shared variance components is the paper's central theoretical contribution and forms the basis for its recommendations on averaging strategies.

W2. The practical experiments do not sufficiently establish the added value of the proposed framework over existing ensemble strategies.

  • W2.1. Section 5.3 improves performance by expanding ensembles and introducing architecturally diverse models, both of which are established techniques. The authors do not demonstrate that their variance decomposition leads to better configuration choices than simpler approaches, such as selecting members based on prediction-error correlation or validation performance (Krogh & Vedelsby, 1994; Abe et al., 2023).

  • W2.2. Computational overhead: Adding networks and augmented views increases training and/or inference costs, but the reported performance gains are not systematically evaluated against these additional costs. This makes it difficult to assess whether the proposed strategy provides a meaningful practical advantage.

  • W2.3. Without demonstrating that the framework enables better or more efficient decisions than existing alternatives, its practical contribution remains unclear, despite the reported improvements.

Questions for authors

Q1: Assumption 4.1(ii) approximates the predictions of an EMA teacher by averaging the predictions of its historical student checkpoints. How closely does this approximation hold in practice, particularly for larger EMA decay rates or across different stages of training?

Q2: Can the authors demonstrate that the proposed variance decomposition leads to better averaging configurations than simpler alternatives, such as selecting ensemble members based on pairwise prediction-error correlation or validation performance? Ideally, this comparison would use a fixed training or inference budget.

Q3: What are the additional training and inference costs of the ensemble expansions in Section 5.3?

Q4: How sensitive are the inferred private and shared variance components to violations of Assumptions 4.1 and 4.2, particularly equal member variance and the assumed covariance structure?

Reviewer_cLXnOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
1
Soundness
3
Read review
Paper summary

The paper unifies the variance reduction prediction of ensemble averaging in self-training methods : averaging over training steps, averages over network predictions and averages over different views of the input. They show that a measurable error correlation permits to bound variance reduction after averaging, and the task performance when no further label is added. Under some assumptions (equal variances, uncorrelated components, first-order linearization for weight averaging), the authors prove their claim and compute the bounds mathematically. They further separate two channels through which a teacher affects performance: a readout channel, limited by the correlation bound and linked to mIoU through an empirically calibrated rate, and a target channel (pseudo-label quality), which is not limited by it. This insight permits to optimize self-training and the authors apply it mainly on segmentation datasets and extend their claim on a classification dataset (CIFAR-100). They make a strong focus on Pascal VOC and Cityscapes, and apply the method on ADE20K, COCO, Pancreas-CT, and Left Atrium datasets using state-of-the-art self-training methods.

Critical strengths
  1. The theory can be confronted with experiments through quantities easy to measure As claimed, the paper unifies step EMA, network averaging and view averaging. They propose to compute pairwise correlations ρ between member’s error fields, the effective ensemble size Neff from the variance over recorded error fields, and compare it with the theoretical model-based formula giving Neff as a function of ρ, the number of members M and the parameters of the model.

  2. The mathematical assumptions and proofs are sound, associated with an exhaustive literature search and sound experiments As an example of the usefulness of their work, the mathematics give the counterintuitive proof that for averaging over steps ensembling over raw students gains more than ensembling between teachers. For the practical use of their work, they empirically show a gain increase for ADE20K and COCO when using a more decorrelated member, as expected by the theory presented in the paper.

Critical weaknesses
  1. The improvements over state-of-the-art methods are modest and there is no significance test of the gain increase Improvements in performance range from 1 to 5 %. However, using more decorrelated members significantly raises the gain of performance for ADE20K and COCO dataset, which would be a strength. However, it remains the question if the gain increase is significant : indeed, from my understanding, only standard deviations among 3 seeds are provided.

  2. The paper is hard to read A potential user of the method in a practical case would find the article hard to read, hence hard to use, even if the paper presents practical quantities to measure. The paper would gain in having an easier description of the context of the work, and in having some practical diagrams or codes to help a potential user in making use of the method.

  3. The practical use of the method is not clear One of the main interests of the method lies in its practical use. The authors make little effort in explaining either by providing code or by giving diagrams how their method can be used to tailor or to make practical bounds on the number of members needed to improve performance by tackling the private variance.

Questions for authors

Clarification requests :

  • I find the entry point of the article difficult to follow. On figure 1, the authors could add a form of pipeline of how the bounds are computed and used. I also think that the first part of the introduction could be made more explicative on how a practitioner would use the proposed bounds.
  • Given the premisses, it would be good to have at least in the Appendix a rule of thumb/a diagram or a method of how a future user of self-training can optimize the choice of the added members.
  • I would like to know the significance of the reported increase particularly for the datasets with most performance gains (ADE20K, COCO)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.