- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies the ensembling/averaging methods used in self-training (averaging over training steps, networks, or augmented views) and suggests that all three remove the same kind of error, so their benefits are not independent and are limited by how correlated the members' errors are.
Critical strengths
The area targeted by the paper is interesting. A fine grained decomposition of ensembling that accounts for common (non-independent) methods such EMA and models their specific case is a topic I would like to see studied. Also the specifics of semi-supervised learning could create and interesting problem solving relatively to ensembling in the general classification setup.
Critical weaknesses
- Overall I found the paper extremely difficult to read, understand, and follow. This is a subset of the issues I had: (1) the actual task or problem setting is never defined. This seems very unusual to me. For example, a paper working in standard regression would typically start with something like "we have an input , a target , a function etc etc". But we don't seem to have this here so its hard to understand the problem setting. (2) In preliminaries terms like "probe set" and "pixelwise error field" are just dropped without explaining what they are. I'm not familiar with these terms, and they seem to assume they mean something to the reader. (3) some notation is used in preliminaries without being defined. For example in Definition 3.1. This is essential to everything that follows and is not defined. The symbol. The subscript notation seems an abuse of notation that i don't get. for example is not clear to me. What do the parentheses mean exactly? (4) The intro of the paper is not reader friendly. Lots of sentences that seem plausible at a skim, but on carefully inspection I don't clearly understand after reading and re-reading multiple times. For example, I don't know what this sentence means: "In this paper, we show that the three average types reduce variance in the same prediction error component in a manner bounded by a measurable error correlation ρ, such that the performance gains produced by each type overlap." I get the same feeling throughout the whole intro section. Also "identify the private and shared variance of each average type", am i expected to know what private and shared variance are?
- I spent time trying to understand the theory but struggled to connect it to existing ideas clearly. For example, there seems to be an idea of a shared variance component which remains even as you increase ensemble size and reduce variance in this way. I'm not clear how this is different to the bias component of a standard bias-variance decomposition of the ensemble setting. I know such decompositions already exist for ensembles (see e.g. [1] which is cited but not properly reconciled). I get the impression this is very close to being just the bias component. Also, as far as I can tell the main theorem of the paper 4.5 is effectively just expanding on an assumption the consequences that directly follow from it. So its more of an empirical claim than a theorem and this is not tested empirically.
[1] Wood, Danny, et al. "A unified theory of diversity in ensemble learning." Journal of machine learning research 24.359 (2023): 1-49.
Questions for authors
None beyond a request to respond to the issues described in the weakness section.