- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Subagging is the process of training a copy of the base learner on each of many random subsets of size drawn without replacement from the training set of size , and averaging the predictions. The fraction is called the subsampling ratio. The paper studies this setup for a symmetric and square-integrable base learner, and, using the Hoeffding–ANOVA decomposition, it derives an exact finite-sample variance formula for the subagged model. This result shows that subagging multiplies each order- component of the base learner variance by a factor , interpreted as a "spectral low-pass filter" that suppresses high-order interactions. Addressing the bias-variance tradeoff, the authors argue that the optimal subsampling ratio (minimizing the authors' bound on the MSE) is lower (or equal) when more of the base learner’s variance comes from high-order interactions (which the authors describe as a heavier high-order spectrum).
The paper proposes CGAS, a method to choose based on cross-validation, where the grid depends on the base learner's complexity: for low-complexity learners (e.g., shallow trees), and for high-complexity learners (e.g., deep trees). CGAS is evaluated on nine OpenML regression datasets against fixed , in two regimes: depth-3 trees on the original labels, and unlimited-depth trees on labels with added Gaussian noise, with standard deviation of 1.5 times that of the labels. CGAS outperforms both fixed ratios on all nine datasets in the high-complexity regime, while it is comparable or slightly better in the low-complexity regime.
Critical strengths
-
A clean, explicit quantification of how subagging reduces variance (each order- component is scaled by ), which explains why this process suppresses variance from high-order interactions much more strongly than low-order ones. However, its significance is limited as this formula is classical (see Weakness 1).
-
The empirical experiments suggest that using a fixed subsampling ratio can be suboptimal, while tuning it according to the model's complexity (for example, using lower ratios for a complex base learner on noisy data) may lead to better results. In particular, in the high-complexity regime, CGAS improves over both fixed ratios on all nine datasets. This is useful for practitioners.
Critical weaknesses
- The main theoretical result of the paper (Theorem 3.1) is not novel, despite being presented in the abstract as ‘the first exact finite-sample variance decomposition for subagging’. As the authors note (Section 2.2), the subagged estimator in Equation (1) is a U-statistic of order with kernel . Since Equation (1) averages over all size- subsets, it is a complete one. Theorem 3.1 is exactly the classical variance formula obtained from Hoeffding’s H-decomposition. Indeed,
(The first expression for in Equation (5) is a typo. The correct form, derived in Appendix F, is used above.)
Thus, Equation (4) is , the standard expression (e.g., Peng, Coleman & Mentch, "Rates of convergence for random forests via generalized U-statistics," Electron. J. Stat. 16(1), 2022; see Equation (18) in arXiv:1905.10651v3). Since the main result is already known, the paper must be judged mainly on Section 4 and the experiments.
- The experiments don't test the main claim of the paper, that is governed by the base learner's interaction complexity. This is because CGAS restricts the search range in advance by tree depth (which is the paper’s proxy for complexity). This guarantees the shift of in Figure 1 by construction. Searching the same full grid for all base learners would test this directly.
Together, these leave the paper’s contribution resting mainly on Theorem 4.5, which is not actually tested by the experiments.
Questions for authors
- What is the difference between Theorem 3.1 and the classical H-decomposition variance formula?
- Does the complexity-dependent shift in survive when using a single shared grid for all base learners?