- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces MBoost, a minimax boosting method meant for strong base rules like neural networks. The key idea is an uncertainty set where each rule's expected error can differ from its training error by at most . Through duality, the minimax problem becomes a convex objective (Eq. 8). In this objective, each rule gets an penalty equal to its own estimated generalization gap. Theorem 2 bounds the risk using these estimates, and Theorem 3 relates it to the Bayes risk. The algorithm estimates on a fresh held-out split in each round. The authors test it on 9 image datasets with ResNet-50, 9 tabular datasets with fitcauto, and a synthetic index-2 task.
Critical strengths
1. Per-rule penalty derived from duality. The core idea is good. For neural networks, a Rademacher penalty over a whole subfamily (as in DeepBoost) is too loose and treats all networks alike. A penalty based on each rule's own held-out gap is more practical, and here it follows directly from the duality rather than being a heuristic. I checked the proofs in Appendices A–D and did not find errors.
2. Calibration. The method also calibrates well. Because the log-loss is strictly proper, the learned rule equals the worst-case conditional distribution. The ECE results in Table 1 and Figs. 3 and 5 are consistent with this.
Critical weaknesses
In short, I find the theory clean, but I could not confirm that the experiments test the proposed method. There are two kinds of gaps.
Gaps between the theory and the algorithm. These are not errors in the proofs. Theorem 2 gives an upper bound only when . The algorithm uses a point estimate, which will be below about half the time. The analysis also uses the true marginal , while Algorithm 1 uses the empirical one. I would like the authors to discuss both, and perhaps try plus a confidence width.
Gaps between the algorithm and the released code. MBoost.py does not run as provided. Line 119 is indented by 9 spaces, so Python raises an IndentationError, and train_files (line 159) is never defined. I therefore cannot tell if this script produced Table 1. Please treat the points below as questions. If they hold for the real experiments, they affect the main results.
-
Frozen backbone. In round 1,
freeze_backbone=Truemakes every ResNet layer non-trainable (lines 38, 96–97). Later rounds reuse that model as it is (line 43). Only the 2-unit head is ever trained. The paper, however, says the pretrained weights are fine-tuned (lines 1050–1051). Were the image base rules only linear heads on fixed features? Were the baselines set up the same way? -
Coefficients applied incorrectly. Both scripts fit the solver on , so the fitted coefficients equal . Predictions and sample weights then use these coefficients directly (py lines 144, 149; m lines 42, 46). In effect, each rule is weighted up by . Rules that overfit more therefore count more, which goes against the purpose of the method. The fitted intercept is also discarded in both scripts. In the Python version, the weight formula on line 144 has the wrong sign: it gives and not as in Alg. 1. The MATLAB version gets this right. Also, liblinear with shrinks the effective penalty to .
-
Held-out leakage. The data is reshuffled in every round (Alg. 1, line 8), but image models start from the previous round's weights (lines 1054–1055). The current held-out points were likely seen in training before. This should make too small. Tabular models are retrained from scratch, so they are fine. A fixed held-out set, or results without warm starts, would settle this.
-
Estimating . With only 100 held-out points, can happen. That leads to division by zero and violates the interior condition of Lemma 4. How do the authors handle it? On Heart, Haberman and Ion, each
fitcautocall gets only about 173–216 points. An ablation on would be useful. -
Experimental setup.
MBoost.mgivesfitcauto30 seconds per round (line 4), but the paper reports 360 seconds. It also uses one 10% holdout, not 10-fold CV. On tabular data, Table 4 shows MBoost running about 15 times longer than the baseline. This conflicts with the claim of no extra overhead (line 447), so a baseline with equal compute is needed. The image ensemble only re-initializes the final layer, which makes it a weak comparison. Finally, several bolded wins in Table 1 lie within one standard deviation. Paired tests over the folds would help.
Questions for authors
Clarification requests (could change my assessment):
- Was Table 1 produced with the released scripts, or with different code?
MBoost.pydoes not run as provided (indentation error at line 119;train_filesundefined at line 159). - Were the image base rules only linear heads on a frozen ResNet-50 backbone, as
freeze_backbone=Truein round 1 implies? Were the baselines set up the same way? - Is the fitted coefficient vector rescaled by before it is used for predictions and sample weights, and is the intercept intentionally discarded? Is the sign of the weight formula in
MBoost.py(line 144) a bug? - How is handled, given the division by and the interior condition of Lemma 4?
- Which setting was used for the tabular experiments:
fitcautowith 30 s per round and a single 10% holdout (as inMBoost.m), or 360 s and 10-fold CV (as in the paper)?
Suggested additional experiments:
- Image results with a fine-tuned backbone and a fixed held-out set (or without warm starts).
- A compute-matched baseline on tabular data.
- An ablation on , and a variant using plus a confidence width.
- Paired significance tests over the folds for Table 1.
I am willing to raise my score if the authors (i) confirm or fix the code behind Table 1, (ii) report results with a fine-tuned backbone and a fixed held-out set, and (iii) include a compute-matched baseline.