Minimax Boosting with Strong Rules

Current boosting methods excel at combining weak rules, such as decision trees, that are well-suited to certain types of tabular data. However, they cannot effectively combine strong rules, such as neural networks, which are required for data with a hierarchical structure. Existing boosting approaches generalize poorly with strong rules because they greedily minimize empirical losses disregarding out-of-sample expected losses. This paper presents MBoost, a minimax approach for boosting that can effectively leverage strong rules. Unlike existing methods, MBoost minimizes worst-case expected losses and accounts for the generalization behavior of base rules. Both theoretical and experimental results show that MBoost methods can achieve excellent performance with strong base rules, overcoming the limitations of current approaches.