Functional Regularized Gradient Boosting with Random Forest Weak Learners

Gradient Boosting is one of the most effective ensemble techniques in predictive modeling, but it often faces issues of overfitting and instability when trained on limited or noisy data. In this study, we present a modified version called Functional Regularized Gradient Boosting with Random Forest Weak Learners (FR-GBRF). The method integrates functional regularization across boosting stages with Random Forest weak learners to improve model stability and generalization. The regularization term controls abrupt functional updates, while the Random Forest base learners reduce variance through averaging multiple decision trees. Theoretical derivations and experiments show that this approach produces smoother convergence, lower variance, and more interpretable results without significant additional computational cost.