An Adaptive Stochastic Trust-Region Method With Variance Reduction

We introduce a class of stochastic trust-region methods for unconstrained nonconvex optimization that integrates variance reduction techniques to accelerate convergence. Specifically, we develop two algorithmic variants, TR-SVRG and TR-SpiderBoost, by incorporating the stochastic variance-reduced gradient (SVRG) (Johnson & Zhang, 2013) and SpiderBoost (Wang et al., 2019) estimators, respectively, into a trust-region framework. Unlike classical trust-region approaches, our methods operate exclusively with stochastic gradient information and eliminate the need for function value evaluations. The trust-region radius is adaptively determined using a radius-control parameter and the variance-reduced gradient estimate. Under standard assumptions, we establish convergence in expectation to a first-order stationary point. Notably, our methods attain iteration and sample complexity bounds that match those of SVRG and SpiderBoost combined with stochastic gradient descent (SGD), while accommodating stochastic Hessian approximations that may be correlated with the gradient estimates, a regime beyond the scope of existing variance-reduction theory. Numerical experiments demonstrate that variance reduction techniques and trust-region methods mutually enhance performance, proper Hessian choice benefits algorithmic performance, and careful tuning of batch size and inner-loop parameters is vital for algorithm efficiency.