AnchorFuse: Stable Model Fusion for Small Validation Groups

Combining pretrained predictors can raise average performance while leaving the weakest group behind. With small validation groups, choosing mixture weights becomes an estimation problem. AnchorFuse forms a probability mixture of frozen predictors by measuring leave-one-model-out contributions at a shared anchor, weighting them by group error structure, and applying one entropy-regularized update. We decompose deletion contributions into an anchor-gradient term and a nonnegative curvature remainder. A shift-invariant sensitivity bound separates score estimation error from the response of the update at fixed hyperparameters. Across four benchmarks, AnchorFuse improves worst-group utility over the strongest evaluated minimax control by 1.50–2.80 points with the full estimation set. When the estimation set is reduced to a nominal one eighth while the selection set is retained, its margins over this control reach 5.95 points on COMPAS and 6.65 points on Clinical. A 27-setting coefficient study, independently retuned component removals, and repeated data partitions further characterize the complete procedure.