acceptodds
Under review as a conference paper at ICLR 2027

AnchorFuse: Stable Model Fusion for Small Validation Groups

Abstract

Combining pretrained predictors can raise average performance while leaving the weakest group behind. With small validation groups, choosing mixture weights becomes an estimation problem. AnchorFuse forms a probability mixture of frozen predictors by measuring leave-one-model-out contributions at a shared anchor, weighting them by group error structure, and applying one entropy-regularized update. We decompose deletion contributions into an anchor-gradient term and a nonnegative curvature remainder. A shift-invariant sensitivity bound separates score estimation error from the response of the update at fixed hyperparameters. Across four benchmarks, AnchorFuse improves worst-group utility over the strongest evaluated minimax control by 1.50–2.80 points with the full estimation set. When the estimation set is reduced to a nominal one eighth while the selection set is retained, its margins over this control reach 5.95 points on COMPAS and 6.65 points on Clinical. A 27-setting coefficient study, independently retuned component removals, and repeated data partitions further characterize the complete procedure.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Ensemble Learning, Worst-Group Robustness, Small-Sample Estimation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_h79VOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies post-hoc fusion of a fixed pool of probabilistic predictors when group-labeled validation data are limited. The proposed method, AnchorFuse, evaluates each model’s contribution by comparing a shared anchor ensemble with the renormalized ensemble obtained after deleting that model. These contributions are weighted using group priorities constructed from residual correlation, predictive diversity, and calibration error. The resulting scores, together with a marginal log-loss penalty, are converted into one global probability mixture through an entropy-regularized update. Group labels are used for estimation and evaluation but are not required at inference.

The theoretical analysis decomposes deletion contributions into gradient and curvature terms and establishes sensitivity bounds for the score-to-weight map. Experiments on Adult, COMPAS, Clinical, and CelebA report improvements in worst-group utility over the evaluated baselines, including under reductions in the score-estimation sample.

Critical strengths

1. Relevant problem and a simple post-hoc procedure.

Improving subgroup performance without retraining the base predictors is a practically relevant problem. AnchorFuse provides an explicit, computationally simple update based on cached predictions. Measuring deletion contributions on the same examples gives an interpretable comparison of each model’s contribution within the anchor ensemble. The gradient–curvature decomposition also helps explain how finite deletion differs from a local gradient calculation.

2. Promising empirical results and several useful controls.

The reported worst-group improvements are substantial across the four benchmarks. The evaluation includes regularized and shrinkage-based minimax controls, structural-coefficient sensitivity analyses, independently retuned component removals, and repeated outer partitions. Also, it is nice that the manuscript explicitly distinguishes conditional update sensitivity from prediction-performance guarantees and acknowledges limitations of the experimental protocol.

Critical weaknesses

1. The Introduction does not adequately motivate or position the contribution.

The Introduction begins with four dataset-specific numerical comparisons before establishing the practical setting, explaining the subgroup-performance objective, or motivating why a fixed predictor pool and limited group-labeled validation data are problematic. It then moves quickly from concavity of the minimax objective to the proposed deletion scores, structural statistics, and entropic update. Consequently, the reader is given details before being given a convincing conceptual explanation for its construction.

Although related work is discussed in Section 6, the Introduction does not sufficiently distinguish the contribution from model averaging, group-robust optimization, and fairness-aware post-processing. The current contribution bullets largely describe the paper’s components rather than explaining their significance.

These are illustration points that the current presentation make the paper very difficult to follow.

2. The central structural-priority construction is insufficiently justified.

The most distinctive ingredient is the group priority in Equation (4), which increases with residual correlation and calibration error and decreases with predictive diversity. Section 3.2 provides intuition based on the squared-error behavior of uniform averaging under restrictive moment assumptions. However, a group being difficult for uniform averaging does not by itself establish that reweighting the available predictors can improve that group, or that the proposed logistic combination would work to improve worst-group accuracy or AUROC.

The theoretical results characterize deletion contributions and propagate score-estimation errors through the update, but they do not establish that the proposed priorities are well aligned with the target objective. In particular, Proposition 3 retains priority-estimation error as an explicit term rather than controlling it.

3. Empirical attribution, baseline verification, and reproducibility need strengthening.

The component comparisons are encouraging, but they do not fully explain the source of the gains. Section 3.3 correctly notes that structural modulation changes both group allocation and total score scale. However, Table 3 compares complete procedures rather than reporting the fixed-input, equal-mass intervention defined in Equation (6). Thus, it does not isolate the contribution of group-specific allocation under matched inputs and hyperparameters.

Questions for authors

1. Structural priorities.

Can the authors demonstrate that the priorities in Equation (4) identify groups with low utility and useful corrective information in the fixed pool? How variable are these priorities across estimation draws, particularly in the two-group benchmarks, and how do prevalence weights affect the smallest groups?

2. Mechanism-specific comparisons.

Can the authors report the fixed-input, equal-mass comparison in Equation (6), preferably with paired uncertainty estimates? A gradient-only comparator would also help determine whether finite-deletion curvature adds value: the current no-LOO ablation removes both the gradient and curvature terms.

3. Validation scarcity and tuning.

What happens when both DstatD_{\mathrm{stat}} and DhpD_{\mathrm{hp}} are reduced? Please report realized per-group sample counts and distinguish variability from estimating scores at fixed hyperparameters from variability of the complete tuned procedure.

4. Baseline verification and implementation.

Can the authors verify the minimax solutions using objective-gap diagnostics or an independently checked solver and provide benchmark reproduction code and permissible derived artifacts? Please also clarify whether contribution pooling was used in the reported experiments and specify the corresponding settings and any fallback inputs actually invoked.

Reviewer_Bn3SOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The article presents a heuristic aggregation algorithm that can be based on group structure, construct a set of weights through leave one out and a comprehensive index, and achieve the fusion of pre-trained models.

Critical strengths
  1. This method is intuitive and easy to understand, and it considers the fusion of multiple statistics.

  2. Based on this method, the classification performance of the worst group in the experiment was indeed improved.

Critical weaknesses
  1. The literature review in the article is not sufficient, and there are also expression problems in the writing (such as the second-to-last line on the third page)

  2. The theory presented in the article lacks persuasiveness. Define w⋆∈arg⁡max⁡w∈ΔKmin⁡gQg(w),w^\star\in\arg\max_{w\in\Delta_K}\min_g Q_g(w), and the theoretical analysis should discuss the distance between w∗w^* and $\hat w $, such as $\min_g Q_g(w^\star) - \min_g Q_g(\widehat w) $. However, existing theoretical results do not provide such a comparative analysis.

Questions for authors
  1. Why does equation 4 use an additive structure to define r^g\hat r_g, is there any intuitive understanding?

  2. The $\hat \Gamma_{k, g} $ generated by the leave-one-out method has a significant correlation among different model kk. Will this correlation between models affect the final aggregation effect?

  3. The result of Proposition 3 still has some estimation terms on the right-hand side, which are not non-random. For example, the upper bound of $ |\hat r_g-r_g| $ is currently not available. Therefore, it is uncertain how persuasive the theoretical result is.

Reviewer_A94MOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I find the presentation difficult to follow, and I think the paper does a poor job of communicating the problem it is trying to solve. Some examples:

In the first paragraph, the authors discuss “uniform averaging,” “overall utility,” “worst-group utility,” and “ensemble benefit” without explaining what is being averaged, what the utility measures, what ensemble is being considered, or what experiments have been performed. Numerical results are presented before the experimental setting has been introduced.

The second paragraph introduces the “concavity of a worst-group log-likelihood” without first defining the likelihood or formulating the optimization problem. The authors then refer to “this estimation problem,” although the problem itself has not been introduced.

The rest of the introduction then introduces concepts such as the “anchor,” deletion contributions, structural modulation, and entropic updates without explaining why they are meaningful or how they relate to the problem being addressed.

This pattern continues in Section 2. The paper frequently introduces notation and concepts without establishing their meaning, motivation, or connection to the preceding discussion. The presentation is mostly a succession of definitions and mathematical statements that do not collectively explain the research question.

For example, in the subsection “Why the small-group regime is difficult,” the authors state that the variance of an empirical mean increases as the sample size decreases. Unless I am missing something, Q^g(w)\widehat Q_g(w) is simply an average of independent random variables under the stated assumptions, so its standard deviation scales as 1/ng1/\sqrt{n_g}. This is a standard statistical fact, and it is unclear whether it establishes any specific difficulty that the proposed method is intended to resolve.

To be clear, the mathematical statements in Section 2 are terse but, as far as I can tell, not incorrect. Each makes sense in isolation. The issue is that quantities are frequently defined without their relevance being established.

I urge the authors to revise the manuscript to (i) clearly state the problem they are solving, (ii) explain why it matters, (iii) define important concepts as they are needed (groups, utilities, the ensemble, and the estimation target), (iv) identify precisely what fails for existing approaches in the small-group regime, and (v) explain why each component of the proposed method responds to that failure.

Critical strengths

None

Critical weaknesses

See summary.

Questions for authors

None.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.