- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies post-hoc fusion of a fixed pool of probabilistic predictors when group-labeled validation data are limited. The proposed method, AnchorFuse, evaluates each model’s contribution by comparing a shared anchor ensemble with the renormalized ensemble obtained after deleting that model. These contributions are weighted using group priorities constructed from residual correlation, predictive diversity, and calibration error. The resulting scores, together with a marginal log-loss penalty, are converted into one global probability mixture through an entropy-regularized update. Group labels are used for estimation and evaluation but are not required at inference.
The theoretical analysis decomposes deletion contributions into gradient and curvature terms and establishes sensitivity bounds for the score-to-weight map. Experiments on Adult, COMPAS, Clinical, and CelebA report improvements in worst-group utility over the evaluated baselines, including under reductions in the score-estimation sample.
Critical strengths
1. Relevant problem and a simple post-hoc procedure.
Improving subgroup performance without retraining the base predictors is a practically relevant problem. AnchorFuse provides an explicit, computationally simple update based on cached predictions. Measuring deletion contributions on the same examples gives an interpretable comparison of each model’s contribution within the anchor ensemble. The gradient–curvature decomposition also helps explain how finite deletion differs from a local gradient calculation.
2. Promising empirical results and several useful controls.
The reported worst-group improvements are substantial across the four benchmarks. The evaluation includes regularized and shrinkage-based minimax controls, structural-coefficient sensitivity analyses, independently retuned component removals, and repeated outer partitions. Also, it is nice that the manuscript explicitly distinguishes conditional update sensitivity from prediction-performance guarantees and acknowledges limitations of the experimental protocol.
Critical weaknesses
1. The Introduction does not adequately motivate or position the contribution.
The Introduction begins with four dataset-specific numerical comparisons before establishing the practical setting, explaining the subgroup-performance objective, or motivating why a fixed predictor pool and limited group-labeled validation data are problematic. It then moves quickly from concavity of the minimax objective to the proposed deletion scores, structural statistics, and entropic update. Consequently, the reader is given details before being given a convincing conceptual explanation for its construction.
Although related work is discussed in Section 6, the Introduction does not sufficiently distinguish the contribution from model averaging, group-robust optimization, and fairness-aware post-processing. The current contribution bullets largely describe the paper’s components rather than explaining their significance.
These are illustration points that the current presentation make the paper very difficult to follow.
2. The central structural-priority construction is insufficiently justified.
The most distinctive ingredient is the group priority in Equation (4), which increases with residual correlation and calibration error and decreases with predictive diversity. Section 3.2 provides intuition based on the squared-error behavior of uniform averaging under restrictive moment assumptions. However, a group being difficult for uniform averaging does not by itself establish that reweighting the available predictors can improve that group, or that the proposed logistic combination would work to improve worst-group accuracy or AUROC.
The theoretical results characterize deletion contributions and propagate score-estimation errors through the update, but they do not establish that the proposed priorities are well aligned with the target objective. In particular, Proposition 3 retains priority-estimation error as an explicit term rather than controlling it.
3. Empirical attribution, baseline verification, and reproducibility need strengthening.
The component comparisons are encouraging, but they do not fully explain the source of the gains. Section 3.3 correctly notes that structural modulation changes both group allocation and total score scale. However, Table 3 compares complete procedures rather than reporting the fixed-input, equal-mass intervention defined in Equation (6). Thus, it does not isolate the contribution of group-specific allocation under matched inputs and hyperparameters.
Questions for authors
1. Structural priorities.
Can the authors demonstrate that the priorities in Equation (4) identify groups with low utility and useful corrective information in the fixed pool? How variable are these priorities across estimation draws, particularly in the two-group benchmarks, and how do prevalence weights affect the smallest groups?
2. Mechanism-specific comparisons.
Can the authors report the fixed-input, equal-mass comparison in Equation (6), preferably with paired uncertainty estimates? A gradient-only comparator would also help determine whether finite-deletion curvature adds value: the current no-LOO ablation removes both the gradient and curvature terms.
3. Validation scarcity and tuning.
What happens when both and are reduced? Please report realized per-group sample counts and distinguish variability from estimating scores at fixed hyperparameters from variability of the complete tuned procedure.
4. Baseline verification and implementation.
Can the authors verify the minimax solutions using objective-gap diagnostics or an independently checked solver and provide benchmark reproduction code and permissible derived artifacts? Please also clarify whether contribution pooling was used in the reported experiments and specify the corresponding settings and any fallback inputs actually invoked.