acceptodds
Under review as a conference paper at ICLR 2027

STABLE REWEIGHTING FOR HETEROGENEOUS EVIDENCE FUSION UNDER SOURCE EXCLUSION

Abstract

How much can excluding a single evidence source change a fused prediction, even when the candidate models remain unchanged? This question is formalized as source-exclusion stability for fixed-support evidence fusion: controlling the largest shift induced in the fused predictive distribution when any one source’s complete evidence contribution is removed. We propose Source-Exclusion Stable Reweighting (SESR), a reweighting objective that, to our knowledge, is the first to directly penalize the largest, rather than average, squared displacement between the all-source mixture weights and each leave-one-source-out mixture weight, jointly optimized on a shared candidate support. We prove a deterministic bound connecting this worst-case weight displacement to predictive total-variation shift, establishing weight-space coupling as a principled surrogate for prediction stability.On five weak-supervision datasets, SESR reduces worst-case prediction shift by 18.9–31.9% relative to Independent Reweighting (IR) and by 15.3–25.9% relative to Average Coupling (AC) across all dataset means, with corresponding weight shift reductions and broadly preserved classification accuracy. These results support source-exclusion stability as an explicit, optimizable objective for heterogeneous evidence fusion.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Weak Supervision, Evidence Fusion, Robust Reweighting, Distribution Shift, Model Stability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1cy7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how stable a mixture of fixed candidate models remains when one source of evidence is removed. It proposes SESR, which learns the weights using all sources together with the weights obtained after removing each source. It also adds a penalty for the largest change in weights. The paper gives a bound that links weight changes to prediction changes. In experiments on five weak supervision datasets, SESR is more stable than IR and AC when using 35 optimization steps, while accuracy is usually similar. However, the more complete optimization results in the appendix show mixed results compared with AC.

Critical strengths
  • The problem is clearly defined, and the bound is correct when the candidate models are kept fixed.

  • SESR consistently improves the reported stability measure over IR and AC.

  • The paper includes some useful analyses and acknowledges cases where other methods perform better.

Critical weaknesses
  • SESR keeps the excluded-source weights close to the original weights, but this may limit how well they fit the remaining evidence. Simply keeping all weights unchanged would give zero prediction shift. It would be useful to compare with this baseline and report how much evidence fit is lost after exclusion.

  • AC selects the largest tested (gamma=10) in all 25 cases, so a wider search range seems necessary. Comparing the methods at similar evidence-fit costs would also be fairer. The 35 optimization iterations seem limited, and I think that convergence results would help .

  • The main experiments use fixed classifiers trained with gold labels. The appendix includes different candidate models on three datasets, but SESR beats AC in only 6 of 15 cases. It is still unclear whether the results apply to other kinds of models or to the scientific uses discussed in the introduction.

  • The theoretical bound follows from standard norm inequalities and does not show why SESR should outperform other coupling methods. The paper could explain its methodological novelty more clearly.

  • Some figures and diagnostic labels need clarification. Internal labels from Figure 2 are difficult to read at page size. Table 2 should give the calculation for “Kish (%)”, and Figure 2 should define “evidence mass” precisely.

Questions for authors
  • How much fit to the remaining evidence is lost with SESR compared with AC and fixed weights?

  • Does SESR still outperform AC with a larger γ\gamma range and sufficient optimization convergence?

  • Which exact versions of the datasets were used, and can the authors provide the full results?

1 comment

We sincerely thank the reviewer for the detailed, constructive feedback and for recognizing the clarity of our problem definition, the correctness of the fixed-support bound, and the value of the additional analyses. Your comments helped us improve the experimental controls, reproducibility, and presentation. We address each concern below.

  1. Frozen weights and remaining-evidence adaptation. We agree that zero prediction shift alone is insufficient evidence of useful robustness. We added Frozen-IR and same-anchor Frozen-SESR baselines (Section 4.3; Appendix C.5). Under the original protocol, Frozen-SESR has zero weight and prediction shift but incurs mean remaining-evidence regret of 0.00263826, versus 0.00005843 for SESR and 0.00002775 for AC. Under high-accuracy optimization, SESR achieves lower mean regret than AC in 24/25 supports, although its maximum regret is higher in 14/25. We also report LOSO Accuracy, NLL, and Macro-F1, and explicitly distinguish evidence-fitting regret from gold-label utility.

  2. Larger γ range, convergence, and comparable evidence-fit costs. We agree. We added high-accuracy optimization, γ expansion to {30, 100, 300}, and matched-support results (Appendices B.6 and C.4). The original 35-step SESR advantage over AC is 25/25; under high-accuracy optimization with reselection it becomes 14/25. The extended γ range does not change the selected high-accuracy outcomes. We explicitly acknowledge this protocol dependence. We also added comparisons at approximately matched nontrivial evidence-fit costs (Appendix C.8). Such matches are available for 14/25 supports on the original seven-point grid and 25/25 on the extended grid, with both favorable and unfavorable outcomes. We do not claim dominance at exactly equal costs.

  3. Candidate heterogeneity and scientific applicability. We agree that the original 24 candidates, constructed from one logistic-SGD pipeline, provide a limited test of generality. We added 15 heterogeneous candidate banks across IMDb, Spouse, and ChemProt, containing 360 candidates from logistic regression, linear SVM, and Naive Bayes (Appendices B.7 and C.7). SESR improves prediction stability over IR and mean remaining-evidence regret over same-anchor Frozen-SESR in all 15 supports, but outperforms AC in only 6/15. We also acknowledge that the models share text features and therefore do not establish broad architectural generalization. IDP ensemble reweighting remains a motivating application; we do not claim direct NMR/SAXS validation.

  4. Methodological novelty and theoretical bound. We agree that the weight-to-prediction bound follows standard norm inequalities and does not establish SESR's superiority over other coupling methods. We clarified this limitation in Section 3.3 and added the selective-coupling KKT result in Appendix A.2. Under the stated assumptions, strictly inactive SESR branches retain their independent remaining-evidence optima at exact optimality, whereas AC generally exerts coupling pressure on all nonzero-displacement branches. This provides a more precise characterization of our methodological contribution.

  5. Figure readability and metric definitions. We revised Figure 2 to improve its organization and source-specific labels. Section 4.3 now gives the explicit Kish (%) calculation, and Appendix D defines vote share, row coverage, and normalized evidence mass.

  6. Dataset versions and numerical reproducibility. We expanded Appendix B with dataset statistics, candidate construction, random seeds, optimization settings, source-loss normalization, and selection rules. The local WRENCH snapshots could not be uniquely matched to an immutable public repository commit, and we now explicitly disclose this limitation rather than claiming an exact version that we cannot verify. We have also organized the numerical results and reproducibility code, including per-support comparisons and the additional optimization protocols. Reproduction from frozen inputs is distinguished from full end-to-end reconstruction, which remains incomplete. We sincerely appreciate these suggestions, which helped us make the paper more transparent about both its contributions and its limitations.

Reviewer_6R29OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Given an a collection of predictors, this paper introduces a method to find ensemble weights that make it so that predictions change little when one particular evidence source is excluded. Specifically they consider

Predictors: h1,...,hNh_1, ..., h_N.

Sources of evidence: s=1,...,Ss = 1, ..., S used to evaluate predictors.

For each source, they compute a loss ℓs,i\ell_{s,i}. This measures how well predictor ii agrees with source labels. If the prediction is given by the ensemble

pw(y∣x)=∑iwihi(y∣x)p_{w}(y | x) = \sum_{i} w_{i} h_{i}( y | x)

then they consider

FA(w)=1∣A∣∑i,swiℓi,s+λKL(w∣∣π0)\mathcal F_{A}(w) = \frac{1}{|A|} \sum_{i, s} w_{i} \ell_{i,s} + \lambda \text{KL}(w || \pi_0)

in other words, we want to find the weighted average of the best predictors while remaining as close to π0\pi_0 as possible.

Their proposed objective is

FS(w(0))+1∣S∣∑sFS−s(w(−s))+γmax⁡s∣∣w(0)−w(−s)∣∣2\mathcal F_{S}(w^{(0)}) + \frac{1}{|S|} \sum_{s} \mathcal F_{S - s}(w^{(-s)}) + \gamma \max_{s} ||w^{(0)} - w^{(-s)} ||^2

That last penalty is their main idea. It pulls the full-source ensemble and the most displaced exclusion ensemble toward one another. Both sets of weights can move and are coupled by this penalty.

Critical strengths

The main objective is clearly presented and it does seem reasonable. When several ensembles fit the evidence well, preferring one that depends less heavily on a single source is sensible. Their formulation explicitly balances evidence fit against that sensitivity.

Critical weaknesses

The demonstrated contribution is fairly narrow. The method uses a simple regularization construction, and the theorem establishes a basic relationship between weight changes and prediction changes. Its significance therefore rests largely on the empirical evidence. Although the experiments consistently show reduced sensitivity, it remains unclear whether a carefully tuned and sufficiently optimized average-coupling baseline could achieve a comparable stability–evidence-fit tradeoff. Average coupling selects the largest tested penalty in all 25 runs, and both coupled methods receive only 35 optimization steps. These observations do not establish that the baseline is underperforming because of tuning or optimization, but they motivate extending the penalty range and checking convergence before drawing strong conclusions about the advantage of maximum coupling. The main tables also report means without measures of variability. Reporting the paired improvements across the five candidate-bank realizations would clarify the consistency and magnitude of the gains. Additional realizations, if computationally feasible, would improve uncertainty estimates, while broader tuning and convergence checks would make the baseline comparison more conclusive.

The related works sections is also poorly written and reads like a list of papers. it would be more helpful if the authors could do a better job explaining other work and how it relates to what is presented. As written it makes it relatively challenging to evaluate this paper relative to other work.

Questions for authors

See above.

1 comment

We sincerely thank the reviewer for the thoughtful assessment of our objective and for highlighting the importance of optimization convergence, fair baseline tuning, and clearer positioning relative to prior work. Your concerns motivated substantial additional analyses and helped us qualify our empirical conclusions more carefully.

  1. Methodological contribution and comparison with AC. We agree that worst-case regularization is an established principle and that our contribution should not be presented as a general minimax innovation. We have clarified that SESR jointly optimizes the all-source solution and every single-source-excluded solution on fixed candidate support. Beyond the difference between maximum and average penalties, our new KKT analysis (Section 3.2; Appendix A.2) establishes a selective-coupling property: strictly inactive SESR branches retain their independent remaining-evidence optima under the stated conditions, whereas AC generally couples all nonzero-displacement branches.

This explains the distinct allocation of adaptation pressure, without asserting universal predictive superiority. 2. AC tuning, optimization convergence, and the γ range. We agree that the original 35-step protocol and AC's selection of γ = 10 at the grid boundary were insufficient to establish an optimization-independent advantage. We therefore added high-accuracy convex optimization and four explicitly distinguished protocols (Appendices B.6 and C.4): A: original 35-step optimization; B: high-accuracy optimization at the originally selected γ; C: high-accuracy optimization with γ reselection; D: high-accuracy optimization with γ extended to include 30, 100, and 300. SESR outperforms AC in 25/25 supports under A, but in 12/25 under B and 14/25 under C and D. Under C/D, the mean paired AC−SESR prediction-shift difference is −0.00057326, and the extended grid leaves the selected results unchanged. These results directly address the reviewer's concern: SESR's original advantage over AC depends on the finite-budget optimization protocol. We now state this limitation prominently in Section 4.2 and the Discussion rather than claiming unconditional superiority. We also report numerical accuracy checks and their limitations, without claiming exact convergence certificates. 3. Variability and paired comparisons. We added paired results across all 25 supports, including mean differences, medians, sample standard deviations, ranges, and win/loss counts (Appendix C.4, Tables 11–13). We additionally report fixed-γ sensitivity and an alternative selection-threshold convention in Appendix C.3. These analyses make the consistency and magnitude of the comparisons more transparent. 4. Related work and positioning.

We revised Section 2 to clarify the relationship between SESR and source valuation, Group DRO, missing-modality robustness, predictive stacking, and consistency regularization. The central distinction is that SESR controls the displacement between jointly fitted mixture solutions under complete source-loss exclusion, rather than directly optimizing source value, worst-group predictive risk, or prediction consistency between augmented inputs. We appreciate the reviewer's critical suggestions. They led us to substantially strengthen the optimization analysis, temper our original conclusions, and present a more balanced comparison with AC.

Reviewer_EBPYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Source-Exclusion Stable Reweighting (SESR) to improve the stability of evidence fusion when individual evidence sources are excluded. SESR jointly optimizes the all-source and leave-one-source-out mixture weights by penalizing their maximum squared displacement. Experiments on five weak-supervision datasets demonstrate improved worst-case prediction stability over independent reweighting and average coupling, with generally comparable classification performance.

Critical strengths
  1. The paper clearly formulates source-exclusion stability and provides a straightforward optimization framework that directly targets worst-case weight displacement.

  2. The experiments include mechanism-matched baselines, sensitivity analyses, and source-structure case studies, offering useful insights into the stability–utility trade-off.

Critical weaknesses
  1. The primary methodological difference from independent reweighting is the introduction of a worst-case coupling penalty, while the distinction from average coupling is essentially replacing the average squared displacement with its maximum. Although this formulation is reasonable, worst-case regularization is an established optimization principle, and the paper does not sufficiently establish why this particular formulation constitutes a substantial methodological advance.

  2. The ultimate objective is to reduce changes in predictive distributions, but SESR optimizes distances between mixture weights. Since the candidate predictions are fixed, prediction-level distances could also be directly incorporated into the optimization objective. The proposed upper bound may be loose when candidate predictions are highly similar, and the paper does not compare weight-space coupling against direct prediction-space regularization.

  3. How does the proposed approach distinguish undesirable prediction instability from legitimate changes caused by removing informative evidence? Could the authors provide more direct evidence that improved stability does not come at the expense of adapting appropriately to the remaining evidence?

Questions for authors

Please carefully checking the weaknesses.

1 comment

We sincerely thank the reviewer for the careful evaluation, encouraging comments on our problem formulation and experimental design, and constructive suggestions. Your feedback helped us clarify the methodological contribution and strengthen the empirical evidence. We address the three concerns below.

  1. Methodological novelty beyond worst-case regularization. We agree that replacing an average penalty with a maximum, by itself, is not a new general optimization principle. Our contribution lies in its application to the jointly optimized all-source and source-excluded evidence-fusion solutions, with a distinct allocation of adaptation pressure. We have strengthened Section 3.2 and Appendix A.2 with a selective-coupling analysis. At an exact SESR optimum, strictly inactive exclusion branches retain their independent remaining-evidence optima under the stated assumptions. In contrast, AC generally applies coupling pressure to every branch with nonzero displacement. This establishes a structural difference in how the two objectives balance stability and adaptation, rather than claiming novelty for the maximum operator itself.
  2. Weight-space coupling versus direct prediction-space regularization. We agree that the original bound is one-sided and may be loose when candidate predictions are correlated. We now explicitly acknowledge this limitation in Section 3.3 and introduce DirectPS-L2 as a prediction-space comparator (Appendix C.6). DirectPS-L2 achieves lower prediction shift than SESR on all five tested supports of both IMDb and ChemProt. Its evidence-fitting regret is higher on IMDb but lower on ChemProt, illustrating that prediction stability and evidence fit need not favor the same method. We therefore present weight-space coupling as a principled surrogate with a different optimization structure, not as universally superior to direct prediction-space coupling.
  3. Stability versus adaptation to remaining evidence. We appreciate this important distinction. We added same-anchor Frozen-SESR and Frozen-IR controls and measured remaining-evidence regret separately from prediction shift and gold-label utility (Section 4.3; Appendix C.5). Under the original protocol, Frozen-SESR achieves zero prediction shift but incurs mean remaining-evidence regret of 0.00263826, compared with 0.00005843 for SESR. This shows that SESR does not obtain its stability simply by freezing all exclusion branches. Under high-accuracy optimization, SESR achieves lower mean regret than AC in 24/25 supports, although its maximum branch regret is higher in 14/25. We also report LOSO Accuracy, NLL, and Macro-F1 to distinguish evidence adaptation from downstream predictive utility. We thank the reviewer again. These suggestions substantially improved the precision of our methodological claims and our evaluation of the stability–adaptation trade-off.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.