- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how stable a mixture of fixed candidate models remains when one source of evidence is removed. It proposes SESR, which learns the weights using all sources together with the weights obtained after removing each source. It also adds a penalty for the largest change in weights. The paper gives a bound that links weight changes to prediction changes. In experiments on five weak supervision datasets, SESR is more stable than IR and AC when using 35 optimization steps, while accuracy is usually similar. However, the more complete optimization results in the appendix show mixed results compared with AC.
Critical strengths
-
The problem is clearly defined, and the bound is correct when the candidate models are kept fixed.
-
SESR consistently improves the reported stability measure over IR and AC.
-
The paper includes some useful analyses and acknowledges cases where other methods perform better.
Critical weaknesses
-
SESR keeps the excluded-source weights close to the original weights, but this may limit how well they fit the remaining evidence. Simply keeping all weights unchanged would give zero prediction shift. It would be useful to compare with this baseline and report how much evidence fit is lost after exclusion.
-
AC selects the largest tested (gamma=10) in all 25 cases, so a wider search range seems necessary. Comparing the methods at similar evidence-fit costs would also be fairer. The 35 optimization iterations seem limited, and I think that convergence results would help .
-
The main experiments use fixed classifiers trained with gold labels. The appendix includes different candidate models on three datasets, but SESR beats AC in only 6 of 15 cases. It is still unclear whether the results apply to other kinds of models or to the scientific uses discussed in the introduction.
-
The theoretical bound follows from standard norm inequalities and does not show why SESR should outperform other coupling methods. The paper could explain its methodological novelty more clearly.
-
Some figures and diagnostic labels need clarification. Internal labels from Figure 2 are difficult to read at page size. Table 2 should give the calculation for “Kish (%)”, and Figure 2 should define “evidence mass” precisely.
Questions for authors
-
How much fit to the remaining evidence is lost with SESR compared with AC and fixed weights?
-
Does SESR still outperform AC with a larger range and sufficient optimization convergence?
-
Which exact versions of the datasets were used, and can the authors provide the full results?
We sincerely thank the reviewer for the detailed, constructive feedback and for recognizing the clarity of our problem definition, the correctness of the fixed-support bound, and the value of the additional analyses. Your comments helped us improve the experimental controls, reproducibility, and presentation. We address each concern below.
Frozen weights and remaining-evidence adaptation. We agree that zero prediction shift alone is insufficient evidence of useful robustness. We added Frozen-IR and same-anchor Frozen-SESR baselines (Section 4.3; Appendix C.5). Under the original protocol, Frozen-SESR has zero weight and prediction shift but incurs mean remaining-evidence regret of 0.00263826, versus 0.00005843 for SESR and 0.00002775 for AC. Under high-accuracy optimization, SESR achieves lower mean regret than AC in 24/25 supports, although its maximum regret is higher in 14/25. We also report LOSO Accuracy, NLL, and Macro-F1, and explicitly distinguish evidence-fitting regret from gold-label utility.
Larger γ range, convergence, and comparable evidence-fit costs. We agree. We added high-accuracy optimization, γ expansion to {30, 100, 300}, and matched-support results (Appendices B.6 and C.4). The original 35-step SESR advantage over AC is 25/25; under high-accuracy optimization with reselection it becomes 14/25. The extended γ range does not change the selected high-accuracy outcomes. We explicitly acknowledge this protocol dependence. We also added comparisons at approximately matched nontrivial evidence-fit costs (Appendix C.8). Such matches are available for 14/25 supports on the original seven-point grid and 25/25 on the extended grid, with both favorable and unfavorable outcomes. We do not claim dominance at exactly equal costs.
Candidate heterogeneity and scientific applicability. We agree that the original 24 candidates, constructed from one logistic-SGD pipeline, provide a limited test of generality. We added 15 heterogeneous candidate banks across IMDb, Spouse, and ChemProt, containing 360 candidates from logistic regression, linear SVM, and Naive Bayes (Appendices B.7 and C.7). SESR improves prediction stability over IR and mean remaining-evidence regret over same-anchor Frozen-SESR in all 15 supports, but outperforms AC in only 6/15. We also acknowledge that the models share text features and therefore do not establish broad architectural generalization. IDP ensemble reweighting remains a motivating application; we do not claim direct NMR/SAXS validation.
Methodological novelty and theoretical bound. We agree that the weight-to-prediction bound follows standard norm inequalities and does not establish SESR's superiority over other coupling methods. We clarified this limitation in Section 3.3 and added the selective-coupling KKT result in Appendix A.2. Under the stated assumptions, strictly inactive SESR branches retain their independent remaining-evidence optima at exact optimality, whereas AC generally exerts coupling pressure on all nonzero-displacement branches. This provides a more precise characterization of our methodological contribution.
Figure readability and metric definitions. We revised Figure 2 to improve its organization and source-specific labels. Section 4.3 now gives the explicit Kish (%) calculation, and Appendix D defines vote share, row coverage, and normalized evidence mass.
Dataset versions and numerical reproducibility. We expanded Appendix B with dataset statistics, candidate construction, random seeds, optimization settings, source-loss normalization, and selection rules. The local WRENCH snapshots could not be uniquely matched to an immutable public repository commit, and we now explicitly disclose this limitation rather than claiming an exact version that we cannot verify. We have also organized the numerical results and reproducibility code, including per-support comparisons and the additional optimization protocols. Reproduction from frozen inputs is distinguished from full end-to-end reconstruction, which remains incomplete. We sincerely appreciate these suggestions, which helped us make the paper more transparent about both its contributions and its limitations.