- Rating
- 1(−1)
- Confidence
- 1(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1(−1)
Read review
Paper summary
This paper studies how to recover the population positive-label rate from multiple models that may suffer from selection bias, under the constraint that training data and selection logs are unavailable but each model's training data sources (source records) are known. The core result is a necessary and sufficient condition: consistent recovery is possible if and only if a separation margin κ_s(W) > 0, in which case the minimax MSE is Θ(m^{-1}); otherwise it is bounded away from zero. The authors further extend the framework to a cross-task setting with shared selection and provide a criterion checkable before model acquisition. Validation is conducted on simulated configurations and on HelpSteer3 human feedback.
I am open to raising my score if the authors can address the concerns above in the rebuttal, particularly by (a) positioning the method against an existing selection-bias correction where such a comparison is possible, and (b) providing analysis or discussion on the robustness of the method when the exponential selection form holds only approximately. The theoretical core of the paper is solid; the main barrier to acceptance is the empirical and practical validation gap.
Critical strengths
-
Complete and rigorous theoretical framework: From single-task to cross-task, from mean identifiability to full-distribution indistinguishability, from necessity to sufficiency with matching bounds, the paper provides a thorough theoretical analysis. The necessary-and-sufficient nature of the κ_s(W) condition gives a clean, actionable criterion: one can determine whether consistent recovery is possible by inspecting only the source records matrix. When κ_s = 0, the proof that even the full observation distributions are indistinguishable—not just the means—is technically careful and rules out the possibility that more sophisticated estimators could bypass the condition.
-
Effective translation from theory to practice: Theorem 2 reduces the recovery condition to a concise graph-theoretic property—whether an ambiguous pair exists—and provides a criterion (Equation 9) checkable before model acquisition. This transformation from an algebraic condition into an operational decision rule is a meaningful contribution. The cross-task shared-selection setting is well-motivated by real data curation pipelines. The design of the experiments also follows the theory rather than decorating it: ten of the twelve pre-specified configurations start at κ² = 0 and are pushed above zero by a single acquired model, and the comparison of one estimator under random versus margin-based acquisition (0.89 → 0.64 → 0.20 RMSE) isolates the acquisition rule as the cause of the improvement.
Critical weaknesses
-
Insufficient empirical validation: The real-data experiments are conducted on a single public benchmark (HelpSteer3), with each source fitted on m = 128 retained records—small relative to the m ≥ 512 regime where the linear label budget is established, so the real-data result tests the pipeline more than the asymptotics. The authors acknowledge the proof-of-concept nature of these experiments, but for a paper that claims to address a practical problem in model reuse, the empirical support is not yet sufficient. Comparisons against standard methods for handling selection bias are absent, and evaluation on further public benchmarks is missing. Additionally, the magnitude of constant factors (κ_s(W)^{-1}) in realistic configurations is not discussed, leaving the practical feasibility unclear.
-
Robustness is discussed for one assumption but not the others: The treatment of source-record errors is more concrete than a bare existence claim—with at most r models whose records list incorrect sources, the parametric rate survives if separation holds after every deletion of up to 2r model rows—but that is a tolerance threshold rather than a degradation rate, and the exponential selection form has no analogous treatment at all. Since the title concerns when bias is correctable, the regime where the assumption holds approximately is the one a practitioner actually occupies, and no bound is given for how the excess MSE grows under a small deviation.
Questions for authors
-
When the selection mechanism deviates from the exact exponential form, can the κ_s(W) framework still be used, perhaps through approximation or error bounds? Even without exact recovery, can an error bound be provided?
-
In your experiments, could you compare against at least one standard method for selection bias? Even if the dataset does not perfectly satisfy those methods' assumptions, such a comparison would help readers understand the relative advantages of your approach.
-
For the specific configuration in Figure 1, what is the approximate magnitude of the constant factor involving κ_s(W)^{-1} in the final MSE for different per-source sample sizes m? Could you also report configurations with κ² small but positive, so that the empirical RMSE can be plotted against κ_s(W)^{-1}?
6 comments
Response to mdVd (2/4): Robustness of selection assumptions
Concern 2: Robustness of the selection assumptions and inaccurate source records (condition (b); Q1, W2)
providing analysis or discussion on the robustness of the method when the exponential selection form holds only approximately.
When the selection mechanism deviates from the exact exponential form, can the framework still be used, perhaps through approximation or error bounds?
Response: We have given error bounds that depend explicitly on the size of deviations in the selection assumptions and source records. These bounds provide guarantees even when a fixed deviation prevents the error from vanishing with more data. We address the selection assumptions here and source-record errors in Part 3.
1. Robustness of the selection assumptions (condition (b); Q1, W2).
Regarding the case where “the exponential selection form holds only approximately,” we would like to clarify that this representation is exact for binary label-dependent retention with positive retention probabilities. Specifically, if positive and negative labels are retained with probabilities , then
=\frac{\theta e^b}{1-\theta+\theta e^b}, \qquad b=\log\frac{r_1}{r_0}.$$ Thus, the exponential form does not impose an additional restriction on these retention probabilities, and the single-task estimator does not need to know $r_1,r_0$, or $b$. We have clarified this in Section 2. The robustness concern remains: source bias may not be exactly sparse, and selection strengths may not be identical across tasks. Below, we relax these two assumptions separately and quantify how the resulting deviations affect estimation error. *Small biases beyond the sparse support.* We allow up to $s$ large source effects and an additional bias of at most $\delta$ in every source: $$q=\theta\mathbf{1}+e+u,\qquad \lVert e\rVert_0\le s,\qquad \lVert u\rVert_\infty\le\delta.$$ With known mixture weights $W$, $\kappa_s(W)>0$, and $\mathbb{E}[V]=Wq$, the sparse least-squares estimator satisfies $$\mathbb{E}\bigl[(\widehat{\theta}-\theta)^2\bigr] \le\frac{4}{\kappa_s(W)^2} \left(\sum_{i=1}^{L}\operatorname{Var}(V_i)+L\delta^2\right),$$ where $L$ is the number of observed models. Thus, the bound separates observation variance from a quadratic deviation term; a smaller margin amplifies both. The estimator does not need to know $\delta$. Under the sampling models in Appendix G.1, for a fixed configuration and sufficiently small $\delta$, the minimax risk is $\Theta(m^{-1}+n^{-1}+\delta^2)$. Here $m$ is the per-source training size and $n$ the student-label budget, with $n^{-1}=0$ for exact model observations. The $\delta^2$ term therefore captures an unavoidable error contribution in this setting, not just an artifact of the upper bound. The numerical check in Table E.15 tests this additional source bias. At $m=32{,}768$, the largest RMSE over 64 perturbation directions is below $\delta$ for margins 0.40 and 0.63 at each tested $\delta\in\{0.25,0.5,1\}$ pp. At $\delta=2$ pp, the largest RMSE is 2.3–3.9 pp. Separately, the human-feedback experiment gives RMSE of 2.18 pp under input-and-label-dependent selection and 2.22 pp under label-dependent selection. *Selection that is not exactly shared across tasks.* We also allow the same biased source to have different selection strengths across tasks. For the collection $O\cup G$ in Figure 1, with one common positively biased source, suppose $$\lvert b^{(2)}-b^{(1)}\rvert\le\varepsilon.$$ Within the parameter range of Theorem G.1, the minimax risk is $$\mathcal{R}_{O\cup G}^{\mathrm{drift}}(m,n;\varepsilon) =\Theta\!\left(m^{-1}+n^{-1}+\varepsilon^2\right).$$ Thus, small cross-task differences contribute an $\varepsilon^2$ term for this configuration (Appendix G.3). This result does not cover arbitrary cross-task designs. Appendix F.1 separately extends the cross-task characterization to known monotone selection channels under its stated conditions.Response to mdVd (3/4): Inaccurate source records
Concern 2 (continued): Inaccurate source records (W2)
The treatment of source-record errors is more concrete than a bare existence claim […] but that is a tolerance threshold rather than a degradation rate […]
Response: The existing result specifies when recovery survives incorrect source records. The new bound instead quantifies how errors in the resulting mixture weights affect estimation.
Let be the recorded weights and the true weights, both nonnegative with rows summing to one. For , , the row-sum condition gives . The mismatch in the observation mean is therefore . With and , the estimator using the recorded satisfies
\le\frac{4}{\kappa_s(W)^2} \left(\sum_{i=1}^{L}\operatorname{Var}(V_i)+\lVert\Delta e\rVert_2^2\right).$$ The bound now varies explicitly with the weight error and source bias, rather than only specifying a tolerance condition (Proposition G.3, Appendix G.5). The margin uses the recorded $W$, while the variances use the true $W'$. Only errors in biased-source columns contribute to the mean-mismatch term; other weight errors may still affect the variances. This is an MSE upper bound, not an exact expression for the increase in MSE. All these results retain the source-mixture observation model. For learned outputs, fitting and distillation errors must also be accounted for through the observation-error terms in Section 3.2.
Response to mdVd (1/4): Standard methods and label budgets
Thank you for specifying priorities (a) and (b). We address condition (a) with comparisons against soft BBSE and MLLS with BCTS (Concern 1; Appendix E.6, Table E.9). For condition (b), Concern 2 (Part 2) shows that the exponential form is not itself restrictive for binary label-dependent retention and gives error bounds for the deviations that remain: small bias in additional sources and cross-task drift (Section 2; Appendices G.1 and G.3). The remaining W1 points on constants, training size, and benchmark coverage are addressed with Q3 in Concern 3 (Part 4).
Concern 1: Comparisons with standard methods (condition (a); Q2, W1)
Response: We have expanded the comparison to include two groups of methods:
Comparison and results. The nine-source HelpSteer3 comparison uses m=128 and 128 new training replicates. Each replicate randomly resplits evaluation contexts mostly reused from earlier analyses. The four biased conditions receive equal weight and both primary comparisons were fixed before execution. The table reports rate RMSE in percentage points (pp); the information requirements are listed separately because the methods use different inputs.
Source correction achieves 2.20 pp rate RMSE without additional unbiased labels, compared with 18.7 pp for Soft BBSE and 28.1 pp for MLLS with BCTS.
At the same 128-label budget, combining source correction with the label mean reduces RMSE from 4.45 pp for labels alone to 2.03 pp. This is also less than half the error of PPI++ (4.48 pp) and Platt scaling (4.46 pp). Adding labels improves source correction from 2.20 to 2.03 pp, showing that source records and unbiased labels can be used together.
Why the label-shift methods have larger errors. The predictions in this experiment carry little label information, making label-shift estimation unstable; the soft confusion matrices are nearly singular. Appendix E.6 provides the diagnostic checks and confidence intervals.