- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Residual-robust Active Statistical Inference (RASI) for allocating labeling budgets when estimated prediction errors are unreliable. The method preserves target-specific loadings and places an uncertainty set around the residual-risk component, optimizing over the full class of feasible sampling policies. The paper derives a geometry-dependent limiting allocation, establishes asymptotic inference guarantees for batch policies, and evaluates the method on synthetic and real datasets.
Critical strengths
-
The paper addresses a relevant limitation of active statistical inference: inaccurate uncertainty estimates can lead to inefficient sampling even when inference remains asymptotically valid. Preserving target-specific information while protecting against residual-risk misspecification is a well-motivated principle, and its development into a practical sampling procedure constitutes a meaningful methodological contribution.
-
The empirical results provide encouraging evidence of the method's effectiveness. RASI improves efficiency over both plug-in sampling and oracle-tuned uniform mixtures in the reported applications, with additional sensitivity analyses supporting the stability of these gains.
Critical weaknesses
-
The treatment of target geometry needs further justification for nonlinear targets. For linear regression, the loading can be estimated using unlabeled covariates, making its separation from residual-risk uncertainty particularly natural. However, logistic-regression and log-odds-ratio loadings also depend on unknown outcome-related quantities estimated from the pilot. These estimates may be unstable with small pilots or rare outcomes. Appendix A.5 allows the fitted design loading to differ from the true loading, which supports inference validity but does not establish efficiency robustness to such errors. A clearer discussion of when geometry can reasonably be treated as reliable would strengthen the extension beyond linear regression.
-
The experiments could better identify the source of the gains. RASI changes both the uncertainty set and the policy search space relative to the mixture baselines. The current comparisons therefore do not fully distinguish the benefit of residual-only robustification from that of a more flexible allocation. A particularly informative comparison would replace the uniform endpoint with the geometry-only allocation and retain a simple mixing strategy. This would help establish how much benefit the full optimization provides beyond choosing a better fallback.
Questions for authors
-
Can the existing experiments or diagnostics clarify its sensitivity to pilot estimation error?
-
Beyond computational simplicity, what motivates choosing Equation (4) over the alternative objective in Appendix A.2? Are there identifiable settings where the resulting policies differ substantially?
-
The related-work discussion could be more complete regarding recent progress on active statistical inference, such as balanced operation [1], application to U-statistics [2] and sequential adaptation [3].
[1] Chen, B., Zhou, Z., Peng, L., & Wang, Z. (2025). Balanced active inference. NeurIPS
[2] Wang, X., Huo, Y., Peng, L., & Zou, C. (2026). Learning U-Statistics with Active Inference. ICML.
[3] Sfyraki, M.-E., & Wang, J.-K. (2026). Revisiting Active Sequential Prediction-Powered Mean Estimation. ICLR
4 comments
Follow-up on Our Rebuttal and Manuscript Revisions
Dear Reviewer CPAN,
We sincerely thank you for your careful review and thoughtful feedback.
As the discussion period is coming to the end soon, we are eager to hear your feedbacks on our rebuttal, where we have provided a point-to-point response. We have also revised the manuscript accordingly. We are more than willing to clarify or discuss more if needed.
Thank you again!
Best,
Authors
Thank you for the detailed response, which addresses most of my concern. Therefore I keep my positive evaluation.
Dear Reviewer CPAN,
Thank you for reviewing our rebuttal and for your positive assessment. We appreciate your constructive feedback, which has helped us improve our manuscript.
Best regards,
The authors
Response to Reviewer CPAN
Thank you for your careful review and positive assessment of our work. We appreciate your constructive feedback and address your comments and questions below.
We agree that, for nonlinear targets, errors in estimating the target geometry can lead to wider or less stable confidence intervals, even when asymptotic validity is preserved under our theoretical assumptions.
To investigate this sensitivity, we added an experiment on CheXpert (Appendix D.2), where the target is a log-odds ratio. We fix the residual proxy and ambiguity radius and vary only the pilot sample size used to estimate the target loading. With 100 pilot samples, the confidence interval is 5.9% wider than with a more accurately estimated loading; with 300 pilot samples, this difference decreases to 0.52%.
These results suggest a small efficiency cost from loading estimation in this CheXpert experiment. More generally, estimating the target loading often reduces to estimating population quantities, such as group means, which can be easier than estimating a residual function. A modest pilot sample may therefore suffice to obtain an accurate loading estimate, although the required sample size will depend on the target and application.
We added comparisons with both arithmetic and geometric mixtures between the plug-in policy and geometry-aware endpoints, including Λ and the large-ambiguity RASI endpoint Λ2/3 (Appendix C).
In the synthetic experiment, the best displayed arithmetic mixture toward Λ2/3 has mean interval width 0.148156, compared with 0.147851 for RASI—a difference of approximately 0.21%. This mixing weight is selected in hindsight from the evaluated grid. In the Census experiment, where mixing weights are selected using pilot data, arithmetic mixing toward Λ2/3 also closely matches RASI and yields slightly narrower intervals at lower budgets.
These results suggest that the geometry-dependent allocation accounts for much of the observed gain. The robust optimization formulation provides a principled derivation of the sampling policy and, in particular, identifies the effective Λ2/3 endpoint. Beyond the choice of endpoint, mixing methods require a choice of mixing rule, such as geometric or arithmetic mixing, which can also affect performance. RASI determines the allocation path through robust optimization, reducing the number of design choices required when applying active inference.
Yes. The new CheXpert experiment described above isolates sensitivity to loading estimation by fixing the residual proxy and ambiguity radius while varying the pilot sample size. The increase in interval width falls from 5.9% with 100 pilot samples to 0.52% with 300 pilot samples, relative to the more accurately estimated loading. This provides direct evidence of limited sensitivity to loading estimation error in this setting.
Our main reasons are computational simplicity and the availability of an explicit fallback policy in the large-ambiguity regime. For a fixed residual-risk function, the two objectives differ by a policy-independent term and therefore yield the same optimal policy. However, this equivalence does not generally hold after taking the worst case over the ambiguity set, so the two robust formulations can produce different sampling probabilities.
To assess the practical effect, Table 1 compares the two formulations with separately selected radii. At a 20% evaluation-label budget, their mean 90% confidence-interval widths are 0.1485 versus 0.1484 for synthetic regression, 179.44 versus 179.15 for Census, and 0.3021 versus 0.3017 for CheXpert, respectively. The widths are similarly close across all reported budgets. Thus, although the formulations are mathematically distinct, their empirical efficiency is very similar in these experiments.
Thank you for pointing this out. We have incorporated them into the revised manuscript (Appendix F).