acceptodds
Under review as a conference paper at ICLR 2027

Robust Active Statistical Inference: Separate Geometry from Uncertainty

Abstract

Active statistical inference reduces labeling costs by combining inexpensive proxy labels with selectively queried ground-truth labels for a downstream inferential target. The sampling policy determines which labels to acquire based on the estimated error of the proxy labels. However, such policies can be sensitive to misspecification. When the error of proxy labels is severely underestimated in a region where it is in fact large, the resulting query probabilities can be small, substantially inflating the estimator's variance and confidence-interval width. To mitigate this issue, we introduce Residual-robust Active Statistical Inference (RASI), which exploits a structural decomposition of the optimal sampling policy into a target-specific loading determined by the geometry of the estimand and an unobserved residual-risk component approximated by an uncertainty proxy. RASI preserves the former and robustifies only the latter. We formulate policy design as a distributionally robust optimization problem over plausible residual risks and derive minimax sampling policies over an ambiguity set. Our analysis shows that mixing with uniform sampling, a common robustness strategy in active statistical inference, is generally not the appropriate robust fallback because it discards target-specific geometry. We extend RASI to smooth functions of parameters defined by outcome-affine estimating equations, covering linear and logistic regression coefficients and log-odds ratios. Across synthetic and real-data experiments, RASI protects against misspecification of the uncertainty proxy while retaining the efficiency gains of active sampling when the proxy is informative.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Active statistical inference, prediction-powered inference, distributionally robust optimization, optimal sampling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_CPANOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Residual-robust Active Statistical Inference (RASI) for allocating labeling budgets when estimated prediction errors are unreliable. The method preserves target-specific loadings and places an uncertainty set around the residual-risk component, optimizing over the full class of feasible sampling policies. The paper derives a geometry-dependent limiting allocation, establishes asymptotic inference guarantees for batch policies, and evaluates the method on synthetic and real datasets.

Critical strengths
  1. The paper addresses a relevant limitation of active statistical inference: inaccurate uncertainty estimates can lead to inefficient sampling even when inference remains asymptotically valid. Preserving target-specific information while protecting against residual-risk misspecification is a well-motivated principle, and its development into a practical sampling procedure constitutes a meaningful methodological contribution.

  2. The empirical results provide encouraging evidence of the method's effectiveness. RASI improves efficiency over both plug-in sampling and oracle-tuned uniform mixtures in the reported applications, with additional sensitivity analyses supporting the stability of these gains.

Critical weaknesses
  1. The treatment of target geometry needs further justification for nonlinear targets. For linear regression, the loading can be estimated using unlabeled covariates, making its separation from residual-risk uncertainty particularly natural. However, logistic-regression and log-odds-ratio loadings also depend on unknown outcome-related quantities estimated from the pilot. These estimates may be unstable with small pilots or rare outcomes. Appendix A.5 allows the fitted design loading to differ from the true loading, which supports inference validity but does not establish efficiency robustness to such errors. A clearer discussion of when geometry can reasonably be treated as reliable would strengthen the extension beyond linear regression.

  2. The experiments could better identify the source of the gains. RASI changes both the uncertainty set and the policy search space relative to the mixture baselines. The current comparisons therefore do not fully distinguish the benefit of residual-only robustification from that of a more flexible allocation. A particularly informative comparison would replace the uniform endpoint with the geometry-only allocation and retain a simple mixing strategy. This would help establish how much benefit the full optimization provides beyond choosing a better fallback.

Questions for authors
  1. Can the existing experiments or diagnostics clarify its sensitivity to pilot estimation error?

  2. Beyond computational simplicity, what motivates choosing Equation (4) over the alternative objective in Appendix A.2? Are there identifiable settings where the resulting policies differ substantially?

  3. The related-work discussion could be more complete regarding recent progress on active statistical inference, such as balanced operation [1], application to U-statistics [2] and sequential adaptation [3].

[1] Chen, B., Zhou, Z., Peng, L., & Wang, Z. (2025). Balanced active inference. NeurIPS

[2] Wang, X., Huo, Y., Peng, L., & Zou, C. (2026). Learning U-Statistics with Active Inference. ICML.

[3] Sfyraki, M.-E., & Wang, J.-K. (2026). Revisiting Active Sequential Prediction-Powered Mean Estimation. ICLR

4 comments
AuthorsOpenReview ↗

Response to Reviewer CPAN

Thank you for your careful review and positive assessment of our work. We appreciate your constructive feedback and address your comments and questions below.

The treatment of target geometry needs further justification for nonlinear targets.

We agree that, for nonlinear targets, errors in estimating the target geometry can lead to wider or less stable confidence intervals, even when asymptotic validity is preserved under our theoretical assumptions.

To investigate this sensitivity, we added an experiment on CheXpert (Appendix D.2), where the target is a log-odds ratio. We fix the residual proxy and ambiguity radius and vary only the pilot sample size used to estimate the target loading. With 100 pilot samples, the confidence interval is 5.9% wider than with a more accurately estimated loading; with 300 pilot samples, this difference decreases to 0.52%.

These results suggest a small efficiency cost from loading estimation in this CheXpert experiment. More generally, estimating the target loading often reduces to estimating population quantities, such as group means, which can be easier than estimating a residual function. A modest pilot sample may therefore suffice to obtain an accurate loading estimate, although the required sample size will depend on the target and application.

The experiments could better identify the source of the gains.

We added comparisons with both arithmetic and geometric mixtures between the plug-in policy and geometry-aware endpoints, including Λ\Lambda and the large-ambiguity RASI endpoint Λ2/3\Lambda^{2/3} (Appendix C).

In the synthetic experiment, the best displayed arithmetic mixture toward Λ2/3\Lambda^{2/3} has mean interval width 0.148156, compared with 0.147851 for RASI—a difference of approximately 0.21%. This mixing weight is selected in hindsight from the evaluated grid. In the Census experiment, where mixing weights are selected using pilot data, arithmetic mixing toward Λ2/3\Lambda^{2/3} also closely matches RASI and yields slightly narrower intervals at lower budgets.

These results suggest that the geometry-dependent allocation accounts for much of the observed gain. The robust optimization formulation provides a principled derivation of the sampling policy and, in particular, identifies the effective Λ2/3\Lambda^{2/3} endpoint. Beyond the choice of endpoint, mixing methods require a choice of mixing rule, such as geometric or arithmetic mixing, which can also affect performance. RASI determines the allocation path through robust optimization, reducing the number of design choices required when applying active inference.

Q1. Can the existing experiments or diagnostics clarify its sensitivity to pilot estimation error?

Yes. The new CheXpert experiment described above isolates sensitivity to loading estimation by fixing the residual proxy and ambiguity radius while varying the pilot sample size. The increase in interval width falls from 5.9% with 100 pilot samples to 0.52% with 300 pilot samples, relative to the more accurately estimated loading. This provides direct evidence of limited sensitivity to loading estimation error in this setting.

Q2. Beyond computational simplicity, what motivates choosing Equation (4) over the alternative objective in Appendix A.2? Are there identifiable settings where the resulting policies differ substantially?

Our main reasons are computational simplicity and the availability of an explicit fallback policy in the large-ambiguity regime. For a fixed residual-risk function, the two objectives differ by a policy-independent term and therefore yield the same optimal policy. However, this equivalence does not generally hold after taking the worst case over the ambiguity set, so the two robust formulations can produce different sampling probabilities.

To assess the practical effect, Table 1 compares the two formulations with separately selected radii. At a 20% evaluation-label budget, their mean 90% confidence-interval widths are 0.1485 versus 0.1484 for synthetic regression, 179.44 versus 179.15 for Census, and 0.3021 versus 0.3017 for CheXpert, respectively. The widths are similarly close across all reported budgets. Thus, although the formulations are mathematically distinct, their empirical efficiency is very similar in these experiments.

Q3. The related-work discussion could be more complete regarding recent progress on active statistical inference, such as balanced operation [1], application to U-statistics [2] and sequential adaptation [3]

Thank you for pointing this out. We have incorporated them into the revised manuscript (Appendix F).

AuthorsOpenReview ↗

Follow-up on Our Rebuttal and Manuscript Revisions

Dear Reviewer CPAN,

We sincerely thank you for your careful review and thoughtful feedback.

As the discussion period is coming to the end soon, we are eager to hear your feedbacks on our rebuttal, where we have provided a point-to-point response. We have also revised the manuscript accordingly. We are more than willing to clarify or discuss more if needed.

Thank you again!

Best,

Authors

Thank you for the detailed response, which addresses most of my concern. Therefore I keep my positive evaluation.

Dear Reviewer CPAN,

Thank you for reviewing our rebuttal and for your positive assessment. We appreciate your constructive feedback, which has helped us improve our manuscript.

Best regards,

The authors

Reviewer_YcWBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RASI, a robust active statistical inference method. The main insight is, as the title says, to separate the target-specific geometry from the uncertainty of proxy prediction errors, and only apply robust optimization to the latter. The authors derive a minimax sampling policy and extend it to different statistical targets. Experiments on synthetic and real datasets show improvements in statistical efficiency when the uncertainty proxy is misspecified.

Critical strengths
  1. This paper's motivation is clear and straightforward. I think the problem of unreliable uncertainty proxies in ASI is important.

  2. The overall method is well motivated, and the idea of keeping the target geometry while making the residual risk robust makes sense to me.

  3. The theoretical analysis is relatively solid, and the experimental results are quite encouraging. The paper also considers both linear and nonlinear statistical targets.

Critical weaknesses
  1. The authors optimize a slightly different objective from the actual additional variance. I understand this can help to make the optimization easier, but I am not sure how much this difference matters in practice.

  2. The method assumes that the target-specific geometry is relatively reliable, but this part also needs to be estimated for some targets. Also, the robustness radius is selected from pilot data without a guarantee that the true residual risk is covered.

  3. I think the current results are good, but it would be useful to better understand where the improvements come from. In particular, I would like to see whether a simple geometry-aware mixing strategy can achieve similar results, and how RASI behaves under different levels of proxy misspecification.

Questions for authors
  1. Why do you robustify E[Λv/π]E[\Lambda v/\pi] instead of the actual additional variance E[(1/π−1)Λv]E[(1/\pi-1)\Lambda v]? Would the two objectives lead to different results in practice?

  2. How sensitive is RASI when the target loading is not accurately estimated? I think this could be important for nonlinear targets with a small pilot set.

  3. Could the authors explain a bit more about the choice of the robustness radius? I am curious whether RASI would lose some efficiency when the uncertainty proxy is already quite accurate.

  4. Why not compare with a simple mixture between the plug-in policy and the geometry-only policy? This would help show the additional benefit of the full minimax optimization.

  5. Can you test different levels of proxy misspecification, from a good uncertainty proxy to a very inaccurate one? This would help us better understand the trade-off between efficiency and robustness.

5 comments
AuthorsOpenReview ↗

Response to Reviewer YcWB (Part 1/2)

Thank you for your careful review and positive assessment of our work. We appreciate your constructive feedback and address your questions below.

Q1. Why robustify the inverse-propensity objective rather than the actual additional variance? Do the two formulations differ in practice?

Our main reasons are computational simplicity and the availability of an explicit fallback policy in the large-ambiguity regime. For a fixed residual-risk function, the two objectives differ by a policy-independent term and therefore yield the same optimal policy. However, this equivalence does not generally hold after taking the worst case over the ambiguity set, so the two robust formulations can produce different sampling probabilities.

To assess the practical effect, Table 1 compares the two formulations with separately selected radii. At a 20% evaluation-label budget, their mean 90% confidence-interval widths are 0.1485 versus 0.1484 for synthetic regression, 179.44 versus 179.15 for Census, and 0.3021 versus 0.3017 for CheXpert, respectively. The widths are similarly close across all reported budgets. Thus, although the formulations are mathematically distinct, their empirical efficiency is very similar in these experiments.

Q2. How sensitive is RASI to errors in estimating the target loading, particularly for nonlinear targets with a small pilot set?

We agree that, for nonlinear targets, errors in estimating the target geometry can lead to wider or less stable confidence intervals, even when asymptotic validity is preserved under our theoretical assumptions.

To investigate this sensitivity, we added an experiment on CheXpert, where the target is a log-odds ratio. We fix the residual proxy and ambiguity radius and vary only the pilot sample size used to estimate the target loading. With 100 pilot samples, the confidence interval is 5.9% wider than with a more accurately estimated loading; with 300 pilot samples, this difference decreases to 0.52%.

These results suggest a modest efficiency cost from loading estimation in this CheXpert experiment. More generally, estimating the target loading often reduces to estimating population quantities, such as group means, which can be easier than estimating a residual function. A modest pilot sample may therefore suffice to obtain an accurate loading estimate, although the required sample size will depend on the target and application.

AuthorsOpenReview ↗

Response to Reviewer YcWB (Part 2/2)

Q3. How is the robustness radius selected, and can RASI lose efficiency when the uncertainty proxy is already accurate?

We select the radius using pilot data by minimizing an empirical estimate of the policy-dependent variance criterion. To reduce overfitting in residual-risk prediction, we use out-of-fold predictions from the residual model. Only these predictions are cross-fitted; the loading estimates, radius scale, and budget normalization use the full pilot. This is an empirical tuning procedure and does not guarantee that the ambiguity set contains the true residual-risk function.

When the proxy is accurate, a small radius keeps RASI close to the plug-in policy; at zero radius, the two coincide. A positive radius can incur an efficiency cost, so empirical tuning does not guarantee the absence of such a cost in every setting. The new ablation in Appendix D.1 examines this tradeoff directly, as discussed in our response to Q5.

We also added a theoretically certified alternative in Appendix E. Under boundedness assumptions, an independent calibration sample and a concentration inequality yield a radius whose ambiguity set contains the true residual-risk function with probability at least 1−η1-\eta. This guarantee can be conservative and may produce wider intervals, which motivates our use of empirical tuning in the main experiments.

Q4. Can a simple mixture between the plug-in and geometry-only policies achieve similar performance?

We added comparisons with both arithmetic and geometric mixtures between the plug-in policy and geometry-aware endpoints, including Λ\Lambda and the large-ambiguity RASI endpoint Λ2/3\Lambda^{2/3} (Appendix C).

In the synthetic experiment, the best displayed arithmetic mixture toward Λ2/3\Lambda^{2/3} has mean interval width 0.148156, compared with 0.147851 for RASI—a difference of approximately 0.21%. This mixing weight is selected in hindsight from the evaluated grid. In the Census experiment, where mixing weights are selected using pilot data, arithmetic mixing toward Λ2/3\Lambda^{2/3} also closely matches RASI and yields slightly narrower intervals at lower budgets.

These results suggest that the geometry-dependent allocation accounts for much of the observed gain. The robust optimization formulation provides a principled derivation of the sampling policy and, in particular, identifies the effective Λ2/3\Lambda^{2/3} endpoint. Beyond the choice of endpoint, mixing methods require a choice of mixing rule, such as geometric or arithmetic mixing, which can also affect performance. RASI determines the allocation path through robust optimization, reducing the number of design choices required when applying active inference.

Q5. How does RASI behave across different levels of proxy misspecification?

We added a controlled sensitivity experiment on Census (Appendix D.1), holding the fitted outcome predictor and target geometry fixed while varying the accuracy of the residual-risk input. We perturb the realized absolute residuals with Gaussian noise at six levels, then clip at zero and square the result.

At zero perturbation, the input equals the realized squared residual, the robustness radius is zero, and RASI coincides exactly with the plug-in policy. As the perturbation increases, RASI retains higher relative CI-width efficiency than the plug-in policy and both variance-oracle uniform-mixture benchmarks at every positive noise level considered.

This experiment illustrates the efficiency–robustness tradeoff: RASI recovers the plug-in allocation at the exact-input endpoint and protects efficiency as the input becomes less reliable.

Thank you again for your feedback. These comments have helped us clarify the method’s design choices and strengthen the empirical analysis.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal and Manuscript Revisions

Dear Reviewer YcWB,

We sincerely thank you for your careful review and thoughtful feedback.

As the discussion period is coming to the end soon, we are eager to hear your feedbacks on our rebuttal, where we have provided a point-to-point response. We have also revised the manuscript accordingly. We are more than willing to clarify or discuss more if needed.

Thank you again!

Best,

Authors

Reviewer_LJoAOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper extends the Active Statistical Inference that was developed by Zrnic (2024). ASI assumes we have budget on the number of labels that we can collect and we use ML model to identify which data points to label so that the budget is well utilised. ASI uses the cheap ML prediction for all observations and selects which true labels should be acquired. With this acquisition it corrects prediction errors by using the inverse-probability weights to estimate a target and construct asymptotically valid confidence intervals. The sampling rule prioritises observations whose prediction errors matter most for that target estimation and is based on the quantifying the target geometry and residual risk.

The authors argue that bluntly substituting an residual risk with estimated value can be problematic: if the estimate understates the prediction error in the important region, the policy acquires too few labels leading to to large variance and wide confidence interval. This is motivation for the authors to introduce their more robust approach.

The core idea is in my opinion in choosing a sampling policy that remains useful even when the estimated prediction-error risks are wrong. It does this by allowing several possible residual-risk functions, evaluating each policy against its worst case, and retaining the target geometry throughout.

The paper provides relevant experiments on the real datasets and provides ablation study for the linear regression problem.

Critical strengths
  • The paper addresses a relevant problem and provides promising ideas and direction.

  • It makes good point for active strategies that the labeling decisions should consider both how wrong a prediction might be and how much that error would affect the final result. Aims to identify and preserves the useful information while other estimates may be unreliable.

  • The method by its design can adjust individual sampling probabilities directly while keeping the overall budget. This allows to find allocations that cannot be reached by simply mixing an existing sampling rule with uniform sampling.

  • Provide demonstration on different tasks: the linear regression model, classifier, regression including theoretical assessments

Critical weaknesses
  • I found the writing flow and clarity of the paper rather poor. While authors present interesting research direction and tackle relevant problem, I would appreciate if the terms are well defined, the Fig 1. should clearly demonstrate the problem that the authors try to solve, the caption below the figure should be succinct enough to clearly understand all details of the figure, figure should be placed close to the relevant section. Figure 2. would be more relevant to use.

  • The core difference is between relatively reliable target geometry, �0��0�, and unreliable residual risk, �1��1�. This is well motivated for linear regression, where the geometry depends on covariates observed for everyone. For the nonlinear extensions, however, the geometry also depends on unknown outcome quantities. The paper does not treat this problem which I believe is crucial for the performance of the method.

  • There should be stronger statistical justification for why the chosen worst-case cost represents the robust approach when the geometry gain result concerns that particular cost; it does not establish lower actual variance for every plausible residual-risk function.

  • Experimental design: the paper argues that moving toward uniform sampling discards useful geometry. Its main comparisons then use mixture methods whose endpoint is uniform sampling. This does not separate the benefit of retaining geometry from the benefit of the particular ambiguity set and full optimisation.

Questions for authors
  1. Can you provide the numerically study of the minimax policy for the actual added variance (E[(1/π−1)Λv])(\mathbb E[(1/\pi-1)\Lambda v]), and characterise when and how it differs from the policy produced by the proposed surrogate objective?

  2. What formal robustness statement can be attached to the pilot-selected radius? Please consider ambiguity-set coverage, an oracle/regret guarantee, or nested pilot tuning evaluated on independent query realizations, including all calibration labels in the total budget.

  3. How does RASI behave under misspecification of (Λ)(\Lambda), active propensity clipping, and nonlinear-estimator boundary failures?

  4. Can you elaborate more on the robustness statement can be attached to the pilot-selected radius?

3 comments
AuthorsOpenReview ↗

Response to Reviewer LJoA (Part 1/2)

Thank you for your careful review and thoughtful feedback on our work. We address your comments and questions below.

Weakness 1

The paper begins with linear regression as a motivating example, develops the general framework, and then addresses implementation details such as radius selection. We agree that the original Figure 1 contained too much information for the introduction. In the revision, we moved the original Figure 2 into the introduction as the new Figure 1, where it illustrates the problem and the benefit of preserving target geometry more directly.

Weakness 2

We agree that, for nonlinear targets, errors in estimating the target geometry can lead to wider or less stable confidence intervals, even when asymptotic validity is preserved under our theoretical assumptions.

To investigate this sensitivity, we added an experiment on CheXpert, where the target is a log-odds ratio. We fix the residual proxy and ambiguity radius and vary only the pilot sample size used to estimate the target loading. With 100 pilot samples, the confidence interval is 5.9% wider than with a more accurately estimated loading; with 300 pilot samples, this difference decreases to 0.52%.

These results suggest a modest efficiency cost from loading estimation in this CheXpert experiment. More generally, estimating the target loading often reduces to estimating population quantities, such as group means, which can be easier than estimating a residual function. A modest pilot sample may therefore suffice to obtain an accurate loading estimate, although the required sample size will depend on the target and application.

Weakness 3

In Proposition 3.1 (geometry gain of target-aware fallback), we characterize the large-ambiguity gain of π∞\pi_\infty over uniform sampling, evaluating each policy under its own worst-case residual-risk function. This worst-case comparison is the minimax notion of robustness adopted in our work.

Different plausible residual-risk functions can favor different sampling sample policies. A minimax policy therefore protects against worst-case risk while allowing performance tradeoffs across individual residual-risk functions. It need not achieve lower actual variance for every plausible residual-risk function. Accordingly, Proposition 3.1 establishes a gain under the stated worst-case criterion, without asserting uniform variance dominance.

Weakness 4

We added comparisons with both arithmetic and geometric mixtures between the plug-in policy and geometry-aware endpoints, including Λ\Lambda and the large-ambiguity RASI endpoint Λ2/3\Lambda^{2/3} (Appendix C).

In the synthetic experiment, the best displayed arithmetic mixture toward Λ2/3\Lambda^{2/3} has mean interval width 0.148156, compared with 0.147851 for RASI—a difference of approximately 0.21%. This mixing weight is selected in hindsight from the evaluated grid. In the Census experiment, where mixing weights are selected using pilot data, arithmetic mixing toward Λ2/3\Lambda^{2/3} also closely matches RASI and yields slightly narrower intervals at lower budgets.

These results suggest that the geometry-dependent allocation accounts for much of the observed gain. The robust optimization formulation provides a principled derivation of the sampling policy and, in particular, identifies the effective Λ2/3\Lambda^{2/3} endpoint. Beyond the choice of endpoint, mixing methods require a choice of mixing rule, such as geometric or arithmetic mixing, which can also affect performance. RASI determines the allocation path through robust optimization, reducing the number of design choices required when applying active inference.

AuthorsOpenReview ↗

Response to Reviewer LJoA (Part 2/2)

Question 1

Our main reasons are computational simplicity and the availability of an explicit fallback policy in the large-ambiguity regime. For a fixed residual-risk function, the two objectives differ by a policy-independent term and therefore yield the same optimal policy. However, this equivalence does not generally hold after taking the worst case over the ambiguity set, so the two robust formulations can produce different sampling probabilities.

To assess the practical effect, Table 1 compares the two formulations with separately selected radii. At a 20% evaluation-label budget, their mean 90% confidence-interval widths are 0.1485 versus 0.1484 for synthetic regression, 179.44 versus 179.15 for Census, and 0.3021 versus 0.3017 for CheXpert, respectively. The widths are similarly close across all reported budgets. Thus, although the formulations are mathematically distinct, their empirical efficiency is very similar in these experiments.

We therefore distinguish mathematical equivalence from empirical similarity: our chosen objective provides a tractable robust formulation with an explicit geometry-dependent fallback, but does not generally minimize worst-case added variance. The experiments support similar interval-width performance in the evaluated settings.

Question 2

We select the radius using pilot data by minimizing an empirical estimate of the policy-dependent variance criterion. To reduce overfitting in residual-risk prediction, we use out-of-fold predictions from the residual model. Only these predictions are cross-fitted; the loading estimates, radius scale, and budget normalization use the full pilot. This is an empirical tuning procedure and does not guarantee that the ambiguity set contains the true residual-risk function.

When the proxy is accurate, a small radius keeps RASI close to the plug-in policy; at zero radius, the two coincide. A positive radius can incur an efficiency cost, so empirical tuning does not guarantee the absence of such a cost in every setting. The new ablation in Appendix D.1 examines this tradeoff directly.

We also added a theoretically certified alternative in Appendix E. Under boundedness assumptions, an independent calibration sample and a concentration inequality yield a radius whose ambiguity set contains the true residual-risk function with probability at least 1−η1-\eta. On this event, the robust criterion upper-bounds the true inverse-propensity cost for every feasible policy. This containment guarantee applies to the separately calibrated radius. It can be conservative and may produce wider intervals, which motivates our use of empirical tuning in the main experiments.

Question 3

For misspecification of Λ\Lambda, please refer to our previous response to Weakness 2

For propensity clipping, the Census and CheXpert experiments enforce query probabilities in [0.01,1][0.01,1], while the main synthetic experiment uses [0.005,1][0.005,1]. The bounds are enforced during policy optimization. At an expected labeling budget of 5%, setting the lower bound to 0.05 would force every feasible policy to assign probability 0.05 to every observation, eliminating policy differences. We therefore use a smaller positive floor.

For nonlinear-estimator boundary failures, the log-odds-ratio estimator requires both evaluation groups to be nonempty and their corrected group means to lie strictly inside (0,1)(0,1). In the new CheXpert geometry-sensitivity experiment, we observed no invalid final intervals among 600,000 unique simulated intervals across methods, budgets, and geometry-pilot sizes. Our evaluation procedure checks this condition explicitly and counts invalid intervals as noncoverage.

Pilot boundaries are a separate issue. Four of the 100 geometry pilots of size 100 had no events in the younger age group, leading to the fixed prevalence-clipping rule used for policy construction. No larger geometry pilot required this clipping. The final corrected group means are not clipped.

Question 4

Please see response to Question 2 above. The empirical pilot-selected radius is used to tune efficiency. Appendix E provides a separate calibration procedure with a high-probability ambiguity-set containment guarantee under its stated assumptions. Asymptotic validity of the resulting inference follows under the regularity conditions in our inference theorem and does not require the empirical radius to satisfy that containment guarantee.

Thank you again for your feedback, which has helped us clarify the theoretical claims and strengthen the empirical analysis. We would be grateful if you reconsidered your score in light of these points.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal and Manuscript Revisions

Dear Reviewer LJoA,

We sincerely thank you for your careful review and thoughtful feedback.

As the discussion period is coming to the end soon, we are eager to hear your feedbacks on our rebuttal, where we have provided a point-to-point response. We have also revised the manuscript accordingly. We are more than willing to clarify or discuss more if needed.

Thank you again!

Best,

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised Manuscript and Summary of Changes

We thank the reviewers for their careful reading and constructive feedback. We have revised the manuscript to incorporate this feedback. All additions and revisions are marked in blue in the updated manuscript. The main updates are:

  1. Comparison of the two robust objectives (Appendix A.2, Table 1). We added numerical comparisons between minimizing the worst-case inverse-propensity cost and the worst-case added variance, with separately selected radii. The two formulations yield very similar confidence-interval widths across the reported experiments and budgets.

  2. Geometry-aware mixing baselines (Appendix C). We added arithmetic and geometric mixtures toward both Λ\Lambda and Λ2/3\Lambda^{2/3} endpoints. These comparisons show that preserving target geometry accounts for much of the observed improvement over uniform mixing, with arithmetic mixing toward the Λ2/3\Lambda^{2/3} closely matching RASI in the studied settings.

  3. Additional sensitivity experiments (Appendices D.1–D.2). We added a controlled residual-risk perturbation experiment on Census to examine sensitivity to residual-risk misspecification while holding the predictor and geometry fixed. We also added a CheXpert experiment that isolates loading-estimation error by varying the geometry-pilot size while holding the residual proxy and ambiguity radius fixed.

  4. A certified alternative for radius selection (Appendix E). We added an independent calibration procedure that provides high-probability ambiguity-set containment under stated boundedness assumptions. We distinguish this guarantee from the empirical radius-selection procedure used in the main experiments.

  5. Expanded related work (Appendix F). We added discussion of balanced active inference, active inference for U-statistics, and sequential prediction-powered inference.

  6. Improved presentation and clarified scope (Introduction and Remark 3.2). We moved the synthetic illustration to the Introduction and clarified the distinction between gains from preserving target geometry and the additional gains from full optimization.

We appreciate the reviewers’ suggestions, which have helped us strengthen the empirical analysis and make the scope of our claims more precise.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.