- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes OptAPI, a framework combining active sampling and prediction-powered inference under a limited labeling budget. The authors introduce a D-optimality criterion to guide sampling and power tuning, derive optimality conditions, and develop a two-stage algorithm. Theoretical results and experiments are provided to support the proposed approach.
Critical strengths
-
The paper addresses a practically relevant problem, particularly when obtaining reliable labels is expensive.
-
D-optimality is a reasonable choice for multivariate inference, as it accounts for the joint covariance structure.
-
The paper is generally well organized and includes theoretical analysis, simulations, and real-data experiments.
Critical weaknesses
-
The methodological novelty is somewhat limited. The estimator largely follows the existing active inference and PPI framework, with D-optimality being the main extension. It is not entirely clear what new methodological insights are obtained beyond adapting a classical optimal design criterion.
-
The theoretical optimality claims appear stronger than the results established. In particular, the conclusion states that the estimator attains the minimal asymptotic variance, while the main theoretical results characterize optimality under fixed values of the other design parameter.
-
The empirical advantages are not fully convincing. The multivariate mean experiment is constructed to highlight scale disparities, and the main efficiency metric is directly aligned with the D-optimality objective. The evidence for broader improvements is less clear.
Questions for authors
-
In Section 3.3, the sampling policy is constructed with , although Theorem 1 shows that the optimal policy depends on . How sensitive is the final estimator to this initial choice? For example, would using or a pilot-based estimate lead to substantially different results?
-
Theorem 2 considers unrestricted optimization over , whereas the implementation restricts to . What motivates this choice? Could it lead to efficiency loss when the optimal lies outside this interval?
-
In Section 3.3, the estimated covariance matrix combines a pilot covariance with an active-sampling correction. Is this matrix always positive semidefinite? If not, how is the log-determinant optimization handled when adding a small is insufficient?
-
The paper motivates D-optimality through joint confidence ellipsoids, while Theorem 4 establishes marginal confidence intervals. Can the authors clarify the theoretical justification for the joint confidence regions evaluated in Section 4?
4 comments
Response to Reviewer ySZx (2/2, Q1 - Q4)
Q1: Sensitivity to
Thank you for the insightful comment. We would like to clarify that the final estimator is not sensitive to . The initial power only shapes the design, and Stage II tunes again on the acquired labels (Section 3.3).
Appendix G.6 varies , including , and efficiency and coverage are almost the same for all values (Figure A5). A pilot-based estimate restricted to is also in this range, so we expect the same result. We use as the default because it matches active inference (Zrnic and Candès, 2024), where the Stage I residual is the prediction error (see also our response to Reviewer ALYQ, W3).
Q2: Why restrict to ?
Thank you for this important question. Validity does not require the restriction, but convexity does.
The estimating equation, Eq. (3.2), is unbiased for every , so Theorems 3–4 hold for any limit of . For canonical GLMs, the curvature weights of the empirical Hessian are . These weights are nonnegative for every labeling outcome only when , which is the range from the classical estimator () to the fully prediction-powered estimator (). PPI++ also clips to and reports that the difference from the unrestricted estimator is usually small in practice (Angelopoulos et al., 2026, Section 6.2). Efficiency is lost only if the unrestricted optimum lies outside . We will emphasize the constrained problem in the revised version, whose interior solutions satisfy Theorem 2.
Q3: Is the estimated covariance positive definite?
We appreciate this question. Yes, the matrix is positive semidefinite by construction. We have with weights , because . Since is an empirical covariance, . Thus, can be singular but never indefinite.
For any , , so the log-determinant is always defined. The ridge only shifts slightly, and Theorems 3–4 hold for any limit of (see our response to Q2). No singular occurred in our experiments. For numerical stability, we will compute the log-determinant with a Cholesky factorization and increase only if it fails. We will include this argument in Section 3.3.
Q4: Joint confidence ellipsoids and Theorem 4
We agree that this justification should be explicit. Theorem 3 gives joint asymptotic normality of . With a consistent estimator whose limit is positive definite, the joint Wald region satisfies , and its volume is proportional to . This is the region that Section 4 evaluates and that the D-criterion minimizes. The marginal intervals of Theorem 4 do not give simultaneous coverage. We will update this joint region in Theorem 4 and Remark 4.
All corresponding clarifications, results, and discussion will be incorporated into the revised paper. We appreciate your thoughtful and constructive feedback, and welcome any further discussion.
References
-
PPI: Anastasios N. Angelopoulos, Stephen Bates, Clara Fannjiang, Michael I. Jordan, and Tijana Zrnic. Prediction-powered inference. Science, 382(6671):669–674, 2023.
-
PPI++: Anastasios N. Angelopoulos, John C. Duchi, and Tijana Zrnic. PPI++: Efficient prediction-powered inference. The Annals of Applied Statistics, 20(3):2235–2252, 2026.
-
Active inference: Tijana Zrnic and Emmanuel J. Candès. Active statistical inference. In Proceedings of the 41st International Conference on Machine Learning (ICML), 2024.
I appreciate the authors' detailed response and clarifications. However, my main concern regarding optimality remains. Although the authors have acknowledged this limitation, the theoretical results establish only conditional optimality for each design parameter, rather than joint optimality of the proposed two-stage procedure. I believe this limits the strength of the paper's theoretical contribution.
Therefore, I maintain my original rating.
Response to Reviewer ySZx
Thank you for your continued engagement and for clarifying your remaining concern. We recognize that joint global optimality would strengthen the theoretical contribution. Our separation of the two stages is motivated by two considerations.
1. Distinct roles. Both stages target the same criterion, , but they operate with different information. The policy allocates the label budget before any new outcomes are observed, whereas calibrates how much the estimator relies on the predictions once the labels are collected. After acquisition, can still be retuned, but the probabilities under which the labels were acquired cannot. The working value therefore only shapes , and Figure A5 shows nearly identical efficiency and coverage for , which supports this design in practice.
2. Computational trade-off. For a fixed , the D-optimal policy has no closed form (Theorem 1). It is computed by the MM algorithm (Proposition C.1), where each step requires a covariance factorization and a budget-constrained water-filling pass over the whole pool. A profiled joint search over would solve this policy problem repeatedly, once per candidate value. The two-stage design avoids this outer loop while targeting the same criterion. In a multiwave extension, newly acquired labels could inform both the next policy and its prediction weight, and we will discuss this direction and its theoretical challenges as future work in Section 5.
We will make this rationale and the scope of our guarantees explicit in Section 3.3. We sincerely appreciate your time and hope these clarifications and revisions adequately address your concerns. We would be grateful if you would consider revisiting your assessment, and we welcome further questions and discussion.
Response to Reviewer ySZx (1/2, W1 - W3)
We thank the reviewer for the careful review and for recognizing D-optimality as a natural criterion for joint uncertainty. We clarify each point below.
W1: What is new beyond adopting D-optimality?
We do not claim novelty for D-optimality or for the prediction-powered estimator. Our contribution is the covariance-dependent design that arises when we combine them with active label acquisition.
OptAPIchooses both which points to label and how much to rely on the prediction model f. PPI (Angelopoulos et al., 2023), PPI++ (Angelopoulos et al., 2026), and active inference (Zrnic and Candès, 2024) are special cases (Section 3.1).For a fixed λ, the D-optimal policy (Theorem 1) is πopt(x)∝[E{∥Δλ∥(Vλπ)−12∣X=x}]1/2. The A-optimal rule uses a fixed weighting matrix, but this rule whitens the residual scores by the covariance that the policy itself induces. Thus, the policy has only an implicit definition, and we compute it with a budget-constrained MM algorithm with a closed-form water-filling step (Proposition C.1). Theorem 2 shows that the D-optimal power also depends on this covariance, which couples the two choices. The criterion also targets the volume of the joint Wald region, which is proportional to det(Σ)1/2 (see our response to Q4). We will state these contributions in Section 1.2 (see also our response to Reviewer ALYQ, W1).
W2: Optimality claims
We agree that "attains the minimal asymptotic variance" overstates our results. Theorems 1–2 characterize each design parameter given the other, not their joint optimum. The paper establishes the following.
We will replace the overstated claim with "a two-stage design that targets the D-optimal asymptotic covariance." Joint optimization of (π,λ) is future work.
W3: Empirical gains
Thank you for this point. Section 4.1 is constructed to show when D-optimality matters, not to claim universal superiority. Scale disparity is a change of units. The D-optimal design does not change under it, but the A-optimal design does, so Section 4.1 isolates this mechanism.
The two metrics answer different questions. MSE is a trace criterion, so it favors A-optimal designs. The effective sample size neff measures the volume of the joint confidence region, which Section 4 evaluates (see our response to Q4). The real datasets have no constructed scale disparity. On the housing data,
OptAPIhas the smallest MSE at nearly all budgets and the largest neff (Figure 4). On the census data,OptAPIhas the largest neff at every budget (Table A2):OptAPIOn the census data, the A-optimal methods have a slightly smaller MSE. This reflects the trade-off between joint volume and marginal accuracy (see our response to Reviewer ALYQ, W1). We will state this trade-off and report per-coordinate interval widths together with neff, MSE, and coverage.