acceptodds
Under review as a conference paper at ICLR 2027

Minimax Acquisition Under a Shared Bounded Bias: Critical Windows and Sparse Portfolios

Abstract

Cheap model predictions can reduce the cost of estimating population means, but collecting more predictions does not remove a shared calibration bias. We study how to allocate a limited budget among trusted labels, predictions, and correlated paired observations when this bias lies in a known interval. Existing prediction-powered methods combine predictions with trusted labels to obtain valid inference, while recent minimax allocation theory optimizes acquisition under unrestricted prediction bias. These results leave unresolved how a finite bias bound changes which auxiliary observations are worth buying and how spending on them should scale with the budget. For Gaussian observations and continuous allocations, we reduce the exact fixed-design minimax risk over all measurable estimators to a bounded-normal-mean problem and derive two cost-normalized information criteria that distinguish regimes in which auxiliary data offer no improvement, optimal auxiliary spending remains bounded, or its share of the budget approaches one. We also derive sharp second-order risk expansions in the bounded-spending and critical regimes, characterize a 1/B-wide transition window for total budget B, and prove that an exact optimum can be attained using at most three observation types.

Primary area:
learning theory
Keywords:
prediction-powered inference, minimax estimation, optimal experimental design, data acquisition, bounded bias, uncertainty quantification

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_CrCWOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies budget allocation among trusted labels, cheap predictions, and paired observations for mean estimation under a Gaussian model with known costs, covariances, and a known bound on shared bias. It reduces the exact fixed-design minimax risk to a bounded-normal-mean problem and identifies regimes where trusted-only acquisition is optimal, auxiliary spending remains bounded, or the optimal auxiliary budget share approaches one. The authors also derive second-order risk expansions and a 1/B-wide critical transition, establish that an exact continuous optimum can use at most three observation types, and illustrate the theory through analytical examples.

Critical strengths

None

Critical weaknesses

The paper's utility is limited by the COMPLETE OMISSION of a real-data application. In the era of AI, particularly for the problem addressed here, I believe the authors could EASILY identify candidate datasets from public sources such as the UCI Machine Learning Repository and demonstrate how their framework would operate in practice. If obtaining suitable data for a fully real-world application is difficult, the authors should AT LEAST construct a semi-synthetic dataset grounded in publicly available real observations and apply the proposed method to it.

The absence of ANY such application raises serious questions about the practical substance of this research: is this a GENUINELY APPLICABLE method, or merely a theoretical combination of ideas from several existing works? The authors have not convinced me that the proposed framework meaningfully addresses the real-world problem used to motivate the paper.

Questions for authors
  1. Can the authors support their findings with a REAL-DATA APPLICATION in the revision, rather than relying exclusively on analytical examples? Please demonstrate how trusted labels, model predictions, and paired observations correspond to actual data, and explain how the shared-bias bound, covariance matrices, bias loadings, and acquisition costs would be specified or estimated BEFORE making the allocation. The comparison should assess whether the proposed procedure improves mean estimation over trusted-only acquisition and relevant baselines under the SAME TOTAL BUDGET, including any costs of obtaining the required inputs. Please distinguish empirical findings from guarantees that depend on the assumed Gaussian model. Such evidence would materially affect my assessment of the paper's PRACTICAL RELEVANCE.

  2. If obtaining suitable data for a fully real-world application is difficult, can the authors AT LEAST construct a SEMI-SYNTHETIC DATASET grounded in publicly available real observations and evaluate the proposed method on it in the revision? Please clearly distinguish the real and simulated components and compare performance with relevant baselines under the SAME TOTAL BUDGET.

Reviewer_CGmvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper explores a problem sometimes referred to as "active inference": Under a certain budget for collecting labels from multiple sources of varying quality (e.g., from human annotators, from cheap model predictions, etc), what is the optimal way in which to spend the budget to estimate a mean? This paper considers a setting where all observations are are Gaussian, and explores how the dynamics of the problem change depending on the characteristics of the data sources, identifying different "regimes" (Theorem 3) where spending the entire budget on "trusted" (i.e., bias-free) observations is optimal, all the way to a regime where spending a fixed proportion on auxiliary (i.e., bounded bias) observations is optimal, with a regime in the middle where the budget allocated to auxiliary observations is positive but not a constant share.

Critical strengths

I view the strength of the paper as (potentially) delivering useful insights and intuition about the problem of active inference. I appreciate the fact that the results are "non-asymptotic" in the sense that they identify an interesting boundary between the two more "obvious" regimes (where auxiliary information is useless on the one hand, vs where auxiliary information has some constant share of the budget), whereas a lot of asymptotic analysis in this general literature tends to point to one of the two obvious ones. The general shape of the results is intuitive to me, given prior work in this general area, e.g., the idea that second-order effects create interesting edge cases between regimes.

Critical weaknesses

To contextualize my comments below: I was excited to read this paper, based on the abstract and the introduction, which is not something I can say for other papers in my reviewing batch. My core concern is that (a) I believe there are some cool insights in this paper, but (b) I cannot seem to parse out what those insights are, exactly, because of issues of clarity and presentation. I would also note that the results do not necessarily suggest a practical algorithm, given the strength of some assumptions: In my view, that would be more than balanced out by providing some clear insights, but it does raise the bar for educating the reader on the insights, vs just proving some results.

Clarity and Presentation

The paper at times seems close to delivering some insights, but I wish they were clearer! In our current age of AI in mathematics, my view is that proving some results is of limited utility if the explanations are not relatively self-contained, with intuitions being well-explained and clear to a reader with the right background. I am reasonably familiar with the literature here (both on PPI and Active Statistical Inference, as well as much of the work cited in the introduction like Rosenman & Miratrix 2023), so I would expect it to be easier to follow the work.

I will give some non-exhaustive examples of why I found it difficult to follow along.

First, let us start with Section 2.1, where the core notation and model is introduced. I was immediately confused by the notation in Equation (1), and how it relates to the motivation stated in the introduction. Each observation is Gaussian with mean [fjgj]⊤[θδ][f_j g_j]^\top [\theta \delta], but what is fjf_j and gjg_j mean to represent? If the authors stated that fj=1f_j = 1, then this would be more intuitive (i.e., observations have some bias gjg_j scaled by δ\delta), so my confusion is really the role of fjf_j. Then it is stated that "the loadings, covariances, and costs are known", but I did not follow which aspects of the model correspond to loadings and costs. It is then stated that "predictions may have mean θ+ℓjδ\theta + \ell_j \delta", which further confused me: If fj=1f_j = 1, then this would be true with gj=ℓjg_j = \ell_j, but why the new notation?

Then there is the introduction of pairs: "A pair contains both measurements" (both measurements of what? Are trusted labels and predictions paired up and jointly observable?), and later when the "total information" is introduced, it is stated that "each pair contributes once through its full covariance", but I still am not entirely clear on what the pairs are. I also didn't have much intuition for what the total information represents, and the basis for defining the "information" as Fj⊤σj−1FjF_j^\top \sigma_{j}^{-1} F_j?

There is the additional issue of clarity that the theorems don't clearly link back to their referents: For instance, Theorem 1 references a,b,ca, b, c and jumps into discussion of the minimax squared risk for estimating θ\theta, and it took me a second read to realize that these constants come from Equation (2).

Because I'm a bit confused on what this all means, I find it difficult to interpret what bb represents, and why it is therefore intuitive that on lines 138-140 that if b=0b = 0, no bias correction is needed, etc.

There are other examples to give additional flavor, a smattering of confusions I had while reading the introduction:

  • The intro could do a slightly better job explaining what Γ0\Gamma_0 and Γ∞\Gamma_\infty are, and why the constant value of 1 appears (because it is "per unit cost"?), and what the meaning of information is?
  • The optimal risk is just 1 / B plus other lower-order terms? I would expect there to be some constant for the baseline variance of the outcome, so is this already "scaled" to have unit variance in some way?

Practical Significance

As I stated above, I do not think that a paper like this has to present a practical algorithm to be useful, since this is a theoretical paper as opposed to a methodological paper. But just to be clear, there are limitations of the proposed approach along those lines: The assumption of Gaussian observations, known covariances and bias loadings, known bounds on δ\delta, etc. I presume the Gaussian assumption is "load-bearing" in the sense that it allows for a lot of the results to be as clean as they appear to be.

Questions for authors

Can you walk me through the model in a way that clarifies the points raised above?

Reviewer_JY3COpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper studies how to allocate a limited budget to buy trusted labels and model predictions to minimize the worst-case MSE for estimating population means, assuming Gaussianity, known covariance matrices, and a known bound on the shared bias.

Critical strengths

The paper uses information matrices per unit cost to link the costs of trusted labels and model predictions to the statistical information they provide, making it easier to analyze budget allocation.

Critical weaknesses
  • Strong Gaussian assumption: The paper assumes that model predictions follow a Gaussian distribution, which is hard to verify for black-box models.
  • Limited empirical evaluation: The paper does not provide simulation studies with estimated covariances, a misspecified bias bound, or non-Gaussian data. It also lacks an empirical comparison with Multiple-prediction-powered inference (MultiPPI).
Questions for authors

MultiPPI has already established minimax optimality with known covariances and without a Gaussian assumption. What technical challenges does this paper additionally address, and can its results be extended to non-Gaussian settings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.