acceptodds
Under review as a conference paper at ICLR 2027

Beyond Point Predictions: Distribution-Informed Prediction-Powered Inference

Abstract

Prediction-powered inference (PPI) typically relies on point predictions on unlabeled data. When predictive distributions are available as in a wide range of applications, including predictions from LLMs, we introduce distribution-informed prediction-powered inference (DiPPI), a general framework for further improving statistical efficiency by using predictive distributions as auxiliary information in the spirit of PPI. We characterize the optimal use of this information through score calibration, derive the oracle efficiency for a finite-dimensional representation of the predictive distribution, and provide theoretical guarantees for positive learning with the cross-fitted DiPPI estimator. Through simulations and three real-data applications, we show that DiPPI achieves better efficiency than PPI methods based on point predictions. These gains arise when the predictive distribution contains score-relevant information that is partially lost in point predictions.

Primary area:
learning theory
Keywords:
Prediction-powered inference, Predictive distributions, Statistical efficiency, Semi-supervised inference, Uncertainty quantification

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_ZS73OpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

DiPPI uses predictive distributions rather than only point predictions to improve statistical inference with limited labels. It learns the conditional expectation of the estimating function and combines this with residual correction, cross-fitting, and matrix calibration. The theory characterizes when distributional information improves asymptotic efficiency. Simulations and Wine, Forest, and Politeness applications evaluate estimation error, confidence-interval width, and coverage.

Critical strengths

The strongest point is the explicit condition for when a predictive distribution adds information beyond a point summary. This makes the efficiency claim precise rather than assuming richer predictions always help. Simulations and three real-data applications support the practical benefit through shorter confidence intervals while retaining coverage close to the nominal level.

Critical weaknesses

The methodological step beyond RePPI appears relatively small: the main change is to give the auxiliary regression richer prediction features. The oracle efficiency comparison confirms that richer conditioning information can help, but does not by itself establish a substantial advance in the inference procedure. The experiments support the usefulness of distributional features more clearly than the need for a distinct statistical framework.

Questions for authors

Which part of DiPPI goes beyond supplying richer auxiliary features to a RePPI-style procedure? Please clarify whether the main advance is a different estimator, additional guarantees, or the characterization of when distributional features help. This distinction matters to my assessment of the methodological novelty.

1 comment
AuthorsOpenReview ↗

Response to Reviewer ZS73

We thank the reviewer for the positive assessment, and for identifying the "explicit condition for when a predictive distribution adds information beyond a point summary" as the paper's strongest point. We address the reviewer's question below.


Relationship to RePPI (Question 1 and Weakness 1)

"Which part of DiPPI goes beyond supplying richer auxiliary features to a RePPI-style procedure? Please clarify whether the main advance is a different estimator, additional guarantees, or the characterization of when distributional features help."

The main advances are the characterization of when distributional information improves efficiency, the newly added procedure for selecting Φ(Q)\Phi(Q), and additional theoretical guarantees. We detail these contributions below.

  • (i) When distributional information helps (Section 2.2). We start with the recognition that when more information is available than point predictions for the outcome, we should take advantage of the information beyond the point predictions. Eq. (5) quantifies the oracle efficiency gain from using QQ rather than a point prediction Y^=T(Q)\widehat{Y}=T(Q). The gain is zero if and only if E[Uθ⋆(X,Y)∣X,Q]=E[Uθ⋆(X,Y)∣X,Y^]\mathbb{E}[U_{\theta^\star}(X,Y)\mid X,Q]=\mathbb{E}[U_{\theta^\star}(X,Y)\mid X,\widehat{Y}]. Thus, a gain requires additional information about the estimating score beyond the covariates and the point prediction.

  • (ii) How to select Φ(Q)\Phi(Q) (new Section 2.5). We propose a data-driven procedure that selects Φ(Q)\Phi(Q) from a fixed finite set of candidates, including the predictive mean as a baseline. Specifically, we fit DiPPI using each candidate and select the one with the smallest estimated asymptotic variance for the parameter of interest. Proposition 5 establishes selection consistency and asymptotically valid Wald intervals constructed using the selected representation. All simulations and the Wine and Politeness experiments now use this procedure.

  • (iii) General estimating equations (Theorems 1 and 2). Our theory covers locally identified roots of smooth estimating equations without requiring a convex loss. The proofs establish existence and local uniqueness of a root through a contraction argument. Appendix E (p. 37) illustrates why this extension matters through nonconvex location estimation and instrumental-variable estimating equation examples.

  • (iv) Covariance estimation and inference (Theorem 4). Theorem 4 explicitly proves consistency of the cross-fitted covariance estimator in Eq. (10) and asymptotic validity of Wald intervals. The proof accounts for fold-specific corrections and the dependence induced by reusing the same unlabeled sample across folds.

We hope these clarifications and revisions address the reviewer's concerns. We would be happy to address any remaining questions.

Reviewer_UHrBOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes distribution-informed prediction-powered inference (DiPPI), which uses predictive distributions or their finite-dimensional summaries as auxiliary information for estimating population parameters. It estimates conditional scores using distributional features, then applies three-fold cross-fitting and matrix calibration. The paper characterizes the oracle efficiency gain over point predictions and establishes asymptotic normality, consistent covariance estimation, and asymptotic non-inferiority to labeled-only estimation under stated conditions. Simulations and three applications on Wine, Forest, and Politeness show generally improved estimation accuracy and shorter confidence intervals with approximately nominal coverage.

Critical strengths
  1. Clear characterization of when distributional information helps.

Equation (5) quantifies the oracle covariance improvement over a point summary and identifies the exact condition for no improvement. This provides a useful distinction between predictive uncertainty itself and information relevant to the target estimating equation.

  1. Controlled experiments support the proposed mechanism.

The simulations isolate information carried by variance and skewness. The designs help establish that the gains arise from information discarded by point summaries.

Critical weaknesses
  1. The methodological advance over RePPI appears limited.

RePPI already uses conditional-score estimation, three-fold cross-fitting, and the same covariance-based matrix calibration. Although the proposed information source is useful, the paper does not sufficiently establish which methodological or theoretical contributions go beyond this extension, limiting the case for substantial novelty.

Questions for authors
  1. Relationship to RePPI: For loss-based estimating equations, is DiPPI equivalent to applying RePPI with Φ(Q)\Phi(Q) as a multivariate surrogate? If not, what precisely differs, and which theoretical results require arguments beyond substituting this richer auxiliary input? I would be willing to revise my rating if the authors clearly address this concern.

  2. How were the Wine and Politeness probability vectors obtained? Please specify the probability extraction procedure, prompts, and number of model calls per observation.

3 comments
AuthorsOpenReview ↗

Response to Reviewer UHrB

We thank the reviewer for the positive assessment of our theoretical characterization and supporting experiments. Below, we address your questions regarding our contributions beyond RePPI and provide details on constructing the predictive distributions. Please also see our general response for a summary of the main revisions.


Relationship to RePPI (Question 1 and Weakness 1)

"For loss-based estimating equations, is DiPPI equivalent to applying RePPI with Φ(Q)\Phi(Q) as a multivariate surrogate? If not, what precisely differs, and which theoretical results require arguments beyond substituting this richer auxiliary input?"

Yes. For convex-loss and a fixed Φ(Q)\Phi(Q), DiPPI uses the same score regression, three-fold cross-fitting, and matrix calibration as RePPI, with Φ(Q)\Phi(Q) serving as a multivariate surrogate. RePPI's asymptotic normality guarantee apply in this setting with a multivariate surrogate. The main differences and our contributions are to characterize when distributional information improves efficiency over point predictions, provide a data-driven procedure for selecting Φ(Q)\Phi(Q), extend the theory to general smooth estimating equations, and establish consistency of the cross-fitted covariance estimator and asymptotic validity of Wald intervals.

  • (i) When distributional information helps (Section 2.2). We start with the recognition that when more information is available than point predictions for the outcome, we should take advantage of the information beyond the point predictions. RePPI works with a point prediction, and its authors note as a limitation that this may discard information in richer model outputs such as logits and uncertainty scores (Ji et al., 2026, Section 5). Eq. (5) quantifies the oracle efficiency gain from using QQ rather than a point prediction Y^=T(Q)\widehat{Y}=T(Q). The gain is zero if and only if E[Uθ⋆(X,Y)∣X,Q]=E[Uθ⋆(X,Y)∣X,Y^]\mathbb{E}[U_{\theta^\star}(X,Y)\mid X,Q]=\mathbb{E}[U_{\theta^\star}(X,Y)\mid X,\widehat{Y}].

  • (ii) How to select Φ(Q)\Phi(Q) (new Section 2.5). We propose a data-driven procedure that selects Φ(Q)\Phi(Q) from a fixed finite set of candidates, including the predictive mean as a baseline. Specifically, we fit DiPPI using each candidate and select the one with the smallest estimated asymptotic variance for the parameter of interest. Proposition 5 establishes selection consistency and asymptotic validity of the Wald interval constructed using the selected representation.

  • (iii) General estimating equations (Theorems 1 and 2). Our theory covers locally identified roots of smooth estimating equations without requiring a convex loss. The proofs establish existence and local uniqueness of a root through a contraction argument. Appendix E (p. 37) illustrates why this extension matters through nonconvex location estimation and instrumental-variable estimating equation examples.

  • (iv) Covariance estimation and inference (Theorem 4). Theorem 4 explicitly proves consistency of the cross-fitted covariance estimator in Eq. (10) and asymptotic validity of Wald intervals. The proof accounts for fold-specific corrections and the dependence induced by reusing the same unlabeled sample across folds.

We have clarified this relationship in Remark 2, added representation selection and Proposition 5 in Section 2.5, and added the estimating-equation examples in Appendix E.


Construction of probability vectors (Question 2)

"How were the Wine and Politeness probability vectors obtained? Please specify the probability extraction procedure, prompts, and number of model calls per observation."

Thank you for this question. In our experiments, we use GPT-5.4 nano with temperature 11, reasoning effort none, and top_logprobs=5, which returns at most five candidate-token log-probabilities per call. Since this is insufficient to reliably obtain probabilities for all 20 Wine ratings or 25 Politeness ratings in a single call, we recursively split the rating range into lower and upper groups labeled A and B. At each split, we extract and normalize the first-token probabilities of A and B to estimate their conditional probabilities, which are then multiplied along each path to construct QQ. This requires 19 calls per Wine review and 24 per Politeness request, before retries. Exact prompts and details of handling missing token probabilities are provided in Appendix B.1 (pp. 14–16).

We hope these clarifications and revisions address the reviewer's concerns about the contributions beyond RePPI and the construction of QQ. We respectfully ask the reviewer to reconsider the rating based on the revised manuscript and would be happy to address any remaining questions.

Thanks for the clarification. I have raised my score.

Thank you so much. We are glad that our response addressed your concerns, and we appreciate your constructive feedback.

Reviewer_1HiSOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary
  • The paper studies prediction-powered inference (PPI), where a small labeled sample and a large unlabeled sample are combined with predictions that a fixed pretrained system makes on both samples.

    • In PPI, the predictions on the unlabeled sample add information, and the labeled sample is used to correct the bias from prediction errors.
    • The goal is valid and more efficient inference on a target parameter θ⋆\theta^\star defined by an estimating equation E[Uθ⋆(X,Y)]=0\mathbb{E}[U_{\theta^\star}(X,Y)]=0.
  • The authors observe that many prediction systems, including LLMs, output a predictive distribution QQ in addition to a point prediction.

    • They argue that features of QQ beyond its mean or mode, such as dispersion, skewness, or class probabilities, may carry information about the estimating function UU that a point prediction loses. A predictive distribution QiQ_i is obtained for every observation ii, whether labeled or unlabeled.
  • The paper proposes a framework DiPPI, which learns the conditional expectation of UU given XX and a finite dimensional summary of the predictive distribution, Φ(Q)\Phi(Q), using the labeled sample with three-fold cross-fitting.

    • The learned score predictor is additionally calibrated by a matrix estimated on the held-out labeled fold, which is also used for inference.
    • The calibrated predictor is evaluated on the held-out labeled fold and on the unlabeled sample to correct the estimating equation.
    • The final estimator θ^DiPPI\hat\theta^{\mathrm{DiPPI}} is the average of the three fold-specific estimators, weighted by fold size.
  • Theoretically, Theorem 1 and Eq. (5) show that the optimal correction based on QQ is at least as efficient as the one based on a point prediction Y^=T(Q)\hat Y = T(Q).

    • For the cross-fitted DiPPI estimator actually used, Theorem 2 and Corollary 3 show that it is asymptotically normal and, under regularity conditions, asymptotically no less efficient than labeled-only estimation.
  • In simulations where QQ carries information beyond the point prediction, and on three real datasets (Wine, Politeness, and Forest CoverType), DiPPI is reported to give shorter confidence intervals than labeled-only estimation, PPI++, and RePPI, with coverage near the nominal level.

Critical strengths
  1. The method seems to be simple to apply in practice.

    • When a usable predictive distribution is already available, DiPPI is straightforward to implement by replacing the point-prediction input in RePPI’s score regression with Φ(Q)\Phi(Q).
    • It also works with any suitable regression learner.
  2. This work is equipped with theoretical justification of DiPPI.

    • Specifically, the paper establishes asymptotic normality, consistent variance estimation, and asymptotically valid confidence intervals.
    • It also shows that, under regularity conditions, DiPPI is asymptotically no less efficient than labeled-only estimation, and that it attains the minimum asymptotic covariance among corrections based on (X,Φ(Q))(X,\Phi(Q)) when the conditional expectation of the score is estimated consistently.
    • The proofs in Appendix B appear correct.
  3. The experiments are consistent with the theory.

    • The simulations consider outcome-relevant variance, heterogeneous reliability, and skewness separately.
    • DiPPI also performs similarly to RePPI when the distribution carries no extra score-relevant information, consistent with the oracle equality in Eq. (5).
Critical weaknesses
  1. It is unclear whether the gains come from the distributional structure of QQ or simply from a richer input.

    • As far as I can tell, theoretical results in this work treat QQ as a generic auxiliary variable and would hold unchanged for any vector observed in both samples, such as embeddings or several point predictions, so the theory does not separate the two.
    • The only distribution-specific element of the method is the choice of Φ(Q)\Phi(Q).
      • Section 2.3 gives examples and a practical starting point, but no systematic way to choose Φ(Q)\Phi(Q) and no comparison of alternatives in the experiments.
    • The experiments do not separate them either.
      • In Wine and Politeness, DiPPI uses a 20- or 25-dimensional probability vector, while RePPI uses a single number.
      • The experiments show an advantage over RePPI's single summary, but not whether a few summaries, such as the mean and the standard deviation of QQ, would give most of this gain.
  2. The construction of QQ is insufficiently specified.

    • For Wine and Politeness, the paper only says that GPT-5.4 nano is used to "construct a predictive distribution" over the ratings (Line 404-405, Line 469).
    • However, the paper does not report the following:
      • whether QQ comes from token log-probabilities, repeated sampling, or verbalized probabilities,
      • the prompt, including whether it contains XX,
      • how probability mass outside the rating tokens is handled (for example, mass placed on tokens such as "The" or on out-of-range numbers such as "100" in Wine).
    • Since DiPPI uses the shape of QQ itself, how QQ is constructed can affect the results.
    • For example, if QQ is reconstructed solely from a single top_logprobs list at the rating-token position, the returned alternatives are capped at 20 tokens. This need not cover all 25 ratings, and truncation and renormalization may alter the estimated dispersion or tails.
    • The paper should clarify how QQ is constructed, including whether top_logprobs list is used or not.
  3. (Suggestion) Although the theory shows that DiPPI is at least as efficient as RePPI in the oracle sense (Eq. (5)), it is not clearly stated where the gain over RePPI comes from.

    • The Introduction states that "when the predictive distributions capture (some level of) heterogeneity in the uncertainty of the point predictions, DiPPI delivers additional efficiency gains over RePPI" (Line 77-78).

      • If I read correctly, from Line 185-194, it is implied that, for means and linear or logistic regression, using the full QQ improves on using only its mean m(Q)m(Q) only if E[Y∣X,Q]≠E[Y∣X,m(Q)]\mathbb{E}[Y \mid X,Q] \neq \mathbb{E}[Y \mid X,m(Q)].
      • For the claim in Line 77-78 to hold, heterogeneous uncertainty would therefore have to imply E[Y∣X,Q]≠E[Y∣X,m(Q)]\mathbb{E}[Y \mid X,Q] \neq \mathbb{E}[Y \mid X,m(Q)]. This is not shown in the paper.
    • Indeed, heterogeneous uncertainty alone is not enough.

      • Suppose QQ satisfies the conditional mean-calibration property E[Y∣X,Q]=m(Q)\mathbb{E}[Y\mid X,Q]=m(Q).
      • Then, by the tower property, E[Y∣X,m(Q)]=E[E[Y∣X,Q]∣X,m(Q)]=m(Q)=E[Y∣X,Q]\mathbb{E}[Y\mid X,m(Q)]=\mathbb{E}[\mathbb{E}[Y\mid X,Q]\mid X,m(Q)]=m(Q)=\mathbb{E}[Y\mid X,Q], so by Eq. (5) there is no gain over RePPI with the predictive mean, no matter how much the spread of QQ varies across observations.
      • In other words, mean calibration constrains only the center of QQ, not its spread, so the spread of QQ can still vary widely across observations -- this case shows that heterogeneous uncertainty in QQ does not imply E[Y∣X,Q]≠E[Y∣X,m(Q)]\mathbb{E}[Y \mid X,Q] \neq \mathbb{E}[Y \mid X,m(Q)].
    • So, for means and linear or logistic regression, DiPPI improves on RePPI with the predictive mean only when the conditional bias of m(Q)m(Q) depends on the shape of QQ beyond (X,m(Q))(X, m(Q)).

      • The abstract and Section 2.2 are consistent with this, but the sentence in the Introduction suggests that heterogeneous uncertainty alone suffices. I suggest revising Line 77-78 to state this condition.
Questions for authors
  1. In Wine and Politeness, how do DiPPI with Φ(Q)=(m(Q),sd(Q))\Phi(Q)=(m(Q),\mathrm{sd}(Q)) and RePPI with both the mean and the mode of QQ compare? Since both use a two-dimensional summary of QQ, this comparison would help distinguish the effect of the spread of QQ from that of simply adding a second summary.

  2. Apart from the choice of Φ(Q)\Phi(Q), does any part of the theory rely on QQ being a distribution rather than a generic auxiliary vector?

  3. How was QQ obtained from GPT-5.4 nano (method, prompt, whether XX is included, model settings), and how was probability mass outside the rating tokens handled? Are the results sensitive to these choices?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 1HiS (Part 1): Question 1 and Weakness 1

We thank the reviewer for the careful assessment of the method, proofs, and experiments. Below, we clarify when distributional information improves efficiency, how to choose Φ(Q)\Phi(Q), and how QQ is constructed, using the theoretical results and additional experiments to address the reviewer's concerns. Please also see our general response for a summary of the main revisions.

Question 1 and Weakness 1: Comparison and selection of distributional representations

"Weakness 1: Section 2.3 gives examples and a practical starting point, but no systematic way to choose Φ(Q)\Phi(Q) and no comparison of alternatives in the experiments."

We agree. We have added a data-driven rule for choosing Φ(Q)\Phi(Q) and a comparison of alternative representations (Section 2.5; Appendices A.1 and B.2).

Specifically, given a fixed set of candidate representations Φ1(Q),…,ΦJ(Q)\Phi_1(Q),\ldots,\Phi_J(Q), we run DiPPI separately for each candidate, estimating the conditional score using (X,Φj(Q))(X,\Phi_j(Q)) and computing the corresponding asymptotic covariance estimator Σ^jDiPPI\widehat\Sigma_j^{\mathrm{DiPPI}}. All candidates use the same labeled folds and unlabeled sample. For a prespecified target a⊤θ⋆a^\top\theta^\star, we select

a^\top\widehat\Sigma_j^{\mathrm{DiPPI}}a,$$ and use $\Phi_{\widehat j}(Q)$ for the final estimator and confidence interval. Proposition 5 shows that, when the minimizer is unique, selection is consistent, and the Wald interval remains asymptotically valid. For $\Phi_1(Q),\ldots,\Phi_J(Q)$, we recommend a small candidate set, fixed before examining the labeled outcomes. Include the predictive mean as a point-based baseline, then adding the standard deviation to capture dispersion and quantiles to capture asymmetry and tail behavior. For an outcome with $K$ possible values, $K-1$ probabilities retain the full distribution and can be included. We evaluate this procedure against fixed representations in the simulations and Wine and Politeness. In the variance-driven simulation, mean and standard deviation perform similarly to the higher-dimensional quantile representations and are selected in 80–87.5% of replications. When skewness is also relevant, quantile-based representations are selected in 89.5% of replications. In Wine and Politeness, the full probability vector achieves lower RMSE than the two-dimensional summaries, while data-driven selection achieves similar performance. The full vector is selected in 99.8% and 87.6% of replications, respectively. Full results are reported in Appendices A.1 (pp. 12–13) and B.2 (pp. 16–17). --- > "Weakness 1: $\ldots$ no comparison of alternatives in the experiments. $\ldots$ The experiments show an advantage over RePPI's single summary, but not whether a few summaries, such as the mean and the standard deviation of $Q$, would give most of this gain." > Question 1: "In Wine and Politeness, how do DiPPI with $\Phi(Q)=(m(Q),\operatorname{sd}(Q))$ and RePPI with both the mean and the mode of $Q$ compare?" To address this question, we compare different representations of $Q$ to examine whether the efficiency gains depend on which features are used. In Wine and Politeness, we consider six candidate representations: $\Phi_1=m(Q)$, $\Phi_2=(m(Q),\operatorname{mode}(Q))$, $\Phi_3=(m(Q),\operatorname{sd}(Q))$, $\Phi_4=(q_{0.1}(Q),q_{0.25}(Q),$ $q_{0.5}(Q),q_{0.75}(Q),q_{0.9}(Q))$, $\Phi_5=(\Phi_3(Q),\Phi_4(Q))$, and $\Phi_6= (\text{full probability vector})$. $\Phi_2$ and $\Phi_3$ provide the requested comparison. The table reports results at $r=0.10$ over 200 replications. Relative RMSE is measured against labeled-only estimation; "Selected" uses the data-driven choice of $\Phi(Q)$. | Data | Metric | $\Phi_1$ | $\Phi_2$ | $\Phi_3$ | $\Phi_4$ | $\Phi_5$ | $\Phi_6$ | Selected | |---|---|---:|---:|---:|---:|---:|---:|---:| | Wine | Relative RMSE | 0.957 | 0.911 | 0.931 | 0.921 | 0.919 | 0.856 | 0.856 | | | CI width | 0.598 | 0.577 | 0.577 | 0.572 | 0.569 | 0.509 | 0.509 | | | Coverage (%) | 96.0 | 98.5 | 95.5 | 96.0 | 95.5 | 94.0 | 94.0 | | Politeness | Relative RMSE | 0.958 | 0.972 | 0.935 | 0.939 | 0.935 | 0.912 | 0.927 | | | CI width | 1.674 | 1.673 | 1.676 | 1.673 | 1.675 | 1.640 | 1.632 | | | Coverage (%) | 95.5 | 95.0 | 95.5 | 95.5 | 95.5 | 95.0 | 95.0 | Mean and mode give lower RMSE than mean and standard deviation in Wine, while the reverse holds in Politeness. Neither two-dimensional representation performs as well as the full probability vector $\Phi_6$, which achieves lower RMSE and shorter confidence intervals in both datasets. The selection procedure achieves comparable performance to the full vector, selecting it in 100% of Wine and 68% of Politeness replications at $r=0.10$ (99.8% and 87.6% pooled across all labeled ratios). The comparisons are reported in Appendix B.2 (pp. 16–17, Figure 6 and Tables 3–4).
AuthorsOpenReview ↗

Part 2: Question 2 and Weakness 3

Question 2: Theoretical Scope of DiPPI

"Apart from the choice of Φ(Q)\Phi(Q), does any part of the theory rely on QQ being a distribution rather than a generic auxiliary vector?"

"Weakness 1: It is unclear whether the gains come from the distributional structure of QQ or simply from a richer input."

Thank you for your helpful comments. Our theoretical guarantees apply to general auxiliary variables and do not require QQ to be a probability distribution. The oracle gains in Eq. (5) depend on score-relevant information provided by QQ and lost in the point summary T(Q)T(Q), instead of relying on QQ being a distribution.

Our focus is on the information lost when a predictive distribution is reduced to a mean or mode. As discussed in Section 2.2, one way to use QQ is to predict the score by directly integrating it with respect to QQ, which requires QQ to be a distribution. For mean estimation and the usual linear and logistic regression scores, this prediction uses only the mean m(Q)m(Q) and need not be optimal. The optimal correction instead depends on E[Y∣X,Q]\mathbb{E}[Y\mid X,Q], which may depend on other features of QQ. These features can improve efficiency when they help predict the error of m(Q)m(Q) beyond (X,m(Q))(X,m(Q)). Eq. (5) quantifies the resulting oracle efficiency gain, and the new comparisons in Appendix B.2 examine how different representations of QQ perform in practice.


Weakness 3: Clarification of the Efficiency Gain Statement

"The abstract and Section 2.2 are consistent ..., but the sentence in the Introduction suggests that heterogeneous uncertainty alone suffices. I suggest revising Line 77-78 to state this condition."

We agree, and thank the reviewer for pointing this out. Heterogeneous predictive uncertainty alone does not guarantee an efficiency gain and our original writing was imprecise. We have revised it to: "Moreover, we quantify the oracle efficiency gain over RePPI based on a point summary. There is a gain when conditioning on the predictive distribution changes the conditional expectation of the estimating score given the covariates and the point summary."

AuthorsOpenReview ↗

Part 3: Question 3 and Weakness 2

Question 3 and Weakness 2: Construction of QQ and sensitivity analysis

Weakness 2: "The construction of QQ is insufficiently specified."

Question 3: "How was QQ obtained from GPT-5.4 nano (method, prompt, whether XX is included, model settings), and how was probability mass outside the rating tokens handled? Are the results sensitive to these choices?"

Thank you for pointing out the need for more details. We have added the complete construction procedure, model settings, and prompts in Appendix B.1 (pp. 14–16), together with a sensitivity analysis in Appendix B.3 (p. 18).

Construction of QQ.

  • Method. In our experiments, the GPT-5.4 nano API supports at most five top-token log-probabilities per call (top_logprobs=5), which is insufficient to obtain probabilities for all 20 Wine ratings or 25 Politeness ratings in a single call. We therefore recursively split the rating range into lower and upper groups labeled A and B. At each split, we ask the model to choose between A and B and normalize their first-token probabilities to estimate the conditional probabilities of the two groups. We then multiply these probabilities along each path to construct the full rating distribution QQ. This requires 19 queries per Wine review and 24 per Politeness request, before retries.

  • Prompts and covariates. Wine prompts contain the review text and covariates, including the price and location. Politeness prompts contain the request text; the hedging indicator is not supplied separately because it is determined by that text. Each query uses a system prompt and a user prompt, without conversation history or examples. Please see more details in Appendix B.1 (pp.15-16).

  • Model settings. We use gpt-5.4-nano-2026-03-17, temperature 11, reasoning effort none, and top_logprobs=5.

  • Other tokens and missing probabilities. When both A and B probabilities are available, we renormalize over these two labels, excluding mass assigned to other tokens. If a label probability is missing, we retry the query up to three times. If only one label probability pp remains available, we approximate the missing probability by 1−p1-p. This fallback is an approximation of an exact renormalization over the two labels.

Sensitivity analysis.

We compare three constructions of QQ for Politeness, including the original binary partition, the binary partition with paraphrased system and user prompts, and a ternary partition requiring 12 queries per request. Model settings are held fixed. The table reports results over 200 replications per labeled ratio rr; the three values in each cell follow this order.

Sensitivity analysis on Politeness (200 replications per rr; coverage in %).

rrRMSECI widthCoverage (%)
0.100.387 / 0.390 / 0.3731.632 / 1.613 / 1.60795.0 / 95.0 / 95.5
0.150.336 / 0.326 / 0.3321.335 / 1.330 / 1.32496.0 / 94.5 / 95.0
0.200.333 / 0.338 / 0.3301.151 / 1.148 / 1.14291.0 / 91.5 / 92.5
0.250.255 / 0.251 / 0.2531.027 / 1.026 / 1.02197.5 / 95.0 / 94.5
0.300.278 / 0.275 / 0.2760.938 / 0.936 / 0.93390.5 / 91.5 / 91.5

The three constructions yield similar RMSE, confidence-interval widths, and coverage across the labeled ratios. These results suggest that the performance of DiPPI is not highly sensitive to the prompt wording or partition scheme considered.

We hope these clarifications and revisions address the reviewer's concerns about the choice of Φ(Q)\Phi(Q), the theoretical scope and efficiency gains of DiPPI, and the construction of QQ. We respectfully ask the reviewer to reconsider the rating in light of the revised manuscript and would be happy to address any remaining questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for their careful and constructive feedback. Our starting point is that many prediction systems, including LLMs, provide more than a point prediction for the outcome, and that inference should take advantage of this additional information. Existing PPI methods, including RePPI, typically take a point prediction as the input. We ask when the rest of the predictive distribution helps, and how to use it. The reviewers raised several questions, which we answer below. Changes in the updated manuscript are marked in blue.


1. DiPPI versus RePPI with a richer surrogate, and the role of QQ as a distribution (new Remark 2 and Appendix E)

For loss-based estimating equations and a fixed Φ(Q)\Phi(Q), DiPPI shares the score regression and matrix calibration steps of RePPI, with Φ(Q)\Phi(Q) serving as a multivariate surrogate. Our asymptotic theory does not require QQ to be a probability distribution or to be calibrated. Beyond this, we characterize exactly when a predictive distribution improves efficiency over a point prediction and provide a data-driven procedure for selecting Φ(Q)\Phi(Q) with theoretical guarantees (Section 2.5). Our theory also extends beyond RePPI's convex-loss setting to locally identified roots of general smooth estimating equations. Appendix E (p. 37) illustrates why this extension matters through nonconvex location estimation and instrumental-variable estimating equations. Finally, Theorem 4 explicitly proves consistency of the cross-fitted covariance estimator and validity of Wald intervals, accounting for fold-specific corrections and the shared unlabeled sample. We have added Remark 2 (p. 6) to clarify this.


2. Data-Driven Selection of Φ(Q)\Phi(Q) (new Section 2.5, Proposition 5)

The usefulness of different features of QQ depends on the estimating score and the inferential target. We therefore propose a data-driven procedure (in Section 2.5) that selects Φ(Q)\Phi(Q) from a fixed finite set of candidates, including the predictive mean as a baseline. Specifically, we fit DiPPI using each candidate and select the one with the smallest estimated asymptotic variance for the parameter of interest. Proposition 5 establishes selection consistency and asymptotically valid Wald intervals. All simulations and the Wine and Politeness experiments now use this procedure.


3. Comparisons of alternative representations (updated Sections 3–4 and new Appendices A.1 and B.2)

To examine which features of QQ are useful and how representation choice affects efficiency, we have expanded the experiments to compare DiPPI using data-driven selection with fixed representations that combine mean, mode, standard deviation, and quantiles. Wine and Politeness additionally include the full probability vector as a candidate.

In Wine and Politeness, the full probability vector achieves lower RMSE than the two-dimensional summaries, while data-driven selection achieves similar performance. The full vector is selected in 99.8% and 87.6% of replications, respectively. Detailed results are provided in Appendices A.1 and B.2 (pp. 12–13 and 16–17).


4. Construction of QQ and sensitivity analysis (updated Section 4 and new Appendices B.1 and B.3)

We construct QQ from GPT-5.4 nano's first-token log-probabilities using recursive binary queries. Each query compares lower and upper rating groups; multiplying the conditional probabilities along each path gives the probability of each rating. This avoids requiring all 20 Wine or 25 Politeness rating tokens to appear in a single top_logprobs list. To assess whether prompt wording and partition choices affect inference, we compare the original binary procedure, binary queries with paraphrased prompts, and a ternary partition in Politeness. The three constructions yield similar RMSE, confidence-interval widths, and coverage rates across the labeled ratios considered, suggesting limited sensitivity to these choices. Full construction details and sensitivity results are provided in Appendices B.1 and B.3, respectively.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.