acceptodds
Under review as a conference paper at ICLR 2027

What Does a Predictive Gain Certify? Auditing Information Claims under Approximate Inference

Abstract

A better predictor need not use new information: it may simply make better use of information already available. We study when a held-out predictive gain supports a claim of additional conditional information. Our audit connects three requirements: attributing the gain, validating the baseline's approximation error, and accounting for uncertainty in the final decision. We prove that probability reports and labels alone cannot distinguish some informative and non-informative problems. For a known categorical baseline, we derive an exact reverse-to-forward KL conversion requiring only its smallest probability and one scalar root, avoiding exponential subset enumeration. A failure-aware audit incorporates uncertain error bounds and supports both betting and fixed-sample confidence statements. For sampled categorical reports, simultaneous intervals and a simplex constraint avoid the universal-bound failure caused by unseen classes. On natural WDBC, Digits, and Wine labels, an independent witness sample yields positive simultaneous information bounds in six of nine fixed analyses; all three Wine analyses remain inconclusive. A learned-representation audit checks attribution on continuous 64-dimensional inputs. A deterministic deeper layer passes the predictive-gain test in every 8,000-case repetition despite having zero additional information; the corrected audit makes no positive information claim. With an added informative signal, the betting audit detects it in 98% of repetitions, compared with 100% for conditional randomization using its exact feature sampler.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Approximate inference, conditional mutual information, predictive gain, hypothesis testing, proper scoring rules, variational inference, audi

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_NKW8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This work studies when a gain in predictive log score supports a claim of additional information when adding new input, a common move in machine learning to improve model performance. But improvements from adding new input may just mean the original model was a poor approximation. The work provides a mathematical proof that we need a guaranteed upper bound on how wrong the original model is (a “witness”) in order to claim additional information

From there they show that the usual diagnostics, such as reverse-KL error, can look promising even when the gain reflects no new information. For categorical predictions, they derive an exact reverse-to-forward KL bound requiring only the smallest baseline probability and the solution of one scalar equation. The audit combines this error bound with allowances for witness failures and sampling uncertainty to give a lower confidence bound on conditional information. Controlled experiments show that it avoids certifying improvements from a deterministic representation while detecting an added informative signal. On WDBC, Digits, and Wine, it produces positive certificates in six of nine analyses.

Critical strengths
  1. The main theorems and proofs in the work appear correct, as far as I was able to verify. The paper makes a clear and formal distinction between a model making better predictions and an input adding information, and what better predictions establishes about the new input. This is a point often overlooked in empirical research, such as when interpreting feature or representation gains. The separation example makes the issue of standard checks concrete.
  2. The exact categorical bound is a useful theoretical contribution. It replaces a search over class splits with solving a single scalar equation, and Supplementary Table A4 shows that the generic divergence bound is approximately 1.5–2.4 times larger in the tested settings.
Critical weaknesses
  1. The practical applicability of the results seem to be limited, as it is unclear how often a useful witness can actually be obtained. The method needs a justified upper bound on the baseline’s error, which is also the hardest requirement to satisfy. Every experiment gets it from something convenient: a known teacher model (learned representation), an enumerable posterior (the Bayesian study), or a context with only two groups, so the probabilities can be estimated directly (natural labels). These examples help explain the method, but leave open how to obtain a sufficiently tight witness for richer inputs when the true outcome distribution is unknown.
  2. The experiments do not clearly establish the downstream benefits of the main theoretical results. The successful natural-label experiments construct their witnesses directly from outcome counts, while the learned-representation experiment uses known outcome probabilities. I don’t think these results necessarily show whether the new categorical bound makes otherwise inconclusive information claims certifiable. A comparison holding the predictors, approximation budget, and audit data fixed would help establish how much the sharper bound changes the final conclusion.
Questions for authors
  1. Beyond the settings demonstrated in the paper, where else do you expect a sufficiently tight witness to be obtainable in practice? For example, what additional assumptions, validation data, or error guarantees would you need to construct one for the time-series application?
  2. How much difference does the categorical bound make to the final audit result? With the predictors, approximation budget, and audit data held fixed, how much does it improve the information lower bound over a generic divergence bound, and how often does it turn an inconclusive result into a positive certificate?
Reviewer_MbtrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper derives lower bounds on conditional information from predictive gains by accounting for baseline approximation error and sampling uncertainty. Its main technical contribution is an exact categorical reverse-to-forward KL conversion, extended to sampled probabilities.

Critical strengths

The fixed-reference categorical bound is a clean result: the worst case increases the least-probable class, reducing computation to one scalar root. The supplement provides substantive proofs and comparisons against generic divergence bounds. The paper also carefully distinguishes predictive improvement from additional information and reports negative results.

Critical weaknesses
  1. The main practical bottleneck remains unresolved. A useful certificate requires a justified, sufficiently tight baseline-error bound. The experiments obtain this from a known teacher, an exactly enumerable posterior, or separate labels within two discrete contexts. These validate the accounting but do not demonstrate its usefulness for an intractable baseline. In particular, the natural-label study certifies information beyond a binary thresholded summary, not beyond the original continuous input. This restriction should be explicit in the abstract.

  2. The experiments do not establish the practical benefit of the main categorical result. The successful natural-label and representation audits bypass the categorical conversion. For sampled distributions the bounds remain loose: at 100 classes and 10,000 draws, the mean bound is 0.764 nats against actual error of 0.047. Being below the universal bound is insufficient. The paper should show when the improvement changes the final certification decision.

  3. The sampled-bound comparison does not isolate the proposed construction. The old method uses Hoeffding intervals; the new method uses Clopper–Pearson intervals and additional constraints. The supplement acknowledges this, but the separating comparison is left in machine-readable results. Report comparisons using identical intervals and identify which bound components determine the result.

  4. The empirical conclusions remain narrow. The six positive natural-label bounds come from three overlapping splits each of WDBC and Digits; all Wine bounds are inconclusive. Sensitivity to the split proportions and correction cap is not reported. The quantitative TV Floodgate comparison is useful but shows a tradeoff: at 2,000 cases its variants detect the signal in 100% of trials versus 30% for the proposed audit; at 8,000 cases the proposed audit gives larger bounds. Neither establishes a general advantage.

  5. The main paper and supplement describe a separate code-and-results package but neither links to it. I can't find it in the supplementary paper. Please provide an anonymous repository link.

Questions for authors

A) Can the full procedure produce a useful certificate with a practically obtained baseline-error bound?

B) Does the categorical improvement change certification decisions under matched confidence intervals?

C) Are the natural-label conclusions stable across reasonable split proportions and correction caps?

Reviewer_cmoSOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper audits held-out predictive gains by attributing the gain, validating the baseline's approximation error, and accounting for uncertainty in the final decision.

Critical strengths
  • The proposed framework is well-motivated and can have various applications especially in safety critical settings like healthcare.
  • The theoretical analysis is supported with empirical results.
Critical weaknesses
  • The introduction is difficult to read because it starts with mathematical notation without defining the full notation. For example, what is G?
Questions for authors
  • Can you move some of the related work section prior to the methodology? This can help the reader understand preliminaries before proceeding with the rest of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.