acceptodds
Under review as a conference paper at ICLR 2027

Per-Input Prior Adequacy: Sources of Information and Conditions of Failure

Abstract

Tabular foundation models (TFMs) acquire their entire competence from a synthetic prior sampled at meta-training time, and their prediction errors are far from uniform across inputs. Prior evaluation scores a whole dataset with one number, and such a score cannot say where a model is misaligned. Applying those scores per input does not help either. A coverage-style score is uninformative, and a paired improvement ratio is blind to uniform prior gaps because it is a rate statistic. To address this, we propose the reference-averaged log-error ratio (, RA-ratio), a per-input, training-free diagnostic composed of two components, the excess loss of the model over a Bayesian reference and an average of that excess over the reference's prior strength. First, for each input we take the model's log error in excess of that of a closed-form Bayesian reference, the Bayes gap of in-context learning theory. Second, we average that excess over a grid of reference strengths, which removes the reference's strength as a tuning knob rather than selecting it. Third, we validate the diagnostic causally, injecting meta-training tasks from the region it flags and verifying that the error falls there and not in an off-target region. Notably, the RA-ratio attains a mean rank correlation of with the per-point error on regression datasets with seeds, ahead of every reference-free baseline we tried including three conformal variants, and replicates on a second architecture. On a held-out classification family, prior-based per-input diagnostics help where the model's own uncertainty has stopped resolving difficulty, and hurt where it already succeeds.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Tabular Foundation Models, Pointwise Prior Adequacy, Uncertainty Quantification

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_XnxkOpenReview ↗
Rating
1
Confidence
1
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates how to evaluate the adequacy of synthetic priors in TFMs, such as TabICL and TabPFN. While existing approaches primarily evaluate prior adequacy at the dataset level, the authors aim to identify individual inputs where the model's prior may be inadequate. To this end, the authors propose RA-ratio, which compares the model's prediction error against a Bayesian linear regression reference, averaged over different reference prior strengths. Experiments show that the proposed metric achieves a high Spearman correlation with per-input prediction errors, outperforming several uncertainty-based baselines. Furthermore, the authors conduct controlled prior-injection experiments, demonstrating that adding meta-training tasks from the identified regions can reduce prediction errors.

Critical strengths
  1. The paper identifies limitations of existing improvement-rate-based metrics and presents controlled experiments to illustrate them. The authors are also transparent about negative findings and revisions to their earlier conclusions.
Critical weaknesses
  1. According to Eq. (3), the proposed RA-ratio explicitly includes the model's prediction error, which requires access to the ground-truth label. Yet the main evaluation measures the Spearman correlation between the RA-ratio and this same prediction error. This raises concerns about potential circularity, since the proposed metric already includes the quantity against which it is evaluated. In addition, comparing this label-dependent metric with uncertainty-based baselines that do not require ground-truth labels may be unfair.
  2. The paper is difficult to follow, especially for readers without a strong background in tabular foundation models. It offers too little background while introducing many specialized concepts and technical terms, making its motivation and methodology unnecessarily hard to understand. Some parts of the organization also seem unfinished or unclear. For example, Sections B.2 and B.3 contain headings but no content, while Appendix A includes a lengthy revision process that seems unusual for a conference submission. In addition, Section 5.5 refers to pre-registered criteria G1–G5 but does not clearly define them.
Questions for authors
  1. Could the authors clarify whether they use the same prediction error as in the RA-ratio to calculate the Spearman correlation in the main experiments? If so, how do they justify this, given that the proposed metric already includes the quantity it evaluates?
  2. Computing the RA-ratio appears to require access to ground-truth labels. How can the proposed method be used when these labels are unavailable? Could the authors clarify its practical advantages over directly analyzing prediction errors?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Xnxk

Circularity. Yes — the same per-input error appears in ℓˉ\bar\ell (Eq. 3) and in the Spearman target of Table 1. We concede the endpoint and reframe: Table 1 becomes a reference-context comparison with label dependence stated; the abstract will lead with the controlled and causal validation instead of ∣ρ∣=0.76|\rho|=0.76. The fairness point about label-free baselines is also accepted; §3.1 now states the usage scope (revised PDF).

Labels unavailable. ℓˉ\bar\ell is a post-hoc diagnostic for the setting where labels exist and one asks where the model's failure is model-attributable rather than reference-solvable difficulty. Raw error ranks error perfectly but cannot attribute it. On the controlled causal experiment this attribution is now measured: per-point ℓˉ\bar\ell ranks which points targeted repair will fix at ρ=−0.953\rho=-0.953, against −0.489 for raw error (global response), so the reference normalisation carries repair-relevant information that raw error does not. When labels are unavailable, the applicable component of this work is the scope rule (§5.5): reach for reference-based diagnostics only where the model's own uncertainty has stopped ranking difficulty.

Presentation. Accepted in full: B.2/B.3 will receive prose, G1–G5 will be defined in Appendix C, the process appendix moves to the code repository, and §1–§2 will add background for readers without TFM experience.

Reviewer_XnxkOpenReview ↗

Response to Authors

Thank you for the response. I appreciate that the authors have acknowledged the fundamental issues with the original evaluation and made substantial efforts to address them. However, the changes are quite extensive, including a significant restructuring of the paper's central claims and supporting evidence. I feel the revised work would be better suited for a new submission in the next conference cycle, where it can receive a fresh, thorough review.

Reviewer_jecHOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a per-input diagnostic of synthetic-prior adequacy for tabular foundation models. It first observes that a log-error improvement-rate statistic cancels multiplicative error gaps that are constant across support sizes. It then defines a level statistic comparing the model's realized error with that of a Bayesian linear reference, and averages this log-error ratio over six reference prior strengths. The resulting RA-ratio reportedly achieves mean absolute within-task rank correlation .759 with regression error across 22 datasets and replicates on TabPFN 2.5 and two non-PFN learners. A controlled prior-injection experiment improves a flagged region, while classification experiments characterize when the diagnostic ranks losses better than predictive entropy.

The motivation is worthwhile, and the intervention and negative-result analyses are useful directions. However, the main evaluation reuses the outcome it is intended to diagnose, and several theoretical and causal interpretations exceed what the construction establishes.

Critical strengths
  1. Localizing failures within a dataset is a meaningful complement to dataset-level prior evaluation.
Critical weaknesses
  1. The primary ranking evaluation directly reuses the target error. Section 3.1 defines regression error as the absolute deviation from the true query target, and Eq. (3) gives ℓˉi=log⁡eM,i−1∣K∣∑κ∈Klog⁡eR,i,κ.\bar\ell_i=\log e_{M,i}-\frac1{|\mathcal K|}\sum_{\kappa\in\mathcal K}\log e_{R,i,\kappa}. The main evaluation then correlates this score with the same per-input model error eM,ie_{M,i}. Thus, the score already contains the quantity being ranked. Let's assume that query labels are available, then the correlation is not a clever measurement of how effective the proposed metric is. For example, the raw model error or its logarithm can achieve Spearman correlation 1 with the evaluation target (for example, setting the reference to be a constant). The logic here is that: when you don't know the label and define some metric, the correlation between the metric and the real error may indicate that the metric is useful; but now you already know the label, then the correlation doesn't imply the effectiveness of the metric.

  2. The author may have chosen an improper reference model. The repeated identification of Eq. (2) with the Bayes gap of Wakayama and Suzuki is not justified. Their Proposition 3.1 defines a nonnegative expected squared discrepancy from the Bayes-optimal in-context predictor. The term "Bayes" in their context means "Bayes-optimal", meaning that it is any method that achieves the lowest posterior risk, no matter what the method is. The present statistic is a difference of log realized absolute errors against a potentially misspecified linear reference. The term "Bayes" in the current context is "Bayesian model" that uses some specific Bayesian linear model. Being Bayesian under the chosen NIG model does not make the reference Bayes-optimal for the target distribution. See the original Bayes-gap definition. The definition of Eq. (2) is thus not the Bayes gap.

Questions for authors
  1. What advantage remains over raw-error, constant-reference, and shuffled-reference controls under an independently meaningful endpoint?
1 comment
AuthorsOpenReview ↗

Response to Reviewer jecH

On the endpoint. We agree, and concede the point rather than defend it: the Table 1 correlation reuses the outcome and cannot measure discrimination beyond raw error. The validation we stand on is §5.2 (known-gap construct), §5.4 (causal injection with specificity), and Fig. 3 (knob invariance) — see global response. On your specific question, what advantage remains over raw-error, constant-reference, and shuffled-reference controls under an independently meaningful endpoint, we now have the numbers. Using the per-point error change from targeted injection as the endpoint (512 paired points on the true-gap mechanism): ρ(ℓˉ,Δ)=−0.953\rho(\bar\ell,\Delta)=-0.953 vs −0.489 for raw error; width and ψ\psi are wrong-signed (+0.167, +0.394). The constant-reference control returns exactly ρ=1.000\rho=1.000 on the error endpoint, confirming your degeneracy argument; that degeneracy is why the counterfactual endpoint is the right one, and on it raw error does not come close to ℓˉ\bar\ell. The shuffled-reference control (400 permutations) shows the real reference re-ranks beyond the entire shuffled range on 5g (0.763 vs 0.817–0.889), so the re-ranking it injects is structured rather than noise.

On the reference model. You are right: Wakayama & Suzuki's Bayes gap is excess over the Bayes-optimal in-context predictor, and a NIG Bayesian linear reference is not Bayes-optimal for the target distribution. The repeated identification was an error of wording, not of construction, and every instance will be corrected (abstract, §3.3, contributions, conclusion). The statistic remains a reference-relative excess in log error; what it licenses is prior-relative adequacy claims (§B.9).

Reviewer_noqhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper looks at whether synthetic pretraining prior inadequacy in tabular base models (TFMs) can be identified for individual inputs rather than just across the entire dataset. The authors note that coverage based diagnostics and improvement rate statistics may well miss errors which are either very localised or spread out.

To deal with this they propose the Reference-Averaged Log-Error Ratio (RA-ratio). This is a training-free diagnostic which compares the model's actual prediction error to a Bayesian reference, the reference being averaged over different strengths of the reference prior. They also discuss the limitations of rate-based statistics and consider when the reference-based diagnostics actually provide more information than the model's own uncertainty.

They tested their method on 22 regression datasets and found mean absolute within-task Spearman correlations of 0.759 on TabICL and 0.745 on TabPFN 2.5 (both outperforming the reference-free baselines). Carefully designed experiments show that improvement rate statistics can sometimes fail to fully offset errors, whereas targeted meta-training interventions can reduce error in a weak region by 43.8%. Additional classification experiments look at when the diagnostic is of help and identify cases where predictive entropy is actually more useful.

Critical strengths
  • The paper offers a very clear failure analysis with controlled validation. There's a good mathematical explanation of how improvement rate statistics can mask uniform error gaps, backed up by controlled synthetic experiments. This is a very major limitation of the current evaluation methodology.
  • The study is a very thorough empirical investigation and makes its limitations very clear. It looks at 22 regression datasets, reproduces the results on TabPFN 2.5, uses matched averaging controls, looks at reference family sensitivity, applies focused interventions and does classification experiments. It also reports negative results and is upfront about the cases where predictive uncertainty actually provides the better information.
Critical weaknesses
  • The primary evaluation doesn't in any way confirm prior inadequacy. Equation (3) defines the RA-ratio in terms of the observed model error and Table 1 gives the correlation of that RA-ratio with the observed model error. A high correlation is not necessarily the case and the experiment in no way shows how much of the information on prior mismatch is due to reference normalisation. An observed error statistic would rank the evaluation target perfectly but would tell you nothing about prior mismatch. The absolute Spearman correlations reported also mask whether the rankings are in the expected direction. To support the main interpretation we need an independent measure of known or counterfactual prior inadequacy.
  • The paper has very little in the way of methodological innovation and the causal attribution is not really established. The reference normalised error construction, the prior strength averaging and the targeted repair approach are all based on well known principles [a], [b], [c]. It's conceivable that combining these would be useful but the causal experiments don't really establish that RA ratio guided localisation is superior to simply selecting regions informed by observed error or on model unpredictability. Recent work on synthetic prior evaluation has focused on isolating the effect of the prior by making systematic changes to the synthetic task distributions and using matched training protocols [d], [e]. The experiments here do touch on this to some extent but don't really disentangle prior inadequacy from reference model mismatch or general predictive difficulty. The link between the proposed realised log error ratio and the theoretical Bayes gap also needs to be explained [a].

References

[a] https://arxiv.org/abs/2510.10981

[b] https://arxiv.org/abs/1008.1550

[c] https://arxiv.org/abs/2607.16122

[d] https://arxiv.org/abs/2606.29241

[e] https://arxiv.org/abs/2605.18971

Questions for authors
  1. Construct validity: as the RA-ratio directly incorporates the observed prediction error, can the authors assess it against an independent measure of prior inadequacy (e.g. known prior mismatch regions or counterfactual error decrease achieved by altering the synthetic pretraining prior)? Can the authors quantify the information provided by reference normalisation beyond the observed error and report signed rank correlations?
  2. Causal localisation: In the specific intervention experiments, were the regions selected independently using the RA-ratio or were they selected on the basis of knowledge of the controlled synthetic construction? Can the authors compare RA-ratio guided selection with recorded error, uncertainty based and random selection under matched intervention budgets, reporting both the improvement in the target region and the off-target degradation?
  3. Theoretical justification along with robustness: Under what assumptions does the reference normalised log error statistic map to the theoretical Bayes gap [a]? How robust are the findings across several reference families, averaging grids and support set choices? Can the authors be more specific as to whether the diagnostic is actually sensitive to synthetic prior mismatch rather than simply to irreducible noise or reference misspecification?
1 comment
AuthorsOpenReview ↗

Response to Reviewer noqh

Q1 (construct validity). The known-mismatch measure is the gapR arm of §5.2 (constructed uniform gap; ℓˉ\bar\ell median +1.40 vs −0.29 covered), and the counterfactual error decrease is Table 2 (−43.8% in the flagged region with off-target and equal-compute controls). We have also computed the per-point version: on gapR, ρ(ℓˉ,Δ)=−0.953\rho(\bar\ell,\Delta)=-0.953 vs −0.489 for raw error (512 paired points; details in the global response), and the signed rank correlations over the main 22-dataset suite are ℓˉ\bar\ell +0.759 (positive on 66/66 dataset-seed rows) and ψ\psi −0.302 (negative on 94%). On information beyond observed error: §B.6 bounds it — flexible references at matched effective dof retain 0.625–0.663, so part of what the RA-ratio detects is "hard for a linear model"; we therefore claim prior-relative adequacy, not isolation of the prior gap alone (§B.9), and the revision will state this in the abstract.

Q2 (causal localisation). The injected arms add tasks from the constructed gapR mechanism (x∈[3,5]x\in[3,5] under the gap function); the intervention target is therefore defined by the construction. What the statistic does independently is detect that region: the level statistic ranks gapR above both non-gap mechanisms (Table 3, F1 detection; ℓˉ\bar\ell median +1.40 vs −0.29) without using the construction labels. The per-point analysis above goes further than region-level selection: it shows per-point ℓˉ\bar\ell predicts which individual points the injection repairs (−0.953), roughly twice as well as per-point raw error (−0.489). The matched-budget selection-rule comparison you describe (ℓˉ\bar\ell-guided vs observed-error-guided vs unpredictability-guided vs random region selection, retraining per rule) is the natural next step; we commit to reporting it in the revision whatever it shows.

Q3 (theory and robustness). We accept the correction on Eq. (2): it is not the Bayes gap of Wakayama & Suzuki, and the revision (already uploaded) renames it "reference-relative excess log error" with the Bayes-optimal vs Bayesian-model distinction stated explicitly in §3.3. Robustness across reference families, averaging grids and support sets is in §B.5–B.6 and Table 5 (level spans 0.073–0.680 over κ0\kappa_0; ℓˉ\bar\ell constant per noise level; four reference families). On sensitivity to irreducible noise vs prior mismatch: the σ\sigma axis of Fig. 3 is the gap knob, and the new per-point counterfactual analysis is our direct answer — statistics that merely track irreducible difficulty (width, ψ\psi) do not rank repair (+0.167, +0.394, wrong sign), while the reference-relative statistics do.

On novelty ([a]–[c]). The paper credits all three components (§2, Appendix A); the delta we claim is the per-input resolution and the causal test at input level — [c] (CRAFT-style targeted repair) operates on capability clusters, not per-input prior flags. We will make this delta explicit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.