- Rating
- 1
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates how to evaluate the adequacy of synthetic priors in TFMs, such as TabICL and TabPFN. While existing approaches primarily evaluate prior adequacy at the dataset level, the authors aim to identify individual inputs where the model's prior may be inadequate. To this end, the authors propose RA-ratio, which compares the model's prediction error against a Bayesian linear regression reference, averaged over different reference prior strengths. Experiments show that the proposed metric achieves a high Spearman correlation with per-input prediction errors, outperforming several uncertainty-based baselines. Furthermore, the authors conduct controlled prior-injection experiments, demonstrating that adding meta-training tasks from the identified regions can reduce prediction errors.
Critical strengths
- The paper identifies limitations of existing improvement-rate-based metrics and presents controlled experiments to illustrate them. The authors are also transparent about negative findings and revisions to their earlier conclusions.
Critical weaknesses
- According to Eq. (3), the proposed RA-ratio explicitly includes the model's prediction error, which requires access to the ground-truth label. Yet the main evaluation measures the Spearman correlation between the RA-ratio and this same prediction error. This raises concerns about potential circularity, since the proposed metric already includes the quantity against which it is evaluated. In addition, comparing this label-dependent metric with uncertainty-based baselines that do not require ground-truth labels may be unfair.
- The paper is difficult to follow, especially for readers without a strong background in tabular foundation models. It offers too little background while introducing many specialized concepts and technical terms, making its motivation and methodology unnecessarily hard to understand. Some parts of the organization also seem unfinished or unclear. For example, Sections B.2 and B.3 contain headings but no content, while Appendix A includes a lengthy revision process that seems unusual for a conference submission. In addition, Section 5.5 refers to pre-registered criteria G1–G5 but does not clearly define them.
Questions for authors
- Could the authors clarify whether they use the same prediction error as in the RA-ratio to calculate the Spearman correlation in the main experiments? If so, how do they justify this, given that the proposed metric already includes the quantity it evaluates?
- Computing the RA-ratio appears to require access to ground-truth labels. How can the proposed method be used when these labels are unavailable? Could the authors clarify its practical advantages over directly analyzing prediction errors?
2 comments
Response to Authors
Thank you for the response. I appreciate that the authors have acknowledged the fundamental issues with the original evaluation and made substantial efforts to address them. However, the changes are quite extensive, including a significant restructuring of the paper's central claims and supporting evidence. I feel the revised work would be better suited for a new submission in the next conference cycle, where it can receive a fresh, thorough review.
Response to Reviewer Xnxk
Circularity. Yes — the same per-input error appears in ℓˉ (Eq. 3) and in the Spearman target of Table 1. We concede the endpoint and reframe: Table 1 becomes a reference-context comparison with label dependence stated; the abstract will lead with the controlled and causal validation instead of ∣ρ∣=0.76. The fairness point about label-free baselines is also accepted; §3.1 now states the usage scope (revised PDF).
Labels unavailable. ℓˉ is a post-hoc diagnostic for the setting where labels exist and one asks where the model's failure is model-attributable rather than reference-solvable difficulty. Raw error ranks error perfectly but cannot attribute it. On the controlled causal experiment this attribution is now measured: per-point ℓˉ ranks which points targeted repair will fix at ρ=−0.953, against −0.489 for raw error (global response), so the reference normalisation carries repair-relevant information that raw error does not. When labels are unavailable, the applicable component of this work is the scope rule (§5.5): reach for reference-based diagnostics only where the model's own uncertainty has stopped ranking difficulty.
Presentation. Accepted in full: B.2/B.3 will receive prose, G1–G5 will be defined in Appendix C, the process appendix moves to the code repository, and §1–§2 will add background for readers without TFM experience.