- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper finds having a context that performs well does not mean we can identify or use it in practice. Over fixed bank of 16 context authors report development headroom, headroom splits as H = G + S, pre-inference policies gains mostly come from choosing the same rule again and again and post-inference methods shows consistent gains.
Critical strengths
- The paper uses strong controls and separates different evidences, revealing that P2's gains is because of choosing a fixed rule.
- The paper compares complex selection method against probability averaging, showing extra computation can bring limited benefit and harm performance on many queries.
Critical weaknesses
- Novelty is limited, paper mainly reports negative results, without proposing new method or how theory can be turned into practical gains.
- Experiments use small support pools and hand-build candidate banks and does not reflect read-world use of TabPFn and TabICL on larger datasets. It's unclear whether findings generalize to real settings.
Questions for authors
- On fresh holdout, how does always using CLASS_BALANCED_KNN compare with the development selected fixed reference (STANDARDIZED_KNN/RANDOM_SLOT_6) in NLL?
- Does the conclusion, little added value from per-query selection beyond a good fixed rule or averaging, hold with larger support pools say 1000-3000 rows, and larger K?
5 comments
Follow-up on the revised experiments and remaining concerns
Thank you again for the detailed review. We wanted to briefly follow up because the revision now directly addresses the two experimental questions you raised: the fresh-holdout fixed-CBKNN comparison and the 1,000–3,000-row / larger-K setting, in addition to the new feature-dimension construction in Sec. 7.
If you have had a chance to look at the revision, we would greatly appreciate knowing whether these results resolve your main concerns, or whether there is a remaining issue that you think is still central to your assessment.
Thank you for the revision. The additional experiments addresses my scalability concern and shows that row-level selection offers limited benefit over simpler methods. The fixed-CBKNN results suggest P2/P3 gains mainly comes from rule quality, which should be clarified in the abstract.
My novelty concern still remains, and I suggest narrowing the claims in Section 7 regarding natural missingness. Given the careful experiments, I have raised my presentation and overall scores to 3.
Response to Reviewer YMMC
We thank the reviewer for the careful reading and the two concrete questions. Section and appendix numbers refer to the revised PDF. All numbers are NLL differences with 95% dataset-bootstrap intervals unless stated; positive favours the first-named method.
W1: "mainly negative results, without a new method or how theory turns into practical gains."
The paper's contribution is the evaluation logic: a selection strategy must beat the strongest alternative that uses the same information, and hindsight headroom does not tell whether it will. The revision adds a case where this logic leads to a gain. Row-level signals failed to identify the right context; the query's observed-feature set does identify what its context should contain. Constructing the context from it (Sec. 7) improves on full support by +0.051 [+0.018, +0.096] on four backbones and beats a mismatched construction of the same form by +0.063 [+0.024, +0.117]. We state its limits plainly: the underlying principle dates to 2007, the cost is 16 to 55 times that of full support, and on TabPFN-3.5 a cost-matched NaN-masking ensemble comes close. ICRM constructs contexts along the feature dimension; it does not rescue row selection, whose negative findings stand.
W2: small support pools and hand-built banks.
We added pools of 1,024 to 3,072 rows with K = 256/512 on thirteen datasets and four backbones (Sec. 6.2, App. K), and compared against full support, which uses no bank at all. App. G.2 varies how the bank is composed (resampled candidate identities; mostly random candidates): the changes are largely unresolved, so we do not claim invariance to bank construction. ICRM needs no candidate bank.
Q1: always using CLASS_BALANCED_KNN versus the development-selected reference on the fresh holdout.
CBKNN is better on all four backbones: TabPFN v2 +0.043 [+0.001, +0.100], TabPFN-3.5 +0.046 [+0.001, +0.106], TabICL v1 +0.068 [+0.027, +0.116], TabICLv2 +0.036 [+0.010, +0.064] (App. K.2). P2 selects that rule on 99.22% of queries, and no additional gain of P2 over fixed CBKNN was established. One caveat: on the ten additional CC18 datasets (App. K.2) the CBKNN advantage over the development reference is close to zero for the two TabPFN backbones (about -0.002 and -0.000) and positive for the two TabICL backbones (+0.035 and +0.050, only the last interval excluding zero), so we do not claim a general ranking of fixed rules.
Q2: does "little added value beyond a good fixed rule or averaging" hold with 1,000 to 3,000 rows and larger K?
Yes, at pool 3,072 and K = 512 (thirteen datasets, first seed).
(1) P2 adds no established gain over fixed CBKNN on any backbone (e.g. TabPFN v2 +0.0005 [-0.0001, +0.0015]).
(2) Predictive-centrality selection and averaging are equivalent within the prespecified margin of 0.01 NLL on all four backbones (differences 0.0014 to 0.0030; all 90% intervals inside the margin).
(3) Averaging beats a strong 64-estimator class-balanced context only on TabICLv2 (+0.029 [+0.005, +0.056]).
(4) Full support (all 3,072 rows) improves on averaging on three backbones (+0.014 to +0.018) but not on TabICLv2.
The limitation is that this primary comparison uses one seed.