- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies whether the full-context predictions of a frozen tabular foundation model can be approximated by aggregating predictions obtained from disjoint context shards. It introduces the convex decomposition deficit to characterize reconstruction error and provides a theoretical analysis separating representation reconstruction error from the effect of a nonlinear readout. The proposed method, Adaptive Mass Pooling (AMP), assigns query-dependent weights to shard predictions using the number of training examples from each shard within a query-specific nearest-neighbor radius.
Critical strengths
Relevant problem formulation and useful diagnostic analysis.
Critical weaknesses
W1.The experiments do not substantiate the central context-scaling motivation. A central practical motivation is to bypass context-length limitations by partitioning data and aggregating local predictions. However, the largest evaluated training context contains only 4,496 examples, on optdigits. This is the maximum training-set size in the evaluation, not an established context limit of the underlying models. More importantly, the authors explicitly acknowledge that full-context inference remains feasible on all evaluated datasets and that capacity constraints are simulated. Consequently, the paper does not demonstrate successful inference beyond an actual model or hardware capacity bottleneck.
W2.The methodological novelty is limited. AMP is essentially a query-dependent convex ensemble whose weights are normalized powers of local neighbor counts. The paper itself acknowledges close precedents: DENSE aggregates predictions across bounded contexts using query-dependent similarity weights, while Nadaraya–Watson regression admits an exact decomposition using each subset’s share of kernel mass.
W3. The experimental development and reproducibility details are insufficient.
W4. The manuscript appears insufficiently finalized, with inconsistent numerical reporting and unresolved presentation issues. Numerical precision varies without an explained reporting convention. For example, Table 3 reports a concrete result as , while a fish-toxicity result is reported as , using different precision for both the means and standard deviations. Different precision can be justified by scale, but the manuscript provides no rationale for these choices. A consistent, stated convention is particularly important when some reported differences are very small.
Questions for authors
Please refer to Weakness.
3 comments
W1: Context Scaling Motivation, Reconstruction Evaluation, and Large Scale Benchmarks
Methodological necessity of exact reconstruction evaluation: A core goal of our study is to test whether shard predictions can faithfully reconstruct the exact full context predictive distribution . Computing reconstruction metrics (TV and CDF MSE) and the diagnostic Oracle lower bound strictly requires the ground truth full context output , which is unobtainable when a dataset exceeds server capacity. Evaluating manageable datasets under controlled capacity limits is therefore a methodological necessity for exact reconstruction analysis and verifying Theorem 1, rather than an experimental limitation.
Fixed shards vs. active retrieval: In Section 3.2, active retrieval methods like LoCalPFN (Thomas et al., 2024) act as a special case of our framework that builds a single local context per query to directly shrink (Theorem 2) and (Theorem 1). Since LoCalPFN already established active retrieval, we focus on the complementary setting where data reside on fixed shards to isolate query adaptive aggregation weights (full partitioning protocols are detailed in the Appendix).
Expanded 10 seed and large scale benchmarks: Nonetheless, to directly validate context scaling and address the reviewer's concern in good faith, we expanded all runs from 5 to 10 random seeds and added large scale datasets that exceed our regular server's memory limits, renting a high memory server specifically to obtain the full context reference baselines. On these benchmarks, AMP runs smoothly within shard limits, consistently outperforms Equal and FedAvg, and recovers full context accuracy.
W2: Theoretical Contribution and Distinction from NW and DENSE
Theoretical framework for frozen TFMs: Rather than proposing a Nadaraya Watson (NW) estimator or an arbitrary pooling heuristic, we establish a theoretical framework showing how frozen Tabular Foundation Models (TFMs) under context partitioning naturally motivate convex reconstruction. Building on recent TFM interpretability findings, early layers map inputs into a shared latent space via (Balef et al., 2026), attention acts as normalized kernel smoothing defining a discrete measure for readout (Tsai et al., 2019), and middle to late layers are well approximated by dataset specific affine maps that commute with normalized weighted integration (Cannistraci and Vogt, 2026). Under this abstraction, Theorem 1 proves that a frozen TFM's partitioning error decomposes into representation reconstruction error and nonlinear readout dispersion .
Role of NW regression and Theorem 2: In this view, classical NW regression serves only as an idealized boundary case ( without a nonlinear readout ), while Theorem 2 shows that approximating unknown TFM attention ratios motivates local mass as a tractable proxy.
Fundamental distinctions from DENSE: Our approach differs from DENSE in three core aspects: (1) DENSE targets LLMs rather than TFMs; (2) DENSE's heuristic weight averages across entire shards, diluting local signals in large buckets and risking zero or negative weights from sign cancellation, whereas our Theorem 2 local mass strictly guarantees valid convex weights in ; and (3) we explicitly reconstruct the full context predictive distribution from partitioned shards.
W3 & W4: Reproducibility Details and Numerical Formatting
Hyperparameter tuning protocol (W3): We added full reproducibility protocols to the main text and Appendix. Hyperparameters and are tuned strictly via five fold cross validation on the training split without accessing test labels or test full context outputs.
preprocessing and tree based similarity (W3): For preprocessing, statistics are fit strictly on training features: missing numerical values are median imputed (0 if all missing) and standardized via (, divisor 1 if ), while categorical features are one hot encoded in with a missing bin (unseen test categories set to 0) into a float64 matrix. Moreover, using tree leaf cooccurrence similarity as an alternative proxy captures structure and consistently improves over raw in both reconstruction fidelity and downstream accuracy.
Standardized numerical formatting (W4): We have standardized all manuscript tables to enforce uniform decimal precision across all reported means and standard deviations, ensuring consistent presentation across all datasets (including fish toxicity and concrete).
Response to Reviewer hFdx: Thank You for Your Constructive Feedback and Rigorous Evaluation
We thank the reviewer for the constructive feedback. We have substantially revised the manuscript and Appendix to address the concerns. The main changes are as follows: