acceptodds
Under review as a conference paper at ICLR 2027

Adaptive Aggregation for Partitioned Tabular In-Context Learning

Abstract

Tabular foundation models perform in-context prediction by conditioning directly on labeled examples, but scaling them to large contexts is constrained by computational and memory requirements. This work studies whether full-context predictions of frozen tabular foundation models can be approximated from multiple shards, where the context is either partitioned to satisfy capacity constraints or naturally distributed across sources. We develop a theoretical framework that characterizes when such reconstruction is possible and how reconstruction error depends on partition structure and aggregation weights, motivating a simple training-free, query-adaptive pooling strategy. Experiments on real-world tabular classification and regression tasks examine when shard-level aggregation provides a useful approximation to full-context prediction and how its behavior varies across partitioning regimes. Together, these results provide a general perspective on reconstructing full-context predictions from partitioned contexts in tabular in-context learning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Tabular foundation models, in-context learning, adaptive aggregation, distributed inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_hFdxOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies whether the full-context predictions of a frozen tabular foundation model can be approximated by aggregating predictions obtained from disjoint context shards. It introduces the convex decomposition deficit to characterize reconstruction error and provides a theoretical analysis separating representation reconstruction error from the effect of a nonlinear readout. The proposed method, Adaptive Mass Pooling (AMP), assigns query-dependent weights to shard predictions using the number of training examples from each shard within a query-specific nearest-neighbor radius.

Critical strengths

Relevant problem formulation and useful diagnostic analysis.

Critical weaknesses

W1.The experiments do not substantiate the central context-scaling motivation. A central practical motivation is to bypass context-length limitations by partitioning data and aggregating local predictions. However, the largest evaluated training context contains only 4,496 examples, on optdigits. This is the maximum training-set size in the evaluation, not an established context limit of the underlying models. More importantly, the authors explicitly acknowledge that full-context inference remains feasible on all evaluated datasets and that capacity constraints are simulated. Consequently, the paper does not demonstrate successful inference beyond an actual model or hardware capacity bottleneck.

W2.The methodological novelty is limited. AMP is essentially a query-dependent convex ensemble whose weights are normalized powers of local neighbor counts. The paper itself acknowledges close precedents: DENSE aggregates predictions across bounded contexts using query-dependent similarity weights, while Nadaraya–Watson regression admits an exact decomposition using each subset’s share of kernel mass.

W3. The experimental development and reproducibility details are insufficient.

W4. The manuscript appears insufficiently finalized, with inconsistent numerical reporting and unresolved presentation issues. Numerical precision varies without an explained reporting convention. For example, Table 3 reports a concrete result as 4.309±0.0694.309 \pm 0.069, while a fish-toxicity result is reported as 0.9564±0.010.9564 \pm 0.01, using different precision for both the means and standard deviations. Different precision can be justified by scale, but the manuscript provides no rationale for these choices. A consistent, stated convention is particularly important when some reported differences are very small.

Questions for authors

Please refer to Weakness.

3 comments
AuthorsOpenReview ↗

Response to Reviewer hFdx: Thank You for Your Constructive Feedback and Rigorous Evaluation

We thank the reviewer for the constructive feedback. We have substantially revised the manuscript and Appendix to address the concerns. The main changes are as follows:

  • We added large scale experiments beyond our regular server's memory capacity and expanded all evaluations from 5 to 10 random seeds, demonstrating the scalability and effectiveness of AMP.
  • We clarified the theoretical contributions of our convex reconstruction framework, its distinction from NW regression and DENSE, and the connection between the theoretical results and AMP.
  • We provided detailed experimental protocols, including hyperparameter selection, data preprocessing, and partitioning strategies, and introduced a random forest based similarity measure to improve reconstruction and predictive performance.
  • We standardized the numerical precision across all tables to ensure consistent and accurate reporting.

W1: Context Scaling Motivation, Reconstruction Evaluation, and Large Scale Benchmarks

Methodological necessity of exact reconstruction evaluation: A core goal of our study is to test whether shard predictions can faithfully reconstruct the exact full context predictive distribution f(x,D)f(x, \mathcal{D}). Computing reconstruction metrics (TV and CDF MSE) and the diagnostic Oracle lower bound Ef(x,D(K))\mathcal{E} _ f(x, \mathcal{D}^{(K)}) strictly requires the ground truth full context output f(x,D)f(x, \mathcal{D}), which is unobtainable when a dataset exceeds server capacity. Evaluating manageable datasets under controlled capacity limits is therefore a methodological necessity for exact reconstruction analysis and verifying Theorem 1, rather than an experimental limitation.

Fixed shards vs. active retrieval: In Section 3.2, active retrieval methods like LoCalPFN (Thomas et al., 2024) act as a special case of our framework that builds a single local context per query to directly shrink ∥aD−λkaDk∥\|a _ {\mathcal{D}} - \lambda _ k a _ {\mathcal{D} _ k}\| (Theorem 2) and Vλ\mathcal{V} _ \lambda (Theorem 1). Since LoCalPFN already established active retrieval, we focus on the complementary setting where data reside on fixed shards to isolate query adaptive aggregation weights (full partitioning protocols are detailed in the Appendix).

Expanded 10 seed and large scale benchmarks: Nonetheless, to directly validate context scaling and address the reviewer's concern in good faith, we expanded all runs from 5 to 10 random seeds and added large scale datasets that exceed our regular server's memory limits, renting a high memory server specifically to obtain the full context reference baselines. On these benchmarks, AMP runs smoothly within shard limits, consistently outperforms Equal and FedAvg, and recovers full context accuracy.

W2: Theoretical Contribution and Distinction from NW and DENSE

Theoretical framework for frozen TFMs: Rather than proposing a Nadaraya Watson (NW) estimator or an arbitrary pooling heuristic, we establish a theoretical framework showing how frozen Tabular Foundation Models (TFMs) under context partitioning naturally motivate convex reconstruction. Building on recent TFM interpretability findings, early layers map inputs into a shared latent space U\mathcal{U} via ψ\psi (Balef et al., 2026), attention acts as normalized kernel smoothing defining a discrete measure νD(x,⋅)\nu _ {\mathcal{D}}(x, \cdot) for readout u(x,D)=∫X×Yψ(xi,yi) νD(x,d(xi,yi))u(x, \mathcal{D}) = \int _ {\mathcal{X}\times\mathcal{Y}} \psi(x _ i, y _ i)\, \nu _ {\mathcal{D}}(x, d(x _ i, y _ i)) (Tsai et al., 2019), and middle to late layers are well approximated by dataset specific affine maps that commute with normalized weighted integration (Cannistraci and Vogt, 2026). Under this abstraction, Theorem 1 proves that a frozen TFM's partitioning error decomposes into representation reconstruction error Eu\mathcal{E} _ u and nonlinear readout dispersion L2Vλ\frac{L}{2}\mathcal{V} _ \lambda.

Role of NW regression and Theorem 2: In this view, classical NW regression serves only as an idealized boundary case (Ef≡0\mathcal{E} _ f \equiv 0 without a nonlinear readout F\mathcal{F}), while Theorem 2 shows that approximating unknown TFM attention ratios aD/aDka _ {\mathcal{D}}/a _ {\mathcal{D} _ k} motivates local mass as a tractable proxy.

Fundamental distinctions from DENSE: Our approach differs from DENSE in three core aspects: (1) DENSE targets LLMs rather than TFMs; (2) DENSE's heuristic weight wi=1∣Bi∣∑xj∈Bicos⁡(xje,xe)w _ i = \frac{1}{|\mathcal{B} _ i|} \sum _ {x _ j \in \mathcal{B} _ i} \cos(x _ j^e, x^e) averages across entire shards, diluting local signals in large buckets and risking zero or negative weights from sign cancellation, whereas our Theorem 2 local mass nk(x0,T)/nn _ k(x _ 0, T)/n strictly guarantees valid convex weights in ΔK−1\Delta^{K-1}; and (3) we explicitly reconstruct the full context predictive distribution f(x,D)f(x, \mathcal{D}) from partitioned shards.

W3 & W4: Reproducibility Details and Numerical Formatting

Hyperparameter tuning protocol (W3): We added full reproducibility protocols to the main text and Appendix. Hyperparameters TT and α\alpha are tuned strictly via five fold cross validation on the training split without accessing test labels or test full context outputs.

L2L _ 2 preprocessing and tree based similarity (W3): For L2L _ 2 preprocessing, statistics are fit strictly on training features: missing numerical values are median imputed (0 if all missing) and standardized via zj=(xj−μj)/σjz _ j = (x _ j - \mu _ j)/\sigma _ j (ddof=0\text{ddof}=0, divisor 1 if σj=0\sigma _ j=0), while categorical features are one hot encoded in {0,1}\{0,1\} with a missing bin (unseen test categories set to 0) into a float64 matrix. Moreover, using tree leaf cooccurrence similarity as an alternative proxy captures X→yX \to y structure and consistently improves over raw L2L _ 2 in both reconstruction fidelity and downstream accuracy.

Standardized numerical formatting (W4): We have standardized all manuscript tables to enforce uniform decimal precision across all reported means and standard deviations, ensuring consistent presentation across all datasets (including fish toxicity and concrete).

Reviewer_k9HfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to combine predictions from separate data shards to approximate the full-context prediction of a frozen tabular foundation model. It defines a convex decomposition deficit and gives error bounds based on representation error and the effect of a nonlinear readout. The proposed method, Adaptive Mass Pooling (AMP), weights each shard by the number of samples near the query.

The experiments cover nine datasets and two foundation models. AMP achieves lower mean TV reconstruction error than size-weighted pooling in 14 of 18 dataset–model settings. The gains vary across tasks. Better reconstruction also does not always lead to better prediction of the true labels.

Critical strengths

1. The reconstruction problem is well defined and useful.

The convex decomposition deficit gives a clear reference for evaluating aggregation. The Oracle results help show whether better weights could improve reconstruction while keeping the shard predictions fixed. This is a useful way to study the limits of prediction-level aggregation.

2. The controlled comparison provides some credible positive evidence.

The aggregation rules use the same data splits and cached shard predictions. This helps isolate the effect of the weights. The study includes two models and five random seeds. AMP reduces mean TV error in 14 of 18 settings. On optdigits, the reduction is about 11.5% for TabPFN and 12.6% for TabICL. These results suggest that local mass can be useful in some settings.

Critical weaknesses

1. The theory does not yet explain why AMP should work for the tested models.

Theorem 2 concerns differences between full-context and shard-level attention weights. AMP instead uses Euclidean neighbor counts, followed by a power controlled by α. The paper gives no condition or evidence showing that these counts approximate the weights needed for reconstruction. It also does not explain how α affects either term in Theorem 1.

Equation (3.3) assumes a shared embedding map across the full context and all shards. The paper does not establish this representation for the tested models. Having softmax attention weights alone does not establish that the whole model has this form.

The method’s added value also needs a stronger test. DENSE already studies query-dependent weighting of predictions from separate contexts, as the paper acknowledges. Shared-kernel mass weighting is another stated precedent. The useful contribution here may be the reconstruction framework and the specific mass proxy, but comparisons with uniform and size-weighted pooling do not isolate that contribution. See DENSE, Sections 3.2–3.3.

I suggest comparing AMP with average-similarity weighting and kernel-mass weighting on exactly the same shards. A mechanism analysis should check whether local mass tracks useful reconstruction weights. The theory should also state clearly which model structures it covers.

Finally, Section 5 gives too strong an explanation for nonzero Oracle error. This error shows that the full prediction lies outside the convex hull of the shard predictions. A nonlinear readout can cause this even when the representation is fully decomposable. The result alone does not identify cross-shard interactions as the cause.

2. The experiments leave important questions about reproducibility and practical value.

The paper does not report the actual shard counts, capacity limits, shard size distributions, T and α values, or how these parameters were selected. Feature scaling and categorical feature handling are also unclear. These details matter because AMP relies directly on distances. For regression, the paper needs to explain how predictive distributions are represented and aligned, and how TV and CDF-MSE are calculated.

The experiments use only nine small datasets, with 598–4496 training samples. Full-context inference is feasible for all of them. There are no results on actual over-capacity data, naturally distributed shards, or system costs. This leaves the main scaling motivation untested.

The downstream losses can also be substantial. On optdigits, full-context accuracy is 0.993 for both models, while AMP reaches 0.967 and 0.976. Full-context inference has lower RMSE and MAE in all six regression settings. On concrete with TabPFN, AMP increases RMSE from 4.309 to 5.401, about 25%. Such losses need to be assessed together with measured resource savings.

For revision, I suggest reporting the full configuration first. Then add sensitivity results for T, α, feature scaling, and shard structure. Paired differences or confidence intervals would help assess the smaller gains; separate standard deviations do not answer that question. At least one real capacity-limited experiment should report accuracy, runtime, peak memory, and communication cost.

Questions for authors

Clarification requests that could affect my assessment:

  1. What are the exact shard counts, capacity limits, shard size distributions, T and α values, and parameter selection rules? Were all choices made without using test labels or test full-context outputs?

  2. How are features processed before computing distances? For regression, how are the predictive distributions represented and aligned across shards? How exactly are TV and CDF-MSE computed?

  3. How does Equation (3.3) map to the tested TabPFN and TabICL models? Does the shared embedding map remain unchanged when the context changes? If this is an approximation, what supports its accuracy?

  4. Do the current results include evidence that local mass tracks attention mass or useful reconstruction weights? Are there existing results for other query-dependent weighting rules or non-random partitions that could help establish AMP’s added value?

The additional baseline, sensitivity, and capacity-limited experiments proposed above are suggestions for revision. They are separate from these requests to clarify the current method and evidence.

3 comments
AuthorsOpenReview ↗

Response to Reviewer k9Hf: Thank You for Your Detailed Review and Suggestions

We thank the reviewer for the constructive feedback. We have revised the manuscript and Appendix to clarify the theoretical framework and strengthen the experimental evidence. The main changes are as follows:

  • We clarified the roles of ψ\psi and uu, provided further theoretical motivation and supporting evidence, and elaborated on the distinction between our framework and DENSE.
  • We added a section connecting the theoretical results to AMP, explaining the motivation for local mass weighting and the selection of hyperparameters.
  • We expanded the experiments from 5 to 10 random seeds, added large scale benchmarks beyond our regular server's capacity, and investigated random forest based similarity as an alternative weighting criterion.
  • We provided detailed experimental protocols, including data preprocessing, partitioning strategies, hyperparameter selection, and evaluation metrics, to improve reproducibility.

W1, Q1, Q3: Theoretical Abstractions, DENSE Comparison, and Hyperparameters

Role and architectural generality of ψ\psi and uu: Upfront, ψ\psi and uu are analytical abstractions, never estimated in practice, used solely to bound how context sharding affects prediction reconstruction. Recent TFMs diverge sharply: TabPFN-3 uses column wise encoding then row compression, while LimiX-2 keeps cell level representations throughout its backbone with distinct label conditioning and readouts, so a model specific derivation lacks generality.

Grounding in TFM representations and deep refinement: Balef et al. (2026) show early TFM layers perform latent mapping for residual stream computation, motivating embedding labeled pairs via ψ\psi into a shared Hilbert space U\mathcal{U}. Following Tsai et al. (2019), who cast attention as normalized kernel smoothing, full context weights aD(x,(xi,yi))a _ {\mathcal{D}}(x, (x _ i, y _ i)) define a discrete measure νD(x,⋅)\nu _ {\mathcal{D}}(x, \cdot) for readout u(x,D)=∫X×Yψ(xi,yi) νD(x,d(xi,yi))u(x, \mathcal{D}) = \int _ {\mathcal{X}\times\mathcal{Y}} \psi(x _ i, y _ i)\, \nu _ {\mathcal{D}}(x, d(x _ i, y _ i)). This also captures deep layers: Balef et al. show middle to late layers iteratively refine features, while Cannistraci and Vogt (2026) find replacing half the blocks with dataset specific affine maps costs ≤0.002\le 0.002 median AUROC across eight models; as shared affine maps commute with normalized weighted integration under fixed weights, multi layer refinement reduces to uu. Classical Nadaraya Watson regression admits an exact shard wise convex decomposition under a shared kernel, motivating Definition 2 and local mass as a proxy for aD/aDka _ {\mathcal{D}}/a _ {\mathcal{D} _ k} (Theorem 2, Section 3.3).

Fundamental distinctions from DENSE: First, DENSE (Sections 3.2, 3.3) targets LLMs rather than TFMs. Second, DENSE's heuristic weight wi=1∣Bi∣∑xj∈Bicos⁡(xje,xe)w _ i = \frac{1}{|\mathcal{B} _ i|} \sum _ {x _ j \in \mathcal{B} _ i} \cos(x _ j^e, x^e) is mathematically flawed: shard wide averaging dilutes local signals in large buckets, and sign cancellation risks zero or negative weights. Conversely, we build a framework in Theorem 2: using local mass nk(x0,T)/nn _ k(x _ 0, T)/n strictly guarantees valid convex weights in ΔK−1\Delta^{K-1}. Third, the tasks are distinct: we reconstruct the full context predictive distribution f(x,D)f(x, \mathcal{D}) from partitioned shards.

Hyperparameter calibration and distribution recovery: Here, α\alpha calibrates empirical counts to optimal weights; TT and α\alpha are tuned via 5 fold CV on the training split without test labels or test full context outputs (Appendix). Since TFMs natively output predictive distributions, each shard yields an estimate pk(y∣x)p _ k(y\vert{}x) whose convex combination p^(y∣x)=∑k=1Kλkpk(y∣x)\hat{p}(y\vert{}x) = \sum _ {k=1}^K \lambda _ k p _ k(y\vert{}x) remains a valid distribution; we directly compute TV and CDF MSE between p^(y∣x)\hat{p}(y\vert{}x) and full context p(y∣x)p(y\vert{}x) to measure recovery. Optimizing Oracle weights against full context outputs confirms convex combinations recover full context predictions with minimal error, with residual error captured by Eu\mathcal{E} _ u and L2Vλ\frac{L}{2}\mathcal{V} _ \lambda (Theorem 1).

Sharding configuration, regression alignment, and Section 5: We also report the exact sharding configuration (Algorithm 2 in Appendix B.1) and added a full sensitivity analysis over TT and α\alpha across all small scale datasets (Figure 2 and Appendix C). For regression tasks, we align the shard level predictive distributions (bucketed or quantile representations) onto a common support grid to compute the convex mixture p^(y∣x)\hat{p}(y\vert{}x) and evaluate TV and CDF MSE directly against the full context distribution. Finally, we agree with the reviewer's remark on Section 5: nonzero Oracle error reflects that f(x,D)f(x, \mathcal{D}) lies outside the convex hull of {f(x,Dk)}\{f(x, \mathcal{D} _ k)\}, which can arise from both cross shard interactions and readout nonlinearity (L2Vλ\frac{L}{2}\mathcal{V} _ \lambda in Theorem 1); we have refined the discussion in Section 5 accordingly.

W2, Q2, Q4: Large Scale Benchmarks, Downstream Gap, Tree Similarity, and Preprocessing

Fixed shards vs. active retrieval: In Section 3.2, active retrieval like LoCalPFN (Thomas et al., 2024) directly shrinks ∥aD−λkaDk∥\|a _ {\mathcal{D}} - \lambda _ k a _ {\mathcal{D} _ k}\| (Theorem 2) and Vλ\mathcal{V} _ \lambda (Theorem 1). As LoCalPFN thoroughly studied active retrieval, we use random sharding to model decentralized data on fixed shards, isolating query adaptive weights.

Expanded 10 seed and large scale benchmarks: We expanded all runs from 5 to 10 seeds, confirming consistent gains over Equal and FedAvg, and added large datasets exceeding our server's memory (using a high memory server for full context baselines); on these tasks, AMP runs within shard limits, beats shard baselines, and recovers full context accuracy.

Downstream gap and tree based similarity: Trailing full context inference on some tasks is expected: beyond the irreducible residual L2Vλ\frac{L}{2}\mathcal{V} _ \lambda (Theorem 1), raw L2L _ 2 is an imperfect proxy for aD/aDka _ {\mathcal{D}}/a _ {\mathcal{D} _ k} (Theorem 2). Better proxies shrink this gap: replacing L2L _ 2 with tree leaf cooccurrence similarity captures X→yX \to y structure and consistently beats L2L _ 2 in reconstruction fidelity and downstream accuracy.

L2L _ 2 preprocessing protocol and open question: For L2L _ 2 preprocessing, we fit strictly on training features: missing numerical values are median imputed (0 if all missing) and standardized via zj=(xj−μj)/σjz _ j = (x _ j - \mu _ j)/\sigma _ j (ddof=0\text{ddof}=0, divisor 1 if σj=0\sigma _ j=0); categorical features are one hot encoded in {0,1}\{0,1\} with a missing bin (unseen test categories set to 0), concatenated into a float64 matrix. Ultimately, distance heuristics are not our core focus; rather, our work asks an open question: what external proxies best approximate a TFM's implicit in-context similarity?

Reviewer_5VdXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces AMP, a method for addressing the context length limitations of Tabular foundation models. By dividing the dataset into subsets, AMP proposes applying the model to each part of the dataset individually, and then combining the predictions using a convex combination motivated by theoretical results.

The presentation and writing of the paper need improvement. The experimental setup lacks many details (T, alpha, how is the partitioning of the data performed). Furthermore, the transition from theory to the proposed algorithm is too abrupt. The approximation used should be better justified as a replacement for the attention-based metric. Finally, while the paper aims to improve the efficiency of tabular models, it does not present results on the effectiveness of its method or an end-to-end comparison of execution times with other baselines.

Critical strengths

1/The authors propose a method for scaling tabular foundation models to large training datasets. The proposed approach can indeed lead to substantial speedups by processing each subset of the data in parallel and aggregating the distributions using the proposed approach.

2/ The paper presents several interesting theoretical results to motivate their framework.

Critical weaknesses

1/ Further explanations and improvements to the paper's writing are needed. The connection between the theory and the algorithm requires further explanation. Which psi embedding is used in practice? How is the dataset partitioned? Does the Tabular Foundation Model satisfy the proposed assumptions (definition of f)?

2/ The experimental results do not appear to be sufficient to justify adopting the method. The datasets used contain only a small number of samples. This number of samples allows for direct evaluation of the model. In addition, the gains appear to be quite small compared to the other baselines.

3/ The proposed AMP procedure, while interesting, seems a bit weak. The authors propose computing euclidean distances between an observation of interest and nearby points, and then using the ratio of these points (distributed across shards) to learn the weights. Euclidian distance-based procedures (KNN, etc) have significant limitations in high-dimensional settings + runs into computational issues when the number of data points is large. The paper is positioned to help tabular foundation models in large data settings, so it is unclear if this weighting method would apply. At the very least, some computational feasibility study is needed, showing a case where tabular foundation models exceed the context limit but the AMP metrics can still be computed.

Questions for authors

See concerns above.

1/Could you clarify which u and psi are used? 2/How is the data split determined? 3/How do you justify using neighbor counts based on the Euclidean distance between inputs? 4/What are the time savings of AMP compared to other baselines? 5/ What is the motivation behind using just the inputs to determine distance. For tabpfn for example, does require context examples where the response is known to learn the relationships between X and y, before predictions can be made on X alone. It may be interesting to explore if learning distances based on some metric that involves the response may be helpful, or learning shards/partitions based on that as well. Tree-based methods may be a natural candidate here. Existing work (https://openreview.net/pdf?id=0ZALcXRKAc) use tree-based models to learn context information to provide foundation models before prediction on tabular datasets.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 5VdX: Thank You for Your Constructive Feedback and Valuable Expertise.

We thank the reviewer for the constructive comments. We have revised the manuscript to improve the clarity of the methodology and strengthen the experimental evaluation. The main changes are as follows:

  • We simplified the theoretical presentation by removing redundant results and notation, and provided further explanations and supporting evidence for the key concepts ψ\psi and uu.
  • We clarified the connection between the theoretical framework and AMP, explaining the motivation for local mass weighting, the choice of similarity measures, and the partitioning strategy.
  • We added large scale experiments to demonstrate the computational feasibility of AMP and expanded the evaluation from 5 to 10 random seeds.
  • Following the reviewer's suggestion, we introduced a random forest based similarity measure to incorporate feature response relationships, improving both reconstruction fidelity and predictive performance.

We particularly appreciate the reviewer's valuable suggestion regarding tree based similarity, which has helped us further improve our method and strengthen its empirical evaluation.

W1/Q1:

Architectural generality of ψ\psi and uu: The abstractions ψ\psi and uu are designed to analyze the convex decomposition deficit across diverse architectures. Recent TFMs diverge sharply in design: TabPFN-3 performs column wise encoding followed by row compression, whereas LimiX-2 maintains cell level representations throughout its backbone, alongside distinct label conditioning and prediction head designs. Consequently, a layer by layer derivation tied to any single implementation lacks architectural generality.

Grounding in TFM representations and deep refinement: Balef et al. (ICML 2026) show that early TFM layers perform latent mapping (projecting inputs into representations suited for subsequent residual stream computation), motivating us to embed labeled examples via ψ\psi into a common Hilbert space U\mathcal{U} shared across architectures. Next, following Tsai et al. (2019), who cast attention as normalized kernel smoothing, we introduce full context dependent probability weights aD(x,(xi,yi))a _ {\mathcal{D}}(x, (x _ i, y _ i)) to define an attention induced discrete measure νD(x,⋅)\nu _ {\mathcal{D}}(x, \cdot) and represent the task adapted readout via u(x,D)=∫X×Yψ(xi,yi) νD(x,d(xi,yi))u(x, \mathcal{D}) = \int _ {\mathcal{X}\times\mathcal{Y}} \psi(x _ i, y _ i)\, \nu _ {\mathcal{D}}(x, d(x _ i, y _ i)). This single stage aggregation also captures deep TFM computation: Balef et al. reveal that middle to late TFM layers perform iterative representation refinement, while Cannistraci and Vogt (2026) find that replacing half of the Transformer blocks with dataset specific affine maps incurs at most 0.002 median AUROC loss across eight architectures. Because shared affine transformations commute with normalized weighted integration under fixed weights, multi layer ICL refinement structurally reduces to the aggregated representation uu.

Connection to Nadaraya Watson and AMP weights: Classical Nadaraya Watson (NW) kernel regression admits an exact convex decomposition across shards under a shared kernel (zero decomposition deficit). This both motivates our convex decomposition deficit (Definition 2) as an exact benchmark and, since the exact TFM attention ratios aD/aDka _ {\mathcal{D}}/a _ {\mathcal{D} _ k} are unknown across shards, inspires using query local mass as a proxy similarity measure to construct aggregation weights (Theorem 2 and Section 3.3).

W1/Q2:

Decentralized sharding vs. active retrieval: In our theoretical framework (Section 3.2), active locality based retrieval such as LoCalPFN (Thomas et al., NeurIPS 2024) acts as a special case that directly reduces ∥aD−λkaDk∥\|a _ {\mathcal{D}} - \lambda _ k a _ {\mathcal{D} _ k}\| (Theorem 2) and Vλ\mathcal{V} _ \lambda (Theorem 1), shrinking the error bound. Since LoCalPFN has thoroughly evaluated active retrieval, we use uniform random sharding to simulate the complementary setting where data are naturally decentralized across fixed shards, isolating the effect of query adaptive weights.

W2:

Expanded 10 seed evaluation: We expanded all experiments from 5 to 10 random seeds, confirming consistent gains over Equal and FedAvg.

Large scale benchmarks: We added large scale datasets that exceed our regular server's memory and context limits (for which we rented a high memory professional server to run full context benchmarks). On these large datasets, AMP runs smoothly within per shard capacity limits, outperforms shard baselines, and effectively recovers full context performance.

W3/Q3/Q4/Q5:

Metric agnostic design & L2L _ 2 (Q3 & W3): By Theorem 2 and Remark 2, λk(x)\lambda _ k(x) approximates the attention ratio aD/aDka _ {\mathcal{D}} / a _ {\mathcal{D} _ k} via shard to total local mass. Neighbor counting decouples AMP from any specific metric; standardized L2L _ 2 (inspired by LoCalPFN) was simply a training free default.

Tree based similarity (Q5 & W3): Using tree leaf cooccurrence as the similarity criterion incorporates X→yX \to y relationships and consistently outperforms L2L _ 2 in both reconstruction fidelity and downstream accuracy in our new experiments. Thank you for this suggestion, which also raises an intriguing open question: what proxy metrics best approximate a TFM's implicit in-context similarity?

Runtime & feasibility (Q4 & W3): AMP has virtually identical runtime to FedAvg and Equal. All shard baselines share the dominant cost of running the frozen TFM per shard; AMP merely replaces FedAvg's static numerator nkn _ k with the local neighbor count nk(x0,T)n _ k(x _ 0, T) (Algorithm 1; FedAvg is T=n,α=1T=n, \alpha=1), adding negligible overhead. Moreover, on the large scale datasets where full context inference exceeds our regular server's capacity, AMP executes smoothly on standard hardware.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.