acceptodds
Withdrawn from ICLR 2027

What Do Tabular Foundation Models Compute In Context? In-Situ Representation Refinement through Attention-Gated Updates

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A tabular foundation model must discover which distinctions matter for each new table without updating its parameters. We develop in-situ representation refinement: support labels guide changes to the episode's representations, improving the information available to later queries. A regularized leave-one-out objective yields a support correction and its query extension. The leading term separates attention-based reading from state-dependent scaling, motivating RefineICL: an attention-gated, FFN-free contextual stack with selected low-rank feature interaction and typed memory. A direct intervention tests the role of evolving support states: removing one intermediate support update while preserving the block's query output increases final query cross-entropy in all 72 tested episodes. RefineICL-L24 reaches 0.93836 OVR-AUC and 0.87173 accuracy on AMLB29. A benchmark-informed continuation reaches 1644.8 Elo on the 38-dataset TabArena snapshot, 31.4 Elo above TabPFN-3 under the same evaluation. It also improves all four reported metrics over TabPFN-v3 on both TabZilla views. In a matched 100K-update depth grid, an expanded FFN gives no consistent validation benefit and uses 60.2% more peak inference memory at L8. These results connect learning within a forward pass to representation refinement and show how this view guides a competitive, memory-efficient model.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Tabular Learning, In-Context Learning, Foundation Models, Representation Refinement, Attention-Gated Updates, Scaling Laws

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_KuxyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

In the proposed approach, the authors introduce an adaptation of the contextual representations in order to support the prediction of unlabeled queries. For that, the authors introduce an in-situ representation refinement step, in which supporting labels provide information for modification of hidden representations within a single forward pass, and without updating the pretrained model parameters.

The manuscript formulates a leave-one-out objective to learn representations of supporting samples, i.e., each support sample is predicted from other supporting samples. The modified representation can then be used also for unlabeled queries via a kernel-based formulation.

Next, the authors propose separating two branches, a attention-like weighted aggregation of supporting samples and a scaling factor.

Critical strengths

S1: Adaptive representation refinement

The idea of questioning the representations of support examples is in itself interesting, rather than solely relying on the implicit predictive performance of the model.

Critical weaknesses

W1: Missing theoretical justification

If the argument from the proposed pipeline is that updating a training sample’s representation during a forward pass can give a lift for later predictions, then I’d assume that the same thing also happens in standard transformer. A conventional architecture also updates its internal representation at every layer. With the intervention in the mid-stack, it seems that the experiments show evidence that these representation transformation matter, but why is it considered that this is unique for the proposed approach? Maybe a clearer distinction between the benefits of the gate would shine some light on this claim, but currently I do not see the fundamental difference.

W2: Missing ablation on the block selection

The next thing, I’m wondering is the assumption of the mid-stack intervention. In related work like in exit networks, we see investigations where the major changes happen in the first and the last few layers depending also on the downstream task. So what was the reason behind the design choice to pick a mid-stack intervention. I’m missing a thorough ablation on this point. And also why just a single intervention and not k-many of them?

W3: Flaws in the AI-generated text

we also see flaws in the presentation of the paper. In equation 2 there is a rendering error or a missing package. The text in the figures (e.g., Fig2) are not properly aligned and overlap sometimes with other elements, i.e., seems highly llm generated. Equation 11 is a rendering error again etc. .

The conclusion is also just a paragraph instead of a separate section. No limitations are discussed.

W4: Recommended statements are missing

According to the guidelines, it is also recommended to put a ethics statement and a reproducibility statement in the text, both are missing.

W5: Figure 4 and its experimental setup

“We permute the attention output, gate, or their paired update across queries before residual addition.” The authors discuss now that those permutations reduce the overall accuracy. However, isnt this behavior also expected in any neural architectures that are fed with some contextual information, i.e., where each token receives a state-dependent information. So the experiment shows that updates are well-aligned with some query representations. But what the experiment does not show is what the claim of the paper is, i.e., it does not evaluate the proposed representation refinement heuristic from just some ordinary processing. So, what is the difference for example to some conventional attention-based architectures or why is it assumed that the refinement mechanism outperforms any other contextual processing?

W6: Component analysis

So, at the end of the day, it is not clear from the experimental setup which components now account for the improved performance w.r.t. established TFMs. Maybe the authors can work on this analysis in a major revision of the manuscript. Appdx F.3 suggest that the contextual gate has stronger impact than individual feature-interaction. But does this component better performance than other design choices under comparable conditions… that is not clear to me.

Questions for authors
  • Q1: Can the authors provide a discussion on the justification of the proposed approach and how it fundamentally delineates from conventional modification we see in ordinary transformer architectures? (W1)

  • Q2: Did the authors conduct experiment on the placement of the proposed intervention? In the manuscript the authors chose for a mid-stack intervention, but did the authors conduct an ablation on this? (W2)

  • Q3: Can the authors provide a better understanding of the benefit of the proposed approach compared to conventional attention-based architectures? (W5)

  • Q4: Did the authors conduct a more thorough analysis of the contributions of each of the models’ components to investigate which part actually could provide a lift? (W6)

Reviewer_dKPFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates what tabular foundation models compute during in-context learning, focusing on how labelled support examples influence the representations used to predict unseen queries. The authors propose in-situ representation refinement, where support labels guide successive updates to the representations of examples without updating the pretrained model parameters.

The theoretical motivation comes from a regularized leave-one-out objective that encourages support examples to have representations that better reflect their class relationships. The authors derive a support-state correction and extend it to unlabeled queries through a kernel-based approximation. This leads to a decomposition between attention-based information retrieval and state-dependent update magnitude.

Motivated by this decomposition, the paper introduces RefineICL, a tabular foundation model with an attention-gated contextual stack that omits standalone expanded feed-forward networks (FFNs). The architecture additionally uses low-rank feature interactions, row compression, and typed-memory retrieval.

The authors evaluate RefineICL across AMLB29, TabArena, OpenML-CC18, Grinsztajn, and TabZilla classification benchmarks.

Critical strengths
  1. The strongest contribution is the targeted intervention in Section 6.2. The authors preserve the query representation after block 12 while replacing the updated support representations with their previous states, allowing later blocks to execute normally. This increases final query cross-entropy in all 72 evaluated synthetic episodes. This is a particularly valuable because it separates the effect of updating support representations from the block's immediate update to the query. Additional evidence comes from improved nearest-prototype classification across deeper layers.

  2. The proposed architecture achieves strong performance across several established tabular classification benchmarks. RefineICL-L24 exceeds TabPFN-v3 on some datasets, while obtaining the highest Elo among the listed sub-200M-parameter ICL models in the TabArena snapshot.

Critical weaknesses
  1. The theoretical representation-refinement mechanism is not convincingly shown. The proposed architecture is motivated by a regularized leave-one-out objective, whose locally optimal correction can be extended to queries through kernel similarities. However, the actual network does not explicitly compute this objective, its gradient, or the derived update operator. The support-state intervention convincingly demonstrates that support updates matter, but it does not establish that the specific leave-one-out optimization interpretation explains their contribution. This distinction is consequential because the paper's central scientific contribution is framed around understanding what TFMs compute in context, rather than merely proposing a competitive architecture. A stronger connection between the derived correction and learned updates, or a more explicit separation of theoretical motivation from mechanistic identification, would improve the paper.

  2. Unclear relationship between representation refinement and the proposed optimization mechanism. The paper motivates RefineICL through a gradient-based representation-refinement objective, but the actual architecture implements learned attention-gated residual transformations rather than explicitly optimizing this objective during inference. It remains unclear whether the proposed mechanism constitutes a distinct form of in-context optimization or simply a learned representation transformation induced by the modified architecture.

Questions for authors

Same as weakness

Reviewer_55BvOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies tabular in-context learning as task-specific representation refinement. It derives a regularized correction to support representations and extends it to unlabeled queries, motivating RefineICL: an attention-gated architecture with low-rank feature interactions, typed memory, and no standalone expanded contextual FFN. Validation includes public classification benchmarks, controlled FFN comparisons, and interventions examining how evolving support states contribute to query predictions.

Critical strengths
  1. The support-update intervention is a useful mechanism test. Preserving a block’s query output while reverting its support output worsens subsequent predictions across all 72 tested episodes. This provides relatively direct evidence that evolving support representations contribute to prediction.

  2. The proposed architecture achieves competitive results across several classification benchmarks. The controlled FFN comparison also raises an interesting practical question about whether expanded FFNs justify their memory cost in tabular foundation models.

Critical weaknesses
  1. The motivation needs sharper presentation. The abstract introduces several concepts before clearly explaining the underlying problem. I found it difficult to identify what existing explanations of tabular ICL leave unresolved and how the proposed refinement account addresses that gap.

  2. Figure 1 could communicate more efficiently. It occupies substantial space, but provides limited architectural detail beyond the surrounding text. A more compact overview with clearer information flow would be more helpful. Some notation needs a local definition. In Equation (2), I assume (\mathbb{1}[y_j=y_i]) is an indicator function. Please define it explicitly rather than leaving readers to infer its meaning.

  3. The scaling evidence is difficult to interpret. L24 uses width 1024, whereas L12 uses width 2048, and their training histories also differ. Although the paper acknowledges this, the comparison leaves the effects of depth and width unclear. I would like to see depth scaling at fixed width, alongside a controlled width comparison.

  4. The FFN result deserves deeper analysis. Removing an expanded FFN could offer an appealing combination of fewer parameters and competitive performance. However, the reported effects are mixed: FFNs worsen validation CE at L2/L4 but improve it at L8. The small-model, single-seed comparison does not explain when or why this happens. It also changes FFNs across multiple modules together. Further analysis should distinguish optimization-budget effects, regularization, and interactions with the multiplicative gate.

  5. Figures 3–5 do not sufficiently distinguish RefineICL from other TFMs. In particular, Figure 3B demonstrates sensitivity to a representation bottleneck, while Figure 4B and Figure 5 demonstrate label-conditioned task adaptation. These observations are informative, but they do not directly motivate the specific gating, memory, or FFN choices. I’m still missing a clear comparison showing where RefineICL behaves differently or offers a consistent advantage over existing TFMs. The support-update intervention is more directly relevant to the central claim and deserves greater emphasis.

Questions for authors
  1. Does RefineICL show consistent scaling gains when depth is increased at fixed hidden width and under a clearly specified training budget?

  2. Why does adding an FFN help at some depths but hurt at others? Does this behavior persist across training seeds, longer training, and module-specific FFN ablations?

  3. Would other TFMs exhibit similar results under the probes in Figures 3–5? Which results specifically support the proposed architecture, rather than general ICL behavior?

  4. Can a matched comparison between attention-plus-FFN, scalar-gated attention, and vector-gated attention clarify the contribution of the proposed update structure?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.