acceptodds
Under review as a conference paper at ICLR 2027

LoopICL: Looping a single transformer block to solve tabular tasks

Abstract

Tabular foundation models using in-context learning have recently surpassed gradient-boosted trees on predictive tabular tasks. However, recent mechanistic insights suggest that parameters in these models are largely redundant. We introduce LoopICL, a looped transformer whose core design decouples parameter count from computational depth. LoopICL consists of a single block, processing data through two coupled streams: a cell stream capturing per-cell feature representations and a row stream capturing in-context example representations, jointly refined through within-column and cross-column attention. During pre-training, we vary loop counts, allowing the block to be unrolled for a varying number of iterations at test-time and use a learned exit-gate to automatically exit. In its standard setting, LoopICL performs competitively with TabICLv2 on TabArena and TALENT at the same computational cost (FLOPs), while using nearly 90% fewer parameters. Furthermore, its recurrent design enables users to also trade off inference cost and performance, providing a resource-aware TFM.

Primary area:
foundation or frontier models, including LLMs
Keywords:
tabular foundation model, tecurrent Transformer, test-time computation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_z6SPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether a shared transformer block can replace a stack of independent layers for tabular in-context learning. LoopICL repeatedly applies the same block and adjusts the number of iterations to control inference cost. The model achieves competitive performance with fewer parameters, and the experiments examine when more iterations help and how it generalizes to longer inputs.

Critical strengths
  1. The three research questions are sensible and worth studying. The paper examines whether recurrence can replace independent layers, how behavior changes with more loops, and whether the resulting model can compete with strong tabular models (Section 1).

  2. The experiments address these questions well. The model is competitive with strong non-recurrent Transformers, even though it is not the best overall. The study also shows when additional loops improve performance and tests generalization to longer inputs (Section 4; Appendix D.1).

  3. The paper also identifies when additional loops stop helping. The experiments show performance degradation when updates are poorly scaled or too many loops are used, providing useful guidance on the limits of recurrent computation (Figures 4 and 5).

Critical weaknesses
  1. [Minor] The conceptual novelty is modest. Looped Transformers and their benefits have been studied extensively in other settings, and the paper acknowledges an earlier tabular example (Section 2). I see this as a useful extension rather than a new idea.

  2. [Medium] The practical motivation for reducing parameter count needs to be clearer. The paper does not establish that parameter storage is a main bottleneck for existing tabular models (Sections 1 and 4.3). This makes the practical benefit less compelling, although the research question itself remains worthwhile.

  3. [Medium] It is not yet clear that weight sharing itself improves generalization. The recurrent and stacked models use different training-depth schedules, and the main comparison is made relatively early in training. The long-context performance gap also narrows after further training. These results do not isolate weight sharing from the effects of the training setup (Section 4.1; Appendix D.1).

Questions for authors
  1. What practical bottleneck does reducing parameter count address? Please clarify when parameter storage, rather than computation or intermediate representations, limits existing tabular models.

  2. How much does LoopICL's new architecture add beyond simply looping an existing tabular model? A baseline that retains an existing model's input encoder and prediction head, but replaces its independent ICL layers with a shared recurrent block, would help isolate the value of the dual-stream design.

2 comments
AuthorsOpenReview ↗

Thank you. Here is our response.

Dear reviewer z6SP, Thank you for your feedback and for raising interesting questions.

Response to Q1 and W2: What practical bottleneck does reducing parameter count address? We note that, in general scenarios, computation is the main bottleneck. A small parameter count is relevant for (1) federated learning scenarios where model size impacts communication cost and (2) embedded devices which have limited storage. However, we emphasize that our main motivation is application-agnostic: To show that computational depth (rather than parameter count) drives performance in TFMs.


Response to Q2 and W1: How much does LoopICL's new architecture add beyond simply looping an existing tabular model? (see also response to reviewer Jtqt) To study this, we trained a model that encodes columns/rows only once and then performs recurrent row-level ICL. As can be seen in Section 4.1 and Figure 3, this baseline underperforms LoopICL, highlighting the benefit of recurrently updating the cell stream along the row stream. In addition to the dual-stream, we propose early exit gates (solely based on support samples) and distributional encoding.


Response to W3: It is not yet clear that weight sharing itself improves generalization. Thank you for bringing this up. Our empirical results show that recurrent models learn to generalize (to held-out tasks from the prior in Figure 4 and to larger contexts in Figure 9) faster than stacked models. However, as you point out, final performance may be similar, and we updated the conclusion accordingly. This does not change our main finding that computational depth matters more for predictive performance than parameter count.

Thank you for the response and the revision. I will keep my positive score.

Reviewer_JtqtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper asks whether a tabular ICL model can keep strong performance while sharing parameters across computation steps. LoopICL applies a single Transformer block repeatedly to coupled cell and row representations, adds distributional input conditioning, and supports either a fixed loop budget or a learned early exit. The authors run ablations on architecture and scaling, analyze the learned representations, and compare against leading tabular models on TabArena and TALENT. With 12 loops, the model is competitive with TabICLv2 while using far fewer parameters.

Critical strengths

S1. LoopICL reaches competitive results on both TabArena and TALENT with far fewer parameters than the leading tabular foundation models (about 3.25M vs. tens of millions). On TALENT, the 12-loop model has a higher Elo than TabICLv2 (1719 vs. 1666). I find it convincing that parameter sharing still works at this scale.

S2. The loop-count and residual-scaling experiments are useful for understanding how accuracy changes as more computation is added. The early-exit results are also practical, since the inference cost can be adjusted per dataset.

Critical weaknesses

W1. Is the recurrent cell-stream update actually needed? LoopICL is competitive, but I can't tell from the results whether the repeated updating of the cell stream is what helps. A simple baseline would settle this: encode columns/rows once, then run recurrent row-level ICL on top. This is especially relevant since the gap between the recurrent and stacked models shrinks a lot after Stage 2 (Appendix D.1).

W2. Inference cost is not fully shown. The parameter savings are clear. But each LoopICL iteration re-runs the column-wise, cross-column and readout computation, whereas TabICLv2 runs its encoders once, so I'm not convinced inference compute is comparable. The TALENT timings on the same hardware help, yet full-model FLOPs and peak GPU memory are missing, and the TabArena runtimes come from different hardware, so they can't answer this. Fewer parameters don't necessarily mean a smaller memory footprint at inference.

Questions for authors

Q1. Section 4.2 suggests that α=1/L\alpha=1/L remains useful at higher loop counts, yet the main benchmark evaluates only α=1\alpha=1. Why was α=1\alpha=1 chosen for the main results? The paper's argument for it is that it allows exiting without fixing LL in advance. But in practice a maximum budget LL is often fixed anyway, and in that setting α=1/L\alpha=1/L seems like a natural alternative. Specifically:

(a) Could α=1/L\alpha=1/L also be paired with an exit gate? If there is a technical obstacle (for example, the scale of the accumulated output changing across iterations), please explain it.

(b) If it can, how would $ \alpha=1/L $ with an exit gate compare with $ \alpha=1 $ at the same performance–latency budget? A comparison on the same hardware, for example on TALENT, would be most informative.

(c) If $ \alpha=1/L $ is better at higher loop counts but is not used in the main benchmark, is the reported α=1\alpha=1 setting a conservative choice, or is there a cost (such as worse performance at small $ L $ or less robust early exit) that is not shown?

Q2. How was the 12-loop configuration chosen? Section 4.3 calls it the best-performing fixed-loop setting, which suggests it was selected on the evaluation benchmarks themselves. Please clarify whether a separate validation set was used. Also, what loop-count distribution is used in Stage 2? The main text says Stage 2 runs for 30K steps, while Appendix C says 40K.

Q3. The paper describes Balef et al. (2026) as a small-scale proof of concept for recurrent tabular models. Beyond scale, what exactly is new in LoopICL's contribution (e.g., the coupled cell/row streams, distributional conditioning, the exit gate)? I would like the "first ... at scale" claim to be tied to specific methodological differences.

2 comments
AuthorsOpenReview ↗

Thank you. Here is our response.

Dear reviewer Jtqt, Thank you for your feedback and for raising interesting questions.

Response to Q1. We thank the reviewer for this insightful question. With α=1/L\alpha=1/L, performance will peak around L iterations (no matter what L is set to; see Figure 5 right). The model distributes computations across all iterations and, as a result, the performance of exiting at loop n<L depends on L; i.e., for a high L, each loop contributes little, and for a low L, the individual contribution is larger. This policy generalizes beyond the pre-training limit and, thus, α=1/L\alpha=1/L is preferred if we want to spend exactly L iterations on the solution. In contrast, α=1\alpha=1 results in a model that greedily solves the task in as few loops as necessary within the pre-training limit, making early exit effective.

(a) With α=1/L\alpha=1/L, progress is distributed evenly across updates and does not converge (i.e., iteration 1 and L−1L-1 contribute similarly to the solution). Effective early-exit mechanisms rely on detecting convergence and thus are not expected to trigger early exit in this case. Noteworthy, this is not a technical issue, but rather a methodological one.

(b) We added an additional experiment in Appendix D.2, where we assume the oracle early exit exists; in this case, as seen α=1\alpha=1 is a better choice compared to α=1/L\alpha=1/L.

(c) The setting α=1/L\alpha=1/L generalizes to loop counts beyond the pre-training setting. Concretely, with α=1\alpha=1, performance for large L decreases, whereas with α=1/L\alpha=1/L it remains stable (see the blue line vs. the others in Figure 5, left). However, most importantly, the best achieved predictive performance is comparable across both settings (see stars in Figure 5, left) Thus, our primary motivation for choosing α=1\alpha=1 is its compatibility with early exit and faster convergence.


Response to Q2. We assume this question refers to the setting investigated in Section 4.3 and specifically to assessing the early-exit gate (if not, please let us know). For this, we wanted to assess our early exit relative to the most competitive fixed-loop setting, which is 12 loops according to TALENT and Tabarena. For completeness, we add more number of loops for comparison in Appendix D.7. For Stage 2, we use the distribution as for Stage 1 (see Figure 2), cutting off at 8 loops. The number of epochs is 30K; we correct this number in the appendix.


Response to Q3. The main methodological differences are (1) training with a different number of loops to enable adapting computational depth at test time and (2) using an early exit gate (solely based on support samples) to enable resource efficiency. Furthermore, we introduced two coupled streams and distributional encoding to stabilize training and improve performance. We refined our claim to “the first recurrent tabular ICL architecture at scale enabling automatic adaptation of computational depth at test time [...]”


Regarding W1. We thank the reviewer for the suggestion. We add the proposed baseline, which encodes columns/rows only once (and also looping it) and then performs recurrent row-level ICL. This baseline underperforms LoopICL, highlighting the benefit of recurrently updating the cell stream. We also clarify that the recurrent and stacked models share the same two-stream architecture, a distinction we will make clearer in the revised paper.


Regarding W2. We thank the reviewer for raising this concern. We add asymptotic cost of inference and hidden-state memory consumption estimates to Appendix A.6. Our two-stream architecture has a higher peak memory compared to single-stream ones since it preserves the feature space (the term CRE in hidden-state memory in Table 2 Appendix A.6). The inference cost on the TALENT benchmark is mitigated by two factors: LoopICL uses only one layer for column-wise and cross-column processing per loop step (compared to three in TabICL) and 8 estimators (compared to 32 in TabICL). We will clarify these trade-offs in the revised paper.

Thank you to the authors for the detailed explanations. After carefully reading your response, I have no further questions. I will maintain my original score.

Reviewer_Dh3bOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines the use of a recurrent transformer architecture for tabular in-context learning. The paper has a core method contribution (LoopICL) which is a recurrent transformer model based on a looped transformer block, with empirical analysis on the mechanism of the model. Experiments indicate that the LoopICL model performs competitively with TacICLv2 on tabular benchmarks such as TabArena with fewer parameters.

Critical strengths

The principal strength in this paper is its experimental results, which will likely be of interest for the community given recent focus on scaling TFMs to larger architectures (e.g. TabPFN-3) and tasks, making questions of parameter efficiency of core interest in this community. The trade-off of inference cost is an additional benefit which may see practical interest.

The core methodological insight for the paper (that a shared recurrent block can match the performance of larger TFMs) will be of interest to the tabular community, and highlights the potential for further efficiency improvements. It is possible that the result may also be of interest within the broader looped transformer community.

The overall motivation for the method is clearly described, with related literature appropriate, contemporary, and clearly explained. Overall the text is written clearly. The method and experimental details is explained clearly. While figure presentation has significant issues that should be addressed during the rebuttal, the overall presentation of the paper is relatively clearly organised.

Critical weaknesses

Figure presentation has critical issues. The text for figures is in general much too small to be read in print-out or on a screen without significant zooming in. This is an issue for almost all figures (both in the main paper and the Appendix) and presents a significant presentation issue in the current manuscript. Many of the figures (e.g. Figure 5) are not clear to read and should be reorganised to make the key message to the reader. This is overall the most significant issue with the current manuscript and should be addressed to recommend acceptance.

Questions for authors

While the literature for recurrent tabular models is well described, it would be useful to provide additional description on how the proposed method differs from these approaches.

It would be useful to include table comparisons of LoopICL relative to benchmark methods. Currently results are only presented in visual form, making the core claim of a 90% parameter improvement less observable.

During the discussion period it would be useful for the authors to commit to improving the figures. I am happy to increase my score on this basis.

3 comments
AuthorsOpenReview ↗

Thank you. Here is our response.

Dear Reviewer Dh3b,

Thanks for your positive assessment. We increased figures throughout the paper and added a results table for the results on Tabarena/TALENT (Table 3 and Table 4 in Appendix D.6) and clarified how our approach differs from recurrent tabular models.

Regarding the table comparison, we added Appendix A.6, explaining the differences in feature-space representation, parameters, and inference complexity.

I thank the authors for their amendments to the paper. There is a notable improvement.

I find Figure 5 and 6 difficult to understand. Could the authors speak to this?

In general the presentation is still not clear for many figures. For example, in Figure 8 it is clear that text has been cut-off or occluded ('Faster worse perf' etc). I strongly recommend the authors to review each figure manually, as the level of polish currently impedes understanding and is below standard conference level. I also recommend checking tables (e.g. Table 3 in the Appendix) as this has clearly been scaled and does not fit with the template text size.

AuthorsOpenReview ↗

Thank you for your feedback

We thank the reviewer for carefully inspecting the revised manuscript and for highlighting remaining presentation issues. We manually reviewed all figures and tables and improved their readability. In addition, we also restructured existing content, as seen in the uploaded PDF, in the experiment section to make our key message clearer:

  • We unified the structure across all experiment sections (4.1;4.2; 4.3)
  • We moved results for α=1/sqrt(L)\alpha=1/sqrt(L) to Appendix to declutter plots
  • We added a specific answer box for each research question.

Regarding your questions related to Figures 5 and 6: Figure 5 illustrates two main findings.

  • Left panel: Without residual scaling (α=1\alpha=1; blue curve), the model performs strongly within the pretraining range (i.e., performance is consistently good between 4 and 16 loops, peaking at 12). In residual scaling, α\alpha controls the convergence rate; for example, a lower α\alpha (i.e., a higher value for LL) leads to slower convergence.
  • Right panel: The violin plots show the distribution of optimal loop counts across datasets. With residual scaling, the optimal loop count increases with the available budget L and closely follows the line (y=L), suggesting that the model effectively utilizes the available computational budget.

Figure 6 complements this performance analysis by showing how internal representations evolve across loops. With residual scaling, the model updates its representations in a structured manner across iterations (validating our observation that the model learns to use available budget). Without residual scaling, most of the representation change occurs in the initial iterations, followed by smaller refinements, motivating our early-exit gate, which exploits greedy task solving and the resulting convergence behavior.

We hope that this resolves any open concerns on your side regarding clarity and presentation of results. Thank you again for your time and effort.

Comments on the paper

1 comment

First of all, we thank all reviewers for their effort, positive assessment, and helpful feedback on our work. In addition to our response, we ran additional experiments and updated the PDF (highlighting additional text in blue) with this message. We hope we have addressed your concerns and would be grateful for any further discussion or feedback you would like to share.


Here are the major changes to our paper in response to your comments:

  • Adding a single-stream baseline [Jtqt, z6SP; line 315-326]
  • Increasing font size in all figures and reorganizing them [Dh3b]
  • Add a comparison of inference complexity, memory usage, and parameter count [Jtqt, Dh3b; Table 2]
  • Add a section in the Appendix to discuss why we choose α=1\alpha =1 for early exit [Jtqt; Appendix D.2]
  • Rephrase architectural contributions [z6SP, Jtqt; line 78-87]
  • Unifying the structure across all experiment sections (4.1;4.2; 4.3) [Jtqt, z6SP, Dh3b]
  • Moving results for α=1/sqrt(L)\alpha=1/sqrt(L) to Appendix to declutter plots [Dh3b]
  • Add a specific answer box for each research question.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.