- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether a shared transformer block can replace a stack of independent layers for tabular in-context learning. LoopICL repeatedly applies the same block and adjusts the number of iterations to control inference cost. The model achieves competitive performance with fewer parameters, and the experiments examine when more iterations help and how it generalizes to longer inputs.
Critical strengths
-
The three research questions are sensible and worth studying. The paper examines whether recurrence can replace independent layers, how behavior changes with more loops, and whether the resulting model can compete with strong tabular models (Section 1).
-
The experiments address these questions well. The model is competitive with strong non-recurrent Transformers, even though it is not the best overall. The study also shows when additional loops improve performance and tests generalization to longer inputs (Section 4; Appendix D.1).
-
The paper also identifies when additional loops stop helping. The experiments show performance degradation when updates are poorly scaled or too many loops are used, providing useful guidance on the limits of recurrent computation (Figures 4 and 5).
Critical weaknesses
-
[Minor]The conceptual novelty is modest. Looped Transformers and their benefits have been studied extensively in other settings, and the paper acknowledges an earlier tabular example (Section 2). I see this as a useful extension rather than a new idea. -
[Medium]The practical motivation for reducing parameter count needs to be clearer. The paper does not establish that parameter storage is a main bottleneck for existing tabular models (Sections 1 and 4.3). This makes the practical benefit less compelling, although the research question itself remains worthwhile. -
[Medium]It is not yet clear that weight sharing itself improves generalization. The recurrent and stacked models use different training-depth schedules, and the main comparison is made relatively early in training. The long-context performance gap also narrows after further training. These results do not isolate weight sharing from the effects of the training setup (Section 4.1; Appendix D.1).
Questions for authors
-
What practical bottleneck does reducing parameter count address? Please clarify when parameter storage, rather than computation or intermediate representations, limits existing tabular models.
-
How much does LoopICL's new architecture add beyond simply looping an existing tabular model? A baseline that retains an existing model's input encoder and prediction head, but replaces its independent ICL layers with a shared recurrent block, would help isolate the value of the dual-stream design.
2 comments
Thank you for the response and the revision. I will keep my positive score.
Thank you. Here is our response.
Dear reviewer z6SP, Thank you for your feedback and for raising interesting questions.
Response to Q1 and W2: What practical bottleneck does reducing parameter count address? We note that, in general scenarios, computation is the main bottleneck. A small parameter count is relevant for (1) federated learning scenarios where model size impacts communication cost and (2) embedded devices which have limited storage. However, we emphasize that our main motivation is application-agnostic: To show that computational depth (rather than parameter count) drives performance in TFMs.
Response to Q2 and W1: How much does LoopICL's new architecture add beyond simply looping an existing tabular model? (see also response to reviewer Jtqt) To study this, we trained a model that encodes columns/rows only once and then performs recurrent row-level ICL. As can be seen in Section 4.1 and Figure 3, this baseline underperforms LoopICL, highlighting the benefit of recurrently updating the cell stream along the row stream. In addition to the dual-stream, we propose early exit gates (solely based on support samples) and distributional encoding.
Response to W3: It is not yet clear that weight sharing itself improves generalization. Thank you for bringing this up. Our empirical results show that recurrent models learn to generalize (to held-out tasks from the prior in Figure 4 and to larger contexts in Figure 9) faster than stacked models. However, as you point out, final performance may be similar, and we updated the conclusion accordingly. This does not change our main finding that computational depth matters more for predictive performance than parameter count.