- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether a model gain inference capabilities across representations. This paper construct a latent parity task, where the embeddings can change but the latent geometries remain the same. Theoretically, this paper demonstrate that Bayesian prediction requires supports, while additive predictions requires to achieve a constant loss. Empirical results on looped ICL and Set ICL suggests persistent representations benefit but they cannot transfer to unseen ones completely. Additional elimination transitions trained on Set Transformer experiments are discussed.
Critical strengths
- This paper provides a well-controlled in context learning setup, where intervene the embedding does not change the latent prediction rule.
Critical weaknesses
-
The hierarchy structure defined in Section 3.1 is insufficiently explored. The main comparisons regarding sample efficiency focus on Bayesian predictor and additive predictor lies in . However, Transformer models are much more expressive than class since they can have higher order label interactions. Therefore, the theoretical results shows limited relevance to empirical results.
-
The experiments section requires further clarification in the main text. For instance, Section 6 introduces elimination transitions and recurrent Set Transformers, but without clear description of new transitions, model architecture, etc. Further explanation on these essential components are necessary.
Questions for authors
Please refer to the weaknesses part.
-
In Theorem 6, the sample complexity results are only about Bayes and . Is it possible to develop similar sample complexity results for general ?
-
Could the authors further explain how theoretical results in Section 4 connected to provided empirical results?
4 comments
Thank you again for your constructive review. As the discussion period draws to a close, we would welcome your thoughts on whether the revisions address your original concerns, and whether they warrant an updated assessment.
In particular, we have made the general interaction-order sample-complexity results explicit (Section 4.2), strengthened their connection to the empirical analyses of learned inference (Section 5), and substantially expanded the methodological explanation of recurrent elimination and its controls (Section 6).
We would be happy to clarify any remaining questions before the discussion closes.
Thank you for the additional results. My remaining concern is that the bounds for give limited insight into how many support examples a trained Transformer needs to achieve a fixed risk as the latent rank grows. I will maintain my current score.
Thank you for clarifying your concern. We agree that deriving support-sample complexity bounds for trained Transformers would be an important next step towards a fuller learning-theoretic understanding of ICL.
Our work provides a foundation for that question by separating statistical opportunity from learned, reusable inference. Even when tasks require the same underlying computation and have identical oracle support requirements, our experiments show that endpoint-trained Transformers can exploit the available evidence under one representation without reliably transferring that ability to another. Process supervision improves this transfer, demonstrating that the way inference is learned matters alongside the number of supports available.
We believe establishing this distinction is a necessary step towards explaining trained-model sample efficiency: before characterising how many examples a model needs, we must distinguish the information those examples provide from the inference capability the model has actually acquired.
We hope this clarifies how our results contribute to a broader learning-theoretic understanding of ICL.
Interaction Hierarchy and Empirical Connection
We thank the reviewer for recognising the value of the controlled representation intervention and for the questions about the theoretical hierarchy and its empirical interpretation. We have revised Sections 3–6 to make these connections and the experimental methodology more explicit.
1. General interaction orders and sample complexity. Our theory is not restricted to additive predictors. Theorem 6 characterises optimal risk at every interaction order: R(Vq)=Pr(kmin>q). Theorems 8–9 (Appendix D.4) further establish finite-length bounds and an interaction-order phase transition. In particular, for any fixed q≥1 and target risk ϵ∈(0,1), these bounds imply nϵ(Vq)=Θ(2r/q). We revised this and make the general-order consequence explicit in Section 4.2.
2. Connection between theory and experiments. We agree that Transformers can exploit higher-order label interactions. Our question is how much of this statistical opportunity they learn to exploit, and whether that capability transfers across representations. The interaction hierarchy provides an oracle benchmark, not a restriction on Transformer expressivity. V1 is an optimal additive reference, not a restriction on the learned models. The gap R(V1)−R⋆ measures the total statistical benefit available beyond additive label dependence, and our empirical metric η measures the fraction of this opportunity recovered by a trained model. Positive population-level recovery therefore certifies useful nonadditive label dependence, without claiming to identify the model's internal interaction order or inference algorithm.
The experiments further investigate what prevents expressive learners from recovering this statistical advantage. Section 5.1 shows that both contextual architectures can exploit joint-label evidence when its useful organisation is supplied, but struggle when they must discover the relevant relations themselves. At rank 6, revealing the exact route increases recovered opportunity from 0.002 under ordinary endpoint training to 0.998 (Appendix B.3, Figures 13–14). This implicates relation discovery as an important bottleneck, rather than an inherent inability to combine labels.
Additionally, Appendix B.3 (Figure 13) evaluates performance conditional on the minimum required interaction order kmin. These results examine learned performance at different required interaction orders, rather than relying solely on the aggregate Bayes–additive comparison. Recovery also deteriorates with rank at fixed kminindicating that interaction order alone does not explain the empirical difficulty. Importantly, Proposition 4 establishes that the entire oracle interaction-risk hierarchy is unchanged across embeddings. The representation experiments test differences in learners' ability to exploit the same statistical opportunity, rather than differences in available information. We have clarified these connections in Sections 4–5.
3. Elimination transitions and architecture. We agree that Section 6 previously relied too heavily on the appendix. The revised section now describes the mutable state, pivot selection and XOR elimination updates, and the recurrent four-layer Transformer that predicts successive states. Training combines final-answer, state-transition and pivot supervision, with teacher forcing annealed away. Evaluation is fully autonomous, without teacher-provided transitions or routes. We also clarify the distinction between architecture-matched endpoint controls and the stronger curriculum comparison (full implementation in Appendix A.6).
4. Connection to real-data experiments. We have additionally evaluated two pretrained tabular ICL models across five real datasets (new Section 7, Table 2). These experiments extend the central question motivating our theory: whether learned inference remains effective when the representation changes but the predictive information available in context is preserved. Orthogonal feature rotations provide an information-preserving intervention, and we observe dataset-dependent changes in predictive performance and in the benefit obtained from additional support examples. Unlike latent parity these tasks do not admit the same tractable Bayes and interaction-order benchmarks. The new experiments test the broader relevance of representation sensitivity, rather than validating the specific interaction-order mechanism or measuring the recovery of its theoretical advantage.
We hope these revisions resolve the concerns about theoretical scope, empirical relevance and methodological clarity, and would welcome your reconsideration of the overall assessment in light of the revised manuscript.