- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper claims to introduce a theory by which to characterise ICL's limits.
Critical strengths
The paper is grammatically well written, as well as well-motivated. Exploring when ICL breaks is decidedly an interesting problem.
Critical weaknesses
I find a few issues with this paper. First, the writing is extremely unscientific and subjective, and technically I would argue that the results/experiments (which are underspecified) do not support the thesis. But, more importantly, the claims are very strong for what the paper shows (or it is already known).
I do appreciate the effort the authors put into this specific problem, which--at least to me--is compelling. However, as it stands, the authors need to put in more effort in this work so it meets the bar. Here's how I would improve this paper:
Regarding writing:
- The paper is severely informal for an article showcasing theory.
- For example, Eq. 1. There is a statement that , but then the rest of the equation is entirely underspecified (is a vector? A complex matrix? A tensor over ? (L127). On the same line, is a function mapping what to what? What is the observability index? These are just examples from the first paragraph.
- The proof style is not formal. There are two propositions (without proofs or justifications) and then a theorem (unrelated) without even a proof sketch. Full proofs do belong in the appendix, but you need at least a sketch. The propositions are stated to be the 'ingredients' for the treatment,
- You need to introduce your background better. The introduction starts throwing in terminology without providing an intuitive (or motivational) background first. The conclusion is essentially symbolic in this work. It is meant to tell us what the outcomes of your work are, not closing a narrative (which is as it reads now).
- The writing itself is unscientific:
- The section titles are leading. You should put the interpretation of the results in the discussion section, not in the results headers. For example,L350: 'Head to head with Dai (2025): the boundary is latent memory depth'
- There are quite a few references missing with respect to how in-context learning has been characterised. I would argue that most of the works (including the ones missing) that have done that also include the limitations for ICL.
- Do not start an abstract with a question.It is not just style (which is unacademic), it simply does not fulfill the definition of an abstract: a condensed version of your paper. Clip out numbers and keep it readable.
- Claims are too strong or obvious:
- Additionally:
- It is unclear to me what are your experiments. You have the models (without citations), and then a slew of numbers without actually telling the reader what you measured, how you measured, which hypothesis you are testing, etc.
Fundamentally, I think that this paper lacks positioning, and it is not objective enough to be a scientific contribution. The problem is interesting but it really seems that it needed considerable more time to be a paper at the level of ICLR.
Questions for authors
I think one of the most fundamental questions I have for the authors is how does this compare to previous work. I see some minor comparisons in the related work section, but it does not fully position it with respect to other existing paradigms/explorations--what are they lacking? What are your shortcomings?
Response to Reviewer aALS (1/2): W1 & Q1
Thank you for a review that located exactly where the claims outran the evidence. Your points are of three kinds: a gap in the proofs (W1/Q1), a distinction drawn too sharply (W2/Q2), and interpretations beyond the data (W3/Q3, Q4). We narrowed the first, redefined the second, and measured or withdrew the third. This part covers W1/Q1; part 2 the rest.
Reply to W1 & Q1 (Theorem 3 and Corollary 2). You are right on all three counts: HKZ's Condition 1 needs the single-symbol emission matrix to have rank r, impossible for r = 4 with 3 symbols; their Condition 3 (positivity) is needed for the KL guarantee; and nothing in the cited results connects a single trajectory to their i.i.d. sample size. A complete proof was out of reach in the discussion period, so we narrowed the claims as you suggested (App. A, "Discrete sequences"):
We state the small constants rather than hide them. Part 2 follows.