- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes an information-theoretic view of in-context learning using the language of information ratchets in thermodynamics. They map the ideas to an ICL setting to draw parallels and develop new understandings. In particular, they introduce the "learning toll" idea, which measures how much information the context contains about the latent task.
The paper uses this in two ways. Firstly, it interprets the gap between an ideal oracle and the in-context learner as an unavoidable cost of learning the task, with additional KL divergence measuring model inefficiency. This seems to harken back to the standard principles of excess loss in learning theory. Second, it compares a finite-pool memorizer against a learning of the underlying task family and derives the threshold for when memorizing the task pool stops being useful. The paper then validates this over several synthetic cases and experiments on some post-trained LMs.
Critical strengths
-
The paper is mathematically clean and the experiments are reasonably thorough. The derivations generally follow cleanly from the setup and existing information-theoretic results, and I did not find any obvious unjustified leaps. They also conduct experiments across a range of task families, contexts, model sizes, etc. which I consider an adequate experimental coverage.
-
The finite-pool memorizer vs. family-level learner comparison is interesting. Putting both predictors on the same information-theoretic ledger and deriving a task-side threshold from their loss difference is a neat idea, and the empirical agreement is surprisingly strong.
-
I do find the information-ratchet framing quite fun. It gives a real parallel for us to draw intuition and it's always interesting to see cross field comparison like this to give us possible future insights as well.
Critical weaknesses
-
The novelty does honestly seem a bit overstated, especially relative to existing information-theoretic ICL work and Wurgaft et al. A lot of the early theory is standard information-theoretic ICL machinery translated into information-ratchet language, which the authors themselves state in the appendix. The genuinely interesting part seems to be the information-theoretic characterization of the loss gap between the finite-pool memorizer and the family-level learner and the resulting threshold. However, Wurgaft et al. already consider essentially the same two strategies, with the transition explained through a loss/complexity tradeoff. My current understanding is that the main advance here is a cleaner characterization of the loss side of that competition. The model-dependent complexity side is not derived from the mutual-information analysis and instead seems to come from the assumed storage cost c_pen, which the experiments fix to one nat per context. I’m not yet convinced that the paper’s “one quantity answers both questions” framing reflects a comparably broad new insight, rather than a cleaner bookkeeping of one part of an existing story.
-
The presentation is honestly harder to follow. I understand that the ratchet language is cute but it introduces a lot of secondary vocabulary. It could be better maybe with a section drawing a comparison and having the intuition and then straight up just using information theoretic language throughout. Also, there is a bit too little discussion of future observations information ratchets could provide. If they give us further intuition and this line of reasoning can be continued further then great, otherwise the story revolving around them this heavily is a bit lackluster.
Questions for authors
-
What are the clear theoretical contributions here beyond existing information-theoretic ICL work if you take away the information-ratchet language?
-
What is the main advance over Wurgaft et al., especially given that c_pen is assumed instead of being derived?
-
What further predictions or insights could the information-ratchet framing potentially provide beyond the results in this paper?
3 comments
Thank you for the detailed response and additional experiments. I appreciate the clarifications, and I agree that the information-theoretic characterization of the loss gap and the resulting threshold are genuine contributions. The additional experiments also provide useful support for the theoretical results.
However, my main concern remains the significance of the advance relative to existing work, particularly Wurgaft et al. My view is that theoretical results under these kinds of simplified assumptions are often valuable primarily for the intuition and understanding they provide, rather than their direct applicability to real-world models. While the new threshold result gives a cleaner quantitative characterization of the competition between the two Bayesian predictors, I don't find that it substantially advances our understanding of why models select one strategy over another beyond what Wurgaft et al. already established. The additional experiments are useful, but agreement in the loss-based threshold does not necessarily demonstrate a change in the learned strategy.
I do want to emphasize that I find the information-ratchet perspective interesting, and I think the current results are legitimate contributions. However, much of the intuition and the additional predictions seem to follow naturally from standard information-theoretic reasoning. I would be interested to see this perspective developed further, especially if it can lead to new insights or predictions that are less obvious from existing frameworks.
Overall, I think this is interesting and worthwhile work, and I would encourage the authors to continue developing it. However, at this stage I still find the additional conceptual advance somewhat limited for ICLR, so I will maintain my original rating.
Thank you for the follow-up and the generous assessment.
One clarification on the learned strategy, with details in our reply to Reviewer mS7M. The new-task reading uses the trained model's own predictions on unseen tasks, not the loss margin. Where a model cannot hold its pool, it becomes an in-context learner while the ideal memorizer is still more than a nat away, so its learned strategy changes. In all sixteen settings this happens at or below the computed threshold, to within 0.05 nats.
Again, thank you for your reply.
Thank you for the constructive review. We agree with your reading of where the advance lies: it is the loss side of the competition, made exact. We adopt your suggestion on presentation, and the answers below use no ratchet vocabulary.
Q1. The theoretical contributions without the ratchet language
Theorem 1 and Proposition 1 are standard, and the revision will credit them where they are stated. Three results are new.
The revised introduction will claim these three results and no more.
"One quantity answers both questions" is meant literally. I(θ;X1:T) is the floor of question 1, and up to a constant it is the logarithm of the pool size in question 2. The nats a context must spend to learn its task are the same nats it has for telling stored tasks apart.
Q2. The advance over Wurgaft et al. [1], given that cpen is assumed
The two strategies and the loss-against-complexity form are theirs, and we do not derive the complexity side. Three things are new.
Q3. What the ratchet framing predicts beyond this paper
It has already produced one prediction that a count of tasks does not make. A ratchet prices a pool by its entropy, in the same nats as the toll. Skewing a pool of 4,096 tasks should then select memorizing with no task removed. It does (Figure 18).
The framing adds one variable that the Bayesian accounts leave out, the machine's memory. A ratchet can use only the correlations that its state keeps (Boyd et al. [2]). Two predictions follow.
The networks of Section 4 keep far more of their context than these families need (Appendix G), so the test needs models with a quantized or explicitly bounded state. The revised Discussion will state these predictions.
Presentation
The ratchet will stay as the frame in Sections 1 and 2. Section 2 will end with a table that maps each ratchet term to its information-theoretic name. Sections 3 and 4 will then use the standard names.
References
[1] Wurgaft, Lubana, Park, Tanaka, Reddy, and Goodman. In-context learning strategies emerge rationally. NeurIPS, 2025.
[2] Boyd, Mandal, and Crutchfield. Leveraging environmental correlations: The thermodynamics of requisite variety. Journal of Statistical Physics, 2017.