acceptodds
Under review as a conference paper at ICLR 2027

Language Models as Information Ratchets: An Information-Thermodynamic Perspective on In-Context Learning

Abstract

In-context learning raises two questions usually studied apart: how much a prompt can teach a model, and how much task diversity pretraining needs before a model learns in context rather than memorizing. Reading a language model as an information ratchet—a machine that turns predictions into work—makes one quantity answer both: the mutual information between task and context, the learning toll. The toll is a floor no model can beat. It is tight for transformers trained on a fresh task in every sequence, so their whole loss curve follows from the task family alone; frozen language models pay several times it, and what they overpay is a failure to keep learning, not a wrong prior. The toll also counts tasks: a context tells apart only about of them, so beyond that memorizing stops paying. Trained transformers switch where it says, until the pool outgrows what they can hold, and that point is set by the task family and its prior, not by the number of a task's parameters. Post-training a pretrained model shows the same selection.

Primary area:
interpretability and explainable AI
Keywords:
in-context learning, large language models, information theory, emergence, Bayesian inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UaFoOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes an information-theoretic view of in-context learning using the language of information ratchets in thermodynamics. They map the ideas to an ICL setting to draw parallels and develop new understandings. In particular, they introduce the "learning toll" idea, which measures how much information the context contains about the latent task.

The paper uses this in two ways. Firstly, it interprets the gap between an ideal oracle and the in-context learner as an unavoidable cost of learning the task, with additional KL divergence measuring model inefficiency. This seems to harken back to the standard principles of excess loss in learning theory. Second, it compares a finite-pool memorizer against a learning of the underlying task family and derives the threshold for when memorizing the task pool stops being useful. The paper then validates this over several synthetic cases and experiments on some post-trained LMs.

Critical strengths
  1. The paper is mathematically clean and the experiments are reasonably thorough. The derivations generally follow cleanly from the setup and existing information-theoretic results, and I did not find any obvious unjustified leaps. They also conduct experiments across a range of task families, contexts, model sizes, etc. which I consider an adequate experimental coverage.

  2. The finite-pool memorizer vs. family-level learner comparison is interesting. Putting both predictors on the same information-theoretic ledger and deriving a task-side threshold from their loss difference is a neat idea, and the empirical agreement is surprisingly strong.

  3. I do find the information-ratchet framing quite fun. It gives a real parallel for us to draw intuition and it's always interesting to see cross field comparison like this to give us possible future insights as well.

Critical weaknesses
  1. The novelty does honestly seem a bit overstated, especially relative to existing information-theoretic ICL work and Wurgaft et al. A lot of the early theory is standard information-theoretic ICL machinery translated into information-ratchet language, which the authors themselves state in the appendix. The genuinely interesting part seems to be the information-theoretic characterization of the loss gap between the finite-pool memorizer and the family-level learner and the resulting threshold. However, Wurgaft et al. already consider essentially the same two strategies, with the transition explained through a loss/complexity tradeoff. My current understanding is that the main advance here is a cleaner characterization of the loss side of that competition. The model-dependent complexity side is not derived from the mutual-information analysis and instead seems to come from the assumed storage cost c_pen, which the experiments fix to one nat per context. I’m not yet convinced that the paper’s “one quantity answers both questions” framing reflects a comparably broad new insight, rather than a cleaner bookkeeping of one part of an existing story.

  2. The presentation is honestly harder to follow. I understand that the ratchet language is cute but it introduces a lot of secondary vocabulary. It could be better maybe with a section drawing a comparison and having the intuition and then straight up just using information theoretic language throughout. Also, there is a bit too little discussion of future observations information ratchets could provide. If they give us further intuition and this line of reasoning can be continued further then great, otherwise the story revolving around them this heavily is a bit lackluster.

Questions for authors
  1. What are the clear theoretical contributions here beyond existing information-theoretic ICL work if you take away the information-ratchet language?

  2. What is the main advance over Wurgaft et al., especially given that c_pen is assumed instead of being derived?

  3. What further predictions or insights could the information-ratchet framing potentially provide beyond the results in this paper?

3 comments

Thank you for the constructive review. We agree with your reading of where the advance lies: it is the loss side of the competition, made exact. We adopt your suggestion on presentation, and the answers below use no ratchet vocabulary.

Q1. The theoretical contributions without the ratchet language

Theorem 1 and Proposition 1 are standard, and the revision will credit them where they are stated. Three results are new.

  1. The gap between memorizing and learning is a difference of two mutual informations (Lemma 5): Δ(K,T)=I(θ;X1:T)−E I(κ;X1:T∣pool)\Delta(K,T)=I(\theta;X_{1:T})-\mathbb E\,I(\kappa;X_{1:T}\mid\text{pool}), exactly.
  2. A two-sided bound on where the gap closes (Theorem 2): Λπ(T)−c≤ln⁡K∗≤Λπ(T)+O(1)\Lambda_\pi(T)-c\le\ln K^\ast\le\Lambda_\pi(T)+O(1), with the upper constant explicit through an exact Rényi redundancy of the Dirichlet-categorical family (Lemma 7). The threshold becomes a number computed from the task family before training.
  3. The split of a pretrained model's shortfall into prior lost and learning lost (Eq. 6), which needs the exact floor.

The revised introduction will claim these three results and no more.

"One quantity answers both questions" is meant literally. I(θ;X1:T)I(\theta;X_{1:T}) is the floor of question 1, and up to a constant it is the logarithm of the pool size in question 2. The nats a context must spend to learn its task are the same nats it has for telling stored tasks apart.

Q2. The advance over Wurgaft et al. [1], given that cpenc_{\rm pen} is assumed

The two strategies and the loss-against-complexity form are theirs, and we do not derive the complexity side. Three things are new.

  1. The loss side becomes a law. In Wurgaft et al. the constants of the trade-off are fitted to trained runs, so the threshold is located after training. We show that the loss difference is exactly a difference of mutual informations and bound where it closes. The threshold is then set by how much a context reveals about its task, and its change with context length, family, and prior is predicted before training. Two families with almost the same number of parameters grow at 2.76 and 1.45 nats per ee-fold of context, and two priors on one family differ by a factor of two. Trained models follow both (Figure 6a,b).
  2. cpenc_{\rm pen} is the level at which a bound is read. Theorem 2 holds at every level c>0c>0, and it needs no rule about training: the ideal memorizer minimizes the pretraining loss, so no model can gain more than Δ(K,T)\Delta(K,T) over the in-context learner by fitting the pool. Changing the level from 0.5 to 2 nats shifts the threshold by about a nat and leaves its slope in the mutual information between 0.98 and 1.07 (Table 1). The contrasts in item 1 therefore do not depend on the level. We will state the theorem without the rule. The model's side is measured, as the number of tasks a trained model stores, and architecture, training steps, and width move it (Section 4.2, Appendix G).
  3. New in this response: a test that uses no storage cost. The computed threshold is an upper limit that trained transformers reach on tasks they never saw (Tables A and B in the comment to all reviewers).

Q3. What the ratchet framing predicts beyond this paper

It has already produced one prediction that a count of tasks does not make. A ratchet prices a pool by its entropy, in the same nats as the toll. Skewing a pool of 4,096 tasks should then select memorizing with no task removed. It does (Figure 18).

The framing adds one variable that the Bayesian accounts leave out, the machine's memory. A ratchet can use only the correlations that its state keeps (Boyd et al. [2]). Two predictions follow.

  • A memory floor. A model that stays on the loss floor at every later position must carry Λπ(T)\Lambda_\pi(T) nats about the task in its state. A model whose state holds fewer nats cannot stay on the floor.
  • A memory cap on the switch. Such a model can tell apart at most about eCe^{C} tasks when its state holds CC nats, so its switch is capped near ln⁡K≤C\ln K\le C even when the toll is larger. The same number says how far a context can be compressed, by pruning or summarizing, before in-context learning degrades.

The networks of Section 4 keep far more of their context than these families need (Appendix G), so the test needs models with a quantized or explicitly bounded state. The revised Discussion will state these predictions.

Presentation

The ratchet will stay as the frame in Sections 1 and 2. Section 2 will end with a table that maps each ratchet term to its information-theoretic name. Sections 3 and 4 will then use the standard names.

References

[1] Wurgaft, Lubana, Park, Tanaka, Reddy, and Goodman. In-context learning strategies emerge rationally. NeurIPS, 2025.

[2] Boyd, Mandal, and Crutchfield. Leveraging environmental correlations: The thermodynamics of requisite variety. Journal of Statistical Physics, 2017.

Thank you for the detailed response and additional experiments. I appreciate the clarifications, and I agree that the information-theoretic characterization of the loss gap and the resulting threshold are genuine contributions. The additional experiments also provide useful support for the theoretical results.

However, my main concern remains the significance of the advance relative to existing work, particularly Wurgaft et al. My view is that theoretical results under these kinds of simplified assumptions are often valuable primarily for the intuition and understanding they provide, rather than their direct applicability to real-world models. While the new threshold result gives a cleaner quantitative characterization of the competition between the two Bayesian predictors, I don't find that it substantially advances our understanding of why models select one strategy over another beyond what Wurgaft et al. already established. The additional experiments are useful, but agreement in the loss-based threshold does not necessarily demonstrate a change in the learned strategy.

I do want to emphasize that I find the information-ratchet perspective interesting, and I think the current results are legitimate contributions. However, much of the intuition and the additional predictions seem to follow naturally from standard information-theoretic reasoning. I would be interested to see this perspective developed further, especially if it can lead to new insights or predictions that are less obvious from existing frameworks.

Overall, I think this is interesting and worthwhile work, and I would encourage the authors to continue developing it. However, at this stage I still find the additional conceptual advance somewhat limited for ICLR, so I will maintain my original rating.

Thank you for the follow-up and the generous assessment.

One clarification on the learned strategy, with details in our reply to Reviewer mS7M. The new-task reading uses the trained model's own predictions on unseen tasks, not the loss margin. Where a model cannot hold its pool, it becomes an in-context learner while the ideal memorizer is still more than a nat away, so its learned strategy changes. In all sixteen settings this happens at or below the computed threshold, to within 0.05 nats.

Again, thank you for your reply.

Reviewer_mS7MOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces an information-theoretic framework for in-context learning, using mutual information between the task and context to characterize both the Bayesian prediction limit and the transition from task memorization to in-context learning.

The authors derive a task-diversity threshold under certain assumptions and evaluate their framework on synthetic tasks, pretrained LLMs, and post-training experiments. The results provide empirical support for the proposed connection.

Critical strengths
  1. The paper connects the Bayesian prediction limit and the memorization-to-learning transition through mutual information. Although the underlying information-theoretic tools are largely established, the connection provides a useful quantitative perspective on two related ICL problems.

  2. The authors evaluate multiple synthetic task families, pretrained LLMs, and post-training interventions. In particular, varying task diversity and training conditions provides meaningful evidence for the proposed relationship between task information and memorization behavior.

Critical weaknesses
  1. The main information-theoretic decomposition follows established results, and the memorization-to-learning transition has already been studied in prior work. While the paper connects these two questions through mutual information, it is unclear what fundamentally new insights this unified perspective provides.

  2. The theoretical analysis does not fully support the claimed prediction of the memorization-to-learning transition. Theorem 2 assumes a storage-cost-based selection rule rather than deriving it from Transformer training dynamics. Moreover, the reported agreement between predicted and measured switching points relies on closely related loss criteria. As a result, the experiments do not fully establish that the proposed information-theoretic quantity predicts an actual change in learning strategy.

Questions for authors
  1. Given that Bayesian ICL limits and memorization-to-learning transitions have both been extensively studied, could the authors clarify what new theoretical insights or predictions their unified framework provides beyond existing approaches? In particular, what can we understand or predict using this framework that previous results could not explain?

  2. Theorem 2 assumes that pretraining selects memorization when its advantage exceeds a fixed storage cost. Could the authors clarify the justification for applying this rule to actual Transformer training, and how the predicted threshold should be interpreted given that the model's distances to the ideal memorizer and learner cross at a substantially larger task-pool size?

  3. In the Markov task family, the first symbol is sampled independently of the task, suggesting i1=0i_1=0 under Equation (4). Is the first transition instead used as the reference point when computing learning gain and efficiency in the experiments?

4 comments

Thank you for the careful reading. In short:

  • You are right that the measured switch in the submission is not independent of the prediction. We now test the prediction with a criterion that shares nothing with it, and it holds (Table A in the comment to all reviewers).
  • Theorem 2 can be stated without the selection rule, and we will state it that way.
  • The threshold is the pool size at which a model trained on the pool becomes an in-context learner of new tasks. How long the model keeps storing its pool beyond that point is a property of the model, and the crossing in Figure 5c measures it.
  • Yes, the Markov family uses the first transition as the reference.

Q1 / W1. What the framework predicts that earlier results could not

In one sentence: the task-diversity threshold is the number of tasks that a context can tell apart, about eI(θ;X1:T)e^{I(\theta;X_{1:T})}, and this number is computed from the task family before any model is trained.

Theorem 1 and Proposition 1 are known results, and the revision will credit them where they are stated. The transition itself is also known. The new object is one relation: the gap between the ideal memorizer and the ideal in-context learner is exactly a difference of two mutual informations (Lemma 5), and it closes at ln⁡K∗=Λπ(T)+O(1)\ln K^*=\Lambda_\pi(T)+O(1) with explicit constants (Theorem 2, Lemma 7). Three predictions follow, each made before training.

  1. Where the transition is. Earlier accounts need quantities fitted to trained models. The kinetic account derives a power law in context length whose exponent comes from a memorization law fitted to trained models on one task family (Nguyen and Reddy [1]). The normative account fits its constants to the trained runs (Wurgaft et al. [2]). Here the location is a number computed from the task family.
  2. Which property of the data moves it. The answer is how much a context reveals about its task. It is not the number of task parameters.
ContrastFrom parameter countFrom the toll, before trainingTrained transformers
Growth of ln⁡K∗\ln K^* per ee-fold of context, for Dice - 8 sides (dim 7) and Hidden Markov - 2 states (dim 8)3.5 and 4.02.76 and 1.452.10 and 1.53 (Figure 6b)
Switch under Dir(0.3) relative to Dir(1), same familyequalabout twiceabout twice (Figure 6a)

A parameter count gives the wrong order in the first contrast and no difference in the second. Trained transformers follow the toll in both.

  1. When the model matters and when it does not. The switch has two limits: the tasks a context can tell apart, and the tasks a trained model can store. Comparing them says in advance whether architecture and training budget will move the switch. At T=32T=32 a transformer, a linear-attention model, a GRU, and an LSTM store 211 to 495 tasks, and all four switch at 26 to 27 tasks, against a computed 28. At T=128T=128 the computed value is 203, which is close to what the attention models store. The two attention models now switch at 133 and 134 tasks, and the GRU and the LSTM, which store 420 and 642, switch at 162 and 178. Accounts that tie the threshold to a model's kinetics or capacity do not say when the model stops mattering.

The same unit gives two results on pretrained LLMs. A frozen model's shortfall splits into prior lost and learning lost, and instruction tuning raises only the second (Section 3.2). The threshold is a statement about the entropy of the pool, so a skewed pool of 4,096 tasks should select memorizing with no task removed, and it does (Figure 18).

Q2 / W2. The selection rule, the loss criteria, and the crossing at a larger pool

(a) The selection rule is not needed. The content of Theorem 2 is a property of the task family. For every level c>0c>0, let Kc(T)K_c(T) be the pool size at which the two ideal predictors come within cc nats per context on pretraining sequences. Then ln⁡Kc(T)≥Λπ(T)−c\ln K_c(T)\ge\Lambda_\pi(T)-c for every family, and ln⁡Kc(T)≤Λπ(T)+c2(A)−cπ(A)+ln⁡2ec−1+o(1)\ln K_c(T)\le\Lambda_\pi(T)+c_2(A)-c_\pi(A)+\ln\frac{2}{e^{c}-1}+o(1) for Dirichlet-categorical families (Steps 2-4 of the proof). The ideal memorizer minimizes the pretraining loss. Therefore no model, whatever its architecture or optimizer, can gain more than Δ(K,T)\Delta(K,T) nats per context over the ideal in-context learner by fitting the pool, and beyond KcK_c it can gain at most cc. The rule in the submission is this bound read at one level. We will state the theorem in this form and move the rule to a remark. Whether training reaches the bound is then an empirical question, answered in (b) and (c).

(b) The loss criteria. We agree with this criticism. The measured switch of the submission is the pool size at which the margin, less the model's distance to the memorizer, falls to one nat. It contains the margin, so it cannot exceed the computed switch, and it equals the computed switch whenever the model is the memorizer. Its agreement shows only that training collects the saving. Table A replaces it with an independent reading. The measured quantity is the trained model's own loss on tasks it never saw, and the computed quantity involves no trained model. The two agree to 0.11 nats wherever the model can store the pool, and the computed value is an upper limit in all sixteen settings. The agreement does not depend on the level: at 0.5 and 2 nats it is 0.13 and 0.23 nats. Theorem 2 is proved for pretraining sequences. On new tasks the same law holds numerically, with a slope of 0.98 to 1.07 and an offset 0.2 to 0.9 nats higher (Table 1).

(c) Actual transformer training. Table B follows the switch through training. It rises to the computed value and stays there where the model can store the pool, and it is still rising at the last step where it cannot. Four architectures arrive at the same value at T=32T=32 (item 3 above).

(d) The crossing at a larger pool. The two quantities answer different questions.

  • K∗K^\ast is where the two ideal predictors become the same predictor to within a nat. Past it, the ideal memorizer of a pool is itself an in-context learner of new tasks, because the stored tasks are too close together for a context to separate and the memorizer averages over the nearby ones. A model trained on the pool therefore generalizes to new tasks past K∗K^\ast whether or not it still stores the pool. Table A measures this.
  • The crossing in Figure 5c asks which of the two ideal predictors the model is nearer to. Where the model can store the pool, the two distances cross between nearly identical predictors: at the crossing each distance is at most a quarter of a nat (median 0.06 nats, against tolls of 1.8 to 5.1 nats). Where it cannot, the crossing lies at 1.2 to 2.1 times the number of tasks the model stores. The crossing therefore reads the model's storage. It is the second limit of Section 4.2.

Raventós et al. [3] showed that below the threshold a trained model follows the Bayes predictor of its pool, and we see the same: on new tasks the distance between the two is a median 2.5% of the memorizer's own excess loss. What the toll adds is where that predictor itself turns into an in-context learner. The change at K∗K^* is thus a change in what the best solution of the pretraining objective does on new tasks, and trained models follow it. The revision will state this reading in Section 4.1, report the switch on new tasks as the main result, and present the crossing as the measure of storage.

Q3. The first symbol of the Markov family

Yes. The first symbol is uniform, so i1=0i_1=0, and the first position that carries information about the task is the first transition. All gains and efficiencies of this family use it as the reference, gt=i2−itg_t=i_2-i_t, with i2=ln⁡3−(ψ(4)−ψ(2))=0.27i_2=\ln 3-(\psi(4)-\psi(2))=0.27 nats exactly. The other seven families use t=1t=1. Eq. (4) will read gt=it0−itg_t=i_{t_0}-i_t, with t0t_0 the first position that carries task information, and Appendix C.4 will list t0t_0 for each family. No reported value changes.

References

[1] Nguyen and Reddy. Differential learning kinetics govern the transition from memorization to generalization during in-context learning. ICLR, 2025.

[2] Wurgaft, Lubana, Park, Tanaka, Reddy, and Goodman. In-context learning strategies emerge rationally. NeurIPS, 2025.

[3] Raventós, Paul, Chen, and Ganguli. Pretraining task diversity and the emergence of non-Bayesian in-context learning for regression. NeurIPS, 2023.

Thank you for the response. The Markov clarification resolves my concern, and the new evaluation provides stronger empirical support.

However, the threshold seems to tell us when the memorizer performs similarly to the ideal learner, rather than when a trained model changes its learning strategy. I remain unconvinced that the unified framework provides substantial additional insight over existing work. And I also agree with the other reviewer about the presentation. The ratchet analogy is interesting, but the extra terminology often makes the underlying ideas harder to follow, especially for readers without a thermodynamics background. I suggest the authors make the paper more accessible to the broader ML community. I will raise my rating, but still lean toward rejection.

Thank you for reading our response and for raising your rating. We address the two remaining points.

What the threshold says about a trained model's strategy

You are right about what K∗K^* is. It is the pool size at which the ideal memorizer of the pool predicts new tasks as the ideal in-context learner does, to within a nat per context. This is why K∗K^* can be computed before training. What it says about a trained model depends on whether the model can hold the pool. In both cases it is a statement about the model's own predictions on new tasks, which is the criterion of Raventós et al. (2023).

  • A model that can hold the pool changes strategy at K∗K^*. Below K∗K^* such a model predicts new tasks as its ideal memorizer does. Its distance from the memorizer is a median 2.5% of the memorizer's own distance from the learner. It therefore assigns a new context to a stored task, and it stops doing so where the memorizer stops. In these eight settings the measured switch agrees with K∗K^* to 0.11 nats (Table A). The network needs no second algorithm for this transition, because the best memorizer of a large enough pool is itself an in-context learner.
  • A model that cannot hold the pool changes strategy earlier, and K∗K^* bounds where. In the other eight settings, a trained model becomes an in-context learner of new tasks while its ideal memorizer is still more than a nat away. For example, on Dice - 8 sides at T=128T=128 the model switches at 916 tasks, against a computed 10,676. This is a change of strategy by the model itself. In all sixteen settings K∗K^* is a ceiling on it, exceeded by at most 0.05 nats. More training moves the switch up toward the ceiling: four times the steps take it from 291 to 737 tasks on Dice - 4 sides at T=512T=512, against a computed 1,554 (Table B).

The framework therefore separates two kinds of transition before training: the ones the task family fixes, whose location it predicts, and the ones that reflect the model's limits, which it bounds.

What the unified view adds

The view gives two results that earlier accounts do not give before training for a standard transformer.

  • Where the model can hold the pool, the transition is a property of the task family. Its location is the number of tasks a context can tell apart, ln⁡K∗=Λπ(T)+O(1)\ln K^*=\Lambda_\pi(T)+O(1). Its growth with context length differs between two families with almost the same number of parameters: 2.76 and 1.45 nats per ee-fold of context, and 2.10 and 1.53 in the trained transformers (Figure 6b).
  • The same number answers both questions. Λπ(T)\Lambda_\pi(T) is the floor on what any model loses on a new task (Theorem 1), and it sets the task diversity at which the memorizer becomes a learner (Theorem 2). The worth of a context and the diversity that in-context learning needs are therefore one quantity, computed from the task family.

Presentation

We agree, and the final version will go further than the term table.

  • The abstract and introduction will state both questions and both results in standard terms: mutual information, KL divergence, and Bayes-optimal predictors.
  • Section 2 will start from the loss decomposition and introduce the ratchet in one paragraph, as the reading that motivates it. The derivation of work will move to an appendix.
  • Sections 3 and 4 will use only standard names, plus the learning toll, the learning gain, and the learning efficiency, each defined in one sentence where it is first used.

Thank you again for the detailed engagement.

Reviewer_ZqcaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Inspired by Maxwell’s demon, this paper develops an information-ratchet framework to study two important questions in in-context learning: how much a prompt can teach a model, and how diverse pretraining tasks must be before a model learns from context rather than memorizing tasks. The paper draws two main conclusions. First, the “learning toll” lower-bounds a model’s excess loss, regardless of its size or training method. Second, a context can distinguish only about e^{toll} tasks; beyond this scale, memorizing the pretraining task pool becomes less beneficial than learning from context. The authors use synthetic experiments with trained and pretrained models to test these predictions.

Critical strengths
  1. The paper addresses two important and interesting questions in in-context learning and offers answers from a novel theoretical perspective.
  2. The synthetic experiments provide empirical support for the theoretical predictions.
Critical weaknesses
  1. Although the paper evaluates pretrained language models, its main conclusions are supported primarily by synthetic tasks. Figure 19 also reports cases in which the proposed measure does not generalize to broader in-context learning performance. The limitations and scope of the approach could be discussed more thoroughly.
Questions for authors
  1. Could the authors verify the theoretical predictions on more controlled real applications instead of synthetic tasks?
3 comments

Thank you for the positive assessment and for the questions on scope.

Real applications

An exact floor needs a task whose generating process is known. This is why the task families are synthetic: each claim can then be tested to a few hundredths of a nat. Real models and real text enter in three places, the last of which is new.

  • Frozen LLMs. Thirty public models from 0.5 to 33 billion parameters are scored against the floor in Section 3.2, and fifty in Appendix F.
  • Post-training of production models. Qwen2.5-0.5B and Qwen2.5-1.5B are post-trained on sequences that are half text (Section 4.3). A pool of four tasks moves them to the memorizing side while their loss on held-out text improves, and sixty-four or more make them near-ideal learners.
  • New: an exact floor on a real-text task. The label-mapping task of Appendix I uses real news text (AG News). Each prompt shows demonstrations of the form "Text: ... / Label: X" under a random assignment of the four classes to four labels. The assignment is the task, and its generating process is known, so the floor is exact here too. The ideal learner needs eight demonstrations to learn the assignment (0.008 nats left). We re-read the records of the 36 frozen models against this floor. After 64 demonstrations the best model (Qwen2.5-32B) is still 0.29 nats above it and has taken 79% of the learning gain. The median model has taken 38%. The floor takes the class of each text as known, so part of this excess is reading the text and the rest is learning the assignment. This is the finding of Section 3.2 on real text: frozen models pay far more than the task requires, and the floor says by how much.

The second prediction, the switch, is tested on real models by the post-training of Section 4.3, and now on tasks the trained model never saw, which is how in-context learning is used (Table A in the comment to all reviewers). We have not tested it on a real-text task. Label mappings over real text make that test possible, because their toll can be computed, and the Scope paragraph will name it as the next step.

Author note (delete before posting). If E1 is run and its prediction P1 passes, replace the two sentences "We have not tested ... next step." with the E1 sentences of experiments.md, Section 6.

Figure 19

Figure 19 fixes what learning efficiency measures. It measures one ability: how closely a model tracks the posterior over a process that it has been told nothing about, on an absolute scale with an exact floor. Across models it agrees with the label-mapping gain and with 16-shot accuracy on real text (Spearman correlations of 0.57 and 0.63), and model scale carries this agreement. Within one lineage, post-training moves learning efficiency and the label-mapping gain in opposite directions in eight of nine steps. The two are therefore distinct abilities, and post-training raises one while it lowers the other. A single few-shot score cannot show this, and an absolute scale can. The revision will state this reading in the main text.

Limitations and scope

The revised Discussion will have a Scope paragraph with three statements. The exact toll requires a task with a known generating process. Learning efficiency measures posterior tracking specifically, as Figure 19 shows. Predicting the number of tasks a model stores from its architecture and training budget is left to later work.

Presentation

You marked the paper as difficult to follow. Section 2 will end with a table that maps each ratchet term to its information-theoretic name, and Sections 3 and 4 will use the standard names.

Thanks for the additional results. I would suggest the author to further improve the paper with these suggestions. I choose to maintain my score.

Thank you for your reply. Yes, we will improve the paper based on all comments we received.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Comment to all reviewers

We thank the three reviewers. Two of them ask whether the predicted switch is tested by a criterion that is independent of the prediction, and whether it holds for actual training. We answer with one analysis of the runs already in the paper. No model was retrained.

Definition: the switch on new tasks

The switch on new tasks is the pool size at which a predictor's loss on tasks outside the pool comes within one nat per context of the ideal in-context learner. This is the criterion of Raventós et al. (2023). We read it twice. The computed value applies it to the ideal memorizer of the pool, before any training (the "diss." columns of Table 1). The measured value applies it to the trained transformer. The computed value contains no trained model, and the measured value contains no margin and no pretraining sequence.

The paper states that the switch has two limits (Section 4.2): the tasks a context can tell apart, which the toll computes, and the tasks a trained model stores (the paper's "holds"). The second is the pool size up to which the model still reproduces the ideal memorizer on its pretraining sequences. It is measured on pretraining sequences, so it is independent of the measured column, which uses new tasks only. The column "Limit" says which of the two is smaller in each setting.

Table A. The switch on new tasks, in tasks

Three seeds per setting, with the range over seeds in parentheses.

FamilyTToll (nats)ComputedMeasuredModel storesLimit
Dice - 4 sides162.29.810.1 (9-11)212context
323.12826 (23-29)211context
644.07063 (58-73)199context
1285.0203134 (126-152)181model
2566.0538232 (218-249)176model
5127.01,554291 (263-346)166model
Dice - 8 sides163.52725 (22-27)272context
325.1154118 (114-120)363context
647.11,235465 (458-471)424model
1289.210,676916 (899-926)470model
25611.5116,2041,462 (1,380-1,531)468model
Hidden Markov - 2 states161.87.68.0 (7-9)100context
322.71919 (14-25)115context
643.95046 (43-52)132context
1285.1221140 (127-149)138model
2566.7920519 (423-622)112model
  • Where the context is the limit, the computed and measured values agree. In these eight settings the root-mean-square gap is 0.11 nats in ln K, and the largest gap is a factor of 1.3. They span three families and computed switches from 8 to 154 tasks.
  • Where the model is the limit, it becomes an in-context learner earlier, as the two-limit account says. Within each family the shortfall grows as the computed value moves further above what the model stores, from a factor of 1.5 (Dice - 4 sides, T = 128) to a factor of 80 (Dice - 8 sides, T = 256).
  • The computed value is an upper limit in all sixteen settings. The measured switch exceeds it nowhere by more than 0.05 nats.

Table B. The measured switch on new tasks along training

Setting2.5k5k10k15k20k25k stepsComputed
Dice - 4 sides, T = 169.29.29.710.09.510.19.8
Dice - 4 sides, T = 3223242527272628
Dice - 8 sides, T = 1621232324242527
Hidden Markov - 2 states, T = 3218181818181919
Dice - 4 sides, T = 51267881492142422911,554
Dice - 8 sides, T = 12819525146462782391610,676

Each entry is the mean of three seeds. Training moves the switch up to the computed value and not past it. In the first four rows, where the context is the limit, it has arrived and stays. In the last two, where the model is the limit, it is still rising when training ends. One run of Dice - 4 sides, T = 512, trained four times longer (one seed) reaches 737 tasks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.