acceptodds
Under review as a conference paper at ICLR 2027

A Simpson's Paradox in In-Context Learning Curves

Abstract

A language model's average prediction loss often falls as it reads further into a document. This decrease is commonly used to measure in-context learning, but later positions also contain different tokens. We find a striking reversal on web text: average loss falls while loss rises in every group defined by how often a token has appeared before. Later positions contain more repeats, which have lower loss than first occurrences; their increasing share lowers the average. This is a Simpson's paradox. Across eight text domains and two Pythia sizes, domain accounts for 99% of the variance in a standard position-based score, compared with 0.2% for size. Controlled synthetic experiments also change the score more through the data than through model size. Yet models do benefit from context, and continue improving after this score stabilizes. We therefore recommend reporting token-group losses alongside the average, and measuring context benefit by comparing predictions of the same tokens with different amounts of context. An early loss decrease on first-occurrence tokens provides an additional diagnostic of improvements that the conventional score misses.

Primary area:
interpretability and explainable AI
Keywords:
in-context learning, language models, per-position loss, Simpson's paradox, scaling laws, evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_HGU2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper decomposes changes in prediction loss with document position into composition and dilution effects. Composition captures changes in the proportions of tokens with different prior occurrence counts, while dilution captures changes in the average loss within each count group. For Pythia on FineWeb, the authors observe that average loss decreases while count-conditional losses increase, which they identify as a Simpson’s paradox. They examine these measurements across model sizes, text domains and training checkpoints.

Critical strengths

None.

Critical weaknesses

The paper does not sufficiently clarify what scientific conclusion follows from the loss decomposition. The observed reversal is compatible with a simple model performing successful inference from context. Consider the add-one predictor qt(xt+1∣x≤t)=ct(xt+1)+1t+V,q_t(x_{t+1}\mid x_{\leq t})=\frac{c_t(x_{t+1})+1}{t+V}, where ct(x)c_t(x) counts occurrences of token xx among the first tt tokens and VV is the vocabulary size. Its loss at fixed count cc is ℓt+1=log⁡(t+V)−log⁡(c+1),\ell_{t+1}=\log(t+V)-\log(c+1), which increases with position. This predictor is also the Bayesian posterior predictive for a document-specific categorical distribution drawn from a uniform Dirichlet prior. In that setting, expected aggregate loss decreases as the predictor infers the document’s distribution, despite increasing loss at every fixed count. Thus, the reversal alone does not establish that the aggregate improvement gives a misleading impression of context use.

The authors acknowledge that the observed reversal does not imply a failure to use context. However, this leaves the scientific and practical implications of the decomposition underdeveloped. In particular, the recommendation to report token-group losses needs a concrete demonstration of what these measurements allow us to infer.

In addition, in my opinion, the issues raised in the following "questions for authors" section are not mere presentation issues, but rather indicators that the current submission doesn't meet the scientific standard required for publication, independently of the strength of the contribution.

Questions for authors
  1. Can the authors demonstrate a concrete use of the token-group losses to assess context use or distinguish models? In particular, what conclusion would be missed or incorrectly drawn from the aggregate loss, and how would the group losses resolve it despite the changing targets within each group?

  2. How should the controlled synthetic experiments in section 4.3 change our interpretation of the position-based score? The Bayes-optimal score itself depends on the synthetic distributions, indicating that the available benefit of context depends heavily on the data. How do these experiments distinguish a limitation of the score from a genuine difference in the predictive information supplied by context?

  3. Terminology should be consistent between the main text and figures, particularly “first occurrences” versus “novel” and “within-group change” versus “dilution.” Why use different terminology for the main text and figures/figure captions?

  4. The schematic graphics in Figures 2 and 3 are more elaborate than necessary; simpler illustrations or a direct report of the relevant measurements, possibly on a graph, would make the argument easier to follow.

  5. Why do the authors report rates per e-fold of position instead of just position?

Reviewer_NzXLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates how well ICL score, a position based loss score, measures in-context learning. Tokens are grouped by their appearance frequency earlier in the document. On the FineWeb dataset, average loss goes down with position while loss within every group goes up. The authors call this a Simpson's paradox which stems from repeated tokens at later positions which are therefore easier to predict. Experiments are run on 13 models and 8 text domains. They also claim the score misses improvements later in training, and recommend comparing loss on the same tokens with short and full context.

Critical strengths
  • Good spread of data tested, 13 models and 8 domains. This breadth helps strengthen results.
  • Paper demonstrates ICL scores across domains may differ, and therefore can't be compared. The scores differ a lot even when the benefit from context is about the same, so a score difference between domains does not mean a difference in in-context learning.
Critical weaknesses
  • On the FineWeb dataset, the 2nd control seems to show that the falling average is from the model using text more than 64 tokens back, so the "mixture shift fools the average" narrative doesn't appear to be fully supported.
  • The claims that the score "misses" improvements aren't fully tested. As it stands, you can't tell if there's a problem with the score vs models improving the same amount at each position.
Questions for authors
  • For Pythia-160M on FineWeb, how much does context beyond 64 tokens help at the early and late positions in the ICL score?
  • When tokens are grouped only by a property of the target (e.g. corpus frequency), does the reversal persist?
  • Given that Control 1 shows rising group losses, what insight should be drawn from count-group losses?
  • Can you add, for each corpus and both Pythia sizes, the ICL score of the 64-token window model and the loss reduction from full context on the same tokens?
  • Why do StackExchange and FreeLaw gain almost the same from context beyond 64 tokens, but have ICL scores of opposite sign?
  • How do the ICL score and the loss reduction from context beyond 64 tokens change across the Fig. 9 checkpoints? Please give standard errors for the differences between checkpoints.
  • In the abstract, does "continue improving after this score stabilizes" refer to overall loss?
  • When first occurrences are matched on corpus frequency, does T_new still grow with model size?
Reviewer_fKRoOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates Simpson's paradox in the evaluation of in-context learning (ICL) curves. The authors observe that although the average prediction loss of language models decreases with increasing context position, the loss within individual token groups, defined by their previous occurrence counts, can actually increase. The authors argue that this phenomenon is mainly caused by the growing proportion of repeated tokens at later positions. Therefore, conventional position-based ICL scores do not necessarily reflect a model's ability to use contextual information. Through experiments across different model families, text domains, and synthetic datasets, the paper demonstrates this effect and suggests using first-occurrence token losses and fixed-target context comparisons as complementary evaluation methods.

Critical strengths

One major strength of this paper is that it revisits a commonly used evaluation method from a simple but important statistical perspective. Previous studies often interpret the decrease in prediction loss with increasing context length as evidence of in-context learning. However, the authors point out that this comparison does not control for differences in token distributions across positions. By grouping tokens according to their previous occurrence counts, the paper demonstrates that the overall loss can decrease even when the loss within each group increases. This finding is meaningful for understanding ICL curves and suggests that changes in average prediction loss should not be directly interpreted as improvements in contextual learning ability.

Another strength is that the experimental analysis is relatively comprehensive. The authors do not simply present Simpson's paradox as an isolated observation but further investigate its implications across different settings. They evaluate several model families, including Pythia, GPT-2, SmolLM2, and Qwen2.5, and analyze variations across eight text domains. The results show that, for the two evaluated Pythia model sizes and eight domains, text domains explain approximately 99.3% of the variance in ICL scores, while model size explains only 0.2%. The authors also conduct controlled experiments using synthetic datasets with different token repetition patterns. These experiments provide additional evidence that data distribution has a substantial influence on the conventional ICL score.

A further strength is that the authors do not simply conclude that language models fail to benefit from longer contexts. Instead, they distinguish between changes caused by token composition and improvements resulting from actual context utilization. By restricting the context window and comparing predictions for the same target tokens, the authors show that models can indeed benefit from more distant contextual information. In addition, the early loss reduction for first-occurrence tokens provides a complementary perspective on model improvements that may not be captured by the conventional ICL score. Therefore, the paper not only identifies a limitation of an existing metric but also provides useful suggestions for future evaluation.

Critical weaknesses

The first limitation is that some of the stronger conclusions are based on a relatively limited range of model sizes and text distributions. For example, the finding that text domains explain 99.3% of the ICL score variance is mainly based on eight domains and two Pythia models, while the Transformers used in the synthetic experiments are relatively small. Although the authors acknowledge these limitations, it remains unclear whether similar observations hold for much larger modern language models, longer context windows, or instruction-tuned models. Additional experiments involving larger models and more diverse context lengths would strengthen the generalizability of the conclusions.

The second concern is that the proposed first-occurrence token loss metric may still be affected by statistical confounding factors. Although this metric avoids the changing proportion of repeated and first-occurrence tokens, it still compares different target tokens at different document positions. As the position changes, token difficulty, topic distribution, and local contextual structure may also change. Therefore, this metric seems more suitable as a complementary diagnostic rather than a direct measure of contextual learning ability. The authors already acknowledge this issue, but further analysis of the relationship between this metric and fixed-target context benefits would make the evaluation discussion more convincing.

Finally, the analysis of the relationship between repeated tokens and contextual learning remains mainly statistical. The authors use simple predictors, such as the count cache, to show that token repetition patterns can explain a substantial part of the conventional ICL score. However, this does not necessarily mean that neural language models rely on the same internal mechanisms. Different models may use different attention patterns and representations to incorporate contextual information, and the current experiments do not clearly distinguish these mechanisms. Therefore, while the paper provides convincing evidence about the limitations of the evaluation metric, its explanation of the internal mechanisms underlying contextual learning remains limited.

Questions for authors

Can the conclusion that text domains dominate the variation in ICL scores be further validated using larger language models and longer context windows?

Is there a consistent relationship between the early loss reduction for first-occurrence tokens and the context benefit measured using fixed target tokens? Could the authors further clarify the practical value of this complementary metric for comparing language models?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.