acceptodds
Under review as a conference paper at ICLR 2027

In-context learning across the logistic family

Abstract

Prior theoretical studies of in-context learning (ICL) in transformers have formulated task inference in a regression setting, where tasks are weight vectors in parameter space and the data are independently sampled from a known distribution. In contrast, time series prediction involves observations generated from an evolving latent system that can exhibit temporal dependencies while the available representation may only preserve part of the underlying state. We study ICL in transformers trained to perform next token prediction on trajectories generated by the logistic family. Using the logistic family as a controlled setting, we investigate how training-task diversity, context length, and observation resolution affect model performance while retaining the latent structure of the task. At a fixed trajectory budget, generalization to new maps emerges sharply beyond a critical level of training-task diversity. The resulting generalization structure depends on representation: finer resolutions require greater task diversity to generalize but attain lower limiting error. Under restricted training support, transfer is strongly asymmetric. Models trained on intervals containing chaotic dynamics often predict lower-complexity systems accurately, whereas reverse transfer is poor. Our results show that in-context generalization depends jointly on task diversity, observation representation, and the dynamical regimes represented during training.

Primary area:
learning on time series and dynamical systems
Keywords:
In-Context Learning, Dynamical Systems, Time-series prediction, Transformers, Generalization, Tokenization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_68V6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper extends the notion of task diversity in in-context learning, traditionally studied for function regression, to the ICL of dynamical systems. They restrict to the clear case of logistic maps and empirically explore how performance (including generalization) depends on various factors such as task diversity, context length, and observation resolution. Like the standard ICL case, they also identify a task-transition.

Critical strengths

The paper is written well, with clear exposition, figures, and flow. The question is well-motivated and the experiments are conducted appropriately.

The findings on asymmetric transfer are interesting and may inspire future work, seemingly related to importance sampling or curriculum learning for learning dynamical systems.

Critical weaknesses

I am concerned about the novelty of this work. The logistic map is a good toy case and does demonstrate the desired properties, but it alone seems lacking for a complete empirical work. Task diversity is a known idea, this just seems like repeating existing studies in a slightly different setting (which is okay in general, but the setting here is overly simple and the results are consequently not unexpected).

The same transformer architecture is used throughout the experiments, without any results or ablation studies about expressivity, capacity, or training.

Questions for authors
  1. I understand that in the r<1r<1 regime trajectories converge to the fixed point at zero; why does that automatically suggest you should exclude r<0.5r<0.5 for the experiments?

  2. The uniform bin tokenization seems unmotivated (though this could be my lack of knowledge). Why is it used at all here? Why not just predict directly on the real-valued trajectory? I’m familiar with the necessity of tokenization in some settings but I don’t see it here, especially since you’re training from scratch. You even use squared loss for one choice of the objective. The only additional benefit I see is the tradeoff of resolution with generalization, but that doesn’t seem especially surprising: the bin tokenization smooths the data which prevents overfitting.

  3. Can you justify the terminal one-step objective over a sequential loss? I would think that the goal of ICL for a Markovian dynamical system is the approximate trajectories through its autoregressive generation, not just the next point. It feels like the benefit of one model to process variable context lengths at inference is discarded here.

  4. Since the logistic map is Markovian, is this substantively different from the standard ICL setup (x1,fr(x1),x2,fr(x2),…,xL,fr(xL),xL+1),(x_1, f_r(x_1), x_2, f_r(x_2), \ldots, x_L, f_r(x_L), x_{L+1}), just with xi+1=fr(xi)x_{i+1}=f_r(x_i) (i.e. the context inputs are generated by the dynamic)? Of course the logistic map here is quadratic (not linear), which makes it a bit more complex, but it is still just a one-parameter map, and there have been studies of ICL of nonlinear functions in more general cases.

1 comment

We would like to thank the reviewer for their careful consideration and helpful comments, we believe that the questions asked have enabled us to significantly improve the paper.

The same transformer architecture is used throughout the experiments, without any results or ablation studies about expressivity, capacity, or training.

To address this concern, we trained new square-loss models at every resolution with widths ranging from 32 to 256, covering 0.05M to 3.2M parameters. The task threshold appears at every width and only width 32 is limited by capacity. The results can be seen in (Appendix A.1)

I understand that in the regime trajectories converge to the fixed point at zero; why does that automatically suggest you should exclude for the experiments?

For r<1r < 1 the orbit converges to the fixed point at zero, and for r<0.5r < 0.5 this convergence happens within about six steps. Thus, at 6464 bins, over 9696% of context windows are all zeros followed by a zero target. These tasks are indistinguishable from one another in token space, so they add trivially predictable examples but no task diversity. We keep [0.5,1)[0.5, 1) to represent this regime. This is clarified in §1 ("Why the logistic family?").

The uniform bin tokenization seems unmotivated. Why is it used at all here? Why not just predict directly on the real-valued trajectory?

With real values, two consecutive observations determine the task exactly, r=xn+1/(xn(1−xn))r = x_{n+1} / (x_n (1 - x_n)), and a transformer given continuous inputs reaches the mean squared error floor quickly. Tokenizing the space reduces observation precision and makes in-context inference non-trivial, and it is the object of our study. Uniform bins have also been used in tokenized time-series foundation models such as Chronos. We added this motivation to §1 and the Methods.

Can you justify the terminal one-step objective over a sequential loss?

We trained the same models with the loss at every position (Appendix A.2). They generalize to new tasks at smaller m, but reach a much worse error. The terminal objective therefore gives both the cleaner measurement and the more accurate models.

We attribute the worse error to the short-context positions. Although the map is first order in x, the tokenized process is not Markov, thus the next token depends on several preceding tokens, because they constrain where x lies within its bin. Predictions made early in the context therefore carry a large irreducible error. Under an every-position loss, these positions dominate the training signal, and the accuracy of long-context predictions suffers.

Since the logistic map is Markovian, is this substantively different from the standard ICL setup?

For one-step prediction, knowing r gives the next state exactly, but the dynamics determine what data the context contains. In regression ICL, inputs are drawn i.i.d. from the same distribution for every task, so every new example adds information. Here, a fixed-point context is one value repeated and a period-2 context has only two distinct values, so more context adds nothing once the orbit repeats. Thus, the period of the orbit (or lack thereof in the chaotic regime) limits what the context reveals. The invariant measure for a given task determines which states the dynamics will visit, which produces coverage effects with no counterpart in regression ICL. Finally, tokenization introduces non-Markovian behavior into the dynamics. For each map, the state �0��0� depends only on �1��1�, but two states assigned to the same token can fall into different output bins after applying �2��2�. A given token tn corresponds to a single bin and thus the preceding token sequence can further constrain the location of �3��3� within tnt_n. It follows that the distribution of �4��4� may depend on more tokens than �5��5�, even though the underlying map is deterministic and first order. Thus, tokenization affects both the difficulty of one-step prediction and the value of additional context. Coarse tokenizations collapse many continuous states onto the same token, while finer tokenizations preserve more information but create a larger set of tokens and transitions between tokens.

Reviewer_FWzsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper empirically studies in-context learning (ICL) in transformers trained to predict trajectories generated by the logistic family of dynamical systems. Each system is governed by a single parameter rr and evolves according to xt+1=r⋅xt(1−xt)x_{t+1} = r \cdot x_t(1-x_t). This family exhibits different dynamical behaviors, including fixed points, periodic orbits, and chaos (depending on rr). The trajectories are discretized into tokens by partitioning the state space [0,1][0,1] into equally sized bins, and the transformer is trained to predict the next observation from the preceding trajectory.

The authors investigate how the number of distinct training systems, observation resolution (i.e., the number of bins used to discretize the state space), context length, and training-task distribution affect generalization to unseen systems.

The main findings are: (1) increasing the number of distinct training systems leads to a sharp improvement in generalization to unseen systems, similar to previous observations in ICL for regression; (2) finer observation resolution (more bins) requires greater task diversity to achieve generalization but leads to lower prediction error when sufficient task diversity is available; (3) increasing context length has little effect on performance; and (4) generalization across dynamical systems is asymmetric, with models trained on chaotic regimes often transferring better to simpler regimes than vice versa. The authors further investigate this asymmetry by analyzing similarities between the states and transitions represented in training and evaluation trajectories.

Critical strengths
  1. Controlled experimental setting. The logistic family provides a simple and interpretable setting in which different factors affecting generalization can be studied independently (See the weaknesses).

  2. Systematic empirical investigation. The paper explores several factors affecting generalization, including task diversity, observation resolution, context length, and training support.

Critical weaknesses
  1. Limited novelty and unclear main contribution. Most of the paper's findings are consistent with existing results or natural intuitions. In particular, the transition from memorization to generalization as task diversity increases has already been studied in the ICL literature (e.g., Raventós et al., 2023). Similarly, the tradeoff between observation resolution and the amount of training data required, as well as the advantage of training on systems that explore a richer set of states, seem relatively intuitive. While the paper extends these observations to a different setting, it is unclear what fundamentally new understanding is gained.

  2. Very simple setting without correspondingly deep insights. The paper considers only a one-dimensional deterministic family governed by a single scalar parameter, with noise-free observations. While such a simplified setting can be valuable for isolating fundamental phenomena, I would expect it to enable a deeper theoretical or mechanistic understanding that could provide insights into more general dynamical systems. For example, the authors could investigate whether the observed generalization threshold is governed by more general properties of the underlying dynamics, rather than being specific to the logistic family, and whether these findings are expected to extend to higher-dimensional systems, multiple task parameters, or noisy observations. Similarly, understanding why context length has little effect in this setting could help clarify whether this is a consequence of the simple deterministic dynamics. Without such insights, it is difficult to assess the broader significance of the results for ICL in more general dynamical systems.

Questions for authors
  1. The authors suggest that chaotic systems improve generalization by visiting more states and transitions. However, different systems have different transition rules even when visiting the same states. Could the authors clarify how broader state coverage helps predict unseen systems?

  2. Can the authors explain why increasing the context length does not improve generalization? Does this suggest that the model does not rely on substantial in-context task inference, or is a short context simply sufficient to identify the underlying dynamics? Have the authors also experimented with even shorter context lengths (e.g., L\in {1,2,5} )?

  3. Do the authors have any evidence regarding the algorithm implemented by the transformer? In particular, does it infer the underlying task parameter rr from the context, or rely on simpler strategies such as local transition prediction or pattern matching?

One possible experiment would be to compare the transformer against an explicit parameter-estimation baseline and a simple transition-based predictor. Another experiment would be to construct pairs of trajectories generated by different values of rr, such that their final states xtx_t are approximately identical, but their preceding contexts differ. Since the correct next state depends on rr, one could examine whether the transformer's predictions change accordingly. This could help determine whether the model uses the preceding context to identify the underlying dynamics, rather than relying primarily on the last observed token.

1 comment

We would like to thank the reviewer for their careful consideration and helpful comments, we believe that the questions asked have enabled us to significantly improve the paper.

However, different systems have different transition rules even when visiting the same states. Could the authors clarify how broader state coverage helps predict unseen systems?

Broader state coverage helps predict unseen systems because systems with trajectories that explore more of the state space will naturally also cover more transitions which is what determines generalization given the behavior of the Markov models. This is why chaotic maps transition first at every resolution (Fig. 3c). Furthermore, k-gram Markov models can only reproduce transitions seen in training, and their transition to the performance floor mirrors that of the transformer (Fig. 4).

Can the authors explain why increasing the context length does not improve generalization? Does this suggest that the model does not rely on substantial in-context task inference, or is a short context simply sufficient to identify the underlying dynamics? After new analyses suggested by the reviewer, we find that context does improve performance up to a short history. We added models trained with L∈(1,3,5)L \in (1, 3, 5) (Fig. 5a).

With L=1L = 1, the new-task error is close to that of a 1-step Markov model L=3L = 3 and 55 show no crossover and plateau above longer contexts. From L=10L = 10 on, all context lengths reach the same error. These results reflect how much history next-token prediction requires in this setting.

Do the authors have any evidence regarding the algorithm implemented by the transformer? In particular, does it infer the underlying task parameter from the context, or rely on simpler strategies such as local transition prediction or pattern matching?

We compared the transformer with k-gram Markov models fitted to the same training data (Fig.4). These models predict the next token from the empirical statistics of the last kk tokens. The 1-gram predictor, which sees only the current token, is far worse than the transformer at every mm and resolution. Furthermore, the Markov models also show a transition in performance, at nearly the same number of training tasks as the transformer under cross-entropy, and they reproduce how the crossover shifts with resolution (Fig.4a). Beyond the transition, The transformer is more accurate than every Markov model, by a margin that grows with resolution (Fig. 4). The Markov predictors stop improving after four to six tokens, since longer histories do not occur frequently enough to be estimated. The model therefore does not rely solely on simpler strategies such as local transition prediction or pattern matching. Its data requirement is set by coverage of token transitions, but beyond the transition point, it uses its context more effectively than counting allows. We leave a full mechanistic analysis of the transformer’s algorithm to future work.

Reviewer_pqmjOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies in-context learning (ICL) for dynamical systems. They specifically consider a controlled setup where each sequence (trajectory) is drawn from a logistic dynamic system xn+1=rxn(1−xn)x_{n+1}=rx_n(1-x_n), parameterized by the scalar rr. The transformer is trained on quantized sequences and only sees a finite number of systems (rr) at training time. They empirically measure how the diversity of the training set (number of distinct rr values considered for training), the tokenization precision, and the range of parameters rr considered for training affect generalization performance, i.e., the next-token prediction accuracy of the model for unseen systems.

Critical strengths

The paper considers a simple family of dynamical systems to study in-context learning in transformers. This controlled setting allows for systematic control over factors that affect generalization performance, which might not be cleanly doable for more practical time-series data.

Critical weaknesses
  1. Missing details: Some parts of the paper are not described clearly. To give an example, around line 330, what exactly is the measure used to estimate the threshold m∗m^*? Also, in the experiments with synonym tokens, these tokens introduce ambiguity, and it is unclear how this ambiguity is handled in the experimental design (e.g., which synonym token is chosen when constructing sequences, computing the loss, etc.).

  2. Sections are not well-connected:

    • The paper introduces two training setups at the beginning, training with CE or MSE loss, but the experiments do not consistently report results for both. The impact of task diversity is studied under both CE and MSE, while the impact of context length is only studied under MSE, and out-of-support transfer is discussed for CE. The corresponding out-of-support transfer experiment for MSE is deferred to the appendix, without a proper discussion of the differences in the main text. The observations in these two setups appear quite different, with weaker out-of-support transfer under MSE, but right now the only mention of it is one line at the end of Section 5.1.
    • The paper begins by defining several properties of the dynamical systems, but the empirical observations are barely connected back to these properties. In the current presentation, the only connection I can identify is a brief discussion at the end of Section 5.1.
  3. Scope: I mentioned the simplicity of the data model as a strength, and I do believe it can be a strength if leveraged effectively to establish new results or insights. However, I find the study presented in the paper quite limited. The paper considers only a single dynamical system, which raises the natural question of how much of the observations are specific to this setup. Also, the benefit of such a simple model is to either provide concrete theoretical grounding for empirical observations or to leverage its low training cost and controllability for more extensive empirical investigations. Since the paper does not provide theoretical results, I would expect a stronger empirical study. However, the current empirical findings seem rather incremental: the transition in generalization as task diversity increases is not new and has been reported in several similar studies. The other discussions of the impact of vocabulary size and out-of-support generalization are not sufficiently concrete or in-depth (see questions below).

Questions for authors
  1. In the experiments with CE loss, is the output vocabulary always fixed at 64, or does it change with the input vocabulary size? If it is fixed, is this a standard choice when training time-series models? I have a similar question about the MSE setup, where the model is trained on tokenized inputs but predicts a continuous scalar output.

  2. Regarding the discussion at the end of Section 5.1 (line 425 onward): I find the notation and explanation somewhat unclear. Apart from that, I have two questions:

    • The specific out-of-support transfer behavior is only observed for the interval starting exactly at 3. Why doesn't the same argument hold for any interval of rr starting above 3?
    • Similar question for MSE: Why doesn't the same argument apply to the MSE experiment reported in the appendix?
  3. I find the statement "Finer tokenizations require more training maps to generalize" somewhat misleading. My main concern is that the paper seems to conflate the error arising from the training regime (say, limited task diversity) with the error introduced by quantization, since tokenization is not lossless in this setup. This raises the question of what the appropriate measure of generalization should be. If we consider the actual mean error/loss, models with larger vocabularies reach the final loss achieved by smaller-vocabulary models with fewer training tasks in most cases (the lighter solid curves cross the darker dashed lines at smaller values of mm.)

1 comment

We would like to thank the reviewer for their careful consideration and helpful comments, we believe that the questions asked have enabled us to significantly improve the paper.

What exactly is the measure used to estimate the threshold? Also, in the experiments with synonym tokens, these tokens introduce ambiguity, and it is unclear how this ambiguity is handled in the experimental design (e.g., which synonym token is chosen when constructing sequences, computing the loss, etc.).

We have added the definition to the Appendix A.3. Let E(m)E(m) be the new-task error after training on m tasks, averaged over the evaluation grid, and let E∞=1/2[E(4000)+E(8000)]E_{\infty} = 1/2[E(4000) + E(8000)] be its large-m plateau. The threshold task count m∗m^{*} is the number of tasks at which the new-task error has completed half of its decline from m=100m = 100 to the plateau on a logarithmic scale:

log⁡E(m∗)=1/2[log⁡E(100)+log⁡E∞],\log E(m^{*}) = 1/2 [\log E(100) + \log E_{\infty}],

interpolating linearly in (log⁡m,log⁡E)(\log m, \log E). Because m∗m^{*} is measured against each configuration's own plateau, it locates where generalization to new tasks sets in. Absolute errors across resolutions are compared directly in Fig. 2.

For the synonym tokens, we added the construction details to the synonym paragraph (§5):

To investigate this, we increase the vocabulary size while holding resolution fixed, by giving each bin $j \in {2, 4, 8, 16} $ interchangeable 'synonym' tokens. This grows the input vocabulary from 6464 to 64j64j. For every occurrence of a bin in a context window, one of its synonyms is drawn uniformly at random; these draws are fixed when the training set is built, and evaluation contexts receive new draws. Each synonym belongs to exactly one bin, thus only the number of input embeddings grows. Synonyms apply to the inputs only and the targets and the cross-entropy loss remain on the original 64 output bins.

Sections are not well-connected.

We improved the framing of the paper substantially (please see the response to all reviewers), and report results for both loss setups.

In the experiments with CE loss, is the output vocabulary always fixed at 6464, or does it change with the input vocabulary size? If it is fixed, is this a standard choice when training time-series models?

We fix the output resolution at 6464 bins so that varying the input resolution doesn’t affect the prediction target. This allows us to compare cross-entropy across input resolutions. The scalar-output experiment tests the same input-resolution manipulation under a different prediction objective.

Regarding the discussion at the end of Section 5.1 (line 425 onward): I find the notation and explanation somewhat unclear. Apart from that, I have two questions....

State overlap is necessary but not sufficient. Intervals starting above 3 also contain period-2 states that coincide with fixed points of maps with r<3r<3, yet they do not transfer. What distinguishes [3,3+w][3,3+w] is that it contains the bifurcation. Just above r=3r = 3, the two branches of the period-2 orbit nearly coincide, so these training trajectories look like fixed-point trajectories at the token level, repeating the same or neighboring tokens. Intervals starting above 33 contain only well-separated branches. This is consistent with our other results, which indicate that the model must have seen the transitions of the systems it predicts, not only their states.

Under square loss the model is scored on the exact next state. For a fixed-point system the exact next state is the fixed point, which depends on rr. A model that has learned near-fixed-point token patterns from training maps just above r=3r = 3 can predict the right bin but not the exact value for an unseen rr, which likely explains why the low-loss region is weaker under square loss.

I find the statement "Finer tokenizations require more training maps to generalize" somewhat misleading.

The statement holds relative to each model's own plateau, which is how the threshold task count is defined. Finer resolution raises the threshold task count but lowers the attainable error.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to all reviewers/AC

We would like to thank all the reviewers for their careful consideration and helpful comments, we believe that the questions asked have enabled us to significantly improve the paper. In this response, we would like to address some common themes that came up in the reviews, as well as explain some of the major changes we made to the paper.

Novelty/incremental results

Several of the reviews noted that our results seemed to be an incremental study on top of prior working including that of Raventós et. al [1]. We acknowledge that we did not make the differences clear, but believe our paper diverges from earlier work in two key ways.

First, we study a heterogeneous task family. In the often-studied linear regression setting (or even in many nonlinear settings) changes in the parameters of the task family lead to tasks that are similar in kind (in linear regression the tasks are even spherically symmetric). For the logistic map, however, changing the parameter rr yields trajectories that are qualitatively quite different from one another (from fixed points, to periodic orbits, to chaos). This heterogeneity can have important implications for generalization behavior. We detail the potential impacts in section 2.1 and investigate what happens for the logistic map in new experiments (Figure 3).

Second, we study lossy transformations of the data (changes in input resolution). Prior work feeds the data into the model directly, but natural data can be noisy and/or not available in full resolution. We study in a toy model how these perturbations to the data affect models’ ability to generalize. We again detail the potential impacts to generalization ability in section 2.1, and update our discussion to emphasize the results we see.

We also investigate the interplay between these two features of the data. In Figure 3, we see that the effect of resolution change on the model’s generalization ability depends crucially on the dynamical regime the data was drawn from. The heterogeneity of the data exposes that different types of data are affected differently by data transformation.

We thank the reviewers for highlighting this important point, and have updated the manuscript substantially to emphasize the new behaviors the logistic map allows us to study.

Simple setting

Several reviewers point out that we study fairly simple data. This is correct, but it is also what allows us to cleanly explore how transformers generalize. The logistic map is an example of a simple system that still exhibits heterogeneous data, and allows us to perturb the inputs to the model in meaningful, simple ways. Natural language or image data, for example, is no doubt far more heterogeneous, and the number of input transformations is dramatically increased. However, this complexity is also what makes it so difficult to study in a controlled way. Our analysis of the logistic map allows us to tackle these features of real data in a setting where we can be sure of our conclusions. We note also that studying the behavior of transformers in simple settings is by no means a new idea, for example see [1,2,3].

Other major changes

In addition to Figure 3 and the improved framing relative to prior work, we add Figure 4, which compares transformers trained to do next-token prediction on Logistic map trajectories with k-gram Markov models fit to the same dataset. We see that across a range of input resolutions, transformers outperform Markov models, extracting more signal from the same data. We thank reviewer FWzs for this helpful suggestion that we believe meaningfully improves the paper.

[1] Allan Raventós, Mansheej Paul, Feng Chen, and Surya Ganguli. Pretraining task diversity and the emergence of non-Bayesian in-context learning for regression. In Advances in Neural Information Processing Systems, volume 36, 2023. [2] Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? A case study of simple function classes. In Advances in Neural Information Processing Systems, volume 35, pp. 30583–30598, 2022. [3] Chase Goddard, Lindsay M. Smith, Vudtiwat Ngampruetikorn, and David J. Schwab. When can in-context learning generalize out of task distribution? In International Conference on Machine Learning, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.