- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper extends the notion of task diversity in in-context learning, traditionally studied for function regression, to the ICL of dynamical systems. They restrict to the clear case of logistic maps and empirically explore how performance (including generalization) depends on various factors such as task diversity, context length, and observation resolution. Like the standard ICL case, they also identify a task-transition.
Critical strengths
The paper is written well, with clear exposition, figures, and flow. The question is well-motivated and the experiments are conducted appropriately.
The findings on asymmetric transfer are interesting and may inspire future work, seemingly related to importance sampling or curriculum learning for learning dynamical systems.
Critical weaknesses
I am concerned about the novelty of this work. The logistic map is a good toy case and does demonstrate the desired properties, but it alone seems lacking for a complete empirical work. Task diversity is a known idea, this just seems like repeating existing studies in a slightly different setting (which is okay in general, but the setting here is overly simple and the results are consequently not unexpected).
The same transformer architecture is used throughout the experiments, without any results or ablation studies about expressivity, capacity, or training.
Questions for authors
-
I understand that in the regime trajectories converge to the fixed point at zero; why does that automatically suggest you should exclude for the experiments?
-
The uniform bin tokenization seems unmotivated (though this could be my lack of knowledge). Why is it used at all here? Why not just predict directly on the real-valued trajectory? I’m familiar with the necessity of tokenization in some settings but I don’t see it here, especially since you’re training from scratch. You even use squared loss for one choice of the objective. The only additional benefit I see is the tradeoff of resolution with generalization, but that doesn’t seem especially surprising: the bin tokenization smooths the data which prevents overfitting.
-
Can you justify the terminal one-step objective over a sequential loss? I would think that the goal of ICL for a Markovian dynamical system is the approximate trajectories through its autoregressive generation, not just the next point. It feels like the benefit of one model to process variable context lengths at inference is discarded here.
-
Since the logistic map is Markovian, is this substantively different from the standard ICL setup just with (i.e. the context inputs are generated by the dynamic)? Of course the logistic map here is quadratic (not linear), which makes it a bit more complex, but it is still just a one-parameter map, and there have been studies of ICL of nonlinear functions in more general cases.
We would like to thank the reviewer for their careful consideration and helpful comments, we believe that the questions asked have enabled us to significantly improve the paper.
To address this concern, we trained new square-loss models at every resolution with widths ranging from 32 to 256, covering 0.05M to 3.2M parameters. The task threshold appears at every width and only width 32 is limited by capacity. The results can be seen in (Appendix A.1)
For r<1 the orbit converges to the fixed point at zero, and for r<0.5 this convergence happens within about six steps. Thus, at 64 bins, over 96% of context windows are all zeros followed by a zero target. These tasks are indistinguishable from one another in token space, so they add trivially predictable examples but no task diversity. We keep [0.5,1) to represent this regime. This is clarified in §1 ("Why the logistic family?").
With real values, two consecutive observations determine the task exactly, r=xn+1/(xn(1−xn)), and a transformer given continuous inputs reaches the mean squared error floor quickly. Tokenizing the space reduces observation precision and makes in-context inference non-trivial, and it is the object of our study. Uniform bins have also been used in tokenized time-series foundation models such as Chronos. We added this motivation to §1 and the Methods.
We trained the same models with the loss at every position (Appendix A.2). They generalize to new tasks at smaller m, but reach a much worse error. The terminal objective therefore gives both the cleaner measurement and the more accurate models.
We attribute the worse error to the short-context positions. Although the map is first order in x, the tokenized process is not Markov, thus the next token depends on several preceding tokens, because they constrain where x lies within its bin. Predictions made early in the context therefore carry a large irreducible error. Under an every-position loss, these positions dominate the training signal, and the accuracy of long-context predictions suffers.
For one-step prediction, knowing r gives the next state exactly, but the dynamics determine what data the context contains. In regression ICL, inputs are drawn i.i.d. from the same distribution for every task, so every new example adds information. Here, a fixed-point context is one value repeated and a period-2 context has only two distinct values, so more context adds nothing once the orbit repeats. Thus, the period of the orbit (or lack thereof in the chaotic regime) limits what the context reveals. The invariant measure for a given task determines which states the dynamics will visit, which produces coverage effects with no counterpart in regression ICL. Finally, tokenization introduces non-Markovian behavior into the dynamics. For each map, the state �0� depends only on �1�, but two states assigned to the same token can fall into different output bins after applying �2�. A given token tn corresponds to a single bin and thus the preceding token sequence can further constrain the location of �3� within tn. It follows that the distribution of �4� may depend on more tokens than �5�, even though the underlying map is deterministic and first order. Thus, tokenization affects both the difficulty of one-step prediction and the value of additional context. Coarse tokenizations collapse many continuous states onto the same token, while finer tokenizations preserve more information but create a larger set of tokens and transitions between tokens.