acceptodds
Under review as a conference paper at ICLR 2027

Continuous Latent Contexts Enable Efficient Online Learning in Transformers

Abstract

Large language models (LLMs) exhibit a strong capacity for in-context learning: Given labeled examples, they can generate good predictions without parameter updates. However, many interactive settings go beyond static prediction to online decision-making, in which effective behavior demands adaptation over long multi-turn horizons in response to feedback, and efficient algorithms in these domains must use compact representations of what they have learned. Recently, continuous transformer architectures with latent chain of thought have shown promise for offline iterative tasks such as directed graph-reachability. Motivated by this, we study whether continuous latent context tokens equip transformers to more effectively realize online learning. We give explicit constructions of constant-depth transformers that implement two foundational online decision-making procedures – the weighted majority algorithm and -learning – by storing their algorithmic state as linear combinations of feature embeddings, using a small number of latent context tokens. We further train a small GPT-2-style transformer with latent contexts using a multi-curriculum objective that does not directly supervise the latent states. On long synthetic online prediction sequences, this model outperforms comparable discrete-state baselines, while comparisons with prompted LLMs show that textual state summaries can also improve adaptation. Our results suggest that continuous latent contexts provide a simple and effective persistent state for transformers to implement online learning algorithms.

Primary area:
foundation or frontier models, including LLMs
Keywords:
latent contexts, transformers, online decision making, online learning, realizability, efficient constructions

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_pQt5OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper shows that having a few soft tokens allows a transformer to represent weighted majority and tabular q learning. It then implements these in very small transformers using backprop-through-time finding the learned recurrent transformers achieve good performance. Weighted majority performance is compared with existing large scale transformers showing that a specialized transformer with a recurrent state can perform well. Other experiments compare with a discrete one-token transformer showing better performance with the modified transformer.

Critical strengths

The proofs appear correct and the experimental results are believable.

Critical weaknesses

The implications of this paper appear primarily informational since the dependence on backprop through time makes the algorithm difficult to parallelize and no approach to a scalable algorithm is offered.

There's significant missing related work about existing similar implementation in Recurrent Memory Transformer (Bulatov, Kuratov, Burtsev, NeurIPS 2022) and related literature. People have also thought about recurrent state as an online learning algorithm. See for example the test time training work several of which are referenced here https://yueatsprograms.github.io/ttt/home.html . Note also that this is spelled out in the title of the Longhorn work which is cited but not discussed in relation to the thesis. There is prior work on getting transformers to explore by creating summaries in Krishnamurthy, Harris, Foster, Zhang & Slivkins, “Can Large Language Models Explore In-Context?” (NeurIPS 2024). Another very related one is: Lin, Bai, & Mei, Transformers as Decision Makers: Provable In-Context RL via Supervised Pretraining (ICLR 2024)

In the end, I'm left with relatively little to consider new here since similar methods have been tested and similar conclusions drawn previously. (1) Recurrent state is learnable, as per RMT and TTT papers. (2) Recurrent state can function as online learning per Longhorn. (3) Summaries help LLMs, as per the Krishnamurthy et al paper. And of course trained transformers also don't have a regret guarantee. The strongest point seems to be the constructions which are nice but seem unsurprising.

Questions for authors

How many seeds were used in conducting the Q-learning experiments and what was the variance across seeds?

3 comments
AuthorsOpenReview ↗

Response to Reviewer pQt5 (part 1)

Dear Reviewer pQt5,

We thank you for your review. We address your concerns about novelty and scalability below, and we have expanded the related work in the revised PDF, which we will submit in the next few hours.

What we see as new. Our main result is a specific computational one: a fixed-depth transformer can run classical online decision-making algorithms by keeping their state in a constant number of continuous context tokens, with parameters independent of the horizon. Theorems 3.1 and 4.1 give explicit two-layer constructions:

  1. Weighted Majority: one latent token stores the expert log-weights as coefficients in an embedding basis. A unit-temperature softmax head then computes WMA's normalized vote exactly, and a residual write performs the log-weight update.
  2. Tabular Q-learning: ∣A∣|A| latent tokens store the action-indexed columns of the Q-table. Attention and MLP layers assemble the current value, the greedy Bellman target, and the TD update.

We think this is less expected than it may look, for two reasons:

  1. Existing transformer constructions of online RL algorithms read the full interaction history (details below). Keeping a horizon-independent persistent state changes how history is stored and updated, and our constructions use O(T)O(T) total work and memory independent of TT
  2. Our experiments show that whether training finds such a state depends on the state being continuous. In our new memory-budget ablation (Figure 12), discrete contexts with up to 16 tokens (132 bits per round) stay at majority-vote regret, even though a finite-state MW reaches regret 4.8 with 12 bits. In the same training signal, the continuous context learns the MW statistic, and causal interventions show the model's decisions depend on it (Figures 10-11).

Backpropagation through time. Training cost and run-time cost are separate questions. Our constructed transformers need no training at all since their weights are given in closed form. At runtime, both the constructed and the trained models only run forward passes. Per-round work and state size don't depend on the horizon, giving O(T)O(T) total work without keeping the history. Nonetheless, we agree that training is sequential, like any recurrent memory (RMT included), and a parallel training method is a universal open problem.

Regret guarantees. We agree that trained models do not inherit MW's worst-case guarantee, and we mentioned this explicitly in the paper, stating that it tracks the average-case distribution. They are trained on in-distribution log-loss, so they are optimized for the training distribution, not for adversarial sequences. An adaptive adversary drives the trained model's regret to 47 within 500 rounds, versus MW's ln⁡4\ln 4 bound (Fig. 13). Our claims about trained models are therefore distributional: in distribution, they achieve low regret and closely track the reference learners (e.g., within 0.0015 of MW's predicted probabilities on data where MW is Bayes-optimal, Table 1). The worst-case guarantees belong to the constructions of Theorems 3.1 and 4.1, which implement WMA and Q-learning exactly.

AuthorsOpenReview ↗

Response to Reviewer pQt5 (part 2)

Related work.

  1. Recurrent Memory Transformer. RMT shows that memory tokens carried across segments are an effective architecture. We give an explicit realization of WMA and Q-learning (state encoding, reads, updates, and proofs of correspondence), and we test which statistics training recovers. The write rule also matters: an RMT-style overwrite (c→Wctxhc\to W_{ctx}h) is unstable across seeds on our task (regret 15.9 and 137.5 at T=1000T=1000), while our additive write is stable (Fig. 12).
  2. TTT and Longhorn. These develop online-learning mechanisms as sequence layers. TTT runs learning updates on an inner model's fast weights; Longhorn derives an SSM transition from an online associative-recall objective. We show how standard attention and MLP computations, plus persistent context writes, implement the predictions and feedback updates of specific decision-making algorithms. The shared online-learning view motivates the connection, but the mechanisms, target algorithms, and guarantees differ. We now discuss Longhorn in this context.
  3. Lin, Bai & Mei. They construct transformers that run online RL algorithms and analyze supervised pretraining. In their setting, the whole interaction trajectory is the context (so length grows with TT), and their constructions use ReLU attention. Ours carry out the online recurrence through softmax attention and persistent latent tokens whose number, dimension, and update depth don't depend on the horizon.
  4. Krishnamurthy et al. Their exploration setup feeds the model sufficient statistics computed outside it. Our note experiments ask the model to create and maintain its own summary, including a history-free setting where that summary is the only information passed between rounds. Using given statistics and testing whether a model builds and keeps them are different questions. Their setting is bandits; ours is full-information expert prediction.

In summary, each of the works above establishes one ingredient: recurrent memory is trainable, recurrent state can act as an online learner, and summaries help LLMs. Our contribution is to connect them with exact constructions and controlled experiments: a few continuous tokens suffice to run regret-minimization and RL algorithms with constant depth, and training without state supervision recovers that state when it is continuous. We believe this opens a principled direction for studying algorithm learning in transformers.

Seeds in Q-learning. We trained 3 seeds for the continuous and discrete versions; As we mention in Appendix 4.2.5, the teacher-forced agreement is 30%, and the ϵ\epsilon-greedy returns are 35% for ϵ=0.05\epsilon=0.05, 38% at 0.10.1, 44% at 0.20.2, and 47% at 0.30.3.

Of course, we would be happy to answer any further questions you may have for us, and if you find that we have addressed all your concerns, we hope you will consider raising your score.

Thank you,

Authors

Thanks for your response. I'm planning to stick with my review, because I still do not find the results surprising or impactful enough that I would recommend it to other folks.

Reviewer_TFQ3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether continuous latent context tokens enable constant-depth transformers to maintain persistent algorithmic states for online decision-making. The authors first provide explicit 2-layer causal transformer constructions (in the hard-attention limit β→∞\beta \to \infty w/o LayerNorm, alongside finite-temperature error bounds) that implement the Weighted Majority Algorithm (WMA) using a single latent token to store expert log-weights in superposition, and tabular QQ-learning using ∣A∣|\mathcal{A}| latent tokens to store action-indexed columns of the QQ-table. They then train small GPT-2 decoders (n=4n = 4 experts and ∣S∣≤8,∣A∣≤4|\mathcal{S}| \le 8, |\mathcal{A}| \le 4 MDPs) w/o direct state supervision, showing that continuous residual-write contexts track target online statistics and outperform a single-token discrete baseline as well as zero-shot prompted LLMs (DeepSeek-V3 and Qwen-3-14B).

Critical strengths

The explicit 2-layer constructions for WMA (Thm. 3.1) and tabular QQ-learning (Thm. 4.1) are clean and intuitive. Encoding expert log-weights λi,t=log⁡wi,t\lambda_{i,t} = \log w_{i,t} as coefficients in an orthonormal expert basis so that a single unit-temperature softmax attention head directly outputs the WMA prediction p^t\hat{p}_t, while updating the state via a linear residual write, is a neat connection between continuous CoT superpositions and classical online learning updates. The numerical verification and robustness sweeps in App. E (testing finite β\beta, residual noise σ\sigma, non-orthogonal embeddings, and buffer overlap ρ\rho) are thorough and reassuring.

I also really liked the mechanistic probing and causal intervention experiments in Sec. 5.1 and App. F.1.2. Showing that a linear probe recovers cumulative expert losses from ⟨zt⟩\langle z_t \rangle w/ R2=1.00R^2 = 1.00 and retention ρ=1.000\rho = 1.000 beyond the training horizon, and backing this up w/ latent state transplants between sequences (Fig. 10) and directional steering along probe vectors (Fig. 11), gives strong evidence that the trained model genuinely relies on the latent slot as a persistent state rather than shallow heuristics.

Critical weaknesses

My main reservation is that the comparison against the trained "discrete transformer" baseline in Sec. 5.1 and 5.2 is structurally unfair from an information-capacity standpoint. Per App. F (line 1684), the discrete baseline is restricted to writing a single vocabulary token embedding per slot (arg⁡max⁡vuv⊤h\arg\max_v u_v^\top h, trained via straight-through Gumbel-softmax) over a tiny task vocabulary (∣V∣≈12|\mathcal{V}| \approx 12 for WMA and ∣V∣≈18|\mathcal{V}| \approx 18 for QQ-learning). A single discrete token from a ~15-token vocab can pass at most log⁡2(15)≈4\log_2(15) \approx 4 bits of state across rounds, whereas a single continuous vector in R64\mathbb{R}^{64} or R256\mathbb{R}^{256} carries thousands of bits of floating-point precision. It is information-theoretically impossible to compress 4 unbounded cumulative expert losses or an 8-state real-valued QQ-table column into a single discrete token ID. In realistic discrete CoT (and in the paper's own LLM note experiments in Sec. 5.3), a discrete scratchpad uses O(n)O(n) or O(∣S∣)O(|\mathcal{S}|) tokens, not 1 token. Coupled w/ high-variance Gumbel-softmax gradients backpropagated through 200 recurrent steps, the discrete baseline's collapse to majority vote (R2=0.00R^2 = 0.00) reflects an artificial 1-token bottleneck rather than an inherent flaw of discrete representations.

Second, the architecture relies on external hard-coded memory addressing and scales linearly in embedding dimension w/ problem size. In the QQ-learning setup (App. F.2.3, line 1916), the selective write to slot catc_{a_t} (Ct+1[a,:]=Ct[a,:]+WctxhtUPD+bctxC_{t+1}[a, :] = C_t[a, :] + W_{\text{ctx}} h_t^{\text{UPD}} + b_{\text{ctx}} only for a=ata = a_t, leaving other slots untouched) is executed externally by indexing ata_t in the training loop rather than learned by the transformer. Moreover, storing non-interfering superpositions requires d=O(n)d = O(n) or d=O(∣S∣+∣A∣)d = O(|\mathcal{S}| + |\mathcal{A}|), and the per-round input sequence length still scales as 4n+44n + 4 or 3∣A∣+93|\mathcal{A}| + 9. All trained runs are limited to toy dimensions (n=4n = 4 experts, ∣S∣≤8,∣A∣≤4|\mathcal{S}| \le 8, |\mathcal{A}| \le 4, 2 seeds).

Finally, there is a notable gap between the theoretical algorithms and what the trained models actually learn. In Sec. 5.1, the model trained on i.i.d. expert data learns the Bayes-optimal posterior voter for that prior rather than WMA; when trained on the hidden-expert distribution where MW is Bayes-optimal (App. F.1.2, Fig. 12), one of the two seeds (seed 42) fails to converge to MW (regret 21.5 vs. 1.51), and even the successful seed breaks down on adversarial label sequences (regret 47 vs. MW's ln⁡4\ln 4 bound). Thus, the model learns an average-case in-distribution heuristic rather than a worst-case no-regret algorithm. Similarly, in closed-loop QQ-learning (App. F.2.5, Fig. 17), the latent slot norm grows linearly w/ tt instead of converging, and greedy closed-loop execution only gets 27% of optimal return (near random at 29%).

Questions for authors
  1. To make the continuous vs. discrete transformer comparison fair, what happens if the discrete baseline is allowed to emit a short sequence of MM discrete tokens per round (e.g., M=8M = 8 or 1616 tokens, matching the text-note setup in Sec. 5.3) or uses a larger codebook (e.g., VQ or multi-head discrete latents) so it actually has enough bits to represent the cumulative counts?

  2. In App. F.1.2 (Fig. 12), why does seed 42 fail to learn MW on the hidden-expert distribution (regret 21.5 vs. 1.51 for seed 43), and have you evaluated across more than 2 random seeds? Also, since the trained model incurs high regret (47) against an adaptive adversary, wouldn't it be more accurate to frame the learned behavior as amortizing Bayes-optimal filtering rather than learning a no-regret online algorithm?

  3. In the QQ-learning experiments (App. F.2.3), the update loop externally indexes ata_t to only overwrite slot Ct+1[at,:]C_{t+1}[a_t, :]. If the model instead has to update all ∣A∣|\mathcal{A}| context slots simultaneously via its own learned projection (without external action-slot indexing), can it still learn to route the TD update solely to the executed action's slot?

  4. In App. E.2 (Fig. 9c, d), tied read-out (R=Φ⊤R = \Phi^\top) causes errors to compound rapidly through the recurrent latent write under random unit embeddings or minor buffer overlap (ρ=0.01\rho = 0.01), whereas dual read-out (R=Φ+R = \Phi^+) stays exact. Since standard weight-tied transformers do not enforce dual pseudo-inverse projections, how do the trained GPT-2 models avoid this recurrent interference over 1,000 steps?

3 comments
AuthorsOpenReview ↗

Response to Reviewer TFQ3 (part 1)

Dear Reviewer TFQ3,

We thank you for your detailed and constructive review. We have run a number of experiments to answer your questions and will provide them in the revised manuscript, which we will submit in the next few hours. Please find our response below:

Q1. What happens if the discrete baseline is allowed to emit a short sequence of MM discrete tokens per round or uses a larger codebook?

  1. We ran this experiment (Figure 12, now in Section 5.1) on the actual GPT-2-style discrete transformer (which is not constrained by the information-theoretic bottlenecks) with K∈1,2,4,8,16K\in\\{1,2,4,8,16\\} context tokens from a 311-token vocabulary (up to 132 bits per round), trained with the same data and curriculum, all stay at majority-vote regret (141–148 at T=1000T=1000 and have a majority vote of 144).
  2. Moreover, we have provided a new and stronger discrete baseline of finite-scalar quantization (FSQ) [Mentzer et al. ICLR 2024] which improves on majority vote but remains far from the continuous context: it has regret 70 with 37 bits, and 131 and 87 (two seeds) with 101 bits, against −21.1-21.1 for the continuous transformer. We also provide a baseline of a multi-head discrete latent (which seems to underperform a lot, and our probes show that its latents decode almost none of the algorithmic statistics and that the probe falls to R2≈0.02R^2\approx 0.02 by round 10001000.
  3. The task needs few bits: a hand-designed finite-state MW that stores clipped loss gaps reaches regret 14.6 with 8 bits and 4.8 with 12 bits (MW: 4.9). A single context token carries 8.3 bits, so even one token can hold a far better learner than the majority vote that training finds.
  4. The continuous context needs little precision: projected onto 8 principal directions and stochastically rounded to 10 bits each (80 bits), it reaches −20.5-20.5 vs −21.1-21.1 at full precision.

So the gap is one of learnability, as you suggest: a discrete state with ample capacity is not found by straight-through gradients over long recurrent horizons, while a continuous state is. The revised text makes this the claim.

Q2. Why does seed 42 fail to learn MW, and have you evaluated across more than 2 random seeds? Wouldn't it be more accurate to frame the learned behavior as amortizing Bayes-optimal filtering?

We agree with your point on the framing and have adopted it. Importantly, the two parts of the paper make separate claims. The constructions of Theorems 3.1 and 4.1 implement the worst-case algorithms exactly (and thus also inherit the worst-case regret guarantees). On the other hand, the trained models are optimized on in-distribution log-loss, so they approximate the Bayes-optimal predictor for their training distribution and carry no worst-case guarantee. We connect these points theoretically in our manuscript: on hidden-expert data, the MW predictor that our construction implements is the global minimizer of the population log-loss, and any trained model with excess risk at most ϵ\epsilon is within ϵ\epsilon of MW in expected KL divergence (Corollary C.3). Indeed, empirically, the trained model's probabilities there are within 0.0015 of MW's (Table 1).

Moreover, we re-ran our experiment with 8 seeds on the hidden-expert data. Seed 42 is the only seed that seems to only learn MW partially (distance 0.065, regret 21.5), and we believe this is a statistical outlier since the other 7 seeds do converge to MW. We find, however, that if we decrease the learning rate and continue training on seed 42, the validation loss drops from 0.547 to 0.510 in one epoch. It then matches MW as closely as any seed: distance 0.0014, regret 1.42, probe R² 0.996. So then, all the seeds reach MW.

AuthorsOpenReview ↗

Response to Reviewer TFQ3 (part 2)

Q3. If the model instead has to update all AA context slots simultaneously via its own learned projection, can it still learn to route the TD update solely to the executed action's slot?

Yes. We ran this ablation (and provide it in the Appendix). At every step, the model now writes to all slots through a gate it computes itself: g=softmax(Wroutehupd)g=\mathrm{softmax}(W_{route} h^{upd}) and ca←ca+gaWctxhupdc_a \gets c_a + g_a W_{ctx} h^{upd} for every slot aa.

The executed action ata_t is still an input token, but nothing outside the network uses it to choose a slot. To route correctly, the model must carry the action's identity to the ⟨Update⟩\langle\text{Update}\rangle token through attention and turn it into a gate.

Training. Early in training, the gate used for the write is mixed with the true address, gtrain=λ onehot(at)+(1−λ) gg_{\text{train}} = \lambda\,\mathrm{onehot}(a_t) + (1-\lambda)\,g, together with an auxiliary loss on the gate. λ\lambda is annealed to 0. All reported numbers use the model's own gate with λ=0\lambda=0, with checkpoints taken only after λ\lambda reaches 0. We provide our results below:

Learned routingExternal routing (paper)
Gate mass on the executed action's slot (∣A∣=2/3/4\lvert\mathcal{A}\rvert = 2/3/4)1.000 / 1.000 / 0.9931 (given)
Teacher-forced validation accuracy93.2%92.3%
Agreement with Q-learning (α=0.2,γ=0.9\alpha=0.2, \gamma=0.9)) with all steps and steps of Fig. 160.958 / 0.7540.951 / 0.718
Best-fitting (α,γ)(\alpha,\gamma)(0.2, 0.9)(0.2, 0.9)
Probe R2R^2: Q-table / visit counts0.89 / 0.980.71 / 0.89
Closed-loop return, ϵ=0.2\epsilon=0.2 (% of optimal)45.6%44.4%

The model learns the address: over 99% of each write goes to the executed action's slot, and it matches the externally routed model on every measure. A second run that removes the routing aid earlier learns a softer gate (mass 0.77–0.86) and performs equally well (92.6% accuracy, 44.9% return), so exact routing is not required.

Q4. How do the trained GPT-2 models avoid this recurrent interference over 1,000 steps?

The interference in Appendix E.2 arises when the read-out is forced to be tied, R=Φ⊤R=\Phi^\top. The trained models have no such constraint: the write map Wctx,bctxW_{ctx},b_{ctx}, and every attention projection that reads the context are learned separately, so training is free to find a read-out that inverts the write, as the dual read-out does. Moreover, the state stays clean in practice: the MW-state probe holds R2=0.99−1.00R^2=0.99-1.00 on rounds 201-400, beyond the training horizon, and regret keeps falling to T=1000T=1000.

Closed-loop Q-learning. We rebuilt the Q-learning data so that the teacher's labels depend on bootstrapping (reward noise, 100-200-step episodes, shifted-sign and goal-state tasks). A myopic γ=0\gamma=0 learner matched the teacher on 97.6% of steps in the earlier data and differs on 17% now (see Appendix F.2.1). On the new data:

  • With ϵ=0.2\epsilon=0.2 exploration, the continuous transformer reaches 44% of optimal return, versus 34% (discrete), 29% (random) and 55% (tabular Q-learning), and keeps improving well past the training horizon (Fig. 4).
  • Greedy return is low for every learner, tabular Q-learning included (27% vs. 30%), so it reflects missing exploration.
  • The context norm grows because learned writes have roughly constant size. Reads are unaffected, since every pre-norm block normalizes the context first (Appendix F.2.5).

Scaling. The embedding dimension d=O(n)d=O(n) matches the size of the algorithm's own state: for instance, WMA keeps nn weights, and each round's input already has length O(n)O(n). The quantity that stays constant is what matters for long interactions: depth, heads, and state size are independent of the horizon TT. The trained settings are small by design, so that the learned state can be decoded and intervened on; we state this in the limitations.

In summary, the revised manuscript now emphasizes the distinction between what the architecture can represent, which the constructions settle with constant depth and a horizon-independent state, and what training finds, where a continuous state is learned, and a discrete one with the same capacity is not. We believe this gives a principled basis for studying how transformers acquire and maintain algorithmic state over long interactions.

Of course, we would be happy to answer any further questions you may have for us, and if you find that we have addressed all your concerns, we hope you will consider raising your score.

Thank you,

Authors

Thanks for the detailed response, my most concern has been addressed, I will maintain my positive score.

Reviewer_VmKXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether transformers can implement online learning algorithms by storing information about the past in continuous latent context vectors. The authors provide explicit constructions to implement two classical online learning algorithms, Weighted Majority and tabular Q-learning, via standard GPT-like transformer architectures. They also train small transformers on synthetic expert-prediction and reinforcement-learning tasks without directly supervising the latent states. The paper includes several mechanistic interpretability experiments to evaluate whether the learned strategies of the models follow the optimal theoretical constructions. A few experiments showing that state-of-the-art open-weight models such as DeepSeek-V3 and Qwen3-14B benefit from natural-language state summaries conclude the paper.

Critical strengths

The paper is well written, the proofs of the theoretical claims seem correct to me, and the experimental results are varied and mostly convincing except for some concerns that I have (see below).

The theoretical results are interesting additions to the mechanistic interpretability research lines on LLMs implementing optimal in-context estimators. The experiments do a good job to probe whether the models truly learn something resembling the proposed theoretical constructions.

The additional experiments on open-weight models, showing that models such as DeepSeek-V3 and Qwen-3-14B perform differently under the same conditions, but benefit from additional unsupervised notes about the context history, are fascinating, and help the paper move beyond the simple toy models used for the theoretical results toward more practical scenarios.

Critical weaknesses

In my opinion, the experimental part of the paper is the one that can be improved further. In particular, I question the authors on their choice of baselines to which they compare the continuous transformer. More specifically:

  1. The discrete transformer baseline is quite restrictive in using a single vocabulary token in place of the continuous latent token. For example, in the "experts" experiment of Sections 5.1 and F.1, the vocabulary size is only 10, which seems extremely restrictive.
  2. The Bayes-optimal baseline in the "experts" experiment seems wrong, as the experts are sampled uniformly at random from [0.3,0.9][0.3, 0.9] while the posterior is uniform on the entire [0,1][0,1] interval.

The second issue is that the experiments do not directly prove that transformers learn the proposed theoretical constructions in practice. They only provide indirect evidence that the latent context vectors are used actively to store information about the past history. Nonetheless, this is still worthwhile.

Questions for authors

I would like to ask the authors the following clarification questions:

  1. The Bayes-optimal baseline for the experiment in Section 5.1 and F.1 is not truly optimal as it samples the expert's parameter uniformly on [0,1][0,1] instead of [0.3,0.9][0.3, 0.9]. Could you provide numerical results for the correct Bayes-optimal expert and discuss its performance compared to the continuous transformer?
  2. The discrete transformer baseline stores its state using a single vocabulary token per context slot, while the continuous model has access to an entire real-valued vector. This creates a substantial difference in information capacity, particularly for tasks requiring the storage of multiple cumulative statistics like the ones considered in the paper. Do you believe that the advantage of the continuous transformer would persist against more capable discrete models?
  3. In the "experts" experiment, the continuous transformer beats the Weighted Majority algorithm but falls short of matching the Bayes optimal estimator (which seems not be even the actual optimal estimator, see Q1). Why do you think this is the case? It would be nice to understand why the continuous transformer cannot match the optimal in this setting.
  4. The task of online learning by means of compressed history token seems very naturally suitable for alternative LLM architectures such as state-space models (Mamba and similar models), that function by updating a hidden state vector sequentially. Do you think that architectures such as Mamba would have better performance than the continuous transformer that you consider in this paper?

I will be happy to increase my score if my questions are addressed satisfactorily.

3 comments
AuthorsOpenReview ↗

Response to Reviewer VmKX (part 1)

Dear Reviewer VmKX,

We thank you for your careful reading. We respond to each of your comments below and will provide an updated manuscript in the next few hours. :

The contribution of the constructions. Prior theory on continuous chain-of-thought shows that latent tokens help with offline problems such as graph reachability, where superposition lets a transformer explore many paths at once. We show that the same mechanism plays a different role in online decision-making: it acts as a persistent algorithmic state. To our knowledge, Theorems 3.1 and 4.1 are the first explicit constructions of this kind, and they establish three things:

  • Constant depth and a horizon-independent state. Existing constructions of online learning and RL algorithms in transformers read the whole interaction history, so context length and per-round work grow with TT. Ours run WMA and tabular Q-learning with two layers, O(1)O(1) heads and a fixed number of latent tokens, for O(T)O(T) total work and memory independent of TT.
  • A natural fit between attention and online learning. Storing expert log-weights as coefficients in an embedding basis lets a single unit-temperature softmax head compute WMA's normalized vote exactly, with the update as a linear residual write. Q-learning is harder, since it needs a maximum over actions and a selective update of one table entry; the construction handles both with standard attention and ReLU gating. Both remain accurate at moderate finite temperatures (Lemma 3.3, Appendix E).
  • A testable prediction. The constructions say exactly which statistic the latent state should hold. This turns "does the model learn the algorithm?" into a question we can test with probes and causal interventions, which we do below.

Q1. The Bayes-optimal baseline samples the expert's parameter uniformly on [0,1][0,1] instead of [0.3,0.9][0.3,0.9].

Thank you for catching this. Our reference learner used a Beta(1,1) prior, while qualities are drawn from U[0.3,0.9]U[0.3,0.9]. With the correct truncated-uniform prior, its regret at T=1000T=1000 is −28.3-28.3 (previously, −27.9-27.9, which is only a small change), so the conclusions are unchanged. We have updated our Figure 2, Table 2, and the text.

Q2. Do you believe that the advantage of the continuous transformer would persist against more capable discrete models?

Yes, and the new memory-budget ablation (Fig. 12, now in Section 5.1) shows the advantage is not about capacity:

  1. Discrete GPT-2 style transformer with K∈1,2,4,8,16K\in\\{1,2,4,8,16\\} context tokens from a 311-token vocabulary (up to 132 bits per round), trained with the same data and curriculum, all stay at the majority-vote baseline (regret 141-148 at T=1000T=1000 with majority vote: 144).
  2. Moreover, we have provided a new and stronger discrete baseline of finite-scalar quantization (FSQ) [Mentzer et al. ICLR 2024] which improves on majority vote but remains far from the continuous context: it has regret 70 with 37 bits, and 131 and 87 (two seeds) with 101 bits, against −21.1-21.1 for the continuous transformer.
  3. Twelve bits are enough in principle: a hand-designed finite-state MW that stores clipped loss gaps reaches regret 4.8 (MW: 4.9).
  4. The continuous context needs little precision: projected onto 8 principal directions and stochastically rounded to 10 bits each (80 bits), it reaches −20.5-20.5 versus −21.1-21.1, versus −21.1-21.1 at full precision. The finding is therefore about learnability. Under the same training signal, a continuous state learns to carry the loss statistics. A discrete state with ample room learns them only partially (FSQ) or not at all (vocabulary tokens).

Q3. Why do you think the continuous transformer falls short of the Bayes estimator?

The probes give a specific answer. The context stores each expert's cumulative loss linearly (probe R2=1.00R^2=1.00, retention ρ=1.000\rho=1.000, still 0.99−1.000.99-1.00 on rounds 201–400), which is the state our construction predicts. The Bayes vote is a nonlinear (log-odds) function of those counts: a linear probe for the log-odds reaches R2=0.90R^2=0.90 on the rounds it was fitted on (31-95) and turns negative beyond them. So the model holds the right sufficient statistic and computes the nonlinear vote when it reads the state. That read-out was trained on sequences of up to 200 rounds, and its approximation loosens as counts grow, which matches the gap to Bayes widening with the horizon: 1.1, 4.0, and 7.2 at T=200,500,1000T=200,500,1000 (Table 2). Where Bayes and majority vote disagree, the model follows Bayes on 82% and 94% of rounds (two seeds).

AuthorsOpenReview ↗

Response to Reviewer VmKX (part 2)

Q4. Do you think that architectures such as Mamba would have better performance than the continuous transformer?

We expect SSMs to do well on this task, and we see that as support for our thesis. WMA's update λt+1=λt+η(1−ℓt)\lambda_{t+1} = \lambda_t + \eta(1-\ell_t) is a linear recurrence, the form SSMs implement, and the same structure underlies our additive latent write. Figure 12 includes two recurrent baselines trained on the same data and curriculum, a GRU (64-dim state) and an LSTM (two 32-dimensional states), which reach −26.7-26.7 and −23.9-23.9. These are gated RNNs, not SSMs, but they point the same way: what matters is a continuous, additively updated state. A Mamba baseline with a matched state size also seems to learn the Bayes optimal estimator, and its regret is −27.2-27.2, which has similar performance to our continuous transformer. Our interest in the transformer is practical: it is the architecture of current LLMs, and a few latent context tokens give it this state without changing the backbone (as in COCONUT). We now discuss the connection to SSMs as amortized online learners (Longhorn) in the related work.

How direct is the evidence? Beyond probing, two causal interventions show that the model's decisions depend on the decoded state (Figs. 10–11):

  • Continuing sequence BB from sequence AA's context token makes the model follow AA’s best expert on 92-93% of disagreement rounds (control: 6–7%).
  • Steering the context along an expert's loss direction lowers how often the next decision follows that expert from 61% to 27%; a random direction of the same norm leaves it at 42–43%.

On data where MW is Bayes-optimal, the model's predicted probabilities are within 0.0015 of MW's (Table 1). Together, these show that training recovers the state the construction uses and that the model relies on it. Whether the learned circuit matches the construction weight for weight is a natural next question, and the constructions give a reference to compare against.

In summary, our work connects recurrent latent contexts in transformers to the states of online learning algorithms. The constructions show that a few continuous tokens suffice to run regret-minimization and RL algorithms exactly, with constant depth and a horizon-independent state. The experiments show that, without any latent-state supervision, gradient-based training recovers the same algorithmic state and behaviour comparable to the target algorithms. As persistent online adaptation is central to agentic systems, we believe this opens a principled direction for studying algorithm learning in transformers.

Of course, we would be happy to answer any further questions you may have for us, and if you find that we have answered your concerns, we hope you will consider raising your score.

Thank you,

Authors

I thank the authors for their thorough response, which I find satisfying. In particular, the new experiments are particularly welcome. Consequently, I increased my score to 3. I suggest the authors to update their submission to include the new changes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.