acceptodds
Under review as a conference paper at ICLR 2027

Rethinking at Fixed Points in Looped Language Models

Abstract

A looped language model whose recurrent states converge to fixed points can replace its recurrent trajectory with the terminal state, in both training and inference. Near a fixed point, the endpoint can stand in for the whole trajectory and unlock a series of abilities: truncated backpropagation in training, terminal key-value cache reuse that shrinks the cache for a loop- recurrent core at almost no performance loss, distillation for up to faster prefill, and faster post-training updates that compute gradients from saved rollout states instead of replaying the trajectory. To better shape fixed points, we improve two existing techniques: depth sampling, whose depth prior we propose to learn from training signals with constraints; and input injection, which we force orthogonal update so that every recurrence receives the input intact. At M, M, and B parameters, the learned depth prior lowers perplexity under terminal cache reuse by – relative to Huginn's fixed prior; at B, it matches the full-cache downstream accuracy of fixed-depth training. Orthogonal injection lowers perplexity by up to relative to leading injection schemes.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Looped Language Models, Efficiency, Fixed Points

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_r4xoOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

In looped language models, achieving fixed points in recurrence can improve efficiency in training and decoding by allowing the terminal end-point to be a replacement for the full trajectory. Thus it can give a number of benefits such as faster decoding by shrinking the KV cache, faster prefill in distillation and faster rollout scoring in RL. This paper proposes two techniques to shape fixed points: learning a depth prior and orthogonal input injection. The proposed approach for learning a depth prior improved upon a fixed-depth scheme and a fixed prior scheme in terms of validation perplexity by 1.1-1.9%. It speeds up scoring and backpropagation in RL by 1.5x with a slight degradation in quality. A distilled 2-block student based on the learned prior model is 1.5-1.7% faster than the teacher in terms of prefill latency while keeping most of its quality though there is quite a bit of variation across evaluation tasks. For input injection, the current baseline (Parcae's injection) uses a method whereby the portion of the input that the fixed point retains depends on the state. The paper proposes a method to remove this component so that each recurrence is passed the same input. This yields improvements by around 0.7-1.6% in terms of perplexity.

Critical strengths
  • Proposes 2 well-motivated approaches for improving the efficiency of looped language models. Learning prior improves upon fixed-depth and fixed-prior in terms of perplexity/downstream accuracy, faster RL and prefill latency in distillation. Orthogonal injection improves upon Parcae's injection in terms of perplexity, though the gains are small.
  • Trains models at 3 scaled: 100M, 400M and 1.6B parameters
Critical weaknesses
  • The paper could benefit from additional analysis: 1) Why Fixed depth works as well as the proposed scheme for OBQA on M and L scales? 2) Why Huginn-Linear and Parcae-Decay works better than orthogonal injection for ARC-C and OBQA respectively?
  • The improvements in perplexity are rather small: 1.1-1.9% for learning depth prior and 1.6% for better input injection.
Questions for authors
  • L182: Shouldn't h_r be a function of t? i.e. h_r^t?
  • L183: Clarify this phrase: "prefix-dependent input to its update".
3 comments
AuthorsOpenReview ↗

Response to Reviewer r4xo

Thank you for the positive assessment and the precise questions.

W1. Gaps on OBQA and ARC-C.

  • To test whether these gaps are real, we computed paired bootstrap intervals over questions (10,000 resamples). At our scales, accuracy on 500 (OBQA) or 1,172 (ARC-C) questions is too coarse: every 95% interval contains zero.
  • We therefore further score correct-answer bits per byte (BPB) on the same questions, a continuous metric with a higher signal-to-noise ratio than accuracy on most benchmarks and more reliable for comparing small models (Heineman et al., 2025; Magnusson et al., 2025):
GapScaleAccuracy Δ (points, ↑) [95% CI]BPB Δ (↓) [95% CI]
OBQA: learned prior − fixed depthM−2.40 [−5.20, +0.40]−0.077 [−0.101, −0.052]
OBQA: learned prior − fixed depthL0.00 [−3.20, +3.00]−0.095 [−0.117, −0.073]
ARC-C: OrthoInj − Huginn-LinearM−0.94 [−2.99, +1.11]−0.036 [−0.048, −0.023]
OBQA: OrthoInj − Parcae-DecayM−2.00 [−4.80, +0.80]+0.018 [−0.006, +0.043]

By BPB, our method is better in three of the four cases; in the fourth, the interval contains zero, so we count it as a tie.

W2. Size of the perplexity gains.

The gains are consistent: the learned prior lowers Val PPL relative to Huginn’s fixed prior, and OrthoInj lowers it relative to Parcae-Decay at every scale in the paper. Both changes are nearly free: the learned prior adds at most 1.6% training FLOPs, and neither adds inference cost.

The PPL gain carries over to downstream tasks. At 1.6B, with both models under terminal KV sharing (Table 2), the learned prior is at or above Huginn's fixed prior on all ten tasks:

1.6BLAMB.HSPIQAARC-EARC-COBQASciQGSM8KDROPMBPP+
Huginn's fixed prior55.1760.8073.5669.7438.7423.8093.3047.6134.1127.00
Learned prior55.1761.5773.7271.0940.4424.4093.6047.9234.5627.25
Δ0.00+0.77+0.16+1.35+1.70+0.60+0.30+0.31+0.45+0.25

Averaged over the seven tasks, BPB shows the same pattern:

BPB Δ (↓) [95% CI]100M400M1.6B
Learned prior − Huginn's fixed prior−0.003 [−0.008, +0.003]−0.011 [−0.015, −0.006]−0.059 [−0.063, −0.054]
OrthoInj − Parcae-Decay−0.009 [−0.014, −0.004]−0.036 [−0.041, −0.031]–

Every interval excludes zero except the learned prior at 100M, a tie, and its gain grows with scale. Under terminal sharing, fixed-depth training drops from 59.90 to 56.90 on the 7-task average and from 50.64 to 21.23 on GSM8K; the learned prior reaches 60.00, matching the full-cache average of fixed-depth training.

Q1. L182: shouldn't hrh_r be a function of tt, i.e., hrth_r^t?

Yes. Lemma 1 fixes the current token tt and drops the index; the revision writes htrh_t^r in Lemma 1 and Section 3.2.

Q2. L183: clarify "prefix-dependent input to its update".

C<trC^r_{<t} is everything that token tt's update at recurrence rr reads from earlier positions: the keys and values of tokens before tt in each physical attention layer at that recurrence, plus any other recurrent state they carry. The revision states this definition explicitly.

Thanks for providing the additional analyses.

Thank you for reading our response and for supporting the paper.

Reviewer_vxqDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Looped language models apply the same layers repeatedly before predicting the next token. This paper argues that, when their internal states settle, the final state can replace some of the work or storage associated with those repetitions. It proposes a learned distribution of training loop counts and orthogonal input injection (OrthoInj), and tests shared attention caches, distilled prompt processing, and reuse of saved states during reinforcement learning (RL).

The findings about training and cache sharing are useful. My main concerns are whether the learned prior improves on the authors’ stronger fixed-prior configurations, and whether the efficiency gains justify their quality and training costs.

Critical strengths

1.The training changes show useful improvements:

The learned depth distribution lowers reported validation perplexity at all three model sizes. With the authors' preferred setting, average task accuracy improves for the small and large models and is almost unchanged for the medium model (Table 2). An additional medium-model comparison improves WikiText perplexity from 13.16 to 12.77 and the four-task accuracy average from 47.29% to 47.78% (Table 10).

  1. The input change has supporting tests:

OrthoInj improves average task accuracy for the small and medium models. Among the input schemes compared, it scores highest on all three generation tasks for the medium model (Table 3). Table 12 separates the projection and normalization changes for the small model and finds a validation-perplexity benefit from projection in both settings.

3.The cache-sharing results have direct supporting tests:

For the medium models, the paper compares the same trained models with and without sharing, measuring both perplexity and task accuracy (Figure 5b; Table 8). The sampled-depth models retain their reported task averages and WikiText perplexity, supporting cache savings with little change in these measured results.

The faster prompt processing and RL updates add value:

The distilled prompt-processing model gives better average task accuracy than the cheaper single-loop alternative (Table 4). Reusing saved rollout states also reduces RL scoring and backward time. The authors provide timing checks and report total RL time, making the scope of that saving clear.

Critical weaknesses
  1. The smaller gains need repeat training runs:

Each setting is trained with one random seed, so the paper does not establish how reliably the gains would repeat. Without cache sharing, some small-model fixed-distribution settings in Table 13 have lower WikiText perplexity than the learned prior in Table 7. A closer comparison on both perplexity and accuracy would help. Keep the temperature constant while comparing other methods. This is a remaining question about the size and source of the benefit, rather than evidence that learning the distribution fails.

  1. The practical speed–quality trade-off needs broader testing:

For the medium model, distilled prompt processing cuts latency from 495.8 to 299.3 ms for 8K prompts at batch size eight. Average task accuracy falls by 2.30 points, and GSM8K falls from 15.62% to 8.95% (Table 4). The cheaper alternatives lose more average accuracy, so the result is useful, but its value depends on the workload. Total student-training cost, generation speed, and tests with other prompt lengths and batch sizes would help.

  1. Training and evaluation overlap needs clarification:

Appendix D.1 says document-level separation of MIXVAL from training is unverified. D.2 says WikiText was not cleaned to remove possible document overlap with pretraining data. This does not show that leakage occurred, but clearer split construction and overlap checks would strengthen confidence in the results.

Questions for authors
  1. The gains look smaller, do the small-model gains persist across additional training seeds and comparisons with similarly tuned fixed distributions?
  2. What is the total computation or time needed to train the prompt-processing student, and how does the speed–quality trade-off change across workloads?
  3. How were training and validation documents separated, and what overlap checks can be provided?
6 comments
AuthorsOpenReview ↗

Response to Reviewer vxqD

Thank you for the detailed review. We answer Q1–Q3 and the remaining points of W1 below.

Q1. Do the small-model gains persist across seeds and against similarly tuned fixed distributions?

100MSeedsVal PPLWikiText PPL7-task avg.
Fixed PLN-556.697 ± 0.01619.95 ± 0.2941.50 ± 0.29
Tuned PLN (σ=0.35\sigma=0.35)36.669 ± 0.00420.09 ± 0.4541.17 ± 0.15
Learned prior56.663 ± 0.03819.75 ± 0.1941.42 ± 0.28
Parcae-Decay36.776 ± 0.00320.16 ± 0.2241.37 ± 0.08
OrthoInj36.668 ± 0.02519.82 ± 0.2041.89 ± 0.29
  • Yes. The learned prior and OrthoInj have lower mean validation and WikiText PPL than their baselines. The learned prior's downstream average is within one seed standard deviation of Fixed PLN-5's, and OrthoInj's is higher than Parcae-Decay's (41.89 vs. 41.37).

  • The tuned PLN is the capped PLN closest to the learned distribution in KL, trained at the learned prior's learning rate. It matches the learned prior's validation PPL, but its mean WikiText PPL and downstream average are the weakest of the three depth priors. Its σ\sigma is also read off the learned run; without it, finding σ\sigma needs one full pretraining run per candidate.

W1. Some fixed-distribution rows in Table 13 have lower WikiText PPL than the learned prior in Table 7.

  • Those rows come from an earlier sweep that used antithetic sampling: each depth draw is paired with a mirrored one (quantiles uu and 1−u1-u), so draws come in shallow–deep pairs instead of independently. Every depth-sampling model in Tables 2 and 7, including the learned prior, draws depths independently, so those rows were not comparable with Table 7.

  • The revision replaces those rows with independent-draw runs. Table 13's conclusion is unchanged (the backward window barely affects quality), and every row now has higher WikiText PPL than the learned prior: 21.32, 21.47 and 21.15 vs. 21.12.

  • On temperature: all arms share one decoding protocol (greedy for GSM8K and DROP; eight samples at temperature 0.8 for MBPP+), and perplexity and the choice tasks involve no sampling.

Q2. What does the prefill student cost to train, and how does the trade-off change across workloads?

Training FLOPs100M400M
Final student≈1.2×10191.2\times10^{19}≈1.4×10201.4\times10^{20}
Teacher pretraining7.3×10197.3\times10^{19}9.2×10209.2\times10^{20}
Student / teacher≈16%≈15%
  • The student trains on a quarter of the teacher's tokens. Like the teacher's own prefill under terminal sharing, it writes only the four terminal KV banks, a 3× smaller cache than the full one, and the same teacher decodes from them, so per-token generation speed is unchanged. Across prompts of 512–8K tokens and batch sizes 8 and 32, the 400M student cuts time-to-first-token by 1.26–1.87×.

  • Because both prefill paths feed the same decoder, a server can choose one per request: the student for long prompts with short outputs, where prefill dominates latency, and the teacher's own prefill for long multi-step generation, where the student loses more.

As a comparison, the cheaper prefill paths lose more (400M, 8K prompts at batch size 8):

Prefill pathPrefill latency (ms)Downstream avg.
Teacher (R=5R=5)495.849.07
Distilled student299.346.77
One recurrence (R=1R=1)245.941.19
Dense D4229.844.08

Q3. How were training and validation documents separated, and what overlap checks can be provided?

Each of the 19 MixVal source files was cut out of its source before the remaining documents became training data. Two checks on the data:

  • Exact matching against all 146M training documents the runs read finds 0.16% of MixVal documents, covering 0.15% of scored Val tokens.
  • Excluding these tokens changes Val by at most 0.07%; gaps between models shift by at most 0.01 percentage points, and no ordering in Tables 2 and 3 changes.

WikiText-103 is not filtered against the pretraining data, so as an external check we also score the public Paloma test set (mean PPL over its C4, Dolma, Falcon RefinedWeb and RedPajama subsets, 64 packed 8K-token sequences each), where the learned prior and OrthoInj keep their gains:

Depth prior100M400M1.6B
Huginn's fixed prior (PLN-5)24.3816.4712.19
Learned prior23.8516.1112.02
Change−2.19%−2.17%−1.41%
Input injection100M400M
Parcae-Decay24.3616.31
OrthoInj23.9816.20
Change−1.57%−0.66%

As the rebuttal period draws to a close, I encourage reviewers to engage with the authors’ responses and revised manuscript, clearly identifying which concerns have been addressed and which remain unresolved.

Given the limited time remaining, please prioritize evaluating the evidence and clarifications already available in the current revision. If you nevertheless consider additional experiments essential to support the paper’s claims, please specify what is needed and why. Even if the authors cannot complete these experiments within the remaining time, a discussion of the underlying concerns and the available evidence would help inform our final assessment.

Thank you all for your time and constructive engagement.

Thanks. The repeated runs support the perplexity improvements, although the learned prior’s task accuracy remains similar to the fixed baseline. The OrthoInj ppl results are encouraging.

The added training costs and timing results make the speed–quality trade-off clearer. The overlap checks also reduce my concern about the evaluation data, though WikiText remains unfiltered.

I want to keep my weak-accept recommendation. The remaining limitations are the modest gains, limited replication beyond the small model, and the accuracy loss from faster prompt processing. I do not require further experiments for my current recommendation.

Reviewer_FXo4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how fixed-point behavior in looped language models can reduce training and inference costs. It examines truncated backpropagation, sharing terminal KV caches, distilling prefill, and reusing rollout states for RL updates. It also proposes a learned training-depth distribution and orthogonal input injection. Experiments on models from 100M to 1.6B non-embedding parameters show modest perplexity improvements, a 3x reduction in recurrent KV banks at depth five, and faster prefill with some accuracy loss.

Critical strengths
  1. The paper gives a useful overall view of how fixed points can improve the efficiency of looped language models, covering training, KV caching, prefill, and RL updates. The connection between these techniques is interesting and well motivated.

  2. The experiments are fairly extensive. The authors evaluate the learned depth prior at three model scales, compare input injection methods, and report both performance and efficiency results. They also acknowledge important tradeoffs, including accuracy loss in distilled prefill.

Critical weaknesses
  1. The quality of the writing can be improved; some places are confusing, especially to readers who are not familiar with loop LLMs.

  2. Some efficiency claims are overstated or unclear. The abstract claims a 5x KV-cache reduction, but Table 2 shows four banks instead of twelve, or 3x. The theoretical result assumes convergence, while Appendix A.1 shows that many prefixes need over 100 loops to converge, far beyond the five loops used in practice. Also, the reported 1.5x RL speedup applies to the update stage; the end-to-end improvement is much smaller.

  3. It is not fully clear which design choices account for the gains. OrthoInj changes input normalization as well as the projection, making the contribution of projection harder to isolate. The perplexity gains are fairly small, and each configuration uses only one training seed. Additional controlled comparisons and variance estimates would strengthen the conclusions.

Questions for authors
  1. Why does the abstract claim a 5x KV-cache reduction when the main experiment shows 3x?

  2. How closely do states and KV caches approach fixed points at the practical depth of five loops? Can you relate convergence measurements to the observed benefit of KV sharing?

  3. Can you compare OrthoInj and Parcae with identical normalization and initialization, changing only the projection?

  4. How consistent are the small perplexity gains across different training seeds? Would a tuned fixed depth distribution perform similarly to the learned one?

  5. Can you emphasize end-to-end latency and memory savings, rather than only the prefill or RL update stages?

1 comment
AuthorsOpenReview ↗

Response to Reviewer FXo4

Thank you for the careful reading and feedback.

Q1. About inconsistency of KV cache reduction

The 5× counts the recurrent core's cache, as the abstract says ("for a loop-5 recurrent core"): at R=5 the two core blocks store 10 KV banks without sharing and 2 with it. Table 2 counts the whole cache, which also keeps one unshared bank each for the prelude layer and coda layer: 12 → 4, or 3×. The revision names the cache next to each number.

Q2. How does convergence relate to the benefit of KV sharing?

At R=5 the states are close to but not at a fixed point. We measure the distance by the relative residual ρ5,t=∥ht5−ht4∥2/∥ht4∥2\rho_{5,t}=\lVert h_t^5-h_t^4\rVert_2/\lVert h_t^4\rVert_2, averaged over validation tokens, and by the same ratio for the keys and values the recurrent core writes (KV ρ5,t\rho_{5,t}). With sampled depths the state still moves by 5–7% at the fifth recurrence, with fixed-depth training by about 30%, and the cost of KV sharing follows:

PriorMean ρ5,t\rho_{5,t} (100M / 400M / 1.6B)Mean KV ρ5,t\rho_{5,t} (100M / 400M / 1.6B)Val PPL cost of KV sharing (100M / 400M / 1.6B)
Fixed R=50.33 / 0.31 / 0.270.26 / 0.17 / 0.12+3.26% / +3.98% / +3.54%
Fixed PLN-50.050 / 0.046 / 0.0510.050 / 0.054 / 0.047−0.01% / −0.01% / +0.10%
Learned prior (λH=0.01\lambda_H=0.01)0.049 / 0.057 / 0.0710.050 / 0.060 / 0.088+0.05% / +0.17% / +0.42%

Sharing is nearly free once the state moves little at the last recurrence; full convergence is not required (Spearman 0.93 over these nine checkpoints, p < 0.001; 0.78, p = 0.017, for the KV banks).

Q3. Further ablations about OrthoInj and Parcae

  • Parcae-Decay injects ζtr=Λhtr+qt\zeta_t^r=\Lambda h_t^r+q_t with qt=Δ⊙Wetq_t=\Delta\odot We_t (ete_t: normalized prelude output).
  • OrthoInj changes two things: it drops that RMSNorm and projects the carryover, ζtr=QqtΛhtr+qt\zeta_t^r=Q_{q_t}\Lambda h_t^r+q_t.

We cross the two changes, sharing initialization, data, sampled depths and learning rate (native Val PPL without KV sharing, 100M, 3 seeds):

Prelude normalizationParcae-Decay+ projection
RMSNorm6.689 ± 0.0156.689 ± 0.018
None6.673 ± 0.0246.618 ± 0.026

The projection without prelude RMSNorm has a synergy effect, where it lowers validation PPL in all three seeds.

Q4. Are the small gains consistent across seeds? Would a tuned fixed depth distribution perform similarly?

100MSeedsVal PPLWikiText PPL7-task avg.
Fixed PLN-556.697 ± 0.01619.95 ± 0.2941.50 ± 0.29
Tuned PLN (σ=0.35\sigma=0.35)36.669 ± 0.00420.09 ± 0.4541.17 ± 0.15
Learned prior56.663 ± 0.03819.75 ± 0.1941.42 ± 0.28
Parcae-Decay36.776 ± 0.00320.16 ± 0.2241.37 ± 0.08
OrthoInj36.668 ± 0.02519.82 ± 0.2041.89 ± 0.29
  • Yes. The learned prior and OrthoInj have lower mean validation and WikiText PPL than their baselines. The learned prior's downstream average is within one seed standard deviation of Fixed PLN-5's, and OrthoInj's is higher than Parcae-Decay's (41.89 vs. 41.37).

  • The tuned PLN is the capped PLN closest to the learned distribution in KL, trained at the learned prior's learning rate. It matches the learned prior's validation PPL, but its mean WikiText PPL and downstream average are the weakest of the three depth priors. Its σ\sigma is also read off the learned run; without it, finding σ\sigma needs one full pretraining run per candidate.

Q5. Can you emphasize end-to-end latency and memory?

We serve the 400M model at R=5R=5 on one H200 with 8K prompts. Decoding takes the same time per step with or without terminal KV sharing, about 23 ms at both 2K and 8K context, because every recurrence reads one KV bank per layer either way; a request's end-to-end latency therefore differs mainly in time-to-first-token (TTFT):

400M, 8K contextTTFT, batch 8 (ms)TTFT, batch 32 (ms)KV cache, batch 32 (GiB)Peak decoding memory, batch 32 (GiB)Max batch per H200
Full cache6922,7619.613.9440
Terminal sharing4961,9773.27.51,328
+ distilled prefill3001,180———

RL (1.6B, R=6R=6, GSM8K): as you note, reuse targets the update. It makes the update 1.5× faster than recomputation and 2× faster than full BPTT, and halves the memory the update needs beyond weights and optimizer state (11.3 GiB at any depth vs. 21.3 GiB for full BPTT at R=6R=6). Rollout, about 90% of a run, is unchanged, so the end-to-end saving is 1.9–3.7 of 62–65 minutes (3–6%).

W1. Writing for readers new to looped models. The revision adds an overview figure (Figure 1) and a notation box on page 2.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for their constructive feedback. The reviewers find the connection between fixed points and efficiency useful and well motivated (FXo4, r4xo), the experiments extensive across three scales (FXo4, r4xo), and the cache-sharing results directly supported (vxqD).

The revised PDF marks all changes in blue and collects the new results in Appendix F. The main text adds an overview figure (Figure 1) and a notation box for readers new to looped models, states which cache each KV reduction counts (5× for the recurrent core, 3× for the whole cache), and describes the RL gain as an update-stage speedup.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.