acceptodds
Under review as a conference paper at ICLR 2027

Jacobi Recirculation: Parallelizing Recirculation with Fixed-Point Iteration

Abstract

A growing line of work lets a transformer reuse its own deeper representations in later computation, increasing its effective depth without adding parameters. One recent approach is recirculation, which feeds the state of each token at a deeper layer back into a shallower layer, thereby rewriting the key–value cache for later tokens to attend to. However, this token-wise recursion makes processing a known sequence expensive in prompt prefill, evaluation, hyperparameter selection, and the training of its auxiliary mixer. In this work, we propose Jacobi recirculation, which replaces the recursion with a few rounds of parallel updates over all token positions. We prove that the iteration is exact after finitely many rounds, converges geometrically under a local contraction condition, and yields convergent training gradients under additional regularity. Experiments across model families and sizes show that two to three rounds suffice to reproduce sequential recirculation, even though the pretrained models were never trained for this iteration. When used for prompt prefill before generation, it also preserves downstream accuracy. In our experiments, it speeds up single-sequence prefill by more than an order of magnitude and reduces mixer training from hours to minutes. Together, these gains make recirculation substantially cheaper to use and to study. [Code](https://anonymous.4open.science/r/jacobi-recirculation-review-5846/README.md) is available.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Recirculation, Recurrent Connections, LLM Efficiency, Looped Transformer

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5wakOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Jacobi recirculation, replacing token-sequential recirculation with a small number of parallel updates over a known sequence. It establishes finite-round exactness and conditional convergence guarantees, and empirically demonstrates that two to three rounds often provide a sufficiently accurate approximation for evaluation, mixer training, and prompt prefill. The method substantially reduces the cost of these computations, with particularly strong gains in mixer training.

Critical strengths
  1. The method substantially reduces a genuine computational bottleneck in recirculation, particularly during mixer training. The training gains remain substantial when each method uses its largest feasible batch on a single GPU. This provides a concrete practical benefit beyond the favorable single-sequence prefill setting.

  2. The approximation is evaluated extensively across models, mixing configurations, sequence lengths, numerical precisions, and downstream tasks. The rounding-scale comparisons, batch-scaling analysis, and optimized sequential baseline strengthen the evidence that the reported gains preserve useful behavior and are not limited to one favorable configuration.

Critical weaknesses
  1. The broader significance remains conditional on the usefulness of recirculation itself. The paper convincingly makes recirculation cheaper, but does not establish its accuracy–cost competitiveness against alternative adaptation or additional-computation strategies; Appendix A explicitly leaves matched-compute comparisons to future work. The strongest demonstrated benefits concern training and known-sequence processing, while end-to-end generation gains are approximately 1.1–1.3×. I regard this as a limitation of the demonstrated practical scope, rather than a failure to support the stated acceleration claims.

  2. The algorithmic novelty is relatively limited. Parallel fixed-point iteration for sequential computations is an established approach, as the paper acknowledges. The principal advance appears to be its formulation and careful validation for recirculation on frozen pretrained models, especially the empirical finding that a small number of rounds can support both evaluation and training. This is useful, but the distinction from existing parallel fixed-point methods limits my assessment of the methodological contribution.

Questions for authors

Which parts of the convergence and gradient analyses rely specifically on the structure of recirculation, rather than following from general results for causal fixed-point iterations? Clarifying this distinction would help assess the technical contribution beyond the empirical finding that a few rounds suffice.

I also have two minor suggestions for improving the figures. In the caption or nearby legend of Figure 1(C), please clarify that “Evaluation” refers to perplexity evaluation without autoregressive decoding. In Figure 3, please reposition the legends in panels A and C so that they do not obscure the plotted data, and make the clustered SR and JR mixer markers in panel A easier to distinguish. These are presentation suggestions and are not central to my overall recommendation.

2 comments
Reviewer_5wakOpenReview ↗

Follow-up on the Author Response

Thank you for the response. My question about which parts of the analysis are specific to recirculation has been answered, the figure suggestions have been implemented, and Tables R1–R2 address my concern about generation gains by showing when the end-to-end speedup becomes substantial. The general response also clarifies the contribution relative to prior work on parallel fixed-point methods.

These clarifications resolve my technical questions, but do not change my overall assessment of the contribution's significance, which remains tied to the practical value of recirculation itself. I view this remaining reservation as a question of scope rather than technical soundness, and I do not consider additional experiments essential to complete my assessment. That said, my overall assessment remains positive. The early interest in recirculation, including independent reimplementations, is encouraging, and the practical value of this work could grow if recirculation gains wider adoption. Taking the clarifications and additional results into account, I maintain my rating.

Thank you for the follow-up. We are glad that our response resolved your technical questions, and we appreciate your positive assessment.

Reviewer_ebWzOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper replaces token-sequential recirculation with parallel fixed-point rounds over a known sequence. It proves finite-round exactness and conditional forward/gradient convergence, then shows that two or three rounds often preserve sequential recirculation's task performance. Experiments evaluate prefill, mixer training, and generation across model families and GPU platforms.

Critical strengths
  1. The method makes recirculation substantially easier to train and evaluate, with large mixer-training and low-batch prefill gains. The paper clearly separates these from smaller end-to-end generation gains.
  2. Validation is thorough: cache/logit errors, rounding controls, trained-mixer comparisons, generation outcomes, and measurements across AMD and NVIDIA hardware support the main empirical result.
Critical weaknesses
  1. Reliable selection of the round count remains unresolved. Appendix E.4 reports a two-round batch-gradient cosine of 0.02 at one checkpoint, and additional rounds need not improve individual gradients monotonically. Despite favorable final training results, these cases limit confidence in uniformly reliable low-round training.
  2. Acceleration is concentrated in known-sequence processing. Autoregressive decoding remains sequential, and total generation speedups are only 1.1–1.3×, limiting the scope of the otherwise substantial gains.
Questions for authors
  1. How frequently do large batch-gradient errors occur across checkpoints and seeds, and can an observable residual identify them without computing the sequential reference?
  2. What is the end-to-end quality/latency trade-off at K=3 in the generation settings where K=2 loses accuracy?
Reviewer_SMdMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Jacobi Recirculation (JR), a method for parallelizing the token-wise sequential computation required by recirculation in pretrained language models. The authors formulate recirculation as a causal fixed-point problem and approximate its sequential solution with a small number of parallel Jacobi-style iterations. They provide theoretical results establishing finite-round exactness, geometric convergence under local contraction conditions, and convergence of gradients through finite unrolling. Experiments across Gemma 3, Llama 3.1, and Qwen3 models show that two to three rounds generally preserve the task performance and training outcomes of sequential recirculation. The method achieves substantial acceleration in low-batch prompt prefill and auxiliary mixer training, with smaller improvements in full autoregressive generation.

Critical strengths

[S1] Substantial practical acceleration with careful systems evaluation. The method directly addresses a significant computational bottleneck of sequential recirculation. It achieves large speedups in single-sequence prefill and auxiliary mixer training, while preserving comparable task performance. The paper also provides complete evaluation and generation workloads, analyzes throughput as a function of batch size and sequence length, and presents experiments on multiple GPU architectures. These results make the contribution practically meaningful rather than merely demonstrating theoretical parallelizability.

[S2] Comprehensive theoretical and empirical characterization of approximation accuracy. The paper provides a clear fixed-point formulation, proves finite-step exactness from causality, and derives conditional guarantees for forward convergence and gradient approximation. The experiments cover different model families, precisions, mixing strengths, loop lengths, layer configurations, and training scenarios. In particular, the comparisons with sequential computation and the analyses of numerical rounding, gradient similarity, and generated-answer agreement provide substantial support for the proposed method.

Critical weaknesses

[W1] Novelty. Reformulating sequential neural computation as a fixed-point problem and applying parallel Jacobi iterations has been explored in prior work, particularly [Song et al., ICML’21], [Santilli et al., ACL’23], and [Wu et al., EMNLP’25]. Therefore, the main novelty would be adapting this framework to recirculation and demonstrating its practical effectiveness. Although the paper derives recirculation-specific bounds, the sufficient contraction assumptions are not verified for the evaluated pre-trained models. Thus, the theory characterizes conditions under which the approach works rather than establishing why two to three rounds are sufficient for the actual architectures explored.

[W2] Fidelity and practical benefits. The large reported speedups primarily concern low-batch prompt prefill and mixer training. In the evaluated GSM8K generation setting, where sequential decoding remains unchanged, the complete runtime improves by only approximately $\times$1.1. For example, similar task accuracy does not imply equivalent generated outputs. In other words, with two rounds, exact generated-text agreement is only 16-45% on GSM8K. Section E.4 in App. also identifies a checkpoint where the two-round batch gradient has cosine similarity as low as 0.02 with the exact gradient, while an additional round corrects this example. These findings suggest that a fixed small iteration count may not be uniformly reliable, especially for sensitive training or generation workloads.

Questions for authors

[Q1] Convergence assumptions in practice. The theoretical convergence guarantees require local sensitivity bounds and a contraction condition, which are not verified for the evaluated models. Could the authors provide empirical estimates of the local update Jacobian norm or spectral radius across different mixing strengths, layer pairs, and models? This would help establish whether the proposed theory explains the observed rapid convergence.

[Q2] Reliability of finite-round gradient approximation. E.4 in App. reports that the two-round batch gradient can differ substantially from the exact gradient. How frequently do such discrepancies occur across training steps and batches, and what is their effect on convergence or final mixer quality? A comparison of two-round and three-round mixer training under the same optimization settings would help assess the reliability of the default iteration count.

[Q3] Practical speed-accuracy trade-off. Could the authors provide a speed-accuracy Pareto comparison across iteration counts under realistic end-to-end generation workloads, including different prompt-to-generation length ratios and batch sizes? In particular, when would two or three rounds provide a practically significant advantage over sequential recirculation, given that the largest measured speedups are concentrated in prefill and mixer training?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.