acceptodds
Under review as a conference paper at ICLR 2027

Universal YOCO for Efficient Depth Scaling

Abstract

The rise of test-time scaling has remarkably boosted the reasoning and agentic proficiency of Large Language Models (LLMs). Yet, standard Transformers struggle to scale inference-time compute efficiently, as conventional looping strategies suffer from high computational overhead and a KV cache that inflates alongside model depth. We present Universal YOCO (YOCO-U), which combines the YOCO decoder-decoder architecture with recursive computation to achieve a synergistic effect greater than either alone. Built on the YOCO framework, YOCO-U implements a Universal Self-Decoder that performs multiple iterations via parameter sharing, while confining the iterative process to shallow, efficient-attention layers. This combination yields a favorable capability-efficiency tradeoff that neither YOCO nor recursion achieves independently. The YOCO architecture provides a constant global KV cache and linear pre-filling, while partial recursion enhances representational depth with limited overhead. Together, YOCO-U improves token utility and scaling behavior while maintaining efficient inference. Empirical results confirm that YOCO-U remains highly competitive in general and long-context benchmarks, demonstrating that the integration of efficient-attention architectures and recursive computation is a promising direction for scalable LLMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Architecture

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Q4jSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

YOCO-U introduces weight-shared recursion into YOCO’s efficient-attention self-decoder while retaining the cross-decoder’s shared global KV cache. The objective is to improve effective depth and parameter utility without the global-cache growth of recursive full-attention Transformers. The paper evaluates language-modeling quality, architectural alternatives, scaling behavior, and inference efficiency.

Critical strengths
  1. The integration offers a useful quality–efficiency trade-off. Table 2 reports a 4.45-percentage-point average improvement over YOCO under the stated equal-FLOPs comparison.

  2. The dense architecture comparison achieves approximately the same average accuracy as RINS while reporting substantial long-context throughput advantages.

  3. Loop-position, KV-sharing, and depth–width ablations address relevant design choices. The combination of MoE experiments, dense comparisons, and serving measurements gives the contribution practical relevance.

Critical weaknesses
  1. Compute accounting needs clarification. Figure 2 labels its horizontal axis as training FLOPs, although the main analysis uses tokens multiplied by executed layers. Appendix F estimates FLOPs at 2048 tokens rather than the actual 8192-token training length. The exact equal-FLOPs budgets and the criterion for removing one validation-loss point per model should also be specified.

  2. Quality and efficiency are demonstrated over different context-length ranges. The retrieval and book/code quality experiments extend to 8K, whereas serving measurements reach 128K/256K. The latter do not establish that quality is maintained at those lengths.

  3. Fixed-depth recursive training should be distinguished from increasing the depth of one checkpoint at inference. The additional cost relative to YOCO is also context dependent: decoding throughput is about 28% lower at 8K, versus about 5% lower at 256K. These qualifications limit broad negligible-overhead and test-time-scaling interpretations.

Questions for authors
  1. Can you provide the exact token/step budgets, FLOP estimates at the actual training length, and unfiltered validation-loss measurements, together with the outlier-removal criterion?

  2. Is a limited quality evaluation at 32K or 128K available under settings corresponding to the serving measurements?

  3. Please align the main scaling claims with the fixed-recursion setting acknowledged in Appendix G, or provide a single-checkpoint inference-depth sweep if available.

2 comments

Thank you for clarifying the training budgets. I am keeping my rating at 3.

Thank you for acknowledging our clarifications on the training budgets and for your constructive suggestions on improving the precision of our experimental claims.

Reviewer_eWUMOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the author proposes Universal YOCO (YOCO-U), which improves language model efficiency by combining YOCO’s shared KV cache architecture with recursive computation. YOCO-U loops YOCO's sliding-window bottom half three times. It gets looped-model quality with YOCO-level KV cache. The engineering seems clean. Experiments show improved data efficiency and downstream performance: under an equal training-compute budget, YOCO-U raises average accuracy across eight benchmarks from 41.78% to 46.23%.

Critical strengths

The design is simple, easy to adopt, and comes with real serving numbers from a vLLM-style engine.

The 2×2 setup (Transformer / YOCO, with and without recursion) is clear.

Exact per-loop computation with no shared-KV approximation is sound and the evaluation also reported equal-FLOP results.

Critical weaknesses

Weaknesses

  • PLT and MoR KV sharing go after the same memory problem and are neither compared against nor discussed that way. PLT is filed under "parallel scaling."
  • The test-time-scaling motivation is never delivered: the loop count is fixed at 3 in both training and inference.
  • Headline numbers are overstated:
    • "62% fewer tokens" is about 24% less compute once the doubled FLOPs per token are counted.
    • "Only 5% decode slowdown" holds only at 256K; it's 28% slower at 8K.
    • The advantage shrinks at larger scale (Fig 5).
  • The evidence is thin: one seed, gains of about 1 point at 20B tokens, and the RINS baseline config doesn't match RINS.
  • The math SFT gains (MATH 41.5→74.9) are too big to accept without explanation.
Questions for authors
  1. Head-to-head against PLT and RINS with MoR KV sharing: is exact per-loop computation better than reusing the first loop's KV?
  • Use the same data, tokens and roughly 2× FLOPs, and match KV cache within about 10%.
  • Report quality, prefill and decode throughput, and cache size.
  1. Sanity checks on the math SFT result.
  • Evaluate with sampling (avg@k) instead of greedy decoding only.
  • Also SFT the equal-FLOP YOCO checkpoint.
  • A 33-point MATH gap from about 2× compute seems strange.
Reviewer_Df3bOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper claims that test-time scaling has boosted LLM reasoning, but that standard Transformers scale inference compute inefficiently. Looping the whole network is expensive, and the KV cache grows with depth. To address this, the paper presents Universal YOCO (YOCO-U). It combines the YOCO decoder-decoder design with recursive computation. YOCO splits the model into a Self-Decoder, which reads the input with efficient (sliding-window) attention and produces one compact global KV cache, and a Cross-Decoder, which reuses that single cache across all of its layers. The key idea is to make only the shallow Self-Decoder recursive, iterating it T times with shared parameters. This adds representational depth without adding parameters, and crucially without touching the one-piece KV cache or the linear pre-fill. So YOCO-U aims to get the capability of looped models while keeping YOCO's cheap serving. Experiments span a 10B/1.3B-activated MoE trained on 300B tokens, a controlled 1.3B dense comparison against Universal Transformer, RINS and ParScale, long-context perplexity and needle-in-a-haystack, and scaling sweeps over both model size and loop count. The headline findings are that YOCO-U beats non-recursive YOCO (+4.45 average at equal FLOPs), matches the strongest recursive baseline RINS on capability (48.3 average), and does so at a fraction of the serving cost (roughly 38× less KV cache, and 10×/5× higher pre-fill/decode throughput than RINS at long context).

Critical strengths
  • The central design choice is simple and well-motivated. Recursion is confined to the shallow, efficient-attention Self-Decoder. That one restriction is what makes the whole thing work. It buys Universal-Transformer-style depth without regenerating per-layer KV caches, and without re-running the global attention. So YOCO-U keeps YOCO's O(N) one-piece cache and linear pre-fill while still scaling depth. It is a clean idea, and the paper states exactly why it holds.

  • The inference-efficiency story is strong and carefully measured. At long context YOCO-U keeps essentially YOCO's memory, while RINS needs about 38× more KV cache (542 MB versus 20,480 MB at 256K). Decoding gives up only about 5% throughput versus non-recursive YOCO. Pre-fill is roughly 10× a vanilla Transformer and 20× RINS. This matters because the point of recursion is to pay compute for capability. Here the serving cost barely moves, which is the real win.

  • The ablations are the right ones, and they actually support the design. Looping the Self-Decoder beats looping the Cross-Decoder (48.25 versus 47.34), and sharing the KV cache matters (48.25 versus 46.41). There are also controlled comparisons at 1.3B dense against Universal Transformer, RINS and ParScale, plus parameter scaling from 300M to 10.8B and loop-count sweeps over T. The design-level conclusions (bottom-block recursion beats all-block, recursive beats parallel) are useful and reasonably backed.

Critical weaknesses
  • The novelty is mostly a combination of existing pieces, and the framing oversells it a little. At its core YOCO-U is a looped, depth-sharing transformer. It reuses the shared Self-Decoder T times to add effective depth at a fixed parameter count. That is exactly the looped / recurrent-depth idea, and it is a crowded and active space. Universal Transformer, RINS, Mixture-of-Recursions, the recurrent-depth "encode-think-decode" line, and recent looped models such as Ouro and Nanbeige all do a version of this. The paper cites several of them (Universal Transformer, RINS, Mixture-of-Recursions, Koishekenov et al.), but it only benchmarks against Universal Transformer, RINS and ParScale. So the genuinely new part is narrow.

  • The headline capability result is parity, not a win. YOCO-U matches RINS at 48.3 average. So the actual selling point is efficiency at equal capability, not better capability. On top of that, the lead comparison in Table 2 is "Equal Steps" (+5.3 over YOCO), but at T=3 that is about 2× the FLOPs. The more honest "Equal FLOPS" row is +4.45. Leading with equal-steps conflates extra compute with the architecture, and the reader has to dig for the equal-FLOP number.

  • In all honesty, the scaling evidence cuts against the method at the scales that matter most. The parameter sweep shows the YOCO-U advantage narrowing, and becoming "near-comparable" to non-recursive variants above about 10B activated parameters. The authors report this plainly, which is good. But it suggests the benefit may wash out at the scale of large frontier models, and the big 300B-token result is a single MoE configuration rather than a trend.

Questions for authors
  • Table 2 leads with "Equal Steps" (+5.3), but at T=3 that is about 2× the FLOPs of YOCO, and the "Equal FLOPS" row is +4.45. Which is the intended headline? It would be good to make the equal-FLOP comparison the primary one, since equal-steps mixes extra compute with the architecture.

  • The parameter sweep shows the advantage narrowing to "near-comparable" above ~10B activated parameters. Is there any evidence the benefit persists at larger scale, or is partial recursion mainly a small-to-mid-scale win?

  • No seed variance is reported, and some gaps are small (0.033 loss; 48.3 versus 48.3 against RINS). Could at least the controlled 1.3B comparisons be run with a few seeds, so the margins can be read against noise?

3 comments
AuthorsOpenReview ↗

Additional Clarification on Architecture and Scaling

Thank you again for the time and care you put into reviewing our work. We appreciate your recognition of the design and serving results, and your questions about the contribution and scaling. Our earlier reply was brief, so we would like to explain these points more fully, drawing on the existing results below.

1. Looping the Self-Decoder adds depth while keeping the global KV cache unchanged.

The following comparisons use 1.3B models.

ComparisonResult
Loop placement (Table 5)Avg. accuracy: Self-Decoder 48.25, Cross-Decoder 47.34
Extra KV cache vs. YOCO (Appendix D)+20 MB across tested 8K–256K contexts
KV cache at 256K (Appendix D)YOCO-U 542 / YOCO 522 / RINS 20,480 MB

Table 5 helps explain why we chose this layout. The Self-Decoder uses efficient attention, while the Cross-Decoder still reads the full context from a shared global cache. More loops add local computation and local cache storage, but do not add full-attention layers or enlarge the global cache. Prefill also remains linear in sequence length. So matching RINS's reported average accuracy comes with much lower cache requirements. The 256K figures measure serving cost. We have not tested quality parity at that length.

2. Figure 5's right panel compares YOCO-U with non-recursive YOCO using twice as many independent parameters.

All YOCO-U models in this sweep use T=3T=3: the first half runs three times and the second half once. This doubles parameter computation without adding independent parameters. The right panel matches this parameter computation, with both model families trained on 20B tokens (Appendix C). The left panel compares models at the same independent parameter count.

As the curves get closer, YOCO-U is catching up with a model that has twice as many independent parameters. Similar loss here means getting comparable quality from half the parameters. It does not mean that the gain over a model of the same size is disappearing. We do not yet know whether this continues at frontier scales.

3. All eight benchmarks improve with half as many training tokens at the matched compute budget.

These results use 150B training tokens for YOCO-U and 300B for YOCO at the same model size. We match the budget using tokens × executed layers, which omits attention.

ComparisonResult
Equal FLOPs (Table 2)+4.45 points on average, gains on 8/8 tasks
GSM8K, equal FLOPs (Table 2)YOCO 38.06 → YOCO-U 50.49

Under this budget, spending compute on recursion works better than training on more tokens. We have not measured how much these results vary across pretraining seeds.

Thank you for the response. After taking that into consideration along with the other reviewer comments and scores, I still feel the value-add of YOCO-U over what YOCO and Loop Transformer already provide is incremental. However, given the additional experiments showing improvements in performance at lower token budget compared to YOCO are impressive. I will raise my rating to a 3.

Thank you for reconsidering our work and recognizing the significance of the empirical improvements! We sincerely appreciate your thoughtful evaluation and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.