acceptodds
Under review as a conference paper at ICLR 2027

Mobius Learning: Cyclic Depth Folding in Transformers

Abstract

Transformer-based language models organize computation along an ordered depth axis, where shallow and deep blocks often develop distinct representational roles. We introduce , a training architecture that allows the same block parameters to serve multiple depth roles. Through cyclic depth folding, different data streams traverse shared block groups in cyclically shifted orders, jointly training each group in shallow and deep positions, a phenomenon we call depth-role superposition. In mixture-of-experts (MoE) language-model pretraining, combines effectively with looping, which reuses parameters through repeated passes over the same block sequence. At eight loop passes, achieves higher five-task zero-shot accuracy than standard looping across all four evaluated model sizes. It also supports effective data reuse: under eightfold repetition, raises five-task zero-shot accuracy from 46.13% to 48.29% relative to standard looping for the largest evaluated model. These findings establish depth-role superposition as a viable training mechanism that combines with looping and improves downstream performance under repeated-data training.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Representation learning, New training architecture

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VLCsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces Mobius Learning, a training method that removes the fixed association between Transformer blocks and their position in the network. The model is divided into groups of blocks, and different batches traverse these groups from different cyclic starting points, so the same group can act at both shallow and deep positions during training. The authors call this depth-role superposition. They show that this works well with looped Transformers (though performance varies with number of loops), where the same blocks are already reused over multiple passes, and also helps when training data is repeatedly reused. Experiments on MoE language models pretrained on FineWeb show improvements over standard looping, especially at greater loop depths and under heavy data repetition. The paper also provides a theoretical analysis suggesting that training blocks across multiple depth roles allows learned computations to transfer between roles.

Critical strengths

The core idea is interesting - using different blocks as entry points for the sequence allowing them to access shallow and deep representations.

Critical weaknesses

1 - The benefits over standard looping appear only after many rounds of looping 2 - Even then, the accuracy is not really much better over standard looping 3 - Experimental scale is very small though I can understand if this is due to compute concerns but validation on somewhat larger models would have been helpful especially considering these are MoEs so the actual flops/token are even smaller. 4 - The writing uses very BIG words to describe really SIMPLE things. Words like depth-role SUPERPOSITION / cyclic folding are really not needed and only confuse the reader.
5 - The diagram on Page 1 - part C serves absolutely no purpose and is in fact confusing. 6 - The experimental choices are confusing as well - more detail in questions but basically - why use MoEs? why compare to looping and why study reuse?

Questions for authors
  • Why is it called Mobius learning?
  • Why all the heavy terminology - cyclid depth folding / superposition and what not? It would be easier for future readers if these words were not used
  • Why use MoEs? You're already training really small models? why complicate the idea further by introducing MoEs? it would be easier to understand the method if it was done across simple dense transformers no? Is there any reason for this?
  • Why compare with looping? Depth folding is not parameter reuse rather allowing blocks to have access during training to both shallow + deep representations
  • Why talk about data reuse? Especially when in your model scales / data scale there is no reason to worry about it. Like what's the motivation at all?
  • Regarding mobius parallelism it says Local Embedding + Head/Loss. Does this mean each replica has a separate embedding / head with separate parameters or do you just have a single embedding / head and its replication across each device?
Reviewer_hcFJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Möbius Learning, which jointly trains shared Transformer block groups under cyclically shifted execution orders. This allows each group to occupy multiple source-relative depth positions. The method is evaluated in MoE language-model pretraining, both with repeated backbone passes and with repeated training data reassigned across cyclic starting points. The paper also provides theoretical results in a controlled causal neural network trained on a binary hidden Markov model.

Critical strengths
  1. Clear and interesting construction. Cyclic shifts provide a simple way to train shared parameters across depth positions while preserving a structured execution order.

  2. Promising results at high loop depth. At eight passes, origin-averaged downstream accuracy exceeds standard looping across all four evaluated model sizes. These results provide evidence beyond selecting the best origin.

  3. Theoretical investigation. The analysis identifies conditions under which multiple entrances and data reassignment improve prediction risk, offering a possible explanation for the empirical behavior.

  4. Transparent reporting. The authors explicitly disclose post-hoc origin selection, single-run experiments, and the limitations of nominal FLOPs estimates.

Critical weaknesses
  1. The data-reuse results are mixed. Across the five scales in Table 2, the average accuracy difference relative to standard looping is approximately −0.68 percentage points under fourfold repetition and −0.11 under eightfold repetition for origin-averaged performance. The corresponding best-origin differences are approximately +0.07 and +0.76 points. These results support gains for some selected predictors, but provide weaker evidence for a broadly reliable data-reuse advantage.

  2. Statistical reliability is unclear. All configurations use a single training run, and several gains are only a few tenths of a percentage point. Without multiple seeds or confidence intervals, it is difficult to distinguish robust improvements from training and evaluation variability.

  3. The mechanism is insufficiently isolated. Additional experiments should distinguish the effects of cyclic-order training, balanced depth-role coverage, and data reassignment. The exact-sharing ablation also reduces independent parameter capacity, limiting its explanatory value. Furthermore, changing the number of block groups between some repetition settings complicates interpretation.

  4. The theory-to-experiment connection remains limited. The analysis uses an attention-free network, structured symmetric initialization, Euclidean gradient descent, weak temporal dependence, and specific training windows. Under full-entrance population training, symmetry also keeps block parameters identical. It remains unclear how closely this mechanism describes randomly initialized, untied MoE Transformer groups trained with Muon.

Questions for authors

See weakness.

Reviewer_WmTBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Mobius Learning, a training scheme in which a Transformer's layers are split into P block groups each placed on its own worker. Each worker injects its local batch at its own group, so data streams traverse the same groups in cyclically shifted orders. Every group is thus trained at every within-pass depth. Gradients from all paths through a group contribute to a single update. The input embedding and output head are tied, shared across paths, and kept as a local copy on each worker, so the loss is computed where the labels live. The method is combined with looping and with data repetition, where repeated data windows are reassigned to different entry points across replays. A theory section on a binary-HMM toy model proves an early-training risk advantage for multi-entrance training and a test-risk benefit from reassigning repeated data across entrances.

Critical strengths
  1. Simple, original idea: multiple shifted entry orders over shared block groups is a distinct point in the space of looped/shared-depth models, clearly distinguished from LayerShuffle, CYCLE sharing, and model-/layer-loop schedules.

  2. Compute-matched comparison against standard looping within each loop depth (L = 1, 2, 4, 8) and across several model sizes.

  3. The paper has an innovative idea.

  4. Ambitious theoretical effort with explicit statements of scope and limitations.

  5. Reasonably transparent reporting of Mean vs Best metrics and of some limitations (single runs, a lost validation hash).

Critical weaknesses
  1. The paper's main weakness is that the empirical evidence is thin. Every result comes from a single run, and the "Best" scores pick the entry point using the same test sets they're reported on. Averaged over entry points, the headline Large 8× gain shrinks to about half a point. The larger gains are concentrated in the smallest tasks, especially SciQ, while HellaSwag favors standard training in every data-reuse comparison. At one loop pass, standard looping beats even the best origin at all three sizes. Möbius wins a couple of earlier cells, but the advantage across all tested sizes appears only at eight loop passes. In the data-reuse experiments, averaged over entry points, it ends up slightly below standard training across model sizes.

  2. The cost of looping is never justified. Eight passes cost roughly eight times the block compute and latency, yet there is no comparison with a larger non-looped model at the same compute. The paper's own data come close only under fourfold repetition: a much larger model at one pass uses about the same training FLOPs and scores about the same as the small eight-pass Möbius model. Under eightfold repetition that larger model falls clearly behind, so the tie does not hold for repeated data in general. Relatedly, each entry point is effectively a different model, and the paper never measures whether they agree or explains how to choose one without the test set.

  3. The theory doesn't explain the experiments. The analyzed model has no attention. Under its symmetric training, Mobius reduces exactly to weight sharing, and for the paper's 12-block, four-group setup that is precisely the exact-sharing configuration the experiments show performs worse. The theorems also cover neither the partition nor the reassignment schedule actually used, and the advantages are proven only in narrow early training windows.

  4. The evaluation is entirely teacher-forced and no text is ever generated, so it is unknown how the model, and its different entry points, behave in actual generation. This matters because Mobius is a new training scheme in which each block group is trained to serve at different depths of the network, and good next-token loss does not by itself guarantee coherent long-form generation. An evaluation on generation tasks is therefore missing, for example dialogue summarization (DialogSum) and similar datasets and compare it with the baselines is important.

Questions for authors
  1. Can you provide sampled generations and a generative evaluation, across entry points and against standard looping?

  2. How does Mobius at L=8 compare to a larger non-looped model at matched training and inference FLOPs, ideally on fresh data? What are the latency and KV-cache costs?

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Manuscript Update

Dear Area Chair and Reviewers,

We appreciate your time, careful reading, and constructive feedback. We were encouraged that the reviewers recognized the simplicity and originality of our idea.

We have revised the manuscript to make the main contribution explicit: Möbius Learning is a completely new training architecture that shows that language models using different cyclic orders of the same Transformer blocks can be pretrained jointly from scratch. That a single set of block weights can learn to serve these different roles is a very surprising result in its own right. The reuse experiments build on this finding, showing that Möbius Learning can outperform standard looping both with repeated passes through the blocks and with repeated training data. The revision also clarifies these comparisons and the scope of the theoretical results.

We genuinely believe this work opens a very important new direction for language model training. We are grateful for the feedback that helped us communicate this contribution more clearly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.