- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper introduces Mobius Learning, a training method that removes the fixed association between Transformer blocks and their position in the network. The model is divided into groups of blocks, and different batches traverse these groups from different cyclic starting points, so the same group can act at both shallow and deep positions during training. The authors call this depth-role superposition. They show that this works well with looped Transformers (though performance varies with number of loops), where the same blocks are already reused over multiple passes, and also helps when training data is repeatedly reused. Experiments on MoE language models pretrained on FineWeb show improvements over standard looping, especially at greater loop depths and under heavy data repetition. The paper also provides a theoretical analysis suggesting that training blocks across multiple depth roles allows learned computations to transfer between roles.
Critical strengths
The core idea is interesting - using different blocks as entry points for the sequence allowing them to access shallow and deep representations.
Critical weaknesses
1 - The benefits over standard looping appear only after many rounds of looping
2 - Even then, the accuracy is not really much better over standard looping
3 - Experimental scale is very small though I can understand if this is due to compute concerns but validation on somewhat larger models would have been helpful especially considering these are MoEs so the actual flops/token are even smaller.
4 - The writing uses very BIG words to describe really SIMPLE things. Words like depth-role SUPERPOSITION / cyclic folding are really not needed and only confuse the reader.
5 - The diagram on Page 1 - part C serves absolutely no purpose and is in fact confusing.
6 - The experimental choices are confusing as well - more detail in questions but basically - why use MoEs? why compare to looping and why study reuse?
Questions for authors
- Why is it called Mobius learning?
- Why all the heavy terminology - cyclid depth folding / superposition and what not? It would be easier for future readers if these words were not used
- Why use MoEs? You're already training really small models? why complicate the idea further by introducing MoEs? it would be easier to understand the method if it was done across simple dense transformers no? Is there any reason for this?
- Why compare with looping? Depth folding is not parameter reuse rather allowing blocks to have access during training to both shallow + deep representations
- Why talk about data reuse? Especially when in your model scales / data scale there is no reason to worry about it. Like what's the motivation at all?
- Regarding mobius parallelism it says Local Embedding + Head/Loss. Does this mean each replica has a separate embedding / head with separate parameters or do you just have a single embedding / head and its replication across each device?
General Response and Manuscript Update
Dear Area Chair and Reviewers,
We appreciate your time, careful reading, and constructive feedback. We were encouraged that the reviewers recognized the simplicity and originality of our idea.
We have revised the manuscript to make the main contribution explicit: Möbius Learning is a completely new training architecture that shows that language models using different cyclic orders of the same Transformer blocks can be pretrained jointly from scratch. That a single set of block weights can learn to serve these different roles is a very surprising result in its own right. The reuse experiments build on this finding, showing that Möbius Learning can outperform standard looping both with repeated passes through the blocks and with repeated training data. The revision also clarifies these comparisons and the scope of the theoretical results.
We genuinely believe this work opens a very important new direction for language model training. We are grateful for the feedback that helped us communicate this contribution more clearly.