- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper systematically studies how loop count, non-recurrent layer placement, and conditioning affect looped language models. Additional recurrence can improve reasoning while hurting knowledge performance, and harder examples do not necessarily benefit more. The authors also propose combining history-state differences with timestep gating.
Critical strengths
S1: Systematic controlled experiments. The paper examines multiple architectural factors and stratifies tasks by reasoning depth, helping clarify when recurrence is useful.
S2: Informative negative results. More loops, more complex state injection, and more conditioning mechanisms do not guarantee better performance. These findings provide useful guidance for model design.
Critical weaknesses
W1: Limited robustness of the conclusions. All training runs use a single seed, and the combined method is mainly evaluated on Qwen 4×7. Its overall score is only marginally higher than the strongest simple history-only variant, limiting support for a broad design recommendation.
W2: imited validation across model scales. The experiments are restricted to small models, and the combined conditioning method is mainly evaluated on Qwen 4×7, with approximately 219M parameters. Its effectiveness at larger scales, such as 7B or above, remains unverified, limiting the generality of the design recommendations.
Questions for authors
Q1: Do the gains from combining history-state conditioning with timestep gating hold across additional seeds or architectures? How does the combination compare with the strongest history-only variant under the same settings?
Q2: Do you have results at larger model scales, or evidence showing how the gains change with model size? Such results would help assess whether the proposed conditioning method remains effective beyond the small-model settings studied here.
7 comments
Response to Reviewer fexH [update]
Update: large-scale results in W2 / Q2
The larger-scale runs announced in our earlier response have finished: Llama BaseLoop with 1.01B unique parameters, training depth (the depth of a 3.5B-parameter non-recurrent model), and 17.31B training tokens. Overall accuracy, with Knowledge / Reasoning in parentheses:
| Method | (train) | () | () |
|---|---|---|---|
| H | 38.99 (63.12 / 29.94) | 38.97 (62.28 / 30.23) | 38.22 (60.53 / 29.86) |
| H+LG | 38.78 (63.00 / 29.70) | 38.82 (62.47 / 29.95) | 38.31 (60.94 / 29.82) |
- History-conditioned recurrence stays robust well beyond the training horizon at this scale.
- At twice the training depth, Overall is unchanged (H: 38.97 vs. 38.99; H+LG: 38.82 vs. 38.78), and Reasoning rises slightly for both.
- At three times the training depth, both models keep more than 95% of their training-depth Knowledge (H: 60.53 / 63.12; H+LG: 60.94 / 63.00).
- For context, at the smaller scale the unconditioned Llama BaseLoop already loses 10.69 Knowledge points at depth (62.80 → 52.11).
- H+LG performs on par with H at every inference budget. Overall differences are at most 0.21 points, well within evaluation error, and H+LG has slightly higher Knowledge at depth (+0.41).
These results show that the knowledge-preserving extrapolation behavior of history-state conditioning, observed at 0.2–0.5B unique parameters, persists at 1B unique parameters. We will add this baseline, together with the results above, to App. E.5 of the revision.
Dear Reviewer fexH,
We would like to kindly follow up on our rebuttal and ask whether our responses have addressed your concerns. We would greatly appreciate any further feedback or questions you may have.
Thank you for your time and consideration.
Response to Reviewer fexH
We thank the reviewer for highlighting the controlled experiments and the value of the negative results. We agree that robustness across training seeds and the scale of the method-specific evidence matter for a design recommendation, and we address both below.
W1 / Q1. Robustness across seeds and architectures
Statistical significance at the original seed. We propagate the lm-evaluation-harness standard errors of the 11 tasks to the Overall score, which gives about 0.36 points per model. We then test the differences at D=84 on Qwen 4×7 (unpaired, which is conservative because all models are evaluated on the same items):
The gains over BaseLoop, LG and the matched H component exceed evaluation sampling error. The 0.06-point margin over scalar history w=4, the strongest history-only variant, does not, and we do not claim a ranking between those two.
Additional training seed. Evaluation error does not capture variation across training runs. We therefore retrained H, H+LG and scalar history w=4 with seed 123, keeping the data order and method definitions fixed. Overall accuracy on Qwen 4×7 (training depth D=28):
Additional architectures. We trained H and H+LG on Llama BaseLoop 4×5 and 5×4 (training depth D=20).
The full table is in our response to Reviewer 3FVq, W3(a). All new runs are added to App. E.3 and E.5.
W2 / Q2. Larger-scale validation and dependence on model size
We are extending the H vs. H+LG comparison to a substantially larger model, the largest we can train within the discussion period:
Training of these models is still running. We will post the results in this thread as soon as they finish, together with a comparison to the smaller-scale trend.