acceptodds
Under review as a conference paper at ICLR 2027

DeepLoop: Depth Scaling for Looped Transformers

Abstract

Looped Transformers scale sequential computation by applying a compact stack of physical blocks for multiple rounds, increasing unrolled depth without increasing stored parameters. This reuse changes the residual-scaling problem: in an untied Transformer, each residual branch receives and applies its own parameter update, whereas in a looped Transformer one shared update aggregates gradients from repeated visits and is read back by those same visits in the next linearized forward pass. We formalize this tied-depth effect through a first-order perturbation bound controlled by a visit-alignment coefficient . The bound recovers the DeepNorm exponent when visits decorrelate, but in the conservative aligned regime it requires the exponent to increase from to as loop count grows at fixed physical depth. The resulting method, DeepLoop, keeps the Post-LN DeepNorm architecture with a normalized branch input and sets and for unrolled depth . On GPT-style looped language models at GPT-2 small and GPT-2 medium scale, DeepLoop is neutral when no physical block is revisited, improves validation loss at every activated loop count, and improves average downstream accuracy in all but one of the evaluated settings. These results show that stable recurrent depth requires residual scaling rules that account for parameter visits, not only nominal layer count.

Primary area:
foundation or frontier models, including LLMs
Keywords:
looped transformers, residual scaling, weight sharing, depth scaling, training stability, initialization, test-time compute, hierarchical reasoning models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ZukMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies residual scaling in looped Transformers where the same physical blocks are visited repeatedly.

The paper shows that the original DeepNorm scaling rule can be insufficient when Transformer blocks are repeatedly reused across loop iterations.

It proposes DeepLoop, which uses a stronger scaling rule and evaluates it on looped language models and a hierarchical recurrent reasoning model.

Critical strengths

The paper identifies why the original DeepNorm scaling can be insufficient when the same Transformer blocks are reused across loop iterations, and formalizes this effect with a loop aware perturbation analysis.

The proposed scaling rule improves performance on looped language models and also gives a clear gain on HRM under a controlled comparison.

Critical weaknesses

A closely related prior work [1] already argues that weight sharing creates correlated residual updates in looped Transformers and derives a 1/N1/N residual scaling rule. DeepLoop’s p=1/2p=1/2 parameterization also yields an update to residual ratio proportional to 1/N1/N, so the incremental novelty over this prior work is unclear.

The main language modeling baseline uses a standard Pre LN block with standard initialization, while DeepLoop uses a Post LN sandwich block together with the proposed residual scaling. Therefore, the reported gains cannot be attributed specifically to the new scaling rule.

[1] On the Residual Scaling of Looped Transformers: Stability and Transferability, Latent & Implicit Thinking Workshop at ICLR 2026.

Questions for authors

Can the authors compare DeepLoop p=1/2p=1/2 with DeepNorm p=1/4p=1/4 under the same Post LN sandwich architecture, while keeping all other training settings fixed?

2 comments
AuthorsOpenReview ↗

Response to Reviewer ZukM: p=1/4 vs p=1/2 in the same block, and relation to [1]

Thank you for the review.

Q (and W2): p=1/2p = 1/2 vs p=1/4p = 1/4 in the same sandwich block; can the gains be attributed to the scaling? You're right that Table 1 compares two full configurations (pre-LN vs sandwich + scaling), so on its own it can't tell you what the exponent contributes. In the submission the only exponent-only evidence was Appendix F, and that sweep started at 0.30. We ran the comparison you asked for (N1/N2 in the common comment). With p=1/4p = 1/4 the looped model trains on 0 of 5 seeds: val loss is 7.68 at step 2,000, which is the unigram floor. With p=1/2p = 1/2 it trains on 5 of 5 (3.72). To check whether this is about weight sharing or just about depth, we also ran both exponents with 36 untied blocks at the same depth and init. Untied, p=1/4p = 1/4 trains on 5 of 5 and is a bit better than p=1/2p = 1/2 (3.56 vs 3.66).

So in this block the DeepNorm exponent works without weight sharing and fails with it, which is what our analysis says should happen. What we still can't do is split the 0.016-0.028 nat gain over pre-LN in Table 1 into a part from the block and a part from the scaling. There is no trained sandwich model with p=1/4p = 1/4 at R≥3R \ge 3 to compare against, and we haven't run the sandwich block with no scaling at all. We will add N1/N2 to the paper, describe Table 1 as a full-configuration comparison, and say this explicitly.

W1: relation to [1]. Thanks for the pointer. We didn't know about [1] and should have cited it. We will add it and discuss it. Our work was done independently: an earlier version of this paper was already under review at another conference in May 2026, before [1] appeared on arXiv (June 2026).

You're right that both papers end up with a branch-to-skip ratio that scales as 1/R1/R at fixed physical depth. We think the two results support each other more than they overlap. [1] works with pre-LN and multiplies each branch by N−1L−1/2N^{-1}L^{-1/2} at run time (NN loops, LL layers). Its argument is an infinite-width one, proved for linear networks under SGD, and the goal is learning-rate transfer across loop counts. We work with the post-norm DeepNorm family, where the branch enters through 1/α1/\alpha after normalization (Lemma A.1) and the two knobs are the skip scale α\alpha and the init gain β\beta. Our bound is finite-width and first-order, and it has the alignment coefficient κR\kappa_R in it, so it gives a range of thresholds p≥(1+γ)/4p \ge (1+\gamma)/4 and falls back to DeepNorm at γ=0\gamma = 0. We also treat the truncated-gradient hierarchical case. The rules are not identical either. In our notation [1] uses a ratio ∝1/(RK)\propto 1/(R\sqrt{K}), while DeepLoop uses 1/(4KR)1/(4KR). If we take our aligned condition MR(β/α)2=O(1)MR(\beta/\alpha)^2 = O(1) with equality, we get exactly 1/(RK)1/(R\sqrt{K}). DeepLoop is more conservative in KK because it keeps DeepNorm's constants.

[1] also measures that residual increments from different passes are positively correlated in looped models and close to orthogonal in untied ones. That fits what we now measure directly on gradients and read-outs (N3): κ^=R\hat\kappa = R at init for the tied model, and orthogonal gradients for the untied one. We will put this comparison into Related Work.

Thank you for the experiment. However, I still think the novelty of the work is limited considering [1] and the bar of ICLR. So, I maintain my rating.

Reviewer_LTbQOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposed a new scaling (DeepLoop) for the residual stream and initialization of certain blocks which extends the main contribution of DeepNet to looped transformers. They showed through both analysis and empirical evidence (appendix F) that the exponent must be of at least a certain size, or else convergence is not likely, and was able to extend the looped transformer to 7x the original transformer size. The main experiment was done on GPT2 with some results on Hierarchical Reasoning Model.

Critical strengths
  1. Very useful for cases where deep transformers are wanted without scaling the parameters linearly up alongside layer count. The modification is easy and trivial in the code, and seems to be the actual "transition" point given appendix data, meaning that the analysis/assumptions seem to be of practical value.
Critical weaknesses
  1. Originality: while I think the result is useful, I'm uncertain whether it's a big enough improvement over DeepNet for ICLR. The results in Table 1/2 are very marginal (to the point where I would like to see extra seeds if possible). In addition, I'm unsure if the theoretical analysis really differs that much from DeepNet. Ostensibly, the current paper uses RMSNorm, but the deepnet paper is essentially equivalent in their proofs when doing the assumption μ(x)=0\mu(x) = 0.

  2. The application to HRR was extremely rushed and not coherent with the paper. It seems tacked on, with not enough of a introductory runway in the main text or the appendix to guide the reader.

Questions for authors
  1. Can you estimate κr\kappa_r? Or have an algorithm to describe how to compute it? it's not obvious to me, and I guess it's mainly because (3.6) and the matrices instead are quite hard to comprehend fully.

  2. Is Assumption 3.1 really stated in the DeepNet paper? I have read that paper in some depth in the past, and just skimmed it but could not find a similar expression/assumption. Similar to point 1, the statement quantities themselves is very hard to parse in text (e.g. "linearized output map from a residual-branch parameter perturbation at that visit and the norm of the corresponding visit-wise effective update") and should probably be accompanied by some sort of equation.

1 comment
AuthorsOpenReview ↗

Response to Reviewer LTbQ: seeds, difference from DeepNet, Assumption 3.1, estimating kappa_R, HRM

Thank you for the careful review and for checking the DeepNet paper.

W1 (first part): gains in Table 1/2 are marginal; more seeds. We don't think the gain is marginal for pretraining. For comparison, giving the baseline two more loop rounds (R=3R = 3 to 5, or 5 to 7) buys 0.008-0.021 nats and costs about 40-60% more compute. DeepLoop gets 0.015-0.028 nats with no extra compute. In fact DeepLoop with RR rounds is about as good as (within 0.001 nats) or better than the baseline with R+2R + 2 rounds. Small: 2.7917 (R=3R = 3) vs 2.7910 (R=5R = 5), and 2.7679 (R=5R = 5) vs 2.7700 (R=7R = 7). Medium: 2.5627 vs 2.5640, and 2.5444 vs 2.5558. That is about 30-40% less compute for the same loss.

On seeds: each cell in Table 1 is one full-length run. We'd like to add seeds but can't afford it right now. One full-length run with R≥3R \ge 3 takes 90 to 620 H200 GPU-hours (Appendix G), the 16 cells took about 3,700 GPU-hours, and two more seeds per cell would be about 7,400 more. Where we could afford seeds, we ran them: the new controlled experiments (N1/N2, N1b) use 5 seeds per cell. There the effect of the exponent is 0/5 vs 5/5, and the s.d. across seeds for runs that train is at most 0.01 nats, below the gaps in Table 1.

W1 (second part): is the analysis really different from DeepNet? Yes. The difference is not RMSNorm vs LayerNorm (Lemma A.1 is only there to derive the 1/α1/\alpha factor for the norm we use). It is what weight sharing does to DeepNet's sum. DeepNet bounds ∥ΔF∥\lVert\Delta F\rVert by 2N2N terms, one per sublayer, each being that sublayer's sensitivity times its own update. With tying, the update of a physical sublayer is a sum over its RR visits, and that same update is then read by all RR visits. The single sum turns into a double sum (Eq. 3.5) with R2R^2 cross terms per physical sublayer. κR\kappa_R says how much of the cross terms survive. DeepNet is the case κR=O(1)\kappa_R = O(1). Aligned visits give κR=Θ(R)\kappa_R = \Theta(R), and that moves the exponent from 1/4 to (1+γ)/4(1+\gamma)/4. DeepNet's analysis has no such term and says p=1/4p = 1/4 is enough at any depth. N1/N2 show that this fails under tying: same depth, same block, same init, and p=1/4p = 1/4 trains on 5/5 seeds untied and 0/5 tied.

Q2: is Assumption 3.1 really in the DeepNet paper? DeepNet does not state it as a named assumption, and we will fix that sentence. But it is exactly the per-sublayer scaling that DeepNet's derivation uses. It combines two quantities from there: the sensitivity factor vi2+wi2/α\sqrt{v_i^2 + w_i^2}/\alpha in their model-update theorem, and the SGD step size in their appendix, which is bounded by η\eta times the same factor. With vi=wi=βv_i = w_i = \beta both are O(β/α)O(\beta/\alpha). We will write it as equations,

∥Ur,j∥≤CUβ/α,∥Gr,j∥≤CGβ/αfor all visits (r,j),\lVert U_{r,j}\rVert \le C_U \beta/\alpha, \qquad \lVert G_{r,j}\rVert \le C_G \beta/\alpha \qquad \text{for all visits } (r, j),

where Ur,jU_{r,j} is ∂F/∂ϕj\partial F/\partial\phi_j through visit rr only and Gr,jG_{r,j} is the update that visit rr alone would produce. We will make clear that this is our abstraction and not a DeepNet assumption.

Q1: can κR\kappa_R be estimated? Yes, and we have now done it (N3). The recipe is: (i) Give each visit its own copy of the block weights, all set to the shared values. One backward pass then gives the per-visit gradients gr,jg_{r,j}, and they sum to the normal tied gradient. (ii) For the read side, perturb only the copy used by visit rr along the update direction δj\delta_j and record how the output changes. That is Ur,jδjU_{r,j}\delta_j. (iii) κ^j=∥∑rUr,jδj∥⋅∥∑rgr,j∥/(R⋅max⁡r∥Ur,jδj∥⋅max⁡r∥gr,j∥)\hat\kappa_j = \lVert\sum_r U_{r,j}\delta_j\rVert \cdot \lVert\sum_r g_{r,j}\rVert / (R \cdot \max_r\lVert U_{r,j}\delta_j\rVert \cdot \max_r\lVert g_{r,j}\rVert). It is 1 for orthogonal visits of equal norm and RR for aligned ones. At init we get κ^=R\hat\kappa = R exactly for R=2R = 2 to 16. After training it is about 1 and barely depends on RR (0.85, 0.93, 0.94 for R=3R = 3, 5, 7 at 100K steps). We will add the procedure and the table to the paper and release the script.

W2: the HRM part feels rushed. The HRM section is there for a reason. It tests the same rule on a different kind of model (two modules, truncated gradients), so it is a transfer test of the analysis and not an add-on. The result is clear: p=1/2p = 1/2 has the best 2-vote accuracy, +3.25 points over vanilla HRM with a seed s.d. of about 0.5, and exponents around 1/4 are no better than the unscaled model (N4). We will add a short introduction to HRM and show the full sweep.

Reviewer_ZdV4OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes adjusting the residual scaling hyperparameters of DeepNorm for looped Transformers, which reuse a small stack of shared blocks across multiple rounds. The authors argue that shared parameters accumulate gradient updates across loop visits and, under the assumption that these updates are fully aligned across rounds, claim that the DeepNorm exponent should be raised from 1/4 to 1/2. The proposed method, DeepLoop, applies this modified exponent to the skip scale α and the initialization gain β within a Post LN architecture with an additional normalization on the branch input. The main experiments compare DeepLoop against a standard Pre LN looped baseline on GPT 2 small and medium models with loop counts from 1 to 7, reporting validation loss and downstream accuracy. The paper additionally reports a short exponent sweep at a single loop count and an application of the same rule to the Hierarchical Reasoning Model on ARC AGI.

Critical strengths

Though the paper provides no evidence that this mechanism explains the reported gains, the observation that shared parameters accumulate updates across loop visits seems reasonable.

Critical weaknesses

The baseline uses Pre-LN, while DeepLoop simultaneously switches to Post-LN, introduces an additional inner normalization layer, and changes the scaling exponent. The central claim of the paper is that the exponent should be p=1/2 rather than p=1/4, yet the full training experiments do not include a DeepNorm baseline with p=1/4 under the same architecture. Consequently, the improvements in validation loss could arise from the Post-LN architecture itself rather than the proposed loop-aware scaling exponent.

The theoretical argument relies on the assumption that gradients and sensitivities across recurrent iterations are highly aligned. However, the authors only analyze the worst-case alignment scenario and acknowledge that measuring actual alignment is left for future work. Computing cosine similarities between gradients from different iterations would be relatively inexpensive using the existing training setup. Without such measurements, it is impossible to determine whether p=1/2 works because of the proposed alignment mechanism or simply because reducing the residual branch magnitude improves optimization stability.

The derivation assumes that summing gradients across R iterations amplifies the parameter update by a factor of R, which holds directly for SGD under perfectly aligned gradients. However, Adam normalizes updates using estimates of the first and second moments of the gradients, so increasing the gradient magnitude does not necessarily produce a proportional increase in the parameter update. The paper does not sufficiently justify whether its claimed R-dependent amplification remains valid under adaptive optimizers such as Adam.

The ARC-AGI experiments compare only the original HRM against a modified version using p=1/2, without including an intermediate DeepNorm baseline with p=1/4. These results therefore demonstrate, at most, that introducing this residual scaling parameterization can improve HRM performance. They do not establish that p=1/2 is more appropriate than p=1/4, which is the central theoretical claim of the paper.

The improvements in validation loss are relatively small, ranging from approximately 0.015 to 0.028, while most improvements in average downstream accuracy are below one percentage point. Individual benchmarks such as OpenBookQA contain only a few hundred evaluation examples, making their accuracy estimates potentially sensitive to sampling variation.

The language modeling experiments are restricted to GPT-2 small and medium scale models with at most seven loop iterations, which limits the generality of the conclusions. The main comparisons use a single random seed, and the paper does not report confidence intervals or statistical significance for the language modeling improvements.

Questions for authors

The paper repeatedly uses strong language suggesting that p=1/2 is the correct scaling choice and that the theoretical prediction has been confirmed experimentally. However, these claims appear stronger than what the experiments support, particularly given the missing DeepNorm baselines discussed above and the modest empirical improvements. Moreover, the theoretical analysis establishes a sufficient condition for bounding a first-order perturbation, rather than a necessary and sufficient condition for training stability. Could the authors clarify why this result is repeatedly characterized as a stability threshold and provide stronger evidence supporting the claim that p=1/2 is the correct choice?

Several prior works have investigated initialization, residual scaling, and normalization strategies for deep or recurrent Transformers, including approaches discussed in the Related Work section. However, the experiments primarily compare DeepLoop against a standard Pre-LN baseline, without evaluating these existing alternatives under comparable settings. Could the authors include stronger baselines, particularly DeepNorm with p=1/4, as well as relevant initialization and normalization strategies from prior work, to establish whether DeepLoop provides improvements beyond existing techniques?

2 comments
AuthorsOpenReview ↗

Response to Reviewer ZdV4 (1/2): same-architecture DeepNorm baseline and measured alignment

Thank you for the detailed review. We ran new experiments for the main concerns (N1-N4 in the common comments). W1-W6 and Q1-Q2 follow the order of the paragraphs in the review. This comment covers W1, W2 and Q2. The next one covers W3-W6 and Q1.

W1 / Q2: the baseline is Pre-LN while DeepLoop changes the norm placement and the exponent together; no DeepNorm p=1/4p = 1/4 baseline in the same architecture. We now have this baseline, and it rules out the alternative explanation. N1 keeps the architecture fixed and changes only the exponent. With the DeepNorm exponent the looped sandwich model trains on 0 of 5 seeds (val loss 7.68 at step 2,000 and 7.67 at step 3,000). With p=1/2p = 1/2 it trains on 5 of 5 (3.72 at step 2,000). So the Post-LN architecture by itself does not produce the gains: with DeepNorm's exponent it does not train at all at R=3R = 3. N2 shows that this failure comes from weight sharing. With 36 untied blocks at the same depth and init, p=1/4p = 1/4 trains on 5 of 5 seeds and is slightly better than p=1/2p = 1/2, as DeepNet prescribes for untied depth.

Sec. 5.1 already says that Table 1 compares the pre-LN default with the complete DeepLoop configuration, and that the exponent alone is tested in the fixed-block sweep of Appendix F. That sweep now starts at p=1/4p = 1/4 (N1b). Longer runs agree with the short ones. With p=1/4p = 1/4 in the same block (constants (3N)1/4(3N)^{1/4}, (12N)−1/4(12N)^{-1/4}), R=1R = 1 trains to 2.8647 at 100K steps, while R=3R = 3 and R=5R = 5 stayed on the floor for 46.6K and 28.3K steps before we stopped them. The diagnostics in the common comment are consistent with this: after the collapse only the last visit gets any gradient.

On other init and normalization schemes: DeepNorm is the right baseline here. It is the rule our analysis modifies, and the DeepNet paper already compares it with DS-Init, Admin, ReZero and Fixup variants, which all diverge at depths where DeepNorm trains. Our pre-LN baseline also uses depth-scaled init on the residual output projections. The 1/N1/N branch multiplier of [1] is independent work for pre-LN, and we will discuss it.

W2: alignment is assumed but never measured. We measured it (N3), and the assumption holds exactly. At init, the gradients that different visits send to a shared block have cosine 1.00, and the coefficient in Eq. (3.6) equals RR for every RR from 2 to 16. In an untied model with the same init, gradients at matching layers are orthogonal. This is the phase that decides whether a run trains: in Appendix F and in N1, a run either gets off the floor within the first 1,000-2,000 steps or it doesn't within the budget. After training the visits have decorrelated (κ^≈1\hat\kappa \approx 1). That is the γ=0\gamma = 0 end of Proposition 3.2, and it fits the fact that smaller exponents reach lower loss once they do train.

You also suggest that a smaller branch might simply help optimization, with or without tying. N2 rules that out. Same branch scale, same block, same depth: p=1/4p = 1/4 trains on 5/5 seeds untied and 0/5 tied. Branch size does not explain the failure. Weight sharing does. The measurements in [1] agree with ours (increments from different passes are positively correlated in looped models and near-orthogonal in untied ones).

AuthorsOpenReview ↗

Response to Reviewer ZdV4 (2/2): Adam, HRM sweep, effect size, seeds, strength of claims

W3: the RR-fold amplification argument assumes SGD; what about Adam? The SGD form is the same simplification DeepNet makes, and DeepNorm is used with Adam in practice. The factor of RR that we rely on does not need SGD, for two reasons.

First, the factor of RR comes from two places and only one of them depends on the optimizer. Whatever the update δϕj\delta\phi_j is, all RR visits read it: ΔF=∑j(∑rUr,j)δϕj\Delta F = \sum_j (\sum_r U_{r,j}) \delta\phi_j. With an Adam-like step the size of δϕj\delta\phi_j is set by the learning rate, so the write-side factor is gone. But then the output change is first order in β/α\beta/\alpha rather than second order. As a rough estimate, ∥ΔF∥≲ηJRa(β/α)\lVert\Delta F\rVert \lesssim \eta J R^a (\beta/\alpha), with a=1a = 1 if the reads are aligned and a=1/2a = 1/2 if they are decorrelated. At fixed KK this is bounded iff p≥a/2p \ge a/2, so we get the same two endpoints, 1/4 and 1/2.

Second, we measured the read side along the sign of the tied gradient, which is the first Adam step (N3). At init, an update read through RR visits moves the output exactly RR times as much as through one visit (checked up to R=16R = 16). After 100K steps the factor is 1.7, 2.4 and 2.8 for R=3R = 3, 5, 7. All experiments in the paper use AdamW, including the new ones where p=1/4p = 1/4 fails only under tying.

W4: HRM has no p=1/4p = 1/4 row. We have the full exponent sweep (N4). p=0.20p = 0.20 and p=0.30p = 0.30, on either side of 1/4, give 36.75% and 37.25%. That is the same as the unscaled model within noise (36.50%, seed s.d. about 0.5). p=1/2p = 1/2 gives 39.75%, a gain of 3.25 points. So on HRM too, p=1/2p = 1/2 clearly beats exponents in the DeepNorm range, which is the comparison you asked for. (There is also a smaller gain near p=0.13p = 0.13. p=1/2p = 1/2 is still the best setting in the sweep.) We will include the full sweep in the paper.

W5: improvements are small; some benchmarks are tiny. We don't think the gain is small for pretraining. Giving the baseline two more loop rounds buys 0.008-0.021 nats at about 40-60% more compute, and DeepLoop gets 0.015-0.028 nats with no extra compute. DeepLoop with RR rounds is about as good as (within 0.001 nats) or better than the baseline with R+2R + 2 rounds at both scales, which is about 30-40% less compute at the same loss (numbers in our reply to Reviewer LTbQ). On downstream, we report the 8-task average because single tasks are noisy (OpenBookQA has 500 questions). The average favors DeepLoop in 5 of the 6 medium-scale settings with R≥3R \ge 3, and at R=7R = 7 DeepLoop wins on 7 of 8 tasks in both shot settings.

W6: small scale, single seed, no confidence intervals. 124M and 350M parameters at 50B tokens per run is in line with related work (for example [1] goes up to 438M parameters and 10B tokens), and the gain holds at both scales. It is largest at 350M, R=7R = 7. Each Table 1 cell is one full-length run. More seeds are a matter of compute: one run with R≥3R \ge 3 costs 90 to 620 H200 GPU-hours, and two more seeds per cell would be about 7,400 GPU-hours. Where we could afford seeds, we ran them. The new controlled experiments use 5 seeds per cell, the effect there is 0/5 vs 5/5, and the s.d. across seeds for runs that train is 0.003-0.010 nats, below the gaps in Table 1.

Q1: the language is too strong; the result is a sufficient condition, not a stability threshold. The paper does state the result as a sufficient condition. Sec. 3.3 calls it "a sufficient tied-depth stability condition" and says explicitly that we do not claim every looped model attains worst-case alignment. "Threshold" refers to the exponent at which the bound becomes uniform in RR. The new experiments back the choice of p=1/2p = 1/2 directly: visits are exactly aligned at init (N3), p=1/4p = 1/4 fails under tying and only under tying (N1/N2), and p=1/2p = 1/2 is the first exponent in the sweep at which every seed trains (N1b). We will make the wording uniform ("sufficient first-order condition") and state the HRM conclusion as "p=1/2p = 1/2 gives the best result in the sweep" instead of "the correct setting".

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.