- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors argue that the choice of comparison baseline matters when accelerating recurrent-depth transformers, which pass through the same block repeatedly. A shortcut cannot be considered an acceleration merely because its output gets close to the final-depth output, since simply running the model honestly up to the same depth may already get closer. The authors propose the native-trajectory criterion (NTC), which requires that, within the same budget, an acceleration method must get closer than the original loop.
Critical strengths
- Looped transformers also demand heavy computation, so accelerating them is an important and practical problem.
Critical weaknesses
- Citations are not properly attached, which makes it hard to tell what the authors’ contribution is versus what it builds on, and the paper is difficult to follow overall. For example, “Anderson-style mixing” is not obvious to me; I only learned that it is an extrapolation technique based on fixed-point iteration by looking it up myself. Such cases kept recurring, making the paper very hard to follow.
- The paper reports that both of its proposed “potential” acceleration methods, DepthPar and Anderson, ultimately fail and perform worse. I recognize that research on failed approaches is also important and a valid contribution, but I feel it falls short of ICLR’s bar. For a negative-result study to be meaningful, it should either reveal that something that seems like it should obviously work in fact does not, or provide a detailed analysis of the causes behind the failure. I do not think this submission satisfies either condition.
- Also, the proposed NTC criteria seems quite obvious and trivial.
Questions for authors
- Is there any analysis of what trajectories the actual tokens follow during the loop? I expected that to come first, but the paper simply proposes two methods and reports that they failed, which left me puzzled.
7 comments
Response to the Authors' rebuttal
I thank the authors for their effort in preparing the rebuttal and the revised manuscript. After carefully reviewing the rebuttal, the revision, and the other reviewers' comments, I summarize my assessment point by point below.
[W1] Clarity of the paper. The added citations are an improvement over the original submission. However, the overall clarity remains insufficient. The revised manuscript is still difficult to follow, much as before, and in my view it requires a substantial, paper-wide rewrite rather than localized edits. Such a revision seems beyond what can reasonably be accomplished within the rebuttal period.
[W2] Reporting of negative results. As another reviewer also noted, negative results alone are not sufficient to constitute a contribution at the level expected for ICLR. Having analyzed the cause of the failure, I would expect the paper to leverage that analysis toward a remedy or an improved method; this aspect remains underdeveloped. Furthermore, Proposition 1 is difficult to assess. The paper does not adequately explain what represents, why the threshold in the condition takes this particular value, or how the empirical values relate to the stated assumptions. As a result, it is very hard to evaluate the soundness of the assumptions and of the conclusions drawn from them. This concern is closely tied to the clarity issue raised in [W1].
[W3] Triviality of the proposed criterion. I am not sure what the authors mean by stating that the criterion is "simple, but it changes conclusions." In particular, how was the absolute threshold of determined? It appears to have been chosen by the authors without justification, and its meaning is unclear. Rather than resolving my concern that the claim is trivial, the rebuttal raises additional concerns related to [W1].
[Q1] Analysis of native trajectories. Figure 1 is presented, but the main text contains no discussion that references or interprets it. It is therefore unclear what the reader is expected to take away from this figure. In its current form, the figure contributes little to the paper's argument.
Overall, while I appreciate the authors' efforts, my main concerns remain unresolved, and I maintain my original rating.
Follow-up: Figure 1, operational threshold, Proposition 1, and tested repairs
Thank you for specifying the remaining concerns. The latest revision adds explicit explanations in Secs. 2, 5 and 6.1; no reported result changed. Two different numbers are at issue: the KL tolerance was an experimental gate, whereas the constant 2 in Proposition 1 follows from the update itself.
[Q1] Figure 1. You are right that the main text should interpret the figure, and Sec. 2 now does. Across its 128 documents, mean KL to R16 falls from 0.0273 at R4 to 0.00237 at R8 and 0.000220 at R12. The reference a shortcut must beat therefore changes with the budget: approaching R16 is not enough when the native prefix is already closer. The figure averages over documents and initializations; it does not claim monotonicity for every trajectory.
[W3] The 0.01 gate is not NTC. The original submission described the median/p95 limits 0.01/0.05 as operational promotion thresholds, not externally validated quality boundaries (original Sec. 4, p. 3). The latest revision states this again in Sec. 2. The limits were one part of a branch-continuation gate. NTC compares candidate and native error at the same budget, and that ordering does not depend on 0.01. The original screen's pooled medians show why a fixed cutoff cannot stand in for it:
| Budget | Native prefix | Anderson |
|---|---|---|
| B4 | 0.0139 | 0.0156 |
| B8 | 6.31×10⁻⁴ | 8.94×10⁻⁴ |
| B12 | 2.95×10⁻⁵ | 3.51×10⁻⁵ |
A median cutoff accepts the native prefix and rejects Anderson only if it lies between the two values in a row. The three intervals are disjoint, so no single cutoff separates all three pairs (Table 8). Paired comparison: at B8, native R8 has lower per-example mean KL on 99.8% of the 1,000 paired examples (Sec. 4), while safeguarded Aitken improves on the native prefix (Fig. 3).
[W2] Proposition 1. For M4/K2 with a shared segment map, identity coarse propagation and all initial boundaries set to , substitution gives , where is the native R16 state (Eq. 5).
- What is. Eq. 6 bounds the residual relative to the target: . It is neither a learned parameter nor a chosen tolerance.
- Why 2. By Cauchy–Schwarz, , so forces negative alignment. The threshold 2 is the magnitude of the coefficient in , not a tuned value.
- How the data relate. The shared map and identity coarse propagation hold by construction for the evaluated Parcae path. The remaining condition, , is checked per document and initialization: all 32 fresh documents satisfy it under both initializations at both scales, with mean of 0.346 and 0.200 (Table 2). uses the native target and off-trajectory segment outputs, not the observed final corrected endpoint, so it is a target-informed diagnostic, not an online predictor (App. C).
[W2] From diagnosis to remedy. One route the initial review named for a negative-result study was "a detailed analysis of the causes behind the failure." Sec. 5 and Fig. 2 give that analysis: a derivation, a check on fresh documents, and interventions. On the further request for a remedy, the diagnosis points to the identity coarse map, and the paper tests that repair. One native coarse step turns the M4/K2 endpoint cosine from −0.992 to 0.992, but once that step is charged its KL is 0.107 at B10, against 0.000765 for native R10 (Tables 3–4). Separately, safeguarded Aitken improves endpoint KL at B4, B8 and B12, by 26.3% at B8 (Fig. 3). The repair works geometrically and it does not beat native computation.
[W1] Clarity. The edits above cover the passages you named. We would welcome pointers to any others.
We would appreciate reassessment in light of these explanations. If a soundness concern remains, could you identify the specific mathematical step, empirical check or assumption, separately from the judgment of significance?
Response to Reviewer vNcG: attribution, a tested explanation of the failure, native trajectories
Thank you for the questions about clarity and explanation. The revision cites and describes each method where it first appears, puts the native trajectories first, and makes the failure mechanism central, with a derived condition, fresh-document tests and interventions.
W1 (citations; what is ours and what we build on). Parareal, Anderson mixing and Aitken extrapolation are established numerical methods; we do not propose them. The Introduction (p. 1) now cites and briefly describes each: Parareal combines coarse propagation with independent fine solves over segments (Lions et al., 2001), Anderson mixing combines recent iterates using their fixed-point residuals (Walker & Ni, 2011), and Aitken extrapolation estimates a sequence limit from consecutive iterates (Brezinski & Redivo Zaglia, 1991). Sec. 3 defines the evaluated variants, and Sec. 9 adds related work on parallel neural solvers and early exit. Our contribution is the controlled evaluation of these methods against the model's own native computation, and the tested conditional mechanism.
W2 (why the negative result is informative). The revision analyzes why the method fails. Sec. 5 (p. 4) derives the second-round endpoint exactly as u4(2)=−2z+δ, where z is the native R16 target and δ collects initial-state, target-shift and off-trajectory terms. Proposition 1 gives a sufficient sign-and-norm condition, and Corollary 1 bounds the angle. The same 32 fresh documents satisfy the condition at 140M and 370M under both initializations, with mean η of 0.346 and 0.200 against a threshold of 2 (Table 2).
On a separate 64-document set, target-informed interventions on the failed endpoint give (Fig. 2, p. 5):
The sign flip removes most of the measured endpoint-KL error. Together with the exact decomposition and fresh-document checks, this links the unresolved boundary's reversed direction to the observed output failure.
W3 (NTC seems obvious). The comparison is simple, but it changes conclusions.
The criterion therefore separates methods; it does not reject all shortcuts.
Q1 (what trajectories do tokens follow?). This analysis now comes first. Sec. 2 and Fig. 1 (p. 2) show KL to the R16 output at every native depth and token position over 128 documents. It is an output-space view; hidden-state geometry is reported for the Parareal boundaries (Sec. 5; Tables 2, 4 and 11).
We would appreciate your reassessing the explanatory contribution, soundness and presentation, and the overall rating. Please tell us if any technical concern remains.
Reference update (12 November 2026): page references now match the latest revision.