acceptodds
Under review as a conference paper at ICLR 2027

When Exact Loops Beat Shortcuts: Deep-Target Approximation Is Not Recurrent-Depth Acceleration

Abstract

Recurrent-depth Transformers reuse a shared block for many sequential updates, making depth a candidate for parallel-in-time acceleration. A shortcut can look successful against the deepest endpoint while remaining worse than the model's native state after the same sequential depth. We formalize this comparison as the Native-Trajectory Criterion (NTC). On a prespecified 1,000-example Parcae-140M screen, identity-coarse Parareal yields pooled median forward KL across one to three corrections, while matched exact recurrence is better at every evaluated path budget. Anderson-style mixing at budget 8 reaches KL to R16, yet exact R8 reaches . Paired evaluation confirms the difference. Boundary diagnostics recover the expected Parareal exactness front but reveal a near-antipodal unresolved final boundary after the second correction. The same ordering and geometry persist in a 370M cross-scale robustness check. Thus, for post-hoc recovery of fixed-depth behavior, deep-target proximity is insufficient evidence of recurrent-depth acceleration.

Primary area:
foundation or frontier models, including LLMs
Keywords:
recurrent transformers, test-time compute, depth parallelism, Parareal, Anderson acceleration, negative results, numerical methods

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vNcGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors argue that the choice of comparison baseline matters when accelerating recurrent-depth transformers, which pass through the same block repeatedly. A shortcut cannot be considered an acceleration merely because its output gets close to the final-depth output, since simply running the model honestly up to the same depth may already get closer. The authors propose the native-trajectory criterion (NTC), which requires that, within the same budget, an acceleration method must get closer than the original loop.

Critical strengths
  • Looped transformers also demand heavy computation, so accelerating them is an important and practical problem.
Critical weaknesses
  • Citations are not properly attached, which makes it hard to tell what the authors’ contribution is versus what it builds on, and the paper is difficult to follow overall. For example, “Anderson-style mixing” is not obvious to me; I only learned that it is an extrapolation technique based on fixed-point iteration by looking it up myself. Such cases kept recurring, making the paper very hard to follow.
  • The paper reports that both of its proposed “potential” acceleration methods, DepthPar and Anderson, ultimately fail and perform worse. I recognize that research on failed approaches is also important and a valid contribution, but I feel it falls short of ICLR’s bar. For a negative-result study to be meaningful, it should either reveal that something that seems like it should obviously work in fact does not, or provide a detailed analysis of the causes behind the failure. I do not think this submission satisfies either condition.
  • Also, the proposed NTC criteria seems quite obvious and trivial.
Questions for authors
  • Is there any analysis of what trajectories the actual tokens follow during the loop? I expected that to come first, but the paper simply proposes two methods and reports that they failed, which left me puzzled.
7 comments
AuthorsOpenReview ↗

Response to Reviewer vNcG: attribution, a tested explanation of the failure, native trajectories

Thank you for the questions about clarity and explanation. The revision cites and describes each method where it first appears, puts the native trajectories first, and makes the failure mechanism central, with a derived condition, fresh-document tests and interventions.

W1 (citations; what is ours and what we build on). Parareal, Anderson mixing and Aitken extrapolation are established numerical methods; we do not propose them. The Introduction (p. 1) now cites and briefly describes each: Parareal combines coarse propagation with independent fine solves over segments (Lions et al., 2001), Anderson mixing combines recent iterates using their fixed-point residuals (Walker & Ni, 2011), and Aitken extrapolation estimates a sequence limit from consecutive iterates (Brezinski & Redivo Zaglia, 1991). Sec. 3 defines the evaluated variants, and Sec. 9 adds related work on parallel neural solvers and early exit. Our contribution is the controlled evaluation of these methods against the model's own native computation, and the tested conditional mechanism.

W2 (why the negative result is informative). The revision analyzes why the method fails. Sec. 5 (p. 4) derives the second-round endpoint exactly as u4(2)=−2z+δu_4^{(2)} = -2z + \delta, where z is the native R16 target and δ\delta collects initial-state, target-shift and off-trajectory terms. Proposition 1 gives a sufficient sign-and-norm condition, and Corollary 1 bounds the angle. The same 32 fresh documents satisfy the condition at 140M and 370M under both initializations, with mean η\eta of 0.346 and 0.200 against a threshold of 2 (Table 2).

On a separate 64-document set, target-informed interventions on the failed endpoint give (Fig. 2, p. 5):

InterventionMean endpoint KL
Unchanged30.0
Flip sign0.586
Correct norm only22.4
Flip sign and correct norm0.147

The sign flip removes most of the measured endpoint-KL error. Together with the exact decomposition and fresh-document checks, this links the unresolved boundary's reversed direction to the observed output failure.

W3 (NTC seems obvious). The comparison is simple, but it changes conclusions.

  • In the original screen, Anderson B8 has pooled median KL 8.94×10⁻⁴, below the absolute limit of 10⁻², yet native R8 reaches 6.31×10⁻⁴ and has lower per-example mean endpoint KL on 99.8% of the 1,000 paired examples (Secs. 2 and 4).
  • On a separate confirmation cohort, Aitken B8 lowers mean KL from 0.00237 to 0.00174 (26.3%), paired 95% interval [−7.32, −5.23]×10⁻⁴ (Fig. 3, p. 7; Table 18).
  • On LoopFormer, correction beats a truncated prefix (KL 0.0913 vs 3.22 at B6) but loses to the trained native schedule (0.0758, at lower latency; Table 6, p. 7).

The criterion therefore separates methods; it does not reject all shortcuts.

Q1 (what trajectories do tokens follow?). This analysis now comes first. Sec. 2 and Fig. 1 (p. 2) show KL to the R16 output at every native depth and token position over 128 documents. It is an output-space view; hidden-state geometry is reported for the Parareal boundaries (Sec. 5; Tables 2, 4 and 11).

We would appreciate your reassessing the explanatory contribution, soundness and presentation, and the overall rating. Please tell us if any technical concern remains.

Reference update (12 November 2026): page references now match the latest revision.

Reviewer_vNcGOpenReview ↗

Response to the Authors' rebuttal

I thank the authors for their effort in preparing the rebuttal and the revised manuscript. After carefully reviewing the rebuttal, the revision, and the other reviewers' comments, I summarize my assessment point by point below.

[W1] Clarity of the paper. The added citations are an improvement over the original submission. However, the overall clarity remains insufficient. The revised manuscript is still difficult to follow, much as before, and in my view it requires a substantial, paper-wide rewrite rather than localized edits. Such a revision seems beyond what can reasonably be accomplished within the rebuttal period.

[W2] Reporting of negative results. As another reviewer also noted, negative results alone are not sufficient to constitute a contribution at the level expected for ICLR. Having analyzed the cause of the failure, I would expect the paper to leverage that analysis toward a remedy or an improved method; this aspect remains underdeveloped. Furthermore, Proposition 1 is difficult to assess. The paper does not adequately explain what η\eta represents, why the threshold in the condition η<2\eta < 2 takes this particular value, or how the empirical values relate to the stated assumptions. As a result, it is very hard to evaluate the soundness of the assumptions and of the conclusions drawn from them. This concern is closely tied to the clarity issue raised in [W1].

[W3] Triviality of the proposed criterion. I am not sure what the authors mean by stating that the criterion is "simple, but it changes conclusions." In particular, how was the absolute threshold of 10−210^{-2} determined? It appears to have been chosen by the authors without justification, and its meaning is unclear. Rather than resolving my concern that the claim is trivial, the rebuttal raises additional concerns related to [W1].

[Q1] Analysis of native trajectories. Figure 1 is presented, but the main text contains no discussion that references or interprets it. It is therefore unclear what the reader is expected to take away from this figure. In its current form, the figure contributes little to the paper's argument.

Overall, while I appreciate the authors' efforts, my main concerns remain unresolved, and I maintain my original rating.

AuthorsOpenReview ↗

Follow-up: Figure 1, operational threshold, Proposition 1, and tested repairs

Thank you for specifying the remaining concerns. The latest revision adds explicit explanations in Secs. 2, 5 and 6.1; no reported result changed. Two different numbers are at issue: the KL tolerance was an experimental gate, whereas the constant 2 in Proposition 1 follows from the update itself.

[Q1] Figure 1. You are right that the main text should interpret the figure, and Sec. 2 now does. Across its 128 documents, mean KL to R16 falls from 0.0273 at R4 to 0.00237 at R8 and 0.000220 at R12. The reference a shortcut must beat therefore changes with the budget: approaching R16 is not enough when the native prefix is already closer. The figure averages over documents and initializations; it does not claim monotonicity for every trajectory.

[W3] The 0.01 gate is not NTC. The original submission described the median/p95 limits 0.01/0.05 as operational promotion thresholds, not externally validated quality boundaries (original Sec. 4, p. 3). The latest revision states this again in Sec. 2. The limits were one part of a branch-continuation gate. NTC compares candidate and native error at the same budget, and that ordering does not depend on 0.01. The original screen's pooled medians show why a fixed cutoff cannot stand in for it:

BudgetNative prefixAnderson
B40.01390.0156
B86.31×10⁻⁴8.94×10⁻⁴
B122.95×10⁻⁵3.51×10⁻⁵

A median cutoff accepts the native prefix and rejects Anderson only if it lies between the two values in a row. The three intervals are disjoint, so no single cutoff separates all three pairs (Table 8). Paired comparison: at B8, native R8 has lower per-example mean KL on 99.8% of the 1,000 paired examples (Sec. 4), while safeguarded Aitken improves on the native prefix (Fig. 3).

[W2] Proposition 1. For M4/K2 with a shared segment map, identity coarse propagation and all initial boundaries set to h0h_0, substitution gives u4(2)=−2z+δu_4^{(2)} = -2z + \delta, where zz is the native R16 state (Eq. 5).

  • What η\eta is. Eq. 6 bounds the residual relative to the target: ∥δ∥≤η∥z∥\lVert\delta\rVert \le \eta\lVert z\rVert. It is neither a learned parameter nor a chosen tolerance.
  • Why 2. By Cauchy–Schwarz, ⟨u4(2),z⟩≤(−2+η)∥z∥2\langle u_4^{(2)}, z\rangle \le (-2+\eta)\lVert z\rVert^2, so η<2\eta < 2 forces negative alignment. The threshold 2 is the magnitude of the coefficient in −2z-2z, not a tuned value.
  • How the data relate. The shared map and identity coarse propagation hold by construction for the evaluated Parcae path. The remaining condition, η<2\eta < 2, is checked per document and initialization: all 32 fresh documents satisfy it under both initializations at both scales, with mean η\eta of 0.346 and 0.200 (Table 2). η\eta uses the native target and off-trajectory segment outputs, not the observed final corrected endpoint, so it is a target-informed diagnostic, not an online predictor (App. C).

[W2] From diagnosis to remedy. One route the initial review named for a negative-result study was "a detailed analysis of the causes behind the failure." Sec. 5 and Fig. 2 give that analysis: a derivation, a check on fresh documents, and interventions. On the further request for a remedy, the diagnosis points to the identity coarse map, and the paper tests that repair. One native coarse step turns the M4/K2 endpoint cosine from −0.992 to 0.992, but once that step is charged its KL is 0.107 at B10, against 0.000765 for native R10 (Tables 3–4). Separately, safeguarded Aitken improves endpoint KL at B4, B8 and B12, by 26.3% at B8 (Fig. 3). The repair works geometrically and it does not beat native computation.

[W1] Clarity. The edits above cover the passages you named. We would welcome pointers to any others.

We would appreciate reassessment in light of these explanations. If a soundness concern remains, could you identify the specific mathematical step, empirical check or assumption, separately from the judgment of significance?

Reviewer_NbUyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses primarily on post-hoc trajectory recovery in recurrent-depth Transformers and proposes the Native-Trajectory Criterion (NTC): for a given fine-transition dependency depth BB (i.e., path budget), a candidate method must be closer to the specified RR-step output than the native BB-step state. This depth represents an idealized sequential-dependency budget rather than a measure of runtime (wall-clock time). In a prespecified screen of 1,000 examples using Parcae-140M with R=16R=16 and M=4M=4, identity-coarse Parareal achieves pooled median forward KL values of 1.07, 29.1, and 0.292 after one to three correction rounds, respectively, all exceeding those of native recurrence at matched budgets. Anderson at B=8B=8 achieves a KL of approximately 8.9×10−48.9\times10^{-4}, which also exceeds the approximately 6.3×10−46.3\times10^{-4} achieved by native R8R8. The authors further examine segment boundaries, observing a nearly antipodal final state after the second correction round, and re-examine the qualitative Parareal pattern on a 370M model from the same Parcae family using 256 post-hoc diagnostic examples.

Critical strengths
  1. The paired controls and statistical reporting are careful. Section 4 fixes the initial state for each example and distinguishes pooled token-level metrics from per-example paired analyses; Section 5.1 reports bootstrap analyses with a fixed random seed. The authors also disclose the overlap involving the smoke-test set, as well as the Anderson implementation issue and the validation of its correction in Appendix C.

  2. The conclusions are appropriately scoped. The limitations in Section 9 explicitly restrict the findings to the architecture family, data, and coarse propagator studied. Even under favorable accounting assumptions—including ideal concurrency and the omission of communication and some solver overhead—the tested configurations fail to satisfy NTC, making the negative results convincing within this restricted setting.

Critical weaknesses
  1. Limited coverage of coarse propagators and configurations restricts the generalizability of the negative results. The main screen uses the identity coarse propagator GG, while the 370M experiment is a post-hoc check within the same model family. Although Appendix A reports screening a probe-based candidate, it does not provide a systematic comparison of multiple effective coarse propagators. Since Parareal’s performance with few correction rounds depends on the compatibility between the coarse and fine propagators, the current evidence does not distinguish an identity approximation that is unsuitable for this model from more general difficulties in trajectory recovery.

  2. The practical applicability of NTC has not been sufficiently demonstrated. In Table 2, native R8R8 is already quite close to R16R16, leaving limited absolute room for further improvement. However, the paper lacks analyses of inputs or settings in which deeper recurrence matters more, and it does not present a positive control that satisfies NTC. It therefore remains unclear how much additional guidance the criterion offers for broader computational allocation problems, or whether it can lead to better downstream task performance.

  3. The mechanistic explanation of the second-round failure remains largely descriptive. The boundary-direction and norm diagnostics in Section 6 are valuable, but the authors also acknowledge that the first-round affine formula cannot predict the second-round reversal. The paper currently lacks interventions on coarse-propagator quality, segment length, or state sensitivity. Consequently, it remains unclear why this phenomenon occurs in the tested configuration and whether it persists under other configurations.

Questions for authors
  1. After accounting for additional sequential costs, are results available for inexpensive non-identity coarse propagators or different segment lengths?
  2. How sensitive is the relative performance of Anderson to its hyperparameters? Was Anderson also evaluated in the 370M check?
  3. Does changing the coarse propagator or segment length eliminate the second-round reversal or shift it to a different correction round? If an analytical explanation is available, could you further clarify the conditions under which this reversal occurs? Minor Issues
  4. When introducing “DepthPar” in Table 1, please clarify its correspondence to the identity-coarse Parareal configuration used in this paper.
  5. Both native recurrence steps and segment boundaries are indexed using subscripts on hh. Consider introducing separate notation for the reference boundary states and explicitly stating that the mm-th boundary corresponds to native step 4m4m.
4 comments
AuthorsOpenReview ↗

Response to Reviewer NbUy: charged coarse-map comparisons, positive control, reversal mechanism, Anderson sensitivity

Thank you for the specific questions. The revision adds charged coarse-map and segment-length comparisons, a positive endpoint control, and a quantitative account of the reversal that is tested on fresh documents and through interventions.

W1/Q1 (coarse propagators, segment lengths, charged cost). Every non-native configuration in the Parcae coarse-map/segmentation grid at B < R loses to its charged native prefix. Costs follow Eq. 3 (p. 3): B=KL+(M−K)gB = KL + (M-K)g and W=KR+(K+1)MgW = KR + (K+1)Mg, where g counts sequential fine calls inside the coarse map. One native step at K2 (R16, M4) costs B10 and W44 and reaches KL 0.107, vs 0.000765 for native R10 (Table 3, p. 6). The grid covers one-native-step, damped, scaled-residual and fitted ridge-16/64 maps at M4, plus M2 and M8 variants (Table 9); fitted-map arithmetic is additional to B and W.

W2 (practical applicability; positive control; where depth matters). The criterion separates methods. On Parcae it rejects identity-coarse Parareal and Anderson and accepts safeguarded Aitken: on 128 confirmation documents, Aitken B8 lowers mean endpoint KL from 0.00237 to 0.00174 (26.3%), paired difference −6.23×10⁻⁴, 95% interval [−7.32, −5.23]×10⁻⁴, and B4 and B12 also improve (Fig. 3, p. 7; Table 18). On predictive value:

  • Average headroom is unresolved. R8-minus-R16 NLL is 0.000726 nats/token, interval [−0.00109, 0.00250] (Sec. 8, p. 9).
  • Headroom is heterogeneous. Four oracle groups of 45/41/16/26 confirmation documents have mean headroom −0.00937, 0.00101, 0.00587 and 0.0146 (App. F). This is a target-informed diagnostic, not an online routing rule.
  • Task accuracy. On 1,024 LAMBADA items, Aitken B8 and R8 both reach 25.4%; the paired difference interval is [−0.439, 0.439] percentage points (Tables 16–17).

The endpoint gain is therefore established, and a task gain is not.

W3/Q3 (mechanism; does the reversal vanish or shift?).

  • Condition. With z the R16 target, Sec. 5 (p. 4) derives u4(2)=−2z+δu_4^{(2)} = -2z + \delta with ∥δ∥≤η∥z∥\lVert\delta\rVert \le \eta\lVert z\rVert. If η<2\eta < 2, Proposition 1 gives negative whole-state alignment and a norm ratio within η\eta of 2; Corollary 1 (App. C) gives cos⁡(u4(2),z)≤−1−η2/4\cos(u_4^{(2)}, z) \le -\sqrt{1-\eta^2/4}. The same 32 fresh documents satisfy the condition at both scales under both initializations, with mean η\eta of 0.346 and 0.200 (Table 2).
  • Interventions. On a separate 64-document set, flipping the endpoint's sign cuts mean KL from 30.0 to 0.586; correcting only the norm leaves 22.4; doing both gives 0.147 (Fig. 2, p. 5).
  • Coarse map. At M4/K2 the endpoint cosine is −0.992 for identity, 0.992 for one native step and 0.790 for ridge-64. Damped propagation is also positive; scaled residual stays negative at −0.165 (Table 4, p. 6).
  • Segment count and round. Identity remains reversed at the tested M8/K2 and M8/K4 endpoints (−0.962 and −0.857), while M4/K1 and M4/K3 are positive (Table 19). In the small-initial-state constant-map limit the M4 coefficients are 4, −2, 2 (Eq. 7, App. C).

Proposition 1 is a sufficient condition; it does not say why the off-path terms are small.

Q2 (Anderson sensitivity; 370M). Tuning narrows the 140M gap to R8 but does not close it. The 12-setting development grid varies history {2, 5}, damping {0.5, 1} and ridge {10⁻⁶, 10⁻⁴, 10⁻²} (Table 21). Selection reduces the 140M confirmation gap from 0.815 to 0.583 (10³ΔKL), interval [0.522, 0.649]. At 370M, with no retuning, the selected gap is 0.44 [0.34, 0.566]. Original and selected settings both lose, in the final-position comparisons too (Table 5, p. 6).

Minor issues. Sec. 3 now identifies "DepthPar" as identity-coarse Parareal, writes boundaries as um(k)u_m^{(k)} and native states as hr⋆h_r^\star, and states that boundary m is native step mL.

We would appreciate an updated assessment and rating in light of these answers. Please tell us if any applicability or soundness concern remains.

Reference update (12 November 2026): page references now match the latest revision.

AuthorsOpenReview ↗

Follow-up on the revised comparisons

Thank you again for your review. Our response and the revised manuscript address the comparisons you requested, and the later updates leave the reported results unchanged.

As the discussion period approaches its close, could you let us know whether any of your principal concerns remain unresolved? We would be glad to address a specific remaining question.

I thank the authors for their response and additional experiments. The newly added comparison of coarse propagators, the analysis of segmentation schemes, and the sensitivity analysis of the Anderson hyperparameters address some of my initial concerns; the authors also clarify the sufficient conditions and use intervention experiments to explain the direction reversal in the second correction round. The safeguarded Aitken method is a useful positive reference, showing that under the proposed comparison criterion some methods can indeed get closer to the deep model’s output. However, the terminal KL improvement has not been shown to translate into downstream task gains, and the native prefix remains faster in the comparisons that were measured, so the practical benefit remains unclear. After considering the authors’ response, I maintain my current score.

AuthorsOpenReview ↗

New measurement on the open questions from your review

Thank you for the careful follow-up. You kept your score because no shortcut shows a downstream or speed benefit. We agree, and a new measurement confirms it. It also answers the other question your review left open: whether there is a setting in which deeper recurrence matters.

We scored next-token loss on 512 held-out C4 documents with Parcae-140M (95% document-bootstrap intervals, nats/token).

ContrastMean [95% interval]
Native R4 − native R80.0220 [0.0197, 0.0243]
Native R8 − native R16−0.0001 [−0.0012, 0.0009]
Identity-coarse Parareal at budget 4 − native R41.265 [1.245, 1.285]
Anderson at budget 4 − native R4 (final position only)+0.0017 [−0.0029, +0.0063]
Aitken at budget 4 − native R4−0.0004 [−0.0009, +0.0001]

Deeper recurrence has measurable predictive value below budget 8. The R4–R8 gap is about 0.6% of the loss and is resolved at both scales (0.0159 [0.0139, 0.0180] at 370M, native steps only). Nothing is resolved from R8 to R16. At budget 4, Parareal is far worse than the native prefix. Aitken's loss difference is not resolved and is about 2% of the R4–R8 gap; its top-1 accuracy difference is small but resolved (+0.06 points [0.01, 0.10]).

The paper does not offer an accelerator. Its contribution is a criterion and a negative result. Your review called the negative results convincing within this restricted setting. The new data adds that native steps capture a measurable predictive gain at budget 4 that no tested shortcut matches in loss. The abstract and Sec. 8 now state that Aitken's endpoint gain comes with no resolved gain in predictive loss, and the full numbers are in Appendix H (Table 29, p. 23).

This is next-token prediction. Only the R4-against-R8 comparison was fixed before any outcome was seen; the other contrasts are a post-hoc follow-up on the same documents, at 140M.

We ask whether that evaluation contribution meets the bar without a practical accelerator.

Reviewer_MMLWOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper challenges the validity of identity-coarse Parareal and Anderson-style mixing as acceleration methods for recurrent-depth Transformers. The authors introduce the Native-Trajectory Criterion (NTC), which requires that a shortcut method not only approximate the deep target h_R^* but actually recover the full R-step computation while preserving fine-transition dependency depth. Using a prespecified 1000-example screen on Parcae-140M, they show that identity-coarse Parareal yields pooled median forward KL 1.07 -> 29.1 -> 0.292 across one to three corrections with exact recurrence better at every evaluated correction, while Anderson-style mixing at budget 8 reaches only 8.94e-4 KL. Paired median Δ = 6.97e-4 over 1000 examples. Critically, boundary diagnostics reveal that the second fine-boundary fails: at k=2, the first unresolved h_3^2 has median cosine -0.0747 (already degraded but not yet antipodal), while h_4^2 reaches median cosine -0.991867 and norm ratio 2.05 -- the second-round extrapolation goes in the wrong direction and overshoots. The same pattern reproduces at Parcae-370M. The authors conclude that deep-target proximity alone is insufficient evidence of recurrent-depth acceleration, and propose an evaluation checklist.

Critical strengths
  1. Important methodological critique. The paper makes a valuable point: deep-target approximation (post-hoc output close to the full-depth target h_R^*) is not the same as recurrent-depth acceleration. The NTC criterion -- which requires preserving fine-transition dependency depth and solving the full R-step computation -- is a useful formalization that the community should adopt. The proposed evaluation checklist (native exact trajectory, fine-transition depth, boundary cosine/norm, separate state memory) is practical.
  2. Prespecified and statistically rigorous experiment design. The 1000-example screen was prespecified before looking at negative results, with fixed gates (median KL <= 0.01, p95 KL <= 0.05, exact early exit at same depth). Paired bootstrap CIs and per-example distribution analysis (99.8% of examples favor exact R8) strengthen the credibility of the negative finding.
  3. Mechanistic explanation of the failure. The boundary diagnostics (Fig. 2) go beyond "the KL is worse" and show exactly why: at k=2, the first unresolved boundary h_3^2 is already degraded (cosine -0.0747) but not yet antipodal, while h_4^2 becomes near-antipodal (cosine -0.991867), causing the second correction to overshoot. This mechanistic understanding i more useful than a mere empirical negative result.
Critical weaknesses
  1. The paper is primarily a negative result with limited positive contribution. While the critique is valuable, the paper does not propose a better acceleration method -- it only shows that existing identity-coarse Parareal and Anderson mixing fail the NTC criterion. The contribution is diagnostic, not constructive. For an ICLR paper, this may be seen as narrow.
  2. Only one architecture family is tested. All experiments use Parcae (140M and 370M), which is a specific looped Transformer with identity coarse propagation. The paper does not test learned coarse maps, different looped architectures (Loopformer, recurrent-depth GANs, etc.), or autoregressive generation. The generality of the boundary failure is unclear.
  3. No wall-clock or FLOP comparison is provided. The paper argues that KL proximity is not acceleration, but it never actually measures end-to-end runtime. Without an efficiency comparison against exact recurrence, the "acceleration" framing is one-sided -- the paper proves these methods are not "true" acceleration, but does not show they are not practically useful.
  4. The identity-coarse setting is a strawman. The paper acknowledges that "a more capable coarse map could close the gap" but does not test any learned or non-identity coarse propagator. Since most practical acceleration methods use learned coarse maps, the negative result on identity-coarse Parareal has limited direct relevance to the broader acceleration literature.
Questions for authors
  1. The prespecified screen uses Parcae-140M with R=16, M=4, and identity coarse propagation. How general are the negative findings? Have you tested non-identity coarse maps (e.g., learned coarse propagators, linear interpolation) that might avoid the boundary antipodal failure?
  2. The boundary failure at k=2 shows that h_4^2 is near-antipodal to the reference (median cosine -0.991867), meaning the second correction extrapolates in the wrong direction. Is this a property of the Parcae architecture specifically, or does it also occur in other looped Transformer models (e.g., Loopformer, DEBERT-style dynamic early exit)?
  3. The paper reports no wall-clock acceleration measurement. The NTC criterion requires fine-transition dependency depth, but in practice, a method that approximates the deep target well even without exact recurrence might still be practically useful if it reduces end-to-end compute. Have you compared wall-clock time of DepthPar vs. exact recurrence under matched settings?
  4. The 370M cross-scale check uses 256 post-hoc examples from the same training distribution. Would the boundary failure pattern hold on out-of-distribution text, or on autoregressive generation (as opposed to the fixed forward-pass setting)?
2 comments
AuthorsOpenReview ↗

Response to Reviewer MMLW: positive control, second architecture, runtime, non-identity coarse maps

Thank you for the review. The revision adds a positive endpoint control, a tested mechanism, a second architecture, measured parallel timing and non-identity coarse maps. Together they show why the choice of native baseline matters: a correction can beat a truncated prefix and still lose to the model's trained native schedule.

W1 (limited positive contribution). The paper now includes a shortcut that improves on the native prefix. On 128 Parcae-140M confirmation documents, development-selected safeguarded Aitken lowers mean endpoint KL from 0.00237 to 0.00174 at B8 (26.3%); the paired difference is −6.23×10⁻⁴, 95% document-bootstrap interval [−7.32, −5.23]×10⁻⁴. The same method also improves at B4 and B12 (Fig. 3, p. 7; Table 18). Its final-position NLL intervals include zero, so predictive quality and runtime are evaluated separately.

W2/Q2 (one architecture; is the reversal Parcae-specific?). We added LoopFormer, a time-conditioned recurrent model. At B6, trained-coarse correction beats the truncated prefix (KL 0.0913 vs 3.22), but the trained uniform schedule reaches 0.0758 and is faster (79.6 vs 209.3 ms; Table 6, p. 7). On the reversal itself: it is confirmed within Parcae at 140M and 370M (Table 2, p. 5), and LoopFormer is not a matched test of it. LoopFormer's K2 endpoints are positively aligned (cosine 0.276 with identity, 0.993 with the trained coarse map; Table 4, p. 6), but with M = 3 that endpoint is the third boundary, which is not antipodal in Parcae either (median cosine −0.0746571; Table 11).

W3/Q3 (wall-clock, FLOPs). Parallel execution speeds up the solver but does not make it competitive with native recurrence. Four segments run on two A10G GPUs; serial and parallel states and logits match in all 384 recorded comparisons, and the parallel solver is 1.69–1.80× faster than the serial one. At K2:

MethodMean latency (ms)Mean KL to R16
Serial Parareal K293.529.5
Two-GPU Parareal K253.429.5
Native R828.50.00271
Native R1650.00

R8 is the comparator at the same ideal dependency depth, not a latency-matched control (Table 7, p. 9). K2 uses 32 fine calls against 8 for R8; these are call counts, not full-program FLOP counts (Table 9).

W4/Q1 (identity coarse map; non-identity alternatives). Better coarse maps can repair the geometry without beating native computation. The Parcae grid covers one-native-step, damped, scaled-residual and fitted ridge-16/64 maps, plus M = 2 and M = 8 variants (Tables 3 and 9). Every non-native configuration in this grid at B < R loses to its charged native prefix: one native step at K2 costs B10 and reaches KL 0.107, vs 0.000765 for native R10 (Table 3, p. 6). Native-step, damped and ridge maps turn the M4/K2 endpoint cosine positive; scaled residual stays negative at −0.165 (Tables 4, 19). Ridge-map arithmetic is additional to the fine-call budget.

Q4 (out-of-distribution text, generation). Both extensions use the 140M checkpoint. On 256 WikiText articles, identity K2 has mean KL 26.75 vs 0.004797 for R8, and its 64-document boundary subset keeps cosine −0.99 (Tables 19, 22). Aitken B8 still improves on R8 there: 10³ΔKL −0.885, interval [−1, −0.775] (Table 22). In greedy generation over 64 prompts, identity K3 diverges from its paired R16 continuation in 0.984 of cases, vs 0.586 for R8 (Table 23). These measure continuation fidelity; the K2 boundary reversal was not measured during free generation.

We would appreciate your reconsidering the contribution, experimental coverage and rating in light of these results. Please tell us which concern, if any, remains unresolved.

Reference update (12 November 2026): page references now match the latest PDF; the substantive answer is unchanged.

AuthorsOpenReview ↗

Follow-up on the revised comparisons

Thank you again for your review. Our response and the revised manuscript address the comparisons you requested, and the later updates leave the reported results unchanged.

As the discussion period approaches its close, could you let us know whether any of your principal concerns remain unresolved? We would be glad to address a specific remaining question.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Revision summary

We thank the reviewers. The revision keeps the original 1,000-example screen and the original boundary diagnostics, and adds targeted post-submission validation. The main results are given here.

ConcernResultWhere
Limited positive contribution (MMLW W1, NbUy W2)Positive control. Safeguarded Aitken at B8 lowers mean endpoint KL from 0.00237 to 0.00174 (26.3%) on 128 Parcae-140M confirmation documents; paired 95% interval [−7.32, −5.23]×10⁻⁴. B4 and B12 also improve.Fig. 3, p. 7; Table 18
Mechanism only descriptive (NbUy W3, vNcG W2)An exact decomposition and a sufficient reversal condition hold on the same 32 fresh documents at 140M and 370M. On a separate 64-document set, flipping the endpoint's sign cuts mean KL from 30.0 to 0.586; correcting only its norm leaves 22.4.Sec. 5; Table 2; Fig. 2, p. 5
Identity-only coarse map (MMLW W4, NbUy W1)Every non-native configuration in the Parcae coarse-map/segmentation grid at B < R loses to its charged native prefix. One native step at K2 repairs the negative alignment but reaches KL 0.107 at B10, vs 0.000765 for native R10.Tables 3–4, p. 6; Table 9
One architecture family (MMLW W2)LoopFormer at B6: trained-coarse correction beats the truncated prefix (KL 0.0913 vs 3.22) but loses to the trained native schedule (0.0758), which is also faster (79.6 vs 209.3 ms).Table 6, p. 7
No wall-clock evidence (MMLW W3)Two-GPU Parareal is 1.69–1.80× faster than serial Parareal. At K2 it takes 53.4 ms with KL 29.5; native R8 takes 28.5 ms with KL 0.00271, and exact R16 takes 50.0 ms.Table 7, p. 9

Predictive relevance. On 1,024 English LAMBADA items, Aitken B8 and R8 both reach 25.4% accuracy; the paired difference interval is [−0.439, 0.439] percentage points (Tables 16–17). The endpoint gain is established; a task gain is not.

Contribution. A tested comparison protocol, a conditional explanation of one failure, and evidence that endpoint fidelity, predictive loss and measured efficiency can rank methods differently. The individual replies cover Anderson sensitivity, segmentation, corpus shift, generation and clarity.

Additions. Blue captions on Figures 1–4 and Tables 2–7, together with the highlighted Proposition 1, identify selected post-submission additions. Table and figure numbers differ from the submission, and the subtitle is now "Evaluating Recurrent-Depth Acceleration".

Reference update (12 November 2026): page references now match the latest revision.

AuthorsOpenReview ↗

Clarification revision: unchanged results and updated page references

We have uploaded a further clarification revision. It adds an explicit interpretation of Figure 1, distinguishes the original operational KL gates from the native-relative criterion, explains the premises and constant in Proposition 1, and connects the existing coarse-map controls to the failure diagnosis. Table 8 restores the original Anderson B4/B12 rows.

No new experiments were run, and previously reported numerical results and conclusions are unchanged. Equation, figure, table and section numbers are retained. Pagination Changes:

ItemPrevious PDFCurrent PDF
Table 2: fresh mechanism checksp. 4p. 5
Table 3: charged coarse-map comparisonsp. 5p. 6
Figure 3: Aitken endpoint/predictive comparisonp. 6p. 7
Table 7: actual two-GPU timingsp. 8p. 9
Section 8: predictive utility and native headroompp. 8-9p. 9

Proposition 1 remains on p. 4, Figure 2 on p. 5, Tables 4-5 on p. 6, Table 6 on p. 7, and Figure 4 on p. 8. Our earlier answers remain applicable; the new follow-up under vNcG addresses the clarification questions directly.

AuthorsOpenReview ↗

Updated PDF: Table 28 added and earlier results are unchanged

We thank the reviewers for the discussion. We have uploaded an updated PDF, and every number and conclusion reported earlier is unchanged.

The one addition is Table 28 (Appendix G, p. 22), summarized in Sec. 8. It responds to the request for an analysis of the native trajectory. Across the 16 native depths it reports the size of each hidden-state update, the ratio of successive updates and the cosine between them, together with the three segment responses that enter Eq. 5. These are summaries of measurements recorded in our completed 6 November run. We did not run the model again, and they are not an independent confirmation.

We also replaced internal shorthand with plain wording, named the dataset and cohort in Sec. 3 and the Figure 1 caption, and stated in Sec. 8 which case Proposition 1 covers.

Taken together, the revisions answer the main requests in the reviews:

  • A positive result. Safeguarded Aitken improves on the native prefix, with 26.3% lower mean endpoint KL at B8 (Fig. 3).
  • An explanation of the failure. An exact decomposition and Proposition 1, checked on fresh documents at two scales and by interventions (Sec. 5; Table 2; Fig. 2).
  • Other coarse maps. Every coarse-map and segmentation variant tested at B < R loses to its charged native prefix (Tables 3 and 9).
  • A second architecture and wall-clock evidence. LoopFormer (Table 6) and measured two-GPU execution (Table 7).

Pagination changes: Table 7 is now on p. 8, where our comments of 12 November say p. 9. All other page numbers in our comments still hold.

We hope these additions address the main concerns, and we would be grateful if they could be taken into account in the final assessments. We are glad to answer any remaining question before the discussion closes.

AuthorsOpenReview ↗

New measurement: predictive value of depth and of the shortcuts at budget 4

Two reviews ask whether deeper recurrence has predictive value on this model, and whether the endpoint results carry over to prediction. We measured both on held-out text.

Setup: 512 C4 English-validation documents, 128 next-token labels each, two paired initializations averaged within document, Parcae-140M. Intervals are 95% document bootstraps, in nats/token.

MethodBudgetMean NLLTop-1 accuracy
Native R443.679833.46%
Safeguarded Aitken43.679433.51%
Identity-coarse Parareal K144.945228.12%
Native R553.666533.68%
Native R663.660933.79%
Native R773.658833.79%
Native R883.657833.84%
  1. Deeper recurrence has measurable predictive value below budget 8. Native R4 is worse than R8 by 0.0220 [0.0197, 0.0243], about 0.6% of the loss, and by 0.38 points of top-1 accuracy [0.26, 0.50]. The successive steps R4→R5, R5→R6, R6→R7 and R7→R8 gain 0.0133 [0.0120, 0.0146], 0.0056 [0.0047, 0.0064], 0.0022 [0.0016, 0.0027] and 0.0010 [0.0006, 0.0014]. R8 and R16 are indistinguishable (−0.0001 [−0.0012, 0.0009]). At 370M the native R4–R8 gap is 0.0159 [0.0139, 0.0180].
  2. Of the two methods the criterion rejects, identity-coarse Parareal at budget 4 is worse than native R4 by 1.265 [1.245, 1.285]. Anderson at budget 4, scored at the final position only, is unresolved (+0.0017 [−0.0029, +0.0063]).
  3. The method the criterion accepts, Aitken, is not resolved against native R4 at budget 4. Its NLL differs by −0.0004 [−0.0009, +0.0001], at most about 4% of the R4–R8 gap, and its top-1 accuracy by +0.06 points [0.01, 0.10].

Budget 4 is one of the paper's three prespecified budgets. A small but resolved predictive gain was available there. Native steps obtain it, and none of the tested shortcuts shows a resolved gain in loss. At budgets 8 and 12 no predictive headroom is resolved, as Sec. 8 already states.

This is next-token prediction, not a downstream task. The shortcuts were scored at 140M only. Only the R4-against-R8 comparison was fixed before any outcome was seen; the Parareal, Aitken, Anderson and R5 to R7 contrasts were chosen afterwards on the same documents, so they are a post-hoc follow-up.

These numbers are in the revised PDF as Appendix H (Table 29, p. 23). The abstract and Sec. 8 now state that Aitken's endpoint gain comes with no resolved gain in predictive loss. Pagination before p. 23 is unchanged, so the page references in our earlier comments still hold. We are glad to answer any question.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.