When Exact Loops Beat Shortcuts: Deep-Target Approximation Is Not Recurrent-Depth Acceleration
Abstract
Recurrent-depth Transformers reuse a shared block for many sequential updates, making depth a candidate for parallel-in-time acceleration. A shortcut can look successful against the deepest endpoint while remaining worse than the model's native state after the same sequential depth. We formalize this comparison as the Native-Trajectory Criterion (NTC). On a prespecified 1,000-example Parcae-140M screen, identity-coarse Parareal yields pooled median forward KL across one to three corrections, while matched exact recurrence is better at every evaluated path budget. Anderson-style mixing at budget 8 reaches KL to R16, yet exact R8 reaches . Paired evaluation confirms the difference. Boundary diagnostics recover the expected Parareal exactness front but reveal a near-antipodal unresolved final boundary after the second correction. The same ordering and geometry persist in a 370M cross-scale robustness check. Thus, for post-hoc recovery of fixed-depth behavior, deep-target proximity is insufficient evidence of recurrent-depth acceleration.
est. 32% chance this paper gets accepted at ICLR 2027.
What do you think this paper will get?
All positions stay anonymous.