acceptodds
Under review as a conference paper at ICLR 2027

Carrying a Length and Counting It Are Different Abilities: A Double Dissociation between Recurrence and Attention in Code-Fence Generation

Abstract

A Markdown code fence closes at the first later run of backticks at least as long as its opener. A large language model (LLM) that wraps its answer must therefore open with a run longer than any inside, carry that length across the content and reproduce it. In LatentMD, a measurement of nine deployed Transformer LLMs, 38.0% of valid outputs were content-correct but boundary-broken, and inference alone cannot tell whether a model lacks the computation or does not use it. To isolate that computation, we built Minimal Boundary Transduction (MBT) and trained recurrent, state-space model (SSM), Transformer and hybrid architectures from scratch on its three synthetic tasks. Across 1,216 runs that pass an in-distribution gate, two extrapolation axes, distance (twelvefold gaps) and count (unseen fence lengths), split the typical solution of each family in opposite directions. At learning rates chosen on Distance, nonlinear or selective recurrence reaches closing accuracy 0.75 to 1.00 on Distance but at most 0.04 on Count. The two linear time-invariant recurrences stay below 0.30 on Distance. Deep attention stacks with learned or no positional encoding reach up to 0.99 on Count but fail on Distance once gaps stretch beyond a factor that depends on depth and encoding. RoPE stacks do not extrapolate count at any depth. In four-layer attention stacks each added recurrent layer lowers count extrapolation. Scored with the unchanged LatentMD protocol, eight SSM and hybrid LLMs show no ordering by attention or SSM share in how unclosed-fence rates grow with content length. They differ in how their unclosed fences end, which one failure rate conceals.

open until 14 Dec 2026

est. 32% chance this paper gets accepted at ICLR 2027.

Reject 68%Accept 32%

What do you think this paper will get?

All positions stay anonymous.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.