acceptodds
Under review as a conference paper at ICLR 2027

Carrying a Length and Counting It Are Different Abilities: A Double Dissociation between Recurrence and Attention in Code-Fence Generation

Abstract

A Markdown code fence closes at the first later run of backticks at least as long as its opener. A large language model (LLM) that wraps its answer must therefore open with a run longer than any inside, carry that length across the content and reproduce it. In LatentMD, a measurement of nine deployed Transformer LLMs, 38.0% of valid outputs were content-correct but boundary-broken, and inference alone cannot tell whether a model lacks the computation or does not use it. To isolate that computation, we built Minimal Boundary Transduction (MBT) and trained recurrent, state-space model (SSM), Transformer and hybrid architectures from scratch on its three synthetic tasks. Across 1,216 runs that pass an in-distribution gate, two extrapolation axes, distance (twelvefold gaps) and count (unseen fence lengths), split the typical solution of each family in opposite directions. At learning rates chosen on Distance, nonlinear or selective recurrence reaches closing accuracy 0.75 to 1.00 on Distance but at most 0.04 on Count. The two linear time-invariant recurrences stay below 0.30 on Distance. Deep attention stacks with learned or no positional encoding reach up to 0.99 on Count but fail on Distance once gaps stretch beyond a factor that depends on depth and encoding. RoPE stacks do not extrapolate count at any depth. In four-layer attention stacks each added recurrent layer lowers count extrapolation. Scored with the unchanged LatentMD protocol, eight SSM and hybrid LLMs show no ordering by attention or SSM share in how unclosed-fence rates grow with content length. They differ in how their unclosed fences end, which one failure rate conceals.

Primary area:
foundation or frontier models, including LLMs
Keywords:
length generalization, Transformers, recurrent neural networks, counting, synthetic benchmarks, structured generation, Markdown code fences

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_Bx2FOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces Minimal Boundary Transduction (MBT), three synthetic tasks over unary token runs that encode the CommonMark fence rule: CLOSE (emit the closing run and stop), VERIFY (judge a completed block) and OPEN (choose a safe opening length). Training recurrent, SSM, Transformer and hybrid models from scratch, it reports that across 1,216 gate-passing runs, at a twelvefold gap scale, nonlinear or selective recurrence carries a fence length across long gaps but does not close unseen lengths, while deep learned-position and NoPE attention shows the reverse; linear time-invariant recurrences fail on distance and RoPE never extrapolates count. Validation includes per-family learning-rate scans, depth/width/budget controls, segment splits, interventions and an attention-layer ladder, plus a bridge scoring eight deployed SSM and hybrid LLMs under the LatentMD protocol, which finds no ordering by SSM share.

Critical strengths
  1. Extensive, well-designed controls. Every family gets a five-rate learning-rate scan, and the paper adds depth, width and parameter-matched arms, a budget ladder, a training-ceiling control, an emission-range control, an interpolation control and a fresh-draw check on learning-rate selection, with gate survivorship reported per cell. Several of these rule out obvious objections; the emission-range control, for example, shows that recurrent models can emit runs of 7 to 9 when trained to, so their Count failure is not a failure of emission alone. This is why the recurrent half of the result is convincing.

  2. Honest scoping and full verifiability. Claims are framed as the typical solution each family finds rather than as representational limits, counterexamples (the AAAM seed at 0.59 on Count, the three attention cells that carry both axes) are disclosed, and per-seed values, checkpoints and analysis scripts are released. This makes the paper's claims checkable, including the concern raised below.

Critical weaknesses

The attention side’s “distance failure” is measured with a metric that counts outcomes the paper itself labels correct as failures. Under CommonMark, a closing fence longer than the opener is a valid close, and the paper’s own taxonomy maps a long close to E0 (Correct) in Figure 2 and Table 39. The headline metric, closing accuracy, instead requires exactly ℓ tokens, and §3.1 states that most of attention’s distance errors are long closes. Under the safe-close rate, which accepts any valid close, the twelve-layer learned-position cell the paper chose for Table 1 scores 0.95 on Distance and 0.73 on Count. That is the paper’s own representative attention cell carrying both axes. The paper justifies the exact-length metric by noting that safe-close would reward always emitting nine tokens. That concern is testable, but the paper reports no measure of how far long closes exceed ℓ. As it stands, the evidence does not show whether attention fails to carry the length or carries it approximately and overshoots. Only the first reading supports the title’s claim that carrying a length and counting it are different abilities. For the Markdown problem that motivates the paper, the second reading would be a correct outcome.

Questions for authors

Please report the distribution of overshoot (emitted length minus ℓ) for the deep attention cells on Distance. Please also restate the dissociation under the safe-close rate alongside exact closing accuracy in Table 1 and Figure 1.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Bx2F

Thank you for the close reading, and for stating the concern precisely enough that we could test it. Line numbers refer to the submitted PDF, and G1 above lists what the revised PDF adds.

W1 / Q1. (Exact versus safe close, and the overshoot distribution)

Your description of the metric is right. Under the validity rule, any close of at least ℓ, the twelve-layer learned-position cell of Table 1 carries both axes at twelvefold gaps, 0.95 on Distance and 0.73 on Count, and the taxonomy labels such a long close E0 (Figure 2 and Table 39). Our §3.1 draws the line where you do, "Attention therefore loses the exact transduction of ℓ" (lines 220 to 221). §2.2 of the revised PDF now carries this sentence, in blue. "Carrying a length means reproducing ℓ exactly. A longer close is a valid Markdown outcome for the outermost fence and is scored separately as a safe close."

Here is the overshoot distribution you asked for, newly decoded from the released checkpoints on the same seeded items. The closing accuracies reproduce Table 1 to its printed precision, so no number in the paper changes. On Distance the twelve-layer cell closes 54.6% of its open items exactly and 40.2% long, with 2.4% short, 2.4% without a fence and 0.4% malformed. Of its long closes 76% overshoot by one token, 17% by two, 5% by three and 2% by four or more, a mean of 1.35 tokens. Closes that reach the decoding cap of nine are 1.4% of open items, so always emitting nine is ruled out.

The table gives the deep attention cells you asked about, the three of Table 1, and two recurrent cells for contrast, in percent of open items over five seeds (28,768 items per cell on Distance). "Long by +1" is the percent of long closes that overshoot by one token. In the two recurrent rows it rests on fewer than 250 closes. The three middle columns are on Distance.

cellDistance exact / safelonglong by +1at cap 9Count exact / safe
Transformer 12L54.6 / 94.840.2761.471.9 / 72.6
NoPE 8L35.4 / 66.331.0553.498.7 / 99.1
NoPE 4L10.1 / 15.25.1700.491.6 / 91.7
Mamba 2L94.9 / 95.60.7520.00.0 / 0.1
GRU 2L99.2 / 99.90.8920.00.0 / 0.0

In the other eight cells long closes are at most 1.5% of open items in the recurrent cells and the two hybrids, 0.9% and 14.3% in the two linear time-invariant cells and 9.7% and 11.6% in the two-layer attention cells. All thirteen cells are in Appendix N (Tables 49 to 52). Figure 15 redraws Figure 1 under the safe rule and Figure 16 draws the distribution.

So overshoot is largest at eight and twelve attention layers, and where the deep attention cells overshoot it is mostly by one. Under the safe rule the recurrent half is unchanged. On the attention side the twelve-layer cell carries both axes, NoPE 8L rises to 0.66 on Distance and NoPE 4L stays at 0.15, so attention's distance limit remains at the shallower depths and becomes a loss of exactness at twelve layers. On Count overshoot is rare, with long closes at most 1.9% of open items.

Exact stays the headline for two reasons. The first is measurement. The safe-close rate cannot tell carrying from always emitting long, and the distribution above separates the two, so we report both. The second is the application. A long close is harmless only for the outermost fence. When an inner fence overshoots by one it closes an outer fence opened with margin one, and in LatentMD this margin decides boundary failure, with boundary-OK rates of 10.3%, 32.4% and 28.4% at required outer lengths of 3, 4 and 5, margins 0, 1 and 2 over three-backtick inner fences (Lee & Kang, 2026, Table 4).

Reference. Lee and Kang (2026). LatentMD: Benchmarking Markdown Boundary Failures in LLM-Generated Text. arXiv:2609.06993.

Reviewer_Bx2FOpenReview ↗

Thanks for testing my suspicions

Interesting results, I have no further questions and raise my rating to 4.

We truly appreciate your reading our response! It is a pleasure to hear that your concern has been addressed. We will make sure to keep the analysis of the additional experiments (Appendix N) in the final version.

Reviewer_M6icOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Minimal Boundary Transduction (MBT), a synthetic benchmark for studying two distinct abilities in code-fence generation: retaining a length across long irrelevant sequences and generating lengths beyond the training range. Across recurrent, state-space, Transformer, and hybrid architectures trained from scratch, the authors report a double dissociation: nonlinear or selective recurrence generally performs well on distance extrapolation but poorly on unseen counts, while sufficiently deep attention-only models with certain positional encodings show the opposite pattern. The paper further analyzes failure mechanisms and compares the synthetic findings with deployed language models.

Critical strengths
  • The paper separates two abilities that are often conflated in length-generalization evaluations, providing a useful benchmark and a more precise analysis of model failures.

  • The experimental study is extensive, with multiple random seeds, learning-rate scans, depth and width controls, and targeted interventions to investigate the mechanisms behind the observed differences.

Critical weaknesses
  • MBT isolates distance and count generalization using unary fence tokens and synthetic filler sequences. However, real LLMs process tokenized delimiters in natural-language contexts, which may introduce different challenges. The extent to which the observed double dissociation generalizes to realistic structured-output tasks remains unclear.

  • The deployed-model analysis does not establish a clear relationship between architectural composition and boundary failures, while differences in model scale, training data, and post-training confound the comparison. Controlled comparisons between models differing primarily in architecture are needed to determine whether the synthetic findings transfer to real LLMs.

  • The main experiments use matched training steps and sample counts, with additional parameter-count and budget controls. However, different architectures incur different computational costs per step. Comparisons under matched FLOPs or wall-clock training budgets would strengthen the conclusions about architectural differences.

Minor Issues / Typos:

  • There is excessive vertical whitespace below the Introduction heading

  • In Figure 4, the rightmost x-axis label appears misaligned and is not centered properly.

Questions for authors
  • Can the authors validate the observed double dissociation under more realistic tokenization and structured-output settings?

  • Can the authors provide controlled comparisons between models that differ primarily in architecture, and directly evaluate count extrapolation on deployed LLMs?

  • Can the authors compare architectures under matched FLOPs or wall-clock training budgets, in addition to matched training steps and sample counts?

5 comments
AuthorsOpenReview ↗

Response to Reviewer M6ic (1/2)

Thank you for the careful review. Line numbers refer to the submitted PDF, and G1 above lists what the revised PDF adds.

W1 / Q1. (Realistic tokenization and structured output)

Tokenization. Please see G2 above.

Structured output. The suite (MBT) removes natural-language text and instruction following on purpose, because inference on a deployed model cannot tell whether the model lacks the computation or does not use it (lines 38 to 40). The realistic setting you describe is what §5 measures on eight deployed SSM and hybrid models against the nine Transformers of LatentMD. What neither gives is a controlled test in natural-language context. That is a different study with controls of its own, and §7 of the revised PDF names it as future work, in blue.

W2 / Q2, first half. (Controlled comparison of deployed models)

We agree that the cohort does not isolate architecture, and the paper says so in §5.3 and §7 (lines 404 to 406, lines 474 to 476). The controlled comparison of the submission is the synthetic attention-layer ladder of Appendix H, and §5.1 now says so in blue. The only public same-data family, the 8B triplet of Waleffe et al. (2024), has a 4,096-token context below the 7,168-token output budget of our frozen protocol, and §5.3 now names it for a future protocol, in blue.

So we ran a new experiment on the closest instruction-tuned pair, Granite-4.0-H-Micro and Granite-4.0-Micro (IBM Granite Team, 2025). Both are 3B models trained on samples of the same corpus under the same regimen, one with 36 Mamba-2 and 4 attention layers, the other with 40 attention layers under RoPE. We decoded 4,427 generations per model under the frozen protocol and fixed the prediction before decoding. If the distance result of the suite transferred, unclosed fences (E2) would grow more slowly with content length in the hybrid.

The prediction fails in the opposite direction. The hybrid's E2 rate rises from 1.8% at up to ten content lines to 17.2% at 41 to 80 lines and 39.5% above eighty, while the Transformer's stays between 0.5% and 5.6% (Table 64, also below). The difference in slope is 0.96 ± 0.24 (p < 0.001). Overall boundary failure is alike, 32.5% and 30.0%, but 9.1% of the hybrid's records are unclosed fences against 1.1% for the Transformer, whose failures are almost all length collisions. On the count side both fail alike. Neither chooses a safe opener or follows the length hint. This is the controlled comparison you asked for, and it does not support a transfer of the suite's distance result. The difference is a failure the suite does not test. CLOSE presents the close point and asks for the length, whereas a deployed outer fence has first to be kept open across the content. Three confounds remain (RoPE against no positional encoding, hidden size 2560 against 2048, samples of one corpus instead of identical batches), and one pair at one size cannot settle transfer. The full result is in Appendix P.2 (Tables 63 and 64), and §5.3 states it in blue.

Q2, second half. (Count extrapolation on deployed LLMs)

The count axis has two counterparts in a deployed model. The first, choosing an opener longer than every run inside the content, can be measured, and it comes out as a near-uniform failure. In LatentMD eight of the nine Transformers chose a safe opener in under 1% of cases (lines 364 to 365). In our cohort seven of the eight models never do, Falcon-H1-34B does in 0.0%, 4.4% and 7.2% of its fenced responses at inner runs of 3, 4 and 5, and both models of the pair never do (new Tables 61 and 62, §5.1 in blue). The measurement records the shared habit of the three-backtick fence and does not separate architectures. The second, closing a fence opened at an unseen length, cannot be measured cleanly. A prompt has to place the long opener first, and compliance with that instruction varies by model, so the close would report instruction following and the closer together. We leave it as the open gap named in §7 (lines 474 to 477).

W3 / Q3. (Matched FLOPs and wall-clock budgets)

Your point is fair, and deep attention did receive more FLOPs. The twelve-layer Transformer costs 142 GFLOP per step, nine times the two-layer Mamba, and 4.3 PFLOP over training against 0.5. The budget ladder shows that this does not explain count (Table 7). Mamba trained for 300,000 steps receives 4.7 PFLOP and scores 0.00 on Count. Two-layer Transformers at 300,000 steps receive 7.5 PFLOP and score at most 0.11, far below the 0.72 of twelve layers at 4.3 PFLOP. Mamba at 12 and 16 layers takes six to eight times the wall-clock of the twelve-layer Transformer and also scores 0.00 (Table 18). Depth decides count here, not compute. Wall-clock has the opposite order because our Mamba scan is unfused. Training takes 1.26 hours per run for the two-layer Mamba against 0.26 for the twelve-layer Transformer (new Table 47).

Minor. (Whitespace and Figure 4 labels)

Please see G1 above.

AuthorsOpenReview ↗

Response to Reviewer M6ic (2/2)

W2 / Q2, first half. (The pair by content length)

This is Table 64 of the revised PDF. It gives the E2 (unclosed fence) rate of the pair by content length, in percent of valid generations with an opened fence, with Wilson 95% intervals in parentheses. Content length is counted on the output, in lines from the outer opening fence to its close or to the end of the output (Appendix J).

content linesH-Micro (hybrid) nE2 %Micro (Transformer) nE2 %
0 to 106851.8 (1.0 to 3.0)7490.5 (0.2 to 1.4)
11 to 203263.7 (2.1 to 6.3)3711.3 (0.6 to 3.1)
21 to 404564.6 (3.0 to 6.9)5231.7 (0.9 to 3.2)
41 to 8047017.2 (14.1 to 20.9)4011.0 (0.4 to 2.5)
over 8017739.5 (32.6 to 46.9)185.6 (1.0 to 25.8)

The clearest row is 41 to 80 lines, where each model has over 400 generations and the intervals do not overlap. Only 18 of the Transformer's generations exceed eighty content lines, so its last interval is wide.

The difference in slope in our response above is the interaction term of a logistic fit of E2 on log content length and model, 0.96 ± 0.24 (standard error).

References. These are the works our response above cites. All three are in the reference list of the revised PDF.

  • IBM Granite Team (2025). Granite 4.0 Language Models. https://github.com/ibm-granite/granite-4.0-language-models
  • Lee and Kang (2026). LatentMD: Benchmarking Markdown Boundary Failures in LLM-Generated Text. arXiv:2609.06993.
  • Waleffe et al. (2024). An Empirical Study of Mamba-based Language Models. arXiv:2406.07887.

Dear Reviewer M6ic,

As the discussion period closes soon, we would like to ask whether any of your concerns remain unaddressed. We would be glad to answer any further questions.

Reviewer_eieGOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks into the problem of LLMs breaking markdown code fences. The outer fence would have to be longer than any run of backticks inside it, then carried across the content and reproduced at the end. The paper proposes Minimal Boundary Transduction (MBT, synthetic tasks suite) and trains recurrent, state-space model (SSM), Transformer and hybrid architectures on three synthetic tasks. The paper reports "double dissociation": nonlinear or selective recurrence carry the fence length over long distances but fail on lengths unseen in training, while attention-only stacks of eight or more layers do the reverse.

Critical strengths

The paper has thorough controls and ablations ruling out any obvious alternative explanations. Before concluding that one architecture family can't do something, the authors check that it wasn't just trained badly. For eg. each family gets its own tuned learning rate, ablations over depth, width, and removing Mamba's selection mechanism, results are repeated across seeds, etc. Code is also released so the finding is very likely reporducible.

Critical weaknesses

-The title claims two abilities- carrying a length and counting it. The evidence however supports something narrower. The distance test stretches the filler twelve fold but at four fold its one sided i.e., deep Transformers handle both carrying and counting. -All synthetic conclusions come from small models (Table 14) -Typically real tokenizers split backticks into chunks of 2 to 3 but the toy uses one token per backtick so this set up doesn't match real input.

Questions for authors

At four fold filler, deep Transformers with learned or no positional encoding handle both carrying and counting. So only one direction of the dissociation remains. Why call it a "double dissociation"? Why not a single dissociation, with attention's carrying limit depending on distance?

4 comments
AuthorsOpenReview ↗

Response to Reviewer eieG

Thank you for the careful review, and for noting the controls. We answer each point under its own heading. Line numbers refer to the submitted PDF, and G1 above lists what the revised PDF adds.

W1 / Q1. (Double dissociation at fourfold gaps)

You are right about the fact. At fourfold gaps the deep learned-position cells both carry and count, 0.78 to 0.98 on distance (the carrying axis) and 0.61 to 0.84 on count, and we say so in §3.1 and Appendix E (lines 243 to 245, lines 1751 to 1755).

We keep the word double for three reasons.

  1. A double dissociation is a crossover at one test condition. Ours is twelvefold gaps with unseen counts, and we fixed that factor before any model was trained (lines 239 to 241, lines 1722 to 1723). At that condition the nonlinear or selective recurrent cells and the hybrids score 0.75 to 1.00 on Distance and at most 0.04 on Count, and attention at 8 and 12 layers scores 0.72 to 0.99 on Count and 0.35 to 0.55 on Distance (Table 1). The crossover is what rules out the reading that count is simply the harder task.
  2. The two limits differ in kind. The attention limit is a distance threshold that depth extends. At two layers the closing accuracy of the learned-position Transformer falls below one half between gap scales 4 and 6. At twelve and sixteen layers it does so between 8 and 16 (Table 29, lines 341 to 344). The recurrent limit is a ceiling that no learning rate, width, depth or budget moves (Tables 6, 7, 14 and 18).
  3. The fourfold success is the first rung of the scale ladder in Table 21, not a second ability. The fourfold value is only the nearest point on a curve that keeps falling as the distance grows. At the twelvefold scale only three of the 132 attention-only cells reach 0.5 on both axes, and none reaches 0.6 on both (Table 19, lines 1644 to 1646). So carrying at fourfold and failing to carry at twelvefold are two points on one distance limit.

We therefore answer the two halves of your question differently. On "a single dissociation" we disagree. The direction that is absent at fourfold gaps is present at twelvefold. On "attention's carrying limit depending on distance" we agree, and it is how we describe the attention side ourselves (lines 241 to 243). The revised PDF states this more plainly. §3.1 carries these sentences in blue, and the Figure 1 caption a shorter form. "The dissociation is measured at the twelvefold gap scale fixed before training. On the attention side the limit is graded and depends on distance. On the recurrent side it is a ceiling."

W2. (Model size)

The models are small by design, and size is an axis we scanned, not a fixed setting. Small models trained from scratch isolate the computation from pretraining exposure and instruction following (lines 76 to 77). The roster (§2.3) spans widths 64 to 512 and depths 1 to 16 (lines 1420 and 191), and the cells Table 14 lists run from 0.10M to 7.09M parameters. What size does is consistent.

  • Width lifts the count of no family. Two-layer Transformers at the anchor learning rate stay at or below 0.16 on Count from 0.20M to 7.09M parameters.
  • Depth lifts the count of attention only, 0.28, 0.68, 0.84 and 0.67 at 4, 8, 12 and 16 layers at the anchor learning rate (Table 14).
  • Nothing lifts the count of recurrence. Mamba at 12 and 16 layers and the tenfold budget ladder stay at 0.00 (Tables 18 and 7), and the largest recurrent cell, Mamba at width 512 with 7.04M parameters, scores 0.73 on Distance and 0.00 on Count.

The new Table 48 (Appendix M) classifies every gate-passing cell by parameter range. Among the cells with at least 1M parameters, none of the 33 recurrent-block cells counts (26 Distance only, 7 neither), and the 54 attention-only cells split into 2 Distance only, 17 Count only, 3 both and 32 neither. Within the roster, then, size does not change the pattern. Beyond the roster our evidence is the deployed cohort of §5, which cannot separate architecture from training data, size and instruction tuning, and one same-recipe pair at 3B (§5.3). The revised PDF adds one sentence to §7 naming the size range the synthetic results cover, in blue.

W3. (Tokenization)

Please see G2 above.

Dear Reviewer eieG,

As the discussion period closes soon, we would like to ask whether any of your concerns remain unaddressed. We would be glad to answer any further questions.

Thanks for addressing my questions W1,W2,W3. I'm willing to increase my score to 3.

Thank you for taking the time to read our response! We are happy to hear that your concerns have been addressed. We will make sure to keep the analysis of the additional experiments (Appendices M and O) in the final version.

Comments on the paper

3 comments
AuthorsOpenReview ↗

G1 (General comment 1). Revised PDF and new appendices

We thank the three reviewers for their careful reviews. We ran the experiments you asked for, and the results are more than the replies can hold. So we put them in a revised PDF and keep each reply short, pointing to its tables. This comment lists what the revised PDF adds.

Revised PDF

The main text ends on page 10, every changed or added main-text sentence is in blue, and line numbers in our replies refer to the submitted PDF (the initial version), which stays in the revision history. Each reply names the blue sentences that answer it. The table and figure numbers of the submitted PDF are unchanged, and the new material follows at the end of the appendix.

  • Appendix M. The compute of the thirteen Table 1 cells (Table 47, for Reviewer M6ic W3) and the parameter range of every gate-passing cell (Table 48, for Reviewer eieG W2).
  • Appendix N. The overshoot distribution and the safe-close rate of the Table 1 cells, with Figure 1 redrawn under the safe rule (Tables 49 to 52, Figures 15 and 16, for Reviewer Bx2F W1).
  • Appendix O. The CLOSE task under merged backtick tokens, from the tokenizer sources and the two closer conventions to the thirteen cells, depth, learning rates, training curves, the widened range and VERIFY (Tables 53 to 60, Figure 17, for Reviewers eieG W3 and M6ic W1).
  • Appendix P.1. The safe-opener rate and the outer-fence lengths of the eight deployed models and the pair (Tables 61 and 62, for Reviewer M6ic Q2).
  • Appendix P.2. The same-recipe pair Granite-4.0-H-Micro and Granite-4.0-Micro (Tables 63 and 64, for Reviewer M6ic W2). Its rows are also added to the cohort tables, Tables 41 to 43.
  • Limitations, tokenization (§7, Appendix J). The submission called the tokenization axis unverified and left the merged-token control to future work (lines 463 to 464 and 2974 to 2975). Both sentences are replaced in blue, and §7 now states the result of Appendix O.
  • Limitations, the two open gaps (§7). The submission named two, a pair of models that differ only in architecture and a count measurement on deployed models (lines 474 to 477). Appendix P narrows both, and §7 restates what remains, in blue.
  • Appendices B, C, E and J. Five more places change, in blue. Four point to the new appendices, and the one in Appendix E adds the Table 1 value to a range.
  • Layout. The space below the Introduction heading is gone, since the main text now uses a ragged bottom, and the Figure 4 axis labels are horizontal and centred under each group (for Reviewer M6ic, Minor).
AuthorsOpenReview ↗

G2 (General comment 2). Tokenization

Reviewer eieG (W3) notes that real tokenizers split a run of backticks into chunks of two or three while the suite (MBT) uses one token per backtick, and Reviewer M6ic (W1) asks whether the dissociation holds under realistic tokenization. One control answers both. The submission named this encoding as the untested axis (lines 463 to 464 and 2974 to 2975), and we ran it. Appendix O of the revised PDF holds the sources, design and full tables (Tables 53 to 60, Figure 17).

Two closers

The fifteen public tokenizers of the deployed models close a fence in two ways (Table 44). Eight (Gemma, Falcon, Jamba, Nemotron) write the closing run with the same pieces as the opener. This is the same-token closer. The other seven (GPT-4o, Llama-3.1, Qwen3, Granite-4.0) merge the last backtick of the closing run with the newline. This is the newline-merged closer. The difference is whether a model can close by copying the opener. For example, o200k_base, the tokenizer of GPT-4o, splits a run of backticks into pairs and ends an odd run with a triple, so a seven-backtick opener is (``)(``)(```), three tokens. A same-token closer repeats these three tokens. In contrast, GPT-4o's own closer is (``)(``)(``)(`\n), four tokens, the last one a backtick merged with the newline. To write it a model must hold the number seven and write it again as 2+2+2+1.

What we ran and found at the paper's range

We retrained the Table 1 cells on the CLOSE task under both closers. Only the encoding of the backtick runs changes. At the paper's training range, ℓ = 3 to 6, the recurrent half does not move. Count is at most 0.5% in every recurrent and hybrid cell under both closers at every learning rate and depth, and Distance stays above 50% wherever it was in the paper (Tables 54 to 56). The attention half changes. Under the same-token closer the deep cells lose most of their count, Transformer 12L from 71.9% to 1.3%, and only NoPE keeps part of it, 52.5% at 12 layers, mostly at ℓ = 7 and 8. Under the newline-merged closer every attention cell counts 0% to 7%, while Distance rises in the Table 1 cells.

Why the count falls, and the control

The drop has two readings. Either attention cannot count under a merged encoding, which would make the attention half of the claim a tokenization artifact, or the paper's training range does not train the rule the merged closer follows. The runs carry the signature of the second reading. With one token per backtick an unseen count is the seen pattern plus one copy. With merged pieces the closers of the training lengths ℓ = 3 to 6 are (3), (2,2), (2,3) and (2,2,2), and those of the unseen ℓ = 7, 8 and 9 are (2,2,3), (2,2,2,2) and (2,2,2,3). So training never asks for a fourth piece, which ℓ = 8 and 9 need, or a triple after two pairs, which ℓ = 7 needs. The runs of Table 59 reach in-distribution accuracy 1.00 and their Count is flat from step 20,000 to 30,000, so this is not under-training. The misses are short closes, 70% to 99%, so the models stop at a seen piece count rather than overshoot. And NoPE's partial success peaks at ℓ = 8, the one unseen length that only extends the all-pairs pattern, while ℓ = 9, three pairs then a triple, scores 0% to 14%. That is the mark of a rule never trained, not of a computation missing. We therefore trained the same cells on ℓ = 3 to 10, whose targets are closers of up to five pieces with both endings, and tested ℓ = 11 to 13 at the same twelvefold gaps, where ℓ = 12 and 13 still need a sixth piece. The recurrent cells went through the same range, so if recurrence counted once trained on the rule, the claim would fall.

Result of the range control

It does not fall (Table 60). Under the same-token closer, NoPE 12L closes the unseen lengths at 99.8%, Transformer 12L at 88.6% and NoPE 8L (lr 1e-3) at 94.6%. Under the newline-merged closer, where copying cannot work, they count at 100%, 79.3% and 99.0%. Their Distance is 40%, 42% and 69% under the first closer and 41%, 67% and 77% under the second, below Mamba, GRU and the RNN under the same closer (78% or above). The recurrent cells given the same range count at 0% to 9.4%, at two layers under both closers and in the deeper cells. Nearly every exact close of theirs is at ℓ = 11, whose same-token closer (2,2,2,2,3) has five pieces like the (2,2,2,2,2) of ℓ = 10 (Table 53) and so recombines seen pieces. At ℓ = 12 and 13 every recurrent run sits at 0% to 4%. Figure 17 redraws Figure 1 for the Table 1 cells under both closers at both ranges.

What this does to the claim

The cause is the range, not the encoding. The revised PDF states this in blue in §7 and, in these words, in §3.2. "Under the merged-token encodings of deployed tokenizers, attention extrapolates count when the training range covers the closer's composition rule and only in part otherwise. Recurrence does not count under any encoding or range we trained." VERIFY keeps the paper's pattern under both closers (Table 58).

AuthorsOpenReview ↗

Thank you to the reviewers and the Area Chair

Dear Reviewers and Area Chair,

Before the discussion period closes, we would like to thank you once more for your time and for the thoughtful discussion. Thanks to your questions, open points of the submission have become results. For example, the tokenization axis was a limitation and the overshoot of long closes was unmeasured. Both are now tested in the paper (Appendices O and N).

The final version will carry all the changes listed in our general comment G1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.