- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper introduces Minimal Boundary Transduction (MBT), three synthetic tasks over unary token runs that encode the CommonMark fence rule: CLOSE (emit the closing run and stop), VERIFY (judge a completed block) and OPEN (choose a safe opening length). Training recurrent, SSM, Transformer and hybrid models from scratch, it reports that across 1,216 gate-passing runs, at a twelvefold gap scale, nonlinear or selective recurrence carries a fence length across long gaps but does not close unseen lengths, while deep learned-position and NoPE attention shows the reverse; linear time-invariant recurrences fail on distance and RoPE never extrapolates count. Validation includes per-family learning-rate scans, depth/width/budget controls, segment splits, interventions and an attention-layer ladder, plus a bridge scoring eight deployed SSM and hybrid LLMs under the LatentMD protocol, which finds no ordering by SSM share.
Critical strengths
-
Extensive, well-designed controls. Every family gets a five-rate learning-rate scan, and the paper adds depth, width and parameter-matched arms, a budget ladder, a training-ceiling control, an emission-range control, an interpolation control and a fresh-draw check on learning-rate selection, with gate survivorship reported per cell. Several of these rule out obvious objections; the emission-range control, for example, shows that recurrent models can emit runs of 7 to 9 when trained to, so their Count failure is not a failure of emission alone. This is why the recurrent half of the result is convincing.
-
Honest scoping and full verifiability. Claims are framed as the typical solution each family finds rather than as representational limits, counterexamples (the AAAM seed at 0.59 on Count, the three attention cells that carry both axes) are disclosed, and per-seed values, checkpoints and analysis scripts are released. This makes the paper's claims checkable, including the concern raised below.
Critical weaknesses
The attention side’s “distance failure” is measured with a metric that counts outcomes the paper itself labels correct as failures. Under CommonMark, a closing fence longer than the opener is a valid close, and the paper’s own taxonomy maps a long close to E0 (Correct) in Figure 2 and Table 39. The headline metric, closing accuracy, instead requires exactly ℓ tokens, and §3.1 states that most of attention’s distance errors are long closes. Under the safe-close rate, which accepts any valid close, the twelve-layer learned-position cell the paper chose for Table 1 scores 0.95 on Distance and 0.73 on Count. That is the paper’s own representative attention cell carrying both axes. The paper justifies the exact-length metric by noting that safe-close would reward always emitting nine tokens. That concern is testable, but the paper reports no measure of how far long closes exceed ℓ. As it stands, the evidence does not show whether attention fails to carry the length or carries it approximately and overshoots. Only the first reading supports the title’s claim that carrying a length and counting it are different abilities. For the Markdown problem that motivates the paper, the second reading would be a correct outcome.
Questions for authors
Please report the distribution of overshoot (emitted length minus ℓ) for the deep attention cells on Distance. Please also restate the dissociation under the safe-close rate alongside exact closing accuracy in Table 1 and Figure 1.
3 comments
Thanks for testing my suspicions
Interesting results, I have no further questions and raise my rating to 4.
We truly appreciate your reading our response! It is a pleasure to hear that your concern has been addressed. We will make sure to keep the analysis of the additional experiments (Appendix N) in the final version.
Response to Reviewer Bx2F
Thank you for the close reading, and for stating the concern precisely enough that we could test it. Line numbers refer to the submitted PDF, and G1 above lists what the revised PDF adds.
W1 / Q1. (Exact versus safe close, and the overshoot distribution)
Your description of the metric is right. Under the validity rule, any close of at least ℓ, the twelve-layer learned-position cell of Table 1 carries both axes at twelvefold gaps, 0.95 on Distance and 0.73 on Count, and the taxonomy labels such a long close E0 (Figure 2 and Table 39). Our §3.1 draws the line where you do, "Attention therefore loses the exact transduction of ℓ" (lines 220 to 221). §2.2 of the revised PDF now carries this sentence, in blue. "Carrying a length means reproducing ℓ exactly. A longer close is a valid Markdown outcome for the outermost fence and is scored separately as a safe close."
Here is the overshoot distribution you asked for, newly decoded from the released checkpoints on the same seeded items. The closing accuracies reproduce Table 1 to its printed precision, so no number in the paper changes. On Distance the twelve-layer cell closes 54.6% of its open items exactly and 40.2% long, with 2.4% short, 2.4% without a fence and 0.4% malformed. Of its long closes 76% overshoot by one token, 17% by two, 5% by three and 2% by four or more, a mean of 1.35 tokens. Closes that reach the decoding cap of nine are 1.4% of open items, so always emitting nine is ruled out.
The table gives the deep attention cells you asked about, the three of Table 1, and two recurrent cells for contrast, in percent of open items over five seeds (28,768 items per cell on Distance). "Long by +1" is the percent of long closes that overshoot by one token. In the two recurrent rows it rests on fewer than 250 closes. The three middle columns are on Distance.
In the other eight cells long closes are at most 1.5% of open items in the recurrent cells and the two hybrids, 0.9% and 14.3% in the two linear time-invariant cells and 9.7% and 11.6% in the two-layer attention cells. All thirteen cells are in Appendix N (Tables 49 to 52). Figure 15 redraws Figure 1 under the safe rule and Figure 16 draws the distribution.
So overshoot is largest at eight and twelve attention layers, and where the deep attention cells overshoot it is mostly by one. Under the safe rule the recurrent half is unchanged. On the attention side the twelve-layer cell carries both axes, NoPE 8L rises to 0.66 on Distance and NoPE 4L stays at 0.15, so attention's distance limit remains at the shallower depths and becomes a loss of exactness at twelve layers. On Count overshoot is rare, with long closes at most 1.9% of open items.
Exact stays the headline for two reasons. The first is measurement. The safe-close rate cannot tell carrying from always emitting long, and the distribution above separates the two, so we report both. The second is the application. A long close is harmless only for the outermost fence. When an inner fence overshoots by one it closes an outer fence opened with margin one, and in LatentMD this margin decides boundary failure, with boundary-OK rates of 10.3%, 32.4% and 28.4% at required outer lengths of 3, 4 and 5, margins 0, 1 and 2 over three-backtick inner fences (Lee & Kang, 2026, Table 4).
Reference. Lee and Kang (2026). LatentMD: Benchmarking Markdown Boundary Failures in LLM-Generated Text. arXiv:2609.06993.