- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper compares two hand-written renderings of the same synthetic box-swap state-tracking problems. One is an unpunctuated, contemporary-authored “controlled Literary Chinese” template with ASCII box and item names (WY). The other is a punctuated controlled English template (EN). Each rendering gets its own small byte-level BPE tokenizer (492 entries, merges allowed across word boundaries). WY comes out at about half the tokens: 7,934 vs 15,920 per pass over the 720-item inventory.
The main experiment trains paired from-scratch Transformer classifiers at 1M, 10M and 100M parameters (20, 40 and 20 seed pairs, with identical initialization within a pair). Each pair trains on the 720 items, and training FLOPs are charged until accuracy on those same 720 items stays at or above 75% for a 7,200-presentation window. At the pre-registered 10M scale all 40 pairs favor WY, with a mean saving of 50.38%. The number of presentations needed is nearly identical (31,244 vs 31,644), so the saving comes from cheaper examples, not fewer of them. 100M is the exception, with ~22% fewer WY presentations. The final classifiers are near chance on 320 challenge items.
A supplement trains 20 pairs of ~11M-parameter decoders on zero/one-swap problems with a configuration-disjoint test split. At equal passes, WY uses 62% fewer training tokens and gets higher complete-output test accuracy (97.86 vs 95.03). At a matched 16M-token budget it scores 98.09 vs 95.15, and it uses ~62% fewer tokens per request. A post-hoc audit traces 39-43% of the quality gap to 16 test items with a layout never seen in training. The authors frame everything as a comparison of rendering/tokenizer/schedule packages, not a language effect.
Critical strengths
- The experimental reporting is exceedingly careful
- The primary scale, endpoint, persistence window and analysis were fixed in advance.
- Seeds are paired with identical initial weights, and runs go to a fixed budget with no score-based stopping or seed replacement.
- Every learning curve is plotted, and all 45 scale/threshold/span cells are reported, including the ones where EN never reaches the criterion.
- The authors kept the earlier, unfavorable generator study in the paper and volunteered the layout audit in Tables J5/J6, which undercuts their own headline. This matters for my assessment because I don’t doubt the numbers. My problems are with what the numbers can tell us.
- Splitting the token saving into presentation count times tokens per presentation (Sec. 4.3, App. G) is the right test for any “compressed encoding learns cheaper” claim, and the paper answers it directly. At 1M and 10M the models need the same number of examples; each one just costs half as much. That stops the paper from claiming sample efficiency it doesn’t have, a check that’s easy to skip in this kind of comparison.
Critical weaknesses
- The primary experiment can’t separate Literary Chinese from “a shorter template,” and it isn’t really measuring learning in the usual sense. Several things stack up:
- It measures memorization. Training and acquisition evaluation use the same 720 items, so the endpoint is the cost of memorizing one inventory. The challenge results confirm this: both packages land below the 61.25% majority-label baseline on the 320 named challenges, with balanced accuracy around 48-50%. = The tokens are near-whole template clauses. The tokenizers are fit to an in-domain corpus with merges across whitespace. The 4-box, 3-swap question in Appendix E is 5 WY tokens and 11 EN tokens before BOS/EOS, and the training inventory uses only 74 WY and 53 EN distinct token IDs. At that granularity the classifier sees a handful of chunk IDs, and most of what makes the two renderings “languages” is gone.
- The compression ratio depends on the tokenizer. It is 55% with the frozen BPE, 26% with cl100k and 45% with o200k (Table E1).
- The headline is close to predictable. Presentation counts are unchanged at 10M and sequences are 4-38 tokens long, so FLOPs are basically proportional to tokens. Using the 10M coefficients in Table A1, the quadratic term is under 2% of per-item cost even at T=38. Once you know the number of examples needed to memorize doesn’t depend on rendering, the 50.38% follows from encoding length alone.
- There’s no compact English or symbolic control. The EN template keeps capitals, punctuation and repeated sentences like “Box C is empty.”, so we can’t tell whether anything about WY matters beyond being terser. The intro’s motivation (an economical, interpretable representation) is never actually tested.
- The learning-rate schedule adds a confound. It is indexed by cumulative FLOPs, so the two conditions sit at different learning rates at the same update. By my arithmetic from Table 2, WY certifies around 0.7S (still in warmup) and EN around 1.5S (past the peak). Even the presentation comparison carries a schedule confound. This is the main reason for my score. Fixing it needs new controls and a held-out evaluation, not a rewrite.
- The generator supplement is where the paper tries to show a quality benefit, and I don’t think it supports one beyond this specific template pair.
- It was redesigned after a negative result. An earlier 20-pair study came out unfavorable to WY (94.10 vs 95.02). The new study changed the task depth, tokenizers, output format, budget and schedule, and evaluates on the same reused test inventory. Fresh seeds don’t fix that.
- Most of the gap sits in one unseen layout. 39-43% of the quality gap comes from 16 test items with all four items in box D, a layout that appears 0 times in training. Every EN model fails all 16 at both checkpoints. Per the discussion around Table J5, the EN input switches to a list form training never showed, while WY just repeats its conjunction. That’s a design asymmetry between the two templates.
- The rest of the gap is small and mixed. On the other 1,008 items the gap is 1.6-1.8 points. In the max-occupancy-3 stratum (208 items), 11 of 20 pairs actually favor EN at both checkpoints (Table J6).
- The token saving reflects authoring choices. The 62% request-token saving mostly comes down to how verbose the authored EN output format is.
A smaller point on presentation, not decisive for me: the paper is hard to read. Nearly every sentence carries one or two disclaimers, and after a while it’s hard to tell which claims the authors want the reader to take away, and this therefore reads as exceedingly AI. The abstract alone has around ten figures and several “not X” clauses. The fact that the paper needs Figure L1 as a “reading map” says something. I’d rather see each claim stated plainly once, with the caveats collected in the limitations section.
Questions for authors
These may need new experiments:
- Do you have any runs with a compact English rendering (no capitals or punctuation, repeated clauses collapsed) or a non-linguistic symbolic code at roughly WY’s token length? If presentation counts to criterion match WY’s, I’d read the classifier result as purely about length. If WY still comes out ahead at matched length, that would change my view a lot.
- Did you try a shared tokenizer, either one BPE fit on both renderings or an off-the-shelf one? Table E1 suggests the length gap drops to 26% under cl100k. Does the acquisition saving drop with it?
- For the generators, what exactly does the EN renderer produce for four items in one box, and does that form appear anywhere in training, dev or tokenizer fitting? If the EN template extended its repeated form the way WY does, would you expect the gap on those 16 items to disappear? A rerun with that change and a freshly regenerated split would be much more convincing than the current development-reused test set, and it should be cheap at 11M parameters.
Clarifications:
- Why index the classifier schedule by FLOPs rather than by updates? An update-indexed schedule would make the presentation comparison clean.
- At 100M, WY needs ~22% fewer presentations in 19 of 20 pairs, while 1M and 10M show essentially no difference. Do you have an explanation, and could it come from schedule phase rather than the rendering?
- Given the challenge results, what do you see as the value of measuring the cost to memorize a fixed inventory? Why not measure acquisition on a held-out split, as you do for the generators?
Thank you for your updates. The paper is slightly more promising as a result, and I have updated the score to reflect that. Best of luck.