acceptodds
Under review as a conference paper at ICLR 2027

Learning Cost and Token Use with Controlled Literary Chinese: A Paired State-Tracking Study

Abstract

We measure learning cost and generation token use separately: shorter examples need not reduce the cost of learning. We compare controlled Literary Chinese (WY) and controlled English (EN) in 80 pre-specified pairs of from-scratch Transformer classifiers across three scales on a fixed 720-item training inventory. Certification requires at least 75% inventory accuracy throughout evaluations spanning 7,200 presentations, charging training through certification. All 80 pairs attain it with lower WY costs; scales are analyzed separately. The 40 primary 10M pairs yield a mean cost difference of −0.7447 S (approximate paired-t 95% interval [−0.7811, −0.7084] S), where S is the scale-specific training-FLOPs unit: descriptive mean paired savings are 50.38%, with similar repetitions. A development-informed supplement trains 20 paired 11.24M-parameter generators. At equal exposure, WY uses 62.12% fewer training tokens; complete-output test accuracy is WY 97.86%, EN 95.03%. At nominal 16-million-token budgets, accuracy is WY 98.09%, EN 95.15%, with 2.64 times as many WY presentations. Input-plus-generated-output tokens per request, including visible steps and errors, are then 61.98% lower. One training-unseen layout accounts arithmetically for 39–43% of generator quality gaps in a post-hoc audit. Classifier transfer is poor, and generator evaluation data were development-reused. The evidence quantifies acquisition cost, checkpoint quality, and visible-output length for these task-specific packages, without isolating language effects or establishing matched-test-quality deployment savings.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
training efficiency, controlled language, Literary Chinese, token efficiency, state tracking

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_caPdOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper compares two hand-written renderings of the same synthetic box-swap state-tracking problems. One is an unpunctuated, contemporary-authored “controlled Literary Chinese” template with ASCII box and item names (WY). The other is a punctuated controlled English template (EN). Each rendering gets its own small byte-level BPE tokenizer (492 entries, merges allowed across word boundaries). WY comes out at about half the tokens: 7,934 vs 15,920 per pass over the 720-item inventory.

The main experiment trains paired from-scratch Transformer classifiers at 1M, 10M and 100M parameters (20, 40 and 20 seed pairs, with identical initialization within a pair). Each pair trains on the 720 items, and training FLOPs are charged until accuracy on those same 720 items stays at or above 75% for a 7,200-presentation window. At the pre-registered 10M scale all 40 pairs favor WY, with a mean saving of 50.38%. The number of presentations needed is nearly identical (31,244 vs 31,644), so the saving comes from cheaper examples, not fewer of them. 100M is the exception, with ~22% fewer WY presentations. The final classifiers are near chance on 320 challenge items.

A supplement trains 20 pairs of ~11M-parameter decoders on zero/one-swap problems with a configuration-disjoint test split. At equal passes, WY uses 62% fewer training tokens and gets higher complete-output test accuracy (97.86 vs 95.03). At a matched 16M-token budget it scores 98.09 vs 95.15, and it uses ~62% fewer tokens per request. A post-hoc audit traces 39-43% of the quality gap to 16 test items with a layout never seen in training. The authors frame everything as a comparison of rendering/tokenizer/schedule packages, not a language effect.

Critical strengths
  1. The experimental reporting is exceedingly careful
  • The primary scale, endpoint, persistence window and analysis were fixed in advance.
  • Seeds are paired with identical initial weights, and runs go to a fixed budget with no score-based stopping or seed replacement.
  • Every learning curve is plotted, and all 45 scale/threshold/span cells are reported, including the ones where EN never reaches the criterion.
  • The authors kept the earlier, unfavorable generator study in the paper and volunteered the layout audit in Tables J5/J6, which undercuts their own headline. This matters for my assessment because I don’t doubt the numbers. My problems are with what the numbers can tell us.
  1. Splitting the token saving into presentation count times tokens per presentation (Sec. 4.3, App. G) is the right test for any “compressed encoding learns cheaper” claim, and the paper answers it directly. At 1M and 10M the models need the same number of examples; each one just costs half as much. That stops the paper from claiming sample efficiency it doesn’t have, a check that’s easy to skip in this kind of comparison.
Critical weaknesses
  1. The primary experiment can’t separate Literary Chinese from “a shorter template,” and it isn’t really measuring learning in the usual sense. Several things stack up:
  • It measures memorization. Training and acquisition evaluation use the same 720 items, so the endpoint is the cost of memorizing one inventory. The challenge results confirm this: both packages land below the 61.25% majority-label baseline on the 320 named challenges, with balanced accuracy around 48-50%. = The tokens are near-whole template clauses. The tokenizers are fit to an in-domain corpus with merges across whitespace. The 4-box, 3-swap question in Appendix E is 5 WY tokens and 11 EN tokens before BOS/EOS, and the training inventory uses only 74 WY and 53 EN distinct token IDs. At that granularity the classifier sees a handful of chunk IDs, and most of what makes the two renderings “languages” is gone.
  • The compression ratio depends on the tokenizer. It is 55% with the frozen BPE, 26% with cl100k and 45% with o200k (Table E1).
  • The headline is close to predictable. Presentation counts are unchanged at 10M and sequences are 4-38 tokens long, so FLOPs are basically proportional to tokens. Using the 10M coefficients in Table A1, the quadratic term is under 2% of per-item cost even at T=38. Once you know the number of examples needed to memorize doesn’t depend on rendering, the 50.38% follows from encoding length alone.
  • There’s no compact English or symbolic control. The EN template keeps capitals, punctuation and repeated sentences like “Box C is empty.”, so we can’t tell whether anything about WY matters beyond being terser. The intro’s motivation (an economical, interpretable representation) is never actually tested.
  • The learning-rate schedule adds a confound. It is indexed by cumulative FLOPs, so the two conditions sit at different learning rates at the same update. By my arithmetic from Table 2, WY certifies around 0.7S (still in warmup) and EN around 1.5S (past the peak). Even the presentation comparison carries a schedule confound. This is the main reason for my score. Fixing it needs new controls and a held-out evaluation, not a rewrite.
  1. The generator supplement is where the paper tries to show a quality benefit, and I don’t think it supports one beyond this specific template pair.
  • It was redesigned after a negative result. An earlier 20-pair study came out unfavorable to WY (94.10 vs 95.02). The new study changed the task depth, tokenizers, output format, budget and schedule, and evaluates on the same reused test inventory. Fresh seeds don’t fix that.
  • Most of the gap sits in one unseen layout. 39-43% of the quality gap comes from 16 test items with all four items in box D, a layout that appears 0 times in training. Every EN model fails all 16 at both checkpoints. Per the discussion around Table J5, the EN input switches to a list form training never showed, while WY just repeats its conjunction. That’s a design asymmetry between the two templates.
  • The rest of the gap is small and mixed. On the other 1,008 items the gap is 1.6-1.8 points. In the max-occupancy-3 stratum (208 items), 11 of 20 pairs actually favor EN at both checkpoints (Table J6).
  • The token saving reflects authoring choices. The 62% request-token saving mostly comes down to how verbose the authored EN output format is.

A smaller point on presentation, not decisive for me: the paper is hard to read. Nearly every sentence carries one or two disclaimers, and after a while it’s hard to tell which claims the authors want the reader to take away, and this therefore reads as exceedingly AI. The abstract alone has around ten figures and several “not X” clauses. The fact that the paper needs Figure L1 as a “reading map” says something. I’d rather see each claim stated plainly once, with the caveats collected in the limitations section.

Questions for authors

These may need new experiments:

  1. Do you have any runs with a compact English rendering (no capitals or punctuation, repeated clauses collapsed) or a non-linguistic symbolic code at roughly WY’s token length? If presentation counts to criterion match WY’s, I’d read the classifier result as purely about length. If WY still comes out ahead at matched length, that would change my view a lot.
  2. Did you try a shared tokenizer, either one BPE fit on both renderings or an off-the-shelf one? Table E1 suggests the length gap drops to 26% under cl100k. Does the acquisition saving drop with it?
  3. For the generators, what exactly does the EN renderer produce for four items in one box, and does that form appear anywhere in training, dev or tokenizer fitting? If the EN template extended its repeated form the way WY does, would you expect the gap on those 16 items to disappear? A rerun with that change and a freshly regenerated split would be much more convincing than the current development-reused test set, and it should be cheap at 11M parameters.

Clarifications:

  1. Why index the classifier schedule by FLOPs rather than by updates? An update-indexed schedule would make the presentation comparison clean.
  2. At 100M, WY needs ~22% fewer presentations in 19 of 20 pairs, while 1M and 10M show essentially no difference. Do you have an explanation, and could it come from schedule phase rather than the rendering?
  3. Given the challenge results, what do you see as the value of measuring the cost to memorize a fixed inventory? Why not measure acquisition on a held-out split, as you do for the generators?
1 comment

Thank you for your updates. The paper is slightly more promising as a result, and I have updated the score to reflect that. Best of luck.

Reviewer_wu4QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper compares controlled Literary Chinese and English representations of synthetic state-tracking problems, measuring training cost and generated token use separately. The primary experiment trains paired Transformer classifiers from scratch at three scales and measures training FLOPs required to maintain an accuracy threshold on a fixed 720 example training inventory. At the primary 10M scale, WY achieves approximately 50% lower acquisition cost with similar numbers of example presentations.

A supplementary experiment trains 20 paired autoregressive generators. WY uses fewer training tokens at equal exposure and achieves higher complete-output accuracy at both equal-exposure and matched-token checkpoints. The paper also examines challenge-set transfer, output lengths, and layout-specific failures. Its conclusions concern the complete representation, tokenization, and training packages rather than an isolated effect of language.

Critical strengths
  • The primary experiment carefully distinguishes shorter inputs from lower cost to reach a learning criterion. Comparing sequence lengths alone would not establish training savings, because a shorter representation could require more repetitions to learn. The paper explicitly measures both quantities. At 10M, WY and EN require approximately 31244 and 31644 presentations, while WY achieves 50.38% lower training-FLOPs cost. All 40 primary pairs favor WY, and the direction persists across the pre-specified 80 to 95% accuracy thresholds and shorter persistence windows. Paired initialization, fixed run budgets, complete seed reporting and separate analyses by scale make this a credible result for the particular inventory and protocol.

  • The reporting is transparent about adverse results and the boundaries of the evidence. The paper reports poor classifier transfer, development reuse of the generator evaluation set, earlier inconclusive or unfavorable generator results, and the concentration of generator errors in particular layouts. It also distinguishes training tokens, FLOPs, presentation counts, and request tokens, and includes malformed outputs in the cost account. These distinctions prevent several common misleading efficiency comparisons. The generator's equal-exposure and matched-token evaluations are useful complementary controls and the figures show the relevant learning trajectories rather than only selected endpoints.

Critical weaknesses
  • The experiments do not yet establish lower cost for learning a generalizable state-tracking capability. The primary endpoint is accuracy on the same 720 examples used for training. Consequently, the strong seed-level replication establishes cheaper acquisition of a fixed inventory, potentially through memorization. The separate challenge evaluation does not provide evidence of successful transfer: at 10M, balanced accuracy is 49.85% for WY and 48.59% for EN (as presented in Table 4). The widespread presence of token IDs unseen during classifier training is a plausible contributor, but also makes this challenge set an imperfect test of compositional generalization. An evaluation with novel configurations and adequate training coverage of the required tokens is missing.

  • The generator supplement provides more encouraging results, but its entire evaluation inventory was reused during development, and the task contains only zero or one swap. Fresh initialization seeds quantify variability under the selected design. They do not remove selection effects from repeated development on those evaluation examples. Furthermore, one unseen initial layout contributes approximately 39 to 43% of the complete-output quality gap. WY retains an advantage on the remaining examples, so this does not explain away the result, but it makes replication across independently generated splits particularly important. The paper acknowledges these limitations, however they substantially restrict the scientific contribution beyond the fixed-package comparison.

  • The comparison does not identify what produces the advantage or establish competitiveness against simpler compact representations. WY and EN differ jointly in wording, punctuation, segmentation and learning-rate trajectories. The generator additionally uses different realized vocabulary sizes. Appendix J5 identifies a concrete asymmetry where the English four-item list introduces a form absent from training, whereas WY extends a repeated conjunction. Thus, part of the quality difference may reflect the chosen rendering rules and their training coverage rather than a broadly useful property of Literary Chinese.

  • A concise English rendering and a language-neutral symbolic representation are essential comparators for assessing the value of the proposed package. Compact English representations and tokenization-based efficiency improvements already have precedents, including work cited by the authors. At the primary scale, the approximately 50% acquisition-cost reduction closely tracks the ~50% per-pass token reduction, while presentation counts barely change. This is a useful measurement, but without stronger controls it offers limited insight into representation design or learning behavior. Matching learning-rate schedules by update count would also help determine how much the observed contrast depends on the compute-indexed policy.

Questions for authors
  • Does the acquisition-cost advantage persist when the stopping criterion is measured on a genuinely untouched set of configurations, with all necessary input tokens represented in training? For the generator, can the result be replicated on fresh configuration splits selected after freezing the rendering and training pipeline?

  • How does WY compare with concise English and a symbolic encoding under comparable tokenizer settings and an update-indexed learning-rate schedule? Which parts of the advantage survive these controls?

  • Can the complete training implementation and the primary per-update cost and accuracy records be included, allowing independent reproduction of the acquisition-cost results rather than only replay of summary statistics?

Reviewer_HeyMOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Essentially the authors wanted to see if it could be possible to train models on fewer tokens to reach the same performance on some tasks. For that they defined a simplified form of Chinese and English (a form of data compression) as their training data that they repeatedly trained over. They deliberately trained on all their dataset (no held out data), on which they reached 99-100% of accuracy. They implemented two scenarios: (i) A classification task for which they trained several transformer-based classifiers (1M, 10M and 100M parameters) to predict whether an object ends up in a particular box after a series of swaps. For that, they used only 720 training samples that they repeatedly trained on. They deliberately evaluated the models on the very same samples, without keeping a held-out test set. The idea was to measure how much computation each representation required to reach a given accuracy threshold (75%, maintained over several evaluations). At the end of training, the models reached almost perfect accuracy on their training data, which is not particularly surprising given the setup. The second scenario consist in (ii) a generative task: Here, they trained smaller, GPT-like models (around 11M parameters) on a slightly different version of the same box-tracking problem. Instead of just predicting True or False, the models had to generate a sequence describing where the object was initially, where it moved after a swap, and whether it ended up in the box mentioned in the question. So it's not really free-form text generation, just the generation of an explanation following a predefined template, with a final answer at the end. They used more training examples and separate development and test sets this time. They then compared the models under two conditions: training on the same number of examples and training with the same token budget (16M tokens). In the second case, the Chinese models could see considerably more examples because their representations were shorter. They also measured how many tokens the models generated when answering the questions.

My first and probably biggest concern on this paper is the writing. I found the paper extremely difficult to read, especially the abstract, introduction and discussion. The paragraphs often feel disconnected, and I had a hard time understanding what the authors actually wanted to demonstrate. Even the central concepts, like "controlled English" and "controlled Literary Chinese", are not introduced in the main paper. I don't understand why I had to reach Appendix E to finally see concrete examples of what they were comparing. My second concern is the scientific contribution itself. The authors essentially show that shorter sequences require less computation to train on, which is hardly surprising. They do try to establish whether this advantage persists when reaching a given accuracy threshold, but their main experiment consists of repeatedly training classifiers with up to 100M parameters on only 720 examples, without any held-out evaluation. So what exactly are we learning here, beyond the fact that their models can fit a tiny training dataset more cheaply? Finally, I have serious concerns about the experimental setup. The classifiers perform close to chance on the separate challenge evaluation, but most of those examples contain token IDs whose embeddings were never trained on the task data. This makes it difficult to understand what their generalization results actually tell us. The generative experiment is somewhat better designed, with separate training and test configurations, but the authors acknowledge that the test data had already been reused during development. Maybe it’s my reviewer fatigue but I struggle to see how these experiments support a meaningful conclusion beyond their very specific synthetic setting.

Critical strengths

the idea of compressing textual data is not new ouf course but despite the unreadability of this paper, i'm sure the authors are onto something that could become very interesting.

Critical weaknesses
  • the writing
  • we can't really interpret the generalization results and what do we learn from the first set of experiments?
Questions for authors

1° What exactly is the classifier's training objective? 2° Why train such large models (110M for the largest) on such small data?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.