acceptodds
Under review as a conference paper at ICLR 2027

SyntaxBPE: Learning Which Code Boundaries to Cross

Abstract

Most code tokenisers derive BPE merge rules primarily from corpus co-occurrence statistics, with limited use of the explicit structure of source code. This leads to a trade-off: frequency-driven merging may cross meaningful program boundaries, while rigid structural constraints can block useful recurring code patterns from being merged. This raises a question: which structural boundaries should constrain tokenisation, and which should remain open to merging? We study the coupled design of boundary policies and BPE vocabularies, and introduce SyntaxBPE, a two-stage tokeniser construction procedure. First, SyntaxBPE groups recurring boundary patterns into deployable operators. A shared model trained on randomised boundary views provides paired code-length estimates, which are used to select a deterministic policy through a lower-confidence-bound rule. Second, program structure guides vocabulary induction by downweighting merge occurrences that cross the remaining parser-visible boundaries. Structural information is used only during construction, leaving the final tokeniser deterministic and parser-free at deployment. Experiments with 1B and 3B code language models show consistent improvements over GPT-style tokenisers under both fixed-token-budget and matched-raw-data comparisons, with gains persisting after instruction tuning. These findings support using program structure as soft guidance while allowing selected cross-boundary patterns.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Code Language Models, Tokenization, Byte-Pair Encoding

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_GwYyOpenReview ↗
Rating
4(+2)
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

SyntaxBPE builds a code tokeniser in two stages.

  1. Stage I trains one 1B calibration model (100B tokens) on views where each candidate boundary operator is randomly split or open. The operators cover case, digits, letter–digit, underscore, punctuation pairs and symbol prefixes. An operator is opened if its simultaneous cluster-bootstrap LCB on the paired BPB gain is positive, and the selected operators are compiled into a fixed regex.
  2. Stage II runs BPE under that regex, discounting merge evidence that crosses the remaining AST boundaries (γ=0).

The deployed tokeniser is plain regex+BPE, with no parser at inference.

Validation: 30K vocabularies; 1B/3B Llama-style models trained for 720B/800B tokens. Against a retrained o200k-style ("GPT-5-style") BPE, the MultiPL-E average improves from 13.9 to 15.8 (1B) and from 20.4 to 22.4 (3B) for base models, and from 28.3 to 31.1 and from 47.3 to 49.6 after SFT. The direction holds under a matched-raw-data control. Ablations remove each stage, compare the calibrated policy against all-split and all-open over a fixed candidate set, and sweep γ.

Critical strengths
  1. Deployable, with a consistent headline gain. All structural information is used at construction time; the output is ordinary regex+BPE with no parser at inference. There are four GPT-5-vs-SyntaxBPE pretraining comparisons (1B and 3B, fixed-token and matched-raw; the 3B matched-raw comparison reuses the SyntaxBPE run), and all four favour SyntaxBPE by 1.9–2.2 points on MultiPL-E, with lower BPB at every reported checkpoint of the fixed-token runs.

  2. A sound calibration design. A single model trained on randomised split/open views, scored with paired views that differ only in the target operator, amortises what would otherwise be one LM per policy. The bootstrap clusters at the randomisation unit and the max-standardised bound is simultaneous. It is the paper's novel idea and is reusable beyond code.

Critical weaknesses

1. The gain is not attributed to calibration. The title claims boundaries are learned, but the main comparison changes many things at once:

  • Opening O2 (digit groups of 1–3) is the existing GPT default.
  • Opening O1 (camelCase) reverts o200k to cl100k behaviour.
  • Several prespecified choices were neither calibrated nor ablated against o200k: space-attached numbers (Table 15: "·512" vs "·" "512"), quote/marker runs, newline attachment.
  • "w/o calibration" swaps the whole regex, so calibrated and fixed decisions are confounded.
  • The only clean selection test (Table 10) is one run at 300B tokens: 9.65 vs 8.46.

There is no hand-written baseline: o200k with standalone delimiters, fused identifiers and the obvious operator whitelist, essentially Tables 8–9. If that heuristic matches SyntaxBPE, the contribution is a better code regex, not learned boundaries.

2. The ablations sit inside the variation the paper's own tables show. Every comparison is a single run, and Appendix E bootstraps only tasks and completions. Fixed-token and matched-raw runs differ only in length, yet single benchmarks swing by 2–6 points: Strict-AST 1B SFT MBPP 36.5 vs 30.9; w/o calibration 1B Python 18.4 vs 15.5. The two near-identical GPT baselines differ by 0.2–0.6 on average. Against that variation sit structure weighting (+0.7 to +1.8 across settings), the γ sweep (9.23 / 9.65 / 9.55) and calibrated vs all-split (+1.19). The Stage II and selection claims rest on these numbers. Stage II also barely changes the calibrated tokeniser (Cross-AST 2.46% → 2.26%, bytes/token 3.488 → 3.478), so the claimed complementarity is unsupported.

Questions for authors

Clarifications

  1. The 1B matched-raw target of 2.3 TB is below both fixed-token exposures (about 2.51 TB for SyntaxBPE and 2.84 TB for GPT-5 at 721.4B tokens), implying about 661B and 584B tokens. Yet the averages barely move (15.9 vs 15.8; 13.8 vs 13.9). Why 2.3 TB? What were the actual token counts? Are these separate runs, or earlier checkpoints of the same constant-LR trajectories?
  2. Calibration windows: are the "64-token prefix" and "next C=32 tokens" counted under the split or the open view? What is the size of the refined decision family (q=3.18 implies about 34 effective tests), and how was the refinement chosen before it was frozen?
  3. Base 1B HE+ is +3.5 [+2.0, +5.0] on 164 tasks. That implies a per-task difference SD of about 0.10, i.e. almost no fully flipped tasks. Can you release per-task outcomes?

Experiments that would change my assessment

  1. A hand-written code-regex baseline (Weakness 1) at 1B.
  2. 2–3 seeds for the main 1B pair and for the Stage I/II ablations.

Suggestions, not decisive

  1. Results at matched inference compute (SyntaxBPE emits about 13% more tokens) and a vocabulary sweep up to about 128K.
4 comments
AuthorsOpenReview ↗

Response to Reviewer GwYy (Part 1/2)

We thank the reviewer for the careful reading and for recognising the deployable design and the calibration procedure. Below, we address attribution to calibration, run-to-run variation, and the requested clarifications.

1. Attribution to calibration and a hand-written code baseline


We thank the reviewer for proposing this direct comparison. We have added a hand-written code regex paired with standard BPE (without structure weighting), with fused identifiers (including camelCase, underscores, and identifier-internal digits), standalone delimiters, and an explicit operator whitelist. It retains o200k-style 1–3-digit numeric groups and whitespace handling, and explicitly specifies quote/marker runs and newline attachment. This provides a stronger test of whether manually chosen code boundaries account for the gains. We have included the comparison and construction details in the revised manuscript (lines 379–391 and 1305–1331).

1B/400B-token comparison; pass@1 (%).

TokeniserMultiPL-E Avg.HE+MBPPMBPP+ClassEval
GPT-5-style BPE11.212.220.824.627.3
Hand-written code regex11.512.721.425.527.1
Calibrated, γ=111.813.223.026.528.6
SyntaxBPE12.414.624.628.029.5

The baseline follows the reviewer's specification: o200k with fused identifiers, standalone delimiters, and a whitelist of standard multi-character operators. We wrote it after completing our calibration analysis, so it is not blind to those findings; if anything, this favours the baseline. It is comparable to or slightly better than GPT-5-style BPE (−0.2 to +0.9 percentage points), confirming that a better code regex accounts for only a small part of the gain. SyntaxBPE still gains a further 0.9–3.2 percentage points on every metric, while the fixed-inventory ablation directly tests the benefit of calibration-based selection.

The hand-written regex achieves 3.394 bytes/token compared with 3.478 bytes/token for SyntaxBPE.

For reproducibility, we have included the complete hand-written pre-tokenisation regex in the revised manuscript (lines 1313–1331):

CODE_REGEX_HANDWRITTEN = (
    # Fused identifiers, with an optional leading space and contraction suffix.
    r""" ?[\p{L}\p{M}_][\p{L}\p{M}\p{N}_]*(?i:'s|'t|'re|'ve|'m|'ll|'d)?"""

    # Numeric groups of one to three digits, as in o200k.
    + r"""|\p{N}{1,3}"""

    # Quote/marker runs, whitelisted operators, and standalone symbols.
    # All symbol chunks share the same optional newline attachment.
    + r"""| ?(?:["'`#]+"""
    + r"""|<!--|>>>=|!==|&&=|&\^=|\*\*=|-->|->\*"""
    + r"""|\.\.\.|\.\.=|/\*\*|//=|<<=|<=>|===|>>=|>>>"""
    + r"""|\?->|\?\?=|\|\|=|!=|##|%=|&&|&=|&\^"""
    + r"""|\*\*|\*/|\*=|\+\+|\+=|--|-=|->"""
    + r"""|\.\*|\.\.|\.=|/\*|//|/=|::|:=|<-|<<|<="""
    + r"""|==|=>|>=|>>|\?\.|\?\?|@=|\^=|\|=|\|\|"""
    + r"""|[^\s\p{L}\p{N}])[\r\n]*"""

    # Whitespace, as in o200k.
    + r"""|\s*[\r\n]+"""
    + r"""|\s+(?!\S)"""
    + r"""|\s+"""
)

2. Training-seed variation and Stage I/II ablations


We thank the reviewer for this suggestion. Despite the pretraining cost, we have added a second training seed for each of the four configurations below and reported the two-run results in the revised manuscript (lines 1713–1727).

1B/300B-token results; pass@1 (%), reported as mean ± standard deviation across two independent training runs.

ConfigurationMultiPL-E Avg.HE+MBPPMBPP+ClassEval
GPT-5-style BPE7.6 ± 0.310.1 ± 0.212.8 ± 0.518.1 ± 0.223.3 ± 0.4
SyntaxBPE, γ=09.7 ± 0.113.1 ± 0.418.0 ± 0.521.5 ± 0.327.2 ± 0.2
Calibrated, γ=18.5 ± 0.311.1 ± 0.214.3 ± 0.419.4 ± 0.125.0 ± 0.5
All-split, γ=08.6 ± 0.211.9 ± 0.516.7 ± 0.120.6 ± 0.425.7 ± 0.3

SyntaxBPE (γ=0) leads across all five metrics, exceeding the reported seed variation and supporting calibration and structure weighting. Small aggregate tokeniser changes can thus accompany meaningful downstream gains.

AuthorsOpenReview ↗

Response to Reviewer GwYy (Part 2/2)

3. Matched-raw budget and checkpoint provenance


We thank the reviewer for identifying this ambiguity. We chose 2.3 TB to compare matched raw-data exposure using checkpoints already available from the existing training trajectories, avoiding additional pretraining. These are earlier checkpoints from the same constant-learning-rate runs, corresponding to 660.6B tokens for SyntaxBPE and 585.1B tokens for GPT-5-style BPE. The nearly unchanged MultiPL-E averages do not establish convergence: benchmark pass@1 need not improve monotonically with training, and the observed changes are only 0.1 percentage points. The key observation is that SyntaxBPE's advantage persists under matched exposure (15.9 vs 13.8), as at the fixed-token endpoints (15.8 vs 13.9). We have clarified the checkpoint provenance and actual token counts in the revised manuscript (lines 1642–1651).

4. Calibration windows and the refined decision family


We thank the reviewer for these questions. Both views share a prefix of up to 64 tokens and continuation of up to 32 tokens, counted under a common open-view background tokenisation; only the target boundary changes. Candidate families draw on GPT-style rules, identifier conventions and an AST-crossing audit. O5/O6 are refined into punctuation-pair and symbol-prefix candidates, with the inventory and matching precedence fixed before held-out confirmatory calibration and downstream evaluation. We have clarified the paired windows and refined decision family in the revised manuscript (lines 166–179 and 992–1015).

For the 592 eligible candidates, q = 4.315 is the empirical 95th percentile of the maximum absolute standardised deviation. We have updated the O1–O4 multiplier and the corresponding lexical LCBs in the revised manuscript (lines 260–282); the Appendix’s bootstrap definition remains unchanged (lines 836–850). All five broad lexical LCBs remain positive, preserving their OPEN decisions, and downstream results are unaffected. The corrected lexical summary and distinction between descriptive parent-family statistics and refined selection decisions are provided in the supplementary clarification (https://github.com/mkdpzth6n2/syntaxbpe-rebuttal-answers/blob/main/calibration_clarification.md).

5. Per-task evaluation outcomes and confidence intervals


We thank the reviewer for this suggestion. We now provide per-task MultiPL-E, HE+, MBPP, and MBPP+ outcomes and updated confidence intervals (https://github.com/mkdpzth6n2/syntaxbpe-rebuttal-answers/blob/main/uncertainty_clarification.md), together with a script reproducing the intervals from the released outcomes. We have included these intervals and the resampling protocol in the revised manuscript (lines 1608–1635).

6. Inference token usage and vocabulary size


We thank the reviewer for raising these practical considerations. The overhead is real: for the same code, SyntaxBPE decodes about 13% more tokens than GPT-5-style BPE (3.939 vs 3.478 bytes/token). Matching inference compute, however, does not have a single definition. An equal token cap matches decoding compute but lets SyntaxBPE emit less code before truncation, whereas an equal code-length cap, as in our current evaluation, matches what each model may emit but lets SyntaxBPE spend more decoding compute. The two cannot be satisfied simultaneously, so any such comparison is conditional on the chosen protocol. Our fixed-token-budget and matched-raw-data settings are the training-side counterparts of these two protocols, and SyntaxBPE improves over GPT-5-style BPE under both.

Larger vocabularies are not necessarily preferable for code generation: DeepSeek-Coder and CodeLlama use approximately 32K tokens, while StarCoder2 uses 49,152 and reports no improvement from expanding to 100K. At approximately 1B scale, vocabulary expansion also creates a substantial capacity tradeoff. In our archived configurations, input embeddings and the untied output layer account for approximately 11.2% of parameters at 30K versus 39.4% at 128K, with the latter using a narrower Transformer. This provides a plausible explanation for the observed performance difference, rather than establishing a general disadvantage of larger vocabularies. We have included the vocabulary-size comparison and the parameter-allocation caveat in the revised manuscript (lines 1759–1771).

1B/400B-token vocabulary-size comparison; pass@1 (%). Higher bytes/token indicates stronger compression.

TokeniserBytes/tokenMultiPL-E Avg.HE+MBPPMBPP+ClassEval
SyntaxBPE 30K3.47812.414.624.628.029.5
SyntaxBPE 128K3.81010.813.420.024.128.3
Reviewer_GwYyOpenReview ↗

Thanks for the context

This context gives me confidence to move the rating to 4. Please add appendices to your paper with these results.

AuthorsOpenReview ↗

Thank You for Your Encouraging Feedback

Thank you very much for your positive feedback and for reconsidering your rating. We sincerely appreciate your constructive suggestions and will incorporate these additional results into our revised paper.

Reviewer_tTrDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This interesting paper investigates improving model vocabulary/tokenization for code language models. The authors first motivate this by noting that existing vocabularies, commonly based on BPE with a pretokenizer, may produce counter-intuitive splits that cross unexpected boundaries, from the perspective of a programming language. This can include leading/trailing parentheses, method "." syntax, etc. The authors propose a two stage approach. After gathering candidate "operators", the authors build a mixed training set including both varieties, open and closed operators. They then train a base model on this and measure model loss under different open/close settings. Based on these measurements, they are then able to select a set of operators to use, which are baked into a static pretokenizer. In the second stage, they apply "hints" from code to the bpe merge process. Finally, they train two models at 1 and 3B sizes using different configurations of this system, and report standard code benchmark results.

Critical strengths
  • Addresses a clear problem in determining BPE splits. Authors carefully inspect existing pretokenizer pipelines and how these affect downstream splits and merges in the actual BPE process.

  • Training the calibration model on mixed operator settings is a very interesting method for comparing two vocabularies. Authors carefully designed this process so that different settings would get fair supervision. Rather than try cross-model or cross-vocabulary analysis, this allows comparing different schemes within a single joint model!

  • Results seem promising at model scales and the operator decisions are intuitively supported.

Critical weaknesses
  • Operator set seems somewhat limited, though this might be to maintain usability across many languages.

  • Why not consider an OSS model or fully open model with trained tokenizer? It seems the main settings are GPT-4 and -5 style merges, rather than also including other pretokenizer strategies.

  • Further analysis for the effects on natural language, or the distribution of natural language inside code documents, would be useful. One concern is that this could over-optimize for code, which could then decrease instruction following or long reasoning abilities in larger or more complex models. This could limit the impact/generality of the method.

  • Existing experiments were far beyond compute optimal but early results showed differentiation between methods. Therefore some training budget could be spent on exploring other operators or the issues above.

Questions for authors

What datamix factors would influence the balance of natural language and code?

Why were models trained to the specific ~800B durations? This doesn't match any scaling law heuristic (i.e. 20 for chinchilla, or much higher for small models), since the models are different sizes and trained for similar durations.

4 comments
AuthorsOpenReview ↗

Response to Reviewer tTrD (Part 1/2)

We thank the reviewer for the positive assessment of SyntaxBPE, in particular the shared-model calibration design and the analysis of existing pre-tokenisers. Below, we address operator coverage, open-tokeniser baselines, natural-language capabilities, the data mixture, and training duration.

1. Operator-set coverage


We thank the reviewer for this observation. Our goal is to learn useful code boundaries within a compact, cross-language search space.

(1) The inventory covers recurring non-whitespace boundary choices. We compared 14 released pre-tokenisers on 200 code windows across 10 languages (805KB; seed 42). At each non-whitespace character boundary, we checked whether some tokenisers split while others allow merging, and whether our operator space can express both choices. For example, Llama 3 keeps getUserName within one span, whereas Llama 4 splits get|User|Name; both disputed positions are controllable by our case-transition operator. Our inventory covers 98.38% of such disagreement positions (44,557/45,291). Thus, the inventory already captures nearly all boundary choices on which existing pre-tokenisers differ.

(2) Calibration selects the boundary policy compiled into SyntaxBPE. The six operator families organise the candidate space. Within this space, calibration compares split and open views under a shared tokeniser and language model to select supported boundary openings. In particular, O5/O6 are evaluated through concrete punctuation-pair and symbol-prefix suboperators, allowing different patterns within the same family to receive different decisions. These calibrated choices, together with explicitly prespecified rules, are then compiled into the final pre-tokenisation regex. We have clarified the refined inventory and the distinction between calibrated and prespecified choices in the revised manuscript (lines 992–1015 and 1335–1342).

2. OSS and fully open tokeniser baselines


We thank the reviewer for this helpful suggestion. Our aim is to isolate the effect of pre-tokenisation boundaries while keeping vocabulary size and tokeniser-training data fixed. Directly adopting released tokenisers would introduce differences in these factors, including vocabularies substantially larger than our 30K setting. We therefore compare their boundary strategies by training a 30K BPE vocabulary on the same code corpus for each method using the public pre-tokenisation regexes. “GPT-4/5-style” refers to these public regexes, with merges learned from our corpus. These regexes are not specific to OpenAI models. Our audit of released configurations shows that the same regex families are used by open models, including fully open ones:

Pre-tokenisation strategyReleased models
GPT-4-style (cl100k), same regexLlama 3/3.1/3.2/3.3, OLMo 2/3/3.1 (fully open), Phi-4/reasoning, DBRX, Granite 4.0/4.1, GLM-4, SmolLM3
GPT-5-style (o200k), same regexLlama 4 Scout/Maverick, gpt-oss-20B/120B, Phi-4-mini/multimodal/mini-reasoning, MiniMax-Text-01
GPT-4-style variants (e.g., single-digit splitting; separate trailing newlines)Qwen2.5-Coder/Qwen3-Coder-Next, Hunyuan, Ling-Coder-lite, Seed-Coder/Seed-OSS, DeepSeek V3/R1/V3.1/V3.2
GPT-5-style-related variants (e.g., single-digit splitting; additional Han-character handling)Mistral Nemo, Nemotron-H/Nano-v2, Apertus (fully open), Kimi/Moonlight

Thus, the GPT-style baselines represent the pre-tokenisation strategies used across the open-model ecosystem. We have emphasised this baseline rationale in the main text and included the family mapping in the appendix (revised manuscript, lines 307–315 and 1244–1278).

To include strategies beyond the GPT-style regexes, we have added two recent tokenisation methods, each trained as a 30K tokeniser on the same corpus: SuperBPE, which relaxes pre-tokenisation boundaries, and SA-BPE, which reweights merge scoring. Both comparisons and their construction details are now included in the revised manuscript (lines 379–391 and 1282–1302).

1B/400B-token comparison; pass@1 (%).

TokeniserMultiPL-E Avg.HE+MBPPMBPP+ClassEval
GPT-5-style BPE11.212.220.824.627.3
SuperBPE10.212.419.222.526.2
SA-BPE10.711.420.122.727.0
SyntaxBPE12.414.624.628.029.5

SyntaxBPE achieves the highest score on all five metrics, leading the strongest alternative on each metric by 1.2–3.8 percentage points.

AuthorsOpenReview ↗

Response to Reviewer tTrD (Part 2/2)

3. Natural-language capabilities and instruction following


We thank the reviewer for raising this concern. To examine whether SyntaxBPE's code-oriented boundaries compromise broader capabilities, we evaluate the final 3B SyntaxBPE and GPT-5-style pretraining checkpoints reported in the paper, both before and after SFT. The base-model evaluation covers knowledge, mathematical reasoning, reading comprehension, commonsense, and language understanding. We have included these results in the main text (revised manuscript, lines 404–426).

CategoryTaskBase SyntaxBPEBase GPT-5-styleΔ (pp)
KnowledgeARC-Challenge34.22%33.79%+0.43
KnowledgeMMLU33.42%32.72%+0.70
KnowledgeOpenBookQA37.20%32.00%+5.20
Math & ReasoningArithmetic57.72%50.90%+6.82
Math & ReasoningGSM8K4.47%2.35%+2.12
Reading ComprehensionBoolQ62.35%60.24%+2.11
Reading ComprehensionDROP22.78%21.98%+0.80
Reading ComprehensionSQuAD50.46%49.44%+1.02
CommonsenseCommonsenseQA41.52%34.15%+7.37
CommonsenseCOPA62.00%53.00%+9.00
CommonsensePIQA52.23%52.50%-0.27
Language UnderstandingHellaSwag30.50%28.80%+1.70
Language UnderstandingLanguage Identification9.56%9.24%+0.32

SyntaxBPE is comparable or slightly higher on 12 of 13 tasks; the only decrease is 0.27 pp on PIQA. This table shows no evidence that SyntaxBPE's code-generation gains come at the expense of natural-language performance in this 3B comparison.

We further evaluate instruction following after SFT using IFEval and Multi-IF, scored with the official rules. T1–T3 denote the respective conversation turns; all scores are percentages, and Δ denotes percentage-point differences. We have included these instruction-following results in the appendix (revised manuscript, lines 1730–1742).

DatasetStrict SyntaxBPEStrict GPT-5-styleΔ (pp)Loose SyntaxBPELoose GPT-5-styleΔ (pp)
IFEval34.5333.69+0.8440.1737.29+2.88
Multi-IF T128.3628.28+0.0831.8831.00+0.88
Multi-IF T228.2027.03+1.1730.2028.32+1.88
Multi-IF T322.6122.21+0.4023.9323.67+0.26

SyntaxBPE is comparable or slightly higher under both strict and loose scoring on IFEval and all three Multi-IF turns; the differences are small (0.08–2.88 pp). Together, the base and SFT results show no evidence of degradation in natural-language capabilities or instruction following, including across multiple conversation turns.

4. Effect of the natural-language/code data mixture


To examine how the data mixture affects the balance between natural language and code, we compare NL:code ratios of 5:5 and 3:7 using 1B SyntaxBPE models pretrained for 400B tokens. We have included this comparison in the revised manuscript (lines 1746–1757). The following table reports code-generation performance (pass@1, %).

MetricNL:code = 5:5NL:code = 3:7Δ (5:5 − 3:7, pp)
MultiPL-E Avg9.0512.37−3.32
HE+11.6514.63−2.98
MBPP17.2824.60−7.32
MBPP+21.2728.04−6.77
ClassEval25.6829.51−3.83

Increasing the natural-language share from 30% to 50% lowers all five code-generation metrics by 2.98–7.32 percentage points, showing that the NL/code sampling ratio is an important factor in the capability balance. The 3:7 mixture provides stronger code performance in this comparison, while the natural-language and instruction-following evaluations above complement the code results when assessing broader capabilities.

5. Choice of training duration


The 720B/800B-token budgets were practical experimental choices, not scaling-law-derived optima. Our goal is a controlled tokeniser comparison: within each model size, all variants receive the same nominal token budget. We have clarified this budget choice and its scope in the revised manuscript (lines 1499–1503). Moreover, intermediate checkpoints from 40B to 720B tokens consistently show higher average code-generation pass@1 and lower validation BPB for SyntaxBPE than GPT-5-style BPE, supporting its benefit across training budgets rather than only at the chosen endpoints (revised manuscript, lines 479–485).

Reviewer_tTrDOpenReview ↗

Thanks for the interesting work!

I appreciate the further work by these authors in answering my questions and the questions of the other reviewers. I'll maintain my positive score and recommendation!

AuthorsOpenReview ↗

Thank You for Your Continued Support

Thank you very much for your encouraging feedback and continued support! We sincerely appreciate your time and thoughtful comments throughout the review process. Your suggestions have been very helpful in strengthening our paper.

Reviewer_ks3eOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes SyntaxBPE, a structure-aware tokenization method for code language models. It addresses the limitations of conventional BPE, which relies primarily on token co-occurrence frequencies and may merge tokens across meaningful syntactic boundaries. Meanwhile, strictly enforcing syntactic boundaries can prevent useful code patterns from being merged. To balance these two approaches, SyntaxBPE introduces a two-stage framework. First, it uses predictive calibration with a shared language model to determine which structural boundaries should remain open for merging. Second, it employs structure-weighted BPE to penalize merges that cross remaining syntactic boundaries during vocabulary construction. The resulting tokenizer is deterministic and does not require a parser at inference time. Experiments on 1B and 3B code language models demonstrate improvements over GPT-style BPE tokenizers across multiple code generation benchmarks, with gains persisting after instruction tuning.

Critical strengths
  1. Well-motivated research problem.

  2. Novel and interesting technical idea.

The paper addresses an important problem in code language model tokenization. I find the core idea of SyntaxBPE interesting and technically novel.

Critical weaknesses
  1. Limited evaluation on larger language models.

The experiments are conducted only on 1B and 3B code language models. Although the results show consistent improvements at these two scales, both models are relatively small compared with modern code language models. It remains unclear whether the proposed tokenization strategy would provide similar benefits for larger models, such as 7B, 14B, or 32B models. Larger models may learn code structures and relationships differently, which could affect the benefits of explicitly incorporating syntactic information into tokenization. Therefore, the current evaluation is insufficient to establish the generalizability of SyntaxBPE across model scales.

  1. Missing comparisons with closely related tokenization methods.

The experimental evaluation mainly compares SyntaxBPE against GPT-4-style BPE, GPT-5-style BPE, and Strict-AST BPE. However, several closely related methods, including SuperBPE, BoundlessBPE, MorphBPE, and SA-BPE, are discussed in the related work but are not included in the experimental comparisons. These methods are particularly relevant to the proposed approach. SuperBPE and BoundlessBPE explore relaxing pre-tokenization boundaries, while MorphBPE incorporates structural information into tokenization, and SA-BPE modifies BPE merge scoring through reweighting. Although the paper discusses the conceptual differences between SyntaxBPE and these methods, such differences alone do not establish the advantages of the proposed approach. It is difficult to determine whether the proposed boundary calibration and structure-weighted BPE offer meaningful improvements over existing tokenization methods.

Questions for authors

See Weaknesses

1 comment
AuthorsOpenReview ↗

Response to Reviewer ks3e (Part 1/1)

We thank the reviewer for recognising the motivation and novelty of SyntaxBPE. We address the two concerns below: generalisation to larger models and comparisons with recent tokenisation methods.

1. Model scale


The original study focused on 1B/3B because of limited compute: each tokeniser requires separate pretraining from scratch. For reference, the model sizes and pretraining budgets of the cited tokenisation studies are summarised below.

MethodLM size (B parameters)Pretraining budget (B tokens)
SuperBPE8.1 (main experiments)330
BoundlessBPE— (no LM trained)—
MorphBPE0.3/16 / 20
SA-BPE0.1015.7
SyntaxBPE (ours)1.1/3720 / 800

To directly address this concern, we have added a 7B/300B-token comparison with GPT-5-style BPE (revised manuscript, lines 379–384 and 398–402). The 7.02B model has 32 Transformer layers, hidden size 4,096, FFN hidden size 13,824, and 32 attention heads with 8 query groups. Training it required 14,131 GPU-hours on 128 AMD MI300X GPUs. We match architecture, optimisation, context length and vocabulary size. Evaluation follows the submission's protocol (pass@1, %, 10 samples, temperature 0.2). We have included the architecture, training configuration, and evaluation details in the revised manuscript (lines 1459–1472, 1494–1498, and 1513–1534).

TokeniserMultiPL-E Avg.HE+MBPPMBPP+ClassEval
GPT-5-style BPE15.817.632.339.535.1
SyntaxBPE18.820.635.142.038.6

SyntaxBPE improves all five metrics at 7B by 2.5–3.5 percentage points. Across scales, its MultiPL-E Avg. gain over GPT-5-style BPE is +1.9 at 1B (15.8 vs 13.9), +2.0 at 3B (22.4 vs 20.4), and +3.0 at 7B (18.8 vs 15.8). The token budgets differ across scales (720B/800B/300B), so these gains are not directly comparable, but we observe no sign that the benefit shrinks as the model grows. We were unable to cover 14B or 32B models because of compute constraints: 7B is the largest scale at which we could pretrain both tokenisers from scratch within the rebuttal period, and we have stated this limitation in the revised manuscript (lines 1494–1503).

2. Recent tokenisation baselines


We thank the reviewer for suggesting these recent tokenisation methods. Our existing GPT-4/5-style baselines are already strong, widely adopted baselines: their public pre-tokenisation regexes, or close variants, are used across most major open-model families, including Llama, Qwen, DeepSeek, OLMo, and gpt-oss. We retrain their BPE vocabularies on the same code corpus with the same 30K vocabulary size, ensuring a controlled comparison. We have made this baseline rationale explicit in the main text and added the open-model family mapping in the appendix (revised manuscript, lines 307–315 and 1244–1278). To further broaden the comparison, we considered all four methods listed by the reviewer:

MethodTokeniser backbone / changeLM training in the original paper?Included in our comparison?Configuration / reason
SuperBPEGPT-5-style BPE, then relaxed whitespace boundariesYesYes27K standard tokens → 3K superword merges; 90% transition ratio
SA-BPESame GPT-5-style pre-tokeniser; source-diversity-weighted merge rankingYesYesPublished merge score Flog⁡R⋅LF\log R\cdot L; 30K vocabulary
BoundlessBPEBoundary relaxation, in the same broad family as SuperBPENoNoSuperBPE selected as the representative boundary-relaxation baseline under limited compute
MorphBPENatural-language morpheme boundariesYesNoNo implementation code in its public repository at the time of writing; no direct code-morpheme counterpart

Both added baselines use a 1B LM backbone, 400B training tokens, and 30K tokeniser vocabularies trained on the same code corpus. We have included the construction details, intrinsic statistics, and downstream results in the revised manuscript (lines 379–391 and 1282–1302), and report the results below.

TokeniserBytes/tokenCross-AST (%)
GPT-5-style BPE3.93911.70
SuperBPE4.27514.00
SA-BPE3.9348.51
SyntaxBPE3.4782.26
TokeniserMultiPL-E Avg.HE+MBPPMBPP+ClassEval
GPT-5-style BPE11.212.220.824.627.3
SuperBPE10.212.419.222.526.2
SA-BPE10.711.420.122.727.0
SyntaxBPE12.414.624.628.029.5

SyntaxBPE consistently outperforms all baselines across the five downstream metrics, improving over GPT-5-style BPE by 1.2–3.8 percentage points. Notably, its lower bytes/token means that it sees less raw training data under the matched 400B-token budget, yet achieves better performance, further highlighting its data efficiency.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary for the Area Chair

Dear Area Chair,

Thank you for your time and consideration of our submission. Following the rebuttal discussion, all three reviewers support acceptance: GwYy raised their rating from 2 to 4, ks3e raised their rating from 2 to 3, and tTrD maintained 3. We appreciate their constructive feedback and summarise the main outcomes below.

  1. Benefits extend to larger models and recent tokenisation baselines. At 7B/300B tokens, SyntaxBPE improves all five code-generation metrics over GPT-5-style BPE by 2.5–3.5 percentage points. Additional 1B/400B-token comparisons with SuperBPE and SA-BPE also favour SyntaxBPE across all five metrics under matched training budgets, vocabulary sizes, and tokeniser-training data.

  2. Broader capabilities are largely preserved. At 3B, SyntaxBPE improves performance on 12 of 13 natural-language tasks, spanning knowledge, mathematical reasoning, reading comprehension, commonsense, and language understanding. After instruction tuning, IFEval and multi-turn Multi-IF scores are consistently slightly higher. These evaluations provide evidence that the code-generation gains are compatible with broadly preserved natural-language and instruction-following capabilities in the tested setting.

  3. Additional controls strengthen attribution to both stages. We added a hand-written code-regex baseline at 1B/400B tokens, which SyntaxBPE outperforms on all five metrics. Two independent training runs per configuration at 1B/300B tokens further strengthen the component comparisons. Ablations holding the candidate inventory or calibrated policy fixed support contributions from calibration and structure weighting, respectively; the full method leads on all five metrics in mean performance.

Together, these findings strengthen the evidence for learning code-tokenisation boundaries and using program structure to guide vocabulary construction, while retaining deterministic, parser-free deployment. The additional results and clarifications are incorporated into the revised manuscript. We appreciate your consideration of this expanded evidence and the reviewers' updated recommendations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.