- Rating
- 4(+2)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
SyntaxBPE builds a code tokeniser in two stages.
- Stage I trains one 1B calibration model (100B tokens) on views where each candidate boundary operator is randomly split or open. The operators cover case, digits, letter–digit, underscore, punctuation pairs and symbol prefixes. An operator is opened if its simultaneous cluster-bootstrap LCB on the paired BPB gain is positive, and the selected operators are compiled into a fixed regex.
- Stage II runs BPE under that regex, discounting merge evidence that crosses the remaining AST boundaries (γ=0).
The deployed tokeniser is plain regex+BPE, with no parser at inference.
Validation: 30K vocabularies; 1B/3B Llama-style models trained for 720B/800B tokens. Against a retrained o200k-style ("GPT-5-style") BPE, the MultiPL-E average improves from 13.9 to 15.8 (1B) and from 20.4 to 22.4 (3B) for base models, and from 28.3 to 31.1 and from 47.3 to 49.6 after SFT. The direction holds under a matched-raw-data control. Ablations remove each stage, compare the calibrated policy against all-split and all-open over a fixed candidate set, and sweep γ.
Critical strengths
-
Deployable, with a consistent headline gain. All structural information is used at construction time; the output is ordinary regex+BPE with no parser at inference. There are four GPT-5-vs-SyntaxBPE pretraining comparisons (1B and 3B, fixed-token and matched-raw; the 3B matched-raw comparison reuses the SyntaxBPE run), and all four favour SyntaxBPE by 1.9–2.2 points on MultiPL-E, with lower BPB at every reported checkpoint of the fixed-token runs.
-
A sound calibration design. A single model trained on randomised split/open views, scored with paired views that differ only in the target operator, amortises what would otherwise be one LM per policy. The bootstrap clusters at the randomisation unit and the max-standardised bound is simultaneous. It is the paper's novel idea and is reusable beyond code.
Critical weaknesses
1. The gain is not attributed to calibration. The title claims boundaries are learned, but the main comparison changes many things at once:
- Opening O2 (digit groups of 1–3) is the existing GPT default.
- Opening O1 (camelCase) reverts o200k to cl100k behaviour.
- Several prespecified choices were neither calibrated nor ablated against o200k: space-attached numbers (Table 15: "·512" vs "·" "512"), quote/marker runs, newline attachment.
- "w/o calibration" swaps the whole regex, so calibrated and fixed decisions are confounded.
- The only clean selection test (Table 10) is one run at 300B tokens: 9.65 vs 8.46.
There is no hand-written baseline: o200k with standalone delimiters, fused identifiers and the obvious operator whitelist, essentially Tables 8–9. If that heuristic matches SyntaxBPE, the contribution is a better code regex, not learned boundaries.
2. The ablations sit inside the variation the paper's own tables show. Every comparison is a single run, and Appendix E bootstraps only tasks and completions. Fixed-token and matched-raw runs differ only in length, yet single benchmarks swing by 2–6 points: Strict-AST 1B SFT MBPP 36.5 vs 30.9; w/o calibration 1B Python 18.4 vs 15.5. The two near-identical GPT baselines differ by 0.2–0.6 on average. Against that variation sit structure weighting (+0.7 to +1.8 across settings), the γ sweep (9.23 / 9.65 / 9.55) and calibrated vs all-split (+1.19). The Stage II and selection claims rest on these numbers. Stage II also barely changes the calibrated tokeniser (Cross-AST 2.46% → 2.26%, bytes/token 3.488 → 3.478), so the claimed complementarity is unsupported.
Questions for authors
Clarifications
- The 1B matched-raw target of 2.3 TB is below both fixed-token exposures (about 2.51 TB for SyntaxBPE and 2.84 TB for GPT-5 at 721.4B tokens), implying about 661B and 584B tokens. Yet the averages barely move (15.9 vs 15.8; 13.8 vs 13.9). Why 2.3 TB? What were the actual token counts? Are these separate runs, or earlier checkpoints of the same constant-LR trajectories?
- Calibration windows: are the "64-token prefix" and "next C=32 tokens" counted under the split or the open view? What is the size of the refined decision family (q=3.18 implies about 34 effective tests), and how was the refinement chosen before it was frozen?
- Base 1B HE+ is +3.5 [+2.0, +5.0] on 164 tasks. That implies a per-task difference SD of about 0.10, i.e. almost no fully flipped tasks. Can you release per-task outcomes?
Experiments that would change my assessment
- A hand-written code-regex baseline (Weakness 1) at 1B.
- 2–3 seeds for the main 1B pair and for the Stage I/II ablations.
Suggestions, not decisive
- Results at matched inference compute (SyntaxBPE emits about 13% more tokens) and a vocabulary sweep up to about 128K.
4 comments
Response to Reviewer GwYy (Part 2/2)
3. Matched-raw budget and checkpoint provenance
We thank the reviewer for identifying this ambiguity. We chose 2.3 TB to compare matched raw-data exposure using checkpoints already available from the existing training trajectories, avoiding additional pretraining. These are earlier checkpoints from the same constant-learning-rate runs, corresponding to 660.6B tokens for SyntaxBPE and 585.1B tokens for GPT-5-style BPE. The nearly unchanged MultiPL-E averages do not establish convergence: benchmark pass@1 need not improve monotonically with training, and the observed changes are only 0.1 percentage points. The key observation is that SyntaxBPE's advantage persists under matched exposure (15.9 vs 13.8), as at the fixed-token endpoints (15.8 vs 13.9). We have clarified the checkpoint provenance and actual token counts in the revised manuscript (lines 1642–1651).
4. Calibration windows and the refined decision family
We thank the reviewer for these questions. Both views share a prefix of up to 64 tokens and continuation of up to 32 tokens, counted under a common open-view background tokenisation; only the target boundary changes. Candidate families draw on GPT-style rules, identifier conventions and an AST-crossing audit. O5/O6 are refined into punctuation-pair and symbol-prefix candidates, with the inventory and matching precedence fixed before held-out confirmatory calibration and downstream evaluation. We have clarified the paired windows and refined decision family in the revised manuscript (lines 166–179 and 992–1015).
For the 592 eligible candidates, q = 4.315 is the empirical 95th percentile of the maximum absolute standardised deviation. We have updated the O1–O4 multiplier and the corresponding lexical LCBs in the revised manuscript (lines 260–282); the Appendix’s bootstrap definition remains unchanged (lines 836–850). All five broad lexical LCBs remain positive, preserving their OPEN decisions, and downstream results are unaffected. The corrected lexical summary and distinction between descriptive parent-family statistics and refined selection decisions are provided in the supplementary clarification (https://github.com/mkdpzth6n2/syntaxbpe-rebuttal-answers/blob/main/calibration_clarification.md).
5. Per-task evaluation outcomes and confidence intervals
We thank the reviewer for this suggestion. We now provide per-task MultiPL-E, HE+, MBPP, and MBPP+ outcomes and updated confidence intervals (https://github.com/mkdpzth6n2/syntaxbpe-rebuttal-answers/blob/main/uncertainty_clarification.md), together with a script reproducing the intervals from the released outcomes. We have included these intervals and the resampling protocol in the revised manuscript (lines 1608–1635).
6. Inference token usage and vocabulary size
We thank the reviewer for raising these practical considerations. The overhead is real: for the same code, SyntaxBPE decodes about 13% more tokens than GPT-5-style BPE (3.939 vs 3.478 bytes/token). Matching inference compute, however, does not have a single definition. An equal token cap matches decoding compute but lets SyntaxBPE emit less code before truncation, whereas an equal code-length cap, as in our current evaluation, matches what each model may emit but lets SyntaxBPE spend more decoding compute. The two cannot be satisfied simultaneously, so any such comparison is conditional on the chosen protocol. Our fixed-token-budget and matched-raw-data settings are the training-side counterparts of these two protocols, and SyntaxBPE improves over GPT-5-style BPE under both.
Larger vocabularies are not necessarily preferable for code generation: DeepSeek-Coder and CodeLlama use approximately 32K tokens, while StarCoder2 uses 49,152 and reports no improvement from expanding to 100K. At approximately 1B scale, vocabulary expansion also creates a substantial capacity tradeoff. In our archived configurations, input embeddings and the untied output layer account for approximately 11.2% of parameters at 30K versus 39.4% at 128K, with the latter using a narrower Transformer. This provides a plausible explanation for the observed performance difference, rather than establishing a general disadvantage of larger vocabularies. We have included the vocabulary-size comparison and the parameter-allocation caveat in the revised manuscript (lines 1759–1771).
1B/400B-token vocabulary-size comparison; pass@1 (%). Higher bytes/token indicates stronger compression.
| Tokeniser | Bytes/token | MultiPL-E Avg. | HE+ | MBPP | MBPP+ | ClassEval |
|---|---|---|---|---|---|---|
| SyntaxBPE 30K | 3.478 | 12.4 | 14.6 | 24.6 | 28.0 | 29.5 |
| SyntaxBPE 128K | 3.810 | 10.8 | 13.4 | 20.0 | 24.1 | 28.3 |
Thanks for the context
This context gives me confidence to move the rating to 4. Please add appendices to your paper with these results.
Thank You for Your Encouraging Feedback
Thank you very much for your positive feedback and for reconsidering your rating. We sincerely appreciate your constructive suggestions and will incorporate these additional results into our revised paper.
Response to Reviewer GwYy (Part 1/2)
We thank the reviewer for the careful reading and for recognising the deployable design and the calibration procedure. Below, we address attribution to calibration, run-to-run variation, and the requested clarifications.
1. Attribution to calibration and a hand-written code baseline
We thank the reviewer for proposing this direct comparison. We have added a hand-written code regex paired with standard BPE (without structure weighting), with fused identifiers (including camelCase, underscores, and identifier-internal digits), standalone delimiters, and an explicit operator whitelist. It retains o200k-style 1–3-digit numeric groups and whitespace handling, and explicitly specifies quote/marker runs and newline attachment. This provides a stronger test of whether manually chosen code boundaries account for the gains. We have included the comparison and construction details in the revised manuscript (lines 379–391 and 1305–1331).
1B/400B-token comparison; pass@1 (%).
The baseline follows the reviewer's specification: o200k with fused identifiers, standalone delimiters, and a whitelist of standard multi-character operators. We wrote it after completing our calibration analysis, so it is not blind to those findings; if anything, this favours the baseline. It is comparable to or slightly better than GPT-5-style BPE (−0.2 to +0.9 percentage points), confirming that a better code regex accounts for only a small part of the gain. SyntaxBPE still gains a further 0.9–3.2 percentage points on every metric, while the fixed-inventory ablation directly tests the benefit of calibration-based selection.
The hand-written regex achieves 3.394 bytes/token compared with 3.478 bytes/token for SyntaxBPE.
For reproducibility, we have included the complete hand-written pre-tokenisation regex in the revised manuscript (lines 1313–1331):
2. Training-seed variation and Stage I/II ablations
We thank the reviewer for this suggestion. Despite the pretraining cost, we have added a second training seed for each of the four configurations below and reported the two-run results in the revised manuscript (lines 1713–1727).
1B/300B-token results; pass@1 (%), reported as mean ± standard deviation across two independent training runs.
SyntaxBPE (γ=0) leads across all five metrics, exceeding the reported seed variation and supporting calibration and structure weighting. Small aggregate tokeniser changes can thus accompany meaningful downstream gains.