acceptodds
Under review as a conference paper at ICLR 2027

When and Where Can Learned Tokenization Be Refined at Test Time?

Abstract

Learned byte-level tokenizers expose an inference-time degree of freedom that fixed subword tokenizers do not: the model can vary how much of its learned latent resolution is used for an observed prefix. We study two forms of test-time refinement—where to restore boundaries inside a reduced native mesh and when to return a prefix to full native resolution. The experiments reveal a clear granularity asymmetry. Fine-grained boundary placement is unstable: dense insertion hurts and first-layer geometry or attention does not yield a reliable long-horizon location rule. Prefix-level allocation is substantially more predictable. A lightweight router that estimates continuation-likelihood gain per added latent improves over a strong gate-count allocation baseline by 0.00553 and 0.00716 BPB on two disjoint Python test sets at matched latent spend. Repetition and local code-structure features are especially useful for ranking prefixes, even though the corresponding signals are too weak to localize individual gates. This suggests that test-time adaptation of learned tokenization is better posed first as a compute-allocation problem over prefixes, and only second as a boundary-placement problem within a prefix.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Test-time adaptation, Byte-level language models

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_38t4OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies inference-time refinement of the internal latent resolution of byte-level models with learned tokenization. Prefix-level decisions about restoring full native resolution are more reliably effective than fine-grained boundary placement, improving over gate-count-based allocation on Python code at matched latent budgets.

Critical strengths

I find the idea of predicting continuation-likelihood gain per added latent from the observed prefix interesting. Such predictability is not obvious, and the matched-budget comparisons and replication on independent Python test sets support the usefulness of this approach.

Critical weaknesses

The main evaluation relies on restoring positions after artificially thinning the native support. It is unclear how well this setting represents practical use under a limited compute budget, making the practical relevance of the results harder to assess.

Questions for authors

Q1. The paper motivates prefix-level allocation under a latent budget. Could you illustrate this with a simple, possibly idealized use case, explaining how the artificial thinning and per-prefix restoration decisions relate to the compute constraints in that setting?

Q2. How do you interpret the larger observed gains on Python than on JavaScript? Could this reflect the router being trained on Python or differences in the evaluation data?

Reviewer_G2fLOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether learned byte-level tokenization can be refined at inference time by selecting which latent positions to restore and which input prefixes should receive additional computation. The authors find that selecting individual positions using attention or geometric information is unreliable, while a learned prefix-level router improves continuation likelihood at a fixed latent budget, particularly on Python code.

Critical strengths
  1. Prefix-level compute allocation is more reliable than individual boundary selection. The paper provides evidence that allocating additional latent resolution across input prefixes is more effective than selecting individual token boundaries using the evaluated heuristics. The negative results on boundary selection are also informative.

  2. Careful evaluation of the routing strategy. The authors compare their router against a strong count-based allocation baseline at matched latent budgets. Table 2 shows consistent improvements in BPB on two independent Python test sets, supported by repository-level confidence intervals.

Critical weaknesses
  1. Limited downstream evaluation. The main improvements are measured using teacher-forced likelihood over 64 future bytes. While BPB is a standard language-modeling metric, it remains unclear whether the proposed method can reduce computation while preserving downstream performance. Figure 4 includes a constrained-choice evaluation, but the router's improvement over count-only allocation is not statistically significant.

  2. Unclear practical efficiency gains. The paper demonstrates a better quality–latent-budget trade-off, but it is unclear whether this translates into better quality at a given inference latency. Appendix H reports slightly higher throughput for the count-only baseline than for the proposed router. A quality–latency comparison across different budgets is missing, making it difficult to assess the practical efficiency gains.

Questions for authors
  1. Preservation of downstream performance. Could the authors evaluate whether the proposed routing strategy can reduce computation while preserving downstream performance relative to full native resolution, for example on code generation or completion tasks?

  2. Quality–compute trade-off. Could the authors provide a quality–latency Pareto comparison between the proposed router, count-only allocation, and fixed-resolution baselines? Such a comparison would help clarify whether the proposed method offers a practical efficiency advantage beyond the improvements at matched latent budgets.

2 comments

Thank you for your thoughtful review!

Why this question matters. Learned byte models choose which byte positions enter their expensive middle network. Once a model is trained, those positions also define a possible test-time choice: we can retain a reduced subset for a prompt and restore the model’s own native positions when finer resolution is valuable. The central question of ActiveMesh is whether the observed prefix tells us when that restoration is worth its cost.

The distinction between boundaries and requests matters here. Attention-based boundary scores appear plausible, yet they do not reliably identify which individual positions improve future prediction. Across requests, however, the value of returning to native resolution varies substantially. Our prefix router captures part of that variation and improves BPB over count-based allocation on two Python test sets at the same added-latent budget. This suggests a useful role for learned tokenization beyond setting one compression pattern during training: its resolution can be allocated according to the needs of each prompt.

Downstream performance. We agree that better likelihood should be tested against a task decision. For a code-continuation ranking task with two available candidates, we added a simple task-aware rule: score both candidates on the reduced mesh and restore native resolution when their scores are close. On 1,024 choices, it reaches 81.69% accuracy, compared with 81.74% for full native and 80.96% for count allocation at the same added-latent spend. It uses 22.7% of the extra native states available beyond the reduced mesh. The route’s advantage over count is +0.73 percentage points [0.15, 1.32]. Candidate ranking gives the policy a direct signal of uncertainty; extending this result to free-form generation is an important next experiment.

Measured efficiency. We also measured prefill quality and time across 10%, 25%, 50%, and 75% escalation. As the route spends more time, its BPB improves from 2.17891 at 93.99 ms to 2.14788 at 101.85 ms.

Thank you for the additional experiments and clarifications! I believe my main concerns have been sufficiently addressed and will maintain my positive rating.

Reviewer_3jdpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies test-time refinement of learned byte-level tokenization. In byte U-Net models, a learned gate selects which byte positions become latent tokens for the middle network. Starting from a constructed reduced mesh that retains half of the model’s native boundaries, the authors examine two forms of refinement. The first is where to restore a fixed number of omitted native boundaries, using recency, attention mass, or a first-layer attention geometry score. The second is when to return an entire prefix to full native resolution under a population-level latent budget. As a results, location rules are found to be unstable and to fade over a 64-byte horizon. In contrast, the prefix router improves next-64-byte BPB over a gate-count allocation baseline on two Python test sets, with smaller gains on JavaScript and no gain on FineWeb.

Critical strengths

Strengths

  • The research question is clearly framed. Separating where to place boundaries from when to allocate resolution, and analyzing each as a distinct decision, is conceptually simple and easy to follow.
Critical weaknesses

Weaknesses

[W1] Limited scale and questionable generalizability. All main experiments use two 147M byte U-Nets trained on about 6B byte exposures drawn from roughly 1B distinct FineWeb bytes. The 73M models also disagree on the sign of the rate effect across seeds. More importantly, the main positive result is obtained on Python code, which is out-of-domain for these models (code BPB ≈ 2.2 vs. ≈ 1.22 on FineWeb). On in-domain FineWeb, the router shows no gain (−0.00001). In addition, only a single architecture is studied, and other learned tokenizers are not considered.

[W2] Small effect sizes and unclear practical benefits. The main gains (0.0055 and 0.0072 BPB) amount to less than 0.3% relative improvement over the count-only baseline (2.24 BPB). They are also much smaller than the remaining gap to full native resolution (0.032 BPB). In terms of efficiency, the router does not outperform the simpler count-only allocation. At the same latent budget, offline count allocation achieves higher throughput. In ordinary mixed batches, the router is only marginally faster than full native resolution due to padding. Since the motivation is a better quality–compute trade-off, the current evidence does not establish a clear practical advantage over simpler baselines.

Questions for authors

[Q1] Does the routing advantage persist when the backbone is trained on code (making Python in-domain) or on mixed data, and at larger model scales? Can the analysis be extended to other learned-tokenization architectures?

[Q2] How does the routing advantage behave for shorter prompts and longer continuation horizons, given that the location results change with both?

[Q3] How sensitive are the conclusions to the fixed design choices, such as the starting mesh ratio, the restoration size and split, and the binary reduced-versus-native action?

[Q4] What properties of the routed prefixes make native resolution more valuable? Could the authors provide an analysis of which prefixes are selected, and why repetition cues predict the gain?

Reviewer_Jj7LOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies byte-level language models (here the goal is to predict the next 64 bytes instead of "next token") with learned tokenization. Basically instead of fixed tokenization, a small network trained with the model groups bytes into tokens. Starting from the model's own tokenization, the authors make a coarser version by dropping about half of the token boundaries, and ask how to spend extra tokens at inference. The paper finds that the granularity of a prompt's tokenization is a knob that can be used to trade compute for quality. In their experiments, choosing whole prompts helps slightly on Python code (but only on python code).

Critical strengths
  1. I found careful experiments on the proposed statements. Test disjoint from train, runs are paired across seeds, bootstrap intervals used, and the gate-count baseline is re-ranked at each run's actual spend.

  2. Negative results are reported. For example, adding boundaries the model would not choose always hurts, and the router gives nothing on web text (Table 2). Both are reported plainly.

Critical weaknesses
  1. Main Concern: The gains are only on python code and on general webtext the gains are non-existent. Is this an accurate reading?

  2. When I read the title and abstract, I expected the model or the tokenizer to be updated at test time, since "refined at test time" and "test-time adaptation" both suggest that. Is inference-time resolution allocation a more accurate framing?

3, The setting is not clear to me. In what practical setting would one drop half of the model's boundaries and then try to win some of them back? (Sec 4.4)

  1. There should be some variation on the 64 bytes. What happens if you predict 8 bytes, 16 bytes, 128 bytes?

  2. I understand compute constraints, but the size of model is 74M-147M The 346M run was exploratory (Section G)? Does this work at larger sizes? Isn't this too small for generative languages?

Questions for authors

Q1. Do the predictions change when you use something other than 64 byte predictions? is this the standard in this domain? If this is varied Q2. How does router architecture matter? Can you do ablations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.