acceptodds
Under review as a conference paper at ICLR 2027

Local Splits, Global Shifts: An Interaction Decomposition of Retokenization in Language Models

Abstract

Different tokenizations of identical bytes can change a language model's predictions. We introduce the first exact local-to-global framework for attributing same-byte retokenization sensitivity to contiguous spans and their interactions. Universal cuts split the tokenization space into the finest independent contiguous blocks. Summing next-token probabilities by first byte makes predictions comparable across tokenizations. Functional analysis of variance separates first-order effects of single blocks from multi-block interactions. Resampling a single block measures its local influence, whose sum upper bounds global variation. Their ratio, the interaction order, is one when effects add and exceeds one when blocks interact. An exhaustive binary-block analysis finds mostly first-order effects with substantial interactions. Across prose, code, and multilingual text, local influence predicts global variation beyond severity and other controls, and prompt-pooled interaction order rises with severity. Independent interventions show that local influence ranks fragile spans whose retokenization is more likely to induce task failures. We derive Local Retokenization Consistency (LRC), a symmetric one-block objective that penalizes local influence while preserving the expected loss of stochastic supervised fine-tuning. In matched-step adaptation of an instruction-tuned model, LRC achieves the highest mean canonical and retokenized accuracy, averaged equally over tasks, and the smallest mean gap among the compared objectives.

Primary area:
foundation or frontier models, including LLMs
Keywords:
retokenization, sensitivity attribution, tokenization robustness

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_LXfROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The critical question being addressed is as follows. When a prompt is retokenized, the output changes from an LLM. Here, does the change in output come from separate spans, each adding its own effect? Or do spans interact, so that how one span is split matters more depending on how its neighbors are split? Different models that have the same total sensitivity could have completely different organization. So how should the measurement be?

The authors go on to give a very nice byte-offset DAG formalism to the problem, and provide clean explanations. The experiments also validate the approach, albeit a few caveats mentioned in the weakeness section.

Critical strengths

The formalism is nice.

The problem statement and experiments are useful.

Critical weaknesses

WikiText is taken from the train split, and HumanEval continuations are the canonical solutions. OLMo 2 and Qwen3 pretraining data most likely contain these. Perhaps, this contamination could shape how sensitive the models look?

In ANOVA, an interaction among k blocks gets weight roughly proportional to (π(1−π))^k. So κ grows with π for almost any function that has some interaction terms. When I ran the authors’ code through a fixed synthetic function with main effects plus small pairwise terms., with no LLM, the pooled κ still rose from 1.015 to 1.029, 1.050, 1.070 and 1.074 as π went from 0.05 to 0.50. So the robustness of the test seems to be compromised.

Questions for authors

The human eval numbers don't seem right. I think 8% for Qwen3 8B Instruct is a bit lower than what we usually see. Could you please cross-verify?

The LRC margins seem to be thin. So do we really have an effect that is worth reporting?

4 comments
AuthorsOpenReview ↗

Response to Reviewer LXfR

Thank you for the careful assessment and for testing our code with a synthetic function. Your test was very helpful, so we begin with it.

1. Interaction order and severity (W2).

Your synthetic example is correct. It shows that κx\kappa_x depends on the product distribution μx\mu_x as well as on the response FxF_x (Section 4). Changing π\pi can therefore raise κx\kappa_x even when the response is fixed, so the rise with severity does not by itself show stronger interactions. We now present the severity curves as measurements under the declared sampler, and we have revised the abstract, contributions, Section 6.3, conclusion and synthetic control discussion accordingly.

The exact identities still hold under every product distribution, and our interaction findings use a fixed distribution for each comparison. Two results illustrate this. First, the exact analysis uses eight binary blocks in 32 prompts at π=0.5\pi=0.5, with other blocks fixed. It assigns 13% to 25% of the variation to interactions, including 3% to 8% at order three or higher (Table 9). At this setting, κ\kappa is 1.176 and 1.322 for OLMo 2 Base and Instruct, compared with 1.360 and 1.366 for Qwen3. Second, in the full product analysis, the two checkpoints within a family share each prompt's sampler at every severity. The OLMo 2 instruct minus base contrast ranges from 0.265 to 0.448, while the Qwen3 contrast stays small (Figure 5). Because these comparisons keep μx\mu_x fixed, they reflect differences between the responses.

2. Possible pretraining overlap (W1).

This is a fair concern. Every measurement compares tokenizations of the same bytes within one prompt, so the content is held fixed, and the exact identities hold for any text. The empirical relationships are also consistent across the evaluated domains. Adding local influence to controls that include canonical confidence raises out of fold R2R^2 from 0.562 to 0.789 (Table 13). The association between local influence and global variation is positive in all 122 reachable model, severity and domain cells across prose, code and six languages (Table 12). In addition, base and instruction checkpoints are compared on identical prompt bytes with matched samplers.

For HumanEval in particular, the reference solutions serve only as teacher forced continuations for the sensitivity measurements. Task generation sees the problem prompt alone, and the adapters are trained on Tulu 3 SFT records (Section 6.1). Familiarity may still affect the magnitude of the measured sensitivity. We have therefore clarified the provenance of each corpus and the scope of the empirical findings.

3. HumanEval accuracy (Q1).

Thank you for flagging this. We checked both the evaluation protocol and the saved completions. The reported score comes from the raw completion interface used in the task evaluations. This interface feeds each prompt as raw token IDs so that canonical and retokenized inputs differ only in tokenization. The 7.89% figure is the three seed mean of the canonical SFT adapter on the 131 test prompts (Table 22). For comparison, the unadapted Qwen3 8B Instruct checkpoint scores 9.76% on all 164 prompts under the same interface (Table 18).

Two checks support this interpretation. First, we rescored the saved LRC seed 0 completions with a scorer that preserves Python indentation and the prompt preamble. Every outcome is unchanged, with 12.21% on 131 canonical prompts and 19.08% on 262 retokenized views. Second, we reran the unadapted checkpoint on all 164 prompts with greedy decoding and a 512 token limit, using the same indentation preserving scorer. It scores 10.37% under raw completion, which is within one problem of Table 18. With the chat template and its stopping conventions, it scores 85.98%.

All methods, seeds and views in Table 2 share the raw completion interface, so they are compared under one protocol. We now document this interface next to the scores and have corrected the description of the thinking setting for raw token ID generation.

4. Value of the LRC gains (Q2).

We believe the effect is worth reporting, for the following reasons. In the submitted comparison, LRC improves canonical and random task macro accuracy by 2.73 and 2.60 points over canonical SFT, with positive gains in all three seeds for both accuracies. Against canonical anchoring, the strongest baseline, the gains are 1.04 and 0.61 points. For the canonical gain, a paired bootstrap over the 1,251 test prompts gives a 95% interval of [0.38, 1.85]. LRC also has the highest mean canonical accuracy on each task and the lowest harmful flip rate, 5.86% (Tables 22 & 23).

Beyond accuracy, LRC is derived from the same local influence used for diagnosis and adds no inference procedure. The paper's main contributions are the decomposition and the diagnosis, and we report the LRC gains at their measured size.

Reviewer_LXfROpenReview ↗

Response

Thank you to the authors for the response. It seems the paper you uploaded is a different one, so I am unable to check the changes.

Thank you for pointing this out, and we sincerely apologize for the oversight. We have now uploaded the correct revision. Please let us know if you have any trouble accessing it or if anything remains unclear. We appreciate your time and feedback.

Authors

Thanks for updating the paper. I maintain my original rating.

Reviewer_9bK9OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper focuses on same-byte retokenization sensitivity by proposing a precise local-to-global decomposition framework based on universal cuts and functional ANOVA and derives the LRC training objective. The paper provides large-scale theoretical derivations and experimental validation to demonstrate that local influence strongly predicts global variation, that interaction order rises monotonically with severity, and that local influence can rank fragile spans whose retokenization is more likely to cause task failures. However, the theoretical derivations are overly convoluted and the paper contains grammatical errors in several places. Also, the core statistical tools like functional ANOVA are adaptations of mature methods rather than new theory. Overall, this paper is a solid, complete analysis paper with clear conceptual and diagnostic value, but some details such as grammatical errors need to be corrected. What’s more, the paper could move some complex derivations to the appendix and explore whether the tools employed can be extended to handle dependent blocks or to guide tokenizer design.

Critical strengths
  1. Conceptually clean and theoretically rigorous. The paper introduces universal cuts to uniquely decompose the same-byte tokenization space into the finest independent contiguous blocks, and uses functional ANOVA to exactly decompose global prediction variation into first-order effects and interactions. The derivations are rigorous, advancing same-byte retokenization sensitivity from a coarse robustness score to span-level attribution.
  2. Unification of diagnosis and training. The same local influence used for diagnosis is also turned into the LRC training objective. The unification of the variables used for diagnosis and training is highly intuitive and effective.
Critical weaknesses
  1. The generalization evidence for LRC remains narrow. LRC is validated only on a limited set of experiments. Its generalization needs further validation.
  2. The paper is highly theoretical, and it does not clearly specify its intended application domains. Although it hints at span-level diagnosis and training, the authors could further discuss where the proposed framework and LRC objective could be applied in practice.
Questions for authors
  1. LRC is validated on only one Qwen3 checkpoint, which the authors acknowledge as a limitation. Do the authors plan to evaluate LRC on additional model families (e.g., OLMo 2) in future work? If so, what challenges do you anticipate?
  2. The paper is highly theoretical. Could the authors more clearly discuss the practical applications of this research in real-world settings, such as multilingual deployment, code generation, or high-stakes domain adaptation?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 9bK9

Thank you for recognizing the conceptual value of the framework and the connection between diagnosis and training. We respond to your two concerns first and then to your comments on presentation.

1. Generalization of LRC (W1, Q1).

The revised paper now states the scope of the diagnostic and adaptation studies explicitly. The diagnostic study covers four checkpoints from two model families across prose, code and six languages. The LRC study adapts Qwen3 8B Instruct and evaluates math, code and reading comprehension over three training seeds. In this comparison, LRC achieves the highest mean canonical and random task macro accuracy among the submitted objectives, improving over canonical SFT by 2.73 and 2.60 points.

To extend the adaptation evidence during the rebuttal, we also evaluated the submitted seed 0 adapters at a second severity, ρ=1.50\rho=1.50. They reach 46.94% random accuracy for LRC and 46.28% for canonical anchoring, the strongest baseline.

To answer your question directly, yes, evaluating LRC on OLMo 2 is our next planned step. The diagnostic experiments already provide its byte mapping MM, block sampler and collision weights, and the objective requires no change to the model architecture. We anticipate three practical challenges: following the model's prompt and stopping conventions, keeping both tokenized branches within its context window, and selecting β\beta on its own development data. In addition, tokenizer differences can change which training examples meet the length limit.

2. Practical applications (W2, Q2).

We have added a concise applications paragraph to the introduction. It is organized around the following three uses.

Robustness testing for multilingual and code prompts. Local influence Λx,i\Lambda_{x,i} identifies the spans whose tokenization changes predictions, while κx\kappa_x describes how these effects combine across spans. Together, they support targeted testing of code, mathematical notation and multilingual prompts. Table 8 shows that block structure and attainable token expansion differ across languages and tokenizers, so matching expected token expansion gives a common severity scale across languages.

Interfaces with noncanonical token histories. Some interfaces expose inputs with identical bytes and different token boundaries (Appendix I). Examples are token ID interfaces, generated token histories whose text maps back to a different tokenization, and conditioning across tokenizers. The framework measures sensitivity in these settings, and its prompt endpoint value is available before generation without a reference continuation.

Domain adaptation. LRC runs two branches during SFT, preserves the expected stochastic SFT task loss and adds consistency between tokenizations that differ in one block. It needs no additional inference procedure. Our evaluation covers math, code and reading comprehension, and the same recipe applies to a specialized domain using that domain's SFT data and tasks.

3. Presentation and contribution.

We agree that functional ANOVA and total effect indices are established tools, as Section 2 states. Applying them requires independent inputs and comparable outputs, and our contribution is to provide both for retokenization. For the inputs, universal cuts give the unique finest contiguous blocks whose tokenizations can be chosen independently (Proposition 1), and the product sampler draws these choices independently. For the outputs, the aligned behavior map compares predictions in a shared next byte space. Together, these give the exact decomposition in Theorem 1, from which we derive the collision stratified estimators and the LRC objective.

To make this connection easier to follow, we now lead with the two block example and explain Δx\Delta_x as global variation, Λx,i\Lambda_{x,i} as the local influence of block ii and κx\kappa_x as interaction order before the equations. We have also moved the estimator derivation of Section 4.1 to the appendix and corrected the grammatical and figure reference errors.

Thank you as well for the two suggested extensions. Both fit the framework, and we now discuss them. For dependent blocks, universal cuts already guarantee that any joint choice of block tokenizations is valid. The exact ANOVA identities, however, use the product distribution, so a sampler that couples neighboring blocks calls for a decomposition for dependent inputs. For tokenizer design, the diagnostics can compare candidate tokenizers with their corresponding models, with the tokenization space and its blocks recomputed for each vocabulary.

Thank you for the clarification. The revised discussion addresses my question about practical applications.

We appreciate your positive feedback and are glad that the discussion addresses your concern about practical applications.

Authors

Reviewer_EPomOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper studies how different tokenizations of identical bytes can affect the prediction of language models. The same text can be split into tokens in many valid ways (for instance, "1200" as either [1200] or [12][00]), and different retokenizations can change the model’s predictions. For this purpose, the authors proposed a local-to-global framework that attributes same-byte retokenization sensitivity to contiguous spans and their interactions. They first apply universal cuts to partition the input text into the finest independent contiguous blocks and then align predictions through next-byte distributions, enabling controlled span-level comparisons, and apply functional ANOVA to separate individual-block effects (first-order effects) from multi-block interactions. Their exhaustive binary-block analysis across prose, code, and six languages finds that most sensitivity comes from individual chunks (~75–87%), with a meaningful share from interactions. Summed local influence predicts overall sensitivity and helps identify fragile spans whose retokenization is more likely to cause task errors. Finally, the authors propose a fine-tuning objective, Local Retokenization Consistency (LRC), which matches predictions for same-byte tokenizations differing in one block. In their tests, it gave the highest average accuracy on both canonical and re-tokenized inputs among the compared objectives.

Critical strengths

Generally, I think it is a good paper on how to make byte-level retokenization effects more interpretable and to improve the training objectives to make models more robust to different same-byte tokenizations.

Framework design: Overall, I think the authors made good design choices in putting together their local-to-global framework. For example, universal cuts ensure that independently selected block tokenizations can be combined into a valid tokenization of the original text, providing the structure needed for the sensitivity decomposition. Aligned next-byte probabilities then allow predictions to be compared across different tokenizations. I also liked how the exact relationships between local influence, global variation, and interaction order make the analysis more interpretable. They distinguish the overall magnitude of sensitivity from whether its effects arise independently or through interactions between spans.

Diagnosis: The validation connects these mathematical quantities to observed model behavior through complementary experiments. The authors exhaustively evaluate all 256 configurations of eight binary blocks on 32 prompts, verifying the decomposition numerically, before extending their measurements to 1,164 prose, code, and multilingual examples across four checkpoints. Shared sampling plans between base and instruction checkpoints and comparisons at matched token expansion strengthen the interpretation of model differences. It is also important to note that the block scoring and intervention evaluation use independent random streams, reducing the risk that localization results only reproduce the samples used to construct the rankings.

Intervention: The framework also offers a training objective through Local Retokenization Consistency (LRC), which turns the sensitivity analysis into a way to improve model behavior. LRC encourages agreement between predictions for two tokenizations differing in one block, while importance weighting preserves the expected stochastic-SFT task loss. I liked this connection between diagnosis and intervention, as it gives the proposed local influence measure a concrete role in training.

Critical weaknesses

Some limitations are already acknowledged by the authors (Appendix A), such as the narrow adaptation setup and the unresolved final-block comparison. They are valid, and addressing them would strengthen the work for sure. The points below are additional:

(1) The results don’t fully convincingly show that LRC works better for the following reasons:

  • I would like to chat further during the rebuttal phase about this. I generally like the idea of proposing the LRC approach as an intervention mechanism, but its advantage over other techniques should be supported with more in-depth analysis.
  • Apart from the gain over canonical SFT, LRC’s advantages over the strongest baseline are about 1 point or less, and mostly within noise. The reported spread reflects only three seeds on a fixed test set. Follow-up: Report paired significance tests over test prompts (bootstrap or McNemar), use more seeds, and evaluate on more models and severities.
  • It is a bit unclear what makes LRC work. LRC changes three things at once compared with the baselines: how training pairs are sampled, how the task loss is weighted, and the added consistency penalty. The paper credits the penalty, but at the chosen weight (λ = 1), the penalty is likely small relative to the task loss, since Hellinger values are on the order of 10^-3, and LRC models are only marginally less sensitive than plain SFT and no less sensitive than canonical anchoring after training (Table 24, seed 0 only). The gains may therefore come from the sampling or weighting rather than the penalty. The comparison may also be unfair: all methods use the same weight even though their penalties measure different quantities on different scales. Follow-up: Add ablations that keep LRC’s sampling and weighting but remove the consistency term, and sweep the penalty weight separately for each method.

(2) Some analysis claims go further than the evidence:

  • The severity trend is partly built in. Interactions only appear when two or more blocks are retokenized at the same time. When few blocks are changed, this rarely happens. When more are changed, it happens often. Interaction order tends to rise with severity for any model with interactions, even if the model itself behaves the same way throughout. The paper’s Figure 4 shows this pattern: interaction order is near 1 when almost no blocks or almost all blocks change, and it peaks in the middle. Follow-up: Estimate how much interaction order should rise from the sampling scheme alone. For example, measure interaction strengths at one severity and use them to predict the other severities. Then report only the part of the observed rise that goes beyond this prediction.
  • Ranking blocks by influence does not clearly beat just picking the last block, and the abstract and main-text Table 3 omit this caveat. Follow-up: Add the final-block and other cheap baselines (e.g., position, token length, rare-token blocks) to the main text, and test whether influence adds information beyond them.
  • Only two model families of similar size are compared, and they behave differently. Their tokenizers, pretraining, and post-training all differ, so the cause can’t be identified. Follow-up: Evaluate OLMo 2’s public intermediate checkpoints (SFT, DPO, RL) to find which stage changes the interaction order, and add more model families and sizes. The Section 6.3 framing suggests general conclusions that two families can’t support.

And just a few typos to fix:

  • Figure 1, middle-panel label: "We tokenization space into independent contiguous blocks" is missing a verb. It should read "We split/factor the tokenization space into…"
  • Section 6.2, around line 391: "Figure 1 (c) shows the interaction shares." Figure 1 has Left/Middle/Right panels, not (a)/(b)/(c). It should be "Figure 1 (right)."
  • Figure 1 caption, last sentence: "The shares in right panel are measured" is missing "the." It should be "The shares in the right panel are measured."
Questions for authors

My questions are generally in line with the weak areas I highlighted in the previous section. I mainly want to know your insights into each:

  • What consistency-coefficient ranges and selection criteria were used separately for LRC, global consistency, and canonical anchoring? How sensitive is their ranking to the different regularizer scales? Does LRC outperform an ablation using the same local pairs and importance-weighted task losses with the consistency coefficient set to zero? This would help isolate the contribution of the proposed regularizer. Does local influence provide additional information about induced failures after accounting for block position and token expansion? How should the inconclusive comparison with final-block selection affect the intended interpretation of the localization results?
  • Severity is matched on average per domain, so individual prompts differ. Do the interaction-order comparisons change if severity is matched per prompt?
  • Does the rise in interaction order with severity go beyond what the sampling scheme alone would produce?
  • What causes the different behavior of OLMo 2 and Qwen3 after instruction tuning? How can you generalize your findings to different post-training phases?
4 comments
AuthorsOpenReview ↗

Response to Reviewer EPom (Part 1 of 2)

Thank you for the careful assessment and the concrete suggestions. We report the requested controls below. This first part covers LRC (W1, Q1), and the second part covers the analysis claims (W2, Q1 to Q4).

1. LRC gains and controls (W1, Q1).

We begin with the size of the gains. In the submitted experiments at ρ=1.25\rho=1.25 over three seeds, LRC improves canonical and random task macro accuracy by 2.73 and 2.60 points over canonical SFT. Against canonical anchoring, the strongest baseline, the gains are 1.04 and 0.61 points. As you suggested, we tested this second comparison with a paired bootstrap over the 1,251 test prompts. It is stratified by task and keeps both views and all seeds of a prompt together. The 95% intervals are [0.38, 1.85] for the canonical gain and [-0.07, 1.32] for the random gain. The canonical gain is also positive in every seed.

You also asked which part of LRC produces the gain. To isolate the penalty's contribution, we ran the ablation you proposed as a fresh control. It keeps the local pairs, importance weights, data order and training settings identical and changes only the consistency weight β\beta:

Mean over three seeds (%)CanonicalRandom
LRC without penalty, β=0\beta=042.8647.17
LRC, β=1\beta=143.0547.33

The control retains nearly all of LRC's accuracy. On average, the penalty adds 0.19 points on canonical inputs and 0.16 points on random inputs. We have added this control to the paper and now report the penalty's increment separately from the full objective.

We also examined the choice of consistency weight. The submission used a shared β=1\beta=1, but the penalties have different scales. LRC targets Λx/m\Lambda_x/m and global consistency targets Δx\Delta_x, with (Λx/m)/Δx=κx/m(\Lambda_x/m)/\Delta_x=\kappa_x/m. We therefore selected β\beta separately for each method from {0.1,1,10}\lbrace 0.1, 1, 10\rbrace on seed 0 development data, maximizing the 50/50 canonical/random task macro accuracy. We then evaluated three seeds on the test split:

MethodSelected β\betaCanonical (%)Random (%)
Canonical anchoring1042.7946.62
Global consistency0.142.7446.65
LRC0.142.8047.08

With this selection, LRC retains the highest mean random accuracy, 0.46 points above anchoring and 0.43 points above global consistency, and it matches both baselines on canonical inputs. Please note that these fresh runs use one GPU with the same effective batch and a different data order from the submission, so we compare methods within each table.

Finally, to check a second severity, we evaluated the submitted adapters at ρ=1.50\rho=1.50. They reach 46.94% random accuracy for LRC and 46.28% for anchoring. LRC thus has the highest random accuracy in the submitted comparison and after coefficient selection, and it stays above anchoring at the second severity.

AuthorsOpenReview ↗

Response to Reviewer EPom (Part 2 of 2)

This second part addresses the analysis claims (W2, Q1 to Q4).

2. Severity and interaction order (W2a, Q3).

We agree with this point. The interaction order κx\kappa_x depends on the product distribution μx\mu_x as well as on the response FxF_x (Section 4), so changing π\pi can raise κx\kappa_x even when FxF_x is fixed. For Q3, this means that the rise under the declared sampler does not by itself show stronger interactions at higher severity. The exact binary slice illustrates this, because reweighting the same 256 evaluations determines κ\kappa at every π\pi by construction. We have revised the abstract, contributions, Section 6.3 and conclusion accordingly, and we now present the synthetic controls as checks of the identities.

Our interaction findings instead use a fixed product distribution for each comparison. At π=0.5\pi=0.5 on the exact binary slice, interactions carry 13% to 25% of the variation, including 3% to 8% at order three or higher (Table 9). Figure 4 compares checkpoint ordering at each π\pi.

3. Localization beyond the final block (W2b, Q1).

Local influence adds information beyond block position, and we tested this in two ways. First, we removed the final block from the frozen candidate sets. Among the remaining blocks, the one with the highest influence gives a divergence of 6.48 (H2×103H^2\times10^3) and a failure rate of 14.89%, compared with 2.76 and 8.66% for a uniformly chosen block. The 95% intervals are [1.00, 7.21] for the divergence difference and [0.87, 11.65] percentage points for the failure difference. Second, we added influence to controls for relative position, a final block indicator, canonical block length and expected expansion. This raises grouped out of fold R2R^2 for divergence from 0.140 to 0.330. Failure average precision changes from 0.209 to 0.215, and the interval for this increment includes zero.

Regarding the direct comparison with the final block, influence ranks the final block first among the 12 candidates in 70 of 144 groups (48.6%). The two selections therefore often coincide, which helps explain why their direct comparison in Table 17 is close. We have added the final block baseline and these controls to Table 3, and the abstract now states that influence identifies fragile spans even when the final block is excluded.

4. Calibration and checkpoint scope (W2c, Q2, Q4).

For Q2, matching severity per prompt preserves the checkpoint pattern. On the high budget subset of 128 prompts, calibrating each prompt to ρx=1.25\rho_x=1.25 gives an instruct minus base contrast of Δκ=+0.448\Delta\kappa=+0.448 for OLMo 2 and −0.058-0.058 for Qwen3. Under domain calibration on the same prompts, the corresponding contrasts are +0.500+0.500 and −0.105-0.105.

For Q4, the two checkpoints within a family share the tokenizer, blocks and sampler, so each contrast reflects the checkpoint difference. Between families, the tokenizer, pretraining and instruction tuning all differ. Section 6.3 therefore now presents checkpoint contrasts within each family, and we have retitled its paragraph. Measuring intermediate OLMo 2 stages and further families is our next step.

Finally, we have fixed the three typos. Thank you for pointing them out.

Thank you for the thorough response and the additional experiments. My major concerns are addressed. Overall, I think the diagnostic framework is a useful contribution, and the clarifications and additional analyses in your response bring the claims into better alignment with the evidence.

The new ablation shows that the consistency penalty itself adds only about 0.2 points, with the no-penalty control retaining nearly all of LRC’s accuracy. After per-method coefficient selection, LRC matches the baselines on canonical inputs and leads by under half a point on random inputs. The paper now reports the penalty’s increment separately. I’d also encourage framing the contribution as the local-pair training scheme as a whole rather than the regularizer alone. Similarly, it would help to note that local influence adds substantial information about divergence beyond the cheap controls, but little detectable information about task failures beyond them.

I look forward to seeing these changes reflected in the final version. I have raised my Soundness score accordingly, and I am keeping my overall rating. Thanks again for the engagement.

Thank you for the thoughtful follow-up and for recognizing the contribution of the diagnostic framework.

We agree with both the framing suggestions and will make them explicit in the final version. We will present LRC as the complete training scheme, including local pair sampling, importance weighting, and the consistency penalty, with the penalty’s increment reported separately. We will also distinguish the substantial improvement in divergence prediction beyond the controls from the limited detectable gain in task failure prediction beyond the same controls.

Thank you again for the constructive discussion.

Authors

Comments on the paper

3 comments
AuthorsOpenReview ↗

Response Summary

We thank the AC and reviewers for their careful assessment and constructive feedback, and for recognizing the framework's conceptual clarity, diagnostic value and connection to training. We completed additional experiments, updated the revised manuscript and clarify three central points below.

An exact framework for retokenization sensitivity

Our framework is the first to attribute LLM retokenization sensitivity exactly to individual spans and their interactions using functional ANOVA. Universal cuts construct the independent inputs it requires, and next byte predictions make outputs comparable. The resulting identities show that summed local influence bounds global variation and lead directly to LRC, which turn the diagnostic quantity into a consistency objective with no additional inference procedure.

How should the interaction findings be interpreted?

In our exact analysis under a fixed sampling distribution, most variation comes from individual spans, and interactions account for 13% to 25%. Across prose, code, and multilingual text, local influence predicts global variation. The revised paper clarifies that interaction order also depends on the sampling distribution, and matched samplers support checkpoint comparisons within each family.

Diagnosis and training

The new controls show that local influence identifies fragile spans even when the final block is excluded (task failure rate of 14.89% v.s. 8.66% for a random block) and improves divergence prediction after accounting for position, block length, and token expansion. We have added these results and the direct final block comparison to Table 3.


Before the discussion period ends, we remain available and welcome any remaining questions.

Dear reviewers,

Please check your assigned papers for author responses. If the authors have addressed any of your questions or concerns, please leave a brief comment for them indicating which points have been resolved and which, if any, remain open. Even if your assessment is unchanged, a short follow-up is helpful.

Best, AC

Dear Reviewers,

The Author-Reviewer Discussion period is coming to an end soon. Please actively participate in the rebuttal discussion with the authors after carefully reading the authors' responses.

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.