- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The critical question being addressed is as follows. When a prompt is retokenized, the output changes from an LLM. Here, does the change in output come from separate spans, each adding its own effect? Or do spans interact, so that how one span is split matters more depending on how its neighbors are split? Different models that have the same total sensitivity could have completely different organization. So how should the measurement be?
The authors go on to give a very nice byte-offset DAG formalism to the problem, and provide clean explanations. The experiments also validate the approach, albeit a few caveats mentioned in the weakeness section.
Critical strengths
The formalism is nice.
The problem statement and experiments are useful.
Critical weaknesses
WikiText is taken from the train split, and HumanEval continuations are the canonical solutions. OLMo 2 and Qwen3 pretraining data most likely contain these. Perhaps, this contamination could shape how sensitive the models look?
In ANOVA, an interaction among k blocks gets weight roughly proportional to (π(1−π))^k. So κ grows with π for almost any function that has some interaction terms. When I ran the authors’ code through a fixed synthetic function with main effects plus small pairwise terms., with no LLM, the pooled κ still rose from 1.015 to 1.029, 1.050, 1.070 and 1.074 as π went from 0.05 to 0.50. So the robustness of the test seems to be compromised.
Questions for authors
The human eval numbers don't seem right. I think 8% for Qwen3 8B Instruct is a bit lower than what we usually see. Could you please cross-verify?
The LRC margins seem to be thin. So do we really have an effect that is worth reporting?
4 comments
Response
Thank you to the authors for the response. It seems the paper you uploaded is a different one, so I am unable to check the changes.
Thank you for pointing this out, and we sincerely apologize for the oversight. We have now uploaded the correct revision. Please let us know if you have any trouble accessing it or if anything remains unclear. We appreciate your time and feedback.
Authors
Thanks for updating the paper. I maintain my original rating.
Response to Reviewer LXfR
Thank you for the careful assessment and for testing our code with a synthetic function. Your test was very helpful, so we begin with it.
Your synthetic example is correct. It shows that κx depends on the product distribution μx as well as on the response Fx (Section 4). Changing π can therefore raise κx even when the response is fixed, so the rise with severity does not by itself show stronger interactions. We now present the severity curves as measurements under the declared sampler, and we have revised the abstract, contributions, Section 6.3, conclusion and synthetic control discussion accordingly.
The exact identities still hold under every product distribution, and our interaction findings use a fixed distribution for each comparison. Two results illustrate this. First, the exact analysis uses eight binary blocks in 32 prompts at π=0.5, with other blocks fixed. It assigns 13% to 25% of the variation to interactions, including 3% to 8% at order three or higher (Table 9). At this setting, κ is 1.176 and 1.322 for OLMo 2 Base and Instruct, compared with 1.360 and 1.366 for Qwen3. Second, in the full product analysis, the two checkpoints within a family share each prompt's sampler at every severity. The OLMo 2 instruct minus base contrast ranges from 0.265 to 0.448, while the Qwen3 contrast stays small (Figure 5). Because these comparisons keep μx fixed, they reflect differences between the responses.
This is a fair concern. Every measurement compares tokenizations of the same bytes within one prompt, so the content is held fixed, and the exact identities hold for any text. The empirical relationships are also consistent across the evaluated domains. Adding local influence to controls that include canonical confidence raises out of fold R2 from 0.562 to 0.789 (Table 13). The association between local influence and global variation is positive in all 122 reachable model, severity and domain cells across prose, code and six languages (Table 12). In addition, base and instruction checkpoints are compared on identical prompt bytes with matched samplers.
For HumanEval in particular, the reference solutions serve only as teacher forced continuations for the sensitivity measurements. Task generation sees the problem prompt alone, and the adapters are trained on Tulu 3 SFT records (Section 6.1). Familiarity may still affect the magnitude of the measured sensitivity. We have therefore clarified the provenance of each corpus and the scope of the empirical findings.
Thank you for flagging this. We checked both the evaluation protocol and the saved completions. The reported score comes from the raw completion interface used in the task evaluations. This interface feeds each prompt as raw token IDs so that canonical and retokenized inputs differ only in tokenization. The 7.89% figure is the three seed mean of the canonical SFT adapter on the 131 test prompts (Table 22). For comparison, the unadapted Qwen3 8B Instruct checkpoint scores 9.76% on all 164 prompts under the same interface (Table 18).
Two checks support this interpretation. First, we rescored the saved LRC seed 0 completions with a scorer that preserves Python indentation and the prompt preamble. Every outcome is unchanged, with 12.21% on 131 canonical prompts and 19.08% on 262 retokenized views. Second, we reran the unadapted checkpoint on all 164 prompts with greedy decoding and a 512 token limit, using the same indentation preserving scorer. It scores 10.37% under raw completion, which is within one problem of Table 18. With the chat template and its stopping conventions, it scores 85.98%.
All methods, seeds and views in Table 2 share the raw completion interface, so they are compared under one protocol. We now document this interface next to the scores and have corrected the description of the thinking setting for raw token ID generation.
We believe the effect is worth reporting, for the following reasons. In the submitted comparison, LRC improves canonical and random task macro accuracy by 2.73 and 2.60 points over canonical SFT, with positive gains in all three seeds for both accuracies. Against canonical anchoring, the strongest baseline, the gains are 1.04 and 0.61 points. For the canonical gain, a paired bootstrap over the 1,251 test prompts gives a 95% interval of [0.38, 1.85]. LRC also has the highest mean canonical accuracy on each task and the lowest harmful flip rate, 5.86% (Tables 22 & 23).
Beyond accuracy, LRC is derived from the same local influence used for diagnosis and adds no inference procedure. The paper's main contributions are the decomposition and the diagnosis, and we report the LRC gains at their measured size.