- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
I'll try to explain the article by explaining what I understand as the most natural real world application. The article is a theoretical one which mentions these applications in passing.
Suppose we have some ML model working on tokens, and we wish to group some tokens together into a cluster of similar tokens in terms of what kind of outcome they produce. We might think of image models giving two patches of pixels which are nearly identical, or similar problems for audio codebooks. Since not every possible patch of pixels makes it into the codebook, we have problems where some patches of pixels have many tokens which could reasonably faithfully represent them, whereas other patches of pixels are only well represented by one token. In particular, a cluster of total model probability p may be represented by a single token of probability p or by many tokens of probability p/n. This property interacts very badly with truncation sampling (such as top-p), where tokens are kept or discarded based on their individual probability, not the probability of the cluster to which they belong.
This problem has been noted, and a potential fix suggested, in prior work. The present work is concerned with theoretical bounds on how bad the phenomenon can be of distorting the mass of sequences of clusters of tokens, even in the case that the sequence of clusters of tokens survives truncation.
Critical strengths
The theory is well explained and the experiments are competently done, I have no issues at all with the proofs/ experimental execution.
Critical weaknesses
I am used to reading both mathematics and machine learning, and so should be in the target audience for this paper, but I find the first two pages almost impenetrable. I did not understand from the abstract or first page that, on a basic level, this article is about splitting the mass of one token into many subtokens, and how this distorts probability mass.
Furthermore, I don't get from reading the article a sense of how practically important the work is. This is, I think, largely a choice of the authors to present their work as being about abstract clusterings of tokens, rather than about a specific type of model. But this presentation assumes that the reader is already familiar with how the basic question affects real world models and is convinced that this question needs solving.
I think I find the two example experiments rather artificial, particularly the first experiment which looks at taking a language model and randomly splitting each token into eight alternative tokens with the same total mass as the original token but the proportion in each alternative token being randomly assigned. I am perfectly convinced that this procedure interacts badly with top-p sampling, but not really convinced that this is a problem.
Questions for authors
I'm afraid I don't really have any questions about your work that would cause me to increase my score. I think there is very substantial value in purely theoretical work, and artificial experiments can be of value, I think some version of your work could be useful. But for a general audience conference like ICLR I think you need to do a lot more to explain (gently) to your reader the problem you are solving and how large it really is.
4 comments
Practical scale and the purpose of the controlled intervention
Weakness 2: how practically important and how large is the effect? Thank you for asking us to distinguish mathematical possibility from natural scale. We added a terminal-action evaluation: legal one-token answer renderings are mapped to the same selected option, and generation stops. It covers all 1,172 ARC-Challenge and 14,042 MMLU questions separately for Qwen3-8B, Qwen2.5-0.5B-Instruct, and OLMo-3-7B-Instruct, at temperature 0.7 and top-p 0.9. Across six cells, mean one-step TV ranges from to . Fewer than 1.63% of questions per cell change support and contribute over 99.99% of each mean; same-support means are below . Section 5.4 (p. 9) and Table 3 (p. 10) report the scale, settings, and support decomposition.
This is a useful practical result: it identifies a concrete natural terminal-action setting with low observed representation sensitivity, while the controlled experiment separately shows that large same-support sensitivity is possible. The result therefore resolves the scale question without conflating typical natural magnitude with worst-case theory. Your image/audio analogy motivates grouping, but approximate similarity alone does not establish identical continuation states.
Weakness 3: why use an artificial random split? Thank you for separating a controlled mechanism test from evidence of prevalence. Its purpose is to hold the model, tokenizer, and untruncated kernel fixed at each shared history, isolating decoder order. Random split imbalance is unnecessary: deterministic uniform eight-way splitting gives mean parent-path TV 0.256 [0.245, 0.267] at 64 tokens over 120 Qwen3-8B families, with support preserved in exact arithmetic and zero recorded one-sided contribution. This experiment was already in the submission; Figure 3 (p. 7) retains its result. The additional natural-action and task studies answer different practical questions rather than relabeling this controlled result as new evidence.
Weakness 2, direct task consequences. We appreciate your practical-impact concern and added the full-task evaluation also requested by 9EtP.
| Policy | ARC-Challenge accuracy | GSM8K accuracy |
|---|---|---|
| Parent-first top-p | 40.27% | 37.00% |
| Quotient-first | 40.27% | 37.00% |
| Uniform refinement | 40.78% | 38.21% |
| Dirichlet refinement | 40.36% | 37.68% |
| Greedy | 43.43% | 43.29% |
The refinement point estimates are slightly higher than parent-first on both tasks. Uniform-minus-parent is +0.51 percentage points on ARC [−0.26, 1.37] and +1.21 on GSM8K [−0.83, 3.26]; neither refinement differs significantly from parent-first after Holm adjustment under the single-seed paired protocol. Thus the experiment directly measures correctness and does not reveal a distinguishable accuracy penalty or gain, rather than interpreting path separation as an accuracy effect. Section 5.3 and Table 2 (p. 9) report these results. Together with the opening changes, these additions address your request for a clear explanation of the problem and its measured scale. We would be grateful for reconsideration of that clarified contribution. Thanks.
Many thanks to the authors for their work. I don't think the new results on task consequences support the case for viewing Quotient first decoding as an important real world solution. I don't think they work against it either, I'm not quite convinced it's the right experiment. Nevertheless, my conclusion remains substantially unchanged from my original review, I repeat my conclusion from before below -
I'm afraid I don't really have any questions about your work that would cause me to increase my score. I think there is very substantial value in purely theoretical work, and artificial experiments can be of value, I think some version of your work could be useful. But for a general audience conference like ICLR I think you need to do a lot more to explain (gently) to your reader the problem you are solving and how large it really is.
I would add that, while this is opinion, you haven't convinced me of the true utility of this work.
Additional measurements of repeated decoding, and the problem and usefulness they help clarify
Thank you for your thoughtful follow-up. The additional three-model study beyond our first two replies directly measures how sensitivity changes when the operation is applied once or repeatedly throughout generation. These controlled measurements make the design implications more concrete, alongside the natural-interface results discussed below. We included the task-accuracy evaluation requested by 9EtP because we thought it might connect the distributional analysis to practical consequences. Your follow-up helps us focus on the problem the theory addresses and the decisions it helps a decoder designer make.
1. The problem in plain language. Consider an application in which several accepted labels select the same terminal action. Their combined probability is the model's probability for that action. Dividing the same mass among more labels changes its representation but not the underlying action probability. Nevertheless, top-p applied to individual labels can change how often the action is selected. If the application specifies top-p on actions, the implementations can disagree even when they permit exactly the same actions. The terminal setting makes the grouping exact without assuming equivalent future model states for different tokens.
The revised opening illustrates this with probabilities and threshold . Direct truncation gives retained action probabilities ; three equal labels per action change them to . The first action becomes 8.3 percentage points more likely. Theorem 3.1 proves that is the exact largest same-support TV discrepancy for those original masses.
2. What the new controlled experiment adds about magnitude. We varied the frequency of the representation-sensitive operation using uniform eight-way refinement, temperature 0.7, top-p 0.9, 60 fixed prompt families per model (20 each from ARC-Challenge, GSM8K, and MMLU), four generation seeds, and both sampling lanes. Mean parent-path TV at 64 tokens is:
| Model | First step only | Every fourth step | After step 32 | Every step |
|---|---|---|---|---|
| Qwen3-8B | 0.0095 | 0.102 | 0.152 | 0.249 |
| Qwen2.5-0.5B-Instruct | 0.0378 | 0.129 | 0.154 | 0.265 |
| Olmo 3 7B Instruct | 0.0155 | 0.128 | 0.170 | 0.289 |
In all twelve settings, the measured discrepancy comes from reweighting shared paths; the recorded one-sided-support contribution is zero. For Qwen3, the first-step-only and every-step 95% intervals are [0.0049, 0.0148] and [0.230, 0.269]. Appendix J.1 and Table 14 report the study. The result shows that initial sensitivity can substantially understate the effect of repeated decoder decisions. Model weights and the untruncated kernel at shared histories remain fixed. This controlled comparison measures the effect under each complete schedule. It shows why assessing the full sequence gives a more informative picture of sensitivity than checking the first decision alone.
For comparison, the earlier natural-scale evidence remains separately identified. The terminal-answer study covers all 1,172 ARC-Challenge and 14,042 MMLU questions for each of the same three models, at temperature 0.7 and top-p 0.9. Conditional on legal answer tokens, mean one-step TV is 0.000057–0.0013. Rare support-changing questions contribute over 99.99% of each mean; same-support means are below . This identifies a natural interface with low sensitivity. Section 5.4 and Table 3 report the finding. One-step action TV and 64-token path TV concern different outcomes and horizons.
3. What useful capability the theory provides. Proposition 2.1 tests whether an implementation preserves the specified outcome policy through its retained fractions. Theorem 3.1 characterizes every attainable same-support law at the original distribution and its exact largest discrepancy. A designer can certify a tolerance without searching through label encodings. When the actual implementation needs correction, aggregation before truncation supplies an exact correction. Section 4 makes these decisions explicit. Together, the theory and new study show how to check a decoder, assess its possible error, and correct it when needed.
Thank you again for reviewing our work. We hope the explanation and additional measurements clarify the problem, its measured scale, and the verification capability provided by the theory. We would appreciate it if you could revisit our responses and reconsider your evaluation in light of the clarified contribution and new evidence. We appreciate your consideration.
The closing explanation request: Splitting before abstraction
Weakness 1: the first pages do not explain the basic operation. Thank you for your constructive feedback. We agree that the opening introduced abstraction before explaining mass splitting. To address this, we have rewritten the abstract and added a short numerical example as the first paragraph of the introduction, before quotient notation. Both are on p. 1.
Weakness 1, the example now used in the opening. We appreciate your request for a direct explanation. Suppose three outcomes have probabilities (0.6,0.3,0.1) and the nucleus threshold is 0.8. Truncating outcomes directly keeps the first two and gives (2/3,1/3,0). Now give each outcome three equal-probability labels. Label-level truncation keeps all three labels of the first outcome but only two of the second. After merging labels, the law is (3/4,1/4,0). The first outcome's probability rises from 66.7% to 75%, an 8.3-percentage-point change, although the same two outcomes remain possible.
Your closing request to explain gently what the analysis contributes. Thank you for asking us to connect the example to its purpose. We have added a practical-design paragraph in Section 4 (p. 6): compare the retained fractions to test invariance, then compute the largest possible same-support distortion from the original nucleus. Theorem 3.1 determines every attainable same-support law and its exact largest error, so the analysis replaces an isolated example with a complete sensitivity calculation. When the application requires preservation, aggregating before truncation enforces its chosen outcome law.
Your closing request for an accessible presentation of the mechanism. We appreciate the distinction between sound mathematics and an accessible introduction. We have also moved the existing uniform-refinement illustration beside its finite-multiplicity result, now Figure 2 (p. 5). It shows how the boundary outcome is retained only in part and how the finite-refinement error approaches the sharp bound. This is an explanatory relocation of an existing analytic figure, not a new experiment. The rewritten abstract, new opening example, relocated illustration, and explicit design paragraph answer your presentation concern while preserving the technical development. Our second comment addresses Weaknesses 2 and 3 in their original order.