- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper defines a cliff token as a position in a sampled reasoning trace where the Monte Carlo estimate of the success probability drops by more than with and being the estimated standard error of the difference between adjacent success probability estimates. Using one trace per problem for seven open models on GSM1K, MATH500 and AIME 2025, it reports that cliff tokens are more common in incorrect traces, that resampling from before the first cliff gives higher pass@k than resampling from after it, and that cliffs need not coincide with the arithmetic error. Cliff tokens are split into deterministic, uncertain and sampled-off types by whether the token was greedy and by next-token entropy. Single-token DPO pairs built at cliff positions (Cliff-DPO) are trained per type on three models. The uncertain and sampled-off subsets give larger gains than the deterministic subset. The token-level rollout data are released.
Critical strengths
- The released per-token rollout data is the most durable contribution. Success probability curves at token granularity are expensive to produce and are not otherwise available, and the paper's analyses, including the checks requested below, can be re-run from them.
- Several design choices are careful. Cliff-del/Cliff-keep uses fresh rollouts independent of the detection rollouts, bootstrap CIs are given for every panel, and the token-matched ablation and the DPO/cDPO baselines are included in the appendix even where they weaken the headline. The writing is clear.
Critical weaknesses
1. The deterministic cliff counts raise concerns about detector calculation and interpretation.
A deterministic cliff is a greedy token with , so the model emits it with probability . Appendix O measures 99.9%. Rollouts from the preceding prefix use the same sampling settings (Table 4), so they redraw that token with probability , and the success probability before it is , where is the mean success probability of the alternatives. The true drop is therefore at most , far below the minimum detection threshold of 0.180 (Table 2). Every deterministic cliff, 9 to 48% of cliff tokens per model (Table 1) and about 200 of the 709 overall (Tables 1 and 2), could be an error of the 64-rollout estimate.
The cause is that the z-test is applied at every position of every trace with no multiple-comparison control, so a per-position false-positive rate well below 1% still yields hundreds of spurious cliffs. The claim that the adaptive threshold "separates statistically significant drops from sampling noise" does not seem to be well supported.
Consistent with this, 7 of 17 Cliff-del/Cliff-keep panels whose pass@1 gap has a 95% CI containing zero (Tables 8 to 10) all belong to Qwen3-8B, Qwen3-4B and Gemma-3-4B, the three models with 44 to 48% deterministic cliffs. The main-text validation is also partly circular: a first cliff can only be detected where at least 12 of the 64 detection rollouts from the preceding prefix succeeded, so Cliff-del pass@64 1 is unsurprising given selection, and comparing resampling before versus after the token re-measures the definition. No controlled experiment at matched-probability non-cliff positions is given.
2. The core idea is the pivotal token of Abdin et al. (2024), and the training evidence is model-dependent and partly confounded by data quantity.
Pivotal Token Search estimates p(success) by rollouts, flags single tokens whose generation changes it by a fixed gap (0.2), and builds DPO pairs with the prefix as context and single chosen/rejected tokens, which is Eq. (6) and (7) of this paper. The stated advance is the adaptive threshold, but it evaluates to [0.180, 0.241] (Table 2), so in practice its detection boundary remains close to the fixed 0.2 it criticizes, and as shown above it does not control noise. Exhaustive per-token rollouts instead of recursive subdivision is a compute choice. Bigelow et al. (2025), cited here only for subsampling, already detect token-level shifts in outcome distributions with a statistical change-point model. Zhu et al. (2026), cited only in the introduction, already report that failures originate at a few transition points that coincide with spikes in token-level entropy, that alternative continuations from those points can still succeed, and give an inference-time method that redirects them using uncertainty signals. That substantially overlaps the motivating findings in Sections 4.3 and 5 and the "decoding-time resampling, backtracking, or branching" the paper lists as future work, although it does not provide the same three-way taxonomy. The entropy findings overlap the known fact that forking tokens are high-entropy (Wang et al., 2025, also cited only in passing), and "only uncertain and sampled-off cliffs respond to training" is the 80/20 finding in DPO form, although the objectives and selected subsets differ. The Abdin overlap is acknowledged, but the implications of the Bigelow, Zhu, and Wang results are not substantively discussed.
The new claim that cliff types differ as training signals is confounded by pair count: the uncertain and sampled-off subsets are 2.4 to 12 times larger than the deterministic one (Table 16). At matched budget (Table 18) the gaps shrink to at most 1.8 points with standard errors up to 1.5, and reverse on GSM8K and GSM1K for Llama-3.2-1B. cDPO (Lin et al., 2025) beats every Cliff-DPO variant on GSM1K for both Llama models (81.7 vs at most 79.8; 41.4 vs at most 40.4) and is omitted from Figure 5. The natural control, single-token DPO at randomly chosen high-entropy positions with no rollouts, is missing. Without it the incremental benefit of rollout-based detection over entropy-based position selection is not isolated.
Questions for authors
- Could you report Cliff-del vs Cliff-keep with fresh rollouts separately for deterministic, uncertain and sampled-off first cliffs? By the argument in Weakness 1 the deterministic gap should be about 0. If it is not, could you explain how a token emitted with probability 0.999 produces a drop above 0.18 under the same sampling policy, and state whether is computed on the raw or the temperature/top-p/top-k distribution and whether rollout prefixes are passed as token ids or re-tokenized text?
- How many positions were tested in total? How many cliffs does the detector return under a no-change null, meaning that the true success probability is the same before and after the token, so the estimated drop comes from sampling noise? To estimate this, for example, you can pool the 64 outcomes from before the token and the 64 from after it, then randomly redistribute them into two groups of 64. Repeating this would show how often random differences alone trigger the detector. How many of the 709 cliffs remain after such correction?
- Could you add Cliff-DPO with the same number of pairs at random positions whose entropy exceeds the taxonomy threshold, with the same chosen-token procedure or simply the greedy token as chosen? Does rollout-based detection add anything over entropy?
9 comments
2. Multiple comparisons: how many cliffs sampling noise can produce.
Thank you for suggesting this analysis. We estimate how many cliff tokens would be detected from rollout noise alone. The analysis covers 1,610 traces and 1,258,178 comparisons between adjacent token prefixes. At each position, we pool the 64 rollout outcomes before and 64 after the token. We calculate how often a random split into two groups of 64 would produce a success probability drop above the adaptive threshold.
Table I. Detections expected when there is no change in success probability.
| Cliff type | Observed cliff detections | Expected cliff detections under no change |
|---|---|---|
| Deterministic cliff | 207 | 185.53 |
| Uncertain cliff | 224 | 93.21 |
| Sampled-off cliff | 272 | 13.65 |
| Subtotal (three types) | 703 | 292.39 |
In Table I, the number of deterministic detections is close to the expected count. Uncertain and sampled-off detections substantially exceed their expected counts. We were able to reproduce their drops with fresh rollouts on incorrect traces (Table H), confirming a positive average drop in success probability for both cliff types.
We will replace “adaptive threshold that separates statistically significant drops in success probability from sampling noise” with “adaptive threshold that accounts for rollout noise at each position”. We will also explicitly state in the camera-ready version that it does not control errors from testing many positions.
3. Novelty: how our study differs from prior work.
Thank you for this careful comparison. Token-level effects on reasoning outcomes, the role of token entropy, and single-token DPO have been studied before. Equations (6)–(7) share the form of Pivotal Token Search (PTS) [3]. We analyze how success probability drops relate to token entropy and whether the sampled token is greedy. We also evaluate cliff positions as training signals using Cliff-DPO.
3-1. Pivotal Token Search: rollout noise and search coverage
Expected noise-induced detections. Using the same 64-rollout success probability estimates and the pooled no-change analysis from Table I, we compare the fixed threshold of 0.2 used in PTS with our z-test-based adaptive threshold.
Table J. Expected cliff detections under no change.
| Token positions | Fixed threshold (0.2) | Adaptive threshold |
|---|---|---|
| All token positions | 1,066.02 | 292.42 |
As shown in Table J, our adaptive threshold substantially reduces expected noise-induced cliff detections at , even though its values are close to the fixed threshold of 0.2.
Comparison with PTS-style recursive search. PTS recursively divides a trace into smaller intervals, stopping when the success probabilities at the two endpoints differ by less than 0.2. PTS acknowledges that recursive search can miss pivotal tokens on non-monotonic success probability curves [3].
Table K. Original cliff detections vs. positions rediscovered by PTS-style recursive search.
| Cliff type | Original detections | Rediscovered by recursive search |
|---|---|---|
| Uncertain cliff | 224 | 35 |
| Sampled-off cliff | 272 | 104 |
Our PTS-style replay uses the available success probability estimates and rediscovers only a subset of the detected cliff tokens (Table K). These results support our choice to estimate success probability at every evaluated token position.
3-2. Connections to Bigelow, Zhu, and Wang
Table L. Comparison with prior work.
| Prior work | Prior findings and methods | Comparison with our study |
|---|---|---|
| Bigelow et al. (2025) [4] | Bayesian change-point detection and survival analysis of alternative-token effects | Their change-point analysis tracks changes in the distribution of all observed answers. Our analysis focuses on drops in success probability. |
| Zhu et al. (2026) [5] | Higher token entropy near reasoning failures, recovery through resampling, and uncertainty-guided branching during inference | To locate the starting point of a reasoning failure, they judge whether reasoning segments are correct. We locate success probability drops at individual tokens, which need not coincide with error tokens (Figure 4). |
| Wang et al. (2025) [6] | RLVR updates focused on high-entropy tokens | We test whether uncertain and sampled-off cliff positions provide more useful training signals than deterministic cliff positions, high-entropy positions, or non-greedy positions in single-token DPO (Figure 5 and Table M). |
We will clarify this scope in the abstract and introduction, add Table L to Related Work, and acknowledge Zhu et al. [5] when discussing future inference-time interventions. We thank you for suggesting this comparison, which has helped us clarify our paper's positioning.
4. Cliff-DPO: comparison with high-entropy and non-greedy selection at the same training budget.
4-1. Full-budget and matched-budget comparisons
Thank you for this comment. In the Cliff-DPO experiment (Figure 5), all cliff types come from the same 7,473 GSM8K training problems and the same cliff-token identification and preference-pair construction pipeline. Each variant uses all available pairs of its type (Appendix Q.3), so the number of preference pairs differs across cliff types.
To control for this difference, we subsample the uncertain and sampled-off subsets to match the update-token budget of the deterministic subset (Table 18). At this small budget, gains over the base models are generally small, suggesting that the Cliff-DPO variants may be undertrained. The uncertain and sampled-off variants show larger gains at their full budgets (Appendix P). We will clarify the limited training budget used in Table 18.
4-2. High-entropy and non-greedy baselines
We compared Cliff-DPO with baseline training on high-entropy and non-greedy token positions under the same preference-pair counts and fine-tuning settings. Both baselines sample uniformly from high-entropy or non-greedy token positions in the same GSM8K training traces. To construct each DPO preference pair, we use the original token as rejected and an alternative with higher estimated success probability as chosen, based on 64 rollouts per candidate. High-entropy positions satisfy . These baselines select positions without cliff detection but use rollouts to construct preference pairs. Table M reports mean accuracy (%) ± SE over three training seeds.
Table M. Matched-budget token-selection baselines.
| Model | Method | GSM8K | GSM1K | MATH500 | Updated tokens |
|---|---|---|---|---|---|
| Qwen3-0.6B | High-entropy | 61.08±0.30 | 55.63±0.52 | 50.40±0.40 | 32,916 |
| Qwen3-0.6B | Non-greedy | 63.79±0.22 | 59.00±0.31 | 51.07±0.29 | 32,916 |
| Qwen3-0.6B | Cliff-DPO (uncertain + sampled-off) | 66.30±0.47 | 63.00±0.28 | 56.10±0.59 | 32,916 |
| Llama-3.2-1B | High-entropy | 44.23±0.36 | 35.93±0.56 | 27.00±1.01 | 62,152 |
| Llama-3.2-1B | Non-greedy | 43.72±0.34 | 36.65±0.45 | 26.93±0.55 | 62,152 |
| Llama-3.2-1B | Cliff-DPO (uncertain + sampled-off) | 45.30±0.48 | 39.40±0.41 | 29.20±0.76 | 62,152 |
| Llama-3.1-8B | High-entropy | 81.27±0.13 | 77.54±0.17 | 49.40±0.61 | 34,084 |
| Llama-3.1-8B | Non-greedy | 82.06±0.41 | 78.15±0.19 | 51.00±0.20 | 34,084 |
| Llama-3.1-8B | Cliff-DPO (uncertain + sampled-off) | 83.90±0.24 | 79.80±0.31 | 49.80±0.42 | 34,084 |
As Table M shows, Cliff-DPO has higher mean accuracy than the high-entropy baseline in all nine settings and than the non-greedy baseline in eight of nine settings at the same training budget. These results suggest that rollout-based cliff detection adds information beyond entropy for selecting training positions. We will include these baseline results in Figure 5.
Comparison with cDPO. Table 16 (Appendix N) shows that cDPO [7] achieves higher GSM1K accuracy on both Llama models. However, our claim in Section 6 is that cliff positions are useful training signals, not that Cliff-DPO outperforms cDPO. Cliff-DPO reaches comparable accuracy while supervising about 1% as many token positions as cDPO, which supports this claim. Figure 5 therefore focuses on the comparison between cliff types. At the same time, we agree with your point that this presentation could be misleading. We will include the cDPO comparison in the main text to clarify this point.
Thank you again for your careful review. We hope these experiments and clarifications address your concerns, and we will incorporate them into the camera-ready version. If you have any additional discussion points or suggestions, we would be happy to address them.
References
[3] Abdin et al. (2024). Phi-4 Technical Report.
[4] Bigelow et al. (2025). Forking Paths in Neural Text Generation.
[5] Zhu et al. (2026). Dissecting Failure Dynamics in Large Language Model Reasoning.
[6] Wang et al. (2025). Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning.
[7] Lin et al. (2025). Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM’s Reasoning Capability.
Thank you for the careful review and for recognizing the value of the released rollout data and our careful design choices. Below, we address your concerns with additional analyses and controlled comparisons.
1. Deterministic cliffs: whether the detected drops reproduce in independent rollouts.
Thank you for your careful reading and this important point. Following your suggestion, we report Cliff-del and Cliff-keep results with independent rollouts separately for deterministic, uncertain, and sampled-off first cliffs. We also compare them with matched non-cliff positions.
Settings. Following Section 4.2 and Figure 3, we report Cliff-del and Cliff-keep results for the first cliff token in each incorrect trace. Note that these rollouts are independent of the detection rollouts. We calculate token entropy from the top 20 raw token probabilities before applying temperature, top-p, or top-k. Rollout prefixes use stored token IDs without re-tokenizing the text. For non-cliff controls, we sample 64 fresh continuations before and after positions in the same trace with similar preceding success probability and token entropy. Non-cliff results include only traces with a matched non-cliff position. The results are as follows:
Table H. First-cliff results and matched non-cliff controls on incorrect traces.
As Table H shows, independent rollouts show no average success probability drop at deterministic cliff detections. The stored probabilities and sampling settings show that only the original greedy token remains after top-p filtering at all 207 deterministic positions. These results indicate that the original deterministic cliff detections were driven by noise in the 64-rollout estimates, and our interpretation of deterministic cliffs needs to be revised. In contrast, the drops at uncertain and sampled-off cliffs reproduce (+0.12 and +0.39), while changes at matched non-cliff positions are close to zero. However, we believe that our cliff taxonomy remains useful. These results show the value of distinguishing deterministic detections from uncertain and sampled-off cliffs when interpreting success probability drops. Our training results further support this distinction. The deterministic Cliff-DPO variants show smaller accuracy gains than the uncertain and sampled-off variants (Section 6.2). Their mean deterministic cliff-token probabilities change by at most 0.06 pp (Table 17). Your comment helps clarify this interpretation and the value of our cliff taxonomy.
We will make the following changes in the camera-ready version:
[Section 1]
[Section 5.2]
[Section 6.2]