acceptodds
Under review as a conference paper at ICLR 2027

Cliff Tokens: Analyzing Failure Trigger Tokens in LLM Mathematical Reasoning

Abstract

Large language models reach high accuracy in mathematical reasoning, but individual traces on the same problem diverge; some arrive at the correct answer while others fail. Prior work localizes such failures at the step, chunk, or sentence level, or identifies tokens where failure has already occurred. These approaches leave open which token triggers failure. We introduce the **cliff token**, a token at which the estimated probability of reaching the correct answer (success probability) drops beyond an adaptive threshold. Across seven models and three mathematical reasoning benchmarks (GSM1K, MATH500, AIME 2025), cliff tokens act as failure triggers. For incorrect traces containing cliff tokens, we compare resampling immediately before and after the first cliff token. Resampling before it shows higher pass@ at the same sample count. We further introduce a cliff taxonomy of deterministic, uncertain, and sampled-off cliffs, defined by greedy choice and token entropy. Additionally, we show that the three types differ as training signals. Using single-token preference optimization at cliff positions (Cliff-DPO), we find that uncertain and sampled-off cliffs show larger accuracy gains than deterministic cliffs on three evaluation benchmarks. We release token-level rollout data and source code to enable further analysis without regenerating costly rollouts: https://anonymous.4open.science/r/cliff_token_data/.

Primary area:
interpretability and explainable AI
Keywords:
large language model, mathematical reasoning, failure trigger

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_nxgbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper defines a cliff token as a position in a sampled reasoning trace where the Monte Carlo estimate of the success probability drops by more than τ+1.645⋅SEt\tau + 1.645 \cdot \mathrm{SE}_t with τ=0.1\tau = 0.1 and SEt\mathrm{SE}_t being the estimated standard error of the difference between adjacent success probability estimates. Using one trace per problem for seven open models on GSM1K, MATH500 and AIME 2025, it reports that cliff tokens are more common in incorrect traces, that resampling from before the first cliff gives higher pass@k than resampling from after it, and that cliffs need not coincide with the arithmetic error. Cliff tokens are split into deterministic, uncertain and sampled-off types by whether the token was greedy and by next-token entropy. Single-token DPO pairs built at cliff positions (Cliff-DPO) are trained per type on three models. The uncertain and sampled-off subsets give larger gains than the deterministic subset. The token-level rollout data are released.

Critical strengths
  1. The released per-token rollout data is the most durable contribution. Success probability curves at token granularity are expensive to produce and are not otherwise available, and the paper's analyses, including the checks requested below, can be re-run from them.
  2. Several design choices are careful. Cliff-del/Cliff-keep uses fresh rollouts independent of the detection rollouts, bootstrap CIs are given for every panel, and the token-matched ablation and the DPO/cDPO baselines are included in the appendix even where they weaken the headline. The writing is clear.
Critical weaknesses

1. The deterministic cliff counts raise concerns about detector calculation and interpretation.

A deterministic cliff is a greedy token with Ht<Hb(0.99)H_t < H_b(0.99), so the model emits it with probability q>0.99q > 0.99. Appendix O measures 99.9%. Rollouts from the preceding prefix use the same sampling settings (Table 4), so they redraw that token with probability qq, and the success probability before it is q pt+(1−q)qˉq \ p_t + (1-q) \bar q, where qˉ\bar q is the mean success probability of the alternatives. The true drop is therefore at most 1−q≤0.011-q \le 0.01, far below the minimum detection threshold of 0.180 (Table 2). Every deterministic cliff, 9 to 48% of cliff tokens per model (Table 1) and about 200 of the 709 overall (Tables 1 and 2), could be an error of the 64-rollout estimate.

The cause is that the z-test is applied at every position of every trace with no multiple-comparison control, so a per-position false-positive rate well below 1% still yields hundreds of spurious cliffs. The claim that the adaptive threshold "separates statistically significant drops from sampling noise" does not seem to be well supported.

Consistent with this, 7 of 17 Cliff-del/Cliff-keep panels whose pass@1 gap has a 95% CI containing zero (Tables 8 to 10) all belong to Qwen3-8B, Qwen3-4B and Gemma-3-4B, the three models with 44 to 48% deterministic cliffs. The main-text validation is also partly circular: a first cliff can only be detected where at least 12 of the 64 detection rollouts from the preceding prefix succeeded, so Cliff-del pass@64 ≈\approx 1 is unsurprising given selection, and comparing resampling before versus after the token re-measures the definition. No controlled experiment at matched-probability non-cliff positions is given.

2. The core idea is the pivotal token of Abdin et al. (2024), and the training evidence is model-dependent and partly confounded by data quantity.

Pivotal Token Search estimates p(success) by rollouts, flags single tokens whose generation changes it by a fixed gap (0.2), and builds DPO pairs with the prefix as context and single chosen/rejected tokens, which is Eq. (6) and (7) of this paper. The stated advance is the adaptive threshold, but it evaluates to [0.180, 0.241] (Table 2), so in practice its detection boundary remains close to the fixed 0.2 it criticizes, and as shown above it does not control noise. Exhaustive per-token rollouts instead of recursive subdivision is a compute choice. Bigelow et al. (2025), cited here only for subsampling, already detect token-level shifts in outcome distributions with a statistical change-point model. Zhu et al. (2026), cited only in the introduction, already report that failures originate at a few transition points that coincide with spikes in token-level entropy, that alternative continuations from those points can still succeed, and give an inference-time method that redirects them using uncertainty signals. That substantially overlaps the motivating findings in Sections 4.3 and 5 and the "decoding-time resampling, backtracking, or branching" the paper lists as future work, although it does not provide the same three-way taxonomy. The entropy findings overlap the known fact that forking tokens are high-entropy (Wang et al., 2025, also cited only in passing), and "only uncertain and sampled-off cliffs respond to training" is the 80/20 finding in DPO form, although the objectives and selected subsets differ. The Abdin overlap is acknowledged, but the implications of the Bigelow, Zhu, and Wang results are not substantively discussed.

The new claim that cliff types differ as training signals is confounded by pair count: the uncertain and sampled-off subsets are 2.4 to 12 times larger than the deterministic one (Table 16). At matched budget (Table 18) the gaps shrink to at most 1.8 points with standard errors up to 1.5, and reverse on GSM8K and GSM1K for Llama-3.2-1B. cDPO (Lin et al., 2025) beats every Cliff-DPO variant on GSM1K for both Llama models (81.7 vs at most 79.8; 41.4 vs at most 40.4) and is omitted from Figure 5. The natural control, single-token DPO at randomly chosen high-entropy positions with no rollouts, is missing. Without it the incremental benefit of rollout-based detection over entropy-based position selection is not isolated.

Questions for authors
  1. Could you report Cliff-del vs Cliff-keep with fresh rollouts separately for deterministic, uncertain and sampled-off first cliffs? By the argument in Weakness 1 the deterministic gap should be about 0. If it is not, could you explain how a token emitted with probability 0.999 produces a drop above 0.18 under the same sampling policy, and state whether HtH_t is computed on the raw or the temperature/top-p/top-k distribution and whether rollout prefixes are passed as token ids or re-tokenized text?
  2. How many positions were tested in total? How many cliffs does the detector return under a no-change null, meaning that the true success probability is the same before and after the token, so the estimated drop comes from sampling noise? To estimate this, for example, you can pool the 64 outcomes from before the token and the 64 from after it, then randomly redistribute them into two groups of 64. Repeating this would show how often random differences alone trigger the detector. How many of the 709 cliffs remain after such correction?
  3. Could you add Cliff-DPO with the same number of pairs at random positions whose entropy exceeds the taxonomy threshold, with the same chosen-token procedure or simply the greedy token as chosen? Does rollout-based detection add anything over entropy?
9 comments

Thank you for the careful review and for recognizing the value of the released rollout data and our careful design choices. Below, we address your concerns with additional analyses and controlled comparisons.

1. Deterministic cliffs: whether the detected drops reproduce in independent rollouts.

Thank you for your careful reading and this important point. Following your suggestion, we report Cliff-del and Cliff-keep results with independent rollouts separately for deterministic, uncertain, and sampled-off first cliffs. We also compare them with matched non-cliff positions.

Settings. Following Section 4.2 and Figure 3, we report Cliff-del and Cliff-keep results for the first cliff token in each incorrect trace. Note that these rollouts are independent of the detection rollouts. We calculate token entropy from the top 20 raw token probabilities before applying temperature, top-pp, or top-kk. Rollout prefixes use stored token IDs without re-tokenizing the text. For non-cliff controls, we sample 64 fresh continuations before and after positions in the same trace with similar preceding success probability and token entropy. Non-cliff results include only traces with a matched non-cliff position. The results are as follows:

Table H. First-cliff results and matched non-cliff controls on incorrect traces.

Cliff typeCliff-del pass@1Cliff-keep pass@1Δcliff\Delta_{\mathrm{cliff}} [95% CI]Δnoncliff\Delta_{\mathrm{noncliff}} [95% CI]
Deterministic cliff0.470.470.00 [−0.03, 0.04]+0.01 [−0.03, 0.04]
Uncertain cliff0.410.29+0.12 [0.08, 0.15]−0.02 [−0.05, 0.01]
Sampled-off cliff0.600.20+0.39 [0.35, 0.44]+0.01 [−0.02, 0.03]

As Table H shows, independent rollouts show no average success probability drop at deterministic cliff detections. The stored probabilities and sampling settings show that only the original greedy token remains after top-pp filtering at all 207 deterministic positions. These results indicate that the original deterministic cliff detections were driven by noise in the 64-rollout estimates, and our interpretation of deterministic cliffs needs to be revised. In contrast, the drops at uncertain and sampled-off cliffs reproduce (+0.12 and +0.39), while changes at matched non-cliff positions are close to zero. However, we believe that our cliff taxonomy remains useful. These results show the value of distinguishing deterministic detections from uncertain and sampled-off cliffs when interpreting success probability drops. Our training results further support this distinction. The deterministic Cliff-DPO variants show smaller accuracy gains than the uncertain and sampled-off variants (Section 6.2). Their mean deterministic cliff-token probabilities change by at most 0.06 pp (Table 17). Your comment helps clarify this interpretation and the value of our cliff taxonomy.

We will make the following changes in the camera-ready version:

  • [Section 1]

    • Add “Independent rollouts reproduce the success probability drops at uncertain and sampled-off cliffs, while the average success probability drop at deterministic cliff detections does not reproduce.” before the sentence beginning “Uncertain and sampled-off cliffs show larger accuracy gains than deterministic ones”.
    • Delete “, as the models already assign near-certain probabilities to deterministic cliffs” from that sentence.
  • [Section 5.2]

    • Delete “This trend suggests that scaling up model parameters shifts the errors from stochastic sampling to deterministic ones.”
    • Replace “This shows that each model family has a specific pattern in its reasoning failures. Overall, these findings show that token-level mathematical reasoning failures vary with both model family and scale.” with “The proportions of detected cliff types vary across model families and scales.”
  • [Section 6.2]

    • Replace “The effect of Cliff-DPO depends on how strongly the model initially favors the cliff token. For deterministic cliffs, the token probability is more than 99%, indicating that the prior is difficult to overcome via post-training. Accordingly, Cliff-DPO changes its mean cliff-token probability by at most 0.06 pp.” with “The deterministic Cliff-DPO variants show smaller accuracy gains than the uncertain and sampled-off variants. Their mean deterministic cliff-token probabilities change by at most 0.06 pp. These findings are consistent with the lack of a reproducible average success probability drop at deterministic cliff detections.”

2. Multiple comparisons: how many cliffs sampling noise can produce.

Thank you for suggesting this analysis. We estimate how many cliff tokens would be detected from rollout noise alone. The analysis covers 1,610 traces and 1,258,178 comparisons between adjacent token prefixes. At each position, we pool the 64 rollout outcomes before and 64 after the token. We calculate how often a random split into two groups of 64 would produce a success probability drop above the adaptive threshold.

Table I. Detections expected when there is no change in success probability.

Cliff typeObserved cliff detectionsExpected cliff detections under no change
Deterministic cliff207185.53
Uncertain cliff22493.21
Sampled-off cliff27213.65
Subtotal (three types)703292.39

In Table I, the number of deterministic detections is close to the expected count. Uncertain and sampled-off detections substantially exceed their expected counts. We were able to reproduce their drops with fresh rollouts on incorrect traces (Table H), confirming a positive average drop in success probability for both cliff types.

We will replace “adaptive threshold that separates statistically significant drops in success probability from sampling noise” with “adaptive threshold that accounts for rollout noise at each position”. We will also explicitly state in the camera-ready version that it does not control errors from testing many positions.

3. Novelty: how our study differs from prior work.

Thank you for this careful comparison. Token-level effects on reasoning outcomes, the role of token entropy, and single-token DPO have been studied before. Equations (6)–(7) share the form of Pivotal Token Search (PTS) [3]. We analyze how success probability drops relate to token entropy and whether the sampled token is greedy. We also evaluate cliff positions as training signals using Cliff-DPO.

3-1. Pivotal Token Search: rollout noise and search coverage

Expected noise-induced detections. Using the same 64-rollout success probability estimates and the pooled no-change analysis from Table I, we compare the fixed threshold of 0.2 used in PTS with our z-test-based adaptive threshold.

Table J. Expected cliff detections under no change.

Token positionsFixed threshold (0.2)Adaptive threshold
All token positions1,066.02292.42

As shown in Table J, our adaptive threshold substantially reduces expected noise-induced cliff detections at N=64N=64, even though its values are close to the fixed threshold of 0.2.

Comparison with PTS-style recursive search. PTS recursively divides a trace into smaller intervals, stopping when the success probabilities at the two endpoints differ by less than 0.2. PTS acknowledges that recursive search can miss pivotal tokens on non-monotonic success probability curves [3].

Table K. Original cliff detections vs. positions rediscovered by PTS-style recursive search.

Cliff typeOriginal detectionsRediscovered by recursive search
Uncertain cliff22435
Sampled-off cliff272104

Our PTS-style replay uses the available success probability estimates and rediscovers only a subset of the detected cliff tokens (Table K). These results support our choice to estimate success probability at every evaluated token position.

3-2. Connections to Bigelow, Zhu, and Wang

Table L. Comparison with prior work.

Prior workPrior findings and methodsComparison with our study
Bigelow et al. (2025) [4]Bayesian change-point detection and survival analysis of alternative-token effectsTheir change-point analysis tracks changes in the distribution of all observed answers. Our analysis focuses on drops in success probability.
Zhu et al. (2026) [5]Higher token entropy near reasoning failures, recovery through resampling, and uncertainty-guided branching during inferenceTo locate the starting point of a reasoning failure, they judge whether reasoning segments are correct. We locate success probability drops at individual tokens, which need not coincide with error tokens (Figure 4).
Wang et al. (2025) [6]RLVR updates focused on high-entropy tokensWe test whether uncertain and sampled-off cliff positions provide more useful training signals than deterministic cliff positions, high-entropy positions, or non-greedy positions in single-token DPO (Figure 5 and Table M).

We will clarify this scope in the abstract and introduction, add Table L to Related Work, and acknowledge Zhu et al. [5] when discussing future inference-time interventions. We thank you for suggesting this comparison, which has helped us clarify our paper's positioning.

4. Cliff-DPO: comparison with high-entropy and non-greedy selection at the same training budget.

4-1. Full-budget and matched-budget comparisons

Thank you for this comment. In the Cliff-DPO experiment (Figure 5), all cliff types come from the same 7,473 GSM8K training problems and the same cliff-token identification and preference-pair construction pipeline. Each variant uses all available pairs of its type (Appendix Q.3), so the number of preference pairs differs across cliff types.

To control for this difference, we subsample the uncertain and sampled-off subsets to match the update-token budget of the deterministic subset (Table 18). At this small budget, gains over the base models are generally small, suggesting that the Cliff-DPO variants may be undertrained. The uncertain and sampled-off variants show larger gains at their full budgets (Appendix P). We will clarify the limited training budget used in Table 18.

4-2. High-entropy and non-greedy baselines

We compared Cliff-DPO with baseline training on high-entropy and non-greedy token positions under the same preference-pair counts and fine-tuning settings. Both baselines sample uniformly from high-entropy or non-greedy token positions in the same GSM8K training traces. To construct each DPO preference pair, we use the original token as rejected and an alternative with higher estimated success probability as chosen, based on 64 rollouts per candidate. High-entropy positions satisfy Ht>Hb(0.99)H_t > H_b(0.99). These baselines select positions without cliff detection but use rollouts to construct preference pairs. Table M reports mean accuracy (%) ± SE over three training seeds.

Table M. Matched-budget token-selection baselines.

ModelMethodGSM8KGSM1KMATH500Updated tokens
Qwen3-0.6BHigh-entropy61.08±0.3055.63±0.5250.40±0.4032,916
Qwen3-0.6BNon-greedy63.79±0.2259.00±0.3151.07±0.2932,916
Qwen3-0.6BCliff-DPO (uncertain + sampled-off)66.30±0.4763.00±0.2856.10±0.5932,916
Llama-3.2-1BHigh-entropy44.23±0.3635.93±0.5627.00±1.0162,152
Llama-3.2-1BNon-greedy43.72±0.3436.65±0.4526.93±0.5562,152
Llama-3.2-1BCliff-DPO (uncertain + sampled-off)45.30±0.4839.40±0.4129.20±0.7662,152
Llama-3.1-8BHigh-entropy81.27±0.1377.54±0.1749.40±0.6134,084
Llama-3.1-8BNon-greedy82.06±0.4178.15±0.1951.00±0.2034,084
Llama-3.1-8BCliff-DPO (uncertain + sampled-off)83.90±0.2479.80±0.3149.80±0.4234,084

As Table M shows, Cliff-DPO has higher mean accuracy than the high-entropy baseline in all nine settings and than the non-greedy baseline in eight of nine settings at the same training budget. These results suggest that rollout-based cliff detection adds information beyond entropy for selecting training positions. We will include these baseline results in Figure 5.

Comparison with cDPO. Table 16 (Appendix N) shows that cDPO [7] achieves higher GSM1K accuracy on both Llama models. However, our claim in Section 6 is that cliff positions are useful training signals, not that Cliff-DPO outperforms cDPO. Cliff-DPO reaches comparable accuracy while supervising about 1% as many token positions as cDPO, which supports this claim. Figure 5 therefore focuses on the comparison between cliff types. At the same time, we agree with your point that this presentation could be misleading. We will include the cDPO comparison in the main text to clarify this point.

Thank you again for your careful review. We hope these experiments and clarifications address your concerns, and we will incorporate them into the camera-ready version. If you have any additional discussion points or suggestions, we would be happy to address them.

References

[3] Abdin et al. (2024). Phi-4 Technical Report.

[4] Bigelow et al. (2025). Forking Paths in Neural Text Generation.

[5] Zhu et al. (2026). Dissecting Failure Dynamics in Large Language Model Reasoning.

[6] Wang et al. (2025). Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning.

[7] Lin et al. (2025). Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM’s Reasoning Capability.

Reviewer_wwNEOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents an interesting finding regarding cliff tokens, which play a crucial role in the reasoning processes of LLMs. The authors conduct an in-depth investigation of these tokens and propose a novel DPO-based training method that specifically optimizes them. The improved performance further highlights the importance of these critical tokens in LLM reasoning.

Critical strengths
  1. The paper's main finding regarding cliff tokens is interesting.

  2. The paper is well-written and easy to follow.

  3. The paper provides a thorough and in-depth investigation of cliff tokens.

  4. The experimental results demonstrate the effectiveness of leveraging cliff tokens to improve LLM reasoning.

Critical weaknesses
  1. Cliff tokens are conceptually similar to critical tokens. It would be beneficial to include a comparison between cDPO, proposed in the critical token work, and the proposed Cliff-DPO in the experiments.

  2. The proposed Cliff-DPO requires time-consuming offline rollouts to identify cliff tokens, which may introduce substantial computational overhead.

  3. Since the entropy distribution of cliff tokens may vary across different models, the fixed threshold (H = 0.0561) may not generalize well to different models.

Questions for authors
  1. Cliff tokens are very similar to critical tokens, as also acknowledged by the authors. I suggest that the authors provide a case study to illustrate the differences between critical tokens and the proposed cliff tokens using the same incorrect reasoning example.
2 comments

Thank you for your positive assessment and thoughtful feedback. We particularly appreciate your recognition of the findings on cliff tokens, the clarity of the paper, the thorough investigation, and the experimental evidence that cliff tokens can help improve LLM reasoning. Below, we address your comments point by point and hope our responses resolve your concerns.

1. Comparison with cDPO: performance and the number of supervised token positions.

Thank you for this suggestion. Appendix N, Table 16 includes cDPO alongside DPO and all five Cliff-DPO variants on Qwen3-0.6B, Llama-3.2-1B, and Llama-3.1-8B. The comparison covers GSM8K, GSM1K, MATH500, and AIME 2025, with mean performance and standard errors. Cliff-DPO obtains comparable improvements using about 1% as many supervised token positions as cDPO, which trains with all response tokens. We will make this comparison more visible in the main text.

2. Computational cost of offline rollouts.

Thank you for raising this important point. We agree that our rollouts are costly. Our goal was to understand which tokens trigger sharp drops in success probability along reasoning traces. To locate individual cliff tokens precisely, we used Monte Carlo rollouts to estimate success probability at each token position. This requires substantial computation, as discussed in Appendix C (4,047 A100 GPU-hours). We have therefore released the rollout dataset so that others can reuse these estimates and extend this line of research without repeating the costly rollouts. Moreover, Table F summarizes the estimated offline rollout costs of constructing Cliff-DPO training data from the 7,473 GSM8K training questions.

Table F. Cliff-DPO training data construction cost.

ModelGPU-hours
Qwen3-0.6B267
Llama-3.2-1B258
Llama-3.1-8B839

We will clarify this training data construction cost more clearly in the paper.

3. Robustness to the entropy threshold across models.

Thank you for raising this concern. We agree that token entropy distribution patterns can differ across models (Appendix H). We set the same threshold, Hb(0.99)H_b(0.99), for all seven models without model-specific tuning (Section 5.1).

To check whether the observed trends depend on this particular threshold, we conduct a sensitivity analysis across all seven models (Appendix J, Table 15). We apply the same alternative thresholds to all models using Hb(p1)H_b(p_1) with p1∈{0.90,0.95,0.999}p_1 \in \{0.90, 0.95, 0.999\}, alongside the default p1=0.99p_1=0.99.

Although the type proportions change, the main trends are broadly preserved across the tested thresholds:

  • Distribution relative to all tokens: the deterministic proportion is lower among cliff tokens than among all tokens, while the sampled-off proportion is higher.
  • Scaling within model families: the shift toward more deterministic and fewer sampled-off cliff tokens as model size increases is generally preserved in the Qwen3 and Llama-3 families.

Thus, the main distributional findings remain broadly consistent across the seven models and tested thresholds without model-specific threshold tuning.

4. Case study comparing a cliff token and a critical token on the same incorrect trace.

Thank you for this suggestion. A cliff token is a token where there is a sharp drop in success probability, while a critical token is one where the success probability has already collapsed to zero. We provide a comparison on the same incorrect reasoning trace below.

Example. Llama-3.1-8B on GSM1K (problem 189).

Question. Arnold has 60 seconds remaining and spends 36 seconds finding a six-move checkmate. How much time can he spend per move?

Model Answer.

Maximum time per move = 60 / 6
Maximum time per move = 10 seconds
So [cliff token, position 72], the most amount of time Arnold can spend making each move in the checkmate sequence in order to not lose [critical token, position 94] is 10\boxed{10} seconds.

Correct Answer. (60−36)/6=4(60-36)/6=4 seconds. (The model ignores the 36 seconds already spent.)

Table G. Comparison between the critical token and the cliff token.

Detected tokenToken positionSuccess probability before tokenSuccess probability at tokenSuccess probability drop
Cliff token: So720.8440.0780.766
Critical token: lose940.07800.078

At So, the drop exceeds the adaptive threshold of 0.193. The critical token is generated 22 tokens later, at lose, where estimated success probability is zero and remains below 0.05 at every subsequent scored prefix. Thus, cliff tokens often precede critical tokens, indicating that they are meaningful precursors to states where estimated success probability reaches zero. We will include this case study in the revised paper to clarify the distinction between cliff tokens and critical tokens.

We hope these clarifications address your concerns. We will incorporate them into the camera-ready version and would be happy to discuss any remaining questions.

Reviewer_iRTeOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies failure-triggering tokens in LLM reasoning. It finds cliff tokens that can cause model to bad answer. The authors analyze cliff tokens across multiple models and propose Cliff-DPO to improve reasoning through token-level preference optimization.

Critical strengths

1.The paper provides a clear definition of cliff tokens.

2.The experiments show that removing cliff tokens can improve performance.

3.The analysis of different cliff types is interesting, and the training experiments suggest that uncertain and sampled-off cliffs provide more useful supervision than deterministic ones.

Critical weaknesses

1.The advantage over existing methods is unclear. Cliff-DPO does not consistently outperform cDPO.

2.Cliff identification is computationally expensive. Estimating success probabilities requires repeated rollouts at individual token positions which cost growing quadratically with reasoning length.

3.Generalization is insufficiently validated. The cross-model experiments only compare token probabilities and rankings, without testing whether the identified tokens remain failure triggers in other models. The paper also lacks cross-model training experiments, and the evaluation is limited to mathematical reasoning, with inconsistent improvements on AIME.

Questions for authors

1.Could the authors compare Cliff-DPO with simpler strategies, such as selecting high-entropy or non-greedy tokens, under the same training budget?

2.What is the total computational cost of constructing Cliff-DPO training data? Could fewer rollouts or a cheaper detection strategy achieve similar results?

3.Do cliff tokens identified by a smaller model also trigger failures in larger models? Can preference data collected from smaller models improve larger models or transfer to non-mathematical tasks?

5 comments

Thank you for the thoughtful feedback and for recognizing the cliff-token definition, the intervention results, and the differences between cliff types. Below, we clarify the scope of our claims and provide additional controlled comparisons, computational-cost information, and generalization experiments.

1. Cliff-DPO: Comparison with cDPO and simpler token-selection strategies.

1-1. Comparison with cDPO.

Thanks for this question. We would like to clarify the role of Cliff-DPO in our paper. As we discussed in Section 6.2, we use Cliff-DPO to test whether the cliff taxonomy differs as training signals. Our main finding is that uncertain and sampled-off cliffs are more effective than deterministic cliffs. We included cDPO in Appendix N as a reference because it shares a similar motivation. As you noted, cDPO achieves higher GSM1K accuracy on both Llama models. Nevertheless, Cliff-DPO reaches comparable accuracy overall while supervising less than 1% as many token positions as cDPO, which trains on all response tokens (Appendix N, Table 16). We will clarify this scope in Section 6 and bring this into the main text.

1-2. Comparison with high-entropy and non-greedy baselines.

Thank you for suggesting this experiment. Following your suggestion, we compared Cliff-DPO with high-entropy and non-greedy baselines under the same preference-pair counts and fine-tuning settings. Both baselines sample uniformly from high-entropy or non-greedy token positions in the same GSM8K training traces. To construct each DPO preference pair, we use the original token as rejected and an alternative with higher estimated success probability as chosen, based on 64 rollouts per candidate. Table A reports mean accuracy (%) ± SE over three training seeds.

Table A. Matched-budget token-selection baselines.

ModelMethodGSM8KGSM1KMATH500Updated token positions
Qwen3-0.6BHigh-entropy61.08±0.3055.63±0.5250.40±0.4032,916
Qwen3-0.6BNon-greedy63.79±0.2259.00±0.3151.07±0.2932,916
Qwen3-0.6BCliff-DPO (uncertain + sampled-off)66.30±0.4763.00±0.2856.10±0.5932,916
Llama-3.2-1BHigh-entropy44.23±0.3635.93±0.5627.00±1.0162,152
Llama-3.2-1BNon-greedy43.72±0.3436.65±0.4526.93±0.5562,152
Llama-3.2-1BCliff-DPO (uncertain + sampled-off)45.30±0.4839.40±0.4129.20±0.7662,152
Llama-3.1-8BHigh-entropy81.27±0.1377.54±0.1749.40±0.6134,084
Llama-3.1-8BNon-greedy82.06±0.4178.15±0.1951.00±0.2034,084
Llama-3.1-8BCliff-DPO (uncertain + sampled-off)83.90±0.2479.80±0.3149.80±0.4234,084

Cliff-DPO has higher mean accuracy than both baselines in eight of nine settings. The non-greedy baseline performs better on Llama3.1–8B/MATH500. These results suggest that cliff-based selection generally provides more useful training signals than high-entropy or non-greedy selection under the same training budget. We will include these results in the camera-ready version.

2. Computational cost and cheaper identification.

We agree that our rollouts are costly. However, our main goal was to understand which tokens trigger sharp drops in success probability along reasoning traces. To locate individual cliff tokens precisely, we used Monte Carlo rollouts to estimate success probability at each token position. This requires substantial computation, as discussed in Appendix C. We have therefore released the rollout dataset so that others can reuse these estimates and extend this line of research without repeating the costly rollouts. Table B summarizes the estimated offline rollout costs of constructing Cliff-DPO training data from the 7,473 GSM8K training questions.

Table B. Cliff-DPO training data construction cost.

ModelGPU-hours
Qwen3-0.6B267
Llama-3.2-1B258
Llama-3.1-8B839

Our finding that uncertain and sampled-off cliff tokens provide useful training signals also suggests a way to reduce computational cost. To identify these tokens, we only need rollouts at high-entropy positions and their preceding prefixes (39.7% of all prefix positions). Developing this approach is beyond the scope of this paper, so we will include it in the future-work discussion in Section 7.

3. Cross-model and non-mathematical generalization.

Thank you for these questions. We would like to clarify the scope of our findings. Our focus in the paper was to show that the cliff token phenomenon is observed across all seven models: there are tokens at which success probability drops sharply, and these tokens act as failure triggers. As noted in our Limitations, whether these findings generalize to larger models and non-mathematical domains remains an open question. To answer your questions, we conducted additional experiments during the rebuttal period.

3-1. Small-to-large transfer of cliff tokens and preference pairs.

We conducted two complementary experiments using models with shared tokenizers.

Cliff token transfer. First, we tested whether smaller-model cliff tokens remain cliff tokens in larger models by applying the original criterion to 64 rollouts from each prefix before and after the token. Of all the cliff tokens identified in smaller models, 19/111 remain cliff tokens in Llama-3.1-8B and 4/124 in Qwen3-8B.

Table C. Small-to-large cliff token transfer.

Smaller model → Larger modelCliff tokens retained in larger model / Cliff tokens identified in smaller model
Llama-3.2-1B → Llama-3.1-8B19/111
Qwen3-0.6B → Qwen3-8B4/124

Preference pairs transfer. Second, we trained Llama-3.1-8B directly on Llama-3.2-1B preference pairs using the existing fine-tuning settings. Table D reports mean accuracy (%) ± SE across three runs. Training Llama-3.1-8B on preference pairs from Llama-3.2-1B shows small or no gains on GSM8K, GSM1K, and MATH500. These results are consistent with our claim that whether a token is a cliff token depends on the model and context.

Table D. Llama-3.1-8B trained on Llama-3.2-1B preference pairs.

ModelGSM8KGSM1KMATH500Updated tokens
Llama-3.1-8B82.20±0.2078.20±0.0748.10±0.29—
+ Cliff-DPO (uncertain)81.93±0.2277.12±0.1147.40±0.1214,886
+ Cliff-DPO (sampled-off)83.14±0.4077.95±0.2448.13±1.7347,266
+ Cliff-DPO (uncertain + sampled-off)83.19±0.2877.26±0.6248.07±0.2962,152

3-2. Out-of-distribution evaluation.

Following your suggestion, we extended our evaluation beyond mathematical reasoning. We chose CruxEval-O (800 questions) for code reasoning and ScienceQA Physics (text-only, 192 questions) for science reasoning, where answer correctness can be automatically verified. We evaluated three Cliff-DPO variants and their baseline models. We use greedy decoding, a maximum of 1,024 generated tokens, and the official execution verifier. We report mean ± SE across three runs. Bold values indicate higher mean performance than the corresponding baseline model.

Table E. Cliff-DPO evaluations on CruxEval-O, ScienceQA Physics (text-only), and AIME 2025.

ModelCruxEval-O (accuracy, %)ScienceQA Physics, text-only (accuracy, %)AIME 2025 (avg@64, %)
Qwen3-0.6B29.71±0.1168.75±0.802.0±1.04
+ Cliff-DPO (deterministic)29.25±0.0769.44±0.462.7±0.09
+ Cliff-DPO (uncertain)32.75±0.7370.66±1.222.7±1.35
+ Cliff-DPO (sampled-off)32.88±0.4568.23±2.463.7±0.19
Llama-3.2-1B1.75±0.1274.13±0.760.2±0.08
+ Cliff-DPO (deterministic)1.92±0.1574.13±0.630.0±0.02
+ Cliff-DPO (uncertain)1.92±0.1876.22±1.510.1±0.05
+ Cliff-DPO (sampled-off)3.00±0.1974.65±0.920.1±0.05
Llama-3.1-8B12.92±0.4296.35±0.300.7±0.20
+ Cliff-DPO (deterministic)13.12±0.2596.18±0.350.6±0.09
+ Cliff-DPO (uncertain)13.71±0.2297.22±0.170.5±0.03
+ Cliff-DPO (sampled-off)13.67±0.2996.18±0.170.5±0.14

On CruxEval-O, the uncertain and sampled-off variants trained on GSM8K improve mean accuracy in all three models. Llama-3.2-1B's low scores reflect frequent answer-extraction failures. On ScienceQA Physics, the uncertain variant achieves higher mean accuracy than the baseline in all three models. These improvements provide preliminary evidence that the training signals from mathematical cliff tokens may also be useful for code and science reasoning. We will include these evaluation results in the camera-ready version.

Also, we reproduce AIME 2025 results from Appendix N, Table 16. On AIME 2025, the uncertain and sampled-off variants have higher mean scores than the baseline on Qwen3-0.6B, while both Llama models remain near zero. We therefore do not base our claims about the effectiveness of different cliff types on the AIME 2025 results.

We believe our additional experiments and clarifications address your concerns, and we will incorporate them into the camera-ready version. Please let us know if there are any points you would like us to discuss further.

References

[1] Lin et al. (2025). Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM’s Reasoning Capability.

[2] Bigelow et al. (2025). Forking Paths in Neural Text Generation.

Comments on the paper

2 comments

Dear Reviewers,

Thank you for your thoughtful reviews and constructive suggestions, which have helped us improve the quality of our work. We would appreciate any further feedback on our responses and would be happy to clarify any remaining questions or concerns.

Best regards, The Authors

AuthorsOpenReview ↗

Summary of Rebuttal Discussion and Revisions

Dear AC and Reviewers,

We thank the reviewers for their constructive feedback, which has helped us clarify the paper's claims and strengthen its empirical support. We summarize the main results and planned revisions below.

1. Validation and interpretation of cliff detections (Reviewer nxgb)

Independent rollouts reproduce positive average success probability drops at uncertain and sampled-off first cliffs on incorrect traces, while matched non-cliff positions show near-zero change (Table H). Deterministic cliff detections show no such drop, since only the original token survives top-p filtering at all 207 positions; their observed count is also close to that expected under no change (Table I). We have therefore recomputed Figure 2 and Table 1 excluding these detections, reporting observed counts alongside counts expected under no change (Tables N–O), and will apply the same exclusion to the other statistics on cliff occurrence and type proportions, including those in the appendices.

We will present the three-way split as a detector diagnostic, reporting deterministic cliff detections separately, and remove the three-failure-type framing from the abstract and introduction. We will also clarify that the adaptive threshold reduces expected noise-induced detections relative to a fixed threshold of 0.2, but does not control errors from testing many positions (Tables I–J).

2. Training contribution (Reviewers iRTe, wwNE, nxgb)

Under matched preference-pair counts and fine-tuning settings, Cliff-DPO (uncertain + sampled-off) achieves higher mean accuracy than high-entropy token selection in all nine evaluated settings and non-greedy selection in eight of nine (Table M). These comparisons will anchor the training claims in the abstract, introduction, and main results. We will also bring the cDPO comparison into the main text.

3. Positioning relative to prior work (Reviewers wwNE, nxgb)

We will incorporate the same-trace comparison of cliff and critical tokens (Table G), the comparison with PTS-style recursive search (Table K), and a Related Work comparison with Bigelow et al., Zhu et al., and Wang et al. (Table L).

4. Scope, cost, and generalization (Reviewers iRTe, wwNE)

We will add the code and science evaluations (Table E), small-to-large transfer results (Tables C–D), and training-data construction costs (Table B). We will state the substantial rollout cost explicitly as a limitation, and note that cliff tokens transfer little across models, consistent with their model and context dependence, which limits the reuse of preference data across models. The rollout data are publicly released, and restricting rollouts to high-entropy positions and their preceding prefixes (39.7% of prefix positions) is a direction for cheaper detection that we leave to future work.

We will carefully incorporate these results and clarifications into the revised version.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.