- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the phenomenon of failure-to-settle — a behavior in which a language model repeatedly proposes and rejects possible responses, ultimately failing to produce a final answer within the generation budget. This behavior was initially observed in queries for nonexistent emojis; the authors subsequently observed similar behavior when translating into rare languages and when working with various character sets. Based on their analysis, they introduce CrashoutBench—a benchmark containing queries for nonexistent elements from various domains that can trigger failure-to-settle – and evaluate several models with open weights using it.
The authors demonstrate that within the model’s internal representations, there is a direction associated with assessing the existence of the queried object. The paper also shows that steering toward the non-existence of an object significantly reduces the frequency of failure-to-settle. At the same time, steering along the commitment vector—associated with the transition from reasoning to an answer—increases the probability of completing generation but does not guarantee the correctness of the answer. The authors also hypothesize that failure-to-settle may be related to the finiteness of the output space and demonstrate that adding special synthetic tokens can prevent some instances of this failure.
Overall, this work consists of an empirical investigation of how information about the response, the duration of reasoning, and available output representations interact.
Critical strengths
-
Steering toward the commitment vector is supported by a substantial set of experiments. For example, in Section 3.2, commitment steering increases the proportion of completed generations from 32.46% to 86.84%, whereas a control intervention with a comparable degree of representation change yields only 35.09%. The authors also test the robustness of the effect to various decoding parameters and demonstrate that commitment influences the completion of generation, rather than merely directly increasing the probability of the response’s ending token.
-
The paper explores a wide range of scenarios beyond the emoji examples in which failure-to-settle is possible. For example, the authors consider 25 different character sets. They also account for various factors affecting the failure rate: decoding parameters and the observation window length. Furthermore, they experiment with the output vocabulary and conduct control experiments using hidden states and random noise.
Critical weaknesses
-
In experiments with synthetic tokens (Appendix A.6), the authors mix hidden states from crashout trajectories with states obtained by explicitly instructing the model to select a replacement for a missing emoji. However, the case where α = 0 is not considered. Therefore, it remains unclear whether the improvement is due to the reconstruction of the missing output representation or to the use of hidden states from a scenario where the model proposes a replacement and is ready to complete generation.
-
In experiments with synthetic tokens (Table 13), nearly all of the improvement is attributable to two concepts—seahorse and lighthouse. They account for 81 out of 88 cases of a net reduction in the number of crashouts. For the remaining eight concepts, the crashout rate remains virtually unchanged on average and sometimes even increases. Furthermore, the model often interprets the added tokens as other objects. Therefore, it is not yet clear to what extent the proposed method for restoring output representations is applicable beyond individual successful examples.
-
The authors define a crashout as the absence of a final response within a fixed token budget. However, this definition does not allow us to distinguish true looping from other cases of prolonged reasoning. Furthermore, a reduction in the crashout rate does not always indicate an improvement in the model’s behavior: for example, in Table 3, strong positive steering reduces the crashout rate to 7%, but results in 93% incorrect answers. The authors acknowledge these limitations and partially investigate the correctness of responses in Appendix A.17.4; however, it would be useful to conduct a more systematic evaluation of the correctness of responses and the nature of incomplete generations on the main benchmark.
Questions for authors
- Did you test generating synthetic tokens exclusively from crashout states (α = 0)? If not, why not?
- Do you have any hypotheses as to why synthetic tokens work effectively for Seahorse and Lighthouse but are practically useless for the other eight concepts?
- In Section 3.2, the AUC value of 0.863 appears to be attributed to the commitment score itself; however, in Appendix A.17.2, it is obtained for a classifier with additional features. What is the predictive power of the commitment score without additional features?
- Section 4.2 states that a random noise token is not selected as the final answer, yet Table 30 shows an example where <RANDOM NOISE> appears precisely in the final answer. How is this discrepancy explained?
3 comments
Part 2
- Reducing crashout does not improve correctness. We agree with the reviewer’s point that less crashing out does not imply more correct answers. Our goal is not necessarily to directly mitigate crashout, but to understand it. We were initially surprised to see that the positive and negative directions of the same “existence” vector both materially reduced crashout (without devolving to incoherent generations). What we found was that the positive direction caused the model to quickly settle on an incorrect answer and the negative direction cause the model to quickly settle on a correct “does not exist” answer. This aligns with the meaning behind each vector direction. The negative direction corresponds to non-existence, thus terminating outputs with "does not exist". The positive direction corresponds to existence, and for sufficiently high magnitudes, it may compel the model to believe in a false candidate to the degree it confidently gives an incorrect answer, and as a result also prevent crashout.
More specific questions:
-
AUC value in predictor. Thanks for catching this. The reported AUC of 0.863 belongs to the combined classifier, not the commitment score alone. We have now evaluated the commitment score alone on the same held-out set: its AUC is 0.890 for predicting a reasoning-to-final transition within 32 tokens. We will correct the attribution and include this standalone result.
-
Random noise token selection. Thanks for flagging this. This is outdated writing (the model does occasionally settle on the random unembedding, see also Table 18 & Table 31), and we will update this in the manuscript.
Thank you for the detailed and transparent response. I appreciate that you engaged with each point directly, and I understand the compute-access situation.
Several clarifications are helpful. The correction of the AUC attribution is important, and the standalone commitment score (0.890) is actually a stronger result than the combined classifier. The clarification of the crashout definition and the additional audit of long rollouts make the framing more precise, and I appreciate the acknowledgment that the random-noise statement was outdated writing. The explanation of why synthetic tokens work for seahorse and lighthouse but not for other concepts is plausible and useful, and I agree that the goal is understanding rather than solving crashout.
My main remaining reservation is the α=0 experiment, which is not resolved. Since this was the key control for separating the semantic signal from a generic stop signal, I would like to see it in the revision. The hypotheses about synthetic-token failures are reasonable but still untested.
Overall, the response addresses several of my concerns and improves my confidence in the paper's framing, but the central control is still missing. I therefore keep my rating at 3 (weak acceptance), and I encourage the authors to include the α=0 results and a more systematic correctness evaluation in the final version.
Part 1
Thank you for your support of our work. Transparently, we have a temporary and unexpected issue preventing access to our normal compute resources (and some of our experimental results), so we have been limited in our ability to run additional experiments during the rebuttal period. Nevertheless, we have done our best to address each of your points in detail below:
Crashout state sweep. This is a good question and the appendix is a bit unclear. We will fix the writing here. Since we don’t currently have access to our results, we can’t answer this question about the full crashout/non-crashout representation combination definitively. That said, our recollection is that using crashout states only (alpha=0) showed similar crashout prevention behavior (strong for seahorse and lighthouse, weak for other emojis). We will update the paper to include all of these results when we regain access to the data. Your concern about confounding the “stop” signal with the semantic signal is valid, and we started using the convex combination in hopes of leveraging this generic stop signal. However, the model frequently selects the synthetic tokens (e.g. the “tapir emoji”) without stopping, implying that a generic stop signal is not sufficient. In our experiments, it seems that the model must correctly recognize the synthetic token to reliably prevent crashout.
Hypothesis for synthetic token failures. Our main hyoptheses as to why some synthetic tokens work while others do not are: i) the Gaussian approximation we use to map to the unembedding spaces is probably more valid for some semantics than others ii) naively reinserting an unembedding as an embedding probably works better for some semantics than others. Unembeddings live in a fundamentally different space than the context embeddings (they can be thought of as linear functionals outputting a similarity score). We should not expect simply using an unembedding vector as an embedding vector to work out of the box; tied models tend to learn representations closer to a separately trained output matrix than an input matrix (Press & Wolf, 2017). So, it is somewhat surprising that it would work at all to simply reinsert a synthetic unembedding as a next-token embedding. We also see that changing the geometry of the mapping by including more tokens to calculate our covariance matrix changes behavior. Still, when the synthetic unembedding fails, it fails in a “reasonable” way: models generally think that the synthetic token is something semantically similar (e.g. “trolley” for “stroller”) and continue the self-correction loop. Additionally, our goal with the synthetic unembeddings was not to solve crashout (since this method is clearly unreliable), but to better understand it. In at least some cases (seahorse and lighthouse), a synthetic token that is interpreted correctly by the model is sufficient to stop crashout, which is consistent with the hypothesis that crashout is a mismatch between the model’s “beliefs” and its actual outputs.
Distinguish true looping. Our definition for failure-to-settle as a failure mode simply considers whether the model is able to terminate within a reasonable budget, which requires making a judgement call as to what a “reasonable budget” is. While a model may not be in a “true loop,” we do think that it’s unreasonable to generate 10,000 tokens just to ultimately say that a nonexistent emoji does not, in fact, exist. However, we did examine longer token horizons in Appendix A.5. We ran an additional audit using GPT-6.1-Terra, of the 10.2% of rollouts that terminated between 1024 and 8192 tokens, our audit found that only 9.1% correctly acknowledged that the requested emoji does not exist, we will include these results in the final manuscript. Also, we don’t think prolonged reasoning is necessarily always a problem. We wouldn’t consider a model trying to solve Navier-Stokes to have crashed out after 1024 tokens. The instances of failure-to-settle that we study are related to a broader class of problems where models don’t manage their reasoning budget well (e.g. struggling with simpler math problems after being trained to do well on hard ones due to prolonged reasoning efforts.
References:
Ofir Press and Lior Wolf. "Using the Output Embedding to Improve Language Models." EACL 2017 (arXiv:1608.05859).