- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper looks at what happens when a model runs out of its token budget before writing a final answer. The proposed fix is simple: if the model is still generating about 130 tokens before a hard output limit, the text "Final answer:" is inserted and the model continues, leaving room for the answer. This is compared with letting the same run continue to the limit. Both versions share the same prompt and the same sampled text up to the insertion point, and the inserted tokens count against the budget, so any difference comes only from runs still going at that point.
The main experiments use Qwen3.5-9B and Ministral-3-8B in their direct (non-thinking) mode, mostly with a 2048-token limit, on MATH and GPQA-Diamond, with three sampled runs per question. An answer counts only if the model stops on its own and gives a complete boxed answer that the scorer accepts. In a preregistered study on 200 unseen MATH questions, the cue improves correct answers by about 7 points for both models at 2048 tokens and by 4.5 points at 4096. On GPQA-Diamond the gains are about 20 points for Qwen and 4.5 for Ministral. The cue also uses slightly fewer tokens overall.
The paper also compares the cue with an adapted version of Sun et al. (2025)'s concluding message and with budget-aware system prompts, and tests which parts of the cue matter. It studies how much room to leave for the answer: a short reserve (64–128 tokens) works better than a long one (512) when compared against letting the model continue. Continuing the original run also works better than discarding it and restarting in thinking mode. Simple accounting identities explain why only unfinished runs can change and why the choice of baseline affects these conclusions.
The evidence includes a development study, a fixed-policy rerun, and several preregistered follow-up studies, analysed with question-level bootstrap intervals and Holm correction.
Critical strengths
-
The experiments are set up carefully. The version with the cue and the version without it start from exactly the same text, and the extra cue words count against the same token limit, so any difference can only come from runs that were still going when the cue was added. The authors check this instead of assuming it. They also repeat the main experiment on new questions that were fixed in advance, and get about the same gain. This makes the main result easy to believe.
-
The paper makes a useful point about how to compare methods. If you compare the cue against a run that was cut off at the same point, leaving a lot of room for the answer looks best. If you compare against a run that was allowed to continue to the full limit, leaving a small amount of room works better, and a follow-up test on new settings confirms this. That is a good lesson for anyone evaluating methods that stop or shorten model outputs.
Critical weaknesses
-
The idea is not new and the gain is narrow. Forcing a final answer near the token limit is already used in s1 budget forcing, Ma et al. (2025) and Sun et al. (2025). When Sun et al.'s message is placed at the same position as the proposed cue, it comes within about two points on MATH, and slightly beats the cue at the 4096-token limit, so most of the advantage comes from leaving more room for the answer rather than from the cue itself. The gain also only comes from runs that would otherwise run out of tokens, so it shrinks as the limit grows. The main 2048-token limit is far below the 32k the model card recommends, and at 2048 tokens the cue does about as well as plain generation with twice the limit. The main comparison also runs with thinking turned off, which doesn't match the "reasoning" framing of the title.
-
Some results are stated more strongly than the evidence supports. On GPQA-Diamond the cue mostly turns runs with no answer into a forced letter, and wrong answers go from 21 to 184. On four-option questions, giving a random letter to those runs would already recover about three quarters of the reported gain. Several preregistered tests also did not hold: the two prose "time's up" cues showed no gain, the short-versus-long reserve test on GPQA with Ministral found no difference, and the first rerun's planned analysis did not meet its own decision rule. The main text and abstract report only the tests that passed. Since the paper relies on preregistration for credibility, every outcome should be stated in the main text.
Questions for authors
Clarifications:
- How does the cue compare with simply raising the token limit, at the same number of generated tokens? What happens at the model card's recommended budget, where few runs hit the limit?
- On GPQA-Diamond, what is the gain over giving a random letter to runs with no answer, or after a standard guessing correction?
- Can the failed preregistered tests be reported in the main text? And why is the first rerun called a replication when its planned analysis did not meet its decision rule?
- Does a late cue also help in thinking mode, for example closing the thinking block and forcing "Final answer:" near the limit?
- Does a lower presence penalty change how often runs hit the limit, and so the size of the gain?
Suggestion (future work): choosing the cue position adaptively, based on whether the run looks unlikely to finish in time.
6 comments
Follow-up for Reviewer AjkK: presence penalty, late cue in thinking mode, and larger caps
Here are the three experiments you asked about: a lower presence penalty, a late cue in thinking mode, and larger caps. All use Qwen3.5-9B on the same 200 MATH questions and three sampling streams as the new experiment of our first reply, with 600 runs per arm. Counts are correct deliveries out of 600, gains are in points, and brackets are 95% question-level bootstrap intervals.
Presence penalty (direct mode, cap 2048). The cap-hit rate changes little with the penalty, and the cue gains at every penalty. Without the cue, continuation reaches the cap in 114 of 600 runs (19.0%) at penalty 0, 124 (20.7%) at penalty 1 and 131 (21.8%) at penalty 2, and 122, 132 and 141 runs are still generating at the cue. The cue gains +5.67 [3.67, 8.00] at penalty 0, with 36 gains and 2 harms, +8.00 [5.17, 11.17] at penalty 1, with 49 and 1, and +8.00 [5.33, 10.83] at penalty 2. Every run whose outcome changes was still generating at the cue.
Late cue in native thinking mode. A late cue helps in thinking mode at every cap. Thinking runs uninterrupted to 8192 tokens and is cut at each cap. The late cue is a six-token close-and-answer cue forced 134 tokens before the cap, only in runs whose thinking is still open there.
| Cap | Uninterrupted thinking | With the late cue | Gain | Thinking open at the cue |
|---|---|---|---|---|
| 2048 | 90 | 391 | +50.17 [44.50, 55.67] | 496 runs (82.7%) |
| 4096 | 206 | 439 | +38.83 [33.67, 44.17] | 368 runs (61.3%) |
| 8192 | 360 | 495 | +22.50 [17.83, 27.33] | 216 runs (36.0%) |
The cue turns 301, 233 and 135 of the open runs into correct deliveries and harms none. The gain follows the share of runs whose thinking is still open at the cue. The paper's early close, at token 1024 with cap 2048, delivers 377 on the same streams, against 391 for the late cue (+2.33 [-2.17, 7.00]).
Larger caps (direct mode, penalty 2). Continuation is one stream run to 16384 tokens and cut at each cap. The cue is placed 132 tokens before each cap.
| Cap | Continuation: correct, mean tokens | Cue: correct, mean tokens | Runs still generating at the cue | Gain |
|---|---|---|---|---|
| 2048 | 441, 1027 | 487, 1001 | 143 | +7.67 [5.17, 10.33] |
| 4096 | 491, 1369 | 520, 1356 | 80 | +4.83 [3.00, 7.00] |
| 8192 | 535, 1709 | 544, 1705 | 25 | +1.50 [0.67, 2.50] |
At cap 16384 every run has finished before the cue position, so the cue is never applied and both arms deliver 549 of 600 with 1775 tokens. The gain is bounded by the number of runs still generating at the cue (Prop. 2), and the cue generates no more tokens than continuation at any cap. The cue at cap 2048 reaches 487 correct deliveries with 1001 mean generated tokens, where continuation needs cap 4096 and 1369 tokens to reach 491.
Thank you for the detailed response, and for running every experiment I asked about. I went through the revised PDF.
A lot is fixed now. The equal-reserve comparison answers my question about where the gain comes from: your cue and the Sun et al. message do the same at every reserve, and the abstract now says this. Table 4 shows every registered test, including the ones that were not met, and the first rerun is no longer called a replication. Section 6.1 gives the random-letter baseline for GPQA-Diamond. Section 6 is also easier to read in three parts.
But my main concern is still there, and the new results make it clearer.
First, if the wording does not matter, then the intervention is the same as in prior work. What is left is the evaluation setup and the reserve finding. I find both useful, but it is a narrow contribution.
Second, in direct mode the gain goes from 7.67 points at 2048 tokens to 1.50 at 8192, and to zero at 16384. That is still only half of the 32768 tokens the model card gives as typical.
Third, the thinking-mode gains look large, but the baseline there is weak. At every cap, thinking with the cue (391, 439 and 495 of 600) is still below plain direct mode with no cue (441, 491 and 535). Also, the late close and the early close do about the same (391 vs 377), so the reserve finding is only shown in direct mode so far.
One small thing: the abstract still gives +20.20 on GPQA-Diamond without the guessing baseline. Only about 5 points of that are above random letters, so the abstract should say it.
I am raising Soundness to 3, since the reporting issues are fixed. I am keeping my rating at 2 because the contribution is still narrow.
Response to Reviewer AjkK
Thank you for the detailed review. We answer your questions in order. For the first one we ran a new experiment: the 200 MATH questions of the paper's first repeat study, three new sampling streams, the same 2048-token cap, on Qwen3.5-9B and the larger Qwen3.8-27B, with our cue and the Sun et al. message at equal answer reserves of 64 to 512 tokens and 600 runs per arm.
"most of the advantage comes from leaving more room for the answer rather than from the cue itself" At equal reserve we find no delivery difference between the two cues: the gain comes from a late cue with an adequate answer reserve, under either wording. Cue minus message is +0.33, +0.83, -0.33 and +0.50 points for 9B and -0.33, +1.00, +0.17 and +0.50 for 27B, and every 95% interval lies within [-1.67, +2.17]. At equal reserve the message starts 10 tokens before our cue, so this compares two policies with the same reserve. The revised PDF reports this in Sec. 6.2, Table 5 and Appendix Table 36.
On the gain shrinking as the limit grows: Prop. 2 limits changes to runs that are still generating at the cue, and this is not the same set as the runs that finally hit the cap. Their share can be measured from plain continuation alone and it bounds the gain. Within that share, the converted fraction varied from 23% to 49% across the settings of Table 7. In the new experiment 27B has 121 of 600 such runs against 141 for 9B, and gains +5.33 [3.33, 7.50] against +8.00 [5.33, 10.83].
"How does the cue compare with simply raising the token limit, at the same number of generated tokens?" The cue at cap 2048 comes close to continuation at cap 4096 and uses fewer tokens. On the questions of the second repeat study, the cue at cap 2048 delivers 507 of 600 (84.50%) with 951.8 generated tokens per run, and continuation at cap 4096 delivers 509 (84.83%) with 1226.1, so the cue uses 22.4% fewer generated tokens. At cap 4096 the cue adds a further +4.50 [2.50, 6.83]. Our follow-up comment below gives continuation up to 8192 and 16384 tokens.
At larger budgets the cue keeps a positive gain: +7.67 [5.17, 10.33] at cap 2048, +4.83 [3.00, 7.00] at 4096 and +1.50 [0.67, 2.50] at 8192, and it generates no more tokens than continuation at any cap. The details are in the follow-up comment below.
"Does a late cue also help in thinking mode" In thinking mode the paper tests an early close: forcing the close-and-answer suffix at token 1024 gains +45.79 points [40.07, 51.52] over uninterrupted thinking at the same 2048 cap. The late close is in our follow-up comment below.
On GPQA-Diamond against a random letter: uniform guessing on the 360 runs where continuation delivers no answer gives 303 expected correct answers. The cue delivers 333, which is +5.05 points above that. With a penalty of 1/3 per wrong letter the gain is +11.05 points, and with a penalty λ the cue is ahead for λ < 120/163 ≈ 0.736.
On reporting the tests that were not met: yes. Table 4 of the revised main text lists every planned test with its outcome. The paper no longer calls the first study a replication, and its +7.41 under the delivery contract is reported as a separate analysis. The second study has delivery as its primary endpoint, and its four primary tests are all met.
On the presence penalty: the experiments used presence penalty 2 in direct mode. Penalties 0 and 1 are in our follow-up comment below.
Choosing the cue position adaptively is a natural extension of the fixed position studied in the paper.