acceptodds
Under review as a conference paper at ICLR 2027

Thinking inside the brackets: Grammar-Constrained Decoding

Abstract

LLM reasoning is a core driver of model performance, but it is costly. Practitioners typically use prompting and fine-tuning as axes of control to reduce the thinking-verbosity of reasoning models. We identify and exhibit grammar-constrained decoding (GCD) as a third axis of control. We structure reasoning traces by a small context-free grammar at decode-time (e.g. open a subgoal, derive, add a fact, check, close) which is compatible with free-text payloads. On GPQA-Diamond and AIME24, GCD on Qwen3.6-27B reduces median generated tokens by 98% and 94%, while holding or improving accuracy over chain of thought. While GCD reliably makes reasoning succinct, we further identify and characterise cases where controlling reasoning is detrimental, or ineffective.

Primary area:
foundation or frontier models, including LLMs
Keywords:
constrained decoding, chain of thought, reasoning traces, test-time compute, grammar-constrained generation, domain-specific language, structured reasoning, token efficiency, inference-time control, LLM harnesses, non-termination, prompt engineering, small open models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fzdzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces grammar-constrained decoding (GCD) for compact reasoning traces. A context-free grammar assigns explicit operators to goals, subgoals, derivations, checks, repairs, and closure, while leaving each line’s natural-language payload unconstrained. A pushdown automaton masks illegal next tokens during decoding. The authors evaluate GCD, Chain-of-Draft prompting (P), ThinkingCap fine-tuning (FT), and their combinations on four benchmarks using 8-bit Qwen3.6-27B. They report 98% and 94% reductions in median generated tokens on GPQA-Diamond and AIME24, with respective accuracy gains of 5 and 3 percentage points over CoT. On MedCalc-Bench and IFBench, GCD cuts generated tokens by 90% and 80% but loses 13 and 25 accuracy points. Budget curves favor GCD at small token budgets but show a lower plateau. The paper acknowledges regressions, limited sampling, a single model and grammar, and token-cap effects.

Critical strengths
  1. The representation is simple and inspectable: online syntax enforcement preserves readable free-text reasoning and explicit control-flow markers.
  2. The authors compare intervention combinations across four task types and report negative results, giving a more useful account than a single favorable benchmark.
Critical weaknesses
  1. The empirical evidence does not establish the marginal contribution of grammar masking. Section 3 says GCD includes a prompt, and Appendix A supplies roughly 870 words of prescriptive instructions: short payloads, no deliberative narration, one independent check, bounded repair, and early stopping. These instructions directly target trace length and stopping behavior. The P baseline is described only as a short Chain-of-Draft prompt; the cited CoD formulation uses a nonbinding five-word-per-step guideline. The paper does not report the same GCD skill prompt with the decoder mask disabled. Consequently, the reported token and accuracy changes support the combined prompt-plus-mask package, but do not show that the mask itself supplies the claimed third axis or is needed for the gains. This is central to the paper’s main empirical claim.
  2. The key gains are not accompanied by uncertainty estimates. The authors acknowledge limited sampling; AIME24 uses two samples per question, while the reported gain is 3 points. No question-level paired or clustered intervals, multiple seeds, or variability analysis are shown. The results should therefore be treated as point estimates in a single-model, single-grammar study.
Questions for authors
  1. Can you report a matched ablation that retains the exact Appendix A skill prompt and settings but disables the decoder mask? Please also provide the exact P prompt and state whether it uses the few-shot examples in the original CoD setup. Would the token and accuracy gains persist under these controls?
  2. Can you report question-level uncertainty for GPQA-Diamond and AIME24, clustering the two AIME samples by source question, and clarify whether any wall-clock latency was measured?
2 comments
AuthorsOpenReview ↗

Reply to Reviewer fzdz

Reply to Reviewer fzdz

Thank you for your review. We have marked every change in the revised PDF (new text is blue and removed text is struck in red) and kept changes minimal.

1. "The paper does not report the same GCD skill prompt with the decoder mask disabled."

It does now. We ran this control on all four benchmarks, using the exact prompt from Appendix A with the same weights, generation settings, token caps and questions; only the mask was switched off. We ran GCD and this control three times each on GPQA-Diamond, MedCalc-Bench and IFBench, and twice on AIME24 (new Table 2 in the paper; Appendix C lists each run).

Benchmark (items)CoTGCD prompt, mask offGCDMask's further cut in tokens
GPQA-Diamond (198)69.7%, 19,120 tokens78.1% [73.6, 82.8], 3,476 tokens, 1.2% parse74.7% [69.7, 79.5], 461 tokens8×
AIME24 (30)68.3%, 26,10673.3% [60.0, 86.7], 16,515, 0.0% parse71.7% [58.3, 85.0], 1,64210×
MedCalc-Bench (300)83.3%, 1,82983.2% [79.1, 87.1], 1,899, 0.1% parse71.7% [66.7, 76.4], 2019×
IFBench (233)63.5%, 4,39044.9% [39.6, 50.2], 4,593, 3.6% parse47.4% [41.5, 52.9], 29116×

Reported accuracy is the mean over runs with a 95% bootstrap interval over questions; on AIME24 the interval resamples problems, keeping both draws of each together. Token counts are median generated tokens. "Parse" is the share of traces that satisfy the grammar rules with the mask off. MedCalc-Bench uses a 300-question subset selected a priori; CoT reports the submission's single run on the same questions.

The last column gives the mask's marginal contribution, a further 8 to 16 fold cut in median tokens beyond the prompt's reduction. The prompt alone does not produce the format: only 0 to 3.6% of its traces parse, and it saves no tokens on MedCalc-Bench or IFBench. The third axis controls cost, and our claims match: 98% and 94% fewer tokens on GPQA-Diamond and AIME24, at accuracy statistically indistinguishable from CoT. We find no accuracy gain from the mask. With it off, the prompt is 3.4 points above GCD on GPQA-Diamond (+5.1, −1.5 and +6.6 in the three runs; none is significant by McNemar's test, smallest p = 0.06, although the last bootstrap interval, [0.5, 13.1], excludes zero) and 1.7 points above on AIME24 (interval [−10.0, 13.3]). On MedCalc-Bench, the prompt is 11.6 points above GCD, significantly so in every run.

2. "Please also provide the exact P prompt and state whether it uses the few-shot examples in the original CoD setup."

Appendix B gives the exact prompt: "Think step by step, but only keep a minimum draft for each thinking step, with 5 words at most." It is the first sentence of the Chain of Draft system prompt, used zero-shot; Section 3 now records this. The prompt-alone arm is the matched control for the mask, so no conclusion about the mask depends on P. We state in Limitations that we chose the P + GCD prompt from two candidates after seeing their AIME24 results (66.7% against 50.0% at a 16K cap). The choice favours the combination, which still scores below P alone on AIME24 (71.7% against 78.3% at the 32K cap), so the finding that GCD composes poorly with prompting does not depend on it.

3. "Can you report question-level uncertainty for GPQA-Diamond and AIME24, clustering the two AIME samples by source question?"

Paired by question, GCD minus CoT is +5.1 points on GPQA-Diamond (95% interval [−1.5, 11.6], McNemar p = 0.17), and +6.6 [−1.0, 14.1] and +3.5 [−3.5, 10.6] in the two further runs. On AIME24 it is +3.3 [−10.0, 16.7], with the two draws clustered by problem (p = 0.71). The intervals are 95% bootstraps over questions. Because they include zero, we replaced "increasing accuracy" with "at accuracy indistinguishable from CoT". The three GCD runs on GPQA-Diamond score 74.7, 76.3 and 73.2. Table 2 gives the intervals, and Appendix C lists every run with its paired test.

4. "Clarify whether any wall-clock latency was measured."

We measured decoding throughput, not end-to-end latency. In a matched throughput test, the mask cost about 2% of tokens per second on a single sequence and 8 to 10% at batch sizes of 8 to 16. Since the mask cuts generated tokens 8 to 16 fold beyond the prompt alone, and more against CoT, the token saving is also a saving in decoding time.

Thanks, the author answered my question and I decided to improve my rating.

Reviewer_ZRzLOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the high latency and cost of verbose Chain-of-Thought traces in Large Reasoning Models by proposing Grammar-Constrained Decoding (GCD). GCD is a training-free, decoder-level intervention that uses a pushdown automaton to enforce a strict structural grammar, forcing the model to replace natural language discourse markers with symbolic shorthands during reasoning. The authors contribute the GCD method alongside a systematic comparison of three length-control axes: decoding constraints, prompting, and fine-tuning. Evaluated on Qwen3.6-27B across GPQA, AIME24, MedCalc, and IFBench, GCD reduces token usage by 80–98%. The results show that GCD maintains or improves accuracy on complex tasks that benefit from extended reasoning, but significantly degrades accuracy on tasks where CoT saturates early, and it composes poorly with prompting and fine-tuning. The authors diagnose this accuracy drop as a consequence of stripping away necessary arithmetic or constraint-checking tokens on simpler tasks, but they do not propose a mechanism to dynamically adjust or prevent this reasoning degradation. Instead, they frame the failure as a known trade-off, recommending that GCD only be applied to tasks that exhibit long test-time scaling.

Critical strengths
  1. A practical, training-free method with massive token reductions. GCD achieves 80–98% token compression with zero fine-tuning. Because it requires no weight updates and integrates directly at the decoding stage, it offers an immediately deployable, plug-and-play solution for reducing the high latency and cost of Large Reasoning Models.

  2. An honest and insightful empirical characterization. The paper rigorously identifies when and why GCD works (on tasks with long test-time scaling) versus when it fails (on tasks where CoT saturates early). This "saturation hypothesis" provides a valuable diagnostic rule for the community, and the finding that GCD interferes with prompting and fine-tuning saves future researchers from costly trial and error.

Critical weaknesses
  1. The method is a "fair-weather" tool that catastrophically degrades accuracy on half the benchmarks without offering a technical fix

  2. The core technical novelty is confounded by heavy prompt engineering, and the paper lacks the ablations needed to prove the decoder constraint is actually the active ingredient.

Questions for authors

First, I would like to know if you have isolated the effect of the decoding constraint from the prompt itself. Your method relies on a very detailed "skill document" prompt. Did you run a baseline where the model receives that exact same prompt but without the pushdown automaton masking the output? If you did, how often did the model follow the rules on its own, and how much of the token reduction was preserved? If the prompt alone does most of the work, it undermines the claim that the decoder-level constraint is the main novelty.

Second, regarding the accuracy drops on MedCalc and IFBench, could you clarify exactly why the reasoning breaks down? Is the strict grammar forcing the model to close its current reasoning scope too early—before it has finished doing the necessary math or checking constraints? A brief example of a trace that went wrong would help illustrate this failure mode. If the grammar's rigidity is the issue, a simple tweak might fix it, whereas if the model simply needs more tokens for these tasks, then your "saturation" hypothesis holds.

Third, you cite CRANE in your related work, which also noticed that strict constraints hurt reasoning and solved it by alternating between free and constrained generation. Could you clarify how your approach fundamentally differs from CRANE, beyond just applying the constraint to the reasoning trace instead of the final answer?

As a suggestion for future work, your grammar includes many different symbols. It would be very helpful to know which specific operators are actually driving the token savings. For example, if just a couple of them are doing all the heavy lifting, you might be able to simplify the grammar, reduce the cognitive load on the model, and potentially fix the accuracy drops.

Finally, since you correctly identified that GCD struggles on tasks where the model does not need long reasoning, a natural next step would be to develop an adaptive mechanism. For instance, you could let the model think freely for the first few hundred tokens and only enforce the strict grammar later, which would help solve the limitation of needing to know the task type in advance.

3 comments
AuthorsOpenReview ↗

Reply to reviewer ZRzL (1/2)

Reply to Reviewer ZRzL

Thank you for your review. We have marked every change we have made in the revised PDF (new text is blue and removed text is struck in red) and kept changes minimal.

1. "Did you run a baseline where the model receives that exact same prompt but without the pushdown automaton masking the output? ... how often did the model follow the rules on its own, and how much of the token reduction was preserved?"

Yes. We ran this control on all four benchmarks, using the identical prompt with the same weights, generation settings, token caps and questions; only the mask was switched off. We ran GCD and this control three times each on GPQA-Diamond, MedCalc-Bench and IFBench, and twice on AIME24 (new Table 2 in the paper; Appendix C lists each run).

Benchmark (items)CoTGCD prompt, mask offGCDMask's further cut in tokens
GPQA-Diamond (198)69.7%, 19,120 tokens78.1% [73.6, 82.8], 3,476 tokens, 1.2% parse74.7% [69.7, 79.5], 461 tokens8×
AIME24 (30)68.3%, 26,10673.3% [60.0, 86.7], 16,515, 0.0% parse71.7% [58.3, 85.0], 1,64210×
MedCalc-Bench (300)83.3%, 1,82983.2% [79.1, 87.1], 1,899, 0.1% parse71.7% [66.7, 76.4], 2019×
IFBench (233)63.5%, 4,39044.9% [39.6, 50.2], 4,593, 3.6% parse47.4% [41.5, 52.9], 29116×

Reported accuracy is the mean over runs with a 95% bootstrap interval over questions; on AIME24 the interval resamples problems, keeping both draws of each together. Token counts are median generated tokens. "Parse" is the share of traces that satisfy the grammar rules with the mask off. MedCalc-Bench uses a 300-question subset selected a priori; CoT reports the submission's single run on the same questions.

On its own, the model follows the grammar in 0 to 3.6% of traces. The prompt retains part of the token reduction on GPQA-Diamond (82%, against 98% for GCD) and AIME24 (37%, against 94%). On MedCalc-Bench and IFBench, it retains none and uses as many tokens as CoT. The mask cuts tokens a further 8 to 16 times on every benchmark, more than the prompt achieves on any of them, so the decoder constraint is the active ingredient for cost. We find no accuracy gain from it. With the mask off, the prompt is 3.4 points above GCD on GPQA-Diamond (+5.1, −1.5 and +6.6 in the three runs; none is significant by McNemar's test, smallest p = 0.06, although the last bootstrap interval excludes zero), 1.7 points above on AIME24 (interval [−10.0, 13.3]) and 11.6 points above on MedCalc-Bench.

2. "Could you clarify exactly why the reasoning breaks down? Is the strict grammar forcing the model to close its current reasoning scope too early? ... A brief example of a trace that went wrong would help."

In a new run we analysed manually, early closure accounts for 2 of the 51 MedCalc-Bench items that GCD gets wrong (of 300) and CoT gets right. The remaining errors comprise 27 miscounted checklists, 17 wrong formulas or constants, 2 sign errors, 2 arithmetic slips and 1 other error. Across the three runs in Table 3, GCD loses 47, 46 and 47 such items. Appendix C gives one trace in full. In that trace, the model computes a Glasgow-Blatchford score, assigns 5 points to a systolic pressure below 90 mmHg where the table assigns 3, and totals 18 against a gold answer of 16; its closing check restates the sum without recomputing it. The same prompt without the mask writes out the table and answers 16 in 2,260 tokens. So it appears that the compact format suppresses the working that these items require. The model rarely closes a scope too early, and token limits are not an active constraint. The free-prefix run in point 5 is consistent with this reading: accuracy rises by 6.0 to 7.3 points when the model writes 300 free tokens before applying the grammar.

On IFBench, the loss is already present with the mask off: the same prompt scores 44.9% against 63.5% for CoT, so the prompt is the likely cause. Its rules require the model to omit the answer format from the trace and to write the answer after the trace closes, where nothing checks it. Between 45 and 47% of GCD's finished answers have 40 words or fewer. The submission's IFBench GCD cell used an earlier version of the prompt, without the forty-word limit; the revision's IFBench cells use the Appendix A prompt, and GCD trails CoT there by 16 points where the submission reported 25.

AuthorsOpenReview ↗

Reply to Reviewer ZRzL (2/2)

3. "A 'fair-weather' tool."

Section 4 now gives practitioners a choice they can make before running anything. GCD suits long, open-ended reasoning towards a short answer: on GPQA-Diamond and AIME24 it removes 98% and 94% of the tokens at accuracy statistically indistinguishable from CoT. The prompt alone or CoT better serves exact procedural work such as MedCalc-Bench. CoT remains best for tasks graded on the form of the response, such as IFBench. This rule takes the place of the saturation reading: the budget curves count where finished traces fall and cannot carry a causal reading, so we state the heuristic by task type, as an observation over four benchmarks.

4. "Could you clarify how your approach fundamentally differs from CRANE?"

The two address different problems. CRANE targets outputs that must be well-formed in a formal language (arithmetic expressions on GSM-Symbolic and first-order logic on FOLIO). Prompted by instructions and few-shot examples, the model opens each constrained window itself, and that window contains formal syntax. CRANE aims for validity and accuracy; in every row of its Tables 1 and 2, it generates as many tokens as chain of thought or more. GCD needs no task-specific output grammar. It enforces one task-agnostic grammar over the discourse structure of the thinking span from the first thinking token, while the payloads remain natural language; closing the root scope ends thinking. The same grammar applies to multiple-choice, integer, clinical and free-form writing tasks, and GCD aims to reduce cost. The control above shows why the decoder must enforce the grammar: left to the prompt, 0 to 3.6% of traces follow it. The related-work paragraph now gives this account in place of "we do the reverse".

5. Your two suggestions.

We did not ablate the operators. We report this under Limitations and evaluate the grammar as a single design. We conducted a brief analysis based on your second suggestion: in a run on the 300-item MedCalc-Bench subset, the same prompt scores 77.7% at a median of 469 tokens with the mask engaged after 300 free tokens. That is 7.3 points above the GCD run we read by hand (95% interval [2.7, 12.0], McNemar p = 0.004) and 6.0 above the mean of the three GCD runs in Table 3, at 2.3 times GCD's tokens. It is 5.7 points below CoT ([−9.3, −2.0], p = 0.006) at about a quarter of CoT's tokens. Appendix C includes analysis of this result. In Further work, we leave open whether a free prefix removes the need to know the task type in advance.

AuthorsOpenReview ↗

(reminder)

.

Reviewer_2r4FOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether you can shorten reasoning traces by controlling their format at decoding time. The authors design a grammar that organizes reasoning into bracketed goals and subgoals, with symbols for steps like deriving a result, adding a fact, checking, and revising. A prompt explains the format to the model, and decoding constraints make sure it's followed. They test this on Qwen3.6-27B and its ThinkingCap fine-tune, comparing it against ordinary chain-of-thought, concise prompting, and combinations of the two across four benchmarks. On GPQA-Diamond and AIME24, median generated tokens drop by 98% and 94%, and accuracy even goes up slightly. The picture is worse on MedCalc-Bench and IFBench, where accuracy falls substantially, and stacking the grammar constraints on top of other shortening methods often makes things worse. So the method is a practical way to cut reasoning length, but how well it works depends a lot on the task.

Critical strengths
  1. It's a simple method that doesn't need any extra training. It constrains the structure of the reasoning but leaves the content mostly in natural language. The grammar, state diagram, and worked example make it easy to follow. The token reductions are large, which makes this a promising way to cut the cost of long reasoning traces.
  2. The evaluation is honest about both what works and what doesn't. The authors compare constrained decoding against prompting and fine-tuning, and against combinations of these, not just against plain chain-of-thought.
Critical weaknesses
  1. The GCD prompt does much more than explain the grammar. It asks for short fragments, discourages repetition, limits checking and repair, and tells the model to stop after a limited amount of work. These instructions alone could explain a lot of the shorter reasoning. Comparing against Chain of Draft doesn't settle this, since Chain of Draft uses a different prompt. The key missing control is the exact GCD prompt with the decoding constraints turned off. Without it, it's hard to tell how much of the gain comes from enforcing the grammar.

  2. The token-budget analysis doesn't support the test-time scaling explanation. Figures 6 and 7 are built from traces that were already generated: a response counts as correct at budget B only if it finished within B and was correct. That's a reasonable way to count how many correct answers finish before a deadline, but it isn't the same as giving the model different budgets and asking it to answer under each one. A curve can go up just because more of the existing correct answers fit under a larger budget. It doesn't show that more reasoning made the answers better. This weakens the paper's explanation of why GCD works on some tasks and fails on others.

  3. The accuracy gains are small (five points on GPQA-Diamond and three on AIME24), and the main results have no confidence intervals or variation across runs.

Questions for authors
  1. See weaknesses
  2. How does the method perform when the exact same GCD skill prompt is used without grammar-constrained decoding?
  3. Does the forty-word final-answer limit conflict with any IFBench requirements? What happens to the IFBench results when that restriction is removed while keeping the reasoning grammar unchanged?
3 comments
AuthorsOpenReview ↗

Reply to Reviewer 2r4F (1/2)

Reply to Reviewer 2r4F

Thank you for your review. We have marked every change in the revised PDF (new text is blue and removed text is struck in red) and kept changes minimal.

1. "The key missing control is the exact GCD prompt with the decoding constraints turned off."

We ran this control on all four benchmarks, using the identical prompt with the same weights, generation settings, token caps and questions; only the mask was switched off. We ran GCD and this control three times each on GPQA-Diamond, MedCalc-Bench and IFBench, and twice on AIME24 (new Table 2 in the paper; Appendix C lists each run).

Benchmark (items)CoTGCD prompt, mask offGCDMask's further cut in tokens
GPQA-Diamond (198)69.7%, 19,120 tokens78.1% [73.6, 82.8], 3,476 tokens, 1.2% parse74.7% [69.7, 79.5], 461 tokens8×
AIME24 (30)68.3%, 26,10673.3% [60.0, 86.7], 16,515, 0.0% parse71.7% [58.3, 85.0], 1,64210×
MedCalc-Bench (300)83.3%, 1,82983.2% [79.1, 87.1], 1,899, 0.1% parse71.7% [66.7, 76.4], 2019×
IFBench (233)63.5%, 4,39044.9% [39.6, 50.2], 4,593, 3.6% parse47.4% [41.5, 52.9], 29116×

Reported accuracy is the mean over runs with a 95% bootstrap interval over questions; on AIME24 the interval resamples problems, keeping both draws of each together. Token counts are median generated tokens. "Parse" is the share of traces that satisfy the grammar rules with the mask off. MedCalc-Bench uses a 300-question subset selected a priori; CoT reports the submission's single run on the same questions.

Both arms receive the instructions you list (short fragments, bounded checking and early stopping), yet only the arm with the mask produces short traces. Without the mask, the model follows the grammar in 0 to 3.6% of traces. The prompt shortens traces on GPQA-Diamond and AIME24 to 18% and 63% of CoT's median, and does not shorten them on MedCalc-Bench or IFBench. The mask then cuts tokens a further 8 to 16 times on every benchmark, more than the prompt achieves on any benchmark. We find no accuracy gain from the mask. With it off, the prompt is 3.4 points above GCD on GPQA-Diamond (+5.1, −1.5 and +6.6 in the three runs; none is significant by McNemar's test, smallest p = 0.06, although the last bootstrap interval excludes zero) and 1.7 points above on AIME24 (interval [−10.0, 13.3]); it uses 8 and 10 times the tokens. On MedCalc-Bench, the prompt is 11.6 points above GCD.

2. "The token-budget analysis doesn't support the test-time scaling explanation."

Accuracy at budget B in Figures 6 and 7 is exactly the accuracy that a method has under a hard generation cap of B. Because decoding is causal, the cap leaves every trace that finishes within B unchanged and fails the rest. A deployment with a token limit faces this deadline; Section 3 now describes the curves in these terms. We also ran the different protocol you describe, budget forcing, in which the model must answer early. Thinking is closed at GCD's median token count and the answer is capped at 64 tokens. The table gives two GCD results: one cut off at the same count with no forced answer, and one run to completion.

Budget (tokens)CoT, forcedPrompt with mask off, forcedGCD, cut off at the budgetGCD, run to completion
GPQA-Diamond (198)4560.5%15.2%43.4%74.7%
AIME24 (30)1,6210.0%3.3%41.7%71.7%

For each benchmark, the budget is GCD's median over all traces in the submission run; both AIME24 draws are pooled.

We removed test-time scaling from Section 4's explanation of where GCD works. Across four benchmarks, the observed pattern instead follows task type: GCD for long reasoning towards a short answer, the prompt alone or CoT for exact procedural work, and CoT for tasks graded on the form of the response.

3. "The accuracy gains are small ... no confidence intervals or variation across runs."

The contribution is the token reduction at maintained accuracy; the abstract and Section 3 now make this claim clear. Across three runs on GPQA-Diamond, GCD minus CoT is +5.1, +6.6 and +3.5 points, with 95% intervals [−1.5, 11.6], [−1.0, 14.1] and [−3.5, 10.6] (McNemar p = 0.17, 0.12, 0.41). On AIME24 it is +3.3 [−10.0, 16.7] (p = 0.71), resampling problems with both draws kept together. The three GCD runs on GPQA-Diamond score 74.7, 76.3 and 73.2.

AuthorsOpenReview ↗

Reply to Reviewer 2r4F (2/2)

4. "Does the forty-word final-answer limit conflict with any IFBench requirements? What happens ... when that restriction is removed while keeping the reasoning grammar unchanged?"

By a static scan of the benchmark's verifiers, 17 of the 233 items carry a constraint that needs more than forty words to pass. We removed the sentence and kept the grammar unchanged: 45.9% against 45.5% on the same items (+0.4, 95% interval [−4.7, 5.6]). Removing the limit has no measurable effect. Other rules in the prompt do work against this benchmark: they require the model to omit the answer format from the trace and to write the answer after the trace closes, where nothing checks it (Appendix C). The IFBench loss is already present with the mask off: the same prompt scores 44.9% against 63.5% for CoT. In the submission, the IFBench GCD cell used an earlier version of the prompt that did not contain this limit, so the limit cannot account for the submitted IFBench result. The revision's GCD and prompt-alone IFBench cells use the Appendix A prompt.

Also updated in the revision. In one run on MedCalc-Bench, letting the model write 300 free tokens before engaging the mask scored 6.0 points above the mean of GCD's runs, and 7.3 above the GCD run that we inspected by hand (Appendix C).

AuthorsOpenReview ↗

(reminder)

.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.