- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces grammar-constrained decoding (GCD) for compact reasoning traces. A context-free grammar assigns explicit operators to goals, subgoals, derivations, checks, repairs, and closure, while leaving each line’s natural-language payload unconstrained. A pushdown automaton masks illegal next tokens during decoding. The authors evaluate GCD, Chain-of-Draft prompting (P), ThinkingCap fine-tuning (FT), and their combinations on four benchmarks using 8-bit Qwen3.6-27B. They report 98% and 94% reductions in median generated tokens on GPQA-Diamond and AIME24, with respective accuracy gains of 5 and 3 percentage points over CoT. On MedCalc-Bench and IFBench, GCD cuts generated tokens by 90% and 80% but loses 13 and 25 accuracy points. Budget curves favor GCD at small token budgets but show a lower plateau. The paper acknowledges regressions, limited sampling, a single model and grammar, and token-cap effects.
Critical strengths
- The representation is simple and inspectable: online syntax enforcement preserves readable free-text reasoning and explicit control-flow markers.
- The authors compare intervention combinations across four task types and report negative results, giving a more useful account than a single favorable benchmark.
Critical weaknesses
- The empirical evidence does not establish the marginal contribution of grammar masking. Section 3 says GCD includes a prompt, and Appendix A supplies roughly 870 words of prescriptive instructions: short payloads, no deliberative narration, one independent check, bounded repair, and early stopping. These instructions directly target trace length and stopping behavior. The P baseline is described only as a short Chain-of-Draft prompt; the cited CoD formulation uses a nonbinding five-word-per-step guideline. The paper does not report the same GCD skill prompt with the decoder mask disabled. Consequently, the reported token and accuracy changes support the combined prompt-plus-mask package, but do not show that the mask itself supplies the claimed third axis or is needed for the gains. This is central to the paper’s main empirical claim.
- The key gains are not accompanied by uncertainty estimates. The authors acknowledge limited sampling; AIME24 uses two samples per question, while the reported gain is 3 points. No question-level paired or clustered intervals, multiple seeds, or variability analysis are shown. The results should therefore be treated as point estimates in a single-model, single-grammar study.
Questions for authors
- Can you report a matched ablation that retains the exact Appendix A skill prompt and settings but disables the decoder mask? Please also provide the exact P prompt and state whether it uses the few-shot examples in the original CoD setup. Would the token and accuracy gains persist under these controls?
- Can you report question-level uncertainty for GPQA-Diamond and AIME24, clustering the two AIME samples by source question, and clarify whether any wall-clock latency was measured?
2 comments
Thanks, the author answered my question and I decided to improve my rating.
Reply to Reviewer fzdz
Reply to Reviewer fzdz
Thank you for your review. We have marked every change in the revised PDF (new text is blue and removed text is struck in red) and kept changes minimal.
1. "The paper does not report the same GCD skill prompt with the decoder mask disabled."
It does now. We ran this control on all four benchmarks, using the exact prompt from Appendix A with the same weights, generation settings, token caps and questions; only the mask was switched off. We ran GCD and this control three times each on GPQA-Diamond, MedCalc-Bench and IFBench, and twice on AIME24 (new Table 2 in the paper; Appendix C lists each run).
Reported accuracy is the mean over runs with a 95% bootstrap interval over questions; on AIME24 the interval resamples problems, keeping both draws of each together. Token counts are median generated tokens. "Parse" is the share of traces that satisfy the grammar rules with the mask off. MedCalc-Bench uses a 300-question subset selected a priori; CoT reports the submission's single run on the same questions.
The last column gives the mask's marginal contribution, a further 8 to 16 fold cut in median tokens beyond the prompt's reduction. The prompt alone does not produce the format: only 0 to 3.6% of its traces parse, and it saves no tokens on MedCalc-Bench or IFBench. The third axis controls cost, and our claims match: 98% and 94% fewer tokens on GPQA-Diamond and AIME24, at accuracy statistically indistinguishable from CoT. We find no accuracy gain from the mask. With it off, the prompt is 3.4 points above GCD on GPQA-Diamond (+5.1, −1.5 and +6.6 in the three runs; none is significant by McNemar's test, smallest p = 0.06, although the last bootstrap interval, [0.5, 13.1], excludes zero) and 1.7 points above on AIME24 (interval [−10.0, 13.3]). On MedCalc-Bench, the prompt is 11.6 points above GCD, significantly so in every run.
2. "Please also provide the exact P prompt and state whether it uses the few-shot examples in the original CoD setup."
Appendix B gives the exact prompt: "Think step by step, but only keep a minimum draft for each thinking step, with 5 words at most." It is the first sentence of the Chain of Draft system prompt, used zero-shot; Section 3 now records this. The prompt-alone arm is the matched control for the mask, so no conclusion about the mask depends on P. We state in Limitations that we chose the P + GCD prompt from two candidates after seeing their AIME24 results (66.7% against 50.0% at a 16K cap). The choice favours the combination, which still scores below P alone on AIME24 (71.7% against 78.3% at the 32K cap), so the finding that GCD composes poorly with prompting does not depend on it.
3. "Can you report question-level uncertainty for GPQA-Diamond and AIME24, clustering the two AIME samples by source question?"
Paired by question, GCD minus CoT is +5.1 points on GPQA-Diamond (95% interval [−1.5, 11.6], McNemar p = 0.17), and +6.6 [−1.0, 14.1] and +3.5 [−3.5, 10.6] in the two further runs. On AIME24 it is +3.3 [−10.0, 16.7], with the two draws clustered by problem (p = 0.71). The intervals are 95% bootstraps over questions. Because they include zero, we replaced "increasing accuracy" with "at accuracy indistinguishable from CoT". The three GCD runs on GPQA-Diamond score 74.7, 76.3 and 73.2. Table 2 gives the intervals, and Appendix C lists every run with its paired test.
4. "Clarify whether any wall-clock latency was measured."
We measured decoding throughput, not end-to-end latency. In a matched throughput test, the mask cost about 2% of tokens per second on a single sequence and 8 to 10% at batch sizes of 8 to 16. Since the mask cuts generated tokens 8 to 16 fold beyond the prompt alone, and more against CoT, the token saving is also a saving in decoding time.