- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper finds that a grammar-constrained speculative verifier matches the grammar-conditional law only if it is given the future-validity-corrected target. It supplies that corrected target to real verifiers and gives exact ways to acquire it from cached probability rows and draft spans. On enumerated JSON tries, an fp32 verifier cuts mean terminal total variation from 0.355 to 0.0072, at a cost of about 0.03 in acceptance.
Critical strengths
- The experiments are well designed.
- The paper content is very rich.
Critical weaknesses
-
The writing is unclear, verbose, and difficult to follow. Please simplify the paper, highlight the core contributions, and revise the manuscript throughout.
-
The authors do not compare accuracy. The base model’s preferences may themselves be wrong, so correcting the sampler toward the conditional law does not necessarily make the outputs more accurate.
-
The speedup setup in Section 5.2 is not realistic. KV cache is enabled in any real online serving system.
-
The evaluation uses small, closed answer sets rather than JSON in the usual sense. The models are also too small for the results to be convincing.
-
Relatedly, these small models have weak instruction following. On a larger LLM, stating the constraint in the prompt may already be sufficient.
Questions for authors
Please see weakness
2 comments
Response to Reviewer QeUz (2/2)
(Continued from comment 1/2; table and proposition numbers refer to the accompanying revision.)
“small, closed answer sets… The models are also too small…” At 27B the two laws remain separated. On Qwen3.8-27B, on the same 3,000 MMLU-Pro questions (2,999 for S2) and without the format stated, a share 0.33 of questions on S2 and 0.70 on S3 has a gap of at least 0.10 (median gap 0.057 and 0.175), and accuracy under the conditional law minus accuracy under the masked law is +4.6 points (95% interval +4.0 to +5.3) on S2 and -9.3 points (95% interval -10.0 to -8.5) on S3, the same directions as on Qwen3.5-9B (+3.9 and -7.9); on S3 the mechanism is that of Section 5.3, the masked law deciding at the option letter, with the accuracy of S1 (53.4%), while the conditional law also weighs each option by the probability of its text (Appendix C.8, Table 34). Each benchmark question defines three JSON events over up to ten options: the option letter, the option text, and both (Section 5.3); the largest event run through the verifier has 2,000 strings (Table 3), and the gap is computed analytically on Dyck events of up to 132,867 strings on real models (Appendix C.2, Table 21). Finite events are what give an exact reference: on tries A–D the terminal law of the implemented verifier has simultaneous 95% upper bounds of 0.013–0.028 on its distance from the conditional law (Table 3).
“On a larger LLM, stating the constraint in the prompt may already be sufficient.” Stating the constraint leaves a measurable gap. With the schema stated, six cells keep gaps of 0.11–0.41 on Qwen3-8B (Appendix B.2, Table 19) and of 0.014–0.150 on Qwen3-14B (Appendix B.3, Table 20), 2–4 of 25 events per prompt keep gaps of at least 0.10 on Qwen3-8B and on Qwen3.5-9B (Table 9), and on the benchmark events with the format stated a share 0.05–0.11 of questions keeps a gap of at least 0.10 on S2 and S3, where the median gap is 0.0019 and 0.0004, on Qwen3.5-9B (Section 5.3, Table 6); on Qwen3.8-27B with the format stated, a share 0.054 of questions on S2 and 0.027 on S3 keeps a gap of at least 0.10 (Appendix C.8). Proposition 5 gives the mechanism. On these dead-end-free finite tries, the gap equals half the mean absolute relative deviation, over paths sampled under the masked law, of the legal mass collected along a path, and it is at most one minus the probability of the event, so it is zero when every such path collects the same legal mass. The paper's result is the criterion and its measurement: it tells when stating the constraint suffices and when it does not.
“The writing is unclear, verbose, and difficult to follow.” In the revision, Figure 1 gives the pipeline. The estimator hierarchy has moved to Appendix C.4, so Section 4 concentrates on what acquired rows certify (Propositions 8 and 9), with the terminal-law bounds collected in Corollary 10. Section 5 answers three questions in order: what an implemented verifier samples (5.1), what exact acquisition costs with a cached prompt (5.2), and when the two laws differ and what that does to accuracy (5.3). The core contributions are the three numbered items of Section 1, and Table 1 maps each main result to its event, reference law, sampler, statistical unit and cost measure.
Response to Reviewer QeUz (1/2)
Thank you for these questions. We added the experiments reported below and revised the manuscript; table and proposition numbers refer to the accompanying revision.
“The authors do not compare accuracy…” The paper's claims concern which law a verifier samples, the grammar-conditional law being the model's own preference given validity; we now measure what that choice does to accuracy. On a fixed sample of 3,000 MMLU-Pro test questions (up to ten options, gold labels from the dataset) we enumerate for every question-event pair the exact conditional law and the exact masked law of Qwen3.5-9B, so accuracy under either law carries no sampling noise. Outputs are JSON objects: the option letter (S1), the option text (S2; 2,999 questions), and a letter-plus-text object (S3). When the format is enforced but not stated, the laws separate (median gap 0.048–0.305; a share 0.77 of questions with a gap of at least 0.10 on S3), and accuracy under the conditional law minus accuracy under the masked law is +0.2 points on S1, +3.9 points (95% interval +3.2 to +4.6) on S2 and -7.9 points (95% interval -8.8 to -7.1) on S3. Section 5.3 gives the mechanism on S3: the masked law decides at the option letter, since the text that follows is forced, and so has the accuracy of S1, while the conditional law also weighs each option by the probability of its text. When the prompt states the format, the median gap is 0.0000–0.0019 and accuracy differs by -0.49 to +0.78 points, and on S2 and S3 a share 0.05–0.11 of questions keeps a gap of at least 0.10. So without the format stated the choice of target moves accuracy by several points on S2 and S3, and the corrected verifier fixes which of two measurably different laws the decoder samples. Section 5.3 and Table 6 report this and the introduction now states the distinction.
“The speedup setup in Section 5.2 is not realistic…” We ran the timing matrix of Section 5.2 with the prompt state cached: all 224 histories, each timed in seven arm-order rotations (1,568 timed runs), 8 events, 7 samplers, a Qwen3.5-9B target with a Qwen3.5-0.8B draft (Table 4). At supports 64 and 256 the faster of CARS and Spec-4 needs 0.034–0.639 of the enumeration wall clock (0.019–0.655 in the submitted, uncached matrix, now Table 29). In every cell of this matrix Spec-4 needs fewer target forwards than CARS, 20–24 against 66–74 at support 256 with the long prompt and 16 outputs (Table 31), and it needs 0.25–1.00 times those of CARS when the submitted pair, Qwen3-8B with Qwen3-1.7B, runs through the same cached interface (Section 5.2). On 200 MMLU-Pro S2 and S3 events with the format stated (first 100 questions, Qwen3.5-9B with Qwen3.5-0.8B, 1 and 16 outputs, same cached interface), Spec-4 is faster than CARS in 390 of 400 cells, with a median wall-clock ratio of 0.44 (Appendix C.8, Table 33). On the eight events of Table 4, the Spec-4/CARS wall-clock ratio in the long-prompt cells is 0.892–1.471 in the cached matrix (Table 4), where the submitted table had 0.568–1.095 (Table 29), and 0.764–1.290 for the submitted pair through the cached interface (Section 5.2).
(Continued in comment 2/2.)