- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies the distributional bias of grammar constrained decoding, where token masking samples from the locally projected distribution rather than the true grammar conditional one. Building on the future validity viewpoint of Nie et al. (2026), it derives an exact per step error identity for any approximate future validity estimator, specializes it to estimators coarsened onto automaton states indexed by parser state and remaining budget, and uses a second order expansion of that identity as an a priori criterion for whether a cheap state level correction suffices for a given grammar. Validation spans exactly enumerable toy grammars, budgeted JSON tasks on models from 135M to 1.7B, a subset of real JSONSchemaBench schemas up to Qwen3-8B, and a speculative decoding composition study with a 135M drafter and a 1.7B verifier.
Critical strengths
First, the theory is done carefully and reported honestly, which is rarer than it should be. The per step Jensen gap identity is verified numerically to machine precision, the failures are reported rather than buried, and the appendix engineering, including the equivalence tests that pin what may change in the sampler implementation, is a model of how to report systems work. Second, the scale analysis is genuinely informative: the coarsening gain is non monotone in model size, the semantic featurization improves monotonically from 135M to 7B, and the taxonomy of distribution correcting decoders as future validity estimators in Section 5 is clean and useful pedagogy for the field.
Critical weaknesses
1、First, and decisively, the method fails exactly where the paper claims it matters, and the paper's own numbers say so. On content dependent schemas the state lookup degrades total variation to 1.000, meaning disjoint support, at every scale, while accuracy barely moves (Table 4). On real JSONSchemaBench schemas the aggregate TV gets worse from masked to corrected at all three scales (0.640 to 0.931, 0.607 to 0.757, 0.508 to 0.696), and the correction helps on only 3 of 15, 6 of 15, and 3 to 5 of 15 schemas. At Qwen3-8B, the largest model tested, the headline improvement in sequence log probability is statistically unchanged. So on the toy grammars where the correction works nobody urgently needed it, and on the real schemas where practitioners would want it the method hurts more often than it helps. Framing this as the criterion working as designed is generous to the point of being backwards: a criterion whose main practical output is do not use this method is a diagnostic, not a solution, and it is not what the abstract advertises.
2、The deployment story compounds this: latency is measured with a 135M drafter and a 1.7B verifier on V100s, a serving configuration from years ago, which undercuts the production framing of the introduction.
3、Third, the novelty and the comparative evidence are thinner than the presentation suggests. The future validity viewpoint, the Doob transform formulation, and the endpoint analysis all come from the (Nie et al., 2026), which is cited throughout; what remains new here is the middle segment, namely coarsening onto automaton states, a per step Jensen gap identity, and the geometric mean table. The identity is a standard telescoping decomposition of the KL between two path measures plus a second order expansion, and Theorem 1 is essentially moment matching. This is competent craftsmanship, but it is the natural next increment of the prior work rather than a qualitatively new idea. The a priori claim also does not survive inspection: estimating the within state variance of log Phi requires the same masked rollouts used to build the table, stabilizing only around N of 100 or more, at which point one could directly measure the corrected TV on held out rollouts and skip the identity. And the main comparative claim rests on beating the authors' own offline reimplementation of ASAp's estimand, explicitly not its online loop, with mean fallback on unseen prefixes; there is no comparison on the real schemas against any actually strong corrector, which is precisely where the state table fails and where a comparison would be informative.
4、The evaluation is also self referential throughout: every fidelity metric is distance to mu star, exactly computable only on enumerable toy languages, and on real schemas the metric collapses to TV of the output length marginal against a self normalized IS reference built from the same 200 masked samples. The paper's own observation that accuracy is preserved while TV degrades begs the question it never answers: does any user visible quantity improve when the bias is corrected? No downstream utility, calibration benefit, or human facing metric appears anywhere.
5、Last but not least, you used GPT-2 for experiment. Are you kidding me? Today is 2026!
Questions for authors
- On the 15 schema JSONSchemaBench subset, for the majority of schemas where the lookup does not help, can you name one realistic deployment scenario where your corrected sampler beats masking on every aggregate metric that scenario cares about?
- How much of the Table 2 margin over the prefix table survives if the prefix baseline is given the same smoothing and fallback tuning you afford your own table, and if it is allowed to key on parser states plus a bounded suffix window rather than exact prefixes?
- Can you show a single downstream task where correcting the sampling law changes a decision a user would make, at any scale at or above 7B?