- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that grammar-constrained decoding guarantees well-formed output but not quantitative requirements. It proposes semantic quota constraints, a decode-time layer on top of a host's structurally valid action set. On a synthetic RNNG-style AST DSL with a 9.2M-parameter native decoder, All-req goes from 0.5% (grammar only) to 92.7% (blocking) to 100% (full layer). The paper also provides in theorem 1 and 2 some theoretical guarantees about the faithfulness of the rollouts and per-prompt termination certificates.
Critical strengths
- The paper has a careful and reproducible measurement of the backend-dependent enforcement, with a careful ablations study.
Critical weaknesses
- The paper has several presentation problems. The baseline measured in the evaluation seems to be degenerate. In table 1, the baseline run hits the 600 actions cap. However, the task itself does not require 600 actions, which cast some doubt on whether the model's temperature is set so high that the model does not output the stopping token. Additionally, the theorems mentioned seem to be mostly definitional from MASK's definition.
- The paper has limited novelty comparing to type-aware constrained decoding, neurologic decoding, monitor-guided decoding, etc, which were not properly discussed or compared. Additionally, the paper does not provide evidence on how the technique can be applied to real-life scenarios, with limited evidence on the repair success.
Questions for authors
- Why does the trained native decoder almost never emit <STOP> under grammar-only decoding (99.5% budget hits) when 99% of references fit within the cap? What is the grammar-only behavior at lower temperature, or with greedy or nucleus decoding? Does the chunked Program root interact with how <STOP> appears in training targets?
- Table 2 shows grammar-only at Valid 0.0% and All-req 0.5%. Is All-req computed on truncated outputs? If so, please report "terminated ∧ All-req" separately, since Theorem 1(2) concerns only terminated outputs.