acceptodds
Under review as a conference paper at ICLR 2027

Semantic Quota Constraints for Specification-Faithful Structured Generation

Abstract

Grammar-constrained generation enforces structural validity but leaves an important class of semantic constraints unenforced: a well-formed object can still miss required fields, undershoot a count, or omit a required operator. We introduce semantic quota constraints, a parameter-free decode-time mechanism that tracks quantitative requirements in parser (or container) state: stopping is blocked while any requirement is unmet, quota-saturated actions are removed, and generation is steered toward unmet requirements. A controlled program-generation environment lets us prove soundness (Theorem 1: every terminating rollout satisfies every parsed requirement) and isolate the mechanism: grammar-only decoding reaches 0.5% All-req, quota blocking—counting validity—92.7%, and the full layer—validity plus steering for timely completion—100.0% across three seeds. The same failure mode appears in deployed structured generation. In a pinned SchemaStore checkout, 691/952 documents (72.6%) contain a cardinality keyword; a pinned constrained decoder exhibits four isolated required/minProperties failures, and a quota adapter closes all four, moving matched full-schema validity from 53/60 to 57/60. At twenty draws the same adapter records 16 repairs and 0 breaks (268/300 to 284/300), a mechanism probe rather than a population estimate. The same masks wrap three frozen instruction hosts; terminating host rollouts satisfy parsed requirements, while residual misses are budget hits whose rate depends on the host.

Primary area:
generative models
Keywords:
constrained decoding, structured generation, grammar-constrained generation, JSON Schema, specification fidelity

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2Gs9OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that grammar-constrained decoding guarantees well-formed output but not quantitative requirements. It proposes semantic quota constraints, a decode-time layer on top of a host's structurally valid action set. On a synthetic RNNG-style AST DSL with a 9.2M-parameter native decoder, All-req goes from 0.5% (grammar only) to 92.7% (blocking) to 100% (full layer). The paper also provides in theorem 1 and 2 some theoretical guarantees about the faithfulness of the rollouts and per-prompt termination certificates.

Critical strengths
  • The paper has a careful and reproducible measurement of the backend-dependent enforcement, with a careful ablations study.
Critical weaknesses
  • The paper has several presentation problems. The baseline measured in the evaluation seems to be degenerate. In table 1, the baseline run hits the 600 actions cap. However, the task itself does not require 600 actions, which cast some doubt on whether the model's temperature is set so high that the model does not output the stopping token. Additionally, the theorems mentioned seem to be mostly definitional from MASK's definition.
  • The paper has limited novelty comparing to type-aware constrained decoding, neurologic decoding, monitor-guided decoding, etc, which were not properly discussed or compared. Additionally, the paper does not provide evidence on how the technique can be applied to real-life scenarios, with limited evidence on the repair success.
Questions for authors
  • Why does the trained native decoder almost never emit <STOP> under grammar-only decoding (99.5% budget hits) when 99% of references fit within the cap? What is the grammar-only behavior at lower temperature, or with greedy or nucleus decoding? Does the chunked Program root interact with how <STOP> appears in training targets?
  • Table 2 shows grammar-only at Valid 0.0% and All-req 0.5%. Is All-req computed on truncated outputs? If so, please report "terminated ∧ All-req" separately, since Theorem 1(2) concerns only terminated outputs.
Reviewer_SxNFOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

Quantitative constraints (e.g, how many properties an object instance has) on generated text can be seen in JSON schemas, for example. However, these quantitative constraints are not addressed by traditional constrained decoding methods. This paper aims to close that gap. In order to do this, four masks are used, one to prevent generation from emitting "stop" before the program has been completed, one to stop us from overshooting requirements, (e.g, mask out for loops after two for loops have been created if we require no more than two), one to try to get a quick path to requirements being satisfied, and one to get a quick path to the ending, provided requirements have been satisfied. With these masks, the authors prove that any generation which emits "stop" must have the requirements satisfied, and furthermore that generation emits "stop" in a finite number of actions. In addition, the authors prove that for some prompts, generation terminates within a certain bound of actions, but this is limited to only certain prompts. Evaluation is conducted with a 9.2M parameter transformer, and it is found that using the masks improves the chances of satisfying all the requirements. Steering is found to slow down generation, but it does lead to fewer actions being taken, as desired.

Critical strengths
  1. The evaluation demonstrates that the techniques used by the paper do work, satisfying the goal of achieving the quantitative requirements within action bounds. Furthermore, having formal guarantees that the requirements are met before "stop" is emitted is nice, because it allows people to better plug code generated by models with these masks enabled into systems that rely on the requirements being satisfied.

  2. Section 6 does provide a reason for this system's existence, and why it might be used. The reader can see that if JSON schemas have these quantitative requirements, and we want to satisfy these requirements with as few actions as possible, then this system might be useful.

Critical weaknesses
  1. Design choices are sometimes not adequately justified. For example, why do the authors pick the specific action-cap requirement that they do in the evaluation?

  2. The main contribution that we should be interested in is steering. However, how steering works is poorly described, and the evaluation finds that it provides only marginal benefits, and that's within a specific action cap. How do the benefits provided by steering change as the action cap is raised? Furthermore, the related work does not acknowledge prior work on steering, so it's unclear how much of a novel contribution this work's implementation of steering is.

Since it's unclear how much of a contribution this represents over prior work I currently recommend a negative score. The stop-blocking and quota blocking could likely be implemented with limited effrt by existing work (e.g., [1]). The remaining contributions would therefore be steering and closure, but this work doesn't acknowledge prior work on steering in the related work, so it's unclear how much of a contribution this is. Furthermore, steering is evaluated on an action cap of 600 and is found to improve Valid from 92.7%+-3.4 to 100%. Why an action cap of 600 is used is unclear, and it is possible that with a larger action cap, quota blocking alone could produce valid programs that satisfy all the requirements, and therefore steering would be unnecessary. Finally, the models used are relatively weak, and with stronger models, the requirements could possibly be satisfied without quota blocking. Quota blocking and steering do give us guarantees that the requirements are satisfied that we wouldn't get even with those stronger models. However, applying constrained decoding might lead to a decrease in output quality if we care about output quality beyond just the satisfaction of the requirements, as is demonstrated by [2], and this is also unacknowledged by this work.

[1] Nagy, Shaan, et al. "ChopChop: A programmable framework for semantically constraining the output of language models." Proceedings of the ACM on Programming Languages 10.POPL (2026): 1905-1932. [2] Park, Kanghee, et al. "Grammar-aligned decoding." Advances in Neural Information Processing Systems 37 (2024): 24547-24568.

Questions for authors
  1. Why did the authors pick the specific number of actions that they did and why did they choose to train their own transformer?
  2. How does the difference between quota blocking and full layer change as the action budget is increased?
  3. Are there more powerful open-source models which could satisfy the quantitative requirements, and if so, does masking result in a decline in output quality (by other metrics then requirement satisfaction?)
  4. Finally, a better description of steering with examples would be nice, as well as a description of how this differs from previous work involving steering
Reviewer_bbZqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a parameter-free decode-time mechanism to track quantitative requirements during generation phase. This mechanism blocks stopping while a requirement is unmet, removes actions that would overshoot an exact count, and steers toward unmet requirements. Experiments show the proposed method greatly outperforms the grammar-only baseline on both requirement satisfaction and action-budget usage.

Critical strengths
  1. The paper targets quantitative requirement in constrained decoding, a common issue in real world usage scenarios.
  2. The ablation study clearly shows the contribution of different parts in the method to the overall performance.
  3. The paper provides a dataset for result reproduction and further study of quantitative constraints in controlled generation.
Critical weaknesses
  1. The main reason for the bad performance of the grammar-only baseline is exhausting the budget, rather than missing some requirements during generation. The baseline is too weak.
  2. This paper does not examine the impact of the method on semantic quality. A response with correct format but meaningless content is also useless.
  3. The experiment result on JSON schema is too narrow (16 repairs only sit on 3 of 15 schemas) to support the transfer claim.
Questions for authors

How are the prompts and ASTs generated?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.