acceptodds
Withdrawn from ICLR 2027

Constrained Decoding: Harmful or Beneficial? Revisiting Its Impact on LLM Performance

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Constrained decoding guarantees that Large Language Models (LLMs) produce outputs conforming to predefined formats, but is often assumed to inherently degrade reasoning. We examine this assumption through a taxonomy of three constraint types: format constraints, which enforce output structure; validity constraints, which mask invalid answer; and reasoning constraints, which restrict reasoning trajectories. We theoretically show that appropriate constraints can preserve or improve performance by eliminating invalid paths. Across four model families and eleven open-source models, causal ablations on diverse benchmarks using state-of-the-art constrained decoding strategies reveal that apparent performance degradation largely stems from non-equivalent prompting and output formats rather than from decoder-level masking. Meanwhile, validity constraints improve reliability and extractability by eliminating invalid answers, while reasoning constraints can improve reasoning and, consequently, accuracy when they enforce beneficial trajectories. These findings show that the impact of constrained decoding depends on the type of constraint and the output space it removes. Our code is available onlineThe code is included in the Supplementary Materials and will be publicly released upon acceptance..

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Constrained decoding, structured generation, syntax-guaranteed generation, format constraints, validity constraints, reasoning constraints

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_DergOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that the accuracy drop blamed on constrained decoding comes from prompt and format differences rather than token masking. It splits constraints into format, validity, and reasoning types, and shows on ten open models that constrained decoding matches or beats unconstrained decoding under matched prompts, with validity constraints giving the largest gains.

Critical strengths

N/A

Critical weaknesses
  • The theory says that if the correct answer is allowed by the constraint, masking other tokens and renormalizing cannot lower its probability. This is correct but close to trivial, and it does not address the main open question in this area, which is why reasoning inside a constrained format (e.g., JSON) hurts accuracy. The paper itself admits that once the constraint touches the reasoning, the effect is undetermined. Yet this derivation takes up a large part of the main text, while the experimental evidence is limited to two tables and one figure, with most results moved out of the main paper. This makes the main claims hard to check from the paper itself.
  • The main empirical finding, that the gap comes from prompt and format differences rather than masking itself, has already been reported in prior work, which the paper acknowledges. The new contribution is therefore mostly a replication on more models.
  • Most of the reported gains look like answer cleanup rather than better reasoning. For LastLetter, the validity constraint forces the answer to be exactly four (or eight) letters, which tells the model the answer length, and the paper’s own examples (“oya” becoming “oyaa”, “h e n e” becoming “hene”, 28.00 becoming 28) are things a better parser could also fix. On the format side, JSON wrapped in a Markdown code fence is counted as a parse failure, so jumps like Gemma2-9B on GSM8K JSON going from 0% to 86.7% mainly reflect the parsing rule. There is no baseline with unconstrained decoding plus stronger post-processing, so it is unclear how much of the gain actually needs constrained decoding.
Questions for authors
  • How much of the validity-constraint gain remains if the unconstrained baseline uses stronger post-processing (e.g., lowercasing, removing spaces, extracting a fixed-length string, casting to integer)?
  • How do the JSON results change if JSON inside a Markdown code fence is counted as valid?
  • Do the conclusions hold with longer reasoning budgets and thinking mode on, which is how these models are usually used?
  • Beyond confirming prior findings on more models, what do you see as the main new finding?
Reviewer_yK3UOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper explored the relationship between constrained decoding and the type of constraints on the outputs. They defined three types of constraints: format constraints, validity constraints, and reasoning constraints, and found improvements with the constrained decoding strategy on LastLetter and its extension tasks.

Critical strengths
  1. This paper provides a valuable insight into understanding the constraint decoding effect with a carefully designed constraint taxonomy.
  2. The comprehensive study on models with different families and sizes shows the non-negative effect of constraint decoding.
Critical weaknesses

The main concern is the writing, which makes it hard to follow. including:

  1. The methodology part is weakly organized and introduces too much basic information about the modeling of constraint decoding.
  2. For line 097, it is hard to understand whether the writing is finished or if something is missing.
  3. In the main body, the authors discuss the role of the RC, but do not find related results in the main body of the paper to support their discussion and findings.
  4. The dataset lacks example illustrations for direct understanding and lacks a discussion about the motivation for the choice of such datasets.
  5. One significant question is whether current strong LLMs need such constrained decoding if they already have enough ability to understand the prompt requirements. SOTA models/agents are needed to support the generality of the finding.
Questions for authors

NA

Reviewer_QHwFOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Constrained Decoding (CD) is a decoding technique to ensure that LLM outputs follow a particular output format, so that they can be parsed easily, by restricting the set of possible tokens at each step. Prior work has found that CD often leads to performance degradation. This work starts by providing a theoretical justification that, assuming the correct answer belongs to the constrained set, CD is guaranteed to perform at least as well as unconstrained. At a high level, during each step of CD the probability mass is renormalized among the set of valid tokens thus redistributing the probability from invalid to valid sequences including the correct answer(s). Through their experiments, the authors show that the drop in performance comes from prompt-level differences between the constrained and unconstrained settings. They test 3 constrained decoding strategies on 11 models from 4 different families using 4 datasets such as GSM8K, AIME2026. They find that the validity constraints show more consistent gains compared to format constraints. This is expected from the theory since the more constrained the set is — assuming the correct answer continues to be present in the set — the more gains can be expected.

Critical strengths

The theoretical analysis in the paper is clear, easy to follow and turns an intuitive idea(restricting the set of possible sequences while ensuring the correct answers belongs to the set increases the correct answer’s likelihood) into something concrete. The analysis makes it easy to understand the more consistent gains in the case of validity constraints over format constraints.

Critical weaknesses

The empirical contributions of the paper are lean. The theory, while useful, is not deep enough to warrant a paper on its own. The key result 5.3 subsection which states that holding the prompt fixed, CD is better than unconstrained also aligns with prior findings of Lee et al. (2026) and is not really new besides the fact that the theory supports it. 5.4 says that Let’s think step by step is useful in some cases and not in others, which is to say inconclusive results, and attribute it to their similarly inconclusive theory regarding reasoning constraints wherein a priori knowledge of which reasoning chains will bring success is unknown. 5.5 states that CD helps preserve output format, which is unsurprising because that is guaranteed by design. In 5.6 the authors present an analysis of token efficiency but they only compare number of output tokens amongst methods without providing the more interesting wall clock time comparison. The results start off with 5.1 stating that json is not the only possible CD output and 5.2 stating json might be a practically better format than natural language — both of these are not essential to the paper’s point and should be a sentence rather than the opening subsections.

Questions for authors

It seems that the standard setting which is greedy decoding was used for majority of the experiments (excluding AIME). This seems to be a different setting compared to the theory which is built only for sampling and in expectation. Greedy decoding has stronger guarantees than sampling when it comes to never worse performance in constrained setting compared to unconstrained — it is at least as good not only in expectation(which is with sampling) but it is also impossible for something that was correct in unconstrained to be incorrect in constrained setting(same tokens are picked in greedy) thus being better at an example level. Please report with sampling

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.