- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper argues that the accuracy drop blamed on constrained decoding comes from prompt and format differences rather than token masking. It splits constraints into format, validity, and reasoning types, and shows on ten open models that constrained decoding matches or beats unconstrained decoding under matched prompts, with validity constraints giving the largest gains.
Critical strengths
N/A
Critical weaknesses
- The theory says that if the correct answer is allowed by the constraint, masking other tokens and renormalizing cannot lower its probability. This is correct but close to trivial, and it does not address the main open question in this area, which is why reasoning inside a constrained format (e.g., JSON) hurts accuracy. The paper itself admits that once the constraint touches the reasoning, the effect is undetermined. Yet this derivation takes up a large part of the main text, while the experimental evidence is limited to two tables and one figure, with most results moved out of the main paper. This makes the main claims hard to check from the paper itself.
- The main empirical finding, that the gap comes from prompt and format differences rather than masking itself, has already been reported in prior work, which the paper acknowledges. The new contribution is therefore mostly a replication on more models.
- Most of the reported gains look like answer cleanup rather than better reasoning. For LastLetter, the validity constraint forces the answer to be exactly four (or eight) letters, which tells the model the answer length, and the paper’s own examples (“oya” becoming “oyaa”, “h e n e” becoming “hene”, 28.00 becoming 28) are things a better parser could also fix. On the format side, JSON wrapped in a Markdown code fence is counted as a parse failure, so jumps like Gemma2-9B on GSM8K JSON going from 0% to 86.7% mainly reflect the parsing rule. There is no baseline with unconstrained decoding plus stronger post-processing, so it is unclear how much of the gain actually needs constrained decoding.
Questions for authors
- How much of the validity-constraint gain remains if the unconstrained baseline uses stronger post-processing (e.g., lowercasing, removing spaces, extracting a fixed-length string, casting to integer)?
- How do the JSON results change if JSON inside a Markdown code fence is counted as valid?
- Do the conclusions hold with longer reasoning budgets and thinking mode on, which is how these models are usually used?
- Beyond confirming prior findings on more models, what do you see as the main new finding?