- Rating
- 2
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates canonical token-path exclusions in constrained decoding, where valid JSON strings may have their canonical token sequences incorrectly rejected. The authors compare four existing constrained decoding backends and identify two implementation issues in the token-enumeration rules of LM Format Enforcer (LMFE) 0.11.3. The proposed repair restores excluded token paths and affects model outputs in selected cases.
Overall, the paper provides a relatively thorough technical investigation and experimental validation. However, I find that its main technical contribution is largely centered on repairing specific implementation issues in an existing decoding framework. Given that tokenizer alignment and valid-token masking have been investigated in prior work, I have concerns about the methodological novelty and generalizability of the contribution.
Critical strengths
S1. The paper clearly distinguishes character-parser acceptance, canonical token-path admissibility, and actual generation behavior. This distinction is useful for understanding different types of rejection in constrained decoding systems.
S2. The authors provide a careful diagnosis and validation of the LMFE implementation issues. The repair preserves the underlying character parser and is supported by holdout evaluations, component ablations, and additional validity checks.
S3. The paired generation experiments demonstrate that token-enumeration issues can affect actual model outputs rather than merely producing differences in static token masks.
Critical weaknesses
W1. Limited methodological novelty and generalizability.
The main technical contribution of this paper is the identification and repair of two specific token-enumeration issues in LMFE 0.11.3. However, tokenizer–grammar alignment and the incorrect masking of valid tokens have already been investigated in previous work, including DOMINO, SynCode, and GreatGramma. The proposed repair mainly relies on fallback trie traversal and modifications to cache usage, rather than introducing a new general-purpose constrained decoding algorithm.
Furthermore, although the paper compares four decoding backends, their observed path rejections do not necessarily share the same underlying cause. For example, XGrammar admits the three initial LMFE failure cases, while many of its other path rejections are attributed to its property-order compilation policy. The cross-backend experiments therefore do not establish that the specific LMFE enumeration defect is shared by other decoding frameworks.
Consequently, it remains unclear what generalizable methodological insight this work provides beyond identifying and correcting implementation issues in LMFE.
W2. Insufficient evidence of practical impact on realistic agent tasks.
Although the authors demonstrate that the repair restores canonical token paths and changes selected model outputs, the practical impact of these exclusions on realistic agent tasks remains unclear.
The 16 generation cases are selected based on identified path exclusions, making them appropriate for validating the repair but not for estimating the frequency of such failures during natural generation. The BFCL-schema evaluation identifies only five distinct affected strings, all of which have admissible alternative tokenizations under the original LMFE implementation.
Moreover, the output changes in the selected agent-state experiments are concentrated in final-answer payloads, and the repair is not evaluated through complete agent executions. Therefore, the paper does not yet establish whether the proposed repair leads to meaningful improvements in real-world agent reliability.
W3. Substantial runtime overhead of the repair.
Although the authors introduce additional caching to reduce the overhead, Appendix L.5 shows that the optimized v3 implementation still incurs approximately 5.85× and 15.05× the original CPU complete-path replay time for Qwen and Llama, respectively.
While these results do not directly represent GPU inference latency, the considerable overhead may limit the practical value of a repair targeting a relatively narrow set of implementation issues.
Questions for authors
Q1. Beyond the specific LMFE repair, what methodological or technical insights can be generalized to other constrained decoding systems? Could the authors further distinguish character-accepted token-path exclusions from compiled-language policy rejections in the other backends to clarify the generality of the investigated issue?
Q2. Could the authors provide further evidence of how frequently this issue affects naturally generated token paths, or clarify its potential impact on complete agent-task performance?
Q3. Could the substantial runtime overhead be reduced through a more selective token-checking mechanism?
5 comments
Thank you for the additional experiments and clarifications.
My primary concern regarding methodological novelty remains unresolved. The proposed approach still mainly identifies and repairs two implementation-specific issues in LMFE, rather than introducing a substantially new or generalizable decoding methodology. Moreover, the additional experiments do not clearly establish the incremental benefits of the proposed repair itself, as several improvements appear to result from schema enforcement in general rather than the specific repair.
Overall, I appreciate the authors' efforts to strengthen the evaluation, but my assessment of the paper's contribution remains unchanged.
What the repair changes under the same schema
Thank you for identifying the remaining attribution issue. The Gemma result shows a benefit from schema enforcement: original LM Format Enforcer and v3 both solve 41/43 requests and emit identical tokens. We attribute that gain to enforcing the schema. The complete-episode comparison in Appendix M.2 tests the repair separately, because both arms already enforce the same schema with the same character parser.
We rechecked all 900 original and repaired episode records against their saved hashes. Each task uses the same checkpoint and prompt in both arms; only token enumeration changes.
| Synthetic AgentGraph panel | Original LMFE | Repair v2 | Gains / losses |
|---|---|---|---|
| Validation: 72 tasks × 5 checkpoints | 282/360 | 316/360 | 34 / 0 |
| Validation pairs where both trajectories are schema-valid | 282/358 | 314/358 | 32 / 0 |
| Development: 18 tasks × 5 checkpoints | 70/90 | 75/90 | 5 / 0 |
| Development pairs where both trajectories are schema-valid | 70/83 | 75/83 | 5 / 0 |
The 32 validation gains with both outputs schema-valid are important here: they cannot be explained simply by turning schema-invalid outputs into schema-valid outputs. Appendix M.2 already reports this breakdown; our earlier reply should have made it explicit. The task panel was fixed before repair outcomes. The table shows raw counts; under the frozen design weights, the validation gain is +9.40 percentage points and is positive at all five checkpoints. A new, post-hoc sensitivity check resamples whole domains and graph motifs while retaining all five checkpoints, giving a central 95% bootstrap range of [+2.50, +17.99] points. This describes the fixed synthetic panel and checkpoints, not deployment-wide benefit. The schema-valid subset is descriptive, since validity itself is an outcome of decoding.
The gains are in the final answer. Executed tool sequences are identical in every pair, and v3 reproduces v2’s traces. On the separate released Llama-3.1-8B BFCL panel, original LMFE and v3 solve 142/178 and 143/178 requests; all outputs are schema-valid, but the one-answer difference is too small to establish a general accuracy advantage (Appendix M.8). We therefore distinguish the observed repair effect on the synthetic panel from the broader benefit of enforcing a schema.
We also ran a new intervention on all five token-divergent pairs in that 178-request Llama panel, selected by token differences without consulting correctness labels. At the first divergence, we either admitted the omitted token once under original LMFE or removed it once under v3, then resumed that arm’s usual masks. Prompts, model, schema, greedy decoding and the 192-token cap stayed fixed. All ten unchanged baselines exactly reproduced the archived tokens.
| On these five requests | Original | Original + one restored token | Full v3 | v3 − one token |
|---|---|---|---|---|
| Officially correct | 1/5 | 1/5 | 2/5 | 1/5 |
| Schema-valid | 5/5 | 5/5 | 5/5 | 5/5 |
For the gained request—the accented city-name argument in Appendix M.8—restoring that token recovered the complete repaired output and its correct score; removing it recovered the complete original output and its wrong score. However, restoration alone also lost another originally correct request. The one-token intervention isolates an answer-content effect while validity stays unchanged; its mixed results do not establish a shortcut to the full repair or a population accuracy gain.
On novelty, the general technical lesson is the need to test token enumeration against the backend’s own live character parser. A rejected canonical token path alone is insufficient: it can reflect a deliberately narrower compiled language, or a token omitted despite the live parser accepting its characters. The cross-backend controls distinguish property-order restrictions in XGrammar and llguidance from LMFE’s parser-accepted enumeration exclusions; Outlines Core’s exclusions remain unattributed. Building on tokenizer–grammar alignment work in DOMINO, SynCode and GreatGramma, our contribution is this parser-relative diagnosis, followed by a parser-preserving intervention and a matched answer-level evaluation. The diagnosis can be applied across stacks, while the specific repair and measured answer gains concern LMFE.
We appreciate your pressing us to separate these claims. The comparison we ask you to consider is the repair against an already constrained decoder, especially the gains where both outputs meet the schema, alongside the explicit limits on tool performance and generalization.
Generality beyond the LMFE patch, complete episodes, and v3 cost
Thank you for the feedback! "Original LMFE" = LM Format Enforcer 0.11.3 as released; "v2" = our repaired enumerator (Section 5.3); "v3" = v2 with a character-set cache whose masks equal v2's at all audited prefixes (App. L.4). New results: Appendix M of the revised PDF (pp. 37–42).
W1/Q1. Beyond an LMFE patch. Two LMFE rules interact (D.5, D.8): a free-text shortcut judges fragments without string context, and later traversal can miss those it rejects; a cache sizes backslash fragments by raw, not decoded, length. Of 160 character-accepted holdout paths, the original excludes 16, fallback-only 16, bypass-only 96 and both changes 0 (Table 11).
We do not claim a new decoding algorithm; DOMINO, SynCode and GreatGramma are prior work (Section 2, Table 21). What transfers is a two-way test, run with the backend's own parser: at a reachable prefix, an excluded token whose characters the parser accepts and that an enumeration-only change restores is an enumeration error; text the compiled language rejects is a policy. In other backends, XGrammar 0.1.27 rejects 48/192 paths with matched schema and text property orders and 192/192 with mismatched ones: a declared-order policy (Table 10). A post-submission CPU replay finds an order policy in llguidance 1.3.0: its 48 holdout rejections are two-field objects, admitted 48/48 when orders match and 0/48 when they differ (M.8, p. 40). Outlines Core's 72/192 remain unattributed; we have no evidence that another backend shares LMFE's defect.
W2/Q2. Complete episodes. 90 synthetic AgentGraph-v2 tasks, fixed before any repair outcome, ran under five Qwen3.5-0.8B checkpoints, which pick tool call or answer at each step. Success is an exact or numeric final-answer match (M.1–M.2, p. 37).
No loss on this fixed panel is an observation, not a guarantee. As your W2 anticipated, the gain is in final answers, not tool execution: all 360 validation and 90 development pairs execute identical tool sequences; on 240 tool-required validation pairs, first tool and arguments match an enabled task-graph action in 200/240 and the complete plan in 156/240 under both (M.4, p. 38).
Natural generation (W2/Q2). At the "first divergence" (first token where the original and v2 generations differ), the original mask excludes v2's token although its parser accepts the added characters, in 270/360 validation and 70/90 development pairs. They come from 58 validation and 19 development prompt-prefix contexts, all at one original-mask state, so they show exposure on generated tokens, not a frequency. Every gain is in such a pair, yet most such pairs (236/270, 65/70) keep their label: co-occurrence, not mediation (M.3, p. 38).
BFCL V4: one Qwen3.5-2B checkpoint, used unchanged although its freeze record carries a failed training-gate marker (M.6, p. 39). Replaying its unmasked tokens through original LMFE finds such an exclusion in 186/345 supported tasks of a 384-task sample; all 545 captured generations took the final-answer route, so this one-checkpoint count is not a traffic estimate. Under free routing (the checkpoint picks call or answer), official scores rise 77/345 → 86/345 (10 gains, 1 loss; category-weighted +2.67 pp, 95% stratified bootstrap interval +0.90 to +4.48 pp). This is no-call format recovery: every gain turns an invalid output into a correct no-call answer. Both constrained arms score 0/250 in the other categories; unmasked generation scores 85/345. With the tool route imposed on 178 call-required tasks, both decoders score 126/178 with identical per-task labels (M.7). Although the five BFCL strings stay reachable by other token paths, the two masks' greedy next token differs at 6/10 teacher-forced prefixes (Table 28).
W3/Q3. Cost. On one A100, one episode at a time, in the 28 of 40 run groups accepted under sampled GPU isolation, mean seconds for original/v2/v3 are 13.42/27.51/20.99 (validation, 264 pairs) and 13.62/24.22/16.15 (development, 54 pairs); timed pairs cover checkpoints 0–2, plus part of 3 on validation (M.5, pp. 38–39). v3 reproduces v2's trace in every pair and takes 1.56× (validation) and 1.19× (development) the original mean time; the median within-pair v3/original ratio is 1.09× on both splits. Times include output-length differences; this is not serving latency. Table 30's 5.85× and 15.05× time only CPU mask queries, not model calls. Selectivity: we have not yet found a narrower check that provably keeps every admitted path.
Revision. The revised PDF states these results and limits in the Abstract, Contributions 1 and 3, new Section 6.3 (p. 9), Sections 7–9 and Ethics, with v3's episode cost (Section 7) and open items (Section 8).