Valid but Excluded: Token-Path Exclusions in Constrained Decoding for LLM Agents

Constrained decoding can exclude the canonical token path of a valid JSON string even when its character parser accepts the complete text and the omitted token's characters at the reachable prefix. We distinguish these canonical-path exclusions from native-stack rejections caused by a narrower compiled language. On a constructed 192-case holdout, four stacks reject different path sets; two reject 48 paths each but disagree on 72. Within LM Format Enforcer, we identify and repair two token-enumeration rules without changing its character parser. The repair restores 16 excluded paths among 160 character-accepted strings while rejecting all 12 invalid controls. A separate BFCL-schema panel extends the result beyond the factorial fixtures: exclusions fall from 5/164 to 0/164 under each of two tokenizer files, on the same five cases. On 16 mask-selected generation cases, exact target copying rises from 0/16 to 16/16 for Qwen and 14/16 for Llama; 61 of 128 selected agent-state pairs also change their final-answer payloads. A larger mask audit finds that all 2,201,656 newly admitted token–prefix memberships pass the live character parser. Full-path CPU replay is substantially slower, even after a cache optimization. These results establish path restoration and selected output sensitivity, not deployment prevalence or end-to-end agent gains.