acceptodds
Under review as a conference paper at ICLR 2027

What The Mask Still Buys: Measuring Which Decoding Constraints Small Language Models Internalize

Abstract

Schema-derived constraints let a small language model emit valid MLIR at inference time: a token-level grammar mask enforces syntax and type domains, and a scope validator rejects use-before-definition. We ask what the mask still buys once a model has been trained to satisfy it, layer by layer, and whether learning validity brings correctness with it. We train students on their own masked-and-verified rollouts (symbolic self-distillation, SSD) from 135M to 3B parameters in two families and measure, per constraint layer, the violation rate it still removes after training, paired per prompt on frozen pools. Validity moves into the weights: type domains internalize completely, syntax up to a structural residual of 5 to 11 points plus a vocabulary residual that shrinks when the training pool covers the missing concepts, and scope up to a small residual that a per-token scope signal mostly removes at 360M. Correctness follows only where the model already samples it. Verifier acceptance under free decoding rises by up to 70 points. Under a gold-differential harness, masked correctness falls at 135M, 360M and Gemma-1B; at 1.7B and 3B sampled pass@1 rises by 7 to 12 points, while the set of prompts solved within eight samples does not grow. We read both as concentration rather than acquisition. A functional reward does not change this on the released pools. Self-distillation raises the untrained model’s pass@1 and lowers its pass@16. Against the 49 of 143 prompts the untrained 360M model solves within 256 masked samples, nine on-policy cells each solve at most one prompt outside that set, while five cells that see a gold program solve 13 to 22, with no overlap between the groups. Supplying one gold program per rollout group restores correctness at a price in validity that an adapter control and a second seed show is not separable from adapter capacity at the scales we compare. As in reinforcement learning with verifiable rewards, self-distillation selects among programs the model already samples: it teaches validity, which the mask makes abundant, and not correctness, which the model must be shown; here the missing program can be supplied and its price measured. Outside the training distribution the student loses validity on held-out StableHLO above 360M while gaining it on a dialect that reuses the training vocabulary, and for Gemma-1B the mask does not repair the loss, leaving the student behind the model it started from with the full stack restored.

Primary area:
reinforcement learning
Keywords:
constrained decoding, grammar-constrained generation, small language models, self-distillation, code generation, MLIR, compilers, reinforcement fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_YCHBOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that while small LLMs could internalize basic coding skills and grammars via self-training, while it is not trivial for them to learn problem-solving skills for unseen situations, and would requires external constraints for decoding to handle complex structural errors. In experiments it is shown that there is a trade-off, where teaching the model with correct solving logic degrades the ability to write valid code.

Critical strengths
  • The paper explains how the models internalize decoding constraints into their weights, instead of treating them as a black box. This may provide insights to readers.
  • The tradeoff shown in the experiments may be interesting, possibly provide positive impact to the community.
Critical weaknesses
  • The presentation and writing quality is, unfortunately, suboptimal and the paper requires a major revision to have a better clarity. For example, abstract provides only overly high-level concepts with terms never introduced (e.g., MLIR), them move to specific experimental setups. The paper, overall, should present what research question is asked, how is it solved, and what is insight. In the current status, it is really hard for readers to extract those information.
  • Small language models would be able to learn basic coding skills through practice, but they would still need external constraints for decoding to avoid structural errors and scale to unseen tasks.
Questions for authors
  • The trade-off between learning problem-solving logic and learning correct grammars is shown. Is this simply a limit of small language models, or merely a gap in the current training paradigm?
Reviewer_1S8LOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

As far as a human reader can understand, this paper considers a setting where LMs for programming additionally have a hard-coded “masking” / filter layer added on top of their output sampler, to enforce syntactic and semantic correctness of the program being outputted. Previous work introduced this paradigm / showed it is useful for getting better code. This work asks: what if you train such a model as a student given a teacher of the same model WITH those additional output-level mechanisms? Can you distill/internalize the output level mechanisms into the weights of the model and therefore have no need for the external gadget anymore? Their findings are that this partially the case: “type domain” masks/enforcements can be dropped, syntax ones only partially, for off-distribution examples masking is especially important.

Critical strengths
  • the basic underlying idea of whether hard-coded constraints / validations can be "transferred"/distilled into the model itself is an interesting / important one, especially for verifiable domains like programming
  • the finding that some of them can be is interesting
Critical weaknesses
  • Overall, this paper is completely unreadable. I almost certain it is entirely generated by AI, having many characteristics of AI writing (tons of jargon introduced without any context / explanation, including seemingly unique coinage etc). The abstract, introduction, related work, and methodology sections of the paper: I read them all and each one is extremely dense, jargon-laden, and practically unreadable. It took significant effort to extract the basic point of the paper, and none of the paper's contributions are motivated / explained at a higher level / given any justification of significance. As it stands this paper is not usable or meaningful for a human reader.
  • highly related to the above - the empirical findings of this paper are not clear / cannot be understood by a reader. Their valence is also completely unclear (as is often the case with AI writing) -- whether the findings are that the mechanisms of sampler-level masking/validation can or cannot be successfully transferred into the weights of a model -- is inconclusive / no lesson is drawn.
Questions for authors

The main issues are summarized in weaknesses above. However more specifically:

  • Abstract is hard to understand / immediately uses jargon (schema-derived, MLIR, use-before-definition). Overall abstract is too dense / hard to read / does not give high-level understanding of problem being studied and contributions. Needs major rewrite to a useful abstract
  • Introduction immediately has this issue too - launches into jargon / hard to understand explanation of previous results. even “compiler intermediate representations” in first sentences is not defined / hard to follow. “Schema-derived stack gates”, “scope validator”, “action masks in policy gradients”, “layered grammar”, “SSA”, “shape consistency” etc all not defined / thrown in immediately. “Pre-registered a class hypothesis” — what does mean? Etc. Intro also needs to be rewritten completely to be motivating, clear and understandable.
  • 70 describes the main method of the paper for the first time (training a model to internalize what the added-on masking + verification does) — I could extract this with effort but it’s written very poorly
  • Of main contributions 1 and 2 are understandable but very barely, major rework needed. 3 is almost completely inscrutable
  • Related work is a rattling off of a TON of papers, it’s overall too long and not a useful section at all. Should be replaced with a shorter section that meaningfully explores / explains the MOST related works, how their findings relate to / precede / complement this work’s findings. Right now any reader will just skip this
Reviewer_1w4NOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Constrained decoding involves constraining the output of language models to satisfy certain syntactic-semantic constraints of a language such as:

  1. a grammar mask to ensure syntactic and typing validity, and
  2. scope validator to avoid use-before-def.

This paper aims to understand (with MLIR as the constraining language) whether small language models trained on their own constrained decoded outputs internalize the different validity aspects/rules to some extent, such that inference-time enforcement of some of these validity rules can be removed without degrading the validity satisfaction rates of the output. The paper also studies whether training to internalize validity rules also improves correctness, i.e., do models that learn some syntactic-semantic aspects of a language also improve in their capacity to generate functionally correct programs in that language. Through experimental evaluation, the paper demonstrates that:

  1. typing rules are completely internalized, whereas grammar and use-before-def rules are internalized to a lesser extent;
  2. internalizing validity improves the pass@1 rates of functionally correct programs the model could already generate prior to training, but it does not transfer to previously unsolved programs, i.e., pass@k does not improve;
  3. training to internalize validity of one dialect is observed to negatively impact performance on other dialects; and
  4. generating more functionally correct programs is positively impacted the most by supplying reference correct programs rather than the training recipes explored in the paper, but doing so is observed to negatively impact validity.
Critical strengths
  1. I appreciate the use of MLIR rather than some toy synthetic language, as it makes the evaluation and the observations more realistic.
Critical weaknesses
  1. Evaluation Soundness. I have some significant concerns about the soundness of the claims being inferred from the evaluation. Here are a few examples:

    • Layer internalization metric does not isolate a layer's individual effects. The paper uses the residual metric: Rk(M′)≜vk(M′,FREE)−vk(M′,Sk)R_k(M')\triangleq v_k(M', FREE) - v_k(M', S_k), to infer if a constraint layer has been internalized by the model (M′M' is the post-trained checkpoint of model MM and vk(M′,S)v_k(M',S) is the violation rate of constraint layer CkCk). This DOES NOT isolate what an individual constraint layer contributes because it is computed relative to the unconstrained post-trained model, so it measures what the stack SkS_k contributes rather than an individual layer CkCk. So a type enforcement layer (C2C2)'s contribution would be conflated with effects from the syntax enforcement layer (C1C1) since the stack S2=C1+C2S_2 = C1 + C2. Therefore, one could easily incorrectly infer that a model which fails in the earliest layers has internalized later layers, and we indeed see the paper making such unsound claims (see next point).
    • Is Typing really completely internalized??? The paper uses the residual metric RkR_k along with Tables 5 and 6 in Appendix D to make these claims. But these claims are unsound. For example: the 3B SSD s0 model (Table 5, last row) has a syntax residual R1=78.4R_1=78.4 and "type" residual of R2=0R_2=0 for the linalg pool, since R2=0R_2=0, which satisfies the paper's criterion of a layer being internalized, it says this model has internalized the typing layer, discounting the fact that the large syntax residual R1R_1 is indicating that the post-trained checkpoint of this model barely produces an output unconstrained that passes the syntax layer. So a model that produces more syntactically correct programs has higher odds of being labeled as not having internalized the typing layer than a model that fails for the syntax layer itself. Any claims of internalization based on the current residual metric can only be made for the syntax layer and not any of the other layers downstream from it.
  2. Incremental Novelty. I believe there are serious problems with the residual metric foundational to the paper's main novelty claims as described before. The remaining ones are incremental, such as:

    • Specializing on one dialect hurts on another. This result is not novel and has been shown by several prior works for both natural and programming languages. Eg: Zhang et al., ExeSQL: Self-Taught Text-to-SQL Models with Execution-Driven Bootstrapping for SQL Dialects (Findings of the EMNLP'25), Yamaguchi et al., Mitigating Catastrophic Forgetting in Target Language Adaptation of LLMs via Source-Shielded Updates (ACL'26).
    • "Concentration rather than acquisition". Yet again, several prior works on RLVR have investigated whether RLVR equips models with new reasoning capabilities or if they simply amplify existing ones, some of which the paper itself cites, such as Yue et al., Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (NeurIPS'25).
  3. Presentation.

    • What is SSA in the Introduction section? It's never been introduced, and it does not have a citation either.
    • Why does the gap Gk(M)G_k(M) use rate instead of vkv_k?
Questions for authors
  1. You already cite quite a lot of work and concede that SSD, RLVR, etc., are not novel. There is also prior work that focuses specifically on training recipes for internalizing syntactic correctness, typing correctness, etc., and work that shows that these may not have carryover to other downstream checks. So how exactly are you positioning your novelty? Are you trying to unify syntax, typing, and semantics internalization within one framework as opposed to prior work that studies them individually?

  2. Could you please address the points raised under Evaluation Soundness in Critical Weaknesses above? Specifically, how can the residual metric be used to interpret anything about the contribution of individual layers when stacking layers is implicit in its definition?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.