acceptodds
Under review as a conference paper at ICLR 2027

GrammarRL: Effective Grammar-Constrained Decoding via Reinforcement Learning

Abstract

Grammar-constrained generation guarantees syntactic validity, but can substantially degrade semantic quality when the model’s preferred outputs are poorly aligned with the imposed grammar. This trade-off is particularly severe when the prompt is underspecified or the model has limited instruction-following ability. Beam search can partially mitigate these failures by exploring multiple valid sequences, but its computational cost grows with beam width, while sequence-level probability is only an imperfect proxy for semantic quality. We introduce GrammarRL, a label-free reinforcement learning method that adapts language models to grammar constraints without requiring annotated data. GrammarRL optimizes the model using two complementary self-supervised rewards derived from its own likelihoods: a direct reward, measuring how likely the constrained output is given the input, and a reverse reward, measuring how well the input can be reconstructed from the generated output. We optimize these rewards with a Reinforce Leave-One-Out (RLOO) objective over groups of grammar-constrained rollouts, augmented with the top-1 beam-search hypothesis and regularized towards a frozen base model. We evaluate GrammarRL on sign language gloss translation, hierarchical text classification, and named entity recognition using Llama models ranging from 1B to 8B parameters. GrammarRL consistently outperforms constrained greedy decoding, with an average improvement of 9.8 points and gains of up to 22.8 BLEU. It matches or outperforms beam search on two of the three tasks while preserving greedy-decoding inference cost. Ablations further show that the two rewards are complementary: either reward alone can underperform the untrained baseline, whereas their combination consistently improves upon it.

Primary area:
reinforcement learning
Keywords:
grammar-constrained decoding, reinforcement learning, grammar aligned decoding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SXPAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

To address the gap between semantic quality and grammar, the paper proposes GrammarRL, a label-free reinforcement learning method that adapts language models to grammar constraints without requiring annotated data. The reward contains a forward term and a backward term. The rollout candidates for the reinforcement learning process is obtained by sampling from the constrained distribution. The paper evaluates GrammarRL on three datasets and three Llama models with different sizes and shows better performance.

Critical strengths

(1) The paper propose a RL method to address the gap between semantic quality and grammer of constrained decoding method and shows much better performance.

Critical weaknesses

(1) The experiment of the paper is not comprehensive. The experiment only performs experiments using Llama and the experiment should be performed on LLMs from another family to show that the method is generalizable. Besides, the idea of the reward is closely related to self-certainty reward used by previous works such as [1]. Compared to these rewards, the main difference is that the rollout candidates by GrammarRL is from the constrained decoding. The paper should include a self-reward baseline, where the rollout candidates are sampled frm the unconstrained distribution to better show the technical contribution of the paper.

[1] Learning to Reason without External Rewards

(2) The paper argues that the samples generated from the constrained distribution can be of low semantic quality. However, the paper uses the samples generated from the constrained distribution as the rollout candidates and the number of rollout candidates are relatively small. In this case, the chance that all rollout candidates are of low semantic quality can be high.

Questions for authors

(1) The reward model is based on the unconstrained distribution since it can address the low semantic quality issue of the constrained distribution. For the sampled rollout candidates, I am wondering how likely the order of log probabilities of these candidates are different between constrained distribution and unconstrained distribution.

1 comment

We thank the reviewer for the suggestions raised:

W1a. Generalizability

We tested Qwen2.5-1.5B/7B-Instruct with the Llama 1B/8B configurations, changing only the model and recalibrating σ\sigma on held-out inputs.

ModelDecodingGloss (BLEU)WoS (Hier. F1)CoNLL (F1-micro)
Qwen2.5-1.5BFrozen, greedy0.363 ± 0.0120.299 ± 0.0090.427 ± 0.011
Qwen2.5-1.5BFrozen, beam (B=3)0.323 ± 0.0200.377 ± 0.0090.688 ± 0.008
Qwen2.5-1.5BGrammarRL0.572 ± 0.0060.474 ± 0.0100.732 ± 0.007
Qwen2.5-7BFrozen, greedy0.565 ± 0.0030.521 ± 0.0080.635 ± 0.005
Qwen2.5-7BFrozen, beam (B=3)0.589 ± 0.0050.547 ± 0.0080.779 ± 0.007
Qwen2.5-7BGrammarRL0.715 ± 0.0080.565 ± 0.0080.792 ± 0.005

The pattern holds at 1.5B and 7B, generalizing beyond Llama: GrammarRL beats greedy and beam on all tasks. We will include this results in the Appendix.

W1b. Self-certainty and internal rewards

We acknowledge that INTUITOR is a relevant related approach, which we will discuss in Related Work as an internal-reward method. Our work focuses on adapting grammar-constrained policies through sequence-level bidirectional rewards computed by a frozen reference model, making a direct empirical comparison beyond its scope. Accordingly, our results do not establish the superiority of GrammarRL over INTUITOR, and we leave their empirical comparison as a direction for future work.

W2. Rollout quality and group size

The update does not require a good candidate in every group: RLOO uses only the relative reward within the group, so the policy moves toward the better candidates even when all of them are imperfect, and improves over successive steps. Moreover, the group always contains the top-1 beam hypothesis, from a higher-probability region of the constrained space, and the two rewards penalize complementary failure modes (Appendix F: bare X outputs and trap tokens on Gloss, empty fields on CoNLL). Empirically, the group size is not a bottleneck: increasing N from 3 to 9 changes performance by at most 0.019 on Gloss and 0.009 on WoS and CoNLL (Section 4.4, Table 8), and N = 3 already improves over the frozen greedy baseline in all nine settings of Table 1.

Q1. Candidate-probability rankings

For 100 held-out inputs per task, we regenerated the rollout groups as in training (3 constrained samples plus the top-1 beam hypothesis), with the initial 1B policy and with the trained one (adapter at step 1,000). As in training, the policy generates the candidates and the frozen model scores them: constrained log-probability (Eq. 1), unconstrained log-probability, and the Eq. 6 reward. Each candidate was scored against the gold output of its input with the sentence-level task metric. For every pair of candidates in a group with different gold quality, a score ranks the pair correctly if it gives the higher value to the better candidate.

Tables report the percentage of pairs in which the score prefers the better candidate. A score unrelated to quality would pick the better one by chance, i.e. in 50% of the pairs; a perfect score would reach 100%. Both log-probabilities are divided by the number of tokens of the candidate. Differences are in percentage points; p-values use a paired, two-sided bootstrap (2,000 input resamples; bold: p < 0.05).

Step 0 — Initial-model candidates

Correctly ranked pairs (%)ConstrainedUnconstrainedEq. 6Unc. − Con.Eq. 6 − Con.Eq. 6 − Unc.
CoNLL72.882.483.8+9.6 (p = 0.003)+10.9 (p = 0.001)+1.4 (p = 0.50)
Gloss74.775.284.1+0.5 (p = 0.85)+9.4 (p < 0.001)+8.9 (p < 0.001)
WoS69.470.580.3+1.2 (p = 0.81)+11.0 (p = 0.026)+9.8 (p = 0.016)

Step 1000 — Trained-policy candidates

Correctly ranked pairs (%)ConstrainedUnconstrainedEq. 6Unc. − Con.Eq. 6 − Con.Eq. 6 − Unc.
CoNLL63.283.076.4+19.8 (p = 0.001)+13.2 (p = 0.093)−6.6 (p = 0.24)
Gloss68.473.078.6+4.5 (p = 0.15)+10.2 (p < 0.001)+5.7 (p = 0.004)
WoS69.676.187.0+6.5 (p = 0.44)+17.4 (p = 0.083)+10.9 (p = 0.29)

Compare columns within each table: trained-policy candidates are more similar and harder to rank. Eq. 6 is never significantly worse than either log-probability; at step 0 it significantly outperforms the constrained score on all tasks and the unconstrained score on Gloss/WoS. After training, its advantage remains significant on Gloss. The unconstrained score significantly outperforms the constrained score on CoNLL and is never significantly worse. We will add this analysis to the appendix.

Reviewer_8zQgOpenReview ↗
Rating
2(+1)
Confidence
4(+2)
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a relatively effective solution to the GCG problem, called GrammarRL, which achieves promising experimental results through the use of rollouts combined with a bidirectional reward scheme. The core idea is to partially shift the search and error correction processes involved in traditional inference to the training phase, using unlabeled reinforcement learning to make the model better adapt to grammatical constraints, thereby enabling the continued use of low-cost constrained greedy decoding during inference. The problem addressed in this paper is both interesting and meaningful.

Critical strengths
  1. The problem is clearly defined, the reasoning is logical, and it addresses a meaningful problem.
Critical weaknesses
  1. Trajectories are sampled from grammar-constrained policies, while the loss function uses unconstrained log-likelihood. This omits the gradients of the grammar-dependent normalization factors required by the constrained policy gradient objective. Furthermore, one of the trajectories is generated deterministically via bundle search. Can the authors provide a precise mathematical interpretation of the resulting gradient estimator and clarify whether it is unbiased, biased but consistent, or merely a heuristic alternative?
  2. Neither beam search nor a naive restricted output vocabulary alters the original capabilities of LLM, but GrammarRL does alter the original model’s capabilities. How should we view and evaluate the impact this has on the model’s original understanding and generation performance? Does the proposed training approach possess a certain degree of transferability?
  3. As can be seen from the results of multiple experiments in Table 1, a bidirectional reward design is not always the best, and there are significant differences in the optimal lambda values across different tasks. This may involve complex parameter tuning or search processes, which is a source of concern for me.
  4. The sampled rollout terms can retain a valid leave-one-out baseline under appropriate independence assumptions, but the deterministic beam trajectory introduces an additional biased update that is not justified as an on-policy policy-gradient term. So I think the authors may need to provide additional theoretical explanation or narrow the scope of this claim.
Questions for authors

plase refer to “weaknesses”

2 comments

W1. Gradient estimator

We thank the reviewer for figuring out this problem. Eq. 8 in the paper contains a typo: renormalized logits are used, so gradients through the grammar normalizers are included. Let

\log\pi^G_\theta(y\mid x)=\log\pi_\theta(y\mid x)-\sum_t\log Z_t(\theta).$$ For the KL term, the constrained-policy log-ratio is $$\Delta^{(i)}=\frac{\log\pi^G_\theta(y^{(i)}\mid x)-\log\pi^G_{\rm ref}(y^{(i)}\mid x)}{|y^{(i)}|}$$ The frozen reference uses the same mask; only rewards (Eqs. 4–5) use the unconstrained reference. We will correct Eqs. 3 and 8 in the paper. Let $J(\theta)=\mathbb E_{y\sim\pi^G_\theta}[r(x,y)]$. The $N$ sampled candidates are i.i.d. from $\pi^G_\theta$; $y^{(N+1)}$ is selected deterministically by beam search. The expected policy-term gradient is $$\begin{aligned} \mathbb E[\hat g]={}&\frac{N}{N+1}\nabla_\theta J(\theta)\\ &+\frac{1}{N+1}\bigl(r^{(N+1)}-J(\theta)\bigr)\nabla_\theta\log\pi^G_\theta(y^{(N+1)}\mid x). \end{aligned}$$ The sampled terms are unbiased: the beam reward is independent of each sample and is a valid baseline. The beam term is not zero-mean, so the estimator is biased for finite $N$; this bias vanishes as $N\to\infty$. ## W2. Original capabilities and transferability We thank the reviewer for raising this point. To assess the impact on general capabilities, we evaluated MMLU, a general-knowledge benchmark, using Meta's official 5-shot setup, without grammar constraints, with and without each of the nine adapters from Table 1. Accuracy changes by at most 0.27 percentage points, and fewer than 5% of answers change. | MMLU benchmark, macro accuracy | Frozen (Meta reported) | Gloss Δ [95% CI] | WoS Δ [95% CI] | CoNLL Δ [95% CI] | |---|---:|---:|---:|---:| | Llama-3.2-1B-Instruct | 49.12 (49.29) | +0.04 [−0.26, +0.32] | −0.11 [−0.39, +0.21] | −0.21 [−0.55, +0.11] | | Llama-3.2-3B-Instruct | 63.42 (63.44) | −0.06 [−0.30, +0.18] | −0.27 [−0.54, −0.01] | −0.18 [−0.44, +0.07] | | Llama-3.1-8B-Instruct | 69.52 (69.43) | −0.22 [−0.42, −0.04] | −0.07 [−0.30, +0.16] | −0.15 [−0.35, +0.06] | The 95% confidence intervals include zero in seven of the nine settings; the two remaining cases show small decreases. Overall, these results indicate that GrammarRL largely preserves the model's general capabilities. Regarding transferability, the training method is task-agnostic and applies across different tasks and model sizes, but each learned adapter is specific to its grammar and requires retraining for a new one (without annotated data). ## W3. Sensitivity to $\lambda$ Although optimal $\lambda$ varies by task (Table 3), main results use an untuned choice of $\lambda=0.5$, midpoint of $\{0.25,0.5,0.75\}$, across tasks, sizes and rollouts (Appendix A). It beats the frozen greedy baseline in all nine settings; neither single-reward variant does so throughout (Table 1). Tuning λ is a one-dimensional search over a few values that yields further gains where needed (CoNLL 1B 0.578 to 0.625 at $\lambda=0.25$, above beam search). Reward scales $\sigma$ are measured once on unlabeled inputs. ## W4. Beam-term interpretation and ablation We thank the reviewer and agree to narrow the claim. Each sampled rollout has an unbiased leave-one-out baseline: the other candidates' mean reward (including the beam) is independent of that sample. The beam's own term, $$A^{(N+1)}\nabla_\theta\log\pi^G_\theta(y^{(N+1)}\mid x),\qquad A^{(N+1)}=r^{(N+1)}-\frac{1}{N}\sum_{i\le N}r^{(i)},$$ is not an on-policy policy-gradient term; its expectation is the bias term above. The full update is therefore biased. It raises the beam hypothesis's likelihood when its reward exceeds the sampled mean, and lowers it otherwise. We retrained the 1B models without the beam term, using the Table 1 seeds, setup and training inputs. Its reward then appears only in sampled baselines, making the estimator unbiased: | Llama-3.2-1B | Gloss (BLEU) | WoS (Hier. F1) | CoNLL (F1-micro) | |---|---:|---:|---:| | Frozen model, greedy | 0.342 ± 0.005 | 0.384 ± 0.006 | 0.435 ± 0.009 | | GrammarRL, no beam term (unbiased) | 0.465 ± 0.006 | 0.412 ± 0.010 | 0.574 ± 0.007 | | GrammarRL, full update (Eq. 8, Table 1) | 0.570 ± 0.014 | 0.394 ± 0.008 | 0.578 ± 0.006 | | Gain from the beam term | +0.105 | −0.018 | +0.004 | Mean ± std over 10 test sets; the last row reports paired differences betwen the biased and unbiased variants. The unbiased variant already achieves substantial gains over the frozen baseline on all three tasks (+0.12, +0.03, +0.14), showing that GrammarRL does not rely on the biased beam term. The beam term further improves Gloss (+0.105) and CoNLL (+0.004), but hurts WoS (−0.018). Its larger benefit on Gloss suggests it helps when sampled rollouts struggle to explore high-reward regions, consistent with the beam having the highest reward in 84/99 initial groups. We will report both variants.

Thanks for your clarification.

The authors have clearly explained the bias introduced by the deterministic beam term and shown that the underlying method still achieves substantial gains. The additional MMLU results also partially alleviate my concerns about degradation of the model’s original capabilities.

However, there is a substantial discrepancy between the original manuscript and the response regarding the use of unconstrained probabilities. Unless I have overlooked it, the authors have not provided supporting code, so this point still requires verification. The formulation of KL regularization by directly adding the log-ratio to the loss also remains insufficiently explained. Furthermore, the claim in the response that the default bidirectional reward outperforms the single-direction rewards in all settings is inconsistent with Table 1.

I will raise my score to 2 accordingly.

Reviewer_4o1MOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Grammar-constrained decoding ensures well-formed outputs, but these outputs can still be semantically incorrect. Beam search and sampling-based methods can improve output quality by exploring more candidates, though this adds computation to every inference call. The paper addresses whether some of this correction can be learned during training, allowing the model to produce better outputs with greedy decoding. GrammarRL combines forward and reverse likelihoods from a frozen reference model to encourage both plausible outputs and preservation of input information. These scores provide a learning signal without annotated targets for the adaptation data. The model is trained with RLOO using grammar-constrained samples and a beam-search candidate, then decoded greedily at test time. Experiments cover gloss translation, hierarchical classification, and constrained named-entity extraction. The reported results show an average gain of 9.8 points over constrained greedy decoding, with performance matching or exceeding baseline beam search on gloss translation and classification.

Critical strengths

GrammarRL uses a frozen model’s forward and reverse likelihoods to learn from adaptation inputs without annotated target outputs. The model requires greedy constrained decoding at deployment. This addresses a setting: applications that need structurally valid outputs, have limited supervision, and cannot afford extensive search for every request.

Critical weaknesses
  1. The added value of RL over simpler adaptation methods is not established. Tables 1 and 2 mainly compare GrammarRL with decoding methods applied to the frozen model. They do not include pseudo-label SFT on the model’s beam outputs, or SFT on candidates selected using the same bidirectional reward. Both alternatives could learn sequence preferences without annotated adaptation targets and retain greedy decoding at inference. These are important controls because transferring such preferences into the model is presented as a central contribution. Without comparisons under similar data and compute budgets, the reported gains establish the benefit of adaptation but leave unclear what the proposed RL procedure contributes beyond simpler self-training or distillation.
  2. The Introduction does not sufficiently connect the stated problem to the proposed reward design. It argues that model likelihood is an imperfect proxy for semantic correctness, but then relies on likelihoods from the same frozen model without clearly identifying when and why these scores can correct the errors motivating the paper. The reverse reward is justified through preservation of source information, yet classification and entity extraction deliberately discard information; higher source predictability therefore needs a clearer connection to task correctness. The explanation of complementarity is also asymmetric: reverse scoring is introduced to address information loss under the direct reward, while the conditions under which the direct reward corrects failures of reverse scoring are less clear. Since the bidirectional reward is a central contribution, these gaps weaken the conceptual justification for the method.
Questions for authors
  1. Why can the frozen reference model provide a useful corrective signal? The second paragraph of the Introduction notes that “high sequence-level probability does not necessarily translate into high-quality outputs,” while the following paragraph proposes rewards from the frozen model to favor continuations that are “both grammatical and correct.” What assumption makes this self-evaluation useful for correcting the errors that motivate the paper?
  2. How does the reverse reward relate to task correctness in classification and entity extraction? The third paragraph of the Introduction motivates the reverse reward through preserving enough source information for the reference model to recover the input. In WoS, however, the output is a parent–child label pair; in CoNLL, it contains only the first entity span of each type. I understand that Eq. (5) evaluates a conditional likelihood rather than requiring exact reconstruction, and that a correct label can make an abstract more predictable without preserving its full content. Could you clarify why higher reverse likelihood should favor correct labels or entity predictions in these settings? Is the intended signal primarily input–output compatibility, and what assumptions or evidence connect it to the paper’s claim about preserving source information?
  3. When should the two rewards complement each other? The third introduction paragraph explains why the reverse reward can counter information loss under the direct reward, and the second contribution states that “each one alone favors a different kind of degenerate output.” Appendix F provides useful evidence of these failure modes for the 1B model. What explains when the direct reward should improve reverse-only training, and when combining the rewards instead creates a trade-off? For example, Table 1 reports 0.622 for reverse-only training versus 0.560 for the default mixture on 8B WoS, and 0.861 for direct-only training versus 0.787 for the mixture on 8B CoNLL. Are these differences mainly due to the mixing weight, or do they indicate limits to the proposed complementarity?
  4. Does the comparison hold when hyperparameters are selected independently of the test data? The caption of Table 2 and Appendix G.3 acknowledge that the best lambda was selected using the same test seeds. This affects the CoNLL comparison in Section 4.3: the default configuration scores 0.577, compared with 0.578 for SMC and 0.586 for beam search, while the selected configuration reaches 0.621. Can you repeat this comparison with lambda selected on a disjoint validation set and all settings fixed before evaluation on an untouched test set? Please also clarify how the baselines scoring and search settings were chosen, and use a comparable validation protocol and tuning budget where those settings are tunable. Whether the reported CoNLL advantage survives independent model selection?
  5. What does RL add beyond pseudo-label SFT? The third paragraph of the Introduction distinguishes GrammarRL from “beam-search imitation,” but Tables 1 and 2 do not establish what RL adds beyond learning from generated candidates. Could you compare GrammarRL with two training baselines: SFT on the frozen model’s top beam outputs, and SFT on candidates selected using the same bidirectional reward in Eq. (6)? These controls would help determine whether the proposed RL procedure provides a meaningful advantage over simpler pseudo-label training.
1 comment

We thank the reviewer for the suggestions raised:

Q1 & Q2

Direct reward scores plausibility, while reverse reward (input predictability given output) provides an input-conditioned signal of semantic relevance, encouraging outputs that are specific to the input rather than generic.

On CoNLL, reverse reduces empty-field errors from 23.71% to 7.88% (λ=0.25), but reverse-only yields misplaced Gloss fragments (THERE-DESC: 140 vs. 33) and incorrect CoNLL spans (F1=54.8).

Importantly, the direct reward does not assume that likelihood is a general proxy for semantic correctness: previous work shows that likelihood-based decoding can favor generic or degenerate outputs (Holtzman et al., 2019) and that model probabilities are weakly correlated with answer correctness (Jiang et al., 2021). Rather, it corrects a specific bias of local constrained decoding: the ideal distribution for a model constrained under grammar G is

πG(y∣x)=π(y∣x)⋅1[y∈L(G)]∑y′∈L(G)π(y′∣x)\pi^{G}(y \mid x) = \frac{\pi(y \mid x) \cdot \mathbf{1}[y \in L(G)]}{\sum_{y' \in L(G)} \pi(y' \mid x)}

where the denominator is constant with respect to yy and the 1\mathbf{1} function is 1 for any valid sequence, but local choices of constrained decoding break this distribution. The direct reward measures (Eq. 4) tries to recover the ideal sequence probability by pointing towards π(y∣x)\pi(y\mid x). GrammarRL transfers this correction to the policy through RL.

Q3

Reward complementarity depends on the prevalence of each reward's failure mode. At 8B, reverse-only WoS no longer collapses onto a few labels (6.3% vs. 41.5% at 1B), while direct-only CoNLL leaves only 7.6% of gold non-empty fields unfilled. Thus, the additional reward becomes less useful and can introduce a counter-signal, explaining why reverse-only performs best on WoS (0.622) and direct-only on CoNLL (0.861).

Q4a

Yes, the comparison holds. The caption of Table 2 and Appendix G.3 are misleading, and we will revise them. All reported comparisons use hyperparameters selected independently of test data, with one exception: the last row of Table 3 explicitly reports the test-best λ, but this should not be interpreted as test-set tuning. The penultimate row indeed reports the a priori default λ configuration. We will clarify this distinction in the revised paper.

Q4b

To ensure a comparable validation protocol, we tuned both the beam-search width BB and GrammarRL's λ\lambda on the same held-out dev set, leaving (N,B)=(3,3)(N,B)=(3,3) and all sampling baselines fixed a priori.

Llama-3.2-1BBeam search (B ∈ {1, 3, 5, 10})GrammarRL (λ ∈ {0.25, 0.5, 0.75}), greedy
Gloss (BLEU)0.397 ± 0.017 (B=3)0.576 ± 0.013
WoS (Hier. F1)0.363 ± 0.008 (B=10)0.407 ± 0.009
CoNLL (F1-micro)0.595 ± 0.008 (B=10)0.625 ± 0.008

Q5

We conducted an ablation using GrammarRL's exact Table 1 setup across two baselines:

  • SFT-beam: trained on the frozen model's top beam (B=3).
  • SFT-select: trained on the highest-reward candidate (Eq. 6) from GrammarRL's step-0 pool (3 samples + 1 top beam).
Model / methodGlossWoSCoNLL
Llama 8B———
SFT-beam0.644 ± 0.0230.501 ± 0.0080.863 ± 0.008
SFT-select0.659 ± 0.0040.523 ± 0.0070.824 ± 0.006
GrammarRL0.754 ± 0.0040.560 ± 0.0060.787 ± 0.008
Llama 1B———
SFT-beam0.446 ± 0.0050.378 ± 0.0060.561 ± 0.008
SFT-select0.438 ± 0.0180.423 ± 0.0060.569 ± 0.009
GrammarRL0.568 ± 0.0150.392 ± 0.0080.579 ± 0.006

Eq. 6-based selection improves WoS over beam pseudo-labels, with SFT-select competitive with GrammarRL. This shows that the proposed reward provides a useful learning signal beyond RL, transferable across training strategies.

To assess whether RL preserves general capabilities better than pseudo-label SFT, we evaluated all three methods on unconstrained 5-shot MMLU across the nine task–size settings. GrammarRL changes accuracy by at most 0.27 points (95% CIs include zero in 7/9 settings), against drops of up to 43 points for SFT-beam and 0.52 points for SFT-select, which selects its targets with the same Eq. 6 reward and thus also largely preserves MMLU. We report 1B results below and will include all sizes in the revision.

AdapterFrozen MMLUGrammarRL Δ [95% CI]SFT-beam Δ [95% CI]SFT-select Δ [95% CI]
Gloss49.12 (49.29)+0.04 [−0.26, +0.32]−0.46 [−0.86, −0.07]−0.45 [−1.06, +0.13]
WoS49.12 (49.29)−0.11 [−0.39, +0.21]−0.23 [−0.58, +0.09]+0.08 [−0.26, +0.44]
CoNLL49.12 (49.29)−0.21 [−0.55, +0.11]−11.88 [−13.04, −10.62]−0.07 [−0.43, +0.29]

Comments on the paper

1 comment
Area_Chair_4yPAOpenReview ↗

Reminder: Author-reviewer discussion period ending soon

Dear Reviewers,

Thank you again for your time and effort in reviewing this paper. This is a friendly reminder that the author-reviewer discussion period is coming to a close.

If you haven't already, please check out the authors' rebuttal and engage in the discussion by leaving comments. Even a brief acknowledgment confirming whether your questions were addressed would be very helpful for our discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.