- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
To address the gap between semantic quality and grammar, the paper proposes GrammarRL, a label-free reinforcement learning method that adapts language models to grammar constraints without requiring annotated data. The reward contains a forward term and a backward term. The rollout candidates for the reinforcement learning process is obtained by sampling from the constrained distribution. The paper evaluates GrammarRL on three datasets and three Llama models with different sizes and shows better performance.
Critical strengths
(1) The paper propose a RL method to address the gap between semantic quality and grammer of constrained decoding method and shows much better performance.
Critical weaknesses
(1) The experiment of the paper is not comprehensive. The experiment only performs experiments using Llama and the experiment should be performed on LLMs from another family to show that the method is generalizable. Besides, the idea of the reward is closely related to self-certainty reward used by previous works such as [1]. Compared to these rewards, the main difference is that the rollout candidates by GrammarRL is from the constrained decoding. The paper should include a self-reward baseline, where the rollout candidates are sampled frm the unconstrained distribution to better show the technical contribution of the paper.
[1] Learning to Reason without External Rewards
(2) The paper argues that the samples generated from the constrained distribution can be of low semantic quality. However, the paper uses the samples generated from the constrained distribution as the rollout candidates and the number of rollout candidates are relatively small. In this case, the chance that all rollout candidates are of low semantic quality can be high.
Questions for authors
(1) The reward model is based on the unconstrained distribution since it can address the low semantic quality issue of the constrained distribution. For the sampled rollout candidates, I am wondering how likely the order of log probabilities of these candidates are different between constrained distribution and unconstrained distribution.
We thank the reviewer for the suggestions raised:
W1a. Generalizability
We tested Qwen2.5-1.5B/7B-Instruct with the Llama 1B/8B configurations, changing only the model and recalibrating σ on held-out inputs.
The pattern holds at 1.5B and 7B, generalizing beyond Llama: GrammarRL beats greedy and beam on all tasks. We will include this results in the Appendix.
W1b. Self-certainty and internal rewards
We acknowledge that INTUITOR is a relevant related approach, which we will discuss in Related Work as an internal-reward method. Our work focuses on adapting grammar-constrained policies through sequence-level bidirectional rewards computed by a frozen reference model, making a direct empirical comparison beyond its scope. Accordingly, our results do not establish the superiority of GrammarRL over INTUITOR, and we leave their empirical comparison as a direction for future work.
W2. Rollout quality and group size
The update does not require a good candidate in every group: RLOO uses only the relative reward within the group, so the policy moves toward the better candidates even when all of them are imperfect, and improves over successive steps. Moreover, the group always contains the top-1 beam hypothesis, from a higher-probability region of the constrained space, and the two rewards penalize complementary failure modes (Appendix F: bare X outputs and trap tokens on Gloss, empty fields on CoNLL). Empirically, the group size is not a bottleneck: increasing N from 3 to 9 changes performance by at most 0.019 on Gloss and 0.009 on WoS and CoNLL (Section 4.4, Table 8), and N = 3 already improves over the frozen greedy baseline in all nine settings of Table 1.
Q1. Candidate-probability rankings
For 100 held-out inputs per task, we regenerated the rollout groups as in training (3 constrained samples plus the top-1 beam hypothesis), with the initial 1B policy and with the trained one (adapter at step 1,000). As in training, the policy generates the candidates and the frozen model scores them: constrained log-probability (Eq. 1), unconstrained log-probability, and the Eq. 6 reward. Each candidate was scored against the gold output of its input with the sentence-level task metric. For every pair of candidates in a group with different gold quality, a score ranks the pair correctly if it gives the higher value to the better candidate.
Tables report the percentage of pairs in which the score prefers the better candidate. A score unrelated to quality would pick the better one by chance, i.e. in 50% of the pairs; a perfect score would reach 100%. Both log-probabilities are divided by the number of tokens of the candidate. Differences are in percentage points; p-values use a paired, two-sided bootstrap (2,000 input resamples; bold: p < 0.05).
Step 0 — Initial-model candidates
Step 1000 — Trained-policy candidates
Compare columns within each table: trained-policy candidates are more similar and harder to rank. Eq. 6 is never significantly worse than either log-probability; at step 0 it significantly outperforms the constrained score on all tasks and the unconstrained score on Gloss/WoS. After training, its advantage remains significant on Gloss. The unconstrained score significantly outperforms the constrained score on CoNLL and is never significantly worse. We will add this analysis to the appendix.