acceptodds
Under review as a conference paper at ICLR 2027

PAVIF: Reinforcement Learning for Instruction Following with the Policy as Its Own Verifier

Abstract

Instruction following is a core capability of large language models (LLMs), yet reinforcement learning for this task often relies on external verifiers to provide reward signals, introducing additional computational cost. This motivates us to investigate whether the policy can serve as its own verifier for instruction-following optimization. Our preliminary study reveals an intriguing phenomenon: verification training for the policy can also enhance its instruction-following potential. Building on this observation, we propose PAVIF, a two-stage reinforcement learning framework that internalizes verification within the policy. In the first stage, the policy is trained to verify its own responses using a self-verification dataset that we construct with responses spanning varying levels of instruction compliance. In the second stage, the policy uses its verification signals to reinforce instruction-compliant responses, turning the increased generation potential into more reliable instruction-following capability. Evaluations across six benchmarks show that PAVIF outperforms open-source baselines, with an average absolute improvement of 2.01% over the corresponding base models. Further analyses show that verification training improves the policy’s verification accuracy, while PAVIF largely preserves general capabilities and reduces per-step training time by 23.2% compared with using an external verifier of the same scale. Our code is available at https://anonymous.4open.science/r/PAVIF-310B.

Primary area:
reinforcement learning
Keywords:
Large Language Models, Instruction Following, Reinforcement Learning, Self-Verification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AJTrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PAVIF, a two-stage reinforcement learning framework for improving instruction following in large language models. In the first stage, the authors remove instruction constraints to generate responses with varying levels of compliance, then use teacher scores to train the policy’s verification ability. In the second stage, the same policy generates and evaluates responses, combining self-verification scores with programmatic rewards for GRPO training. Evaluations cover three Qwen models and six instruction-following benchmarks, alongside analyses of verification ability, general capabilities, and training efficiency. PAVIF improves over the corresponding base models overall, with the clearest gains on the 4B model, and reduces second-stage training time per step relative to a separate verifier of the same size.

Critical strengths
  1. The method is clear and practically useful. Training verification ability within the policy and reusing that policy to provide rewards is a workable training approach. The improvements across several benchmarks on the 4B model, together with lower per-step training time than a separate same-sized verifier, make the approach worth considering.
  2. The evaluation is broad and reports limitations. The experiments cover six instruction-following benchmarks, three model sizes, two model-based judges, and ablations of the training stages and reward components. The authors also evaluate general capabilities and explicitly report that generation optimization does not further improve verification accuracy. These results help establish the scope of the method.
Critical weaknesses
  1. The contribution over prior work remains unclear. Constraint-removal data construction is closely related to MuSC [1], while learning to evaluate responses and using self-generated feedback has precedents in Self-Rewarding Language Models [2]. PAVIF offers a specific combination of these ideas, but the ablations do not clearly establish the added value of verification pretraining and dynamic parameter sharing.
  2. The reliability and consistency of the benefits need stronger support. Most verification gains from Stage 1 disappear after Stage 2, yet the reliability of online self-scores is not directly examined. Improvements over VerIF are also small on the 1.7B and 8B models, with no repeated-run results to establish their stability.

[1] Huang, H., Liu, J., He, Y., Li, S., Xu, B., Zhu, C., Yang, M., & Zhao, T. (2025). MuSC: Improving complex instruction following with multi-granularity self-contrastive training. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 10667–10686). [2] Yuan, W., Pang, R. Y., Cho, K., Li, X., Sukhbaatar, S., Xu, J., & Weston, J. E. (2024). Self-rewarding language models. In Proceedings of the 41st International Conference on Machine Learning, PMLR, 235, 57905–57923.

Questions for authors
  1. Please clarify PAVIF’s main methodological contribution relative to MuSC and Self-Rewarding Language Models. Are there existing comparisons showing that GRPO-based verification training provides additional benefits over verification SFT on the same data?
  2. Is the frozen verifier in Table 2 initialized from the base model or the Stage 1 checkpoint? Are the generation policy’s initialization, training data, and budget otherwise identical to those of full PAVIF? This distinction is important for interpreting the contribution of dynamic self-verification.
  3. Could existing checkpoints or training logs be used to examine agreement between Stage 2 self-scores and independent judgments—for example, the proportion of responses receiving full self-scores despite violating constraints, and how this changes during training? Results on soft constraints would be particularly informative.
  4. Are repeated training runs or other uncertainty analyses available to assess whether the smaller improvements over VerIF are consistent?
4 comments
AuthorsOpenReview ↗

W1 & Q1–Q2: Methodological distinctions and ablations of verification training and dynamic self-verification

W1: The contribution over prior work remains unclear. Constraint-removal data construction is closely related to MuSC [1], while learning to evaluate responses and using self-generated feedback has precedents in Self-Rewarding Language Models [2]. PAVIF offers a specific combination of these ideas, but the ablations do not clearly establish the added value of verification pretraining and dynamic parameter sharing.

Q1: Please clarify PAVIF’s main methodological contribution relative to MuSC and Self-Rewarding Language Models. Are there existing comparisons showing that GRPO-based verification training provides additional benefits over verification SFT on the same data?

Answer: MuSC [1] and Self-Rewarding Language Models [2] both use preference optimization, while PAVIF trains the policy through reinforcement learning using its own verification feedback. MuSC uses constraint removal to construct preference pairs, treating the response to the full instruction as chosen and the response to the reduced instruction as rejected. However, a response may still satisfy an omitted constraint. We therefore assess each response against the full instruction. We generate responses under three levels of constraint coverage (none, partial, and full) to encourage a more balanced distribution across the three verification scores (0, 0.5, and 1).

In Self-Rewarding Language Models [2], the model scores its own responses to construct preference pairs for the next round of DPO. In PAVIF, the same policy generates and verifies responses during reinforcement learning. Under this dynamic self-verification mechanism, the policy’s own verification scores serve as a reward for instruction following. In our ablation study, dynamic self-verification yielded a higher average instruction-following score than using a frozen verifier initialized from the same Stage 1 checkpoint (69.05 vs. 67.93, Table 2).

Table A compares GRPO (PAVIF) with verification SFT (PAVIF+) on Qwen3-4B-Instruct-2507. Both models used the same Stage 1 examples and Stage 2 procedure. PAVIF averaged 69.3 across the six benchmarks, compared with 65.0 for PAVIF+. On InfoBench, PAVIF scored 87.8% against 85.1% for PAVIF+. We will include this comparison in the appendix of the revised manuscript. Owing to potential variation in model judgments across deployments, we reran all evaluations, using gpt-oss-120B as the judge for FollowBench, CFBench, and InfoBench.

Table A: GRPO vs. SFT for verification training.

\hline \smash{\lower0.6em{\textbf{Model}}} & \kern-0.0785em\rlap{\textbf{IFEval}}\kern0.0785em & & \kern2.1957em\rlap{\textbf{Multi-IF}}\kern-2.1957em & & & \kern-0.5245em\rlap{\textbf{IFBench}}\kern0.5245em & & \kern-1.6030em\rlap{\textbf{FollowBench}}\kern1.6030em & & \textbf{CFBench} & \textbf{InfoBench} & \smash{\lower0.6em{\textbf{Avg.}}} \\\\ & \textbf{Pr.S} & \textbf{Pr.L} & \textbf{Turn 1} & \textbf{Turn 2} & \textbf{Turn 3} & \textbf{Pr.S} & \textbf{Pr.L} & \textbf{HSR} & \textbf{SSR} & \textbf{ISR} & \textbf{Overall} & \\\\ \hline \rlap{\textit{Qwen3-4B-Instruct-2507}} & & & & & & & & & & & & \\\\ \text{Base} & 83.5 & 87.6 & 82.1 & 72.1 & 62.9 & 28.3 & 31.0 & 62.6 & 74.6 & 45 & 86.4 & 64.6 \\\\ \text{PAVIF+} & 86.7 & 89.8 & 86.3 & 73.1 & 59.6 & 28.3 & 31.3 & 65.7 & 76.4 & 43 & 85.1 & 65.0 \\\\ \text{PAVIF (Ours)} & 88.4 & 91.7 & 84.3 & 74.7 & 65.3 & 31.7 & 35.3 & 72.9 & 86.1 & 50 & 87.8 & 69.3 \\\\ \hline \end{array}$$ **Q2: Is the frozen verifier in Table 2 initialized from the base model or the Stage 1 checkpoint? Are the generation policy’s initialization, training data, and budget otherwise identical to those of full PAVIF? This distinction is important for interpreting the contribution of dynamic self-verification.** **Answer:** Yes. We initialized both the policy and the verifier from the same Stage 1 checkpoint used in PAVIF. We kept the verifier frozen throughout Stage 2 and held all other training settings fixed, including the training data and number of policy updates. We have clarified this setup in Section 5.4 of the revised manuscript. [1] Huang, H., Liu, J., He, Y., Li, S., Xu, B., Zhu, C., Yang, M., & Zhao, T. (2025). MuSC: Improving complex instruction following with multi-granularity self-contrastive training. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 10667–10686). [2] Yuan, W., Pang, R. Y., Cho, K., Li, X., Sukhbaatar, S., Xu, J., & Weston, J. E. (2024). Self-rewarding language models. In Proceedings of the 41st International Conference on Machine Learning, PMLR, 235, 57905–57923.
AuthorsOpenReview ↗

W2 & Q3: Self-verification reliability and reward inflation during Stage 2

W2: The reliability and consistency of the benefits need stronger support. Most verification gains from Stage 1 disappear after Stage 2, yet the reliability of online self-scores is not directly examined. Improvements over VerIF are also small on the 1.7B and 8B models, with no repeated-run results to establish their stability.

Q3: Could existing checkpoints or training logs be used to examine agreement between Stage 2 self-scores and independent judgments—for example, the proportion of responses receiving full self-scores despite violating constraints, and how this changes during training? Results on soft constraints would be particularly informative.

Answer: Thank you for suggesting a direct assessment of scoring reliability. We examine this from two complementary perspectives.

(1) Verification ability on human-annotated data. We evaluated verification accuracy on a validation set of 300 responses annotated by humans with three-level scores (0, 0.5, and 1). As shown in Table B, verification accuracy was largely maintained after Stage 2 training.

(2) Agreement and over-scoring during training. On soft-only examples sampled from the Stage 2 training dataset, we compared each method's training verifier scores with judgments from the independent judge gpt-oss-120B. Although the frozen-reward variant showed higher agreement at some early checkpoints, PAVIF consistently achieved higher agreement and a lower over-scoring rate across the later checkpoints (Table B2). This supports better scoring reliability in later training, without establishing the absence of reward hacking or generalization to held-out data.

Table B: Verifier agreement and over-scoring during Stage 2.

\hline \textbf{Checkpoint} & \textbf{Agreement with human labels, n (\\%)} & \textbf{Overrate ratio, n (\\%)} \\\\ \hline \text{Stage 1 (step 0)} & \text{139 (46.3\\%)} & \text{95 (31.7\\%)} \\\\ \text{Stage 2, step 50} & \text{136 (45.3\\%)} & \text{100 (33.3\\%)} \\\\ \text{Stage 2, step 100} & \text{136 (45.3\\%)} & \text{108 (36.0\\%)} \\\\ \text{Stage 2, step 150} & \text{142 (47.3\\%)} & \text{104 (34.7\\%)} \\\\ \text{Stage 2, step 200} & \text{129 (43.0\\%)} & \text{112 (37.3\\%)} \\\\ \text{Stage 2, step 250} & \text{137 (45.7\\%)} & \text{104 (34.7\\%)} \\\\ \text{Stage 2, step 300} & \text{132 (44.0\\%)} & \text{111 (37.0\\%)} \\\\ \text{Stage 2, step 350} & \text{140 (46.7\\%)} & \text{109 (36.3\\%)} \\\\ \text{Stage 2, step 400} & \text{143 (47.7\\%)} & \text{105 (35.0\\%)} \\\\ \text{Stage 2, step 450} & \text{144 (48.0\\%)} & \text{109 (36.3\\%)} \\\\ \text{Stage 2, step 500} & \text{139 (46.3\\%)} & \text{108 (36.0\\%)} \\\\ \text{PAVIF (Ours), step 550 (final)} & \text{138 (46.0\\%)} & \text{100 (33.3\\%)} \\\\ \hline \end{array}$$ **Table B2: Agreement and over-scoring relative to gpt-oss-120B on soft-only training examples (all values in %).** $$\begin{array}{lcccc} \hline \textbf{Checkpoint} & \textbf{PAVIF agree.} & \textbf{PAVIF over-score} & \textbf{Frozen agree.} & \textbf{Frozen over-score} \\\\ \hline \text{Stage 2, step 50} & 73.6 & 22.0 & 73.3 & 22.6 \\\\ \text{Stage 2, step 100} & 74.5 & 21.4 & 78.5 & 19.4 \\\\ \text{Stage 2, step 150} & 75.1 & 21.9 & 78.0 & 17.6 \\\\ \text{Stage 2, step 200} & 75.6 & 21.4 & 76.9 & 17.9 \\\\ \text{Stage 2, step 250} & 73.2 & 23.7 & 74.3 & 22.6 \\\\ \text{Stage 2, step 300} & 74.0 & 23.0 & 71.3 & 23.6 \\\\ \text{Stage 2, step 350} & 72.4 & 23.6 & 69.2 & 25.7 \\\\ \text{Stage 2, step 400} & 73.7 & 22.9 & 71.0 & 25.6 \\\\ \text{Stage 2, step 450} & 74.7 & 22.0 & 71.0 & 24.8 \\\\ \text{Stage 2, step 500} & 76.3 & 20.3 & 68.9 & 26.4 \\\\ \text{PAVIF (Ours), step 550 (final)} & 75.4 & 20.8 & 72.1 & 22.1 \\\\ \hline \end{array}$$
AuthorsOpenReview ↗

Q4: Consistency of performance gains over VerIF across repeated evaluations (1/2)

Q4: Are repeated training runs or other uncertainty analyses available to assess whether the smaller improvements over VerIF are consistent?

Answer: We thank the reviewer for highlighting the need to quantify variability in the comparison with VerIF. At each model size, we evaluated one checkpoint per method at temperature 0.7 using three decoding seeds (42, 233, and 1024). PAVIF achieves a higher six-benchmark average than VerIF in eight of the nine evaluations.

Table C: Instruction-following Performance (seed 42).

\hline \smash{\lower0.6em{\textbf{Model}}} & \kern3.3651em\rlap{\textbf{IFEval}}\kern-3.3651em & & & & \kern2.1957em\rlap{\textbf{Multi-IF}}\kern-2.1957em & & & \kern2.9191em\rlap{\textbf{IFBench}}\kern-2.9191em & & & & \kern-1.6030em\rlap{\textbf{FollowBench}}\kern1.6030em & & \kern0.7774em\rlap{\textbf{CFBench}}\kern-0.7774em & & & \textbf{InfoBench} & \smash{\lower0.6em{\textbf{Avg.}}} \\\\ & \textbf{Pr.S} & \textbf{Pr.L} & \textbf{Ins.S} & \textbf{Ins.L} & \textbf{Turn 1} & \textbf{Turn 2} & \textbf{Turn 3} & \textbf{Pr.S} & \textbf{Pr.L} & \textbf{Ins.S} & \textbf{Ins.L} & \textbf{HSR} & \textbf{SSR} & \textbf{CSR} & \textbf{ISR} & \textbf{PSR} & \textbf{Overall} & \\\\ \hline \rlap{\textit{Qwen3-1.7B}} & & & & & & & & & & & & & & & & & & \\\\ \text{Base} & 68.4 & 71.5 & 76.6 & 79.7 & 63.5 & 40.8 & 22.1 & 19.0 & 24.0 & 20.9 & 26.5 & 44.9 & 68.4 & 58 & 24 & 31 & 78.4 & 51.9 \\\\ \text{+ VerIF} & 73.2 & 76.2 & 80.8 & 82.9 & 66.2 & 42.7 & 25.9 & 18.3 & 23.0 & 20.3 & 25.6 & 50.3 & 74.2 & 59 & 25 & 33 & 79.0 & 54.2 \\\\ \text{+ PAVIF (Ours)} & 73.6 & 78.6 & 80.9 & 84.9 & 66.3 & 43.5 & 26.2 & 17.0 & 22.7 & 21.2 & 26.2 & 48.5 & 72.4 & 56 & 23 & 30 & 78.7 & 53.7 \\\\ \hline \rlap{\textit{Qwen3-4B-Instruct-2507}} & & & & & & & & & & & & & & & & & & \\\\ \text{Base} & 83.2 & 87.1 & 88.4 & 91.1 & 78.1 & 60.3 & 43.7 & 31.7 & 34.3 & 33.7 & 37.5 & 67.7 & 84.9 & 71 & 43 & 50 & 84.5 & 66.3 \\\\ \text{+ VerIF} & 87.8 & 90.0 & 91.5 & 93.0 & 82.0 & 63.0 & 46.6 & 31.7 & 34.7 & 34.6 & 38.4 & 68.9 & 85.7 & 71 & 42 & 49 & 86.4 & 67.8 \\\\ \text{+ PAVIF (Ours)} & 86.9 & 90.9 & 91.0 & 93.8 & 81.5 & 63.5 & 47.2 & 33.0 & 37.3 & 35.5 & 39.8 & 68.1 & 85.7 & 73 & 44 & 50 & 85.9 & 68.3 \\\\ \hline \rlap{\textit{Qwen3-8B}} & & & & & & & & & & & & & & & & & & \\\\ \text{Base} & 83.9 & 86.3 & 89.1 & 90.6 & 80.4 & 63.3 & 47.4 & 23.7 & 26.7 & 27.0 & 32.0 & 61.9 & 83.6 & 69 & 37 & 45 & 84.6 & 64.4 \\\\ \text{+ VerIF} & 85.4 & 87.6 & 89.9 & 91.5 & 80.0 & 61.6 & 46.4 & 23.0 & 29.3 & 27.0 & 33.4 & 65.3 & 84.3 & 70 & 38 & 46 & 83.9 & 64.9 \\\\ \text{+ PAVIF (Ours)} & 84.8 & 87.8 & 89.3 & 91.4 & 82.2 & 63.7 & 47.5 & 27.0 & 31.7 & 29.7 & 35.5 & 64.0 & 83.8 & 69 & 36 & 45 & 84.4 & 65.3 \\\\ \hline \end{array}$$
AuthorsOpenReview ↗

Q4: Consistency of performance gains over VerIF across repeated evaluations (2/2)

**Q4 (continued).

Table D: Instruction-following Performance (seed 233).

\hline \smash{\lower0.6em{\textbf{Model}}} & \kern3.3651em\rlap{\textbf{IFEval}}\kern-3.3651em & & & & \kern2.1957em\rlap{\textbf{Multi-IF}}\kern-2.1957em & & & \kern2.9191em\rlap{\textbf{IFBench}}\kern-2.9191em & & & & \kern-1.6030em\rlap{\textbf{FollowBench}}\kern1.6030em & & \kern0.7774em\rlap{\textbf{CFBench}}\kern-0.7774em & & & \textbf{InfoBench} & \smash{\lower0.6em{\textbf{Avg.}}} \\\\ & \textbf{Pr.S} & \textbf{Pr.L} & \textbf{Ins.S} & \textbf{Ins.L} & \textbf{Turn 1} & \textbf{Turn 2} & \textbf{Turn 3} & \textbf{Pr.S} & \textbf{Pr.L} & \textbf{Ins.S} & \textbf{Ins.L} & \textbf{HSR} & \textbf{SSR} & \textbf{CSR} & \textbf{ISR} & \textbf{PSR} & \textbf{Overall} & \\\\ \hline \rlap{\textit{Qwen3-1.7B}} & & & & & & & & & & & & & & & & & & \\\\ \text{Base} & 69.7 & 72.3 & 77.7 & 80.1 & 64.1 & 41.0 & 22.5 & 19.3 & 22.0 & 20.6 & 25.0 & 48.3 & 70.7 & 58 & 24 & 31 & 76.9 & 52.2 \\\\ \text{+ VerIF} & 71.0 & 75.2 & 79.5 & 82.9 & 66.7 & 44.1 & 25.3 & 16.3 & 20.3 & 18.6 & 24.1 & 53.0 & 74.5 & 60 & 27 & 34 & 78.6 & 54.1 \\\\ \text{+ PAVIF (Ours)} & 74.3 & 78.4 & 81.5 & 84.8 & 65.8 & 42.5 & 26.0 & 18.7 & 25.3 & 22.4 & 28.2 & 50.0 & 72.9 & 57 & 25 & 32 & 78.3 & 54.3 \\\\ \hline \rlap{\textit{Qwen3-4B-Instruct-2507}} & & & & & & & & & & & & & & & & & & \\\\ \text{Base} & 84.3 & 88.4 & 89.2 & 91.8 & 78.1 & 59.6 & 44.7 & 30.3 & 34.0 & 33.7 & 38.4 & 66.2 & 84.2 & 72 & 44 & 50 & 84.6 & 66.4 \\\\ \text{+ VerIF} & 86.1 & 88.5 & 90.5 & 92.1 & 82.3 & 63.3 & 47.5 & 31.7 & 34.7 & 34.3 & 38.1 & 66.8 & 85.3 & 72 & 43 & 50 & 87.2 & 67.8 \\\\ \text{+ PAVIF (Ours)} & 86.1 & 90.8 & 90.8 & 93.8 & 81.4 & 63.4 & 47.8 & 31.7 & 36.3 & 34.6 & 39.2 & 70.4 & 87.5 & 73 & 44 & 51 & 86.2 & 68.5 \\\\ \hline \rlap{\textit{Qwen3-8B}} & & & & & & & & & & & & & & & & & & \\\\ \text{Base} & 84.5 & 86.7 & 89.3 & 90.9 & 80.5 & 63.4 & 47.4 & 24.7 & 30.0 & 28.5 & 34.0 & 64.3 & 84.2 & 69 & 38 & 46 & 84.4 & 65.1 \\\\ \text{+ VerIF} & 85.2 & 87.4 & 89.9 & 91.5 & 79.1 & 62.0 & 46.4 & 23.7 & 29.0 & 27.3 & 32.8 & 63.4 & 84.4 & 69 & 38 & 46 & 84.9 & 64.8 \\\\ \text{+ PAVIF (Ours)} & 85.8 & 88.5 & 90.3 & 92.2 & 82.2 & 63.7 & 48.5 & 26.0 & 31.3 & 29.4 & 35.2 & 60.9 & 82.0 & 70 & 38 & 46 & 84.7 & 65.3 \\\\ \hline \end{array}$$ **Table E: Instruction-following Performance (seed 1024).** $$\begin{array}{lcccccccccccccccccc} \hline \smash{\lower0.6em{\textbf{Model}}} & \kern3.3651em\rlap{\textbf{IFEval}}\kern-3.3651em & & & & \kern2.1957em\rlap{\textbf{Multi-IF}}\kern-2.1957em & & & \kern2.9191em\rlap{\textbf{IFBench}}\kern-2.9191em & & & & \kern-1.6030em\rlap{\textbf{FollowBench}}\kern1.6030em & & \kern0.7774em\rlap{\textbf{CFBench}}\kern-0.7774em & & & \textbf{InfoBench} & \smash{\lower0.6em{\textbf{Avg.}}} \\\\ & \textbf{Pr.S} & \textbf{Pr.L} & \textbf{Ins.S} & \textbf{Ins.L} & \textbf{Turn 1} & \textbf{Turn 2} & \textbf{Turn 3} & \textbf{Pr.S} & \textbf{Pr.L} & \textbf{Ins.S} & \textbf{Ins.L} & \textbf{HSR} & \textbf{SSR} & \textbf{CSR} & \textbf{ISR} & \textbf{PSR} & \textbf{Overall} & \\\\ \hline \rlap{\textit{Qwen3-1.7B}} & & & & & & & & & & & & & & & & & & \\\\ \text{Base} & 68.9 & 72.5 & 77.0 & 80.0 & 63.5 & 40.3 & 21.8 & 18.7 & 24.0 & 21.8 & 28.2 & 47.7 & 70.6 & 57 & 24 & 30 & 78.7 & 52.4 \\\\ \text{+ VerIF} & 71.5 & 75.0 & 78.9 & 82.1 & 66.3 & 43.2 & 25.5 & 19.7 & 24.3 & 23.3 & 27.9 & 50.3 & 72.4 & 58 & 26 & 34 & 79.0 & 54.2 \\\\ \text{+ PAVIF (Ours)} & 75.4 & 79.9 & 81.7 & 85.1 & 65.7 & 43.4 & 26.1 & 21.0 & 25.0 & 23.0 & 28.8 & 48.1 & 71.4 & 57 & 24 & 30 & 78.9 & 54.3 \\\\ \hline \rlap{\textit{Qwen3-4B-Instruct-2507}} & & & & & & & & & & & & & & & & & & \\\\ \text{Base} & 83.5 & 87.6 & 88.7 & 91.5 & 78.4 & 60.0 & 44.3 & 31.0 & 33.0 & 33.7 & 36.3 & 65.5 & 84.7 & 71 & 42 & 49 & 86.2 & 66.2 \\\\ \text{+ VerIF} & 86.3 & 88.4 & 90.4 & 91.7 & 82.1 & 62.7 & 46.6 & 31.3 & 35.0 & 33.7 & 37.8 & 66.4 & 84.3 & 72 & 42 & 50 & 88.3 & 67.6 \\\\ \text{+ PAVIF (Ours)} & 86.5 & 89.6 & 90.9 & 92.9 & 81.6 & 62.7 & 46.4 & 32.7 & 38.7 & 34.9 & 41.3 & 69.2 & 86.7 & 74 & 42 & 51 & 85.6 & 68.3 \\\\ \hline \rlap{\textit{Qwen3-8B}} & & & & & & & & & & & & & & & & & & \\\\ \text{Base} & 83.5 & 85.8 & 88.5 & 90.0 & 80.0 & 62.7 & 47.4 & 25.3 & 29.3 & 27.6 & 32.0 & 63.0 & 82.8 & 69 & 37 & 44 & 83.7 & 64.3 \\\\ \text{+ VerIF} & 84.5 & 86.5 & 89.1 & 90.6 & 79.8 & 61.8 & 46.7 & 24.3 & 28.7 & 27.3 & 32.6 & 65.5 & 85.1 & 69 & 38 & 47 & 83.6 & 64.8 \\\\ \text{+ PAVIF (Ours)} & 83.9 & 87.1 & 89.1 & 91.2 & 82.3 & 63.8 & 47.9 & 25.3 & 30.0 & 28.5 & 33.7 & 64.7 & 84.8 & 70 & 39 & 47 & 84.8 & 65.6 \\\\ \hline \end{array}$$
Reviewer_TGR7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PAVIF proposes a two-stage reinforcement learning framework for improving instruction following, in which the policy also serves as its own verifier. The authors construct approximately 12,000 verification examples by generating responses to instructions with different subsets of constraints retained, then scoring those responses against the original instructions using a stronger teacher. Stage 1 uses GRPO to train the policy to predict three-level compliance scores. Stage 2 combines the policy’s own verification scores with programmatic constraint checks to optimize response generation. The paper evaluates three Qwen3 checkpoints across six instruction-following benchmarks, alongside verification and general-capability evaluations. It reports an average improvement of 2.01 percentage points over the starting checkpoints and a 23.2% reduction in Stage 2 per-step training time relative to a separate same-scale verifier under the reported configuration.

Critical strengths
  1. The framework addresses a practically relevant verification problem. Soft instruction constraints often require model-based judgments. Using shared parameters for generation and verification offers a concrete way to reduce the overhead of maintaining a separate online verifier. The paper provides a measured Stage 2 efficiency comparison to support this motivation.

  2. The empirical evaluation has useful breadth. Experiments cover three model sizes and six instruction-following benchmarks, with supplementary evaluation using a second judge. The 4B checkpoint shows a meaningful improvement, and the rule-reward and model-reward ablations provide evidence that these signals are complementary. General-capability evaluations also help characterize the consequences of task-specific training.

Critical weaknesses
  1. The baselines do not adequately isolate the value of verification training or the need for RL in Stage 1. The reported SFT baseline learns instruction-to-answer generation, whereas Stage 1 learns instruction–response verification. Consequently, this comparison changes both the learning task and the optimization method. A direct verification-distillation baseline—SFT on teacher verification outputs, followed by the same Stage 2—would more directly test the proposed design. This is particularly consequential because removing Stage 1 reduces the average score by only 0.27 points in Table 2, although the difference is larger, at 1.05 points, under the alternative evaluation in Table 12. No uncertainty estimates are provided for these differences. Additionally, the ordinary SFT baseline is insufficiently specified and does not establish how PAVIF compares with a strong response-distillation baseline using rejection sampling and explicit constraint validation.

  2. The verification-data distribution and scoring granularity are insufficiently validated. Section 4.2 states that constraint removal produces a “diverse and balanced distribution of verification scores,” but the paper does not report actual violation rates or label distributions for the no-, partial-, and full-constraint conditions. Removing a requirement does not necessarily make the model violate it, particularly for behaviors already favored by the model. Teacher labeling avoids automatically treating such responses as negative examples, but it does not establish that the desired distribution was obtained. Furthermore, the three-level score compresses already available per-constraint judgments: substantially different compliance levels can receive the same reward. Neither the three-level discretization nor the 0.5 threshold is compared with binary full-compliance scoring or the fraction of satisfied constraints. These omissions make it difficult to assess the supervision on which the framework relies.

Questions for authors
  1. For each constraint-removal condition, what fraction of responses violates the original instructions, and specifically the removed constraints? Please provide the resulting counts of scores 0, 0.5, and 1, and clarify how the two responses per condition are selected.

  2. What are its training-set size, instruction sources, response-filtering criteria, and training hyperparameters? Does it use the same instruction inventory as PAVIF, and are teacher-generated responses checked for actual constraint satisfaction before training?

  3. How does PAVIF compare with (a) teacher-verification SFT followed by the identical Stage 2, and (b) teacher-response rejection-sampling SFT using candidates validated against the complete original instructions? Comparisons with matched data access and reported teacher and training costs would help establish the additional value of the proposed RL pipeline.

  4. What motivates the three-level scoring scheme and its 0.5 threshold? Is there evidence that it improves verification or downstream instruction following relative to binary full-compliance scoring or directly using the fraction of satisfied constraints?

4 comments
AuthorsOpenReview ↗

W2-1 & Q1–Q2: Verification-data construction, label distributions, and response-SFT configuration

W2-1: The verification-data distribution and scoring granularity are insufficiently validated. Section 4.2 states that constraint removal produces a “diverse and balanced distribution of verification scores,” but the paper does not report actual violation rates or label distributions for the no-, partial-, and full-constraint conditions. Removing a requirement does not necessarily make the model violate it, particularly for behaviors already favored by the model. Teacher labeling avoids automatically treating such responses as negative examples, but it does not establish that the desired distribution was obtained.

Q1: For each constraint-removal condition, what fraction of responses violates the original instructions, and specifically the removed constraints? Please provide the resulting counts of scores 0, 0.5, and 1, and clarify how the two responses per condition are selected.

Answer: We thank the reviewer for pointing out that our sampling procedure was unclear in the appendix. Each retention condition can yield any of the three scores, and we balance the Stage 1 dataset by pooling responses across conditions and sampling by teacher score. For each instruction, we pooled responses generated with no, partial, or full constraint retention. The teacher scored each response against the complete original instruction, and we retained up to two responses per score (0, 0.5, or 1) for that instruction. When fewer than two candidates received a given score, we sampled additional instructions and repeated the procedure until we had 12,000 verification examples. As Table A shows, all three models had a more even distribution of teacher scores with constraint removal and score-based sampling than with rollouts from complete instructions alone. We will clarify this point in the revised manuscript and add this comparison to the appendix.

Table A: Teacher scores with and without constraint removal.

\hline \textbf{Model} & \textbf{Rollout setting} & \textbf{Score 0} & \textbf{Score 0.5} & \textbf{Score 1} \\\\ \hline \text{Qwen3-4B-Instruct-2507} & \begin{array}{l}\text{w/ constraint removal} \\\\ \text{w/o constraint removal}\end{array} & \begin{array}{c}\text{3,969 (33.1\\%)} \\\\ \text{1,864 (15.5\\%)}\end{array} & \begin{array}{c}\text{4,255 (35.5\\%)} \\\\ \text{3,353 (27.9\\%)}\end{array} & \begin{array}{c}\text{3,776 (31.5\\%)} \\\\ \text{6,783 (56.5\\%)}\end{array} \\\\ \hline \text{Qwen3-1.7B} & \begin{array}{l}\text{w/ constraint removal} \\\\ \text{w/o constraint removal}\end{array} & \begin{array}{c}\text{3,787 (31.6\\%)} \\\\ \text{2,769 (23.1\\%)}\end{array} & \begin{array}{c}\text{4,779 (39.8\\%)} \\\\ \text{5,193 (43.3\\%)}\end{array} & \begin{array}{c}\text{3,434 (28.6\\%)} \\\\ \text{4,038 (33.7\\%)}\end{array} \\\\ \hline \text{Qwen3-8B} & \begin{array}{l}\text{w/ constraint removal} \\\\ \text{w/o constraint removal}\end{array} & \begin{array}{c}\text{3,239 (27.0\\%)} \\\\ \text{1,630 (13.6\\%)}\end{array} & \begin{array}{c}\text{4,446 (37.1\\%)} \\\\ \text{4,370 (36.4\\%)}\end{array} & \begin{array}{c}\text{4,315 (36.0\\%)} \\\\ \text{6,000 (50.0\\%)}\end{array} \\\\ \hline \end{array}$$ **Q2: What are its training-set size, instruction sources, response-filtering criteria, and training hyperparameters? Does it use the same instruction inventory as PAVIF, and are teacher-generated responses checked for actual constraint satisfaction before training?** **Answer:** The response-SFT baseline in Table 1 used the same 20,000 instructions from Ren et al. [1] as PAVIF Stage 2. Qwen3-235B-A22B-Instruct-2507 generated one response per instruction (20,000 teacher calls). We retained all responses for training regardless of whether they satisfied every constraint. We fine-tuned for 5 epochs in BF16 with a per-device batch size of 4 and 4 gradient accumulation steps. We used a peak learning rate of 1 × 10⁻⁵ with 10% warmup followed by cosine decay. [1] Qingyu Ren et al. Instructions are all you need: Self-supervised Reinforcement Learning for Instruction Following. ACL, 2026.
AuthorsOpenReview ↗

W1 & Q3: Comparison with verification-SFT and response-distillation baselines

W1: The baselines do not adequately isolate the value of verification training or the need for RL in Stage 1. The reported SFT baseline learns instruction-to-answer generation, whereas Stage 1 learns instruction–response verification. Consequently, this comparison changes both the learning task and the optimization method. A direct verification-distillation baseline—SFT on teacher verification outputs, followed by the same Stage 2—would more directly test the proposed design. This is particularly consequential because removing Stage 1 reduces the average score by only 0.27 points in Table 2, although the difference is larger, at 1.05 points, under the alternative evaluation in Table 12. No uncertainty estimates are provided for these differences. Additionally, the ordinary SFT baseline is insufficiently specified and does not establish how PAVIF compares with a strong response-distillation baseline using rejection sampling and explicit constraint validation.

Q3: How does PAVIF compare with (a) teacher-verification SFT followed by the identical Stage 2, and (b) teacher-response rejection-sampling SFT using candidates validated against the complete original instructions? Comparisons with matched data access and reported teacher and training costs would help establish the additional value of the proposed RL pipeline.

Answer: We thank the reviewer for suggesting that we compare training paradigms for verification training and evaluate PAVIF against a stronger response-distillation baseline. PAVIF+ uses SFT on the teacher’s verification outputs for the same Stage 1 examples as PAVIF, followed by the same Stage 2 procedure (§5.4). For RS-SFT, we generated five teacher responses for each of the 20,000 instructions. We checked each candidate against all constraints in the original instruction using rules and model judgments, then kept the response with the highest fraction of satisfied constraints.

Regarding the cost of training data construction, PAVIF used 12,000 teacher calls to label verification examples. Generating training responses required 20,000 teacher calls for ordinary SFT and 100,000 for RS-SFT.

Owing to potential variation in model judgments across deployments, we reran all evaluations, using gpt-oss-120B as the judge for FollowBench, CFBench, and InfoBench. Each Avg. weights the six benchmarks equally.

Table B: PAVIF ablations on Qwen3-4B-Instruct-2507.

\hline \textbf{Model} & \kern-0.059em\rlap{\textbf{IFEval}}\kern0.059em & & \rlap{\kern2.233em\textbf{Multi-IF}} & & & \kern-0.502em\rlap{\textbf{IFBench}}\kern0.502em & & \kern-1.573em\rlap{\textbf{FollowBench}}\kern1.573em & & \rlap{\kern0.808em\textbf{CFBench}} & & & \textbf{InfoBench} & \textbf{Avg.} \\\\ & \textbf{Pr.S} & \textbf{Pr.L} & \textbf{Turn 1} & \textbf{Turn 2} & \textbf{Turn 3} & \textbf{Pr.S} & \textbf{Pr.L} & \textbf{HSR} & \textbf{SSR} & \textbf{CSR} & \textbf{ISR} & \textbf{PSR} & \textbf{Overall} & \\\\ \hline \rlap{\textit{Qwen3-4B-Instruct-2507}}&&&&&&&&&&&&&&\\\\ \text{Base} & 83.5 & 87.6 & 82.1 & 72.1 & 62.9 & 28.3 & 31.0 & 62.6 & 74.6 & 71 & 45 & 51 & 86.4 & 66.4 \\\\ \text{PAVIF+} & 86.7 & 89.8 & 86.3 & 73.1 & 59.6 & 28.3 & 31.3 & 65.7 & 76.4 & 69 & 43 & 50 & 85.1 & 66.9 \\\\ \text{PAVIF (Ours)} & 88.4 & 91.7 & 84.3 & 74.7 & 65.3 & 31.7 & 35.3 & 72.9 & 86.1 & 74 & 50 & 56 & 87.8 & 70.9 \\\\ \hline \end{array}$$ **Table C: Response-generation baselines.** $$\begin{array}{lcccccccccccccc} \hline \textbf{Model} & \kern-0.059em\rlap{\textbf{IFEval}}\kern0.059em & & \rlap{\kern2.233em\textbf{Multi-IF}} & & & \kern-0.502em\rlap{\textbf{IFBench}}\kern0.502em & & \kern-1.573em\rlap{\textbf{FollowBench}}\kern1.573em & & \rlap{\kern0.808em\textbf{CFBench}} & & & \textbf{InfoBench} & \textbf{Avg.} \\\\ & \textbf{Pr.S} & \textbf{Pr.L} & \textbf{Turn 1} & \textbf{Turn 2} & \textbf{Turn 3} & \textbf{Pr.S} & \textbf{Pr.L} & \textbf{HSR} & \textbf{SSR} & \textbf{CSR} & \textbf{ISR} & \textbf{PSR} & \textbf{Overall} & \\\\ \hline \rlap{\textit{Qwen3-4B-Instruct-2507}}&&&&&&&&&&&&&&\\\\ \text{Base} & 83.5 & 87.6 & 82.1 & 72.1 & 62.9 & 28.3 & 31.0 & 62.6 & 74.6 & 71 & 45 & 51 & 86.4 & 66.4 \\\\ \text{+ SFT} & 84.7 & 88.2 & 83.2 & 71.4 & 60.6 & 30.0 & 35.3 & 68.4 & 84.1 & 69 & 42 & 49 & 85.7 & 67.7 \\\\ \text{+ RS-SFT} & 83.7 & 87.8 & 84.0 & 71.7 & 62.0 & 30.7 & 34.3 & 66.0 & 83.5 & 69 & 42 & 49 & 86.4 & 67.6 \\\\ \text{+ VerIF} & 86.9 & 89.1 & 83.9 & 73.7 & 64.4 & 30.7 & 32.7 & 68.9 & 84.4 & 72 & 46 & 53 & 86.3 & 68.9 \\\\ \text{+ PAVIF (Ours)} & 88.4 & 91.7 & 84.3 & 74.7 & 65.3 & 31.7 & 35.3 & 72.9 & 86.1 & 74 & 50 & 56 & 87.8 & 70.9 \\\\ \hline \end{array}$$
AuthorsOpenReview ↗

W2-2 & Q4: Rationale and empirical evaluation of three-level scoring

W2-2: Furthermore, the three-level score compresses already available per-constraint judgments: substantially different compliance levels can receive the same reward. Neither the three-level discretization nor the 0.5 threshold is compared with binary full-compliance scoring or the fraction of satisfied constraints. These omissions make it difficult to assess the supervision on which the framework relies.

Q4: What motivates the three-level scoring scheme and its 0.5 threshold? Is there evidence that it improves verification or downstream instruction following relative to binary full-compliance scoring or directly using the fraction of satisfied constraints?

Answer: We explored three-level scoring because, under binary scoring, requiring full compliance can make rewards sparse. Proportional scoring gives every constraint equal weight, even when constraints differ in importance, and relies on the judge to distinguish small differences in compliance. Our three-level scheme gives partial credit when at least half the constraints are met and full credit when all are met. This preserves a training signal for partial success and makes the reward less sensitive to small counting errors within each score band. We supplemented our analysis with two ablations, replacing three-level scoring with binary scoring in one and proportional scoring in the other. Three-level scoring achieved the highest mean LLMBar accuracy under both prompts and the highest InfoBench Overall score among the three schemes (§5.5). The prompts are provided at the end of this rebuttal comment, and we will also include them in the appendix.

Table G: LLMBar accuracy by scoring scheme.

\begin{array}{lcccccccccccc} \hline & \kern-0.3960em\rlap{\smash{\lower0.6em{\textbf{Natural}}}}\kern0.3960em & & \gline{23.5} \rlap{\kern7.7860em\textbf{Adversarial}} & & & & & & & & \rlap{\kern0.3970em\smash{\lower0.6em{\textbf{Avg.}}}} & \\\\ \textbf{Model} & \gline{5.25} & & \gline{5.25} \kern-0.7550em\rlap{\textbf{Neighbor}}\kern0.7550em & & \gline{5.25} \kern-0.6388em\rlap{\textbf{GPTInst}}\kern0.6388em & & \gline{5.25} \kern-0.6270em\rlap{\textbf{GPTOut}}\kern0.6270em & & \gline{5.25} \kern-0.3570em\rlap{\textbf{Manual}}\kern0.3570em & & \gline{5.25} & \\\\ & \textbf{V} & \textbf{NR} & \textbf{V} & \textbf{NR} & \textbf{V} & \textbf{NR} & \textbf{V} & \textbf{NR} & \textbf{V} & \textbf{NR} & \textbf{V} & \textbf{NR} \\\\ \hline \text{IF-Verifier} & 70.5 & 71.5 & 53.7 & \mathbf{50.0} & 54.4 & 50.0 & 57.5 & 61.7 & 53.3 & 51.1 & 57.9 & 56.9 \\\\ \hline \rlap{\textit{Qwen3-4B-Instruct-2507}} & & & & & & & & & & & & \\\\ \text{Base} & \mathbf{88.0} & \mathbf{87.0} & 56.3 & 42.9 & 65.8 & 60.9 & \mathbf{66.0} & 59.6 & 60.9 & \mathbf{58.7} & 67.4 & 61.8 \\\\ \text{+ VerIF} & 87.0 & 85.0 & 53.4 & 40.3 & 65.2 & 58.7 & 62.8 & 59.6 & \mathbf{62.0} & 56.5 & 66.1 & 60.0 \\\\ \text{+ PAVIF} & 87.0 & 84.5 & 56.7 & 43.3 & 68.5 & 62.0 & 64.9 & 58.5 & 60.9 & \mathbf{58.7} & 67.6 & 61.4 \\\\ \text{+ Stage 1 (binary)} & \mathbf{88.0} & 86.0 & 55.6 & 43.3 & 66.3 & 61.4 & 64.9 & 58.5 & 59.8 & \mathbf{58.7} & 66.9 & 61.6 \\\\ \text{+ Stage 1 (prop.)} & \mathbf{88.0} & 85.5 & 56.3 & 43.7 & 66.9 & 62.5 & \mathbf{66.0} & 60.6 & 60.9 & 57.6 & 67.6 & 62.0 \\\\ \text{+ Stage 1 (3-level)} & 87.0 & 85.5 & \mathbf{60.8} & 45.2 & \mathbf{71.2} & \mathbf{63.0} & 62.8 & \mathbf{62.8} & \mathbf{62.0} & \mathbf{58.7} & \mathbf{68.8} & \mathbf{63.0} \\\\ \hline \end{array}$$ **Table H: Instruction following by scoring scheme.** $$\begin{array}{lcccccccccccccc} \hline \smash{\lower0.6em{\textbf{Scoring scheme}}} & \kern-0.0590em\rlap{\textbf{IFEval}}\kern0.0590em & & \rlap{\kern2.2335em\textbf{Multi-IF}} & & & \kern-0.5020em\rlap{\textbf{IFBench}}\kern0.5020em & & \kern-1.5735em\rlap{\textbf{FollowBench}}\kern1.5735em & & \rlap{\kern0.8085em\textbf{CFBench}} & & & \textbf{InfoBench} & \smash{\lower0.6em{\textbf{Avg.}}} \\\\ & \textbf{Pr.S} & \textbf{Pr.L} & \textbf{Turn 1} & \textbf{Turn 2} & \textbf{Turn 3} & \textbf{Pr.S} & \textbf{Pr.L} & \textbf{HSR} & \textbf{SSR} & \textbf{CSR} & \textbf{ISR} & \textbf{PSR} & \textbf{Overall} & \\\\ \hline \rlap{\textit{Qwen3-4B-Instruct-2507}} & & & & & & & & & & & & & & \\\\ \text{Base} & 83.5 & 87.6 & 82.1 & 72.1 & 62.9 & 28.3 & 31.0 & 62.6 & 74.6 & 71 & 45 & 51 & 86.4 & 66.4 \\\\ \text{Binary} & 86.3 & 89.5 & 84.1 & 73.7 & 63.9 & 30.0 & 33.3 & 61.4 & 74.4 & 73 & 42 & 50 & 86.4 & 67.1 \\\\ \text{Proportional} & 86.9 & 89.1 & 84.5 & 73.9 & 64.0 & 29.3 & 33.7 & 61.1 & 74.4 & 72 & 43 & 50 & 85.8 & 67.0 \\\\ \text{Three-level (PAVIF)} & 88.4 & 91.7 & 84.3 & 74.7 & 65.3 & 31.7 & 35.3 & 72.9 & 86.1 & 74 & 50 & 56 & 87.8 & 70.9 \\\\ \hline \end{array}$$
AuthorsOpenReview ↗

Q4 : Verification prompts for binary and proportional scoring

Q4. Verification prompts for the scoring ablations

Self-Verification (English) — Binary scoring

You are a rigorous Instruction-Following Evaluation Expert. Your mission is to extract each explicit constraint from the [User Instruction] and verify whether the [Model Answer] satisfies it.

**Tasks**

1. **Extract Constraints**: Identify all explicit functional, formatting, word count, or content-based constraints in the [User Instruction]. Do not introduce evaluation criteria that are not stated in the instruction.
2. **Point-by-Point Verification**: Assign 1 if the [Model Answer] fully satisfies an extracted constraint, and 0 otherwise.
3. **Detailed Assessment**: Provide specific evidence or reasons for success/failure for every single constraint.
4. **Final Scoring**: Assign a final score of 1 if all extracted constraints are assigned 1, and 0 otherwise.

**Scoring Criteria**
Count each extracted constraint equally. Determine the final score only from the constraint-by-constraint scores.

\boxed{1}: Perfect compliance. Followed all extracted constraints with zero omissions.
\boxed{0}: At least one extracted constraint was not fully satisfied.

**Output Format (Strictly Enforced)**
Below is my evaluation of the response:

1. **Constraint Decomposition and Verification**:
[Constraint 1] && [Score 0/1]: [Briefly explain if the constraint was met and why]
[Constraint 2] && [Score 0/1]: [Briefly explain if the constraint was met and why]
(Continue for all identified constraints)

2. **Detailed Assessment Summary**:
Total constraints: [K]; Satisfied constraints: [M]; Satisfaction ratio: [M/K].
[Summarize which constraints were satisfied or violated, then apply the scoring criteria above.]

Based on my assessment, the final score is:
Score=\boxed{[0 or 1]}

Here is your task input:

[User Instruction]: {question}
[Model Answer]: {answer}
[Your Evaluation]:

Self-Verification (English) — Proportional scoring

You are a rigorous Instruction-Following Evaluation Expert. Your mission is to extract each explicit constraint from the [User Instruction] and verify whether the [Model Answer] satisfies it.

**Tasks**

1. **Extract Constraints**: Identify all explicit functional, formatting, word count, or content-based constraints in the [User Instruction]. Do not introduce evaluation criteria that are not stated in the instruction.
2. **Point-by-Point Verification**: Assign 1 if the [Model Answer] fully satisfies an extracted constraint, and 0 otherwise.
3. **Detailed Assessment**: Provide specific evidence or reasons for success/failure for every single constraint.
4. **Final Scoring**: Assign a final score equal to the proportion of extracted constraints assigned 1.

**Scoring Criteria**
Count each extracted constraint equally. Determine the final score only from the constraint-by-constraint scores.

\boxed{M/K}: M is the number of satisfied constraints and K is the total number of extracted constraints. Substitute the counts and output the exact fraction.

**Output Format (Strictly Enforced)**
Below is my evaluation of the response:

1. **Constraint Decomposition and Verification**:
[Constraint 1] && [Score 0/1]: [Briefly explain if the constraint was met and why]
[Constraint 2] && [Score 0/1]: [Briefly explain if the constraint was met and why]
(Continue for all identified constraints)

2. **Detailed Assessment Summary**:
Total constraints: [K]; Satisfied constraints: [M]; Satisfaction ratio: [M/K].
[Summarize which constraints were satisfied or violated, then apply the scoring criteria above.]

Based on my assessment, the final score is:
Score=\boxed{[M/K]}

Here is your task input:

[User Instruction]: {question}
[Model Answer]: {answer}
[Your Evaluation]:
Reviewer_MrNsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PAVIF, a two-stage GRPO framework where the policy serves as its own verifier for instruction following. In Stage 1, the policy learns to predict three-level compliance scores labeled by a larger teacher model. In Stage 2, the same policy scores its own responses, and this self-verification score is combined with rule-based rewards for hard constraints. On three Qwen3 models and six instruction-following benchmarks, the method improves over the starting checkpoints by about 2 points on average. The authors also report a 23.2% lower per-step time than with a separate frozen verifier of the same size.

Critical strengths
  • Reducing reliance on external verifiers for soft constraints is a practically relevant problem. Keeping a separate judge around during RL adds overhead. If the policy could judge soft constraints well enough on its own, the training pipeline would be simpler and cheaper to run.
  • The experimental evaluation covers six instruction-following benchmarks (IFEval, Multi-IF, IFBench, FollowBench, CFBench, and InFoBench), four general capability benchmarks, and three model scales from 1.7B to 8B. This gives a wide empirical view of how the method behaves.
Critical weaknesses
  • Relative to prior work, the contribution feels somewhat incremental. Using the policy itself to judge constraint satisfaction has been explored in SEIF [1], and the hybrid rule plus LLM reward builds on VerIF [2]. As far as I can see, the main additions are the verification training stage and full parameter sharing. A direct comparison with SEIF would help clarify what these additions bring. So would a comparison with Ren et al. [3], since PAVIF trains on their data and they also propose an RL method for this setting.
  • The improvement over VerIF is small. On Qwen3-8B, the average gap is below 0.2 points under both judges, and PAVIF is behind VerIF on several benchmarks. Without results over multiple runs, it is hard to tell whether the gap is meaningful.
  • The ablations in Table 2 give limited support for Stage 1, which is motivated by the claim in Section 3.2 that verification training alone improves generation. Running Stage 2 directly from the base model (”w/o Stage 1”) gives almost the same average as the full method and is even better on some metrics. Meanwhile, the Stage 1-only model (”w/o Stage 2”) beats the base model on some benchmarks but falls behind on others, such as IFEvalPr.L, Multi-IF, and InFoBench, so the pass@k finding based on 100 prompts per benchmark does not hold consistently in the full evaluation.
  • Verification accuracy on LLMBar [4] goes down after Stage 2, and under the NoRules setting it ends up below the base model. Nothing in Stage 2 seems to keep the verifier accurate, so I'm worried the self-reward becomes less reliable, or more lenient, as training goes on. This sits a bit uneasily with the claim that the gains come from dynamic self-verification.
  • The speedup may depend heavily on the deployment setup. With the same model size and token counts, sharing parameters doesn't reduce the compute needed for verification, and the paper doesn't say where the saving comes from. The method is also about twice as slow as rule-only training, and the cost of teacher labeling and Stage 1 is left out of the comparison.

[1] Ren, Q., He, Q., Zhu, J., Chen, X., Chang, J., Sun, Z., Xia, H., Yu, F., Liang, J., and Xiao, Y. SEIF: Self-evolving reinforcement learning for instruction following. arXiv preprint arXiv:2605.07465, 2026.

[2] Peng, H., Qi, Y., Wang, X., Xu, B., Hou, L., and Li, J. VerIF: Verification engineering for reinforcement learning in instruction following. In EMNLP, 2025.

[3] Ren, Q., He, Q., Chang, P., Zeng, J., Sun, Z., Yu, F., Liang, J., and Xiao, Y. Instructions are all you need: Self-supervised reinforcement learning for instruction following. In ACL, 2026.

[4] Zeng, Z., Yu, J., Gao, T., Meng, Y., Goyal, T., and Chen, D. Evaluating large language models at evaluating instruction following. In ICLR, 2024.

Questions for authors
  • Q1: In Stage 2, the policy judges its own outputs. How do you rule out the policy learning to make its own verification more lenient to get higher rewards? Is the drop on LLMBar a sign of this?
  • Q2: LLMBar is a reasonable choice, but it is pairwise, while your verifier gives pointwise 3-level scores. Could you check the verifier on independently human-annotated multi-constraint responses, using the same 3-level scoring, for Base, Stage 1, and the final model? It would also help to report how well the teacher labels agree with human labels, since the paper only says a subset was reviewed manually.
  • Q3: Could you break down the per-step time for PAVIF, GRPO-4B, and Rule? An external verifier could score one batch while the policy generates the next. Under the same GPU budget, would PAVIF still be faster than that kind of pipelined setup?
8 comments
AuthorsOpenReview ↗

W1: Methodological distinctions and baseline comparisons

W1-1: Relative to prior work, the contribution feels somewhat incremental. Using the policy itself to judge constraint satisfaction has been explored in SEIF [1], and the hybrid rule plus LLM reward builds on VerIF [2]. As far as I can see, the main additions are the verification training stage and full parameter sharing.

Answer: We thank the reviewer for raising this point and would like to clarify the novelty of our work. (1) In SEIF’s self-play framework [1], the Instructor generates harder instructions as the Follower improves. A frozen copy of the previous round’s Follower serves as the Judger. PAVIF, by contrast, asks whether learning to verify constraint satisfaction can itself help a model follow instructions. To this end, we initialize the policy from the trained verifier and share all parameters between the two roles throughout generation training. (2) Unlike VerIF [2] and verl-if (Ren et al. [3]), our policy checks its own responses against the original instruction without a separate list of soft constraints.

W1-2: A direct comparison with SEIF would help clarify what these additions bring. So would a comparison with Ren et al. [3], since PAVIF trains on their data and they also propose an RL method for this setting.

Answer: Table A compares PAVIF, SEIF [1], and Verl-if (Ren et al. [3]) on Qwen3-4B-Instruct-2507. Owing to potential variation in model judgments across deployments, we reran all evaluations, using gpt-oss-120B as the judge for FollowBench, CFBench, and InfoBench.

Table A: Baseline comparison on Qwen3-4B-Instruct-2507.

\hline \smash{\lower0.6em{\textbf{Model}}} & \kern-0.0785em\rlap{\textbf{IFEval}}\kern0.0785em & & \kern2.1957em\rlap{\textbf{Multi-IF}}\kern-2.1957em & & & \kern-0.5245em\rlap{\textbf{IFBench}}\kern0.5245em & & \kern-1.6030em\rlap{\textbf{FollowBench}}\kern1.6030em & & \kern0.7774em\rlap{\textbf{CFBench}}\kern-0.7774em & & & \textbf{InfoBench} & \smash{\lower0.6em{\textbf{Avg.}}} \\\\ & \textbf{Pr.S} & \textbf{Pr.L} & \textbf{Turn 1} & \textbf{Turn 2} & \textbf{Turn 3} & \textbf{Pr.S} & \textbf{Pr.L} & \textbf{HSR} & \textbf{SSR} & \textbf{CSR} & \textbf{ISR} & \textbf{PSR} & \textbf{Overall} & \\\\ \hline \rlap{\textit{Qwen3-4B-Instruct-2507}} & & & & & & & & & & & & & & \\\\ \text{Base} & 83.5 & 87.6 & 82.1 & 72.1 & 62.9 & 28.3 & 31.0 & 62.6 & 74.6 & 71 & 45 & 51 & 86.4 & 66.4 \\\\ \text{+ VerIF} & 86.9 & 89.1 & 83.9 & 73.7 & 64.4 & 30.7 & 32.7 & 68.9 & 84.4 & 72 & 46 & 53 & 86.3 & 68.9 \\\\ \text{+ SEIF} & 82.6 & 87.3 & 81.6 & 71.3 & 62.0 & 30.7 & 34.0 & 63.8 & 75.6 & 73 & 49 & 54 & 86.8 & 67.4 \\\\ \text{+ Verl-if} & 86.0 & 91.7 & 83.5 & 73.3 & 63.2 & 28.7 & 35.0 & 73.2 & 85.3 & 75 & 53 & 59 & 88.1 & 70.6 \\\\ \text{+ PAVIF (Ours)} & 88.4 & 91.7 & 84.3 & 74.7 & 65.3 & 31.7 & 35.3 & 72.9 & 86.1 & 74 & 50 & 56 & 87.8 & 70.9 \\\\ \hline \end{array}$$
AuthorsOpenReview ↗

W2: Comparison with VerIF across repeated evaluations (1/2)

W2: The improvement over VerIF is small. On Qwen3-8B, the average gap is below 0.2 points under both judges, and PAVIF is behind VerIF on several benchmarks. Without results over multiple runs, it is hard to tell whether the gap is meaningful.

Answer: We thank the reviewer for highlighting the small margin over VerIF on Qwen3-8B. We re-evaluated both checkpoints at temperature 0.7 with three decoding seeds (42, 233, and 1024). PAVIF generally achieves a higher six-benchmark average than VerIF across model sizes and decoding seeds (Tables B–D).

Table B: Instruction-following performance (seed 42).

\hline \smash{\lower0.6em{\textbf{Model}}} & \kern3.3651em\rlap{\textbf{IFEval}}\kern-3.3651em & & & & \kern2.1957em\rlap{\textbf{Multi-IF}}\kern-2.1957em & & & \kern2.9191em\rlap{\textbf{IFBench}}\kern-2.9191em & & & & \kern-1.6030em\rlap{\textbf{FollowBench}}\kern1.6030em & & \kern0.7774em\rlap{\textbf{CFBench}}\kern-0.7774em & & & \textbf{InfoBench} & \smash{\lower0.6em{\textbf{Avg.}}} \\\\ & \textbf{Pr.S} & \textbf{Pr.L} & \textbf{Ins.S} & \textbf{Ins.L} & \textbf{Turn 1} & \textbf{Turn 2} & \textbf{Turn 3} & \textbf{Pr.S} & \textbf{Pr.L} & \textbf{Ins.S} & \textbf{Ins.L} & \textbf{HSR} & \textbf{SSR} & \textbf{CSR} & \textbf{ISR} & \textbf{PSR} & \textbf{Overall} & \\\\ \hline \rlap{\textit{Qwen3-1.7B}} & & & & & & & & & & & & & & & & & & \\\\ \text{Base} & 68.4 & 71.5 & 76.6 & 79.7 & 63.5 & 40.8 & 22.1 & 19.0 & 24.0 & 20.9 & 26.5 & 44.9 & 68.4 & 58 & 24 & 31 & 78.4 & 51.9 \\\\ \text{+ VerIF} & 73.2 & 76.2 & 80.8 & 82.9 & 66.2 & 42.7 & 25.9 & 18.3 & 23.0 & 20.3 & 25.6 & 50.3 & 74.2 & 59 & 25 & 33 & 79.0 & 54.2 \\\\ \text{+ PAVIF (Ours)} & 73.6 & 78.6 & 80.9 & 84.9 & 66.3 & 43.5 & 26.2 & 17.0 & 22.7 & 21.2 & 26.2 & 48.5 & 72.4 & 56 & 23 & 30 & 78.7 & 53.7 \\\\ \hline \rlap{\textit{Qwen3-4B-Instruct-2507}} & & & & & & & & & & & & & & & & & & \\\\ \text{Base} & 83.2 & 87.1 & 88.4 & 91.1 & 78.1 & 60.3 & 43.7 & 31.7 & 34.3 & 33.7 & 37.5 & 67.7 & 84.9 & 71 & 43 & 50 & 84.5 & 66.3 \\\\ \text{+ VerIF} & 87.8 & 90.0 & 91.5 & 93.0 & 82.0 & 63.0 & 46.6 & 31.7 & 34.7 & 34.6 & 38.4 & 68.9 & 85.7 & 71 & 42 & 49 & 86.4 & 67.8 \\\\ \text{+ PAVIF (Ours)} & 86.9 & 90.9 & 91.0 & 93.8 & 81.5 & 63.5 & 47.2 & 33.0 & 37.3 & 35.5 & 39.8 & 68.1 & 85.7 & 73 & 44 & 50 & 85.9 & 68.3 \\\\ \hline \rlap{\textit{Qwen3-8B}} & & & & & & & & & & & & & & & & & & \\\\ \text{Base} & 83.9 & 86.3 & 89.1 & 90.6 & 80.4 & 63.3 & 47.4 & 23.7 & 26.7 & 27.0 & 32.0 & 61.9 & 83.6 & 69 & 37 & 45 & 84.6 & 64.4 \\\\ \text{+ VerIF} & 85.4 & 87.6 & 89.9 & 91.5 & 80.0 & 61.6 & 46.4 & 23.0 & 29.3 & 27.0 & 33.4 & 65.3 & 84.3 & 70 & 38 & 46 & 83.9 & 64.9 \\\\ \text{+ PAVIF (Ours)} & 84.8 & 87.8 & 89.3 & 91.4 & 82.2 & 63.7 & 47.5 & 27.0 & 31.7 & 29.7 & 35.5 & 64.0 & 83.8 & 69 & 36 & 45 & 84.4 & 65.3 \\\\ \hline \end{array}$$
AuthorsOpenReview ↗

W2: Comparison with VerIF across repeated evaluations (2/2)

Table C: Instruction-following performance (seed 233).

\hline \smash{\lower0.6em{\textbf{Model}}} & \kern3.3651em\rlap{\textbf{IFEval}}\kern-3.3651em & & & & \kern2.1957em\rlap{\textbf{Multi-IF}}\kern-2.1957em & & & \kern2.9191em\rlap{\textbf{IFBench}}\kern-2.9191em & & & & \kern-1.6030em\rlap{\textbf{FollowBench}}\kern1.6030em & & \kern0.7774em\rlap{\textbf{CFBench}}\kern-0.7774em & & & \textbf{InfoBench} & \smash{\lower0.6em{\textbf{Avg.}}} \\\\ & \textbf{Pr.S} & \textbf{Pr.L} & \textbf{Ins.S} & \textbf{Ins.L} & \textbf{Turn 1} & \textbf{Turn 2} & \textbf{Turn 3} & \textbf{Pr.S} & \textbf{Pr.L} & \textbf{Ins.S} & \textbf{Ins.L} & \textbf{HSR} & \textbf{SSR} & \textbf{CSR} & \textbf{ISR} & \textbf{PSR} & \textbf{Overall} & \\\\ \hline \rlap{\textit{Qwen3-1.7B}} & & & & & & & & & & & & & & & & & & \\\\ \text{Base} & 69.7 & 72.3 & 77.7 & 80.1 & 64.1 & 41.0 & 22.5 & 19.3 & 22.0 & 20.6 & 25.0 & 48.3 & 70.7 & 58 & 24 & 31 & 76.9 & 52.2 \\\\ \text{+ VerIF} & 71.0 & 75.2 & 79.5 & 82.9 & 66.7 & 44.1 & 25.3 & 16.3 & 20.3 & 18.6 & 24.1 & 53.0 & 74.5 & 60 & 27 & 34 & 78.6 & 54.1 \\\\ \text{+ PAVIF (Ours)} & 74.3 & 78.4 & 81.5 & 84.8 & 65.8 & 42.5 & 26.0 & 18.7 & 25.3 & 22.4 & 28.2 & 50.0 & 72.9 & 57 & 25 & 32 & 78.3 & 54.3 \\\\ \hline \rlap{\textit{Qwen3-4B-Instruct-2507}} & & & & & & & & & & & & & & & & & & \\\\ \text{Base} & 84.3 & 88.4 & 89.2 & 91.8 & 78.1 & 59.6 & 44.7 & 30.3 & 34.0 & 33.7 & 38.4 & 66.2 & 84.2 & 72 & 44 & 50 & 84.6 & 66.4 \\\\ \text{+ VerIF} & 86.1 & 88.5 & 90.5 & 92.1 & 82.3 & 63.3 & 47.5 & 31.7 & 34.7 & 34.3 & 38.1 & 66.8 & 85.3 & 72 & 43 & 50 & 87.2 & 67.8 \\\\ \text{+ PAVIF (Ours)} & 86.1 & 90.8 & 90.8 & 93.8 & 81.4 & 63.4 & 47.8 & 31.7 & 36.3 & 34.6 & 39.2 & 70.4 & 87.5 & 73 & 44 & 51 & 86.2 & 68.5 \\\\ \hline \rlap{\textit{Qwen3-8B}} & & & & & & & & & & & & & & & & & & \\\\ \text{Base} & 84.5 & 86.7 & 89.3 & 90.9 & 80.5 & 63.4 & 47.4 & 24.7 & 30.0 & 28.5 & 34.0 & 64.3 & 84.2 & 69 & 38 & 46 & 84.4 & 65.1 \\\\ \text{+ VerIF} & 85.2 & 87.4 & 89.9 & 91.5 & 79.1 & 62.0 & 46.4 & 23.7 & 29.0 & 27.3 & 32.8 & 63.4 & 84.4 & 69 & 38 & 46 & 84.9 & 64.8 \\\\ \text{+ PAVIF (Ours)} & 85.8 & 88.5 & 90.3 & 92.2 & 82.2 & 63.7 & 48.5 & 26.0 & 31.3 & 29.4 & 35.2 & 60.9 & 82.0 & 70 & 38 & 46 & 84.7 & 65.3 \\\\ \hline \end{array}$$ **Table D: Instruction-following performance (seed 1024).** $$\begin{array}{lcccccccccccccccccc} \hline \smash{\lower0.6em{\textbf{Model}}} & \kern3.3651em\rlap{\textbf{IFEval}}\kern-3.3651em & & & & \kern2.1957em\rlap{\textbf{Multi-IF}}\kern-2.1957em & & & \kern2.9191em\rlap{\textbf{IFBench}}\kern-2.9191em & & & & \kern-1.6030em\rlap{\textbf{FollowBench}}\kern1.6030em & & \kern0.7774em\rlap{\textbf{CFBench}}\kern-0.7774em & & & \textbf{InfoBench} & \smash{\lower0.6em{\textbf{Avg.}}} \\\\ & \textbf{Pr.S} & \textbf{Pr.L} & \textbf{Ins.S} & \textbf{Ins.L} & \textbf{Turn 1} & \textbf{Turn 2} & \textbf{Turn 3} & \textbf{Pr.S} & \textbf{Pr.L} & \textbf{Ins.S} & \textbf{Ins.L} & \textbf{HSR} & \textbf{SSR} & \textbf{CSR} & \textbf{ISR} & \textbf{PSR} & \textbf{Overall} & \\\\ \hline \rlap{\textit{Qwen3-1.7B}} & & & & & & & & & & & & & & & & & & \\\\ \text{Base} & 68.9 & 72.5 & 77.0 & 80.0 & 63.5 & 40.3 & 21.8 & 18.7 & 24.0 & 21.8 & 28.2 & 47.7 & 70.6 & 57 & 24 & 30 & 78.7 & 52.4 \\\\ \text{+ VerIF} & 71.5 & 75.0 & 78.9 & 82.1 & 66.3 & 43.2 & 25.5 & 19.7 & 24.3 & 23.3 & 27.9 & 50.3 & 72.4 & 58 & 26 & 34 & 79.0 & 54.2 \\\\ \text{+ PAVIF (Ours)} & 75.4 & 79.9 & 81.7 & 85.1 & 65.7 & 43.4 & 26.1 & 21.0 & 25.0 & 23.0 & 28.8 & 48.1 & 71.4 & 57 & 24 & 30 & 78.9 & 54.3 \\\\ \hline \rlap{\textit{Qwen3-4B-Instruct-2507}} & & & & & & & & & & & & & & & & & & \\\\ \text{Base} & 83.5 & 87.6 & 88.7 & 91.5 & 78.4 & 60.0 & 44.3 & 31.0 & 33.0 & 33.7 & 36.3 & 65.5 & 84.7 & 71 & 42 & 49 & 86.2 & 66.2 \\\\ \text{+ VerIF} & 86.3 & 88.4 & 90.4 & 91.7 & 82.1 & 62.7 & 46.6 & 31.3 & 35.0 & 33.7 & 37.8 & 66.4 & 84.3 & 72 & 42 & 50 & 88.3 & 67.6 \\\\ \text{+ PAVIF (Ours)} & 86.5 & 89.6 & 90.9 & 92.9 & 81.6 & 62.7 & 46.4 & 32.7 & 38.7 & 34.9 & 41.3 & 69.2 & 86.7 & 74 & 42 & 51 & 85.6 & 68.3 \\\\ \hline \rlap{\textit{Qwen3-8B}} & & & & & & & & & & & & & & & & & & \\\\ \text{Base} & 83.5 & 85.8 & 88.5 & 90.0 & 80.0 & 62.7 & 47.4 & 25.3 & 29.3 & 27.6 & 32.0 & 63.0 & 82.8 & 69 & 37 & 44 & 83.7 & 64.3 \\\\ \text{+ VerIF} & 84.5 & 86.5 & 89.1 & 90.6 & 79.8 & 61.8 & 46.7 & 24.3 & 28.7 & 27.3 & 32.6 & 65.5 & 85.1 & 69 & 38 & 47 & 83.6 & 64.8 \\\\ \text{+ PAVIF (Ours)} & 83.9 & 87.1 & 89.1 & 91.2 & 82.3 & 63.8 & 47.9 & 25.3 & 30.0 & 28.5 & 33.7 & 64.7 & 84.8 & 70 & 39 & 47 & 84.8 & 65.6 \\\\ \hline \end{array}$$

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.