- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work presents Verifier-Informed Refinement and Replay Optimization for multi-constraint instruction following. It augments group-relative reinforcement learning with two components: Alias refinement, which uses historical constraint-family difficulty to distinguish responses receiving the same scalar credit, and Peer replay, which adds likelihood supervision on verified successful responses from the same rollout group. Experiments cover five instruction-tuned backbones, four programmatically verified benchmarks, and four semantically judged benchmarks.
Critical strengths
- The motivation is reasonable. Responses with the same aggregate reward can fail different constraints, and successful responses within a rollout group provide potentially useful supervision.
Critical weaknesses
-
The motivation identifies a meaningful problem, but the proposed solutions do not address it as directly as the framing suggests. In Section 3.4, the verifier vector is converted into a difficulty-weighted score and ultimately a scalar advantage for each response. This does recover some distinctions lost by uniform averaging, but it remains unclear whether the benefit reflects learning which constraints need correction or simply changing the relative weights of responses. Section 3.5 similarly adds likelihood supervision on successful samples. Positive-example weighting has precedent in alignment methods such as KTO, which allows separate weights for desirable and undesirable outputs, although its objective differs from Peer replay.
-
The connection between the two components, and their specific relevance to instruction following, is not sufficiently established. Alias refinement could apply to any task whose reward aggregates multiple rubric scores, while reinforcing verified successes is even more general. The existing ablations are useful, yet the shared-pipeline control in Table 3 shows a relatively modest joint gain on Qwen2.5-3B. More diagnostic evidence is needed to identify which instruction-following bottleneck these components jointly resolve. Alternatively, evaluation on another task with multiple rubrics would help substantiate their broader applicability.
-
Most evaluated backbones are from older model families, and the experiments do not establish applicability to long chain-of-thought generation. Appendix B.5 does include Qwen3-4B, but new Qwen and recent models have much higher baseline scores on IFbench and other instruction following benchmarks. Moreover, Table 2 and Appendix B.1 specify a response limit of 1,280 tokens, with length penalties beginning above 1,024 tokens. This setting leaves open how the method behaves when training long-cot models.
Questions for authors
see weakness