acceptodds
Under review as a conference paper at ICLR 2027

Constraint-Decoupled On-Policy Distillation for Multi-Constraint Instruction Following

Abstract

Multi-constraint instruction following requires a model to satisfy every constraint in an instruction simultaneously. We show that the dominant failure mode of current LLMs is partial satisfaction: the all-satisfy rate (ASR) collapses as constraints accumulate, while the per-constraint satisfaction rate stays above 89%. Neither scalar RLVR nor standard on-policy distillation repairs this, because both discard constraint-level structure: RLVR compresses all constraint evaluations into a single reward, and on-policy distillation conditions the teacher on the full instruction, so the student inherits the teacher's own compositional decay. We propose Constraint-Decoupled On-Policy Distillation (CD-OPD), which decouples the instruction into single-constraint prompts, scores each constraint's token-level effect against an unconstrained base, and combines them through a differentiable min-pooling operator. This operator acts as a soft veto—a token violating any constraint is suppressed—while a full-instruction anchor preserves fluency, and the resulting target is distilled into the student along its own on-policy trajectories. Across three teacher–student scales(0.6/1.7/4B) and four benchmarks, CD-OPD improves average ASR over the base student by 4.48%/2.54%/3.57%, reduces the FollowBench decay from 36.6% to 24.5% over one to five constraints, while training substantially faster than RLVR.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM, Instruction Following, On-Policy Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_j9QBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies multi-constraint instruction following and identifies partial satisfaction as a primary failure mode. While models maintain high per-constraint satisfaction, their all-satisfaction rate drops substantially as the number of constraints increases. The paper proposes Constraint-Decoupled On-Policy Distillation (CD-OPD), which decomposes an instruction into single-constraint prompts, estimates the token-level effect of each constraint against an unconstrained prompt, and combines these signals using soft min-pooling. The resulting correction is added to a full-instruction teacher distribution and distilled into the student on its own trajectories. Experiments on three Qwen3 teacher–student pairs and four instruction-following benchmarks show consistent improvements over standard OPD and reduced performance decay as the number of constraints increases.

Critical strengths
  1. CD-OPD offers a clear and well-motivated approach to multi-constraint instruction following. By retaining constraint-level information through decomposition and min-pooling, it directly addresses the partial-satisfaction problem that scalar RLVR and standard OPD struggle with.
  2. The experiments provide useful insights beyond the main performance gains. In particular, the comparisons show the different behaviors of code-based and LLM-based rewards, how standard OPD inherits the teacher’s compositional decay, and how CD-OPD changes performance as the number of constraints increases. The ablations and efficiency analysis further help explain where the gains come from.
Critical weaknesses
  1. The presentation of the core method contains several inconsistencies that make it difficult to verify exactly what was implemented. The main text defines the contrastive signal using raw logit differences and describes zero-mean, unit-variance normalization, while Appendix E uses log-probability differences and uncentered RMS normalization. The claim that min-pooling “guarantees” strict conjunction is also not formally established, and the reported N+1N+1 teacher forward passes appear inconsistent with the N+2N+2 views required by the formulation. These issues suggest that the paper has not been sufficiently proofread and weaken confidence in the technical claims and efficiency analysis.

  2. Although the experiments show consistent improvements, they are not broad enough to support the paper’s general claims. All three teacher–student settings use models from the Qwen3 family and share the same training dataset. The evaluation also does not establish whether the method remains effective or computationally practical when the number of constraints grows beyond the tested range. It is therefore unclear how well the findings generalize across model families, training data, and more complex multi-constraint settings.

Questions for authors
  1. I may be missing something here. The main text uses raw logit differences with zero-mean normalization, while Appendix E uses log-probability differences with uncentered RMS normalization. Since the latter is not shift-invariant, why does the partition-function offset have no effect? Which formulation was used for the reported results?

  2. The paper claims that min-pooling “guarantees” strict conjunction, but it only operates on next-token distributions. Does this guarantee extend to sequence-level constraints? If not, the claim should be stated more cautiously.

  3. I may have misunderstood the computation count, but the method appears to require N+2N+2 teacher views, while the paper reports N+1N+1. Is one distribution reused or omitted, and are all teacher computations included in the reported training time?

  4. All teacher–student settings use Qwen3 models, and constraint counts are tested only within a limited range. Is there evidence that CD-OPD generalizes to other model families or scales to more constraints? Otherwise, the generality claims should be narrowed.

Clear answers to these questions, particularly the first three, would materially affect my assessment. I would consider updating my score if these concerns are satisfactorily resolved.

Reviewer_kei2OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper focuses on training methods for multi-constraint instruction following. It analyzes the limitations of existing approaches and proposes an enhancement to the traditional OPD method by adjusting the teacher distribution based on distributional differences under individual constraints. The experimental results compare the performance of proposed approach with standard OPD and GRPO.

Critical strengths
  1. The paper analyzes the limitations of existing approaches and attempts to address them with a new method, which is a commendable effort.

  2. The paper leverages the observation that models perform well under individual constraints but struggle to satisfy multiple constraints simultaneously. Based on this insight, it proposes using logits from single-constraint scenarios to adjust those under multiple constraints. This approach has certain reference value for related research.

Critical weaknesses
  1. The experimental design uses different training data subsets, such as “Hard,” “Soft,” and the full dataset, across different approaches, undermining the fairness of the comparisons in Table 2. In addition, the paper does not provide essential experimental details, such as the amount of training data and the number of training steps. These omissions hinder reproducibility and weaken the persuasiveness of the experimental results.

  2. Based on the reported results, the proposed CD-OPD does not outperform the traditional GRPO (Hard+Code) baseline. Moreover, CD-OPD requires an additional teacher model, which introduces extra computational costs and further limits its practical value. If the primary goal of the paper is to demonstrate that the proposed variant improves upon standard OPD, the authors should more clearly establish the significance of these improvements and reconsider the overall motivation of the work in light of the competitive performance of existing alternatives.

  3. To demonstrate the generalizability of the proposed approach, the authors should evaluate it across a broader range of domains, tasks, and model families. Currently, the method primarily focuses on improving OPD for the specific task of multi-constraint instruction following, with experiments conducted exclusively on small Qwen3 models (4B parameters or fewer). This narrow evaluation scope makes it difficult to assess whether the approach generalizes beyond the current setting and may limit its broader impact and applicability.

Questions for authors
  1. The results in Table 2 show that, under certain settings, training with OPD or GRPO actually degrades model performance. This finding warrants further investigation to determine whether the degradation stems from an inappropriate experimental setup, an inaccurate reward signal, or other underlying factors.

  2. The word “equation” appears unnecessarily on line 315 and should be removed.

Reviewer_EcqhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Constraint-Decoupled On-Policy Distillation (CD-OPD) to improve multi-constraint instruction following. The method decomposes complex instructions into individual constraints, extracts token-level contrastive signals from a teacher model, and aggregates them through soft-min pooling to construct a refined distillation target. The student is then trained using on-policy KL distillation. Experiments across three teacher–student scales and four instruction-following benchmarks show consistent improvements over standard OPD, particularly as the number of constraints increases.

Critical strengths
  • Well-motivated problem formulation. The pilot studies identify partial constraint satisfaction as a major challenge and provide useful empirical motivation for constraint-aware supervision.

  • Consistent improvements over standard OPD. CD-OPD improves performance across three model scales and four benchmarks. The ablation studies also support the effectiveness of contrastive signals and soft-min aggregation.

Critical weaknesses
  • Limited novelty and insufficient comparisons. The method primarily combines existing techniques, including contrastive log-probability differences, soft-min pooling, and OPD. Comparisons with alternative distribution-composition methods are missing. Moreover, the strongest code-verified RLVR baseline outperforms CD-OPD across all reported benchmarks and model scales.

  • Insufficient theoretical justification. Token-level soft-min pooling does not directly guarantee strict sequence-level constraint satisfaction. Additionally, the claimed equivalence between raw-logit differences and log-probability differences under RMS normalization and soft-min aggregation is not sufficiently justified.

Questions for authors
  1. What is the key methodological novelty beyond combining contrastive log-probability differences, soft-min pooling, and standard OPD? Could the authors compare against alternative distribution-composition methods, such as additive logit composition or product-of-experts?

  2. How does token-level soft-min pooling guarantee strict sequence-level constraint satisfaction? Could the authors also clarify the equivalence between logit differences and log-probability differences after RMS normalization and soft-min aggregation?

  3. Could the authors provide compute-matched comparisons with standard OPD and RLVR, accounting for the additional teacher forward passes? An offline distillation baseline would also help clarify the necessity of on-policy training.

  4. In Table 2 (Qwen3-4B → Qwen3-0.6B), the average ASR improvement of Standard OPD is reported as +1.11, while subtracting the displayed averages gives 1.12.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.