acceptodds
Under review as a conference paper at ICLR 2027

Reinforcement Learning toward Target Behavior

Abstract

Desired model behavior defines a destination for post-training. Reaching this destination through fixed rewards requires anticipating how output scores will shape model behavior. We instead make the joint satisfaction of behavioral requirements the explicit training objective, introducing Reinforcement Learning toward Target Behavior (RLTB). Specifically, RLTB formalizes these requirements as a joint target region in behavior space and minimizes the discrepancy between the policy's current behavior and this region. Each behavior's deviation from its target determines the direction and strength of its corrective feedback: the feedback weakens as the target is approached, can reverse after overshoot, and vanishes upon satisfaction. This feedback scores sampled outputs for training, yielding target-dependent output credit. We implement RLTB through Target-Behavior Group-Relative Policy Optimization (TB-GRPO), which converts this credit into group-relative advantages for GRPO's policy update. To prevent outputs from influencing the feedback used to score them, we estimate each prompt group's feedback from other independent groups using leave-one-prompt-group-out cross-fitting. In mathematical problem solving and factual question answering, models trained with TB-GRPO jointly approach heterogeneous behavioral targets, achieving the smallest joint target gaps among the evaluated baselines. Across the evaluated requests and base policies, the trained models approach their behavioral destinations without case-specific hyperparameter retuning. Together, these results position RLTB as a principled foundation for target-directed post-training.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Target Behavior, Post-Training

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_RcZ2OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes RLTB, which expresses multiple behavioral requirements as a joint interval target region in behavior space. The loss is the scaled squared distance from the behavior statistics to this region, so that a statistic inside its interval produces no gradient while a statistic outside it receives a correction proportional to the signed distance. The Jacobian of Ψ then maps this policy-level correction into weights on the primitive measurements, yielding per-output credit. The method is instantiated as TB-GRPO on top of GRPO, with the within-group standard-deviation normalization removed so that target-dependent magnitudes are preserved, and with leave-one-prompt-group-out (LOPO) cross-fitting used to estimate the weights so that outputs do not influence the feedback that scores them. Experiments report joint target gaps on GSM8K and TriviaQA.

Critical strengths
  1. Disclosure and reproducibility are unusually thorough: complete specification tables (Tables 10 and 12), sensitivity of results to the measurement definitions via two independent checkers (D.1.5), per-baseline search budgets (Table 15, which also shows that TB-GRPO performed no configuration search while RC and CGM-relax did), per-seed readouts (Table 18), and a code revision. A number of unfavorable results are also reported honestly in the appendix.
  2. The design of the paired and interventional measurements is valuable (abstention agreement, and retreat under a challenge that supplies no new evidence), and it incidentally exposes the instability of dual-ascent-based methods under such specifications (C-GRPO, exact variant, on TriviaQA: Mean 10.37±4.74 / Max 57.38±35.96).
Critical weaknesses
  1. Limited novelty, no clear separation between the increment and existing components, and framing that overstates the contribution. Equation (3) is a dead-zone exterior penalty (Π_T is simply coordinate-wise clipping); Equations (4)–(5) are the single-step contextual-bandit special case of the general-utility RL policy gradient (Zhang et al., 2020; Zahavy et al., 2021), obtained by a chain rule; LOPO is an adaptation of cross-fitting (Chernozhukov et al., 2018) and leave-one-cluster-out estimation (Chiang et al., 2022) to this setting; and Equation (7) is RLOO (Ahmadian et al., 2024). Section 5 itself concedes that, for directly averaged features with matched coordinate scales, the point-target discrepancy coincides with CGM-relax's calibration penalty up to a constant factor. All of these sources are cited, so the issue is not attribution: the paper never separates its actual increment (unified credit for interval targets on nonlinear coordinates; the cluster granularity for paired and interventional families) from the pre-existing components. The notation and derivations also occupy space out of proportion to their technical content, and include a considerable amount of material that is unrelated to the method or could be stated far more concisely; relative to the actual increment, the abstract's positioning of RLTB as a "principled foundation for target-directed post-training" is too strong.
  2. The central claims lack corresponding empirical support. (a) LOPO, presented as a selling point: Table 7 reports a training comparison at the actual budget (B = 128, G = 8, point target t = .10 — precisely the affine-weight case in which, by the paper's own Theorem 1, finite-sample unbiasedness applies), giving full-batch plug-in 11.8%, rollout-LOO 10.7%, and LOPO 11.0 / 11.4 / 10.6%, with the paper itself stating that "models trained with all three estimators approach the requested rate" and only one run per alternative. (b) Feedback magnitude and scales: in Table 25, Full and Sign-only give 1.09±0.27 and 1.39±0.17 over three seeds, but on seed 0 Sign-only is better on both Mean and Max (1.22 / 3.62 versus 1.35 / 4.16) and the paper states that "seed 0 favors Sign-only"; the two scale perturbations give 1.17 and 1.38, the same order as Full's own seed-to-seed spread, while the eight scales (values drawn only from {0.05, 0.10} plus 50 for length) come with no selection rule beyond the remark that s_k "controls corrective strength during training," so the claim of no case-specific hyperparameter retuning is unsupported. (c) Baselines: Section 5 explicitly discusses PID-Lagrangian methods (Stooke et al., 2020) and notes that dual ascent accumulates violations as integral feedback, yet never runs it as a baseline, even though RLTB's correction is essentially dead-zone proportional control; and on the single-target sweep CGM-relax's error (Table 23a, 0.8 / 1.9 pp) is no worse than TB-GRPO's (1.1±.6 / 2.3±1.0 pp). (d) Reference convention and scale: Table 21 shows a configured training floor of 75.07% for conditional correctness against a test-split value of 65.34%, a 9.73-pp shortfall, whereas the 0.07-pp gap in main-text Table 1 comes from re-applying the same multiplier to the test-split base reference; the joint specification is validated only at the 1B–1.5B scale, with the only larger model (Qwen2.5-7B) confined to the single-target abstention sweep, and the tasks are limited to rule-verifiable short-answer settings while Theorem 1 presupposes unbiased measurement (E[H_i] = m_θ), which no longer holds once measurements come from an LLM judge or a reward model.
Questions for authors

Please see the Weaknesses above.

6 comments
AuthorsOpenReview ↗

Response to Reviewer RcZ2 (1/5): contribution beyond the existing components

Thank you for recognizing the value of the paired/interventional measurements and the thorough disclosure and reproducibility materials.

Your concerns center on the contribution beyond existing components and the empirical support for the framework's design. We address them in five parts: Part 1 explains expressive joint specifications and unified feedback; Part 2 connects LOPO to optimization through the revised analysis and matched training; Part 3 addresses feedback magnitude, scale selection, and configuration reuse; Part 4 presents the completed PI comparison and the CGM special case; and Part 5 addresses the reference convention, larger models and open-ended generation, and learned evaluators.

Part 1. Separating the contribution from existing components (W1)

We respectfully disagree with the overall assessment of limited novelty. RLTB makes joint behavioral specifications an explicit interface to training, translating desired values and admissible ranges into corrective feedback and output credit. This unified construction enables joint control across heterogeneous measurements and supports alternative feedback controllers.

1.1. Expressive specifications through unified feedback. RLTB organizes behavioral requirements along three composable dimensions: measurement families (single-output, paired, or interventional), statistics (marginal, joint, or conditional), and targets (points, intervals, or upper/lower bounds). A unified target-to-credit construction translates their current discrepancies into feedback for the participating outputs or rollouts, enabling heterogeneous requirements to be optimized jointly within the same training procedure. This combination of expressive specification and unified feedback is a central contribution.

The dead-zone penalty, general-utility policy gradient, cross-fitting, and RLOO supply established mathematical foundations. The direct-mean, matched-scale point-target correspondence with CGM-relax is a recovered special case. The framework contribution lies in the reusable construction connecting specifications, measurements, feedback, and estimation dependencies, and the control capabilities it enables. Stochastic computation graphs provide a related example of a unifying formalism built on established gradient tools (Schulman et al., 2015).

1.2. Equivalent constraints need not give equivalent feedback. For a conditional rate r=n/dr=n/d with d>0d>0, r≤ur\le u and n−ud≤0n-ud\le0 define the same feasible set. Directly penalizing their respective violations gives

L_{\mathrm{cost}}=\frac{[n-ud]_+^2}{2s^2},$$ where $[x]_+=\max(x,0)$. A policy-dependent denominator generally makes their gradients differ. RLTB specifies discrepancy directly in behavioral coordinates and derives the corresponding output credit. Appendix A.6 also shows that some paired/interventional target sets cannot be represented by finitely many fixed single-output expected-cost constraints. General constraints can accommodate richer measurements; RLTB provides a common construction that makes these specifications operational together. **1.3. Joint control enabled by the specification.** On Qwen2.5-1.5B-Instruct and TriviaQA, we hold the training procedure, feedback rule, and budget fixed, and remove the paired-output and interventional requirements while retaining the same four single-output requirements: | Target behavior specification | Shared-four Mean gap (pp) | Retreat (%) | |---|---:|---:| | Four single-output requirements | $0.54 \pm 0.11$ | 38.36–40.35 | | Complete six-requirement specification | $0.46 \pm 0.15$ | **1.55–1.77** | *Mean gaps are mean $\pm$ SD over three training runs; retreat gives their range.* Models trained under both specifications closely control the shared requirements. Adding the paired and interventional requirements enables a substantial additional behavioral benefit: retreat falls from 38–40% to below 1.8%. Part 4 complements this specification ablation by showing that the same credit construction supports effective joint control after substituting a PI feedback controller. **1.4. Positioning and presentation.** Here, “principled” refers to objective-derived feedback and estimator guarantees under explicit assumptions: **a principled framework for deriving training feedback from joint behavioral specifications**. Its objects are measurable policy statistics; parsers and GRPO instantiate measurement and optimization for LLMs. We will clarify the contribution boundary and its relationship to the established components in the manuscript.
AuthorsOpenReview ↗

Response to Reviewer RcZ2 (2/5): LOPO's effect on optimization and target attainment

Part 2. LOPO: corresponding empirical support for its optimization role (W2a)

LOPO has a concrete optimization role: it removes receiving-cluster coupling that can displace the mean-update equilibrium from the requested target. At the reported training budget, the model trained with LOPO also has smaller joint Mean/Max gaps in our matched comparison. These results supply the corresponding empirical support requested for this design.

2.1. Why the original comparison shows close attainment. Theorem 1's affine-weight unbiasedness guarantee applies to the canonical LOPO estimator. Affine weights do not automatically make full-batch plug-in unbiased: the receiving cluster and its estimated feedback remain coupled.

The original B=128B=128, G=8G=8, 10% abstention comparison concerns a directly averaged statistic at a large budget. Close attainment there is consistent with a small coupling effect. The revised manuscript now connects this effect to optimization and separates controlled mean-update equilibria from actual training endpoints.

2.2. Improved joint attainment at the actual training budget. On Qwen2.5-1.5B-Instruct and TriviaQA, we hold the complete six-requirement specification, initialization, prompt stream, cluster weighting, and 1,500-update budget fixed (B=128B=128, G=8G=8). Only inclusion of the receiving cluster in the feedback estimate changes. Here BB counts prompts; each neutral–challenge pair forms one dependency cluster.

Estimator used for trainingHeld-out Mean gap (pp)Held-out Max gap (pp)
LOPO0.371.58
Full-batch plug-in0.482.12

The LOPO-trained model's endpoint has 0.11/0.54 pp smaller Mean/Max gaps. This matched training ablation directly addresses optimization at the reported budget (revised Appendix D.4, Table 25).

2.3. The systematic target displacement that LOPO removes. Revised Appendix B.2, Figure 5, isolates the mechanism using a 20% point target, G=8G=8, and BB independent measurement clusters. Holding the task and sampling design fixed, the zero of the expected canonical update has the following offsets from the target:

Controlled measurementBBPlug-in displacement (pp)LOPO displacement (pp)
Direct mean80.9680.007
Direct mean1280.0610.008
Conditional rate; event-cluster probability p=1/8p=1/889.9170.027
Conditional rate; event-cluster probability p=1/8p=1/81280.6500.007

Entries are estimated absolute offsets of the mean-update equilibrium from the target, comparing both estimators at the same BB. The conditional experiment holds the population denominator fixed with respect to the policy. Figure 5 includes all four tested budgets: 8, 32, 128, and 512.

The direct-mean effect is small at a large budget, consistent with the limited separation in the original comparison. For the sparse conditional measurement, the same coupling causes a much larger displacement, which LOPO removes to the displayed precision. These are mean-update equilibria, distinct from finite Adam training endpoints.

Together, the results identify a systematic optimization error removed by LOPO and show improved joint target attainment in matched training. Revised Appendix A.4.2 connects bias and variance to expected loss decrease for canonical updates under stated smoothness, moment, and step-size conditions.

AuthorsOpenReview ↗

Response to Reviewer RcZ2 (3/5): feedback magnitude, scale selection, and no-retuning scope

Part 3. Feedback magnitude, scales, and case-specific retuning (W2b)

Adaptive feedback translates current target discrepancies into coordinated corrections. Retaining its magnitudes improves joint control on TriviaQA; deliberate changes to relative feedback allocation preserve small joint gaps on GSM8K. Across target requests, the same feedback rule supports reuse of a fixed configuration without per-request hyperparameter search.

3.1. What magnitude and scale do. The declared discrepancy gives behavior-level feedback and primitive-measurement weights

\mathbf w=J_\Psi^\top\mathbf d.$$ The signed gap determines the direction and magnitude of correction. The scale $s_k$ accounts for units and relative corrective strength; the Jacobian maps these corrections into output credit. Positive scales preserve the target region and zero-loss set while affecting training trajectories, finite-budget attainment, and trade-offs if targets cannot all be met. They are explicit design choices, separate from evaluation tolerances. **3.2. Empirical effect of preserving magnitude.** The original GSM8K comparison has the run-to-run variation you identify, including the seed-0 reversal. The following completed comparison is on **TriviaQA**, with Qwen2.5-1.5B-Instruct, the same six requirements, and the same 1,500-update budget: | Training feedback | Mean gap (pp) | Max gap (pp) | Retreat (%) | |---|---:|---:|---:| | Full | $0.31 \pm 0.10$ | $1.22 \pm 0.49$ | **1.55–1.77** | | Sign-only | $0.96 \pm 0.04$ | $5.12 \pm 0.33$ | 6.67–7.26 | *Gaps are mean $\pm$ SD over three matched training runs; retreat gives their range. The retreat upper bound is 1.92%.* All Full models meet the retreat bound; all Sign-only models exceed it. Mean and Max gaps improve in every matched pair. This supports the value of magnitude information for **joint target control**, particularly the interventional requirement. Sign-only takes the sign of the primitive coefficient, removing magnitudes contributed by the discrepancy, scales, and measurement Jacobian together; it tests this complete construction. **3.3. Deliberately changing relative feedback allocation.** On **GSM8K**, we multiply the scales for abstention, format, omission, and length by **0.8**, and the remaining scales by **1.25**, then reverse the assignment. At fixed measurements, the corresponding feedback coefficients scale by **1.5625 and 0.64**. These factors were chosen to cross feedback-allocation ranking thresholds identified on the reference trajectory, testing whether joint control remains effective after changing which requirements receive more feedback. Targets, evaluation criteria, and other training settings stay fixed. The trained runs confirm that allocation changes: omission's absolute coefficient share moves from **22.8% to 31.1%/14.8%**. Yet Mean gaps remain **1.38/1.17 pp**, compared with **1.35 pp** under the original scales. These are feedback-coefficient shares; the endpoint comparison supports effective joint control under the tested reallocations. **3.4. Why adaptive feedback supports no case-specific retuning.** For a point target, $d_k=(t_k-\mu_k)/s_k^2$: changing $t_k$ directly changes the feedback's direction, magnitude, and zero, while scales and optimization settings can stay fixed. Target-dependent weights are computed from current measurements during training, avoiding an additional search to map each requested rate to a fixed reward coefficient. In the **5%, 10%, 15%, 30%, and 50% abstention sweeps**, models trained with TB-GRPO track the requested rates with mean errors of **0.8–1.1 pp** on **Qwen2.5-1.5B-Instruct, Llama-3.2-1B-Instruct, and Qwen2.5-7B-Instruct**. **Scales, optimization, and sampling settings remain fixed across the five requests, with no per-request configuration search.** These results demonstrate reuse of an initially specified configuration as targets change. Initial settings remain explicit: GSM8K uses coarse scales of **0.05/0.10 for rates and 50 tokens for length**. Measurement precision informs the design: TriviaQA's format scale **0.08** follows estimated SD **0.0245** at $B=128$ and the check $\mathrm{sd}(\widehat m)\le s/3$. This check constrains usable scales without uniquely selecting them. We will clarify the initial selection rationale and demonstrated reuse scope in the manuscript.
Reviewer_1YkAOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Requirements such as an abstention rate, correctness among answered questions, instruction compliance, agreement across repeated samples, or withdrawal after an unsupported challenge are written as behavior coordinates μ=Ψ(E[ϕ])\mu = \Psi(\mathbb{E}[\phi]) with point, bound or interval targets. RLTB minimizes the scaled squared distance from μ\mu to the target box; its gradient assigns each output the credit w⊤ϕw^\top\phi with w=JΨ⊤dw = J_\Psi^\top d, which vanishes inside the target and changes sign after overshoot. TB-GRPO estimates ww for each prompt group from the other groups (LOPO), centers credit with a leave-one-out baseline and uses the clipped GRPO update with no KL term. The theory gives unbiasedness when ww is affine and a bias bound of order (B−1)−1/2(B-1)^{-1/2} otherwise. Experiments on Qwen2.5-1.5B use joint specifications on GSM8K (seven rate requirements plus length) and TriviaQA (six), with abstention sweeps on two more base models and ablations of feedback direction and magnitude.

Critical strengths
  1. The formulation covers requirements that fixed reward weights or expected-cost constraints cannot express directly: conditional ratios through the Jacobian, agreement between sampled responses, and retreat under an added challenge. Dependence between the estimated weights and the outputs they score is handled explicitly. The target-switch experiment shows that adaptive signs matter: frozen coefficients leave 79.5 and 94.9 pp of error, while full feedback reaches 0.5 and 0.1.

  2. The TriviaQA result is the strongest evidence. Across three runs the mean gap over six requirements is 0.3 pp, abstention lands at 18.7% against a 20% target, and retreat under challenge falls from 15.8% to 1.6%. Reward combination drives abstention to 0.1%, and C-GRPO collapses to near-total abstention in two of three runs. The abstention sweep tracks five requests on three base models with 0.8–1.1 pp error and no retuning.

Critical weaknesses
  1. The comparison with the closest baseline isolates little. The headline Mean/Max gaps include requirements that some baselines never received, so they partly reward the larger interface. On the coordinates shared with CGM-relax the margin is smaller (1.52 vs. 2.66 pp on GSM8K, 0.67 vs. 1.16 on TriviaQA, Table 20), and on single-target abstention CGM-relax is as good (0.8 vs. 1.1 pp, Table 23). The two methods also run under different conditions: CGM-relax keeps a KL penalty and full-batch moments, and none of its λ values met the selection rule on joint GSM8K, so a fallback was used, while TB-GRPO runs with β = 0. Adding hinge terms for bounds to CGM-relax would be a small change, and the joint reward-combination baseline uses one signed unit per requirement with no search. Conceptually the objective is the classical quadratic exterior penalty for interval constraints, and the paper should discuss it in that light.

  2. Some targets are easier to meet on paper than in substance. The GSM8K correctness floor is set relative to each split's base model: TB-GRPO reaches 65.34% against a test floor of 65.29%, but falls 9.73 pp short of the 75.07% training floor (Table 21). Base correctness itself differs between train and test (62.56% vs. 54.41%), which shows that rate targets depend on the prompt distribution, a limitation the paper does not discuss. GSM8K coherence only checks that the final answer appears among intermediate numbers, and instruction compliance drops from 98.6% to 74.5% under the alternative checker. The main experiments use a 1.5B model.

Questions for authors
  1. If CGM-relax is given hinge terms for the bounds, the same KL setting (β = 0 for both, or KL for both) and the same grouped estimation, does TB-GRPO keep its advantage on the shared coordinates?

  2. How do achieved rates hold up under a shift in prompt distribution, for example training on GSM8K and evaluating on harder problems, with targets held at fixed numerical values?

  3. For the ratio and interval targets, does attainment change noticeably with the number of prompt groups B, as the bias bound in Theorem 1 suggests it should?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 1YkA (1/2): expressive joint control and matched CGM comparisons

Thank you for recognizing the value and contributions of our work and for your constructive feedback. We address your concerns and questions in two parts, covering the framework's contribution and baseline comparisons, the validity and generalization of our behavioral results, and the effects of sampling and estimation on target attainment.

Part 1. Expressive joint control and matched comparisons with CGM-relax (W1 / Q1)

The model trained with TB-GRPO retains smaller Mean/Max gaps on the shared coordinates when CGM-relax is given hinge terms for the bounds, the same KL setting, and the same grouped estimation. Declaring paired and interventional requirements additionally brings retreat under control while retaining small gaps on the shared requirements.

1.1. The CGM-relax special case and KL settings. CGM-relax's calibration penalty is recovered as RLTB's single-output, directly averaged, point-target special case: L=(m−t)2/(2s2)L=(m-t)^2/(2s^2), up to an overall factor when scales are matched. Its comparable single-target abstention performance is therefore expected.

For joint GSM8K training, we turn off KL in CGM-relax and add it to TB-GRPO, keeping the rest of each method's training configuration unchanged. Together with the original runs, this gives both-KL-off and both-KL-on comparisons. The table reports Mean/Max gaps in pp over the five shared requirements:

KL settingCGM-relaxTB-GRPO
Both disabled2.70 / 4.571.89 / 4.16
Both enabled2.49 / 4.841.98 / 2.80

Models trained with TB-GRPO retain smaller Mean/Max gaps in both comparisons. For the comparison with KL enabled, CGM-relax retains its KL coefficient λ=0.001\lambda=0.001, which gave the lowest feature-residual norm in the four-point, 500-update search. None reduced this norm by at least 90% relative to the base, so 0.001 was retained as the fallback (Appendix C.4, Table 13). TB-GRPO's coefficient was fixed at 0.069 before training using the advantage-scale ratio to account for the different feedback scales; measured median KL is 0.028 for CGM and 0.032 for TB-GRPO.

1.2. Hinge terms, grouped estimation, and expressive joint control. We next implement all three changes requested in Q1 on TriviaQA: squared-hinge bounds, zero KL, and complete-cluster LOPO in place of full-batch moments. CGM-relax and TB-GRPO are trained on the same four single-output requirements: abstention, conditional correctness, format violations, and instruction compliance. To test the value of joint specification, we also repeat TB-GRPO training with paired and interventional requirements added. All rows use Qwen2.5-1.5B-Instruct; the remaining training settings stay fixed.

Training method and requirementsShared-four Mean/Max gap (pp)Retreat (%)
Modified CGM-relax, four requirements0.74 / 1.7042.20
TB-GRPO, the same four requirements0.46 / 1.1739.42
TB-GRPO, the same four + paired/interventional0.56 / 1.581.55

Mean/Max always evaluates the same four requirements. Retreat is a training target only in the third row, with an upper bound of 1.92%.

The first two rows answer the shared-coordinate optimization question: the requested changes improve CGM-relax, whose conditional correctness now meets its bound (31.48% against 29.16%), while the model trained with TB-GRPO retains smaller Mean and Max gaps. CGM retains its unit feature weights and expected-cost representation of the conditional bound.

The last two rows isolate what additional behavioral requirements achieve: the model trained with the added requirements meets the retreat bound while retaining small gaps on the shared four. This demonstrates the practical value of jointly controlling behavioral requirements across different measurement families (single-output, paired, interventional), statistics (marginal, joint, conditional), and target forms (points, intervals, bounds) through unified output credit. The quadratic exterior penalty is the loss chosen here; the credit construction also applies to other differentiable losses under appropriate regularity conditions. We will clarify this relationship in the manuscript.

1.3. Hyperparameter search and adaptive feedback. Reward combination requires translating a joint target specification into reward weights, potentially repeating this calibration when the targets change. RLTB computes adaptive feedback directly from target discrepancies. In the five-request abstention sweep, RC searched 15 candidate weight settings while TB-GRPO reused one configuration, demonstrating configuration reuse across requests.

AuthorsOpenReview ↗

Response to Reviewer 1YkA (2/2): behavioral improvement, generalization, and sampling budget

Part 2. Behavioral improvement, generalization, and sampling budget (W2 / Q2 / Q3)

Instruction-compliance gains persist under independent evaluation, and joint control remains effective on larger models and in open-ended generation. Under prompt-distribution shift, the prescribed relative correctness gain is retained. For ratio and interval targets, increasing B markedly improves retreat attainment.

2.1. Alternative evaluation and larger models. Under the alternative checker, instruction compliance rises from 59.4% for the base to 74.5% after TB-GRPO training, a 15.1 pp gain. The trained models score 98.6% and 74.5% under different checkers (Appendix D.1.5).

A policy-blinded semantic audit on 100 matched GSM8K questions also finds compliance increasing from 78% to 96%. GPT-6 Astra and one author independently rechecked the same 60 randomly selected outputs and both agreed on all instruction-compliance labels. Coherence checks whether the final answer appears among intermediate numbers; it measures answer–work numerical consistency, without assessing the validity of every reasoning step.

On GSM8K with Qwen2.5-7B-Instruct, joint training with all eight requirements reduces Mean/Max gaps from 6.48/12.63 to 0.77/1.76 pp.

For open-ended story generation, we adopt the task and eight-profession gender-parity targets from Smith et al., 2026. Even with a gender-balance prompt, Gemma-2-9B-IT has mean absolute occupational imbalance of 0.728; TB-GRPO training with LoRA reduces it to 0.050, a 93% reduction toward the same literature-defined targets.

2.2. Prompt-distribution shift with targets held at fixed numerical values. On harder problems, the other fixed rate targets retain small gaps, while the frozen correctness floors are unmet. We evaluate the trained Qwen2.5-1.5B-Instruct models without further training on 1,319 paired GSM-Plus adding-operation problems, retaining the supplementary instructions. Correctness below is conditional on answering:

DistributionBase correctness (%)Trained correctness (%)
GSM8K test54.4165.34
GSM-Plus adding31.3439.18

Against the fixed test/training floors of 65.29%/75.07%, GSM-Plus shortfalls are 26.11/35.89 pp. The other six rate targets retain their fixed values, with mean gaps at most 3.19 pp per coordinate; abstention changes only from 12.26% to 12.80%. On GSM8K test, the mean per-model shortfall against the frozen 75.07% training floor is 9.73 pp (Appendix D.1.4, Table 19).

The relative requirement asks whether a fixed 20% improvement over the base transfers after a fixed training budget. Base correctness differs between training (62.56%) and test (54.41%), so the same rule gives different numerical floors. On harder problems, the trained models retain approximately 25% improvement, exceeding the requested 20%, whereas a fixed 65.29% floor would require 108% improvement. This supports transfer of the prescribed gain under changing task difficulty.

2.3. Attainment of ratio and interval targets as B changes. Yes, with different effects across requirements: retreat improves markedly as B increases, while correctness meets its floor at every tested budget. With Qwen2.5-1.5B-Instruct, the complete TriviaQA specification, LOPO, G=8G=8, and 1,500 updates:

Prompt budget BBCorrectness ≥29.16%Retreat ≤1.92%
830.92%34.30%
3230.02%8.13%
12830.78–31.26%1.55–1.77%

At equal generation of 96,000 completions, retreat is 19.96% at B=32 versus 34.30% at B=8. For retreat, small B leaves few relevant groups, so rate estimates and feedback can have high variance. For interval targets, a batch estimate can cross a bound even when the model's true rate is within the target interval, triggering extra correction.

Bias deserves separate control because it changes the mean update and can move its equilibrium away from the target. Revised Appendix B.2, Figure 5, isolates this effect: for a ratio point target with a policy-independent denominator, plug-in displacement is 9.92/0.65 pp at B=8/128, while estimated LOPO equilibria stay within 0.03 pp of the target. It also examines extra hinge corrections inside a conditional upper bound, a one-sided interval. B counts prompts in training and independent clusters in Figure 5. Appendix A.4.2 supplements Theorem 1: under its stated conditions, squared bias enters the expected-loss bound at order γ\gamma and variance at order γ2\gamma^2, where γ\gamma is the step size. Removing coupling bias therefore addresses a systematic error that a smaller learning rate alone need not remove, while adequate sampling support keeps feedback stable.

Dear Reviewer 1YkA,

Thank you again for your detailed review. Your suggestions helped us strengthen the comparison with CGM-relax and examine how well the trained models retain the requested behaviors on harder problems. As the discussion period draws to a close, we would be grateful to hear whether the matched comparison and the generalization results address your questions. If there is anything you would like us to clarify, we would welcome the opportunity to do so.

Reviewer_N8B6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Reinforcement Learning Toward Target Behavior (RLTB), a framework for aligning an RL-trained model with desired behavioral statistics—such as abstention, formatting, agreement between paired outputs, or intervention responses—rather than optimizing a single scalar reward. It introduces a projection-based loss over target intervals and TB-GRPO, which uses leave-one-prompt-group-out credit assignment to reduce dependence bias in group-based sampling. The formulation is flexible and the reported target-matching results are promising.

Critical strengths
  1. Clear, useful problem formulation. Treating alignment as matching a vector of measurable behavioral targets is more expressive than a monolithic reward and accommodates bounds, conditional rates, paired generations, and interventions.

  2. The LOPO idea is technically well motivated: it directly addresses leakage from estimating a behavior statistic and assigning credit on the same prompt group.

  3. The paper is transparent about the nonlinear case: exact unbiasedness holds for affine target statistics, while nonlinear statistics retain a characterization of estimation bias.

  4. Experiments span several kinds of requirements and show materially smaller target gaps than the adapted baselines.

Critical weaknesses
  1. The main theorem analyzes a canonical sequence-level score-function estimator, but the reported TB-GRPO implementation also uses token-level normalization, masking/sampler correction, clipped updates, and altered reward scaling. It is therefore unclear to what extent the practical algorithm inherits the claimed optimization properties.

  2. Baseline comparisons are not fully apples-to-apples. Several baselines cannot encode paired-output or interventional constraints, yet are assessed against the complete target specification. This makes it difficult to separate optimization improvements from RLTB’s more expressive target language.

  3. The target suites are largely parser- and heuristic-defined. The paper provides limited evidence that satisfying these proxies improves broader answer quality or avoids undesirable gaming.

  4. The largest joint-training experiments use a 1.5B model and require many sampled completions per update. Scalability and behavior at stronger-model scale remain underexplored.

  5. Some targets are defined relative to the base model and recomputed across splits. This is reasonable for normalization, but makes “meeting the same target” less directly comparable between training and held-out evaluation.

Questions for authors
  1. Can you quantify, through ablations, how much each practical deviation from the canonical LOPO estimator affects target-loss optimization and final target satisfaction? Is there theory that covers the deployed token-level/clipped implementation?

  2. How robust are the results to fixed, split-independent target thresholds and to external quality measures—e.g., answer accuracy, human preference, or adversarial checks for gaming the behavioral parsers?

5 comments
AuthorsOpenReview ↗

Response to Reviewer N8B6 (1/4): practical deviations and final target satisfaction

Thank you for recognizing the expressive behavioral-target formulation, the motivation for LOPO, and the promising target-matching results. We address practical deviations (Part 1), optimization improvements and expressive joint control (Part 2), broader answer quality and gaming (Part 3), stronger-model scale and cost (Part 4), and fixed, split-independent thresholds (Part 5).

Part 1. Practical deviations: target-loss optimization and final target satisfaction (W1 / Q1)

The practical implementation remains closely connected to the canonical estimator: analytical identities explain clipping and token normalization, sampled gradient directions are closely aligned, and the correction/mask training ablation changes Mean/Max target gaps by 0.07/0.40 pp.

1.1. Clipped updates and token-level normalization. Each rollout batch receives one optimizer update. At the collection policy, the policy ratio is one in exact arithmetic, so clipping is inactive at gradient evaluation. With canonical advantages, zero KL penalty, and no sampler correction, Appendix A.5, Eq. (44), establishes

= \frac{BG}{N_{\mathrm{tok}}}\widehat{\mathbf{g}}.$$ Token normalization therefore preserves the sampled gradient direction within each batch. This batchwise identity does not by itself establish unbiasedness after random normalization. Appendix C.1, Table 6, maps the canonical constructions to the implementation. **1.2. Altered reward scaling and masking/sampler correction.** We hold the model state and sampled outputs fixed, then compare gradients with successive implementation choices against the canonical sequence-level gradient. Reward scaling changes the relative contributions of measurement families. The full implementation additionally includes token normalization, sampler correction, and its mask. | Qwen2.5-1.5B-Instruct policy state | Implementation reward scaling only | Full implementation | |---|---:|---:| | GSM8K, base | 1.0000 | 0.9716 | | GSM8K, trained | 1.0000 | 0.9647 | | TriviaQA, base | 1.0000 | 0.9657 | | TriviaQA, trained | 0.9997 | 0.9874 | Entries are cosine similarities to the canonical gradient, averaged over four rollout batches per state; **1 means the same direction**. Token normalization alone preserves direction, as established above. At $G=8$, reward scaling multiplies the paired-to-single relative contribution by $\frac{3}{7}$, changing corrective strength while preserving the target region. Even with all listed implementation choices, the sampled directions remain closely aligned (cosines **0.965–0.987**). **1.3. Final target satisfaction under the same target behavior specification.** We train Qwen2.5-1.5B-Instruct on the complete six-requirement TriviaQA specification, holding initialization, prompt stream, LOPO, and the 1,500-update budget ($B=128$, $G=8$) fixed. The ablation disables only sampler correction and its associated mask. | TB-GRPO training setting | Mean target gap (pp) | Max target gap (pp) | |---|---:|---:| | Correction/mask enabled | 0.37 | 1.58 | | Correction/mask disabled | 0.44 | 1.98 | The differences are **0.07/0.40 pp**. Thus this practical deviation has a limited effect on final target satisfaction in the matched comparison, complementing the gradient-direction checks for target-loss optimization. The theorem's estimation guarantees apply to the canonical estimator. **Appendix A.4.2** additionally relates bias and variance to expected loss decrease for canonical updates under explicit smoothness and moment assumptions; the checks above quantify the connection to the practical implementation.
AuthorsOpenReview ↗

Response to Reviewer N8B6 (2/4): optimization improvements and expressive joint control

Part 2. Optimization improvements and the more expressive target language (W2)

A central contribution of RLTB is the joint control of heterogeneous behavioral requirements—including single-output, paired-output, and interventional targets—through a unified construction that translates target discrepancies into per-output training credit. We address your concern about separating optimization improvements from this expressive capability with three comparisons.

2.1. Optimization on requirements received by both methods. Appendix D.1.3, Table 18, evaluates the same five GSM8K or three TriviaQA equality requirements supplied to both methods:

Training methodGSM8K Mean/Max gap (pp)TriviaQA Mean/Max gap (pp)
CGM-relax2.66/4.911.16/2.12
TB-GRPO1.52/2.880.67/1.31

Models trained with TB-GRPO retain smaller gaps in the submitted configurations when evaluation is restricted to these shared requirements. The advantage therefore cannot be explained solely by evaluating baselines against the complete target specification, including requirements they never received.

2.2. Practical value of paired-output and interventional constraints. We keep the TB-GRPO training procedure fixed on TriviaQA and remove these two requirements. The four remaining single-output requirements are identical.

Target behavior specificationShared-four Mean gap (pp)Retreat (%)
Complete specification0.46±0.150.46 \pm 0.151.55–1.77
Four single-output requirements0.54±0.110.54 \pm 0.1138.36–40.35

Mean gaps are mean ±\pm SD over three training runs; retreat gives their range.

Both settings control the shared requirements closely. Explicitly including the paired-output and interventional requirements also enables retreat control, which does not follow automatically from satisfying the single-output targets. This is a concrete benefit of the more expressive target language for joint behavioral control.

2.3. Unified credit construction with alternative feedback. Within TB-GRPO, we replace the rule mapping each requirement's target deviation to its corrective feedback coefficient with a proportional–integral (PI) rule. PI uses both the current deviation and accumulated past deviations. This substitution occurs before the shared measurement-to-credit construction: the measurement Jacobian, LOPO credit estimation, GRPO trainer, complete target specification, and training budget are retained.

Framework and feedbackTriviaQA Mean/Max gap (pp)GSM8K Mean/Max gap (pp)
TB-GRPO (default feedback)0.31/1.221.093/2.885
TB-GRPO + PI feedback0.16/0.870.760/2.791

Full-specification rate gaps for Qwen2.5-1.5B-Instruct; averages over three training runs except GSM8K PI, which reports one run. GSM8K reuses Ti=200T_i=200 from TriviaQA without further tuning.

Small gaps with both feedback rules show that the unified construction supports effective joint control beyond the default controller. Together, these comparisons establish the practical value of expressivity, quantify shared-target performance, and demonstrate modular feedback. Within each submitted comparison, TB-GRPO uses fixed settings without additional configuration search; CGM-relax used a four-value KL search.

AuthorsOpenReview ↗

Response to Reviewer N8B6 (3/4): broader answer quality and checks for gaming

Part 3. Broader answer quality, external quality measures, and undesirable gaming (W3 / Q2)

The gains extend beyond parser-defined target attainment: semantic assessment and overall answer accuracy support improvements in answer quality, while targeted checks probe undesirable gaming and transfer beyond the training wording.

Your concern about parser- and heuristic-defined targets motivates evaluating the responses with complementary quality measures. Statistical behavior control requires operational measurements: GDC (Khalifa et al., 2021) uses features and classifiers, and GRPO (Shao et al., 2024) uses reward-model scores. RLTB provides a unified target-to-credit construction that can use different observable or evaluator-based measurements under its sampling assumptions. For the reported models, we performed the following checks.

3.1. Semantic assessment beyond the behavioral parser. A policy-blinded audit reads responses to 100 matched GSM8K questions and judges whether they semantically follow the requested instructions. Compliance increases from 78% for the base model to 96% for models trained with TB-GRPO, a paired gain of 18 pp. GPT-6 Astra and one author independently re-evaluated 60 randomly selected outputs from the audit, blinded to policy identities and original labels, and both agreed on all 60 instruction-compliance labels. The improvement survives semantic assessment, with no increase in the audited checker-pass/semantic-fail rate.

3.2. Answer accuracy under joint behavioral control. We additionally report overall answer accuracy on held-out responses, counting abstentions and unparseable outputs as incorrect. This readout does not use split-specific target thresholds.

TaskBase accuracy (%)Accuracy after TB-GRPO training (%)
GSM8K53.1157.32
TriviaQA18.3225.21

Training jointly enforces a lower bound on conditional correctness and the other behavioral requirements; it does not maximize answer accuracy as a standalone objective. These gains show improved task quality under the declared joint specification. They are not estimates of the highest accuracy attainable by the framework.

3.3. Checks for dependence on the training challenge wording. We evaluated four challenge phrasings absent from training. Three met the prespecified effectiveness criterion of increasing base retreat by at least 5 pp relative to its neutral-resampling rate.

ModelsAdditional retreat under effective unseen phrasings (pp)
Base model+5.5 to +19.0
Models trained with TB-GRPO−2.7 to +0.4

Each change is relative to that model's own neutral-resampling condition; ranges cover the effective phrasings and evaluated trained models. Resistance persists under these unseen phrasings, providing evidence against dependence on the single challenge sentence used in training.

3.4. External quality measure: IFEval. We evaluated the Qwen2.5-1.5B-Instruct models trained on GSM8K and TriviaQA using 541 IFEval prompts (Zhou et al., 2023) and the benchmark's own instructions and evaluator. Prompt-level strict scores differ from base by −0.37 to +1.85 pp, with all paired 95% confidence intervals containing zero. This finds no statistically detectable change on the external benchmark, alongside the task-quality and semantic gains above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.