- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes RLTB, which expresses multiple behavioral requirements as a joint interval target region in behavior space. The loss is the scaled squared distance from the behavior statistics to this region, so that a statistic inside its interval produces no gradient while a statistic outside it receives a correction proportional to the signed distance. The Jacobian of Ψ then maps this policy-level correction into weights on the primitive measurements, yielding per-output credit. The method is instantiated as TB-GRPO on top of GRPO, with the within-group standard-deviation normalization removed so that target-dependent magnitudes are preserved, and with leave-one-prompt-group-out (LOPO) cross-fitting used to estimate the weights so that outputs do not influence the feedback that scores them. Experiments report joint target gaps on GSM8K and TriviaQA.
Critical strengths
- Disclosure and reproducibility are unusually thorough: complete specification tables (Tables 10 and 12), sensitivity of results to the measurement definitions via two independent checkers (D.1.5), per-baseline search budgets (Table 15, which also shows that TB-GRPO performed no configuration search while RC and CGM-relax did), per-seed readouts (Table 18), and a code revision. A number of unfavorable results are also reported honestly in the appendix.
- The design of the paired and interventional measurements is valuable (abstention agreement, and retreat under a challenge that supplies no new evidence), and it incidentally exposes the instability of dual-ascent-based methods under such specifications (C-GRPO, exact variant, on TriviaQA: Mean 10.37±4.74 / Max 57.38±35.96).
Critical weaknesses
- Limited novelty, no clear separation between the increment and existing components, and framing that overstates the contribution. Equation (3) is a dead-zone exterior penalty (Π_T is simply coordinate-wise clipping); Equations (4)–(5) are the single-step contextual-bandit special case of the general-utility RL policy gradient (Zhang et al., 2020; Zahavy et al., 2021), obtained by a chain rule; LOPO is an adaptation of cross-fitting (Chernozhukov et al., 2018) and leave-one-cluster-out estimation (Chiang et al., 2022) to this setting; and Equation (7) is RLOO (Ahmadian et al., 2024). Section 5 itself concedes that, for directly averaged features with matched coordinate scales, the point-target discrepancy coincides with CGM-relax's calibration penalty up to a constant factor. All of these sources are cited, so the issue is not attribution: the paper never separates its actual increment (unified credit for interval targets on nonlinear coordinates; the cluster granularity for paired and interventional families) from the pre-existing components. The notation and derivations also occupy space out of proportion to their technical content, and include a considerable amount of material that is unrelated to the method or could be stated far more concisely; relative to the actual increment, the abstract's positioning of RLTB as a "principled foundation for target-directed post-training" is too strong.
- The central claims lack corresponding empirical support. (a) LOPO, presented as a selling point: Table 7 reports a training comparison at the actual budget (B = 128, G = 8, point target t = .10 — precisely the affine-weight case in which, by the paper's own Theorem 1, finite-sample unbiasedness applies), giving full-batch plug-in 11.8%, rollout-LOO 10.7%, and LOPO 11.0 / 11.4 / 10.6%, with the paper itself stating that "models trained with all three estimators approach the requested rate" and only one run per alternative. (b) Feedback magnitude and scales: in Table 25, Full and Sign-only give 1.09±0.27 and 1.39±0.17 over three seeds, but on seed 0 Sign-only is better on both Mean and Max (1.22 / 3.62 versus 1.35 / 4.16) and the paper states that "seed 0 favors Sign-only"; the two scale perturbations give 1.17 and 1.38, the same order as Full's own seed-to-seed spread, while the eight scales (values drawn only from {0.05, 0.10} plus 50 for length) come with no selection rule beyond the remark that s_k "controls corrective strength during training," so the claim of no case-specific hyperparameter retuning is unsupported. (c) Baselines: Section 5 explicitly discusses PID-Lagrangian methods (Stooke et al., 2020) and notes that dual ascent accumulates violations as integral feedback, yet never runs it as a baseline, even though RLTB's correction is essentially dead-zone proportional control; and on the single-target sweep CGM-relax's error (Table 23a, 0.8 / 1.9 pp) is no worse than TB-GRPO's (1.1±.6 / 2.3±1.0 pp). (d) Reference convention and scale: Table 21 shows a configured training floor of 75.07% for conditional correctness against a test-split value of 65.34%, a 9.73-pp shortfall, whereas the 0.07-pp gap in main-text Table 1 comes from re-applying the same multiplier to the test-split base reference; the joint specification is validated only at the 1B–1.5B scale, with the only larger model (Qwen2.5-7B) confined to the single-target abstention sweep, and the tasks are limited to rule-verifiable short-answer settings while Theorem 1 presupposes unbiased measurement (E[H_i] = m_θ), which no longer holds once measurements come from an LLM judge or a reward model.
Questions for authors
Please see the Weaknesses above.
6 comments
Response to Reviewer RcZ2 (2/5): LOPO's effect on optimization and target attainment
Part 2. LOPO: corresponding empirical support for its optimization role (W2a)
LOPO has a concrete optimization role: it removes receiving-cluster coupling that can displace the mean-update equilibrium from the requested target. At the reported training budget, the model trained with LOPO also has smaller joint Mean/Max gaps in our matched comparison. These results supply the corresponding empirical support requested for this design.
2.1. Why the original comparison shows close attainment. Theorem 1's affine-weight unbiasedness guarantee applies to the canonical LOPO estimator. Affine weights do not automatically make full-batch plug-in unbiased: the receiving cluster and its estimated feedback remain coupled.
The original , , 10% abstention comparison concerns a directly averaged statistic at a large budget. Close attainment there is consistent with a small coupling effect. The revised manuscript now connects this effect to optimization and separates controlled mean-update equilibria from actual training endpoints.
2.2. Improved joint attainment at the actual training budget. On Qwen2.5-1.5B-Instruct and TriviaQA, we hold the complete six-requirement specification, initialization, prompt stream, cluster weighting, and 1,500-update budget fixed (, ). Only inclusion of the receiving cluster in the feedback estimate changes. Here counts prompts; each neutral–challenge pair forms one dependency cluster.
| Estimator used for training | Held-out Mean gap (pp) | Held-out Max gap (pp) |
|---|---|---|
| LOPO | 0.37 | 1.58 |
| Full-batch plug-in | 0.48 | 2.12 |
The LOPO-trained model's endpoint has 0.11/0.54 pp smaller Mean/Max gaps. This matched training ablation directly addresses optimization at the reported budget (revised Appendix D.4, Table 25).
2.3. The systematic target displacement that LOPO removes. Revised Appendix B.2, Figure 5, isolates the mechanism using a 20% point target, , and independent measurement clusters. Holding the task and sampling design fixed, the zero of the expected canonical update has the following offsets from the target:
| Controlled measurement | Plug-in displacement (pp) | LOPO displacement (pp) | |
|---|---|---|---|
| Direct mean | 8 | 0.968 | 0.007 |
| Direct mean | 128 | 0.061 | 0.008 |
| Conditional rate; event-cluster probability | 8 | 9.917 | 0.027 |
| Conditional rate; event-cluster probability | 128 | 0.650 | 0.007 |
Entries are estimated absolute offsets of the mean-update equilibrium from the target, comparing both estimators at the same . The conditional experiment holds the population denominator fixed with respect to the policy. Figure 5 includes all four tested budgets: 8, 32, 128, and 512.
The direct-mean effect is small at a large budget, consistent with the limited separation in the original comparison. For the sparse conditional measurement, the same coupling causes a much larger displacement, which LOPO removes to the displayed precision. These are mean-update equilibria, distinct from finite Adam training endpoints.
Together, the results identify a systematic optimization error removed by LOPO and show improved joint target attainment in matched training. Revised Appendix A.4.2 connects bias and variance to expected loss decrease for canonical updates under stated smoothness, moment, and step-size conditions.
Response to Reviewer RcZ2 (3/5): feedback magnitude, scale selection, and no-retuning scope
Part 3. Feedback magnitude, scales, and case-specific retuning (W2b)
Adaptive feedback translates current target discrepancies into coordinated corrections. Retaining its magnitudes improves joint control on TriviaQA; deliberate changes to relative feedback allocation preserve small joint gaps on GSM8K. Across target requests, the same feedback rule supports reuse of a fixed configuration without per-request hyperparameter search.
3.1. What magnitude and scale do. The declared discrepancy gives behavior-level feedback and primitive-measurement weights
\mathbf w=J_\Psi^\top\mathbf d.$$ The signed gap determines the direction and magnitude of correction. The scale $s_k$ accounts for units and relative corrective strength; the Jacobian maps these corrections into output credit. Positive scales preserve the target region and zero-loss set while affecting training trajectories, finite-budget attainment, and trade-offs if targets cannot all be met. They are explicit design choices, separate from evaluation tolerances. **3.2. Empirical effect of preserving magnitude.** The original GSM8K comparison has the run-to-run variation you identify, including the seed-0 reversal. The following completed comparison is on **TriviaQA**, with Qwen2.5-1.5B-Instruct, the same six requirements, and the same 1,500-update budget: | Training feedback | Mean gap (pp) | Max gap (pp) | Retreat (%) | |---|---:|---:|---:| | Full | $0.31 \pm 0.10$ | $1.22 \pm 0.49$ | **1.55–1.77** | | Sign-only | $0.96 \pm 0.04$ | $5.12 \pm 0.33$ | 6.67–7.26 | *Gaps are mean $\pm$ SD over three matched training runs; retreat gives their range. The retreat upper bound is 1.92%.* All Full models meet the retreat bound; all Sign-only models exceed it. Mean and Max gaps improve in every matched pair. This supports the value of magnitude information for **joint target control**, particularly the interventional requirement. Sign-only takes the sign of the primitive coefficient, removing magnitudes contributed by the discrepancy, scales, and measurement Jacobian together; it tests this complete construction. **3.3. Deliberately changing relative feedback allocation.** On **GSM8K**, we multiply the scales for abstention, format, omission, and length by **0.8**, and the remaining scales by **1.25**, then reverse the assignment. At fixed measurements, the corresponding feedback coefficients scale by **1.5625 and 0.64**. These factors were chosen to cross feedback-allocation ranking thresholds identified on the reference trajectory, testing whether joint control remains effective after changing which requirements receive more feedback. Targets, evaluation criteria, and other training settings stay fixed. The trained runs confirm that allocation changes: omission's absolute coefficient share moves from **22.8% to 31.1%/14.8%**. Yet Mean gaps remain **1.38/1.17 pp**, compared with **1.35 pp** under the original scales. These are feedback-coefficient shares; the endpoint comparison supports effective joint control under the tested reallocations. **3.4. Why adaptive feedback supports no case-specific retuning.** For a point target, $d_k=(t_k-\mu_k)/s_k^2$: changing $t_k$ directly changes the feedback's direction, magnitude, and zero, while scales and optimization settings can stay fixed. Target-dependent weights are computed from current measurements during training, avoiding an additional search to map each requested rate to a fixed reward coefficient. In the **5%, 10%, 15%, 30%, and 50% abstention sweeps**, models trained with TB-GRPO track the requested rates with mean errors of **0.8–1.1 pp** on **Qwen2.5-1.5B-Instruct, Llama-3.2-1B-Instruct, and Qwen2.5-7B-Instruct**. **Scales, optimization, and sampling settings remain fixed across the five requests, with no per-request configuration search.** These results demonstrate reuse of an initially specified configuration as targets change. Initial settings remain explicit: GSM8K uses coarse scales of **0.05/0.10 for rates and 50 tokens for length**. Measurement precision informs the design: TriviaQA's format scale **0.08** follows estimated SD **0.0245** at $B=128$ and the check $\mathrm{sd}(\widehat m)\le s/3$. This check constrains usable scales without uniquely selecting them. We will clarify the initial selection rationale and demonstrated reuse scope in the manuscript.
Response to Reviewer RcZ2 (1/5): contribution beyond the existing components
Thank you for recognizing the value of the paired/interventional measurements and the thorough disclosure and reproducibility materials.
Your concerns center on the contribution beyond existing components and the empirical support for the framework's design. We address them in five parts: Part 1 explains expressive joint specifications and unified feedback; Part 2 connects LOPO to optimization through the revised analysis and matched training; Part 3 addresses feedback magnitude, scale selection, and configuration reuse; Part 4 presents the completed PI comparison and the CGM special case; and Part 5 addresses the reference convention, larger models and open-ended generation, and learned evaluators.
Part 1. Separating the contribution from existing components (W1)
We respectfully disagree with the overall assessment of limited novelty. RLTB makes joint behavioral specifications an explicit interface to training, translating desired values and admissible ranges into corrective feedback and output credit. This unified construction enables joint control across heterogeneous measurements and supports alternative feedback controllers.
1.1. Expressive specifications through unified feedback. RLTB organizes behavioral requirements along three composable dimensions: measurement families (single-output, paired, or interventional), statistics (marginal, joint, or conditional), and targets (points, intervals, or upper/lower bounds). A unified target-to-credit construction translates their current discrepancies into feedback for the participating outputs or rollouts, enabling heterogeneous requirements to be optimized jointly within the same training procedure. This combination of expressive specification and unified feedback is a central contribution.
The dead-zone penalty, general-utility policy gradient, cross-fitting, and RLOO supply established mathematical foundations. The direct-mean, matched-scale point-target correspondence with CGM-relax is a recovered special case. The framework contribution lies in the reusable construction connecting specifications, measurements, feedback, and estimation dependencies, and the control capabilities it enables. Stochastic computation graphs provide a related example of a unifying formalism built on established gradient tools (Schulman et al., 2015).
1.2. Equivalent constraints need not give equivalent feedback. For a conditional rate r=n/d with d>0, r≤u and n−ud≤0 define the same feasible set. Directly penalizing their respective violations gives
L_{\mathrm{cost}}=\frac{[n-ud]_+^2}{2s^2},$$ where $[x]_+=\max(x,0)$. A policy-dependent denominator generally makes their gradients differ. RLTB specifies discrepancy directly in behavioral coordinates and derives the corresponding output credit. Appendix A.6 also shows that some paired/interventional target sets cannot be represented by finitely many fixed single-output expected-cost constraints. General constraints can accommodate richer measurements; RLTB provides a common construction that makes these specifications operational together. **1.3. Joint control enabled by the specification.** On Qwen2.5-1.5B-Instruct and TriviaQA, we hold the training procedure, feedback rule, and budget fixed, and remove the paired-output and interventional requirements while retaining the same four single-output requirements: | Target behavior specification | Shared-four Mean gap (pp) | Retreat (%) | |---|---:|---:| | Four single-output requirements | $0.54 \pm 0.11$ | 38.36–40.35 | | Complete six-requirement specification | $0.46 \pm 0.15$ | **1.55–1.77** | *Mean gaps are mean $\pm$ SD over three training runs; retreat gives their range.* Models trained under both specifications closely control the shared requirements. Adding the paired and interventional requirements enables a substantial additional behavioral benefit: retreat falls from 38–40% to below 1.8%. Part 4 complements this specification ablation by showing that the same credit construction supports effective joint control after substituting a PI feedback controller. **1.4. Positioning and presentation.** Here, “principled” refers to objective-derived feedback and estimator guarantees under explicit assumptions: **a principled framework for deriving training feedback from joint behavioral specifications**. Its objects are measurable policy statistics; parsers and GRPO instantiate measurement and optimization for LLMs. We will clarify the contribution boundary and its relationship to the established components in the manuscript.