acceptodds
Withdrawn from ICLR 2027

CABLE: Correlation-Aware Bargaining for Multi-Reward Post-Training

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multi-reward post-training can maximize easy proxy rewards while task accuracy declines. We introduce CABLE (Correlation-Aware Bargaining with Liveness and Equalization), which combines correlation-based reward weighting with explicit control of normalization. It estimates interactions between objectives from token-logit correlations and combines their learning signals before a single actor backward pass. CABLE-L preserves reward-signal amplitudes; CABLE-LE adjusts weights using recent reward levels and adds entropy regularization. Across three training runs on Open-R1 coding with a 1.5B model, CABLE-LE reaches 12.56% LiveCodeBench-v6 pass@1 and a 36.37% four-suite average, improving on GAPO by 0.87 and 1.10 percentage points, respectively. On Math-M3, CABLE-L reaches a 52.21% five-benchmark average, compared with 51.32% for GRPO and 51.18% for the base policy. It also reduces mean response length from the base policy's 9,085 tokens to 3,203 tokens. On a matched actor-core workload, CABLE is over 4.2× faster than a parameter-gradient implementation of the same allocation rule.

Primary area:
reinforcement learning
Keywords:
Multi-objective reinforcement learning, Large language model post-training, Adaptive reward weighting

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_4zmyOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes CABLE for multi-reward post-training, which estimates reward interactions from token-logit correlations to allocate weights across rewards, combined with controls for normalization amplitude and, in the CABLE-LE variant, reward-level equalization. The method is evaluated on code generation and mathematical reasoning with 1.5B models against nine multi-reward baselines.

Critical strengths
  1. Efficient estimation of reward interactions. Estimating reward correlations from token logits requires only one backward pass, making CABLE about 4.2× faster than a parameter-gradient implementation.
Critical weaknesses
  1. Insufficient ablation of the key components. CABLE combines three controls (geometry, activity, and attainment), but the paper does not clearly show the contribution of each. The main text lacks a systematic ablation, and the only component comparison is deferred to the appendix (Table 5). This table is hard to interpret, as it uses development aliases (e.g., CorrNash, GS-Entropy CorrNash) that are not defined in the main text and does not indicate which components each variant includes. More importantly, although the correlation-aware allocation is claimed as a key contribution, no variant replaces it with uniform weights while keeping the other components. In fact, the geometry-only variant falls below the base model on coding (34.06 vs. 35.44). The activity-related controls (CV gating, amplitude preservation, and scalar standardization) are also changed jointly, so their individual effects cannot be distinguished.
  2. Incomplete multi-reward evaluation and task-dependent configurations. Although CABLE targets multi-reward optimization, the evaluation focuses almost entirely on task accuracy. Held-out metrics for the other objectives (e.g., execution reliability and length validity) are not systematically reported across methods, and there is no analysis of trade-offs across rewards. As a result, it is unclear whether the accuracy gains reflect better multi-reward optimization or simply more weight on correctness. On coding, the paper attributes the gains mainly to attainment equalization, which upweights correctness as the lowest-scoring reward (Section 5.4), yet neither the other objectives nor a simple baseline with a fixed higher weight on correctness is evaluated. Moreover, no single configuration works across tasks: CABLE-L falls below the base model on coding, while CABLE-LE falls below it on Math.
  3. Unclear presentation. The paper is difficult to follow: it uses dense notation and many briefly explained custom terms, defers details essential for interpreting the results to the appendix, and is written in a highly compressed style with little interpretation of the results.
Questions for authors
  1. Most trained methods, including GRPO and GDPO, fall below the base model on coding, and several baselines drop by 4-8 points on Math. Could the authors explain this and clarify how the baselines were tuned?
  2. Could the authors provide a clear component-wise ablation in the main text that isolates the contribution of geometry, activity, and attainment on both coding and Math?
  3. Proposition 1 shows that re-standardizing each objective after group normalization amplifies sparse rewards by 1/√f_k, and the introduction presents this as a limitation of separate normalization such as GDPO. However, as we understand it, GDPO applies its second normalization to the aggregated advantage rather than to each objective. Could the authors clarify whether Proposition 1 applies to GDPO? Relatedly, Figure 3 shows GDPO's process reward rising while its correctness collapses, but the paper does not explain the cause. Could the authors clarify what drives this collapse and how CABLE-L avoids it?
  4. CABLE introduces many hyperparameters (e.g., τ, β, the η schedule, transport ranges, and the number of allocation iterations). Could the authors describe how they were selected and provide a sensitivity analysis for the key ones?
  5. The best variant differs by task, and the two tasks also use different normalization and transport settings (Appendix B.1). Which configuration would the authors recommend for a new task?
Reviewer_aFgWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CABLE adapts the weights of multiple rewards using token-logit correlations, activity controls, and optional reward-level equalization. This avoids separate parameter-gradient computations for each objective and uses a single actor backward pass. Experiments study coding and mathematical reasoning, including comparisons with gradient-based reward weighting.

Critical strengths
  1. Estimating objective interactions in token-logit space is a useful engineering contribution. The geometry checks support the approximation, and the actor-core timing comparison shows a clear saving over computing separate parameter gradients.
  2. The coding experiments include three seeds and competitive baselines. The controls on normalization and reward activity make the paper informative beyond the final benchmark scores.
Critical weaknesses
  1. The performance gains are modest and configuration-dependent. CABLE-LE improves coding, whereas the math experiment favors CABLE-L; the math results use one seed and do not establish a clear improvement over GRPO. A clearer selection rule for L versus LE is needed to assess transfer across tasks.
  2. The finite fixed-point procedure is distinct from the reference bargaining optimizer. Appendix B acknowledges possible oscillation and the lack of a convergence certificate. The actual frequency and effect of these cases are not sufficiently quantified to judge robustness.
Questions for authors
  1. Can the authors report residuals or oscillation rates for the implemented solver, and how the results change with the iteration budget?
  2. How should L versus LE be selected without using benchmark outcomes, and how stable is the math comparison across training seeds?
Reviewer_dKFgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies multi-reward reinforcement-learning (RL) post-training for large language models, focusing on the failure mode where easily optimized proxy rewards dominate updates while task correctness stagnates or degrades (Section 1). The authors propose CABLE, which decomposes multi-reward aggregation into three components prior to a single actor backward pass (Section 3, Algorithm 1): (1) activity-preserving normalization via group standardization zikz _ {ik}, a rollout-wide coefficient-of-variation (CV) gate ℓk(n)\ell _ k^{(n)}, a signed transport map T\mathcal{T}, and an exponent-scheduled microbatch scaling (σb,k+ϵ)−ηn(\sigma _ {b,k} + \epsilon)^{-\eta _ n} (Section 3.1 Eq. (1)–(3)); (2) a token-logit interaction geometry that forms a weighted Gram matrix Skℓ=∑iWiaikaiℓS _ {k\ell} = \sum _ i W _ i a _ {ik} a _ {i\ell} from sampled-token score norms Wi=∑tmit∥hit∥2W _ i = \sum _ t m _ {it} \lVert h _ {it} \rVert^2 and runs a fixed 80-step nonlinear map B80(C)\mathcal{B} _ {80}(C) on the resulting cosine correlation matrix CC to obtain baseline weights γ\gamma (Section 3.2–3.3 Eq. (4)–(7)); and (3) a KL-regularized attainment controller that tilts γ\gamma toward under-attained objectives via αk(n,m)∝γk(n,m)exp⁡(−τek(n))\alpha _ k^{(n,m)} \propto \gamma _ k^{(n,m)} \exp(-\tau e _ k^{(n)}) (Section 3.4–3.5 Eq. (8)–(13)). Evaluated on 1.5B policies for code generation (Qwen2.5-Coder-1.5B-Instruct, Table 1) and mathematical reasoning (DeepSeek-R1-Distill-Qwen-1.5B, Table 2), CABLE-LE achieves a 36.37% three-seed four-benchmark average on code (+1.10 points over GAPO) and CABLE-L achieves a 52.21% single-seed five-benchmark average on math while reducing average response length by 64.7% (9,085 →\to 3,203 tokens). A controlled actor-core microbenchmark on a single B200 GPU (Section 5.5 Table 3) shows 4.21×4.21\times–4.23×4.23\times faster geometry acquisition than a matched parameter-gradient implementation (5.74 s / 5.70 s vs. 24.14 s / 24.13 s).

Critical strengths
  1. Low-overhead multi-reward interaction estimation in a single actor backward pass. Computing the weighted reward-interaction Gram matrix in token-logit space (Section 3.2 Eq. (4)–(5)) reuses probabilities already materialized during the actor forward pass and avoids MM separate parameter backward passes. The controlled actor-core benchmark on a B200 GPU (Section 5.5 Table 3) convincingly demonstrates that this reduces actor-core time from 24.14 s / 24.13 s to 5.74 s / 5.70 s (4.21×4.21\times–4.23×4.23\times speedup) at virtually identical peak memory (58.59 GiB vs. 58.57 GiB).

  2. Clear conceptual separation of interaction geometry, reward activity, and attainment level. Framing multi-reward failures through three distinct mechanisms—collinear or conflicting directions, variance-normalization erasure of sparse rewards (Proposition 1), and proxy takeover—provides a structured lens for analyzing multi-reward GRPO variants, and the paper is unusually candid in its Limitations section (Section 7) and fixed-policy bootstrap reporting (Appendix D.2).

Critical weaknesses
  1. Period-2 instability of the executed 80-step allocator (Eq. (7)) and unverified fidelity of the token-logit surrogate. As shown by the 3×33 \times 3 positive-definite counterexample above, the undamped fixed-step update in Eq. (7) can oscillate wildly between odd and even iterations ([0.00076, 0.00076, 0.99848] at step 79 vs. [0.49970, 0.49972, 0.00057] at step 80) and diverge sharply from the theoretical reference solution of Eq. (14) ([0.2484, 0.3128, 0.4387]). Because M=3M = 3 is tiny, solving the convex reference problem in Eq. (14) directly (or via a damped/line-searched solver with a stationarity check max⁡k∣bk(Cb)k−1∣≤tol\max _ k |b _ k (C b) _ k - 1| \le \text{tol}) would incur negligible cost and guarantee the theoretical properties claimed in Section 4. Furthermore, the promised derivation in Appendix B linking Eq. (5) to parameter-gradient geometry is missing, the frozen-policy correlation probe (0.991–0.999) only tests on-policy states where ρit=1\rho _ {it} = 1 and clipping is inactive, and there is no end-to-end training comparison against an exact parameter-gradient CABLE baseline.

  2. Fragile empirical evidence: 1.5B-only scale, variant reversal across tasks, non-significant bootstrap intervals, and entropy confounding.

    • Task-dependent variant reversal: On Code (Table 1), CABLE-LE leads (36.37%) while CABLE-L (34.81%) underperforms the untrained Base model (35.46%). On Math (Table 2), CABLE-L leads (52.21%) while CABLE-LE (50.88%) underperforms the Base model (51.18%), Dynamic Weighting (51.70%), and GRPO (51.32%). No task-independent rule for choosing between CABLE-L and CABLE-LE prior to test evaluation is provided.
    • Statistical uncertainty: Math-M3 uses only a single training seed (1234), and its fixed-policy 95% bootstrap intervals for CABLE-L overlap zero against both Base (+1.03 [-0.49, 2.56]) and GRPO (+0.89 [-0.32, 2.11], Appendix D.2 line 906). On Code (seed 4234), the four-suite bootstrap interval for CABLE-LE over GAPO (+0.96 [-0.75, +2.67]) and SAW (EMA) (+1.13 [-0.16, +2.45]) also crosses zero (Table 6).
    • Confounded component ablation and missing baselines: In single-seed Table 5 (evaluated with an older LCB scorer), adding entropy regularization alone (CORRNASH 34.06% →\to CORRNASH + entropy 35.48%, +1.42%) yields a larger gain than the entire gap between CABLE-LE and GAPO in Table 1 (+1.10%), while the "activity recipe" (36.74% →\to 37.15%, +0.41%) simultaneously bundles CV gating, log-companded transport, and final scalar standardization (and slightly hurts LCB 13.01% →\to 12.93% and MBPP+ 60.85% →\to 59.52%). In addition, MO-GRPO is listed in Section 5.1 but omitted from Table 1, and geometry-only CABLE is cited in Section 5.4 (-2.07 points vs. CABLE-L) but omitted from both Tables 1 and 2.
Questions for authors
  1. Diagnostics and stabilization of the 80-step allocator (Eq. (7) vs. Eq. (14)): On the positive-definite correlation matrix CC with entries (0.724,−0.352,−0.543)(0.724, -0.352, -0.543), Eq. (7) oscillates in a period-2 cycle between γ(79)≈[0.0008,0.0008,0.9985]\gamma^{(79)} \approx [0.0008, 0.0008, 0.9985] and γ(80)≈[0.4997,0.4997,0.0006]\gamma^{(80)} \approx [0.4997, 0.4997, 0.0006], far from the convex reference optimizer γ∗≈[0.2484,0.3128,0.4387]\gamma^* \approx [0.2484, 0.3128, 0.4387] of Eq. (14). Across the microbatches of your actual Code and Math runs, what fraction of updates exhibited negative off-diagonal correlations, non-negligible stationarity residuals max⁡k∣bk(Cb)k−1∣\max _ k |b _ k (C b) _ k - 1|, or period-2 oscillations? Why not directly solve the 3-variable convex problem in Eq. (14) (or use a damped solver with a convergence check)?

  2. Variant reversal (CABLE-L vs. CABLE-LE), inactive ηn\eta _ n in Code, and orthogonal ablations:

    • Why does CABLE-L fall below Base on Code (34.81% vs. 35.46%) while CABLE-LE falls below Base on Math (50.88% vs. 51.18%), and how should a practitioner select between them without test-set tuning?
    • Given that Code training stops at rollout slot 30 / checkpoint 29 (Table 4) while ηn=1−clip[0,1]((n−30)/20)\eta _ n = 1 - \mathrm{clip} _ {[0,1]}((n - 30)/20) in Eq. (25), can you confirm that ηn=1\eta _ n = 1 throughout the entire Code run?
    • Can you provide a multi-seed ablation under the official Table 1 scorer that isolates CV gating, signed transport, attainment tilting, and entropy regularization one factor at a time, along with entropy-matched baselines (GRPO + entropy, GAPO + entropy)?
  3. End-to-end parameter-gradient comparison, missing baselines, and broken references:

    • Can you provide the missing Appendix B derivation promised in Section 3.2 (line 171) and report end-to-end benchmark results for an exact parameter-gradient CABLE run (the 24.1 s route in Table 3)?
    • Please report the missing MO-GRPO results in Table 1 and the geometry-only CABLE results in Tables 1–2, clarify the undefined ToolRL reference in Appendix A.7 (line 754), and fix the missing main-text references to Appendix D and Tables 4–6.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.