acceptodds
Under review as a conference paper at ICLR 2027

GEOMA: Geometric and Econometric Objectives for Multi-Reward Alignment

Abstract

Alignment of large language models is increasingly formulated as optimization over multiple rubric signals. These signals typically exhibit strong statistical dependencies, ranging from redundancy to anti-correlation (e.g., conciseness versus correctness), raising the question of how to robustly convert vector-valued rewards into scalar advantages. While recent state-of-the-art methods like GDPO address scale discrepancies via per-dimension normalization, they ignore reward geometry by treating coordinates as orthogonal. This mishandles correlations: redundant objectives are double-counted, while anti-correlated rewards are dominated by high-variance trade-off directions. We introduce GEOMA (Geometric and Econometric Objectives for Multi-reward Alignment), a framework that decomposes reward aggregation into geometric preconditioning via covariance sphering of reward vectors, and econometric aggregation such as Nash Welfare and SoftMin. We formally characterize these objectives, providing theoretical guarantees for their robustness to reward hacking and signal redundancy. Empirically, we demonstrate that GEOMA outperforms GDPO on Math reasoning and Tool Calling. It further improves alignment on AlpacaEval as a test-time re-ranking strategy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-Objective RL, Reinforcement Learning, Model Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_WCFeOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GEOMA separates multi-reward advantage construction into covariance-based preconditioning and aggregation by a sum, logistic Nash welfare, or SoftMin. It studies whitening and redundancy properties and explains that sum aggregation reduces to a weighted sum of the original rewards. Experiments vary these choices for tool use and mathematical reasoning with 1.5B models. Batch-covariance preconditioning with a sum improves mean math accuracy over GDPO from 46.8% to 48.4% and reduces mean length from 3,897 to 2,518 tokens, while the unnormalized GRPO baseline remains more accurate but longer. Tool-calling improvements in BFCL accuracy are smaller.

Critical strengths
  1. The design space is explicit and easy to study. Remark 4.1 and Proposition B.16 correctly make clear that covariance-plus-sum is a reward-weighting rule, and the duplicate-reward example acknowledges that whitening reduces rather than eliminates double counting. Those qualifications are useful, as they distinguish algebraic properties of the preconditioner from claims about policy behavior.

  2. The accuracy/length results provide a concrete reason to investigate covariance-aware weighting. Batch-covariance with Sum retains or improves GDPO's accuracy on all four math benchmarks while substantially reducing average generation length (Table 2). Reporting GRPO's higher accuracy and longer responses also exposes the trade-off instead of declaring one scalar metric to be the whole task.

Critical weaknesses
  1. Monotonic welfare in the whitened coordinates does not imply that the composed objective protects the original rewards. Section 4 says the ZCA coordinates still correspond to the original rubrics, but whitening mixes those rubrics and need not preserve coordinatewise monotonicity. For a fixed valid covariance Sigma = [[1, 2.9], [2.9, 9]] and ridge 1e-6, P = (Sigma + ridge I)^(-1/2) is approximately [[3.7441, -1.1116], [-1.1116, 0.6777]]. The Sum weights P^T 1 are approximately (2.6325, -0.4338): increasing only the second original reward reduces the scalar score. Near a centered reward of zero, the same increase also reduces both logistic Nash and SoftMin, because their gradients there are positive multiples of P^T 1. Thus the composed objectives can locally prefer a Pareto-inferior original reward vector even with a fixed preconditioner. The welfare monotonicity results in Appendix B concern the transformed coordinates and do not resolve this issue. This directly limits the interpretation that no original rubric is being sacrificed.

  2. The experiments do not yet establish that covariance sphering is preferable to a well-chosen weighting baseline. The strongest math configuration is exactly an adaptive weighted sum, but no tuned positive-weight or comparable adaptive-weight control is reported. BFCL averages span less than one percentage point across configurations, and each configuration is trained once. The five math generations measure sampling variation for a trained model, not training-seed uncertainty. All reported tasks use two rewards, so the experiments also do not directly test the motivating case of redundant rewards in the presence of an independent objective.

Questions for authors
  1. Under what conditions is G(P(r - mu)) monotone in each original reward? Can you address the covariance counterexample above and distinguish protection of transformed coordinates from protection of the original rubrics? If monotonicity is not intended, the alignment interpretation needs to state that limitation explicitly.
  2. How do the covariance-Sum variants compare with tuned nonnegative scalar weights under the same training budget, including the resulting accuracy–length frontier? This comparison would establish what the covariance estimate adds beyond choosing a different trade-off.
  3. Do the gains persist across independent training seeds, and under a controlled three-reward experiment that duplicates one reward while retaining an independent third objective? Please separate training variation from repeated decoding of one checkpoint.
3 comments
AuthorsOpenReview ↗

Original-Reward Monotonicity and the 2x2 Counterexample

We thank Reviewer WCFe for the rigorous technical reading and concrete experimental suggestions, which directly shaped our new proofs and experiments in Appendix G.

Strengths Highlighted by Reviewer WCFe

  1. Explicit, easy-to-study design space: Remark 4.1 and Proposition B.16 clarify that covariance preconditioning with Sum acts as an adaptive reward-weighting rule, while carefully distinguishing algebraic properties of the preconditioner from claims about policy behavior.
  2. Concrete multi-objective gains and transparent reporting: Batch-Cov-Sum retains or improves GDPO's accuracy across all four math benchmarks while substantially shortening generation length, and reporting GRPO alongside it exposes the true multi-objective trade-off rather than collapsing the task to a single scalar.

Weakness 1 & Question 1: Original-Reward Monotonicity and the 2x2 Counterexample

Reviewer Query: Because covariance whitening mixes the original rewards together, the reviewer showed via an explicit 2x2 example (correlation 0.97, scale ratio 3) that an original reward's effective weight can turn negative (-0.43), and asked under what conditions GEOMA preserves each original reward's priority across Sum, Nash, and SoftMin.

"Under what conditions is G(P(r - mu)) monotone in each original reward? Can you address the covariance counterexample above and distinguish protection of transformed coordinates from protection of the original rubrics? If monotonicity is not intended, the alignment interpretation needs to state that limitation explicitly."

Our Response (addressed in Section 4, Section 6.2, and Appendix G.1):

  1. New proofs and audits added (Appendix G.1):
    • Closed-form theorems separating transformed-coordinate welfare (Theorem B.6) from original-rubric monotonicity (Propositions G.1 and G.2).
    • Exact resolution of your 2x2 counterexample via ridge floor and correlation whitening (ZCA-cor).
    • Online correlation and weight audit across every training step of our 1.5B, 7B, and math RL runs.
  2. Global guarantee when objectives conflict: Whenever pairwise covariances are zero or negative, the regularized covariance matrix is a Stieltjes M-matrix, making every entry of its inverse square root non-negative for any K >= 2 across Sum, Nash, and SoftMin.
  3. Exact boundary for positively correlated rewards: For two positively correlated rewards, both weights stay non-negative across every scale ratio whenever correlation is at most 1 / sqrt(2) (about 0.71). In all of our RL runs, batch correlation stays below 0.59, giving zero negative-weight steps.
  4. Resolving your 2x2 example and stating boundaries explicitly: Your matrix (scale ratio 3, correlation 0.967) sits above its 0.854 limit; either ridge >= 0.219 or normalizing variances first via ZCA-cor restores positive weights (+0.71, +0.24). Section 4 also states openly that monotonicity can fail under extreme K >= 3 suppressor correlations or far from the mean under nonlinear objectives at high correlation, and provides diagonal shrinkage toward GDPO whenever strict non-negativity is required.
Regime or Check (Appendix G.1)Effective Original-Reward WeightsExplanation
Conflicting or uncorrelated rewards (all K >= 2)Entrywise P >= 0 (all positive)Stieltjes M-matrix guarantees global Pareto priority
Any two positively correlated rewards (rho <= 0.71)Strictly positive for every scale ratioSign flips require both high correlation and scale skew
Reviewer WCFe matrix (rho = 0.967, scale ratio 3)Raw: (+2.63, -0.43); ZCA-cor: (+0.71, +0.24)Unit-variance scaling before whitening removes the sign flip
All logged RLHF batches (1.5B, 7B, math)Max rho = 0.59; 0% negative-weight stepsEmpirical RL batches stay strictly inside the monotone zone

Findings: Original-rubric Pareto dominance holds globally for conflicting rewards and below 0.71 correlation across all scale ratios, covering 100% of our batch RL steps. Furthermore, Section 4 explicitly states the high-correlation boundary threshold (Appendix G.1).

AuthorsOpenReview ↗

Tuned Non-Negative Scalar Weights on Math

Reviewer Query: Because covariance preconditioning with Sum aggregation acts as an adaptive weighted sum of rewards, the reviewer asked how GEOMA compares against tuned fixed positive weights (1 - alpha) * r_acc + alpha * r_len under the same training budget, and what online covariance estimation adds beyond choosing a static trade-off.

"How do the covariance-Sum variants compare with tuned nonnegative scalar weights under the same training budget, including the resulting accuracy-length frontier? This comparison would establish what the covariance estimate adds beyond choosing a different trade-off."

Our Response (addressed in Section 7.2, Appendix G.3, and Appendix G.4):

  1. New math experiments and audit conducted:
    • 27 checkpoint evaluations on DeepSeek-R1-Distill-Qwen-1.5B (each decoded across 5 seeds) sweeping static weights alpha in {0.00, 0.50, 0.65, 0.80} alongside GDPO, our Batch-Diag-Sum marginal ablation, and an independent training seed of Batch-Cov-Sum.
    • Codebase verification clarifying that Table 2's GRPO row corresponds to the alpha = 0.00 accuracy-only anchor.
  2. Why fixed weights lose calibration mid-training: At initialization, 34% of math rollouts hit the 8,192-token limit (scoring 0 on both accuracy and length), creating +0.44 positive correlation that fades near zero once responses shorten below 2,000 tokens. Fixed weights cannot track this shift, oscillating in late-stage accuracy and stalling above 3,380 tokens.
  3. Why online covariance estimation improves the frontier: Positive early correlation causes Batch-Cov-Sum to boost length compression while truncation is active and automatically relax toward marginal scaling once truncation disappears, improving accuracy and length smoothly without tuning alpha.
Math Checkpoint (DeepSeek-R1-1.5B, 5 Decoding Seeds, Appendix G.3)MATH-500AIME-25AMC-24MinervaMean AccMean LengthExplanation
Untrained base model82.44%26.00%53.33%23.68%46.36%10,418Frequent 8k truncation on hard problems
Accuracy-only anchor (alpha = 0.00, step 200)85.2%27.3%61.8%26.5%50.2%6,848Accuracy upper bound with no length control
Static alpha = 0.50 (step 150 -> step 175)84.44% -> 84.56%28.67% -> 20.67%62.22% -> 56.45%23.75% -> 25.22%49.77% -> 46.72%3,380 -> 3,476Fixed weight drops 3 points late in training
Static alpha = 0.65 (step 200, full budget)83.72%24.67%58.22%23.38%47.50%3,574Full-budget static run stalls above 3,500 tokens
Static alpha = 0.80 (step 175)83.60%22.67%64.00%23.16%48.36%3,545About 400 more tokens than Batch-Cov-Sum at step 150
GDPO baseline (step 200, full budget)83.2%28.0%52.4%23.5%46.8%3,897Per-group scaling trails on both axes
Batch-Diag-Sum ablation (step 100)83.16%24.67%56.89%23.46%47.04%3,100Within 0.3 points of Batch-Cov-Sum at step 100 (47.34%)
GEOMA Batch-Cov-Sum (step 150, new train seed)83.68%28.67%56.00%25.15%48.37%3,135Dominates all late-horizon static runs
GEOMA Batch-Cov-Sum (step 200, full budget)83.9%28.0%57.7%23.9%48.4%2,518+0.9% acc and 1,056 fewer tokens vs alpha = 0.65

Findings: Online covariance weighting adapts automatically outperforming tuned static weights on late-stage accuracy. It additionally saves roughly 400 tokens at step 150 (independent training seed) and over 1,000 tokens at step 200 (Appendix G.3).

AuthorsOpenReview ↗

Training Seeds, Experimental Controls

Question 3: Multi-Seed Training Variance and Controlled 3-Reward & 5-Reward Ladders

Reviewer Query: The reviewer asked whether GEOMA's gains hold across independent training seeds (reported separately from multi-sample decoding of one checkpoint) and under a controlled multi-reward experiment that duplicates one reward alongside an independent constraint.

"Do the gains persist across independent training seeds, and under a controlled three-reward experiment that duplicates one reward while retaining an independent third objective? Please separate training variation from repeated decoding of one checkpoint."

Our Response (addressed in Section 7.1, Appendix G.2, and Appendix G.4):

  1. New multi-seed and duplication experiments conducted:
    • All revised tables strictly separate cross-seed training SD (3 independent training seeds: 42, 101, 202) from repeated-decoding SD (5 evaluation seeds).
    • Controlled K = 3 (2 copies of correctness + 1 format) and K = 5 (4 copies + 1 format) duplication ladders on both Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct, testing GDPO, our Batch-Diag-Sum marginal ablation, Batch-Cov-Sum, and exact-invariance C-Prec-Sum.
  2. Cross-seed training stability: At 1.5B on 2 rewards, Batch-Cov-Sum cuts training-seed SD from 26.3 +/- 13.6 steps (GDPO) to 15.0 +/- 1.0 steps and leads Multi-Turn BFCL-v3 (3.33 +/- 0.26% vs 2.80 +/- 0.14%). Because single-turn BFCL-v3 uses Qwen's native tool syntax and varies by under 1 point across seeds, we drop single-turn superiority claims from the manuscript.
  3. Controlled K = 3 and K = 5 duplication: With 2 initially uncorrelated rewards at 1.5B, covariance is nearly diagonal so Batch-Cov matches our Batch-Diag ablation while halving seed SD. Once correctness is duplicated, marginal scaling weights m copies at m : 1 (collapsing GDPO and delaying Batch-Diag to 29 steps), whereas Batch-Cov-Sum (sqrt(m) : 1) and C-Prec-Sum (exact 1 : 1) stay fast and protect correctness at both 1.5B and 7B.
Controlled Experiment (Steps to 85% Format, Appendix G.2)GDPO BaselineBatch-Diag AblationGEOMA Batch-Cov-SumGEOMA C-Prec-SumExplanation
1.5B, K = 2 (mean +/- training SD, 3 seeds)26.3 +/- 13.615.7 +/- 2.115.0 +/- 1.0Matches Batch-DiagNear-diagonal initial covariance; 11 steps faster than GDPO with 13x lower training SD
1.5B, K = 3 (2 copies of correctness + format)Collapsed20 steps16 steps16 stepsMarginal double-counting collapses GDPO
1.5B, K = 5 (4 copies of correctness + format)Collapsed29 steps20 steps16 steps4x over-counting slows Batch-Diag; C-Prec-Sum shows zero slowdown
7B, K = 3 (2 copies of correctness + format)25 steps20 steps16 steps22 steps (+1.88 cor.)At 7B, Batch-Cov-Sum beats GDPO by 9 steps; C-Prec-Sum reaches highest correctness

Findings: Across independent training seeds and 1.5B-to-7B duplicate-reward ladders, covariance and correlation-precision preconditioning eliminate seed instability and prevent the collapses and slowdowns caused by marginal double-counting (Appendix G.2).


Your note that GEOMA's design space is explicit and easy to study, and that our paper carefully distinguishes algebraic properties of whitening from policy behavior, was a real highlight of the review process for us. Thank you again for your 2x2 counterexample and experimental suggestions, every one of which we implemented and proved in full. If our new monotonicity proofs, tuned static-weight frontier, multi-seed runs, and K = 3 and K = 5 duplication experiments have resolved your concerns, we would be deeply grateful if you would consider raising your score toward acceptance.

Reviewer_9iLrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes GEOMA, a framework for multi-reward LLM alignment that addresses correlations between reward signals and undesirable trade-offs arising from linear reward aggregation. The authors introduce two components: (1) geometric preconditioning through covariance sphering to decorrelate reward signals, and (2) econometric aggregation using Sum, Max Nash Welfare, and SoftMin to balance different objectives. The paper provides theoretical characterizations of the proposed transformations and demonstrates improvements in accuracy–response-length trade-offs and tool-calling performance.

Critical strengths
  1. Well-motivated problem and interesting framework: The paper addresses an important limitation of existing multi-reward alignment approaches, where correlations between reward signals may lead to redundant contributions. Separating geometric preconditioning from econometric aggregation provides a systematic framework for studying these effects.

  2. Theoretical analysis and promising empirical results: The authors characterize the properties of covariance sphering and its relationship to reward reweighting. Experiments demonstrate improved accuracy–response-length trade-offs relative to GDPO and better learning of formatting constraints in tool-calling tasks.

Critical weaknesses
  1. Unclear interpretation of decorrelated rewards: Covariance sphering eliminates linear correlations but does not necessarily remove nonlinear dependencies or semantic redundancy. Furthermore, the transformed coordinates are mixtures of the original rewards, making it unclear whether Nash welfare and SoftMin still represent meaningful trade-offs between the original alignment objectives. The transformation may also fail to preserve Pareto dominance.

  2. Limited novelty and experimental validation: Under linear aggregation, covariance sphering is equivalent to covariance-dependent reward reweighting. The paper does not sufficiently compare against alternative adaptive weighting methods or isolate the benefits of combining sphering with nonlinear aggregation. Additionally, the experiments lack multiple random seeds and uncertainty estimates.

Questions for authors
  1. Since covariance sphering mixes the original reward coordinates and does not necessarily preserve Pareto dominance, how do the authors justify applying welfare-based aggregators to the transformed rewards? What guarantees that the resulting optimization remains consistent with the original alignment objectives?

  2. Given that covariance sphering with Sum aggregation reduces to reward reweighting, what advantages does the proposed approach offer over existing adaptive weighting methods? Can the authors provide experiments that isolate the contribution of decorrelation when combined with nonlinear aggregation?

4 comments
AuthorsOpenReview ↗

Pareto Consistency Under Whitening

We thank Reviewer 9iLr for rating our paper's soundness 3 and presentation 3, and for raising insightful questions on Pareto consistency and nonlinear aggregation. Below we address Question 1 (Part 1), adaptive weighting and multi-seed validation (Part 2), and isolating decorrelation with nonlinear aggregators plus novelty (Part 3).

Strengths Highlighted by Reviewer 9iLr

  1. Well-motivated problem and systematic framework: Separating geometric preconditioning from econometric aggregation provides a clean way to address redundant contributions from correlated rewards.
  2. Soundness (3), theoretical clarity, and multi-objective gains: Characterizing covariance whitening as adaptive reward reweighting clarifies optimization behavior, while experiments improve math accuracy-length trade-offs over GDPO and learn strict tool-calling format constraints.

Weakness 1 & Question 1: Welfare Aggregators on Whitened Coordinates and Pareto Dominance

Reviewer Query: Because covariance whitening mixes the original rewards and removes only linear correlation, the reviewer asked how we justify applying Nash welfare and SoftMin to whitened coordinates and what guarantees consistency with the original alignment objectives (Pareto dominance).

"Since covariance sphering mixes the original reward coordinates and does not necessarily preserve Pareto dominance, how do the authors justify applying welfare-based aggregators to the transformed rewards? What guarantees that the resulting optimization remains consistent with the original alignment objectives?"

Our Response (addressed in Section 4, Appendix C, and Appendix G.1, Propositions G.1-G.2 & Table 4):

  1. New proofs and empirical audits added:
    • Closed-form theorems proving when Sum, Nash, and SoftMin preserve original-rubric Pareto dominance (Propositions G.1 and G.2, Figure 8).
    • Online correlation and weight audits across every step of our 1.5B, 7B, and math RL runs (Table 4, Figure 10).
  2. Global Pareto guarantee when objectives conflict: Welfare objectives (Nash, SoftMin) are designed for conflicting goals (zero or negative pairwise covariance, such as accuracy versus brevity). In this entire regime, the regularized covariance matrix is a Stieltjes M-matrix, making every entry of its inverse square root non-negative for all K >= 2 and guaranteeing global Pareto dominance in the original rubrics.
  3. Positive correlation and nonlinear dependence boundaries: Below 0.71 pairwise correlation, original-reward weights at the mean stay positive across all scale ratios (covering 100% of our batch RL steps, where max correlation is 0.59). Section 4 and Appendix C also explicitly note that whitening removes linear correlation rather than higher-order semantic dependence, and provide diagonal shrinkage toward GDPO whenever strict non-negativity is needed under extreme correlation.
Regime Checked (Appendix G.1, Table 4)Original-Rubric ConsistencyExplanation
Conflicting or uncorrelated rewards (all K >= 2)Global Pareto dominance guaranteedEntrywise non-negative Stieltjes inverse square root
Positively correlated pairs (rho <= 0.71)Positive weights at the mean for all scalesCovers 100% of logged batch RL steps (max rho = 0.59)
High positive correlation away from meanExact dual-cone bound + diagonal shrinkageSmoothly interpolates to Diag-Nash/SoftMin when required
Nonlinear / semantic dependenceStated openly as scope boundarySecond-order whitening targets linear covariance only

Findings: Applying Nash and SoftMin to whitened rewards preserves global Pareto dominance whenever alignment objectives conflict, and remains strictly inside the monotone regime across 100% of our batch RL runs (Appendix G.1).

AuthorsOpenReview ↗

Adaptive Weighting Controls and Multi-Seed Validation

Weakness 2a & Question 2a: Advantages Over Marginal Adaptive Weighting and Multi-Seed Uncertainty

Reviewer Query: Given that covariance preconditioning with Sum acts as reward reweighting, the reviewer asked what advantages GEOMA offers over simpler adaptive weighting methods, and requested multiple random seeds with uncertainty estimates.

"Given that covariance sphering with Sum aggregation reduces to reward reweighting, what advantages does the proposed approach offer over existing adaptive weighting methods? ... Additionally, the experiments lack multiple random seeds and uncertainty estimates."

Our Response (addressed in Sections 7.1-7.2 and Appendix G.2-G.3, Tables 5-6 & Table 9):

  1. New multi-seed and adaptive-weighting experiments conducted:
    • 3 independent training seeds (42, 101, 202) on 1.5B tool calling (Table 5) and 27 five-seed math evaluations (Table 9), separating training SD from decoding SD.
    • Scope-matched marginal adaptive baseline (Batch-Diag-Sum, scaling by each reward's own batch variance) versus Batch-Cov-Sum and exact-invariance C-Prec-Sum across 1.5B, Qwen2.5-7B-Instruct, and K = 3, 5 duplicated rewards (Table 6).
  2. Multi-seed reliability: Batch-Cov-Sum cuts cross-seed format-learning SD from 26.3 +/- 13.6 steps (GDPO) down to 15.0 +/- 1.0 steps and leads Multi-Turn BFCL-v3 (3.33 +/- 0.26% vs 2.80 +/- 0.14% GDPO), while we drop single-turn BFCL superiority claims because native Qwen prompt syntax compresses single-turn scores within 1 point across seeds.
  3. Why off-diagonal covariance beats marginal adaptive weighting: Marginal adaptive methods inspect only individual variances and therefore over-weight m correlated copies of a reward at m : 1. Accounting for off-diagonal correlation prevents duplicate over-counting (sqrt(m) : 1 for Batch-Cov, 1 : 1 for C-Prec) and prevents early correctness drops at 7B scale.
Experiment Setting (Appendix G.2, Tables 5-6)GDPO (Per-Group)Batch-Diag (Marginal)GEOMA Batch-Cov-SumGEOMA C-Prec-SumExplanation
1.5B, K = 2 (3 seeds, steps to 85% format)26.3 +/- 13.615.7 +/- 2.115.0 +/- 1.0Matches Batch-Diag11 steps faster than GDPO with 13x lower seed SD
1.5B, K = 3 (2 copies of correctness + format)Collapsed20 steps16 steps16 stepsMarginal double-counting collapses GDPO
1.5B, K = 5 (4 copies of correctness + format)Collapsed29 steps20 steps16 steps4x over-weighting slows Batch-Diag; C-Prec has zero delay
7B, K = 2 (step-10 correctness)+1.85+0.29+1.71Matches Batch-DiagOff-diagonal correlation prevents a sharp early accuracy dip
7B, K = 3 (2 copies, steps to 85% format)25 steps20 steps16 steps22 steps (+1.88 cor.)Batch-Cov leads GDPO by 9 steps; C-Prec tops correctness

Findings: Across multiple seeds and 7B scale-up, off-diagonal covariance preconditioning cuts seed variance by 13x over GDPO and prevents the collapses, slowdowns, and early accuracy drops suffered by marginal adaptive weighting under correlated or duplicated rewards (Appendix G.2). We hope this addresses the query on advantages marginal weighting.

AuthorsOpenReview ↗

Isolating Decorrelation Under Nonlinear Aggregation

Question 2b: Isolating the Effect of Decorrelation With Nonlinear Welfare Objectives

Reviewer Query: Beyond linear Sum aggregation, the reviewer asked for experiments isolating the specific benefit of decorrelation when combined with nonlinear welfare aggregators (Nash and SoftMin).

"... Can the authors provide experiments that isolate the contribution of decorrelation when combined with nonlinear aggregation?"

Our Response (addressed in Sections 7.1-7.2, Table 3, and Appendix G.1-G.2, Tables 4-5 & Figure 12):

  1. New nonlinear isolation experiments conducted:
    • Paired Nash and SoftMin with Identity, Diagonal, Raw-Covariance, Batch-Covariance, and Correlation-Space (ZCA-cor) preconditioning across multi-seed tool calling, noisy K = 3 duplicate rewards, and mathematical reasoning.
  2. Stabilizing rare constraints under Nash: On 1.5B tool calling, diagonal Nash (Diag-MNW) suffers run-to-run threshold bistability (stalling flat in Figure 3), whereas Batch-Cov-Nash converges reliably in 18 steps and 3-seed Cov-Nash achieves the highest validation correctness across all methods (+1.77 +/- 0.02).
  3. Protecting rare rewards under noisy duplicates: On noisy K = 3 duplicates under Nash (ridge 0.01), raw covariance collapses (8.7% format), whereas correlation whitening (ZCA-cor + Nash) filters duplicate noise and restores 97.5% format compliance.
  4. Stronger compression under SoftMin: On mathematical reasoning (Table 3), adding covariance decorrelation to SoftMin shortens mean responses by 275 tokens compared to diagonal SoftMin (2,330 tokens vs 2,605 tokens).
Nonlinear Aggregator Setting (Appendix G.1-G.2)Diagonal or Uncorrelated BaselineWith GEOMA DecorrelationExplanation
Nash on 1.5B tool calling (K = 2, 3 seeds)Diag-MNW: bistable (stalled in Fig. 3)Batch-Cov-Nash: 18 steps; Cov-Nash: +1.77 cor.Decorrelation stabilizes rare-constraint learning
Nash on noisy K = 3 duplicates (ridge 0.01)Raw Cov + Nash: 8.7% formatZCA-cor + Nash: 97.5% formatRemoves duplicate noise without crushing rare format
SoftMin on math reasoning (Table 3)Diag-SoftMin: 2,605 tokensBatch-Cov-SoftMin: 2,330 tokensContracts positive-correlation axis by 275 tokens

Findings: When combined with nonlinear Nash and SoftMin, decorrelation removes rare-constraint bistability, restores 97.5% format compliance under noisy redundancy, and achieves the tightest response compression on mathematical reasoning (Appendix G.1 and G.2).

AuthorsOpenReview ↗

Novelty of Geometric Multi-Reward RLHF

Weakness 2b: Novelty of Addressing Multi-Reward RLHF Through Reward Geometry

Reviewer Query: The reviewer argued that because covariance preconditioning with Sum aggregation is algebraically equivalent to covariance-dependent reward reweighting, the method's novelty is limited.

"Limited novelty ... Under linear aggregation, covariance sphering is equivalent to covariance-dependent reward reweighting."

Our Response (addressed in Sections 1, 4, 5, 6, and Appendix G.1 to G.3):

We respectfully push back on viewing this equivalence as a limitation of novelty. Proving that linear aggregation after covariance whitening reduces to a closed-form, data-adaptive weighting operator (Remark 4.1) is our own theoretical insight that makes the method interpretable, not a reason to dismiss the method itself. Prior multi-reward RLHF methods rely on manual scalar weights, task-specific gates, or independent single-reward scaling (GDPO), ignoring the joint covariance geometry of rewards altogether. Introducing reward geometry into RLHF is a genuine methodological and theoretical contribution:

  1. A new geometric operator space for RLHF: No prior RLHF method preconditions advantages with the joint inverse-square-root covariance or correlation operator. GEOMA unifies GRPO and GDPO as special cases while introducing Batch-Cov, ZCA-cor, and exact-invariance C-Prec to eliminate multi-rubric double-counting without manual hyperparameter tuning.
  2. Beyond scalar reweighting under nonlinear welfare: As isolated in Part 3, GEOMA is a full 2D/KD operator-by-welfare family, not just Sum; under Nash and SoftMin, preconditioning reshapes the entire multi-objective frontier in ways no scalar reward weight can express.
  3. New alignment theory for RLHF: Composing data-dependent whitening with policy-gradient advantages creates coupled phenomena absent in standalone statistics; we prove exact Stieltjes Pareto-dominance guarantees, the universal 0.71 correlation floor, and correlation-space stability under near-singular redundancy (Appendix G.1).
DimensionPrior Multi-Reward RLHF (GRPO, GDPO, Adaptive Scalars)What GEOMA Introduces
Core MechanismScalar heuristics or marginal per-reward variance onlyJoint covariance and correlation geometry + welfare objectives
Linear RegimeAd-hoc or marginal weights that double-count duplicatesExact closed-form correlation-aware weighting (sqrt(m) and 1:1)
Nonlinear RegimeUnpreconditioned or bistable per-coordinate welfareFull geometric preconditioning for Nash and SoftMin frontiers
Alignment TheoryCoordinatewise assumptions onlyStieltjes Pareto theorem, 0.71 sign-flip floor, ZCA-cor stability

We are grateful for your positive score on our paper's soundness (3) and clear framework. Thank you again for your sharp questions on Pareto consistency, marginal baselines, and nonlinear isolation, all of which we addressed with new proofs and experiments. In case our new results have resolved your concerns, and address your questions on the paper novelty, we would be deeply grateful if you would consider raising your score toward acceptance.

Reviewer_2SuDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GEOMA, a framework for multi-reward alignment in reinforcement learning for large language models. The central idea is to decompose reward aggregation into two components: geometric preconditioning and econometric aggregation. For geometric preconditioning, the authors introduce covariance sphering, which uses the inverse square root of the reward covariance matrix to account for correlations between reward dimensions. This extends the diagonal normalization used in GDPO. For econometric aggregation, the paper considers three objectives: Sum, Nash Welfare, and SoftMin, which differ in how they balance competing reward dimensions.

The paper provides theoretical analyses of covariance whitening, reward redundancy, and the relationship between preconditioning and reward weighting. In particular, it establishes that covariance preconditioning followed by Sum aggregation is equivalent to applying GRPO with covariance-dependent reward weights.

Experiments are conducted on mathematical reasoning and tool-calling tasks using 1.5B-parameter language models. The results suggest that covariance-based aggregation can improve the trade-off between mathematical accuracy and response length compared with GDPO, while also improving the learning of formatting constraints in tool-calling tasks.

Critical strengths
  1. A clear framework for understanding multi-reward aggregation. The paper provides a useful decomposition of multi-reward optimization into geometric preconditioning and aggregation objectives. This distinction helps clarify two different issues: statistical dependencies between reward signals and the desired trade-offs among objectives.

    The theoretical observation that covariance preconditioning followed by Sum aggregation is equivalent to adaptive reward weighting is particularly valuable. It provides a concrete interpretation of what covariance sphering actually changes in the optimization process and helps distinguish the effects of linear and nonlinear aggregation.

  2. Promising improvements in balancing multiple objectives. The experiments suggest that covariance-based methods can achieve a more favorable balance between competing objectives. In mathematical reasoning, Batch-Covariance-Sum improves average accuracy from 46.8% to 48.4% compared with GDPO, while reducing average response length from 3,897 to 2,518 tokens.

    The tool-calling experiments also demonstrate that covariance-based configurations can learn formatting constraints that the standard GRPO baseline fails to learn.

    These observations suggest that accounting for reward dependencies may be useful in practical multi-objective reinforcement learning.

Critical weaknesses
  1. Missing comparisons with simple but strong reward composition baselines. My main concern is that the experimental evaluation does not establish whether the proposed covariance-based framework is necessary to achieve the reported improvements.

The paper motivates GEOMA by highlighting the limitations of simple linear reward aggregation, particularly when objectives conflict or when one objective must satisfy a strict constraint. However, these limitations can also be addressed using relatively straightforward reward engineering techniques, without requiring covariance estimation or matrix preconditioning.

For example, in the tool-calling experiments, where the objective is to maximize correctness while satisfying a binary formatting constraint, several natural alternatives should be considered:

  • Indicator-based gating: (R=r_{\mathrm{correct}}\cdot\mathbb{1}(r_{\mathrm{format}}=1)).
  • Constraint-violation penalties: (R=r_{\mathrm{correct}}-\lambda\mathbb{1}(r_{\mathrm{format}}=0)).
  • Piecewise reward functions: Assigning different reward structures depending on whether the formatting constraint is satisfied.
  • Nonlinear or multiplicative reward composition: Combining correctness and formatting rewards using products or other simple nonlinear transformations. These are natural baselines because they directly encode the desired relationship between the objectives. Moreover, indicator functions and piecewise reward definitions are compatible with GRPO, since the reward function itself does not need to be differentiable.

Similarly, for mathematical reasoning, a simple correctness reward combined with a threshold-based length penalty could be compared against covariance-based aggregation, with the penalty strength appropriately tuned.

The paper does evaluate Sum, Nash Welfare, and SoftMin, but these comparisons do not fully address this concern. In particular, comparing different aggregation functions within the proposed framework is insufficient to demonstrate that covariance preconditioning is preferable to straightforward task-specific reward composition.

I do not assume that these simpler approaches would necessarily outperform GEOMA. Hard gating, for example, may reduce useful learning signals when all sampled responses violate a constraint. Nevertheless, this is precisely why an empirical comparison is necessary.

Without such baselines, it remains unclear whether the reported improvements arise from a genuinely useful treatment of reward covariance or simply from choosing a more suitable effective reward function.

This limitation substantially weakens the experimental evidence supporting the paper's central contribution.

  1. Limited experimental robustness and insufficient evidence of generalization. The experimental evaluation is relatively limited in both model scale and statistical reliability.

All primary experiments are conducted using 1.5B-parameter models, with only 100 training steps for tool calling and 200 steps for mathematical reasoning. More importantly, each configuration is trained only once.

Although the mathematical reasoning results report standard deviations over multiple generations, these do not measure variability across independent training runs.

This distinction is especially important because some of the reported improvements are relatively small. For instance, the best covariance-based configuration achieves 45.28% average accuracy on BFCL, compared with 44.71% for GRPO. Without multiple training seeds, it is difficult to determine whether this difference is statistically reliable.

Furthermore, covariance whitening introduces an additional sensitivity to the estimated reward covariance matrix. Directions associated with small eigenvalues receive greater amplification, potentially increasing sensitivity to estimation noise, especially when reward dimensions are strongly correlated or the group size is small.

Although the paper discusses ridge regularization and batch-level covariance estimation, it does not provide sufficient empirical evidence about sensitivity to covariance estimation errors, regularization strength, or changing reward correlation structures.

Additional experiments with multiple random seeds, larger models, and covariance estimation ablations would significantly strengthen the paper.

Questions for authors
  1. Comparison with explicit reward constraints and nonlinear composition (critical). The paper argues that linear reward aggregation is insufficient when objectives conflict or when strict constraints must be satisfied. Have the authors evaluated simpler alternatives such as indicator-based reward gating, explicit constraint-violation penalties, piecewise reward functions, or multiplicative reward composition? For the tool-calling task, why would covariance-based aggregation be preferable to directly penalizing or gating responses that violate the required format? For mathematical reasoning, how does GEOMA compare with a tuned correctness-plus-length-penalty reward? I consider this comparison important for establishing the practical necessity and significance of the proposed method.
  2. Is covariance sphering beneficial beyond adaptive reward weighting? (critical clarification). Remark 4.1 establishes that covariance preconditioning with Sum aggregation is equivalent to GRPO with reward weights determined by the covariance matrix. Given that some of the strongest mathematical reasoning results are obtained using Sum aggregation, can the authors clarify what experimental evidence demonstrates a benefit from covariance-based weighting compared with simpler adaptive reward weighting strategies? For example, would a tuned weighted sum or an adaptive weighting method based on marginal reward statistics achieve similar results? This distinction would help identify which component of GEOMA is responsible for the observed improvements.
  3. Statistical reliability of the experimental improvements (critical). The paper states that each configuration is trained once. Can the authors provide results across multiple independent training seeds, particularly for the BFCL experiments where the improvements are relatively small? Are the observed differences consistent across runs?
  4. Sensitivity to covariance estimation and regularization (clarification). How sensitive is the proposed method to the covariance estimation strategy, group size, and regularization parameter? In particular, when the reward covariance matrix is nearly singular, how do the authors prevent the inverse-square-root transformation from amplifying noise along low-variance directions? An ablation of these factors would help establish the stability of the method.
4 comments
AuthorsOpenReview ↗

Addressing thorough requests for experiments

We thank Reviewer 2SuD for the thorough review and concrete experimental checklist. We implemented and ran every baseline and ablation you suggested across Appendix G (Parts 1 to 4 below).

Strengths Highlighted by Reviewer 2SuD

  1. Clear framework for multi-reward RL: Separating geometric preconditioning from aggregation objectives cleanly distinguishes statistical dependencies between rewards from desired objective trade-offs.
  2. Valuable theoretical equivalence: Establishing that covariance preconditioning with Sum acts as covariance-dependent reward weighting gives a concrete optimization interpretation and separates linear from nonlinear aggregation.
  3. Promising multi-objective improvements: Batch-Cov-Sum improves math accuracy from 46.8% to 48.4% while cutting length from 3,897 to 2,518 tokens over GDPO, and learns strict tool-calling format rules that GRPO fails to learn.

Weakness 1 & Question 1: Addressing task specific reward composition rules

Reviewer Query: The reviewer asked whether simpler task-specific reward composition (indicator gating, constraint-violation penalties, piecewise rules, multiplicative composition on tool calling, and tuned length penalties on math) matches or beats GEOMA.

"Have the authors evaluated simpler alternatives such as indicator-based reward gating, explicit constraint-violation penalties, piecewise reward functions, or multiplicative reward composition? ... For mathematical reasoning, how does GEOMA compare with a tuned correctness-plus-length-penalty reward?"

Our Response (addressed in Sections 7.1-7.2 and Appendix G.3, Tables 8-9 & Figures 18-19):

  1. New experiments conducted across every baseline you named:
    • Tool calling (Qwen2.5-1.5B, Table 8): 7 GRPO runs testing indicator gating, piecewise lexicographic gating, soft multiplicative gating, and violation penalties (lambda = 3, 6).
    • Mathematical reasoning (DeepSeek-R1-1.5B, Table 9): 27 five-seed evaluations sweeping tuned length weights alpha in {0.00, 0.50, 0.65, 0.80} alongside GDPO and GEOMA (clarifying Table 2's GRPO row as the alpha = 0.00 accuracy-only anchor).
  2. Why hand-crafted tool-calling rules struggle without manual bound engineering: Because failed tool calls score negatively (-3 to +3), raw indicator multiplication assigns 0 to unformatted errors and negative rewards to formatted errors, penalizing formatting. Fixed violation penalties (lambda = 3, 6) also fail (< 8% format) because rare-format variance stays submerged below correctness variance. Manually shifting correctness to [0, 1] lets hard and piecewise gates match GEOMA's 15-step format speed, but zeroes out correctness signals in 97% of early groups (precisely the failure mode you anticipated) and trails GEOMA on Multi-Turn BFCL-v3.
  3. Why tuned math weights lose calibration: Early context truncation (8,192 tokens) creates +0.44 reward correlation that decays near zero once responses shorten, causing static weights to oscillate late in training and stall above 3,380 tokens while Batch-Cov-Sum reaches 3,135 tokens by step 150.
Explicit Composition Baseline (Appendix G.3, Tables 8-9)Format Speed or Math Acc / LengthExplanation
Unshifted indicator gate: r_cor * 1(fmt = 1)Penalizes format on errors (0 > -3)Fails on signed rewards without manual bound shifting
Violation penalties (lambda = 3 and lambda = 6)Collapsed (7.5% and 6.2% format)Rare-format variance stays smaller than correctness variance
Bound-shifted indicator & piecewise lexicographic gates16 steps; 2.63%-2.75% Multi-Turn BFCLZeroes correctness in 97% of early groups; trails GEOMA
Soft multiplicative gate: r_cor * (0.1 + 0.9 * fmt)20 steps; 2.88% Multi-Turn BFCLSlower format learning and lower Multi-Turn than GEOMA
GEOMA Batch-Cov-Sum on tool calling (zero tuning)15.0 +/- 1.0 steps; 3.33% Multi-TurnMatches gated speed without bound tuning; leads Multi-Turn
Tuned static math weights vs GEOMA Batch-Cov-Sum46.7%-48.4% (3,476-3,574t) vs 48.37% (3,135t, step 150)Same best accuracy with ~340-440 fewer tokens

Findings: Explicit gating and penalties break on signed or rare rewards without manual bound engineering, and static math weights drift out of calibration during training; GEOMA matches or beats all of them without task-specific tuning (Appendix G.3).

AuthorsOpenReview ↗

Benefit Over Marginal Adaptive Reward Weighting

Question 2: Covariance Preconditioning Versus Marginal Adaptive Weighting

Reviewer Query: Given that covariance preconditioning with Sum acts as covariance-dependent reward weighting, the reviewer asked what experimental evidence shows a benefit over simpler adaptive weighting based only on marginal reward statistics.

"Given that some of the strongest mathematical reasoning results are obtained using Sum aggregation, can the authors clarify what experimental evidence demonstrates a benefit from covariance-based weighting compared with simpler adaptive reward weighting strategies? For example, would a tuned weighted sum or an adaptive weighting method based on marginal reward statistics achieve similar results?"

Our Response (addressed in Section 5, Sections 7.1-7.2, and Appendix G.2-G.3, Tables 6 & 9):

  1. New marginal-control experiments conducted:
    • Evaluated a scope-matched batch marginal variance baseline (Batch-Diag-Sum, scaling each reward by its batch standard deviation while ignoring off-diagonal correlations) head-to-head against Batch-Cov-Sum and exact-invariance C-Prec-Sum across 1.5B, Qwen2.5-7B-Instruct, K = 3 and K = 5 duplicated rewards, and mathematical reasoning.
  2. When marginal and covariance agree: With 2 initially uncorrelated rewards on 1.5B tool calling, the initial covariance matrix is nearly diagonal, so Batch-Cov-Sum and Batch-Diag-Sum converge at a similar step while Batch-Cov-Sum halves cross-seed SD (+/- 1.0 vs +/- 2.1).
  3. Where off-diagonal covariance clearly wins: Whenever rewards are correlated or redundant, marginal adaptive methods ignore off-diagonal terms and weight m correlated copies at m : 1. Off-diagonal preconditioning prevents duplicate over-counting (K = 3, 5), avoids a sharp early correctness drop at 7B (K = 2), and is 0.3 points above Batch-Diag-Sum on mathematical reasoning at step 100 (47.34% vs 47.04%).
Controlled Comparison (Appendix G.2-G.3)GDPO (Per-Group Marginal)Batch-Diag-Sum (Batch Marginal)GEOMA Batch-Cov-SumGEOMA C-Prec-SumExplanation
1.5B tool, K = 3 (2 copies + format)Collapsed20 steps16 steps16 stepsMarginal scaling double-counts correctness; GDPO fails
1.5B tool, K = 5 (4 copies + format)Collapsed29 steps20 steps16 steps4x over-counting delays Batch-Diag; C-Prec has zero slowdown
7B tool, K = 2 (step-10 correctness)+1.85+0.29+1.71Matches Batch-DiagOff-diagonal terms prevent a 1.42-point early accuracy dip
7B tool, K = 3 (2 copies + format)25 steps20 steps16 steps22 steps (+1.88 cor.)Batch-Cov beats GDPO by 9 steps; C-Prec tops correctness
1.5B math (5-seed eval acc / tokens)46.8% / 3,897t47.04% / 3,100t (s100)48.37% / 3,135t (s150); 48.4% / 2,518t (s200)Matches Batch-Diag at K=2Truncation correlation (+0.44) drives better accuracy-length balance

Findings: Marginal adaptive weighting suffices only when rewards are uncorrelated; under correlated rewards (7B tool calling and math truncation) or redundant rubrics (K = 3, 5), off-diagonal covariance preconditioning consistently outperforms marginal scaling (Appendix G.2 and G.3).

AuthorsOpenReview ↗

Multi-Seed Runs + 7B Scale-Up

Weakness 2a & Question 3: Multi-Seed Training Variance, BFCL-v3 Analysis, and 7B Scale

Reviewer Query: Because each configuration was originally trained once on 1.5B models, the reviewer asked for multiple independent training seeds (especially on BFCL where single-run differences were small) alongside larger-scale validation.

"Can the authors provide results across multiple independent training seeds, particularly for the BFCL experiments where the improvements are relatively small? Are the observed differences consistent across runs?"

Our Response (addressed in Abstract, Sections 1, 7.1, 8, and Appendix G.2-G.4, Tables 5-6 & Figures 13-16):

  1. New multi-seed and 7B experiments conducted:
    • 3 independent training seeds (42, 101, 202) across 5 methods on 1.5B tool calling, evaluated across 32 BFCL-v3 runs (Table 5).
    • 8 RL runs scaling up 4.7x to Qwen2.5-7B-Instruct across K = 2 and K = 3 on 8x H100 GPUs (Table 6).
    • Independent training seed of Batch-Cov-Sum on math plus 27 five-seed evaluations separating training SD from decoding SD (Table 9).
  2. Consistent cross-seed gains on constraint speed and Multi-Turn BFCL-v3: Across 3 training seeds, Batch-Cov-Sum cuts format-learning variation by 13x over GDPO (15.0 +/- 1.0 vs 26.3 +/- 13.6 steps) and consistently leads Multi-Turn BFCL-v3 (3.33 +/- 0.26% vs 2.80 +/- 0.14% GDPO and 2.92 +/- 0.14% GRPO).
  3. Transparent update on single-turn BFCL-v3: Auditing BFCL-v3 showed that single-turn prompts evaluate Qwen2.5 in its native tool syntax rather than our trained XML schema, compressing 3-seed single-turn averages within 1 point (46.24% to 47.17%). We therefore remove all single-turn BFCL superiority claims from the abstract, introduction, Section 7.1, and conclusion.
Multi-Seed & 7B Summary (Appendix G.2, Tables 5-6)Steps to 85% FormatFinal Format ComplianceMulti-Turn BFCL-v3 or Final Cor.Explanation
1.5B GEOMA Batch-Cov-Sum (3 training seeds)15.0 +/- 1.097.5 +/- 0.0%3.33 +/- 0.26% Multi-TurnLowest cross-seed variance and highest Multi-Turn
1.5B GDPO baseline (3 training seeds)26.3 +/- 13.697.1 +/- 0.7%2.80 +/- 0.14% Multi-TurnSeed 202 stalls until step 42 on rare format
1.5B Cov-Nash (3 training seeds)37.7 +/- 11.696.7 +/- 1.4%2.83 +/- 0.14% Multi-TurnHighest 3-seed validation correctness (+1.77)
1.5B GRPO baseline (3 training seeds)> 42 steps38.3 +/- 51.2%2.92 +/- 0.14% Multi-TurnFails to learn format on 2 of 3 seeds by step 50
7B GRPO vs 7B GEOMA Batch-Cov-Sum (K = 2, 3)Never vs 16-17 steps11.3% vs 100.0%+1.91 (unformatted) vs +1.89 / +1.86Gains hold cleanly at 7B parameter scale

Findings: Multi-seed training confirms 13x lower convergence variance and consistent Multi-Turn BFCL-v3 gains, 7B scaling confirms full generalization, and single-turn BFCL claims are aligned strictly with multi-seed evidence (Appendix G.2).

AuthorsOpenReview ↗

Near-Singular Conditioning

Weakness 2b & Question 4: Sensitivity to Estimation Strategy, Group Size, and Near-Singular Covariance

Reviewer Query: The reviewer asked how sensitive GEOMA is to group size, regularization, and covariance estimation noise, and how we prevent the inverse square root from amplifying noise or crushing rare signals when rewards are nearly collinear.

"How sensitive is the proposed method to the covariance estimation strategy, group size, and regularization parameter? In particular, when the reward covariance matrix is nearly singular, how do the authors prevent the inverse-square-root transformation from amplifying noise along low-variance directions?" Our Response (addressed in Section 5, Appendix G.1 Table 4 & Figures 11-12, and Appendix G.2 Table 7):

  1. New conditioning and group-size ablations conducted:
    • 9 RL runs on Qwen2.5-1.5B-Instruct under noisy (eta = 0.05) and 3x-rescaled near-singular K = 3 duplicate rewards across ridge (1e-6, 1e-2) and eigenvalue thresholding (tau = 1e-2) (Table 4, Figure 12).
    • 400-trial covariance error study across group sizes G in {4, 8, 16, 32, 64} comparing Group-Cov, Batch-Cov, and shrinkage (Table 7, Figure 17).
  2. Batch estimation eliminates small-group noise: At standard RLHF group sizes (G = 4, 8), batch pooling cuts Frobenius estimation error by 8x to 10x (1.70 -> 0.17 at G = 4) and produces 0% negative-weight steps across all 1.5B, 7B, and math RL runs.
  3. Correlation-space whitening (ZCA-cor) resolves near-singular sensitivity: Under raw covariance whitening, raising ridge or eigenvalue threshold to 0.01 to damp duplicate noise inadvertently crushes the low-variance rare format signal (< 9% compliance), while 3x rescaling causes O(ridge^-1/2) weight blowup (-199.7). Normalizing marginal variances to 1 before whitening the correlation matrix (ZCA-cor) keeps the rare format diagonal at 1.0 while cleanly damping collinear duplicate noise. | Near-Singular Test on 1.5B (Appendix G.1, Table 4) | Raw Covariance Whitening | GEOMA With ZCA-cor | Explanation | | :--- | :--- | :--- | :--- | | Noisy duplicates (eta = 0.05, Nash, ridge = 0.01) | Collapsed (8.7% format) | 97.5% format (+1.72 cor.) | Damps duplicate noise while keeping format diagonal at 1.0 | | Noisy duplicates (eta = 0.05, Nash, pinv tau = 0.01) | Collapsed (1.3% format) | 97.5% format (+1.73 cor.) | Thresholding in correlation space protects rare constraints | | Exact 3x rescaled duplicate (Sum, ridge = 1e-6) | Weights (+600.1, -199.7, +11.8) | Weights (+0.52, +0.17, +10.9) | Correlation nullspace is orthogonal to 1, preventing blowup | | Noisy 3x rescaled duplicate (Sum, ridge = 1e-6) | Weights (+37.8, -12.3); cor. step 27 | Weights (+0.52, +0.17); cor. step 23 | Bounded positive weights speed up correctness by 4 steps | Findings: Batch pooling removes small-group estimation noise, and correlation-space whitening (ZCA-cor) restores 97.5% constraint compliance and bounded positive weights across near-singular and differently scaled rewards (Appendix G.1).

Thank you for your note that GEOMA's decomposition and adaptive-weighting equivalence are valuable. Thank you for laying out such a clear, thorough experimental roadmap: we implemented every baseline and ablation you requested (indicator gating, violation penalties, piecewise and multiplicative rules, tuned math length weights, marginal adaptive controls, multi-seed runs, 7B scaling, and near-singular conditioning). If these new results, across 30+ new experiments, have addressed your concerns, we would be deeply grateful if you would consider raising your score toward acceptance.

Reviewer_K2v8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GEOMA, a framework for multi-reward alignment that splits reward aggregation into two steps: geometric preconditioning (transforming the reward space before combining dimensions) and econometric aggregation (choosing how to combine them). GRPO and GDPO are presented as special cases. The authors' main proposal, covariance sphering, decorrelates reward dimensions in addition to normalizing their variances. They test on math reasoning and tool-calling tasks with 1.5B models, reporting modest accuracy gains alongside shorter responses compared to GDPO.

Critical strengths
  1. Clean unifying framework. Framing GRPO and GDPO as special cases of a preconditioner × aggregator design space gives a useful map of the multi-reward alignment landscape.
  2. Practical improvement with clear mechanism. Covariance sphering simultaneously improves math accuracy and shortens responses, and enables learning format constraints on tool calling that GRPO misses.
Critical weaknesses
  1. Experiments are too limited. Single runs on 1.5B models with ~1.6% accuracy gains do not provide convincing evidence of a robust improvement. No multi-seed results or larger-scale validation.
  2. Novelty is incremental. Covariance sphering, power means, and Nash welfare are standard tools. The contribution is applying them to RLHF reward aggregation, not introducing new theory.
Questions for authors
  1. Have you tested on larger models (7B+)? Does the advantage persist?
  2. What is the variance across seeds for your main comparison?
  3. Is the shorter response length purely from re-weighting rewards, or does it interact more deeply with the preconditioned geometry?
  4. When would group-wise covariance estimation be preferable to batch-wise?
4 comments
AuthorsOpenReview ↗

7B Scale-Up and Multi-Seed Variance

We thank Reviewer K2v8 for highlighting the clarity of GEOMA's framework and its practical gains on math reasoning and tool calling. Below we address 7B model scaling and multi-seed variance (Part 1), followed by the math length mechanism, group-versus-batch estimation, and theoretical novelty (Part 2). Your highlight that the mechanism is very clear was a significant highlight of the rebuttal for us! thank you.


Weakness 1 & Question 1: Scaling Up to Qwen2.5-7B-Instruct

Reviewer Query: The reviewer asked whether GEOMA's advantages persist beyond 1.5B models when tested at the 7B+ scale.

"Experiments are too limited. Single runs on 1.5B models ... Have you tested on larger models (7B+)? Does the advantage persist?"

Our Response (addressed in Section 7.1 and Appendix G.2):

  1. New 7B experiments conducted:
    • 8 RL training runs on Qwen2.5-7B-Instruct (8x H100 GPUs, batch size 512) testing both standard 2-reward tool calling and a 3-reward duplication setting (2 copies of correctness + 1 format constraint).
  2. Unnormalized GRPO still fails to learn formatting at 7B: Even with a much stronger 7B base model, GRPO stays at 11.3% format compliance through training, whereas covariance preconditioning reaches 100% validation format compliance.
  3. Off-diagonal covariance protects correctness and handles redundancy at 7B: On 2 rewards, Batch-Cov-Sum avoids the sharp early correctness drop suffered by our marginal Batch-Diag-Sum ablation (+1.71 vs +0.29 at step 10). Under duplicated correctness (K = 3), GDPO slows by 12 steps, whereas Batch-Cov-Sum converges fastest (9 steps ahead of GDPO) and exact-invariance C-Prec-Sum achieves the highest peak (+1.92) and final (+1.88) correctness.
Qwen2.5-7B-Instruct Run (Appendix G.2)Steps to 85% FormatStep-10 CorrectnessFinal CorrectnessExplanation
GRPO (2 rewards)Never (11.3% final)+1.86+1.91 (unformatted)High-variance correctness drowns out rare format signal even at 7B
GDPO (2 rewards)13 steps+1.85+1.90Fast at K = 2, but sensitive to reward duplication below
Batch-Diag-Sum ablation (2 rewards)20 steps+0.29+1.87Ignoring post-transition correlation causes a steep early correctness dip
GEOMA Batch-Cov-Sum (2 rewards)17 steps+1.71+1.89Off-diagonal correlation prevents the step-10 correctness drop
GDPO (3 rewards, 2 correctness copies)25 steps (+12 delay)+0.96+1.83Double-counting correctness slows format acquisition by nearly 2x
Batch-Diag-Sum ablation (3 rewards)20 steps+1.78+1.85Batch pooling helps, but still over-weights duplicated correctness 2:1
GEOMA Batch-Cov-Sum (3 rewards)16 steps+1.75+1.86 (peak +1.91)Fastest convergence at K = 3 (9 steps ahead of GDPO)
GEOMA C-Prec-Sum (3 rewards)22 steps-0.80+1.88 (peak +1.92)Exact 1:1 duplicate weight yields highest peak and final correctness

Findings: At the 7B scale, unnormalized GRPO still misses the format constraint, while GEOMA prevents early correctness degradation at K = 2 and converges up to 9 steps faster than GDPO under reward duplication (Appendix G.2).


Question 2: Variance Across Independent Training Seeds

Reviewer Query: The reviewer asked for the variance across independent training seeds on our main comparisons.

"What is the variance across seeds for your main comparison?"

Our Response (addressed in Sections 7.1 and 7.2, Appendix G.2, and Appendix G.3):

  1. New multi-seed experiments conducted:
    • Tool calling (1.5B): 3 independent training seeds (42, 101, 202) across 5 methods, evaluated on BFCL-v3.
    • Mathematical reasoning (1.5B): Independent training seed of Batch-Cov-Sum plus 27 five-seed benchmark evaluations separating training SD from decoding SD.
  2. Order-of-magnitude lower seed variance: Batch-Cov-Sum reaches 85% format compliance in 15.0 +/- 1.0 steps versus 26.3 +/- 13.6 steps for GDPO, and leads Multi-Turn BFCL-v3 (3.33 +/- 0.26% vs 2.80 +/- 0.14%). Single-turn BFCL-v3 uses Qwen's native tool syntax and spans under 1 point across seeds, so we drop single-turn superiority claims.
  3. Reproducible math compression: On an independent training seed, Batch-Cov-Sum climbs smoothly to 48.37% mean accuracy at 3,135 tokens (step 150), outperforming all late-stage static scalar weights (3,476 to 3,574 tokens) and matching Table 2's full-horizon trend (48.4% at 2,518 tokens).

Findings: Across independent training seeds, GEOMA reduces constraint-learning step standard deviation by 13x over GDPO (+/- 1.0 vs +/- 13.6 steps) and reproduces higher accuracy at shorter length on mathematical reasoning (Appendix G.2 and G.3).

AuthorsOpenReview ↗

Math Length Mechanism, Group vs Batch Scope, and Theory

Question 3: Mechanism Behind Shorter Math Responses

Reviewer Query: The reviewer asked whether shorter math responses come purely from adaptive reward reweighting or from a deeper interaction with the preconditioned reward geometry.

"Is the shorter response length purely from re-weighting rewards, or does it interact more deeply with the preconditioned geometry?"

Our Response (addressed in Remark 4.1, Section 7.2, Appendix G.3, and Appendix G.4):

  1. Linear vs nonlinear geometry: Under Sum aggregation, Batch-Cov-Sum acts via online covariance-dependent reward weights (Remark 4.1). Deeper 2D preconditioned geometry enters when whitening is paired with nonlinear aggregators (Nash and SoftMin), which compress responses even further to 2,330 tokens in Table 3.
  2. Why online covariance reweighting beats static weights: Logging 4,096 rollouts per step shows that 34% of initial math responses hit the 8,192-token limit (scoring 0 on both accuracy and length), creating +0.44 positive correlation. Positive correlation automatically boosts length pressure early, then relaxes toward marginal scaling once truncation drops to 0% below 2,000 tokens.
  3. Head-to-head against tuned static weights: Because fixed weights cannot adapt as correlation decays to zero, tuned static weights (alpha = 0.50, 0.65, 0.80) oscillate late in training and stall above 3,380 tokens, whereas Batch-Cov-Sum compresses smoothly to 3,135 tokens (step 150) and 2,518 tokens (step 200) at 48.4% accuracy.

Findings: Sum aggregation shortens responses via an automatic correlation-driven weight schedule induced by early context truncation, while nonlinear aggregators exploit the full 2D whitened space for stronger compression (Appendix G.3).


Question 4: When Group-Wise Covariance Is Preferable to Batch-Wise

Reviewer Query: The reviewer asked under what conditions per-prompt-group covariance estimation is preferable to batch-wide estimation.

"When would group-wise covariance estimation be preferable to batch-wise?"

Our Response (addressed in Section 5 and Appendix G.2):

  1. New 400-trial bias-variance simulation across group sizes G in {4, 8, 16, 32, 64}: We tested homogeneous prompts against heterogeneous multi-domain mixtures where half the prompts have +0.8 reward correlation and half have -0.8 correlation (Simpson's paradox).
  2. Standard small-group RLHF (G <= 8) favors Batch-Cov: At G = 4 or 8, per-group covariance is noisy and rank-limited (rank <= G - 1), yielding 8x to 10x higher Frobenius estimation error than Batch-Cov and frequent negative sample weights.
  3. Large-group heterogeneous mixtures (G >= 16) favor Group-Cov: When opposite prompt-level correlations cancel out in the batch average, Group-Cov overtakes Batch-Cov once G >= 16, while shrinking Group-Cov toward Batch-Cov achieves the lowest error at G = 4 and 8.
Rollout Group Size G (Appendix G.2)Homogeneous Error: Group-Cov vs Batch-CovHeterogeneous (+/- 0.8) Error: Group-Cov vs Batch-CovRecommended Estimator
G = 4 (tool-calling budget)1.70 vs 0.171.10 vs 0.44 (Shrinkage: 0.37)Batch-Cov (or shrinkage)
G = 8 (math budget)0.59 vs 0.080.40 vs 0.47 (Shrinkage: 0.34)Batch-Cov (or shrinkage)
G = 160.29 vs 0.040.22 vs 0.48Group-Cov under prompt heterogeneity
G = 320.17 vs 0.020.13 vs 0.49Group-Cov under prompt heterogeneity

Findings: Batch-Cov is strictly preferable at standard RLHF group sizes (G <= 8), whereas Group-Cov becomes preferable at G >= 16 when diverse prompt domains exhibit opposite reward correlations (Appendix G.2).

AuthorsOpenReview ↗

Novelty of Geometric Multi-Reward Alignment

Weakness 2: Novelty of Addressing Multi-Reward RLHF Through Reward Geometry

Reviewer Query: The reviewer noted that covariance whitening, power means, and Nash welfare are established mathematical tools, questioning whether GEOMA introduces new methodology and theory beyond applying standard primitives.

"Novelty is incremental. Covariance sphering, power means, and Nash welfare are standard tools. The contribution is applying them to RLHF reward aggregation, not introducing new theory."

Our Response (addressed in Sections 1, 5, 6, and Appendix G.1 to G.3):

We view bringing joint reward geometry into multi-reward alignment as a genuine novelty and a core strength of our work, rather than a weakness. Prior multi-reward RLHF methods combine rewards using manual weights, ad-hoc gates, or single-reward scaling (GDPO), ignoring the geometric structure between rewards altogether. Grounding RLHF in clean geometric and welfare principles yields three new contributions:

  1. First geometric framework for multi-reward RLHF: GEOMA replaces ad-hoc reward tuning with joint covariance preconditioning paired with social-choice aggregators, unifying GRPO and GDPO while automatically handling redundant rewards and rare constraints.
  2. New alignment theory for preconditioned RLHF (Section 6 and Appendix G.1): Composing data-dependent whitening with policy-gradient advantages creates coupled effects absent in standalone statistics; we prove exact guarantees for when reward priorities are preserved and how near-duplicate rewards stay stable.
  3. Explaining shifting reward trade-offs in reasoning RL (Appendix G.3): Viewing rewards geometrically uncovers why static weights fail in reasoning RL, as early context truncation creates temporary reward correlations that GEOMA adapts to automatically online.
DimensionPrior Multi-Reward RLHF (GRPO, GDPO, Static Weights)What GEOMA Introduces
Reward ModelingTreats rewards independently or with manual scalar weightsModels the joint covariance geometry of the reward vector
Redundant RewardsDouble-counts correlated rewards and drops rare constraintsAutomatically discounts shared variance without weight tuning
Alignment GuaranteesAnalyzes single scalar or independent coordinates onlyProves exact Pareto-priority and stability guarantees for RLHF
Training DynamicsFixed weights drift out of calibration during trainingAdapts online as correlations shift over training

Thank you for your note that our unifying framework and practical mechanism are clear, which was a genuine highlight of the review process for us. Thank you again for your thoughtful questions, which pushed us to validate GEOMA at 7B scale and across multiple seeds. If our new 7B experiments, multi-seed analyses, and theoretical results have addressed your concerns, we would be very grateful if you would consider raising your score.

Thanks for your extra experiment.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Overview of Revisions and New Experiments

We thank the Reviewers WCFe, 2SuD, 9iLr, and K2v8 for their constructive feedback and careful engagement with our work. Your questions helped us sharpen our theoretical guarantees and expand our empirical evaluation across seeds, reward structures, and model scales.

Strengths Highlighted by the Reviewers

  1. Clean unifying design space: Separating statistical reward dependencies (geometric preconditioning) from objective trade-offs (econometric aggregation) gives a systematic map of multi-reward alignment with GRPO and GDPO as special cases (WCFe, 2SuD, 9iLr, K2v8).
  2. Clear presentation and precise claims: All four reviewers rated the paper clear and easy to follow (WCFe, 2SuD, 9iLr, K2v8), and commended the paper for carefully distinguishing algebraic properties of whitening from claims about policy behavior (WCFe).
  3. Concrete adaptive-weighting equivalence: Showing that covariance preconditioning with Sum aggregation acts as a covariance-dependent reward-weighting rule clarifies what whitening changes in optimization and separates linear from nonlinear aggregation (WCFe, 2SuD, 9iLr).
  4. Better math accuracy and length balance: Covariance preconditioning retains or improves GDPO's accuracy across all four math benchmarks while substantially shortening responses, and transparently exposes the multi-objective trade-off alongside GRPO (WCFe, 2SuD, 9iLr, K2v8).
  5. Reliable constraint learning on tool calling: Covariance-based methods learn strict binary formatting constraints on tool calling that standard GRPO fails to learn (2SuD, 9iLr, K2v8).

New Analyses and Experiments Conducted for the Rebuttal

  1. Original-reward monotonicity and stability proofs: We proved exact conditions showing when covariance preconditioning preserves each original reward's priority across Sum, Nash, and SoftMin, and analyzed correlation-space whitening (ZCA-cor) for nearly redundant rewards.
  2. Multi-seed and 7B tool-calling suite: We ran 51 RL training runs across Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct, paired with 32 BFCL-v3 evaluations, to test multiple random seeds and batch marginal variance baselines.
  3. Redundancy and conditioning ablations: We tested controlled 3-reward and 5-reward duplication ladders alongside noisy and rescaled near-singular reward configurations.
  4. Explicit reward-shaping and tuned static-weight sweeps: We evaluated indicator gating and violation penalties on tool calling, and ran 27 five-seed benchmark evaluations on DeepSeek-R1-Distill-Qwen-1.5B comparing GEOMA against tuned static scalar weights across training horizons.

Summary of Main Findings

  • Conflicting objectives always retain non-negative original-reward weights under covariance preconditioning for any number of rewards, and every training step across our 1.5B, 7B, and math RL runs stays well inside this monotone regime with zero negative weights.
  • Pooling reward covariance across the batch reduces cross-seed format-learning variation from roughly 14 steps under GDPO down to 1 step, and prevents the slowdowns and collapses suffered by marginal normalization under duplicated rewards and at 7B scale.
  • Without manual reward-bound shifting or weight sweeps, Batch-Cov-Sum matches engineered format gates on tool calling and achieves higher math accuracy in shorter responses than tuned static weights, compressing outputs to roughly 2,500 tokens over the full horizon.
AuthorsOpenReview ↗

Reward Monotonicity and Near-Singular Conditioning (Reviewers WCFe, 9iLr, 2SuD, K2v8)

Reviewer Query: Because GEOMA decorrelates rewards before combining them, the reviewers asked: (1) Can decorrelation accidentally flip an original reward's weight negative? (2) When rewards are nearly identical copies or on different scales, does matrix inversion amplify noise or crush rare rewards?

WCFe: "Under what conditions is G(P(r - mu)) monotone in each original reward? Can you address the covariance counterexample above ..."

9iLr: "Since covariance sphering mixes the original reward coordinates ... What guarantees that the resulting optimization remains consistent with the original alignment objectives?"

2SuD: "... when the reward covariance matrix is nearly singular, how do the authors prevent the inverse-square-root transformation from amplifying noise along low-variance directions?"

Our Response (addressed in Sections 4 to 6 and Appendix G.1):

  1. New analyses and experiments conducted:
    • Closed-form proofs for when every original reward keeps a non-negative weight across Sum, Nash, and SoftMin (Appendix G.1).
    • Online correlation and weight logging across all 1.5B, 7B, and math RL runs.
    • 9 near-singular RL runs on Qwen2.5-1.5B testing noisy and 3x-rescaled duplicate rewards.
  2. Positive weights hold across all conflicting rewards and our RL runs: Whenever rewards conflict or are uncorrelated, every entry of the whitening matrix is non-negative for any number of objectives. Even for positively correlated pairs, weights stay positive below 0.71 correlation; across all our RL runs, batch correlation stays below 0.59 with zero negative weights.
  3. Correlation whitening (ZCA-cor) stabilizes near-duplicates: Normalizing each reward to unit variance before whitening decouples scale from correlation, keeping weights positive and preventing regularization from crushing rare format constraints.
Scenario Tested (Appendix G.1)Raw Covariance WhiteningGEOMA With ZCA-corExplanation
Conflicting or uncorrelated rewardsPositive weights guaranteedPositive weights guaranteedStieltjes matrix structure preserves Pareto priority
Logged RL batches (1.5B, 7B, math)Max correlation 0.59 (< 0.71)0% negative-weight stepsAll empirical runs stay inside the monotone zone
Reviewer WCFe 2x2 example (corr. 0.97)Weights (+2.63, -0.43)Weights (+0.71, +0.24)Unit-variance scaling first removes the sign flip
Noisy duplicate rewards (ridge 0.01)8.7% format compliance97.5% format complianceRegularizing correlation protects rare format rewards
3x rescaled duplicate rewardWeights (+600.1, -199.7)Weights (+0.52, +0.17)Invariant to arbitrary reward scaling

Findings: Covariance preconditioning preserves original-reward priority whenever objectives conflict or pairwise correlation stays below 0.71, holding at every step of our RL runs. Under near-duplicate or differently scaled rewards, ZCA-cor keeps weights small, positive, and stable (Appendix G.1).

AuthorsOpenReview ↗

Multi-Seed Reliability, Redundant Rewards, and 7B Scale-Up (Reviewers WCFe, 2SuD, 9iLr, K2v8)

Reviewer Query: The reviewers asked whether GEOMA's gains hold across multiple training seeds and 7B scale, and what off-diagonal correlation adds over normalizing only each reward's own variance, especially when rewards are duplicated.

WCFe: "Do the gains persist across independent training seeds, and under a controlled three-reward experiment that duplicates one reward while retaining an independent third objective?"

2SuD: "... would a tuned weighted sum or an adaptive weighting method based on marginal reward statistics achieve similar results? ... Can the authors provide results across multiple independent training seeds ..."

K2v8: "Have you tested on larger models (7B+)? Does the advantage persist?"

Our Response (addressed in Section 7.1, Appendix G.2, and Appendix G.4):

  1. New experiments conducted:
    • 51 tool-calling RL runs across Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct, paired with 32 BFCL-v3 evaluations.
    • 3-seed replication on 2 rewards, plus 3-reward and 5-reward duplication ladders (2 or 4 copies of correctness + 1 format rule).
    • Ablations comparing per-group GDPO, our batch-marginal ablation (Batch-Diag-Sum), full Batch-Cov-Sum, and exact-invariance C-Prec-Sum.
  2. Multi-seed stability at 1.5B: Batch covariance pooling cuts cross-seed format variation from 26.3 +/- 13.6 steps (GDPO) to 15.0 +/- 1.0 steps and leads Multi-Turn BFCL-v3 (3.33% vs 2.80%). Because single-turn BFCL-v3 uses Qwen's native syntax and varies by under 1 point across seeds, we drop single-turn superiority claims.
  3. Off-diagonal gains under correlation and 7B scale: With 2 initially uncorrelated rewards at 1.5B, the covariance matrix is nearly diagonal so Batch-Cov matches our Batch-Diag ablation while halving seed SD. Once rewards are correlated or duplicated (at 7B or K = 3, 5), ignoring correlation over-counts duplicates and causes GDPO and Batch-Diag to stall, dip in correctness, or collapse.
Experiment Setting (Appendix G.2)GDPO BaselineBatch-Diag AblationGEOMA Batch-Cov-SumGEOMA C-Prec-SumExplanation
1.5B, 2 rewards (3 seeds, steps to 85% format)26.3 +/- 13.615.7 +/- 2.115.0 +/- 1.0Matches Batch-DiagNear-diagonal initial covariance; beats GDPO by 11 steps with 13x lower SD
1.5B, 3 rewards (2 copies of correctness)Collapsed20 steps16 steps16 stepsDouble-counting correctness causes GDPO to fail
1.5B, 5 rewards (4 copies of correctness)Collapsed29 steps20 steps16 steps4x over-weighting delays Batch-Diag; C-Prec has zero slowdown
7B, 2 rewards (step-10 correctness)+1.85+0.29+1.71Matches Batch-DiagOff-diagonal terms prevent a sharp early accuracy dip at 7B
7B, 3 rewards (2 copies, steps to 85% format)25 steps20 steps16 steps22 steps (+1.88 cor.)Batch-Cov converges 9 steps faster than GDPO; C-Prec tops correctness

Findings: Batch covariance preconditioning cuts cross-seed variance by an order of magnitude over GDPO and scales cleanly to 7B. Under correlated or duplicated rewards, off-diagonal preconditioning prevents the collapses, slowdowns, and early accuracy drops of variance-only scaling (Appendix G.2).

AuthorsOpenReview ↗

Explicit Reward Shaping and Static Scalar Weights on Math (Reviewers WCFe, 2SuD, K2v8)

Reviewer Query: We often combine rewards manually by gating correctness on format validity, subtracting a fixed format penalty, or tuning a static weight between math accuracy and length. The reviewers asked whether these simpler rules match GEOMA under the same training budget.

2SuD: "Have the authors evaluated simpler alternatives such as indicator-based reward gating, explicit constraint-violation penalties ... For mathematical reasoning, how does GEOMA compare with a tuned correctness-plus-length-penalty reward?"

WCFe: "How do the covariance-Sum variants compare with tuned nonnegative scalar weights under the same training budget ..."

Our Response (addressed in Sections 7.1 and 7.2, Appendix G.3, and Appendix G.4):

  1. New experiments conducted:
    • Tool calling (Qwen2.5-1.5B): 7 RL runs testing indicator gating, piecewise gating, multiplicative composition, and fixed format penalties.
    • Mathematical reasoning (DeepSeek-R1-1.5B): 27 five-seed evaluations comparing static length weights (0.00, 0.50, 0.65, 0.80) against GDPO and GEOMA (clarifying Table 2's GRPO row as the 0.00 accuracy-only anchor).
  2. Hand-crafted tool-calling rules require manual reward tuning: Fixed format penalties fail to learn formatting (< 8% compliance), and raw indicator gating penalizes formatting whenever correctness is negative. Even after manually shifting rewards to [0, 1], hard gating zeroes out correctness signals in 97% of early groups and trails GEOMA on Multi-Turn BFCL-v3 (2.75% vs 3.33%).
  3. Static math weights drift out of calibration during training: Early context truncation makes accuracy and length positively correlated, and that correlation fades once responses shorten. Fixed weights therefore oscillate late in training and stall above 3,380 tokens, whereas GEOMA adapts online without weight tuning.
Math Method (4-Benchmark Mean, 5 Eval Seeds, Appendix G.3)Mean AccuracyMean LengthExplanation
Untrained base model46.36%10,418 tokensFrequent context truncation on hard problems
Accuracy-only GRPO (weight 0.00, step 200)50.2%6,848 tokensAccuracy upper bound with no length penalty
Static weight 0.50 (step 150 -> step 175)49.77% -> 46.72%3,380 -> 3,476 tokensLoses 3 accuracy points late in training
Static weight 0.65 (step 200, full budget)47.50%3,574 tokensFull-budget static weight stalls above 3,500 tokens
Static weight 0.80 (step 175)48.36%3,545 tokensAbout 400 more tokens than Batch-Cov-Sum at step 150
GDPO baseline (step 200, full budget)46.8%3,897 tokensPer-group scaling lags on both accuracy and length
GEOMA Batch-Cov-Sum (step 150, independent seed)48.37%3,135 tokensBeats all late-horizon static runs on both metrics
GEOMA Batch-Cov-Sum (step 200, full budget)48.4%2,518 tokensSubmitted Table 2 result

Findings: Manual gating and penalties fail on signed or rare rewards without domain-specific reward shifting, and static math weights lose calibration as model length changes over training. Without tuning any weights, Batch-Cov-Sum at step 150 matches the best late fixed-weight accuracy with roughly 340 to 440 fewer tokens (Appendix G.3).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.