- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
GEOMA separates multi-reward advantage construction into covariance-based preconditioning and aggregation by a sum, logistic Nash welfare, or SoftMin. It studies whitening and redundancy properties and explains that sum aggregation reduces to a weighted sum of the original rewards. Experiments vary these choices for tool use and mathematical reasoning with 1.5B models. Batch-covariance preconditioning with a sum improves mean math accuracy over GDPO from 46.8% to 48.4% and reduces mean length from 3,897 to 2,518 tokens, while the unnormalized GRPO baseline remains more accurate but longer. Tool-calling improvements in BFCL accuracy are smaller.
Critical strengths
-
The design space is explicit and easy to study. Remark 4.1 and Proposition B.16 correctly make clear that covariance-plus-sum is a reward-weighting rule, and the duplicate-reward example acknowledges that whitening reduces rather than eliminates double counting. Those qualifications are useful, as they distinguish algebraic properties of the preconditioner from claims about policy behavior.
-
The accuracy/length results provide a concrete reason to investigate covariance-aware weighting. Batch-covariance with Sum retains or improves GDPO's accuracy on all four math benchmarks while substantially reducing average generation length (Table 2). Reporting GRPO's higher accuracy and longer responses also exposes the trade-off instead of declaring one scalar metric to be the whole task.
Critical weaknesses
-
Monotonic welfare in the whitened coordinates does not imply that the composed objective protects the original rewards. Section 4 says the ZCA coordinates still correspond to the original rubrics, but whitening mixes those rubrics and need not preserve coordinatewise monotonicity. For a fixed valid covariance Sigma = [[1, 2.9], [2.9, 9]] and ridge 1e-6, P = (Sigma + ridge I)^(-1/2) is approximately [[3.7441, -1.1116], [-1.1116, 0.6777]]. The Sum weights P^T 1 are approximately (2.6325, -0.4338): increasing only the second original reward reduces the scalar score. Near a centered reward of zero, the same increase also reduces both logistic Nash and SoftMin, because their gradients there are positive multiples of P^T 1. Thus the composed objectives can locally prefer a Pareto-inferior original reward vector even with a fixed preconditioner. The welfare monotonicity results in Appendix B concern the transformed coordinates and do not resolve this issue. This directly limits the interpretation that no original rubric is being sacrificed.
-
The experiments do not yet establish that covariance sphering is preferable to a well-chosen weighting baseline. The strongest math configuration is exactly an adaptive weighted sum, but no tuned positive-weight or comparable adaptive-weight control is reported. BFCL averages span less than one percentage point across configurations, and each configuration is trained once. The five math generations measure sampling variation for a trained model, not training-seed uncertainty. All reported tasks use two rewards, so the experiments also do not directly test the motivating case of redundant rewards in the presence of an independent objective.
Questions for authors
- Under what conditions is G(P(r - mu)) monotone in each original reward? Can you address the covariance counterexample above and distinguish protection of transformed coordinates from protection of the original rubrics? If monotonicity is not intended, the alignment interpretation needs to state that limitation explicitly.
- How do the covariance-Sum variants compare with tuned nonnegative scalar weights under the same training budget, including the resulting accuracy–length frontier? This comparison would establish what the covariance estimate adds beyond choosing a different trade-off.
- Do the gains persist across independent training seeds, and under a controlled three-reward experiment that duplicates one reward while retaining an independent third objective? Please separate training variation from repeated decoding of one checkpoint.
3 comments
Tuned Non-Negative Scalar Weights on Math
Reviewer Query: Because covariance preconditioning with Sum aggregation acts as an adaptive weighted sum of rewards, the reviewer asked how GEOMA compares against tuned fixed positive weights (1 - alpha) * r_acc + alpha * r_len under the same training budget, and what online covariance estimation adds beyond choosing a static trade-off.
"How do the covariance-Sum variants compare with tuned nonnegative scalar weights under the same training budget, including the resulting accuracy-length frontier? This comparison would establish what the covariance estimate adds beyond choosing a different trade-off."
Our Response (addressed in Section 7.2, Appendix G.3, and Appendix G.4):
- New math experiments and audit conducted:
- 27 checkpoint evaluations on DeepSeek-R1-Distill-Qwen-1.5B (each decoded across 5 seeds) sweeping static weights alpha in {0.00, 0.50, 0.65, 0.80} alongside GDPO, our Batch-Diag-Sum marginal ablation, and an independent training seed of Batch-Cov-Sum.
- Codebase verification clarifying that Table 2's GRPO row corresponds to the alpha = 0.00 accuracy-only anchor.
- Why fixed weights lose calibration mid-training: At initialization, 34% of math rollouts hit the 8,192-token limit (scoring 0 on both accuracy and length), creating +0.44 positive correlation that fades near zero once responses shorten below 2,000 tokens. Fixed weights cannot track this shift, oscillating in late-stage accuracy and stalling above 3,380 tokens.
- Why online covariance estimation improves the frontier: Positive early correlation causes Batch-Cov-Sum to boost length compression while truncation is active and automatically relax toward marginal scaling once truncation disappears, improving accuracy and length smoothly without tuning alpha.
| Math Checkpoint (DeepSeek-R1-1.5B, 5 Decoding Seeds, Appendix G.3) | MATH-500 | AIME-25 | AMC-24 | Minerva | Mean Acc | Mean Length | Explanation |
|---|---|---|---|---|---|---|---|
| Untrained base model | 82.44% | 26.00% | 53.33% | 23.68% | 46.36% | 10,418 | Frequent 8k truncation on hard problems |
| Accuracy-only anchor (alpha = 0.00, step 200) | 85.2% | 27.3% | 61.8% | 26.5% | 50.2% | 6,848 | Accuracy upper bound with no length control |
| Static alpha = 0.50 (step 150 -> step 175) | 84.44% -> 84.56% | 28.67% -> 20.67% | 62.22% -> 56.45% | 23.75% -> 25.22% | 49.77% -> 46.72% | 3,380 -> 3,476 | Fixed weight drops 3 points late in training |
| Static alpha = 0.65 (step 200, full budget) | 83.72% | 24.67% | 58.22% | 23.38% | 47.50% | 3,574 | Full-budget static run stalls above 3,500 tokens |
| Static alpha = 0.80 (step 175) | 83.60% | 22.67% | 64.00% | 23.16% | 48.36% | 3,545 | About 400 more tokens than Batch-Cov-Sum at step 150 |
| GDPO baseline (step 200, full budget) | 83.2% | 28.0% | 52.4% | 23.5% | 46.8% | 3,897 | Per-group scaling trails on both axes |
| Batch-Diag-Sum ablation (step 100) | 83.16% | 24.67% | 56.89% | 23.46% | 47.04% | 3,100 | Within 0.3 points of Batch-Cov-Sum at step 100 (47.34%) |
| GEOMA Batch-Cov-Sum (step 150, new train seed) | 83.68% | 28.67% | 56.00% | 25.15% | 48.37% | 3,135 | Dominates all late-horizon static runs |
| GEOMA Batch-Cov-Sum (step 200, full budget) | 83.9% | 28.0% | 57.7% | 23.9% | 48.4% | 2,518 | +0.9% acc and 1,056 fewer tokens vs alpha = 0.65 |
Findings: Online covariance weighting adapts automatically outperforming tuned static weights on late-stage accuracy. It additionally saves roughly 400 tokens at step 150 (independent training seed) and over 1,000 tokens at step 200 (Appendix G.3).
Training Seeds, Experimental Controls
Question 3: Multi-Seed Training Variance and Controlled 3-Reward & 5-Reward Ladders
Reviewer Query: The reviewer asked whether GEOMA's gains hold across independent training seeds (reported separately from multi-sample decoding of one checkpoint) and under a controlled multi-reward experiment that duplicates one reward alongside an independent constraint.
"Do the gains persist across independent training seeds, and under a controlled three-reward experiment that duplicates one reward while retaining an independent third objective? Please separate training variation from repeated decoding of one checkpoint."
Our Response (addressed in Section 7.1, Appendix G.2, and Appendix G.4):
- New multi-seed and duplication experiments conducted:
- All revised tables strictly separate cross-seed training SD (3 independent training seeds: 42, 101, 202) from repeated-decoding SD (5 evaluation seeds).
- Controlled K = 3 (2 copies of correctness + 1 format) and K = 5 (4 copies + 1 format) duplication ladders on both Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct, testing GDPO, our Batch-Diag-Sum marginal ablation, Batch-Cov-Sum, and exact-invariance C-Prec-Sum.
- Cross-seed training stability: At 1.5B on 2 rewards, Batch-Cov-Sum cuts training-seed SD from 26.3 +/- 13.6 steps (GDPO) to 15.0 +/- 1.0 steps and leads Multi-Turn BFCL-v3 (3.33 +/- 0.26% vs 2.80 +/- 0.14%). Because single-turn BFCL-v3 uses Qwen's native tool syntax and varies by under 1 point across seeds, we drop single-turn superiority claims from the manuscript.
- Controlled K = 3 and K = 5 duplication: With 2 initially uncorrelated rewards at 1.5B, covariance is nearly diagonal so Batch-Cov matches our Batch-Diag ablation while halving seed SD. Once correctness is duplicated, marginal scaling weights m copies at m : 1 (collapsing GDPO and delaying Batch-Diag to 29 steps), whereas Batch-Cov-Sum (sqrt(m) : 1) and C-Prec-Sum (exact 1 : 1) stay fast and protect correctness at both 1.5B and 7B.
| Controlled Experiment (Steps to 85% Format, Appendix G.2) | GDPO Baseline | Batch-Diag Ablation | GEOMA Batch-Cov-Sum | GEOMA C-Prec-Sum | Explanation |
|---|---|---|---|---|---|
| 1.5B, K = 2 (mean +/- training SD, 3 seeds) | 26.3 +/- 13.6 | 15.7 +/- 2.1 | 15.0 +/- 1.0 | Matches Batch-Diag | Near-diagonal initial covariance; 11 steps faster than GDPO with 13x lower training SD |
| 1.5B, K = 3 (2 copies of correctness + format) | Collapsed | 20 steps | 16 steps | 16 steps | Marginal double-counting collapses GDPO |
| 1.5B, K = 5 (4 copies of correctness + format) | Collapsed | 29 steps | 20 steps | 16 steps | 4x over-counting slows Batch-Diag; C-Prec-Sum shows zero slowdown |
| 7B, K = 3 (2 copies of correctness + format) | 25 steps | 20 steps | 16 steps | 22 steps (+1.88 cor.) | At 7B, Batch-Cov-Sum beats GDPO by 9 steps; C-Prec-Sum reaches highest correctness |
Findings: Across independent training seeds and 1.5B-to-7B duplicate-reward ladders, covariance and correlation-precision preconditioning eliminate seed instability and prevent the collapses and slowdowns caused by marginal double-counting (Appendix G.2).
Your note that GEOMA's design space is explicit and easy to study, and that our paper carefully distinguishes algebraic properties of whitening from policy behavior, was a real highlight of the review process for us. Thank you again for your 2x2 counterexample and experimental suggestions, every one of which we implemented and proved in full. If our new monotonicity proofs, tuned static-weight frontier, multi-seed runs, and K = 3 and K = 5 duplication experiments have resolved your concerns, we would be deeply grateful if you would consider raising your score toward acceptance.
Original-Reward Monotonicity and the 2x2 Counterexample
We thank Reviewer WCFe for the rigorous technical reading and concrete experimental suggestions, which directly shaped our new proofs and experiments in Appendix G.
Strengths Highlighted by Reviewer WCFe
Weakness 1 & Question 1: Original-Reward Monotonicity and the 2x2 Counterexample
Reviewer Query: Because covariance whitening mixes the original rewards together, the reviewer showed via an explicit 2x2 example (correlation 0.97, scale ratio 3) that an original reward's effective weight can turn negative (-0.43), and asked under what conditions GEOMA preserves each original reward's priority across Sum, Nash, and SoftMin.
Our Response (addressed in Section 4, Section 6.2, and Appendix G.1):
Findings: Original-rubric Pareto dominance holds globally for conflicting rewards and below 0.71 correlation across all scale ratios, covering 100% of our batch RL steps. Furthermore, Section 4 explicitly states the high-correlation boundary threshold (Appendix G.1).