- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes PASP (Positive-Anchored Spectral Projection), a matrix-aware spectral optimizer for LLM reinforcement learning with verifiable rewards (RLVR). In Section 4 (Figure 2, Eq. 3, and Appendix Table 9), the authors decompose policy-gradient matrices by advantage sign, (Eq. 1), and document a striking signed spectral asymmetry: across checkpoints, 7 transformer projection types (q, k, v, o, gate, up, down), and 5 advantage estimators (GRPO, Reinforce++, RLOO, PPO, DAPO), negative-advantage gradients exhibit a 21–89$\times$ higher effective rank than positive-advantage gradients (e.g., at step 50 on Qwen3-8B, the top-8 singular subspace of captures 99.3% of energy vs. only 19.0% of energy).
To prevent high-rank negative updates from drowning out concentrated positive directions under standard Newton–Schulz spectral orthogonalization (, Eq. 2), PASP (Section 5, Figure 3, Eqs. 4–5, Algorithm 1) extracts an orthonormal left-singular anchor from the positive gradient (with live stable-rank estimation , Appendix A.3 Eq. 6), splits into an anchor-aligned component () and an orthogonal residual , and applies the guarded 5-step Newton–Schulz map separately to the combined anchor branch and a rank-truncated residual branch: (Eq. 5). Evaluated over 300 RLVR steps on DAPO-Math-17k across five primary settings (Qwen3-8B/GRPO, Qwen3-1.7B/GRPO, Llama-3.1-8B/GRPO, DeepSeekMath-7B/GRPO, and Qwen3-8B/DAPO, 3 seeds each in Table 1 and Appendix Tables 13–15) on MATH500, AIME24, AIME25, and OlympiadBench (plus secondary evaluations on GPQA-Diamond, AMC23-Chain4, Minerva-Math, MATH, and GSM8K in Appendix Tables 10–11), PASP achieves the highest 3-seed mean accuracy in all five main settings against SGD, AdamW, Muon, and Pion, with its largest gain in the Qwen3-1.7B recipe-transfer setting (+6.5 percentage points over Muon), though paired confidence intervals in four of the five settings remain wide and include zero.
Critical strengths
1. Optimizer architecture directly grounded in a striking, reproducible spectral property of RLVR gradients.
Rather than proposing an ad-hoc tweak to Muon or Pion, the paper first demonstrates that positive-advantage gradients are low-rank and coherent while negative-advantage gradients are high-rank and dispersed across all 21 checkpoint–layer combinations (Figure 2b) and 5 RL estimators (Figure 2c, Table 9). Splitting the update into a positive-anchored branch and a truncated orthogonal negative branch (Eq. 5) directly prevents high-rank negative noise from rotating the singular vectors of , and both component ablations (Table 2: dropping residual truncation loses -2.3 pp; dropping the sign split loses -3.0 pp) and anchor-source ablations (Table 3: Positive 0.4893 vs. Aggregated 0.4612) confirm that each piece of the design contributes.
2. Broad multi-model, multi-seed evaluation with unusually candid statistical and diagnostic reporting in the appendix.
The authors test PASP across four model backbones (Qwen3-8B, Qwen3-1.7B, Llama-3.1-8B, DeepSeekMath-7B), two RL algorithms (GRPO, DAPO), four primary and five secondary benchmarks (Tables 1, 10, 11), and three seeds (42, 43, 44) per main setting, while providing complete per-seed breakdowns and paired- intervals in Appendix G (Tables 13–15) and timing/routing analyses in Appendix C (Figure 6, Tables 6–7).
Critical weaknesses
1. Paired 95% confidence intervals include zero in four of the five main settings.
Appendix G.1 (Table 15, p. 25) reports same-seed paired- 95% confidence intervals ( across seeds 42–44) comparing PASP against the strongest baseline in each of the five main settings. Aside from Qwen3-1.7B/GRPO (+0.065, CI ), all four remaining intervals include zero: Qwen3-8B/GRPO vs. Pion is +0.019 (CI ), Llama-3.1-8B/GRPO vs. Muon is +0.010 (CI ), DeepSeekMath-7B/GRPO vs. Pion is +0.010 (CI , with Pion ahead by 0.007 on seed 44 in Table 14), and Qwen3-8B/DAPO vs. Pion is +0.017 (CI , tied to three decimal places at 0.506 on seed 44). Given only 3 seeds () and an estimated evaluation fluctuation on 30-problem AIME sets (Appendix D.4, p. 20), the advantage over Pion/Muon at the 7B–8B scale is directionally consistent but not yet statistically decisive.
2. Incomplete hyperparameter tuning parity for Muon/Pion and accuracy/speed trade-off between 3-pass and routed PASP.
All main comparisons in Table 1 use a single fixed learning rate (, Appendix D.2) across PASP, Muon, and Pion, and the Qwen3-1.7B experiment (where PASP gets its only statistically significant CI) explicitly transfers the 8B hyperparameter recipe without retuning—causing Pion to collapse to 0.184 (Muon full also drops to 0.369). Moreover, in the single-seed local LR sweep on Qwen3-8B/GRPO (Appendix E.5, Table 12, p. 23), shifting Muon to LR raises its score from 0.4555 to 0.4716 (+1.61 pp) and shifting Pion to LR raises its score from 0.4724 to 0.4763, narrowing PASP's (0.4861) single-seed margin over tuned Pion to just 0.98 percentage points—which is roughly the same size as the -0.88 percentage point drop (0.4909 to 0.4820 in Table 6) incurred when switching PASP from the slow 3-pass implementation ( Muon's step time) to the faster 1-pass sign-routing implementation ( Muon's step time).
Questions for authors
-
Clarification affecting soundness (2-pass DAPO discrepancy & 1-pass routed accuracy):
(a) InQwen3-8B/DAPO(which uses the 2-pass shortcut , Table 5), how large is the relative Frobenius discrepancy at steps 50, 150, and 300 when DAPO's asymmetric clipping (0.2 / 0.28) is active?
(b) In Table 6 (p. 18), 1-pass sign routing drops 2-seed accuracy onQwen3-8Bfrom 0.4909 (3-pass) to 0.4820 (-0.88 pp) because regularizer attribution differs across channels (Appendix B.2, line 889). Can routing be modified to match the 3-pass regularizer split exactly, or does fast 1-pass PASP give up most of its +1.9 pp margin overPion(0.470)? -
Clarification affecting empirical strength (Tuned
Pion/MuononQwen3-1.7Band8B):
In Appendix Table 12 (seed 42 onQwen3-8B),Pionat LR reaches 0.4763 (within 0.98 pp of PASP's 0.4861), and onQwen3-1.7B(Table 1),Pioncollapses to 0.184 under the unretuned8Brecipe. How doPionandMuonperform onQwen3-1.7B(and across seeds 42–44 onQwen3-8B) when each baseline is run at its own best learning rate from the grid? -
Clarification affecting benchmark validity (Decontamination of
DAPO-Math-17k):
Did you verify exact or near-duplicate decontamination between theDAPO-Math-17ktraining set and the nine evaluation benchmarks (especiallyMATH500, fullMATH,AIME24,OlympiadBench,AMC23-Chain4, andMinerva-Math)?
3 comments
Two results finished after our first reply.
W2/Q2. Tuned baselines across seeds. We reran the best rates from the seed-42 grid on seeds 43 and 44. At both sizes these are Pion at and Muon at . PASP keeps its original rate. The table gives three-seed means ± sample standard deviation.
| Model | Seeds | PASP | Pion | Muon |
|---|---|---|---|---|
| Qwen3-8B | 42, 43, 44 | 0.4890 ± 0.0058 | 0.4726 ± 0.0064 | 0.4701 ± 0.0044 |
| Qwen3-1.7B | 42, 43, 44 | 0.463 ± 0.0061 | 0.4252 ± 0.0086 | 0.4145 ± 0.0063 |
PASP is highest on every seed in both settings. At 8B the per-seed PASP scores are 0.4861, 0.4956 and 0.4852, against 0.4763, 0.4762 and 0.4652 for Pion. The rates were chosen on seed 42 only, so this is still a local grid, not an equal tuning budget.
W2/Q1(b). Corrected routing. We built the version that splits the channels before the reduction. Every rank processes its positive responses first and its negative responses second, so each reduce-scatter sums gradients of a single sign. We repeated the earlier probe on the same two batches.
| Final update vs three-pass, mean (range) | Relative error | Cosine |
|---|---|---|
| Old routing | 1.0244 (1.0068 to 1.0421) | 0.5085 (0.5035 to 0.5135) |
| Corrected routing | 0.1273 (0.1135 to 0.1411) | 0.9918 (0.9900 to 0.9936) |
| Three-pass, micro-batch order reversed | 0.1071 (0.1039 to 0.1102) | 0.9943 (0.9939 to 0.9946) |
The last row runs three-pass twice and only reverses the micro-batch order, so it shows how much rounding alone changes the update. Corrected routing is close to that level but slightly outside it. It runs 40% faster per step than three-pass. All main GRPO results still use three-pass PASP.
We appreciate the time you spent on our paper. Your points on tuning and implementation helped us make it more complete. Since the discussion closes soon, we hope the update above is useful, and we would be glad to clarify anything that remains.
Thank you for the detailed review.
W1. Statistical uncertainty. PASP has the highest mean in all five settings. We agree this is not the same as a statistically clear win, since four of the paired 95% intervals include zero. The exception is Qwen3-1.7B (+0.065, CI [0.039, 0.090]), which measures recipe transfer. Each setting has three training seeds, each evaluated once, so the intervals cover both training and decoding variation.
Q1(a). Two-pass DAPO with asymmetric clipping. With DAPO's clipping rule (0.2/0.28), the mean relative Frobenius error of each channel stays below 3%. The table gives results over five draws.
Each draw is an independent 96 × 4 batch, and each metric is averaged over 35 matrices. At step 300, the final spectral update has cosine similarity 0.9922 (0.9911 to 0.9941) and relative Frobenius error 10.55% (9.48 to 11.83%) against three-pass. The two numbers are averaged separately, so one cannot be converted into the other. Both paths use the same sketch for each matrix.
W2/Q1(b). Regularizer attribution and routing accuracy. Not in the current code. The fix we described assumes that each routed increment has a single sign. An eight-GPU measurement shows that this is not true. FSDP2 sums gradients across ranks before each reduced shard is assigned by the local response sign, so both routed channels mix positive and negative parts. In a one-step probe on two batches, with fixed rank and the same sketches, the routed final update has mean cosine 0.5085 (0.5035 to 0.5135) and relative error 1.0244 (1.0068 to 1.0421) against three-pass. These are joint over all trainable tensors, unlike the per-matrix means in Q1(a). Moving the regularizer gradient cannot fix this. A correct version would split the channels before the reduction. We have not built or timed it. We now treat routing as a separate one-backward variant, not as an implementation of PASP, and will revise Section 6.3 and Appendices B.2 and C.1 to match. All main GRPO results use three-pass PASP. Routing accuracy is below.
On these two seeds, routing is 1.89 pp above Pion, compared with 2.77 pp for three-pass, and wins on both. On seed 42 it is also above the best-tuned Pion (0.4818 vs 0.4763). Its median step time is 38% lower than three-pass. The 0.88 pp gap comes from the whole channel mismatch, not only from how the regularizer is assigned.
W2/Q2. Baseline tuning at 8B and 1.7B. At 8B (seed 42, 300 steps, a shared 0.5x/1x/2x grid around 5×10−6 plus one extra point per baseline), the best results are PASP 0.4861, Pion 0.4763 at 2.5×10−6 and Muon 0.4716 at 1×10−5. At 1.7B, the new Pion runs use seed 42 and step 300.
Tuning helps Pion a lot at 1.7B, and PASP is still 3.6 pp ahead. The original three-seed mean of 0.184 reflects the untuned recipe. We do not have a full 1.7B grid for both baselines or best-rate 8B runs on seeds 43 and 44.
Q3. Decontamination. Yes. We checked 16,400 normalized unique training questions against all nine evaluation sets. There are no exact matches. For near matches we used 8-word shingle Jaccard similarity of at least 0.82, then checked by hand. We removed 5 MATH500, 14 OlympiadBench and 7 full-MATH items. AIME24/25, AMC23-Chain4, Minerva-Math, GSM8K and GPQA-Diamond have no detected matches.
We re-evaluated the step-300 checkpoints of all three seeds on the cleaned sets with the original protocol (temperature 0.6, top-p 1.0, 8,192 tokens, decoding seed 0, same templates and scorer) and freshly sampled answers. On three-seed means, PASP is above the designated baseline in 19 of the 20 main benchmark and setting pairs, with one tie on Llama-3.1-8B MATH500, where PASP and hidden-only Muon both score 0.519. Since the answers were resampled, we do not attribute score changes only to the removed items. This audit covers lexical overlap with DAPO-Math-17k. Paraphrases and overlap through SFT or pretraining are outside its scope.
Three cited numbers. Three numbers in the review differ from the submitted PDF. Three-pass and routed PASP take 1.95× and 1.20× Muon's median step time (2213 and 1367 vs 1137 s, Table 7). On Qwen3-1.7B, hidden-only Muon averages 0.398, and all-layer Muon diverges. On Qwen3-8B/DAPO seed 44, PASP and Pion both score 0.481 (Table 14).