acceptodds
Under review as a conference paper at ICLR 2027

PASP: Positive-Anchored Spectral Projection for RLVR

Abstract

Muon has improved large language model pretraining, and Pion extends its spectral approach to reinforcement learning with verifiable rewards (RLVR). We identify a spectral asymmetry: positive-advantage gradients concentrate in a low-dimensional subspace and have lower effective rank than negative-advantage gradients. Muon and Pion transform these gradients after combining them into momentum, without separately controlling the two channels. This motivates PASP (Positive-Anchored Spectral Projection), which uses the current positive gradient to define an anchor subspace at each step. PASP jointly transforms the full positive gradient and the part of the negative gradient within this subspace. It separately transforms a low-rank approximation of the remaining negative gradient and adds the two outputs to form the update. We evaluate PASP on Qwen3 models against AdamW, Muon (hidden-only), and Pion. PASP achieves the highest mean accuracy across four reasoning benchmarks on both Qwen3 models. On Qwen3-8B, it reaches 0.489 ± 0.006, compared with 0.470 ± 0.015 for Pion, where ± denotes the sample standard deviation over three training seeds. On Qwen3-1.7B, the 8B recipes are transferred without retuning. All three PASP runs complete training, averaging 0.463, compared with 0.398 for Muon and 0.184 for Pion. Under DAPO, PASP reaches 0.485, compared with 0.466 for Muon and 0.468 for Pion. It also remains competitive on GPQA-Diamond and AMC23-Chain4, although the gains vary across settings. These results support using advantage sign to guide spectral optimization in the tested RLVR settings.

Primary area:
optimization
Keywords:
RLVR, spectral optimizer, Muon, Pion, advantage sign, policy gradient, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_u3guOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PASP (Positive-Anchored Spectral Projection), a matrix-aware spectral optimizer for LLM reinforcement learning with verifiable rewards (RLVR). In Section 4 (Figure 2, Eq. 3, and Appendix Table 9), the authors decompose policy-gradient matrices by advantage sign, Gt=Gt++Gt−G _ t = G _ t^+ + G _ t^- (Eq. 1), and document a striking signed spectral asymmetry: across checkpoints, 7 transformer projection types (q, k, v, o, gate, up, down), and 5 advantage estimators (GRPO, Reinforce++, RLOO, PPO, DAPO), negative-advantage gradients Gt−G _ t^- exhibit a 21–89$\times$ higher effective rank reff(A)=∥A∥∗2/∥A∥F2r _ {\mathrm{eff}}(A) = \Vert A \Vert _ *^2 / \Vert A \Vert _ F^2 than positive-advantage gradients Gt+G _ t^+ (e.g., at step 50 on Qwen3-8B, the top-8 singular subspace of Gt++Gt−G _ t^+ + G _ t^- captures 99.3% of Gt+G _ t^+ energy vs. only 19.0% of Gt−G _ t^- energy).

To prevent high-rank negative updates from drowning out concentrated positive directions under standard Newton–Schulz spectral orthogonalization (Φϑ\Phi _ \vartheta, Eq. 2), PASP (Section 5, Figure 3, Eqs. 4–5, Algorithm 1) extracts an orthonormal left-singular anchor QtQ _ t from the positive gradient (with live stable-rank estimation rt+∈[8,64]r _ t^+ \in [8, 64], Appendix A.3 Eq. 6), splits Gt−G _ t^- into an anchor-aligned component Gt∥=PtGt−G _ t^\parallel = P _ t G _ t^- (Pt=QtQt⊤P _ t = Q _ t Q _ t^\top) and an orthogonal residual Gt⊥=(I−Pt)Gt−G _ t^\perp = (I - P _ t)G _ t^-, and applies the guarded 5-step Newton–Schulz map separately to the combined anchor branch and a rank-truncated residual branch: Δt=Φ‾(Gt++Gt∥)+Φ‾(Π(Gt⊥))\Delta _ t = \overline{\Phi}(G _ t^+ + G _ t^\parallel) + \overline{\Phi}(\Pi(G _ t^\perp)) (Eq. 5). Evaluated over 300 RLVR steps on DAPO-Math-17k across five primary settings (Qwen3-8B/GRPO, Qwen3-1.7B/GRPO, Llama-3.1-8B/GRPO, DeepSeekMath-7B/GRPO, and Qwen3-8B/DAPO, 3 seeds each in Table 1 and Appendix Tables 13–15) on MATH500, AIME24, AIME25, and OlympiadBench (plus secondary evaluations on GPQA-Diamond, AMC23-Chain4, Minerva-Math, MATH, and GSM8K in Appendix Tables 10–11), PASP achieves the highest 3-seed mean accuracy in all five main settings against SGD, AdamW, Muon, and Pion, with its largest gain in the Qwen3-1.7B recipe-transfer setting (+6.5 percentage points over Muon), though paired confidence intervals in four of the five settings remain wide and include zero.

Critical strengths

1. Optimizer architecture directly grounded in a striking, reproducible spectral property of RLVR gradients.
Rather than proposing an ad-hoc tweak to Muon or Pion, the paper first demonstrates that positive-advantage gradients Gt+G _ t^+ are low-rank and coherent while negative-advantage gradients Gt−G _ t^- are high-rank and dispersed across all 21 checkpoint–layer combinations (Figure 2b) and 5 RL estimators (Figure 2c, Table 9). Splitting the update into a positive-anchored branch Φ‾(Gt++Gt∥)\overline{\Phi}(G _ t^+ + G _ t^\parallel) and a truncated orthogonal negative branch Φ‾(Π(Gt⊥))\overline{\Phi}(\Pi(G _ t^\perp)) (Eq. 5) directly prevents high-rank negative noise from rotating the singular vectors of Gt+G _ t^+, and both component ablations (Table 2: dropping residual truncation loses -2.3 pp; dropping the sign split loses -3.0 pp) and anchor-source ablations (Table 3: Positive 0.4893 vs. Aggregated 0.4612) confirm that each piece of the design contributes.

2. Broad multi-model, multi-seed evaluation with unusually candid statistical and diagnostic reporting in the appendix.
The authors test PASP across four model backbones (Qwen3-8B, Qwen3-1.7B, Llama-3.1-8B, DeepSeekMath-7B), two RL algorithms (GRPO, DAPO), four primary and five secondary benchmarks (Tables 1, 10, 11), and three seeds (42, 43, 44) per main setting, while providing complete per-seed breakdowns and paired-tt intervals in Appendix G (Tables 13–15) and timing/routing analyses in Appendix C (Figure 6, Tables 6–7).

Critical weaknesses

1. Paired 95% confidence intervals include zero in four of the five main settings.
Appendix G.1 (Table 15, p. 25) reports same-seed paired-tt 95% confidence intervals (df=2\mathrm{df}=2 across seeds 42–44) comparing PASP against the strongest baseline in each of the five main settings. Aside from Qwen3-1.7B/GRPO (+0.065, CI [+0.039,+0.090][+0.039, +0.090]), all four remaining intervals include zero: Qwen3-8B/GRPO vs. Pion is +0.019 (CI [−0.031,+0.070][-0.031, +0.070]), Llama-3.1-8B/GRPO vs. Muon is +0.010 (CI [−0.014,+0.034][-0.014, +0.034]), DeepSeekMath-7B/GRPO vs. Pion is +0.010 (CI [−0.041,+0.061][-0.041, +0.061], with Pion ahead by 0.007 on seed 44 in Table 14), and Qwen3-8B/DAPO vs. Pion is +0.017 (CI [−0.034,+0.068][-0.034, +0.068], tied to three decimal places at 0.506 on seed 44). Given only 3 seeds (df=2\mathrm{df}=2) and an estimated ±0.005\pm 0.005 evaluation fluctuation on 30-problem AIME sets (Appendix D.4, p. 20), the advantage over Pion/Muon at the 7B–8B scale is directionally consistent but not yet statistically decisive.

2. Incomplete hyperparameter tuning parity for Muon/Pion and accuracy/speed trade-off between 3-pass and routed PASP.
All main comparisons in Table 1 use a single fixed learning rate (5×10−65\times 10^{-6}, Appendix D.2) across PASP, Muon, and Pion, and the Qwen3-1.7B experiment (where PASP gets its only statistically significant CI) explicitly transfers the 8B hyperparameter recipe without retuning—causing Pion to collapse to 0.184 (Muon full also drops to 0.369). Moreover, in the single-seed local LR sweep on Qwen3-8B/GRPO (Appendix E.5, Table 12, p. 23), shifting Muon to 2×2\times LR raises its score from 0.4555 to 0.4716 (+1.61 pp) and shifting Pion to 0.5×0.5\times LR raises its score from 0.4724 to 0.4763, narrowing PASP's (0.4861) single-seed margin over tuned Pion to just 0.98 percentage points—which is roughly the same size as the -0.88 percentage point drop (0.4909 to 0.4820 in Table 6) incurred when switching PASP from the slow 3-pass implementation (2.39×2.39\times Muon's step time) to the faster 1-pass sign-routing implementation (1.48×1.48\times Muon's step time).

Questions for authors
  1. Clarification affecting soundness (2-pass DAPO discrepancy & 1-pass routed accuracy):
    (a) In Qwen3-8B/DAPO (which uses the 2-pass shortcut G~±=12(G±G∣A∣)\widetilde{G}^\pm = \frac{1}{2}(G \pm G^{\lvert A \rvert}), Table 5), how large is the relative Frobenius discrepancy ∥G2-pass±−G3-pass±∥F/∥G3-pass±∥F\Vert G _ {\mathrm{2\text{-}pass}}^\pm - G _ {\mathrm{3\text{-}pass}}^\pm \Vert _ F / \Vert G _ {\mathrm{3\text{-}pass}}^\pm \Vert _ F at steps 50, 150, and 300 when DAPO's asymmetric clipping (0.2 / 0.28) is active?
    (b) In Table 6 (p. 18), 1-pass sign routing drops 2-seed accuracy on Qwen3-8B from 0.4909 (3-pass) to 0.4820 (-0.88 pp) because regularizer attribution differs across channels (Appendix B.2, line 889). Can routing be modified to match the 3-pass regularizer split exactly, or does fast 1-pass PASP give up most of its +1.9 pp margin over Pion (0.470)?

  2. Clarification affecting empirical strength (Tuned Pion/Muon on Qwen3-1.7B and 8B):
    In Appendix Table 12 (seed 42 on Qwen3-8B), Pion at 0.5×0.5\times LR reaches 0.4763 (within 0.98 pp of PASP's 0.4861), and on Qwen3-1.7B (Table 1), Pion collapses to 0.184 under the unretuned 8B recipe. How do Pion and Muon perform on Qwen3-1.7B (and across seeds 42–44 on Qwen3-8B) when each baseline is run at its own best learning rate from the 0.25×,0.5×,1×,2×0.25\times, 0.5\times, 1\times, 2\times grid?

  3. Clarification affecting benchmark validity (Decontamination of DAPO-Math-17k):
    Did you verify exact or near-duplicate decontamination between the DAPO-Math-17k training set and the nine evaluation benchmarks (especially MATH500, full MATH, AIME24, OlympiadBench, AMC23-Chain4, and Minerva-Math)?

3 comments

Thank you for the detailed review.

W1. Statistical uncertainty. PASP has the highest mean in all five settings. We agree this is not the same as a statistically clear win, since four of the paired 95% intervals include zero. The exception is Qwen3-1.7B (+0.065, CI [0.039, 0.090]), which measures recipe transfer. Each setting has three training seeds, each evaluated once, so the intervals cover both training and decoding variation.

Q1(a). Two-pass DAPO with asymmetric clipping. With DAPO's clipping rule (0.2/0.28), the mean relative Frobenius error of each channel stays below 3%. The table gives results over five draws.

StepPositive error, mean (range)Negative error, mean (range)
500.84% (0.69 to 1.08%)2.31% (1.74 to 3.52%)
1501.15% (0.93 to 1.36%)2.87% (2.06 to 3.91%)
3000.98% (0.81 to 1.27%)2.64% (1.89 to 4.01%)

Each draw is an independent 96 × 4 batch, and each metric is averaged over 35 matrices. At step 300, the final spectral update has cosine similarity 0.9922 (0.9911 to 0.9941) and relative Frobenius error 10.55% (9.48 to 11.83%) against three-pass. The two numbers are averaged separately, so one cannot be converted into the other. Both paths use the same sketch for each matrix.

W2/Q1(b). Regularizer attribution and routing accuracy. Not in the current code. The fix we described assumes that each routed increment has a single sign. An eight-GPU measurement shows that this is not true. FSDP2 sums gradients across ranks before each reduced shard is assigned by the local response sign, so both routed channels mix positive and negative parts. In a one-step probe on two batches, with fixed rank and the same sketches, the routed final update has mean cosine 0.5085 (0.5035 to 0.5135) and relative error 1.0244 (1.0068 to 1.0421) against three-pass. These are joint over all trainable tensors, unlike the per-matrix means in Q1(a). Moving the regularizer gradient cannot fix this. A correct version would split the channels before the reduction. We have not built or timed it. We now treat routing as a separate one-backward variant, not as an implementation of PASP, and will revise Section 6.3 and Appendices B.2 and C.1 to match. All main GRPO results use three-pass PASP. Routing accuracy is below.

SeedPASP, three-passPASP, routingPion
420.48610.48180.4724
430.49560.48230.4539

On these two seeds, routing is 1.89 pp above Pion, compared with 2.77 pp for three-pass, and wins on both. On seed 42 it is also above the best-tuned Pion (0.4818 vs 0.4763). Its median step time is 38% lower than three-pass. The 0.88 pp gap comes from the whole channel mismatch, not only from how the regularizer is assigned.

W2/Q2. Baseline tuning at 8B and 1.7B. At 8B (seed 42, 300 steps, a shared 0.5x/1x/2x grid around 5×10−65\times10^{-6} plus one extra point per baseline), the best results are PASP 0.4861, Pion 0.4763 at 2.5×10−62.5\times10^{-6} and Muon 0.4716 at 1×10−51\times10^{-5}. At 1.7B, the new Pion runs use seed 42 and step 300.

MethodLearning rateMean accuracy
PASP5×10−65\times10^{-6} (original)0.460
Pion2.5×10−62.5\times10^{-6}0.4244
Pion5×10−65\times10^{-6} (transferred)0.211
Pion1×10−51\times10^{-5}0.0388

Tuning helps Pion a lot at 1.7B, and PASP is still 3.6 pp ahead. The original three-seed mean of 0.184 reflects the untuned recipe. We do not have a full 1.7B grid for both baselines or best-rate 8B runs on seeds 43 and 44.

Q3. Decontamination. Yes. We checked 16,400 normalized unique training questions against all nine evaluation sets. There are no exact matches. For near matches we used 8-word shingle Jaccard similarity of at least 0.82, then checked by hand. We removed 5 MATH500, 14 OlympiadBench and 7 full-MATH items. AIME24/25, AMC23-Chain4, Minerva-Math, GSM8K and GPQA-Diamond have no detected matches.

We re-evaluated the step-300 checkpoints of all three seeds on the cleaned sets with the original protocol (temperature 0.6, top-pp 1.0, 8,192 tokens, decoding seed 0, same templates and scorer) and freshly sampled answers. On three-seed means, PASP is above the designated baseline in 19 of the 20 main benchmark and setting pairs, with one tie on Llama-3.1-8B MATH500, where PASP and hidden-only Muon both score 0.519. Since the answers were resampled, we do not attribute score changes only to the removed items. This audit covers lexical overlap with DAPO-Math-17k. Paraphrases and overlap through SFT or pretraining are outside its scope.

Three cited numbers. Three numbers in the review differ from the submitted PDF. Three-pass and routed PASP take 1.95× and 1.20× Muon's median step time (2213 and 1367 vs 1137 s, Table 7). On Qwen3-1.7B, hidden-only Muon averages 0.398, and all-layer Muon diverges. On Qwen3-8B/DAPO seed 44, PASP and Pion both score 0.481 (Table 14).

Two results finished after our first reply.

W2/Q2. Tuned baselines across seeds. We reran the best rates from the seed-42 grid on seeds 43 and 44. At both sizes these are Pion at 2.5×10−62.5\times10^{-6} and Muon at 1×10−51\times10^{-5}. PASP keeps its original rate. The table gives three-seed means ± sample standard deviation.

ModelSeedsPASPPionMuon
Qwen3-8B42, 43, 440.4890 ± 0.00580.4726 ± 0.00640.4701 ± 0.0044
Qwen3-1.7B42, 43, 440.463 ± 0.00610.4252 ± 0.00860.4145 ± 0.0063

PASP is highest on every seed in both settings. At 8B the per-seed PASP scores are 0.4861, 0.4956 and 0.4852, against 0.4763, 0.4762 and 0.4652 for Pion. The rates were chosen on seed 42 only, so this is still a local grid, not an equal tuning budget.

W2/Q1(b). Corrected routing. We built the version that splits the channels before the reduction. Every rank processes its positive responses first and its negative responses second, so each reduce-scatter sums gradients of a single sign. We repeated the earlier probe on the same two batches.

Final update vs three-pass, mean (range)Relative errorCosine
Old routing1.0244 (1.0068 to 1.0421)0.5085 (0.5035 to 0.5135)
Corrected routing0.1273 (0.1135 to 0.1411)0.9918 (0.9900 to 0.9936)
Three-pass, micro-batch order reversed0.1071 (0.1039 to 0.1102)0.9943 (0.9939 to 0.9946)

The last row runs three-pass twice and only reverses the micro-batch order, so it shows how much rounding alone changes the update. Corrected routing is close to that level but slightly outside it. It runs 40% faster per step than three-pass. All main GRPO results still use three-pass PASP.

We appreciate the time you spent on our paper. Your points on tuning and implementation helped us make it more complete. Since the discussion closes soon, we hope the update above is useful, and we would be glad to clarify anything that remains.

Reviewer_zQ8LOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PASP (Positive-Anchored Spectral Projection), a spectral optimization method for reinforcement learning with verifiable rewards (RLVR). The authors observe that positive-advantage gradients exhibit lower effective rank than negative-advantage gradients and introduce a positive-anchored projection mechanism to process these gradient components separately. Experiments across several model families and RLVR algorithms demonstrate improvements over AdamW, Muon, and Pion on mathematical reasoning benchmarks.

While the observed spectral asymmetry is interesting, the experimental evidence does not sufficiently establish its causal relationship with the reported performance gains. Moreover, the improvements over strong baselines are relatively modest in several settings, and the additional computational and memory costs raise concerns about the practical advantages of the proposed method.

Critical strengths
  1. Interesting empirical observation. The paper identifies a spectral difference between positive- and negative-advantage gradients in RLVR. This observation provides a potentially useful perspective on designing spectral optimization methods.

  2. Reasonably comprehensive experimental evaluation. The experiments cover multiple model families, two RLVR algorithms, and several reasoning benchmarks. The ablation studies also provide useful evidence for the proposed components.

Critical weaknesses
  1. Insufficient evidence for the proposed mechanism. Although the paper identifies a substantial spectral asymmetry between positive- and negative-advantage gradients, it remains unclear whether this difference reflects task-relevant optimization structure or is influenced by factors such as token counts and response lengths. Furthermore, PASP operates on current gradients, whereas Muon and Pion transform accumulated momentum. This makes it difficult to isolate whether the improvements arise from the positive-anchored mechanism or other differences in optimization.

  2. Limited practical advantages relative to the additional cost. The performance improvements over Pion and Muon are relatively modest in several settings, with substantial uncertainty in the reported comparisons. Moreover, the main accuracy results rely on a computationally expensive three-pass implementation, while the more efficient sign-routing version achieves lower accuracy and requires additional memory. Given that the experiments primarily focus on mathematical reasoning, the practical benefits and broader generalizability of PASP remain insufficiently established.

Questions for authors
  1. Could the authors clarify whether the observed spectral asymmetry persists after controlling for the numbers of positive and negative tokens and their response lengths? How can the authors establish that this asymmetry is causally related to the performance improvements?

  2. Since PASP transforms current gradients while Muon and Pion operate on momentum, could the authors provide more controlled comparisons to isolate the contribution of positive-anchored projection from differences in momentum handling and spectral transformations?

  3. The main accuracy results use the three-pass implementation, while the more efficient sign-routing version exhibits an accuracy drop and higher memory consumption than Muon/Pion. Could the authors clarify the practical advantages of PASP under matched end-to-end computational budgets?

  4. Could the authors provide stronger evidence that PASP consistently outperforms carefully tuned baselines and generalizes beyond mathematical reasoning to other RLVR settings?

5 comments

Thank you for pointing out the controls we needed.

W1/Q1. Response composition and spectral asymmetry. Yes, the ordering holds under these controls. In an extra AdamW/GRPO probe (seed 42, step 20, 256 prompts with four responses each), the ratio of negative to positive effective rank is as follows.

ConditionNegative/positive rank
Natural sign split1.799
Count-matched subset1.861
Count- and length-matched subset1.891
Length-stratified sign shuffle1.274 (20-draw median)

Composition actually works against the asymmetry. The positive channel has more responses (162 vs 138) and more tokens (1.21M vs 1.08M), yet a lower effective rank, even though a larger sample would normally raise rank. Matching counts, and then counts and lengths, does not remove the gap, and both matched ratios stay above the shuffle median. So counts and lengths do not explain it. The matched ratios come from subsets of the batch, not the full training gradient. The absolute values are not comparable to the paper's sweep (different optimizer, step and aggregation), but the ordering matches. These results support the asymmetry itself. They do not prove that the rank gap causes the gains.

W1/Q2. Momentum and positive anchoring. Momentum does not explain the gap. On Qwen3-8B/GRPO we changed one factor at a time (means over seeds 42, 43 and 44).

Update inputMethodMean accuracy
MomentumMuon (hidden-only)0.456
MomentumPion (whole-matrix)0.470
Current gradientMuon (hidden-only)0.431
Current gradientPion (whole-matrix)0.452
Current gradientPASP, no split0.459
Current gradientPASP, full0.489

Removing momentum hurts both baselines (Muon by 2.5 pp, Pion by 1.8 pp). With the same current-gradient input, single-map methods reach 0.431 to 0.459, and the two-branch design adds 3.0 pp. With both branches fixed, the positive anchor beats random, negative and aggregated anchors on seeds 42 and 43 (0.4893 vs 0.4716, 0.4706 and 0.4612). The spectral maps themselves still differ across methods.

W2/Q3. Accuracy under computational budgets. At equal wall-clock time, three-pass PASP does not beat Pion. The faster routed variant does, but it uses more memory. Times below are from eight H20 GPUs.

MethodMedian step (s)Peak GB/GPUStep (cumulative h)Mean accuracy
PASP, three-pass221334140 (88.0)0.4579
PASP, routing136735240 (92.4)0.4780
Pion108320300 (91.5)0.4697
Muon113720300 (97.8)0.4556

Three-pass PASP is 1.2 pp below Pion at about the same time, so we do not claim an equal-time advantage for it. Its advantage is per rollout, since it reaches Pion's step-300 accuracy at about step 165. Routing scores 0.4752, 0.4777 and 0.4811 at step 240 on seeds 42, 43 and 44 (seed 44 has no step-300 result). It is a one-backward variant that does not reproduce the three-pass update on eight GPUs. At step 300 it is 0.88 pp below three-pass and still above Pion on both matched seeds.

W2/Q4. Tuned baselines, uncertainty and task scope. On Qwen3-8B (seed 42, a 0.5x/1x/2x grid around 5×10−65\times10^{-6} plus one extra point per baseline), the best results are PASP 0.4861, Pion 0.4763 and Muon 0.4716. At 1.7B, tuned Pion reaches 0.4244 versus 0.460 for PASP. Tuning narrows the gap, and PASP stays ahead. We do not have the full 1.7B grid or best-rate 8B runs on seeds 43 and 44.

PASP has the highest mean in all five main settings, but the paired 95% intervals include zero in four of them, so we do not claim a statistically clear win everywhere.

After math RLVR, Qwen3-8B scores 0.480 on GPQA-Diamond, against 0.449 for Pion and 0.397 for Muon (198 questions, one sample each). This tests transfer only. We have not trained on non-math RLVR rewards.

Thank you for the detailed rebuttal. The additional controls and ablations substantially alleviate my concerns regarding spectral asymmetry and momentum-related confounders. However, two concerns remain:

  1. Since the routing variant provides the main evidence for equal-wall-clock advantages, could the authors verify that positive/negative gradient attribution remains correct after distributed gradient reduction?

  2. The routing variant improves accuracy over Pion by only ~0.8 percentage points at comparable wall-clock time, while requiring substantially more memory. This raises questions about its practical benefits.

Overall, the rebuttal has positively influenced my assessment, although the practical advantages of PASP remain only partially established. Given the additional evidence and clarifications, I am willing to raise my score.

Thank you for the follow-up and for reconsidering your score.

1. Sign attribution after distributed reduction. We checked. Attribution was not correct in the routing variant. All main results use three-pass PASP, which computes the two channels in separate backward passes and is not affected. In the routing code, FSDP2's reduce-scatter sums gradients over the eight ranks before each reduced shard is assigned to a channel by that rank's own response sign, so both channels held mixed gradients. We should have stated this more directly in our reply to you.

We have since built a corrected version that keeps the ranks in step. Every rank processes its positive responses first and its negative responses second, with zero-weight padding where a rank has fewer of one sign. Each reduce-scatter then sums gradients of a single sign. We repeated the earlier probe on the same two batches, with a fresh optimizer state, fixed rank and the same sketches. Errors are relative to three-pass PASP.

Final update, mean (range)Relative errorCosine
Old routing1.0244 (1.0068 to 1.0421)0.5085 (0.5035 to 0.5135)
Corrected routing0.1273 (0.1135 to 0.1411)0.9918 (0.9900 to 0.9936)
Three-pass, micro-batch order reversed0.1071 (0.1039 to 0.1102)0.9943 (0.9939 to 0.9946)

The last row runs three-pass twice and only reverses the micro-batch order. The final spectral update is sensitive to the order of floating-point sums, so even three-pass does not match itself exactly. Corrected routing is close to this level but slightly outside it. The reported routing runs used the old code, so we do not use them as evidence for PASP.

2. Practical benefit. We agree that the equal-time gain of the old routing runs over Pion is small. The clearer benefit of PASP is per rollout. Three-pass PASP reaches Pion's step-300 accuracy at about step 165, so it needs about 45% fewer rollouts for the same accuracy.

Corrected routing uses one backward pass. We timed 10 to 20 steps from the same step-300 checkpoint on eight H20 GPUs. The last row also allocates fallback optimizer state only for the fallback parameters, which cover about 15% of the weights.

MethodMedian step (s)Actor update (s)Peak GB/GPU
PASP, three-pass2118120335
PASP, corrected routing126346335
PASP, corrected routing, reduced state128245324

Corrected routing is 40% faster per step than three-pass, and the reduced state lowers peak memory from 35 to 24 GB. Pion's full-run medians are 1083 s per step and 20 GB (Table 7). Those come from different steps, so the comparison is approximate. PASP still costs more time and memory per step than Pion, and we do not yet have an accuracy run for the corrected variant.

In the revision we will replace the old routing code with the corrected one, report these fidelity, time and memory numbers, stop using the old routing accuracy as evidence for PASP, and state per-rollout efficiency as the main practical claim.

Reviewer_FXAtOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper finds that positive-advantage gradients in RLVR are more spectrally concentrated than negative-advantage gradients. PASP uses the positive gradient to define a subspace, splits the negative gradient relative to it, and applies spectral transformations to the resulting branches. On four primary math benchmarks with Qwen3-8B, PASP reaches a mean accuracy of 0.489 versus 0.470 for Pion. Its training is also more stable in the Qwen3-1.7B experiment, where recipes are transferred without retuning

Critical strengths
  • The optimizer design follows a measurable gradient property, which the authors probe across models and advantage estimators.

  • Ablations remove the sign split or residual truncation and compare alternative anchor sources; the positive-gradient anchor performs best in both matched-seed comparisons.

Critical weaknesses
  • The main accuracy results use an implementation requiring three backward passes. The faster single-pass routing version trails it by 0.0088 on average, and signed-gradient attribution under distributed reduction has not been fully validated.

  • Removing the sign split also changes a two-branch spectral update into a single-branch update, so that ablation does not isolate the effect of advantage sign.

  • The evidence centers on mathematical RLVR. Optimizers use their own recipes, and the local learning-rate checks do not establish rankings after equally extensive tuning.

Questions for authors
  • If two spectral transforms and similar compute are retained, how much of the gain remains when advantage signs are shuffled or replaced with a random partition?
  • In distributed training, how closely do the single-pass routing implementation’s signed gradients and final update match the three-pass reference?
  • How do PASP’s benefits and costs change with equal tuning budgets for Muon and Pion, and on non-math tasks?
3 comments

Thank you for the careful review.

W2/Q1. Advantage sign versus two branches. We did not run a random partition with both branches kept. Each variant needs a new 300-step run, which did not fit in the rebuttal period. The closest control keeps both spectral maps, the compute, the true-sign channels and the rank rule, and only swaps the positive anchor for a random basis of the same rank.

Qwen3-8B/GRPO variantSeedsMean accuracy
Full PASP42, 43, 440.489
No split (one spectral map)42, 43, 440.459
Two branches, positive anchor42, 430.4893
Two branches, random anchor42, 430.4716
Two branches, negative anchor42, 430.4706

The random anchor is worse on both seeds (by 1.54 and 2.00 pp) but still above the no-split mean. So part of the 3.0 pp gap to no split comes from having two branches, and part comes from anchoring on the positive gradient.

W1/Q2. Distributed routing fidelity. Not closely. We measured the eight-GPU FSDP2 code at the step-300 checkpoint of the seed-42 routing run, on two batches of 32 responses. In each trial both paths use the same batch, a fresh optimizer state and the same sketches.

QuantityRelative errorCosine
G+G^+ (per trial)0.0522 and 22.82850.9986 and -0.0194
G−G^-, mean (range)0.9183 (0.9105 to 0.9262)0.4305 (0.4136 to 0.4473)
Final update Δ\Delta, mean (range)1.0244 (1.0068 to 1.0421)0.5085 (0.5035 to 0.5135)

G+G^+ swings between trials because each batch has very few responses with nonzero advantage. The final update is stable, and it shows that routing does not reproduce three-pass. The reason is that FSDP2's reduce-scatter sums gradients over ranks before the code assigns each shard to a channel by that rank's response sign. Each channel then holds pieces of the mixed gradient. A correct version would split the channels before the reduction. We have not built or timed it.

We withdraw the claim that routing reproduces three-pass PASP, and we do not claim that it is independent of advantage sign. In the revision we will add this measurement to Appendix C.1, note in Appendix B.2 that the current code does not meet the distributed condition, and present routing in Section 6.3 as a separate, faster variant. All main results use three-pass PASP.

W1/W3/Q3. Tuning, cost and non-math tasks. With local tuning PASP still leads. On Qwen3-8B/GRPO (seed 42, a shared 0.5x/1x/2x grid around 5×10−65\times10^{-6} plus one extra point per baseline), the best results are PASP 0.4861, Pion 0.4763 and Muon 0.4716. This is a local check, not an equal tuning budget. At 1.7B, tuned Pion reaches 0.4244 versus 0.460 for PASP. We do not have best-rate comparisons across seeds.

The cost is real. At equal wall-clock time, three-pass PASP does not beat Pion. Times below are from eight H20 GPUs.

MethodMedian step (s)Peak GB/GPUStep (cumulative h)Mean accuracy
PASP, three-pass221334140 (88.0)0.4579
PASP, routing136735240 (92.4)0.4780
Pion108320300 (91.5)0.4697
Muon113720300 (97.8)0.4556

Three-pass PASP is 1.2 pp below Pion at about the same time. Its advantage is per rollout, since it reaches Pion's step-300 accuracy at about step 165. The routed variant is 38% faster per step than three-pass. It scores 0.4752, 0.4777 and 0.4811 at step 240 on seeds 42, 43 and 44 (seed 44 has no step-300 result), above both baselines at about the same time, but it uses more memory. At step 300 it is 0.88 pp below three-pass.

After math RLVR, Qwen3-8B scores 0.480 on GPQA-Diamond, against 0.449 for Pion and 0.397 for Muon (198 questions, one sample each). This tests transfer only. We have not trained on non-math RLVR rewards.

Three results finished after our first reply.

W2/Q1. Advantage sign versus two branches. We ran the random-partition control on seed 42 with three-pass PASP. It keeps both branches, the rank rule and the compute for 300 steps, and assigns responses to the two branches at random instead of by advantage sign.

Qwen3-8B/GRPO variantSeedsMean accuracy
Full PASP (true sign)42, 43, 440.489
Two branches, random partition420.464
No split (one spectral map)42, 43, 440.459

The random partition is 2.5 pp below full PASP and close to no split. It is lower on all four benchmarks (MATH500 0.796 vs 0.808, AIME24 0.258 vs 0.298, AIME25 0.240 vs 0.260, OlympiadBench 0.561 vs 0.590). Against seed 42 of full PASP (0.4861), the gap is 2.2 pp, about four times the seed standard deviation of PASP (0.0058). So most of the gain needs the true sign, not only two branches. This is one seed, and the other rows are three-seed means.

W1/W3/Q3. Tuned baselines across seeds. We reran the best rates from the seed-42 grid on seeds 43 and 44. PASP keeps its original rate. The table gives three-seed means ± sample standard deviation.

ModelSeedsPASPPionMuon
Qwen3-8B42, 43, 440.4890 ± 0.00580.4726 ± 0.00640.4701 ± 0.0044
Qwen3-1.7B42, 43, 440.463 ± 0.00610.4252 ± 0.00860.4145 ± 0.0063

PASP is highest on every seed in both settings. The rates were chosen on seed 42 only, so this is still a local grid, not an equal tuning budget.

W1/Q2. Corrected routing. We built the version that splits the channels before the reduction. Every rank processes its positive responses first and its negative responses second, so each reduce-scatter sums gradients of a single sign. We repeated the earlier probe on the same two batches.

Final update vs three-pass, mean (range)Relative errorCosine
Old routing1.0244 (1.0068 to 1.0421)0.5085 (0.5035 to 0.5135)
Corrected routing0.1273 (0.1135 to 0.1411)0.9918 (0.9900 to 0.9936)
Three-pass, micro-batch order reversed0.1071 (0.1039 to 0.1102)0.9943 (0.9939 to 0.9946)

The last row runs three-pass twice and only reverses the micro-batch order, so it shows how much rounding alone changes the update. Corrected routing is close to that level but slightly outside it. It runs 40% faster per step than three-pass. All main results still use three-pass PASP.

Thank you again for your careful review. Your questions on routing and the controls made the paper more complete. As the discussion ends soon, we wanted to make sure our follow-up above reached you, and we are happy to answer any remaining question.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.