acceptodds
Under review as a conference paper at ICLR 2027

Spectral-Driven Adaptive-Rank Optimizer for Large Language Model Reinforcement Learning with Verifiable Rewards

Abstract

Reinforcement learning with verifiable rewards (RLVR) is reshaping how large language models (LLMs) acquire reasoning capabilities, yet the optimizers used remain largely inherited from teacher-forcing training. This raises a fundamental question: does RLVR induce a distinct optimization geometry that calls for a different optimizer? We investigate this question through the spectral structure of LLM gradients and find that gradient energy progressively concentrates into a shrinking effective subspace, which is characteristic of RLVR, as it is absent under supervised fine-tuning (SFT) and is largely removed when the verifiable reward is replaced by a random one. Moreover, the rank required to capture a fixed fraction of gradient energy steadily decreases over training, revealing that the effective dimensionality of useful updates itself evolves with training. These findings motivate a new optimization principle: rather than prescribing the update rank externally, an optimizer should adapt its rank to the evolving spectral structure of the gradient. We introduce **Ares** (**A**daptive **R**ank-**E**ffective **S**pectral Optimization), which estimates the effective update rank from the gradient's spectral energy distribution, tracks its evolution, and confines the gradient signal driving each update to the resulting low-rank subspace. Across LLMs of different scales, Ares improves accuracy over widely used optimizers and generalizes across RLVR algorithms and tasks. Results suggest that effective optimization may require optimizers to be designed around the geometry and dynamics of the training process.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Reinforcement Learning with Verifiable Rewards, Optimizer

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_HDpzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the spectral properties of RLVR gradients and observes that their effective rank decreases during training. Based on this observation, the authors propose Ares, which estimates the effective rank from the gradient spectrum and dynamically adjusts the rank of reconstructed gradients before applying AdamW. Experiments on Qwen3 models of different scales show improvements over several optimizers on mathematical reasoning benchmarks.

Critical strengths
  1. The observation that RLVR gradients become increasingly spectrally concentrated is interesting, and the proposed optimizer is well motivated by this finding.

  2. The experimental results are promising, showing consistent improvements over AdamW across different model scales and RLVR algorithms.

Critical weaknesses
  1. The benefit of adaptive rank is not sufficiently demonstrated. In Table 4, removing dynamic rank only reduces average accuracy from 27.2 to 26.6 on Qwen3-1.7B and from 38.5 to 37.6 on Qwen3-4B. This suggests that much of the improvement may come from low-rank gradient filtering rather than rank adaptation itself. Comparisons with fixed-rank variants under the same optimizer would better support the main contribution.

  2. The spectral interpretation needs further justification. Although Ares reconstructs low-rank gradients, the subsequent AdamW momentum and elementwise preconditioning do not necessarily preserve their rank. It is therefore unclear whether the actual parameter updates remain in the claimed low-rank subspace. Also, no multiple-seed results are reported, making the relatively small gains from adaptive rank difficult to assess.

Questions for authors
  1. How does Ares compare with fixed-rank variants using different ranks but otherwise identical settings?

  2. Does the low-rank structure persist in the actual parameter updates after AdamW preconditioning?

  3. Are the improvements from dynamic rank consistent across multiple training seeds?

Reviewer_LWMNOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an interesting optimizer for RLVR. The proposed approach is (i) use randomized SVD to identify the leading singular directions and then reconstruct a low-rank approximation of the gradient. (ii) select the effective rank of gradient under this approximation (keep an EMA over it, since it might be nooisy step-to-step) (iii) truncate the original gradient to project to this lower-dimensional subspace and carry this gradient over to adamW.

Overall I think this is pretty interesting, given the low dimensional nature of RLVR this seems plausible and timely.

However the authors have not cited many prior work, that are directly related and lay ground work on optimization for RLVR. See weakness

Critical strengths

I like the paper's motivation. The paper highlights the per-step gradient's low-dimensionality, and uses it in a clever way for optimizer design. I would love it if authors can further add some discussions on hwo this observation might motivate LoRA choices or even model merging.

Further the results are convincing and strong. The ablations are convincing too. I think table 7 is worth being part of the main paper

Critical weaknesses
  1. Their optimizer requires an approximate SVD for every 2D gradient matrix at every optimizer step, using 20 power iterations. That's substantial additional computation. -- I would appreciate it if the authors can please give a fair comparison across optimizers (at least give fair numbers with respect to adamW). On X axis keep wall clock time and in y validation performance.

  2. The authors while tackling a very timely problem, have not cited any relevant prior work - the direction (optimizers for RL while nascent is not new) please see the following papers, as I believe they are quite important prior work. [1] [2] [3] [4] . Especially l;ooking at the spectrum and drawing insights have been explored well in [2] and [4].

[1] Reinforcement Learning Finetunes Small Subnetworks in Large Language Models

[2] The Path Not Taken: RLVR Provably Learns Off the Principals

[3] Do We Need Adam? Surprisingly Strong and Sparse Reinforcement Learning with SGD in LLMs

[4] ISO: An RLVR-Native Optimization Stack

Questions for authors

See weaknesses please.

Reviewer_S9TuOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Ares, an optimizer designed for RLVR. The authors find that gradients during RLVR training exhibit specific low-rank characteristics—a phenomenon absent in SFT. Based on this finding, Ares uses RBPI to approximate the leading singular subspace of gradient matrices. It then implements an adaptive low-rank selection method to retain the principal components of the gradient matrix for gradient reconstruction. These reconstructed gradients are subsequently used in conjunction with the AdamW optimizer to update model.

In the main experiments, conducted on math tasks using GRPO with Qwen3-1.7B-Base, 4B, and 8B, Ares outperformed the AdamW optimizer by 5.0, 2.7, and 2.3 percentage points, respectively. The paper also conducts ablation studies, analyses of gradient spectral dynamics and optimization overhead, and additional evaluations across RLVR algorithms and downstream task domains.

Despite the notable performance improvements, the paper’s underlying motivation and theoretical rationale are flawed, failing to provide a sufficiently robust justification for its contributions.

Critical strengths
  1. Insightful motivation. The paper examines the differences between SFT and RLVR through the spectral structure of gradients, offering an interesting perspective on optimizer design for LLM post-training. The comparison with SFT and the random-reward control are useful attempts to distinguish the roles of on-policy sampling and informative reward signals in shaping gradient spectral dynamics.

  2. Consistent empirical improvements. Ares achieves higher average accuracy than AdamW across the tested model scales and multiple RLVR algorithms, with gains on mathematical reasoning benchmarks and additional downstream evaluations. These results provide encouraging evidence that its benefits extend beyond a single experimental setting.

Critical weaknesses
  1. The central motivation is inadequately supported. Figure 2 shows that SFT gradients have higher spectral concentration than RLVR. The paper does not explain this observation with its characterization of RLVR-specific low-rank structure. An increasing concentration trend alone does not establish the claimed distinction.

  2. The key optimization principle remains unjustified. Spectral concentration does not imply that low-energy gradient components are uninformative or harmful. The paper doesn't provides sufficient evidence that discarding these components improves optimization rather than removing useful learning signals.

  3. The experimental analysis is insufficient to resolve these concerns. Although the experimental results indicate performance gains, the study fails to isolate the true underlying causes of these improvements or effectively validate the proposed mechanisms.

Questions for authors
  1. [Important] Figure 2: metric definition and interpretation. How is the mean top-32 energy ratio computed? What training steps do the checkpoint indices represent? SFT consistently has a higher ratio than RLVR, indicating greater concentration in the leading 32 components under this metric. This suggests that the energy in SFT aligns more closely with low-rank characteristics. How to understand the difference between "absolute concentration" and the "evolution of concentration throughout the training process"?

  2. [Important] The rationale for truncating tail components. What evidence supports the claim that removing low-energy singular components improves optimization? Spectral concentration alone does not establish that these components are noise rather than useful learning signals. Could the authors provide empirical or theoretical analysis to explain why truncation is beneficial?

  3. Rank-selection alternatives. What are the exact definitions of the Linear and Threshold in Figure 6, and do they use the same temporal smoothing as Ares? I'd like to see the rank-selection trajectories for these strategies similar to Figure 5. Besides, Table 3 indicates that removing Dynamic Rank leads to only a minor performance drop, yet there are significant performance disparities among these different strategies. Why?

  4. Definition of “Adaptive LR.” The paper doesn't specify what is w/o Adaptive LR ablation.

  5. Figure 7: reward definitions and complete training curves. What reward does each panel refer to? Experment settings specifies 300 training steps, and I would like to see the full reward trajectory over these 300 steps, as well as the accuracy throughout the training process.

After the discussion, I have raised my score from 1 to 2.

1 comment

I appreciate the authors' patient responses, which addressed most of my questions; however, I remain somewhat uncertain about the value of low-energy gradients in RLVR and SFT. Due to the additional experiments and explanations provided by the authors, I will consider raising my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.