acceptodds
Under review as a conference paper at ICLR 2027

Selecting Spectral Memory from the Original Gradient

Abstract

Selecting a spectral memory on the gradient used to fit it creates a dependence cost; independent evaluation avoids this dependence only by fitting candidates on noisier views. We keep every memory on the original observation and aggregate its polar action using Stein's unbiased risk estimate (SURE), conditional on the past. We establish the rank-boundary regularity and sequential risk comparison needed for this nonlinear setting. For strictly rectangular matrices, a predictable safety action makes the selected mixture continuously differentiable even at fitted rank loss, although individual corrected scores can have infinite variance. This justifies an exact, computable selection-response term and an expected-risk oracle against the unperturbed candidate rules under predictable feedback. The response is retained explicitly, not assumed small. Spectrum-saturated bounds connect the comparator risk to noise and memory bias without a lower bound on the true singular values. The leading-one oracle applies to the mixture and randomized isometric output; deterministic polar rounding has a factor-two guarantee. Across 272 fixed-plan synthetic streams, the rounded rule improves on one-way splitting in most cells, while symmetric splitting remains competitive and two zero/near-zero second-scale configurations incur substantial adaptation costs. The results identify when original-data fitting is useful and what dependence must still be charged.

Primary area:
learning theory
Keywords:
spectral memory, Stein unbiased risk estimation, online aggregation, polar decomposition, oracle inequalities

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_4Yq5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

As far as I understand, the paper tries to study the gradient-based optimization algorithms from the perspective of how to mix previous gradient informations. The paper designs an algorithm in Algorithm 1 and claims that it is a "safeguarded conditional SURE aggregate over spectral memories". The paper provides guarantee on some expected aggregated distance the author defined as "D(S,P)=∣∣S∣∣∗−⟨S,P⟩D(S, P) = ||S||_* - \langle S, P\rangle". The bounds are in terms of a term BarawB_a^{raw} which they defined also in terms of the distance metric. It seems that the experiment sections provides experimental results on a certain setting that is hard to decode from the writing of the main text, which claims to demonstrate the benefit in terms of the "mean cumulative latent descent loss".

Critical strengths

The paper seems to be studying an important question of how to mix the history in gradient-based optimization algorithms. There is a provided algorithm that, according to the author, seems to provide interesting theoretical and empirical result.

Critical weaknesses

The writing of the paper raises significant concern regarding its clarify and soundness. To start, many notations are not defined. To name a few:

  1. polar(G)\text{polar}(G)
  2. m,rm ,r at the beginning of Section 2.
  3. NN in Eq. (3)
  4. j<kj < k in Eq. (4), possibly due to the lacking of a parathesis.
  5. \mathcaFt−1\mathca{F}_{t-1} is defined as "Let Ft−1\mathcal{F}_{t-1} contain the past" with no clear context of what is "the past"

Terms are used without explicit explanation or definition:

  1. "conditional SURE aggregate" in the intro, with no definition or citation
  2. "candidates", "latent signals" in the intro
  3. "prior" and "temperature" before Eq. (6).

Statement of Algorithm 1 is unclear:

  1. "update every raw memory" need specification
  2. "include the predictable safety action" need specification

Theorem statements are not clear: in Theorem 2, it requires "preceding assumptions" but unclear what are the "preceding assumptions".

Experimental setting is not clearly explained. In Section 5, the first sentence is "The fixed final plan comprises 272 synthetic streams in 16 cells.", but it is unclear from the context what is "final plan", "stream", or "cells" Proofs have many unjustified points, to name a few:

  1. Lemma B.1 "Its expected inverse radius is no larger than the centered value."
  2. Proof of Lemma 1 "the internal symmetric, internal skew, and external coefficients are. …\dots All are bounded by 1/smin."

The above issues are not just clarity issues. They are critical to make the paper's contribution and soundness possible to be evaluated by a human reader. From the current status of the paper, I unfortunately do not believe that the paper is possible to be evaluated in terms of the soundness or the quality. Therefore, simply replying to the example issues above is not sufficient. I recommend:

  1. Re-writing the intro to clearly state the problem being tackled. Make sure terms are sufficiently explained.
  2. Create a detailed mathematical setup and make sure that every notation is clearly defined.
  3. Make formal statement of the assumptions.
  4. Clearly write out the algorithm, without using overly vague or general statements.
  5. Check proofs for insufficiently justified statements.
Questions for authors

I don't think clarifying questions is sufficient to affect my assessment at this point.

Reviewer_qLD7OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(−1)
Read review
Paper summary

The paper studies online selection among polar-normalized updates formed from the current gradient, full history, and exponential moving averages. It assumes additive Gaussian gradient noise with known variance. The method fits all candidates on the original observations and aggregates them using Stein’s unbiased risk estimate (SURE). The analysis gives an expected descent-loss comparison with an explicit correction for selection, together with spectral bounds for the candidate rules. Synthetic matrix streams compare the method with fixed memories and Gaussian splitting controls.

Critical strengths

The paper gives a valid risk comparison for selecting nonlinear memories on the same gradient observations. The result allows rank-deficient true signals and dependence on past observations and actions. It identifies the additional cost of selecting among already fitted candidates, rather than treating unbiased candidate scores as sufficient. This is a useful extension of the existing risk-estimation framework to the stated polar-memory setting.

Critical weaknesses
  1. The main bound does not establish effective adaptation for the fitted memories of interest. Theorem 3 and Corollary 4 retain a selection-response term whose size is not controlled for the EMA and full-history candidates. Being able to compute this term does not establish that the selector stays close to a good candidate. The matching rate in Section 4 concerns a separated library of actions independent of the current observation, where the extra response vanishes; it does not resolve the fitted-memory problem. The authors acknowledge this distinction. Nevertheless, controlling the adaptation cost is central to the proposed selection task, and it remains unresolved in the main setting.

  2. The advance over existing SURE aggregation is narrow. The weighting rule is exactly SURE exponential weighting, and the selected-risk identity specializes [Dalalyan and Salmon (2012)] as the paper explains. The candidate-wise Stein identity in Lemma 1 also follows by applying Hansen (2017, Proposition 1) to the rectangular polar map: it is bounded, smooth outside a rank-deficient set of codimension at least two, and has nonnegative divergence. The remaining contribution is the safeguard and its sequential risk analysis, but without an informative adaptation bound its statistical significance is limited.

  3. The experiments do not establish a compelling advantage over the strongest implemented control. The method has lower mean loss than one-way splitting in 14 of 16 settings, but lower mean loss than symmetric splitting in only 7 of 16. These are not significance counts. In the zero-second-scale setting, its loss is 1.242, compared with 0.550 for symmetric splitting and 0.118 for full history. The changing-signal examples show that adaptation can help, but symmetric splitting achieves similar performance in the three-segment example. All experiments use small synthetic matrix streams with known Gaussian noise; there is no optimization experiment showing a further benefit. The evidence supports a valid alternative, but not a substantial improvement that offsets the limited theoretical advance.

The contribution is a specialized extension of SURE risk. The main theorem does not establish a small adaptation cost for the fitted memories, and the experiments do not establish a convincing advantage over the strongest implemented control. Together, these limitations leave insufficient significance for acceptance.

Questions for authors
  1. Does the paper establish a regime for data-fitted EMA or full-history candidates in which the selection-response term is bounded through assumptions on signal drift, spectrum, and noise, rather than retained as an additional budget? Please distinguish such a result from the current-data-independent library case.

  2. What advantage over symmetric splitting is supported by the existing results? Are there settings with a statistically reliable improvement, or a computational advantage at comparable loss? This would help assess the practical significance of retaining the original-data candidates.

Reviewer_ZmwPOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors study how to choose between different averages of noisy matrix valued gradients, including the current observation, the full history average, and exponential moving averages. Each average is converted into a normalized matrix update using its polar factor.

The method proposed by the authors use Stein's unbiased risk estimate (SURE) to correct the scores used to compare these updates. They also study how choosing between updates using the same noisy observation affects the error. An additional update based only on past observations helps to handle cases in which normalizing the candidates at that time becomes unstable. The authors provide theoretical bounds to compare the method with individual averaging rules, and include an additional term for the cost of choosing between them. The authors test the method on synthetic matrix sequences, comparing it with fixed averaging rules and methods that create two noisier versions of each observation for building updates.

Critical strengths

I find the general question of how much gradient history to retain potentially useful, indeed averaging reduces noise but can become misleading when the gradient changes. I am less able to assess the specialized theoretical contribution.

Critical weaknesses

My main concern is the presentation, which I personally found prohibitively difficult to follow despite substantial effort. The specialized theory is outside my expertise, but I think the way the paper introduces the task, notation, and results also makes it unnecessarily difficult for readers outside this subfield. I list all my concerns below.

  1. The abstract immediately introduces words like "spectral memory," "polar action," "rank-boundary regularity," "selection-response," and "spectrum-saturated bounds" before explaining the concrete problem or even explaining what these words mean. Phrases such as "leading-one oracle" and "deterministic polar rounding" also require background that is not provided before they are used. The introduction gives a brief motivation about gradient averaging, but then moves immediately to technical formulas. The dimensions in equation 1 are specified later, the polar operation is used before its meaning and purpose are explained, and St⁡(m,r)\operatorname{St}(m,r) in section 2 is defined in the appendix. These are a few examples of a more general problem I encountered throughout the manuscript, where technical terminology and notation are not introduced, or without sufficient explanation. I strongly believe that the main text requires substantial rewriting to improve clarity and overall presentation.

  2. I had the same difficulty following section 2 and the algorithm. Terms such as "affine-current," "predictable," and "divergence" are used alongside candidate scores, a safety action, a prior, and a temperature. The later use of "fixed share" adds another component before I could clearly understand the earlier steps. These terms may be familiar to specialists, but I found their roles insufficiently explained here. I had particular difficulty distinguishing how the candidate updates are constructed, how they are compared, and how the final update is chosen. The paper should also explain which corrections actually change the algorithm and which only appear in the error analysis. In my opinion, again, this requires a substantial reorganization and rewriting, rather than local language corrections.

  3. The related work discussion cites other established statistical methods, but the comparison is too compressed for me to assess what has been added. I would appreciate a clearer explanation of what the cited results already establish, why they cannot be applied directly here, and what additional conclusion the new analysis provides compared to a broader literature review. Moreover, the main bound found by the authors includes an additional cost that the authors acknowledge can be large. I found it difficult to understand when the result therefore guarantees performance close to a good averaging rule.

  4. The experiments use synthetic sequences, and the method has mixed results against fixed averaging rules and the splitting alternatives. I would like a clearer explanation of when the proposed method helps, when it does not, and how these outcomes relate to the additional cost in the theory bound provided. I think the authors should provide a more direct interpretation of the results provided.

For the reasons I presented in this section, I lean towards weak rejection. This is mainly because the current presentation prevents me from reliably assessing the method and the significance of its contribution. I think substantial improvements in the introduction, definitions, organization, and explanation of all the results are necessary. I have alerted the AC to my limited subject expertise and I selected confidence 1.

Questions for authors

I list my main questions below in a synthetic way. Please refer to the weaknesses section above for the more detailed and broader discussion on the clarity of the manuscript.

  1. Can the authors explain the task and algorithm through a simple example, defining the memories, polar update, and loss before introducing the statistical corrections?
  2. Which corrections are used to choose the update, and which only appear in the error analysis? Which guarantee applies to the output used in the main experiments?
  3. What additional guarantee is obtained beyond the methods cited by the authors, and when is this extra cost of choosing between memories small enough for that guarantee to be useful?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.