acceptodds
Under review as a conference paper at ICLR 2027

When Does Gradient Fidelity Support Low-Rank Optimization Decisions?

Abstract

Low-rank optimization restricts model updates to a limited set of directions. Gradient fidelity measures the fraction of the current gradient's energy captured by a candidate. We ask when higher fidelity justifies a candidate preference, accounting for update implementation and subsequent training. With candidate geometry fixed, projected descent preserves fidelity rankings, whereas a zero-moment LoRA+AdamW step can reverse them. In the studied regime, its local first-order score is approximately a weighted score of projected-gradient coordinates rather than the squared-energy score used by fidelity. Matched controls isolate adaptive coordinate normalization as a source of the zero-moment reversals. Using projected descent to remove implementation-induced reordering, we compute a first-order future-sensitivity score by differentiating a common reference continuation before evaluating candidate-specific terminal outcomes. The score identifies terminal winners in states in each of two language cohorts and vision states, versus at most per cohort for fidelity. Retrospectively, gains over lowest-fidelity and post hoc best-fixed rules are smaller than gains over fidelity. For a fixed finite candidate set, first-order score differences depend only on future sensitivity along candidate differences. A signed gain and residual bound give a sufficient condition for first-order preservation or reversal of local pairwise ordering. The condition certifies 26 of 60 evaluated pairs as first-order reversals; all 26 also reverse the local ordering in separate terminal evaluations. This geometry also specifies how to assess approximate future sensitivity by comparing induced pairwise-score errors with the reference first-order gaps they could overturn. Together, these results distinguish gradient capture from candidate value and show how update implementation and first-order future valuation can re-rank candidates.

open until 14 Dec 2026

est. 32% chance this paper gets accepted at ICLR 2027.

Reject 68%Accept 32%

What do you think this paper will get?

All positions stay anonymous.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.