acceptodds
Withdrawn from ICLR 2027

Finite-Step Adaptation Decisions: Loss Responses and Retained Directions

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A restricted update can match a full update's loss while omitting directions that affect another execution or target. We study when scalar observations certify those omitted effects. For a radial family containing ordinary and normalized gradient descent, we derive a response law at every finite derivative order. Its leading matrix depends affinely on one gain slope. This transfers two explicit containment designs across the slope interval . The normalized endpoint also admits sharp, realizable cancellation examples. A second observation uses a normal-gradient correction at a saved state. Its loss decrease gives a positive energy measurement. We characterize four-prefix designs by an exact rank condition and derive a finite transfer bound for unobserved steps. A finite residual certificate then allows changing batches, refreshed spaces, and separate validation losses. Reference derivative actions yield a tighter centered interval at additional cost. On stored RoBERTa feature-head trajectories, that interval certifies a nontrivial tolerance decision. The results identify how response structure, observations, and execution cost determine retained-direction adequacy.

Primary area:
optimization
Keywords:
optimization, parameter-efficient adaptation, subspace optimization, low-rank adaptation, gradient methods, finite-step certification, retained directions, observability

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_AsRmOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper tries to answer the question whether restricting an optimizer to a smaller update space changes the result of a finite run by more than a chosen tolerance. It shows how omitted directions can be hidden by agreement in the final training loss, and studies which loss measurements can reveal them. The main results are an affine dependence of the leading loss response on the normalization parameter, and an exact rank condition for a four-checkpoint observation scheme. The paper then uses residual bounds to control the difference between full and restricted runs, including on a separate evaluation objective.

Critical strengths
  1. The paper addresses an interesting question. Matching the first gradient or the final training loss does not guarantee that a restricted run preserves the outcome of full optimization. The examples make this distinction clear.

  2. The theoretical results are more than just standard error propagation. I found the gain-affine response law and the four-checkpoint rank condition interesting. The constructions showing that some observation designs miss nonzero omitted directions also help explain why the choice of measurements matters.

Critical weaknesses
  1. The main experiment does not test the four-checkpoint transfer method. The experiment changes batches and the update space, so it obtains residual information at every stage. The scalar and direct-norm bounds are inconclusive at the chosen tolerance. The successful decisions come from the additional linear correction, which uses a Hessian–vector product at each step. This is a useful result, but it leaves open whether the new observation designs give useful finite certificates.

  2. The finite-transfer assumptions seem difficult to satisfy outside controlled examples. The bound requires a fixed loss and projector, exact retention of E3E_3, and derivative bounds through order five. Its tightness also depends on the calibration matrix and Taylor remainder. The asymptotic results are clear, but it is hard to tell the range of budgets over which the resulting finite bound would be informative.

  3. The benefit over running the full comparator is unclear. In the successful experiment, each candidate requires 64 additional Hessian–vector products, alongside the reference gradients and probes. Running the full comparator takes 64 gradient steps, and its result can be reused across candidate spaces. The paper gives operation counts, but does not establish when certification offers a useful advantage in time, memory, or access requirements.

  4. The current presentation is hard to follow due to the following reasons:

a) The abstract and introduction rely on broad descriptions without clearly stating what is measured. It is very hard for the readers to know what is going on. The roles of the “analyst,” “executor,” and “auditor” are not clear from reading.

b) Many phrases such as “normal norm” and “probe losses” are unnecessarily opaque. Explicit mathematical quantities or plain descriptions will help.

c) Some notation is not defined or is used before its role is explained. For example, Section 3 introduces BB-dependent iterates uˉN(B)\bar u_N(B) before specifying how BB determines the step sizes. Theorem 2 should explicitly identify ((a,b,c,d)) as the four-step profile α\alpha.

In summary, the theory in the paper may be interesting, but its practical implications are not clear and the presentation is hard to follow.

Questions for authors
  1. Can authors give a finite example using the four-checkpoint transfer bound, including the actual remainder constants, calibration conditioning, and resulting target interval?

  2. Can the transfer bound accommodate a small amount of leakage from the lower-order space E3E_3? It is okay to just provide some comments rather than proof.

  3. When is the centered certificate preferable to running one full comparator shared across candidates? What about time and memory comparison on the existing setup?

Reviewer_MVXbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

As far as I can tell, the paper compares gradient descent on a loss FF with gradient descent restricted to a subspace (projector WW), for example a low-rank adaptation. It asks when scalar loss values can certify that the two runs give similar values of a target VV after NN steps. Theorem 1 expands the difference of terminal losses in powers of the step budget BB, and shows that specific step-size profiles detect missing directions as B→0B \to 0. Theorem 2 does the same for loss decreases along the projected-out gradient at saved states. Proposition 1 propagates bounds on the projected-out gradient through the iterations to bound ∣V(xN)−V(yN)∣|V(x_N) - V(y_N)|. The experiment trains a logistic regression head on fixed RoBERTa-large features with 64 training examples and checks which subspace dimensions are certified.

Critical strengths

None.

Critical weaknesses
  1. The paper is very hard to follow. I could not determine the problem being solved until Section 3.
  • The manuscript reads like unedited machine-generated text: dense coined terminology, sentences that are individually grammatical but do not convey a clear argument, and unexplained numerical constants. Combined with the statement that generative AI was used to develop derivations and write proofs (l. 490–491), I could not place confidence in the 39-page appendix.
  • The abstract and introduction rely on undefined terms: "scalar loss agreement", "loss observations", "four-prefix observation designs", "finite transfer", "positive energy measurement", "target-bias guarantees". They never say what is being optimized, by which method, or why.
    • The main text keeps adding new vocabulary ("auditor", "executor", "programs", "profiles", "words", "checkpoint reports") instead of stating the setting: gradient descent versus projected gradient descent on a subspace.
    • The main text is 9 pages, and the derivations take a 39-page appendix. The AI use statement says generative AI was used for "developing and checking derivations and writing proof arguments" (l. 490–491). Given the presentation, I was not able to verify these derivations.
  1. The certificates do not solve a practical problem.
    • Every observation requires the full computation that the restriction is meant to avoid. The terminal response in Theorem 1 needs the full execution uNu_N. The checkpoint report in Theorem 2 and Proposition 1 needs P∇F(y)P\nabla F(y), which requires the full gradient at every probed state (l. 125–131). The experiment uses 64 normal probes per candidate (l. 466). With these, one can simply run the full update.
    • The paper itself notes that when the norm ∥P∇F∥\|P\nabla F\| is available, it is a more direct input than the scalar probes (l. 41–42, l. 467). Computing the probe makes the norm available, so the scalar-observation theory of Theorems 1–2 seems unnecessary.
    • Theorems 1–2 are asymptotic in B→0B \to 0 and rely on specific step profiles (e.g., (12,7,58,21,2)/100(12, 7, 58, 21, 2)/100). They are not used in the experiment.
Questions for authors
  1. What is a concrete setting where the certificate is cheaper than running the full update it is compared against? Please state the cost of the certificate relative to one full gradient step.

  2. Is there any experiment, beyond the logistic head on 64 examples, in which the scalar-observation certificates of Theorems 1–2 lead to a decision?

Reviewer_DbpsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies whether allowing updates only in a chosen subspace preserves the outcome of updating all parameters. Its response analysis covers ordinary gradient descent and normalized variants. Starting from the same point, the two runs produce a restricted state uˉN\bar u_N and a full state uNu_N after NN steps. The goal is to use observed loss values to certify that ∣V(uˉN)−V(uN)∣≤τ,|V(\bar u_N)-V(u_N)|\le\tau, where VV is an evaluation loss, possibly different from the training loss, and τ>0\tau>0 is an acceptable difference chosen in advance.

The authors show that matching the final training loss can hide differences that matter for evaluation. As the total step budget tends to zero, they characterize which combinations of loss measurements detect whether certain directions generated by derivatives of the training loss are excluded. Under additional regularity assumptions, they also derive finite bounds for ordinary GD/SGD by measuring loss decreases from corrections along excluded directions and tracking how update differences accumulate. An experiment with a logistic classifier on fixed features illustrates the resulting certificates.

Critical strengths

1) The example in Section 3 clearly shows that equal final training losses can hide differences on a separate evaluation loss. The gain-affine response law gives a common analysis for ordinary, normalized, and smoothly normalized gradient descent, with containment tests valid across the gain-slope interval [0,1][0,1].

2) The four-checkpoint rank condition is necessary and sufficient within the stated observation class. When it fails, the authors construct losses whose omitted directions go undetected. Equal and geometric step profiles fail this condition, while the authors provide an explicit profile that detects all four omitted derivative components considered in the theorem.

3) The finite certificates account for changing batches, refreshed subspaces, inexact updates, and numerical errors. They distinguish between three outcomes: the restricted run is within the chosen tolerance, it is outside the tolerance, or the available bound is too loose to decide.

Critical weaknesses

1) It is unclear why scalar loss reports are needed. To make a checkpoint correction, the executor already computes the omitted gradient P∇F(y)P\nabla F(y). It could report its norm directly, without extra loss evaluations. The authors acknowledge this, but I would like to see a concrete example where reporting loss values is necessary or more useful.

2) Section 7 gives a useful comparison of scalar, direct-norm, and centered bounds. However, it does not test the finite transfer bound (10). This bound uses four checkpoints to bound omitted gradients at other steps, while keeping the loss and subspace fixed. In the experiment, batches and subspaces change, every step is probed, and successful decisions require the extra linear correction. A small example would help show whether bound (10) remains useful at steps that were not probed, after accounting for conditioning and Taylor errors.

3) The experiments cover only a small, favorable setting: a strongly regularized logistic classifier, 64 training examples, and two batch streams. Although there are 1024 parameters, the full trajectory stays in a space of dimension at most 64. For LoRA, the error term in (69) accounts for the difference between updating the factors and taking a projected full-gradient step. How large is this term in practice, and does it make the bounds too loose? The paper lists the operations needed to compute the certificate. However, comparing the total computation time or cost with running the full update sequence would help show when certification is worthwhile.

Justification of the score I lean toward weak rejection. I find the theoretical results interesting, but I am not yet convinced why scalar loss reports are needed. The experiment also does not test whether four checkpoints give useful bounds at other steps. A concrete use case and an example showing that these bounds remain useful after accounting for the remainder terms and the cost of computing them could change my assessment.

Questions for authors

1) Can you give a concrete application where scalar loss reports are needed or preferable to reporting the norm of the omitted gradient?

2) Can you show an example where the four-checkpoint transfer bound in (10) gives useful certificates at unobserved steps, after accounting for conditioning and Taylor remainders?

3) When is computing the certificate worthwhile compared with running the full comparator, including the cost of probes and linear corrections? For LoRA, how does the update defect in (69) affect the usefulness of the certificate?

4) Which steps in the proofs of Theorems 1 and 2 require new arguments beyond existing B-series expansions, rank arguments, and residual-based error bounds?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.