- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper tries to answer the question whether restricting an optimizer to a smaller update space changes the result of a finite run by more than a chosen tolerance. It shows how omitted directions can be hidden by agreement in the final training loss, and studies which loss measurements can reveal them. The main results are an affine dependence of the leading loss response on the normalization parameter, and an exact rank condition for a four-checkpoint observation scheme. The paper then uses residual bounds to control the difference between full and restricted runs, including on a separate evaluation objective.
Critical strengths
-
The paper addresses an interesting question. Matching the first gradient or the final training loss does not guarantee that a restricted run preserves the outcome of full optimization. The examples make this distinction clear.
-
The theoretical results are more than just standard error propagation. I found the gain-affine response law and the four-checkpoint rank condition interesting. The constructions showing that some observation designs miss nonzero omitted directions also help explain why the choice of measurements matters.
Critical weaknesses
-
The main experiment does not test the four-checkpoint transfer method. The experiment changes batches and the update space, so it obtains residual information at every stage. The scalar and direct-norm bounds are inconclusive at the chosen tolerance. The successful decisions come from the additional linear correction, which uses a Hessian–vector product at each step. This is a useful result, but it leaves open whether the new observation designs give useful finite certificates.
-
The finite-transfer assumptions seem difficult to satisfy outside controlled examples. The bound requires a fixed loss and projector, exact retention of , and derivative bounds through order five. Its tightness also depends on the calibration matrix and Taylor remainder. The asymptotic results are clear, but it is hard to tell the range of budgets over which the resulting finite bound would be informative.
-
The benefit over running the full comparator is unclear. In the successful experiment, each candidate requires 64 additional Hessian–vector products, alongside the reference gradients and probes. Running the full comparator takes 64 gradient steps, and its result can be reused across candidate spaces. The paper gives operation counts, but does not establish when certification offers a useful advantage in time, memory, or access requirements.
-
The current presentation is hard to follow due to the following reasons:
a) The abstract and introduction rely on broad descriptions without clearly stating what is measured. It is very hard for the readers to know what is going on. The roles of the “analyst,” “executor,” and “auditor” are not clear from reading.
b) Many phrases such as “normal norm” and “probe losses” are unnecessarily opaque. Explicit mathematical quantities or plain descriptions will help.
c) Some notation is not defined or is used before its role is explained. For example, Section 3 introduces -dependent iterates before specifying how determines the step sizes. Theorem 2 should explicitly identify ((a,b,c,d)) as the four-step profile .
In summary, the theory in the paper may be interesting, but its practical implications are not clear and the presentation is hard to follow.
Questions for authors
-
Can authors give a finite example using the four-checkpoint transfer bound, including the actual remainder constants, calibration conditioning, and resulting target interval?
-
Can the transfer bound accommodate a small amount of leakage from the lower-order space ? It is okay to just provide some comments rather than proof.
-
When is the centered certificate preferable to running one full comparator shared across candidates? What about time and memory comparison on the existing setup?