- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how learning rates and weight decay jointly control directional steps in scale-invariant SGD. It characterizes feasible forcing paths, solves a noisy spherical alignment benchmark, and establishes stochastic step asymptotics and stationarity conditions. Synthetic experiments test the theory, while image-classification and language-modeling experiments evaluate forcing-based schedules and temporal-order effects.
Critical strengths
- Quantitative theoretical characterization. The reachability conditions and stochastic step classification clarify how learning rates and weight decay jointly govern directional motion. The explicit square-summability boundary and stationarity result provide insight beyond nominal learning-rate analysis.
- Complementary empirical evidence. Synthetic experiments directly test predicted exponents, constants, and heavy-tailed regimes. Paired neural-network experiments report improvements across 14 settings, while the controlled reversal study demonstrates order dependence within its stated scope.
Critical weaknesses
- Limited connection to practical schedule design. The optimal policy assumes a specialized spherical objective and noise geometry, whereas the main experiments use an empirical linear request with sequential clipping. They do not isolate the training benefit of the proposed global projection.
- Insufficient decay-schedule controls. The long-horizon ramp uses zero decay for 61.66% of updates, but comparisons lack tuned constant decay or simple late-stage decay removal. Consequently, the gains do not isolate forcing-path design from changes in decay strength and duration.
Questions for authors
Please refer to the Critical Weaknesses.
Response for Reviewer N25y
1. Weak practical connection and unclear benefit of global projection
The optimal reference problem motivates controlled deviations around B=1 through
Bt⋆=1−t−1+O(t−2).The fixed-amplitude narrow ramp tests a simple forcing family around this reference, with early positive and later negative log forcing. Reachability and approximation then determine how native decay can implement the request.
We conducted the requested matched comparison between sequential clipping and global projection on CIFAR-100/ResNet-50 at 58,650 updates. Both share the forcing request, cosine LR 0.5−0.05, initial WD 0.001, decay bounds [0,0.1], whole-network scope, initialization, sampling randomness, and validation split.
Their log-forcing RMSEs are approximately 7.831529617803×10−4 and 7.831529615095×10−4, respectively. Sequential clipping is already nearly optimal for this request, leaving negligible approximation error to remove. Validation accuracies are 77.40% and 77.14%. Its guarantee concerns optimal feasible approximation under the stated objective.
2. Missing tuned constant-decay and late-removal controls
We have newly added both the constant-decay tuning sweep and an independent decay-removal timing sweep in new section G.6 and G.7 and use the tuned cosine for this part of experiments. The timing experiment separately tests how early or late removal affects performance at a fixed LR/WD configuration (Section G.6, Table 10, and Figure 10).
The late removal affects experiments are done with
λt=0.0011{t<⌊fT⌋},f∈{0,0.1,…,1},T=58,650.The cosine learning rate remains positive and unchanged. The focused early/midpoint/late comparison is:
Removing decay very early or retaining it until late training is less competitive: the 10% and 90% cutoffs trail the midpoint by 1.72 and 1.70 percentage points. The 50% candidate follows directly from the prescribed request's midpoint crossing of B=1, without a separate cutoff search to define that candidate. The results support the forcing construction as a source of competitive candidate timings. However, one can see that without prior knowledge on the Bt, it is hard to directly pinpoint the optimum decay removal timing.