- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Event-Driven Parameter Dynamics (EPD), a framework that places an accumulation, selection and reset process between an optimizer's candidate increments and their application to parameters. The evaluated instance maintains a decayed residual state, releases the largest-magnitude coordinates under a fixed per-tensor budget, and clears released coordinates. It gives elementary properties of these operations and evaluates ResNet-18 and DenseNet-121 on CIFAR-100 with AdamW and momentum SGD. At the default 1% release density, accuracy is within approximately 0.34–1.55 percentage points of dense training. Component ablations and release-density/decay sweeps examine the behavior of this implementation.
Critical strengths
- The paper cleanly separates the base optimizer from the subsequent realization rule and specifies the coordinate-wise instance in sufficient mathematical detail to understand its operation. The dense boundary case is explicit. The discussion also appropriately limits the claims: convergence and system-level benefits are left for future work rather than claimed from the present experiments.
- The empirical reporting is stronger than a single-run demonstration. The main comparisons and component ablations use ten seeds, with paired differences and per-seed results in Appendix D. Testing two architectures and two base optimizers supports the stated CIFAR-100 proof of concept. The decay sweep in Appendix E is also helpful for understanding sensitivity beyond the beta=0 ablation.
Critical weaknesses
-
The distinction from established error-feedback sparsification is not sufficiently demonstrated. Let C denote per-tensor Top-q truncation. Equations (9), (14) and (15) give a=beta*s+u, r=C(a), and s_new=a-C(a). At beta=1, this is the usual residual/error-feedback recursion applied to the candidate increment; with plain SGD it recovers sparsified SGD with memory, up to sign convention. These connections are already acknowledged through citations in the introduction, but the paper does not spell out the equivalence or identify what additional result the new formulation establishes. The evaluated beta=0.95 version introduces residual decay, yet Appendix E stops at beta=0.99 and does not compare against the directly corresponding beta=1 method. The broader framework may be useful as an organizing view, but all experiments concern this closely related canonical instance, and the propositions largely formalize its defining operations. A precise account of the new insight, together with a matched beta=1 error-feedback comparison, is important for assessing the contribution. Relevant primary sources are Stich et al., Algorithm 1 and Karimireddy et al., Algorithm 2, both already cited by the submission.
-
The component ablations establish dependence on the specified components at fixed hyperparameters, but provide limited isolation of the proposed temporal/spatial mechanism. Removing accumulation at p=1% discards unreleased increments, while removing reset repeatedly reapplies past increments. These changes also substantially alter update scale and effective memory. For example, uniform random release without accumulation has expected update approximately p*u, and a persistent coordinate in the no-reset control can accumulate a factor approaching 1/(1-beta). Thus the large gaps in Table 2 do not by themselves distinguish the proposed interpretation from changes in effective step size. The random-release-with-accumulation control is useful, but release policy also changes waiting times and the amount of decay before application. A modest control with separately tuned step scale, accompanied by realized-update norms and inter-release-time statistics, would make the mechanism claim substantially more informative. The authors acknowledge that event-pattern characterization remains open (Section 4); that missing characterization matters especially because it is the central perspective distinguishing this work from the prior residual-feedback formulation.
Questions for authors
- Can you explicitly relate the canonical recursion to Top-q error feedback and report its beta=1 counterpart under the same base optimizer, release budget and training protocol? What specific benefit or behavior is attributable to residual decay or to the broader EPD view?
- Do the ablation gaps persist after a small, comparable learning-rate search or update-scale control? For a representative setting, please report realized-update norms and release-interval distributions, and clarify whether the complete AdamW increment, including decoupled weight decay, is accumulated and sparsified.
- Corollary 1 and Appendix B derive the usual variance expression for a normalized weighted average. Is the result intended for a fixed interval, or for the state-dependent inter-release intervals produced by EPD? The latter introduces data-dependent stopping/selection, so unconditional zero-mean, uncorrelated candidate noise does not automatically remain so within selected intervals. Please make the required conditioning explicit. Also distinguish this normalized statistic from the unnormalized increment actually applied by the algorithm.
- Please reconcile the experiment records: Table D.2 contains five seeds, whereas Section 3 and the Figure 2 caption describe ten unless specified otherwise; the default beta=0.95 column in Table E.1 differs from the canonical EPD column in Table D.3; and p=100% in Table D.2 differs from dense training for the same seed labels in Table D.1. Were these independent reruns or different deterministic settings? A small p=100% versus dense equivalence check and clarification of the runs used in each figure would resolve these questions.