acceptodds
Under review as a conference paper at ICLR 2027

Event-Driven Parameter Dynamics for Spatiotemporal Parameter-Update Application in Deep Learning

Abstract

Standard neural network training typically applies optimizer-generated update signals to model parameters at every iteration. This work introduces a spatiotemporal view of parameter updating and proposes Event-Driven Parameter Dynamics (EPD), which separates candidate-update generation from their application through an intermediate state with accumulation, event-triggered release, and reset. In a canonical coordinate-wise instantiation, accumulation produces temporal integration, release imposes a spatial budget, and reset separates successive inter-release intervals, while conventional dense updating is recovered as a boundary case. Across four model-optimizer configurations on CIFAR-100, EPD maintains final performance close to conventional training even when only a small fraction of coordinates are released at each step, including the 1% regime. Matched-control experiments further show that temporal accumulation, state-dependent selection, and post-release reset each contribute to the observed behavior. Together, these results identify parameter-update realization as a controllable dimension of optimization and provide a concrete basis for studying its temporal and spatial organization separately from candidate-update generation.

Primary area:
optimization
Keywords:
Event-Driven Parameter Dynamics, Neural Network Optimization, Optimization Dynamics, Event-Triggered Updates, Parameter-update applicationalization.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AuyJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Event-Driven Parameter Dynamics (EPD), a framework that places an accumulation, selection and reset process between an optimizer's candidate increments and their application to parameters. The evaluated instance maintains a decayed residual state, releases the largest-magnitude coordinates under a fixed per-tensor budget, and clears released coordinates. It gives elementary properties of these operations and evaluates ResNet-18 and DenseNet-121 on CIFAR-100 with AdamW and momentum SGD. At the default 1% release density, accuracy is within approximately 0.34–1.55 percentage points of dense training. Component ablations and release-density/decay sweeps examine the behavior of this implementation.

Critical strengths
  1. The paper cleanly separates the base optimizer from the subsequent realization rule and specifies the coordinate-wise instance in sufficient mathematical detail to understand its operation. The dense boundary case is explicit. The discussion also appropriately limits the claims: convergence and system-level benefits are left for future work rather than claimed from the present experiments.
  2. The empirical reporting is stronger than a single-run demonstration. The main comparisons and component ablations use ten seeds, with paired differences and per-seed results in Appendix D. Testing two architectures and two base optimizers supports the stated CIFAR-100 proof of concept. The decay sweep in Appendix E is also helpful for understanding sensitivity beyond the beta=0 ablation.
Critical weaknesses
  1. The distinction from established error-feedback sparsification is not sufficiently demonstrated. Let C denote per-tensor Top-q truncation. Equations (9), (14) and (15) give a=beta*s+u, r=C(a), and s_new=a-C(a). At beta=1, this is the usual residual/error-feedback recursion applied to the candidate increment; with plain SGD it recovers sparsified SGD with memory, up to sign convention. These connections are already acknowledged through citations in the introduction, but the paper does not spell out the equivalence or identify what additional result the new formulation establishes. The evaluated beta=0.95 version introduces residual decay, yet Appendix E stops at beta=0.99 and does not compare against the directly corresponding beta=1 method. The broader framework may be useful as an organizing view, but all experiments concern this closely related canonical instance, and the propositions largely formalize its defining operations. A precise account of the new insight, together with a matched beta=1 error-feedback comparison, is important for assessing the contribution. Relevant primary sources are Stich et al., Algorithm 1 and Karimireddy et al., Algorithm 2, both already cited by the submission.

  2. The component ablations establish dependence on the specified components at fixed hyperparameters, but provide limited isolation of the proposed temporal/spatial mechanism. Removing accumulation at p=1% discards unreleased increments, while removing reset repeatedly reapplies past increments. These changes also substantially alter update scale and effective memory. For example, uniform random release without accumulation has expected update approximately p*u, and a persistent coordinate in the no-reset control can accumulate a factor approaching 1/(1-beta). Thus the large gaps in Table 2 do not by themselves distinguish the proposed interpretation from changes in effective step size. The random-release-with-accumulation control is useful, but release policy also changes waiting times and the amount of decay before application. A modest control with separately tuned step scale, accompanied by realized-update norms and inter-release-time statistics, would make the mechanism claim substantially more informative. The authors acknowledge that event-pattern characterization remains open (Section 4); that missing characterization matters especially because it is the central perspective distinguishing this work from the prior residual-feedback formulation.

Questions for authors
  1. Can you explicitly relate the canonical recursion to Top-q error feedback and report its beta=1 counterpart under the same base optimizer, release budget and training protocol? What specific benefit or behavior is attributable to residual decay or to the broader EPD view?
  2. Do the ablation gaps persist after a small, comparable learning-rate search or update-scale control? For a representative setting, please report realized-update norms and release-interval distributions, and clarify whether the complete AdamW increment, including decoupled weight decay, is accumulated and sparsified.
  3. Corollary 1 and Appendix B derive the usual variance expression for a normalized weighted average. Is the result intended for a fixed interval, or for the state-dependent inter-release intervals produced by EPD? The latter introduces data-dependent stopping/selection, so unconditional zero-mean, uncorrelated candidate noise does not automatically remain so within selected intervals. Please make the required conditioning explicit. Also distinguish this normalized statistic from the unnormalized increment actually applied by the algorithm.
  4. Please reconcile the experiment records: Table D.2 contains five seeds, whereas Section 3 and the Figure 2 caption describe ten unless specified otherwise; the default beta=0.95 column in Table E.1 differs from the canonical EPD column in Table D.3; and p=100% in Table D.2 differs from dense training for the same seed labels in Table D.1. Were these independent reruns or different deterministic settings? A small p=100% versus dense equivalence check and clarification of the runs used in each figure would resolve these questions.
Reviewer_e7rPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Event-Driven Parameter Dynamics (EPD), a framework that separates the generation of a candidate update from its application to the parameters. Candidate increments from a base optimizer are accumulated into a per-unit internal state, selectively released through an event rule, and cleared by a reset rule. The authors study a canonical coordinate-wise instance: decayed additive accumulation, per-tensor Top-p magnitude release and hard reset. Dense training is recovered at p = 100%. On CIFAR-100 (ResNet-18 and DenseNet-121, with AdamW and SGD-M), the canonical instance stays within 0.3–1.6 points of dense training, down to a 1% release rate. Matched-control ablations on one configuration indicate that all three mechanisms matter.

Critical strengths

S1 - The general formalism of the paper is clean and clear. Eqs. 1–8 give a precise reminder for update decomposition and the canonical instance (§2.2, Algorithm 1) is easy to reproduce. The paper is well written and the abstraction is a useful way to think about "when and where" updates are applied.

S2 - The ablation is well-designed. The matched controls (Table 2, D.3) and the β sweep (Table E.1) use 10 paired seeds and remove one mechanism at a time. They show clearly that, in the tested setting, accumulation, informed selection and reset each matter. The reserve is that this is done on a single configuration (ResNet-18 + AdamW) and takes up much of an otherwise thin experimental section.

Critical weaknesses

W1 - §2.3 oversells its "properties". Props. 1 and 3 and Cor. 2 follow directly from the definitions, and Prop. 2 is the standard best-q-sparse property of Top-k, stated without citation. Cor. 1, the only non-standard statement, concerns a normalized state that the algorithm never forms. It also assumes uncorrelated increments, which the momentum and Adam updates used in the experiments do not satisfy. No convergence or stability result is given, yet the Conclusion states that “the theoretical and empirical results together show that parameter realization can be treated as an explicit and controllable component of optimization dynamics”.

W2 - The experimental setting is particularly narrow with no trade-off metrics reported. The study covers only CIFAR-100 with two small CNNs. EPD is below dense training in all four settings (−0.34 to −1.55 points, vs. seed std ≈ 0.2–0.3), and the flat accuracy between p = 1% and 100% (Fig. 2e) shows tolerance, not an advantage. No memory, compute, energy or communication measurement is reported, even though EPD adds a d-sized state and a per-step Top-k, and the released code applies the sparse update on dense tensors ("PyTorch may still execute this densely"). There are also no baselines (error-feedback Top-k, sparsified SGD with memory, DGC, block-coordinate or freezing schemes), although several are cited in the introduction. The ablations (β = 0, no reset) therefore mostly re-confirm known error-feedback intuitions rather than isolating something specific to EPD.

Questions for authors

Q1 - What are the memory overhead (one extra d-sized buffer on top of the AdamW/SGD-M states) and the wall-clock overhead of the per-tensor Top-k? In which regime (communication, on-device training, energy) is sparse realization expected to pay off?

Q2 - Why this accumulation, score, budget and reset? The released code contains a soft reset and threshold-based release (hard_threshold) that are not reported; how do they compare?

Q3 - The default β = 0.95 is also the best value in Table E.1, which was swept on test accuracy for ResNet-18 + AdamW, and no validation split is mentioned. Was β chosen on held-out data? Was the learning rate re-tuned for EPD? Is Table 1 last-epoch accuracy (the training script also tracks best test accuracy)?

Q4 - Table D.2 reports 5 seeds, not 10 as stated in §3. At p = 100%, EPD should coincide with dense training (Cor. 2), yet per-seed values differ from the dense baseline by up to 0.78 points (seed 3: 73.56 vs 74.34). What is the actual run-to-run noise of identical algorithms, and how should the paired Δ of Table 1 be read against it?

Reviewer_Y8DXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Event-Driven Parameter Dynamics (EPD), a framework that decouples the generation of optimizer updates from their actual application to model parameters. Instead of applying updates at every iteration, EPD maintains an intermediate state that accumulates candidate updates over time, selectively releases coordinates based on their accumulated magnitudes, and resets the released states.

The proposed framework consists of three mechanisms: temporal accumulation, state-dependent sparse release, and post-release reset. The authors evaluate EPD on CIFAR-100 using ResNet-18 and DenseNet-121 with AdamW and momentum SGD. Results demonstrate that EPD can maintain accuracy relatively close to conventional dense training while updating approximately 1% of parameter coordinates per iteration. Ablation studies further suggest that all three mechanisms contribute to the effectiveness of the proposed method.

Critical strengths
  1. Interesting perspective on parameter updating. The paper introduces a spatiotemporal perspective that separates update generation from update application. This formulation provides a potentially useful framework for studying how the timing and coordinate selection of parameter updates influence optimization dynamics.

  2. Simple and optimizer-agnostic formulation. EPD can be incorporated into existing optimizers without fundamentally modifying their update-generation mechanisms. Its accumulation, release, and reset operations are straightforward and mathematically well-defined, making the method relatively easy to understand and reproduce.

  3. Systematic mechanism-level ablation studies. The experiments investigate the contributions of temporal accumulation, state-dependent release, and post-release reset individually. The results show substantial performance degradation when these mechanisms are removed, providing empirical evidence that their combination is important for maintaining training effectiveness under sparse parameter updates.

Critical weaknesses
  1. Limited novelty compared with existing sparsification and error-feedback methods. The proposed approach closely resembles established Top-k gradient sparsification with error feedback, where unselected updates are accumulated and subsequently applied. Although the authors emphasize the conceptual distinction between update generation and application, the canonical EPD implementation appears largely equivalent to existing residual-accumulation-based sparsification mechanisms. The paper lacks direct comparisons or a rigorous explanation of what fundamentally differentiates EPD from these methods.

  2. Insufficient experimental scale and practical efficiency evaluation. All experiments are conducted on CIFAR-100 with only two relatively small CNN architectures. More importantly, although EPD releases only 1% of parameter coordinates, it still requires dense gradient computation, dense optimizer updates, an additional full-sized accumulation state, and per-tensor Top-k selection. Therefore, sparse parameter realization does not necessarily translate into reduced computation, memory consumption, or training time. Without wall-clock time, GPU memory, and computational overhead measurements, the practical benefits remain unclear.

  3. Limited empirical advantages and theoretical insights. EPD consistently underperforms conventional dense training across the four evaluated configurations, with accuracy reductions of approximately 0.34–1.55 percentage points. Consequently, the experiments establish that sparse update application is feasible rather than demonstrating a clear optimization advantage. Furthermore, the theoretical results mainly characterize the accumulation and selection operations, without providing convergence guarantees, optimization error bounds, or explanations for why high accuracy can be maintained under a 1% release budget.

Questions for authors
  1. What is the fundamental distinction between EPD and existing Top-k sparsification methods with error feedback? In particular, can the authors establish whether the canonical EPD update rule is mathematically equivalent to residual-based Top-k sparsification applied to optimizer-generated increments? Direct experimental comparisons under identical conditions would help clarify the methodological novelty.

  2. Does EPD provide measurable computational or memory-efficiency benefits? Since dense gradients and optimizer updates are still computed at every iteration, and additional accumulation buffers and Top-k selection are required, what are the actual wall-clock training time, peak GPU memory usage, and per-iteration overhead compared with conventional training? Are there realistic deployment scenarios where sparse parameter application provides practical savings?

  3. How well does EPD generalize to larger architectures and more challenging optimization settings? Can the authors provide results on ImageNet-scale training or Transformer-based architectures, particularly with AdamW? Additionally, how does the method behave under different learning-rate schedules, weight decay settings, and release rates, and can the observed performance preservation be explained theoretically?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.