acceptodds
Under review as a conference paper at ICLR 2027

On Gating Dynamics: Spectral Condensation and Distinct Paths of Return

Abstract

Motivated by the possibility that the Edge of Stability (EoS) accompanies a reorganization of neuron participation, we examine how ReLU gating evolves across training regimes. Binary neuron–sample supports let us distinguish how much activity is present, how participation is distributed, and how gates change over time. Across ReLU MLPs on FashionMNIST and CIFAR-10, support density initially rises while raw-support effective rank falls—a spectral condensation. Subsequent rank recovery and density decrease follow distinct paths of return. In shallow FashionMNIST, the higher-rate run recovers rank and renews gate switching near EoS, whereas the lower-rate run remains nearly frozen with little rank recovery. Renewed switching becomes largely balanced exchange, with the net on–off current exhibiting approximate period-two recurrence. Yet deep-layer density–rank return precedes the late increase in gate switching, and returns occur both with a larger low-participation population and, in CIFAR-10, with little such population and weaker class-associated sharing. The SGD examples also exhibit condensation and return, but with gate switching present from the outset. Together, these contrasts motivate a hypothesis that participation structure shapes the available paths of return from spectral condensation, while gate switching influences how those paths are traversed.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
ReLU networks, gating dynamics, activation support, spectral condensation, Edge of Stability, participation redistribution, period-two current, activation sparsity

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_mBwcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tracks binary neuron-sample activation patterns in ReLU MLPs using support density, effective rank, gate turnover, and net activation current. FashionMNIST and CIFAR-10 experiments show initial spectral condensation followed by different return trajectories across depths, learning rates, and optimizers. In the deep example, density-rank return precedes a late switching surge; selected GD runs show approximately period-two current. Participation analyses track previously isolated neurons, while centered and permutation controls show that raw-rank recovery need not imply recovery of more specific support organization. The contribution is descriptive, motivating an open hypothesis about how participation structure shapes reorganization.

Critical strengths

The paper offers an accessible set of tools for describing neuron participation, with explicit definitions of density, effective rank, turnover, and current. Distinguishing total switching from net activation change makes clear why nearly constant density can conceal substantial exchange. The organization into state, flow, and structure helps readers follow the measurements and provides a useful basis for further investigation.

The study combines spectral trajectories, recurrence tests, and individual neuron histories across depth, learning rate, and dataset settings. In the deep example, rank return near epoch 700 precedes the switching surge near 3138, constraining explanations based solely on that later event. Distinguishing periodic current from recurring support also adds descriptive value, although the broader structural interpretation remains unresolved.

Critical weaknesses
  1. The added value of the spectral description remains unclear. The paper acknowledges that raw-rank recovery can reflect changing participation rates. However, Appendix E’s null preserves neuron degrees while changing sample degrees. It therefore does not establish which differences between return paths require support arrangement beyond these marginals. This limits the interpretation of recovery, although the raw-rank observations remain valid. The distinction between raw-rank recovery and centered, null-normalized behavior requires cross-referencing Figure 2, Section 4.1, and Appendix E. Although the caveat is stated, presenting these results together would make the meaning and limits of “return” easier to follow.

  2. The temporal comparison remains limited by observation resolution. Epoch endpoints span one GD update versus approximately 40 SGD updates, roughly matching dataset passes. As the paper acknowledges, switches can reverse between observations. The comparison therefore leaves unresolved how much of the contrast reflects persistent gate changes versus hidden switching.

Questions for authors
  1. For representative Figure 2 trajectories, compare raw and centered ranks against randomized supports preserving both neuron and sample degrees. Use initialization, the raw-rank minimum, and the endpoint, and report variability across randomizations.

  2. Which differences between return paths remain beyond this null? If participation marginals explain the recovery, what additional insight does the spectral description provide?

  3. Could you examine short early and late GD/SGD windows on the same fixed probe set, recording supports at every optimizer update? Indicate training location in both updates and examples processed.

  4. In those windows, compare turnover and current at one-update and common 40-update intervals. Also report accumulated switches per gate to quantify switching missed by endpoint measurements. Which temporal contrasts persist across observation scales?

1 comment
AuthorsOpenReview ↗

Response to mBwc: Spectral Paths and Switching Resolution

Thank you for the careful review and follow-up.

Rebuttal I introduced time-resolved row-permutation controls and addressed your request for a both-degree-preserving null at three checkpoints; Rebuttal II established the five-seed and matched-loss results you acknowledged; and Rebuttal III extended the findings across training subsets, objectives, and architectures.

We have since examined your two remaining concerns.

What remains beyond participation marginals? We extended your requested both-degree control from three checkpoints to the full trajectories of the four representative GD runs (Appendix E.1, Fig. 13). The two nulls closely agree in raw and centered rank, but neither reproduces the observed spectral paths. In shallow FashionMNIST, observed rank reaches its minimum at epoch 400 versus 200 under both nulls; in deep FashionMNIST, the observed minimum is interior while the null minima occur at initialization. The nulls recover more rank, and centered rank relative to its null declines, as you noted. Thus changing participation marginals can reproduce a substantial rebound. What remains distinctive is the observed spectral deficit, the timing of condensation and return, and their rank–density geometry, even when both degree sequences are preserved.

What survives matched observation scales? We included SGD to examine spectral return under continuing gate activity, rather than GD's pronounced freezing and subsequent reactivation. Your sampling concern was justified: epoch endpoints conceal 96.9% and 97.7% of late SGD flip events in FashionMNIST and CIFAR-10. On the same fixed probe, we now compare per-update turnover, accumulated switching, exact 40-update signed current, and turnover at one-data-pass and common 100-update intervals; the distinct switching histories persist (Appendix E.4). SGD reaches its saved rank minimum and partial recovery earlier per data pass, suggesting that training regimes may differ in how readily they access paths of participation reorganization, although the extent of return varies and there is no uniform matched-loss rank advantage.

Together with the reproducibility and generality results, these controls strengthen the evidence for spectral condensation and distinct paths of return, while clarifying their interpretation beyond participation marginals and observation-scale effects.

Reviewer_5T5WOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper tracks the binary neuron × sample ReLU support matrix during training of small MLPs (2×512, 4×512, 2×1024) with full-batch GD (plus a few SGD runs) and MSE loss, on 10k-example subsets of FashionMNIST and CIFAR-10. It reports support density, the effective rank of the raw binary support, gate turnover/current, participation distributions and active-core connectivity. Main observations:

  • Density first rises while effective rank falls ("spectral condensation"), followed by different "returns".
  • In shallow FashionMNIST, the higher learning rate shows renewed gate switching near edge-of-stability (EoS) contact, and the net on–off current oscillates with period two.
  • In the deep network, the return begins before switching increases.
  • SGD shows condensation and return with switching present from the start.

The authors propose, but do not test, a hypothesis that participation structure shapes the paths of return.

Critical strengths
  1. Clear, well-defined observables. The density/turnover/current decomposition (ρt+1−ρt=−Jt\rho_{t+1}-\rho_t=-J_t) cleanly separates "frozen" from "balanced exchange" support dynamics, and it makes a useful point: average sparsity hides substantial gate switching. These measures could be reused by others studying activation-pattern dynamics.
Critical weaknesses
  1. The evidence comes from one or two runs per condition, and the hypothesis is never tested. Each main condition uses initialization seed 42; seed 43 is added only for the shallow and CIFAR runs (Table 2). All runs use one fixed 10k data subset, MSE loss and a constant learning rate, and no uncertainty is reported. The central claim, that participation structure shapes the paths of return, is called "abductive". The intervention that would test it (restricting recruitment of rarely active neurons) is described in Sec. 5 but not run. No observable is linked to loss, accuracy or generalization. Statements such as "the higher-rate run recovers rank, the lower-rate run stalls" therefore rest on two runs and one data draw.
  2. The headline phenomena are largely explained by known or mechanical effects.
    • (a) The raw effective rank is confounded by per-neuron participation rates. The rate (mean) component of the uncentered support has singular value ∥q∥N\|q\|\sqrt{N}, which grows with density and, more strongly, with the spread of per-neuron rates qiq_i. In a quick null simulation (independent Bernoulli supports, 512×10,000, no structure), raising the per-neuron rate s.d. from 0.15 to 0.25 at fixed density lowers the raw effective rank by ≈26 (454 → 428). That is the size of the ≈22-unit "return" the paper reports, whereas density 0.50 → 0.54 alone changes it by only ≈5. The paper's own rate-preserving control (centered rank over a within-row-permutation null, App. E) shows no return: it keeps falling, from .857 to .823, after the raw minimum. "Condensation and return" is therefore not shown to be a change in how the support is arranged, as opposed to a change in participation rates.
    • (b) The period-two current is what GD stability predicts. Along the top Hessian direction, each GD step multiplies the iterate by 1−ηλ=1−2χ1-\eta\lambda = 1-2\chi, which is negative whenever χ>1/2\chi>1/2. In the late windows, χ\chi = .91–.99, so this is a slowly damped sign alternation, and gates near threshold must flip in alternate steps. Jt+1≈−JtJ_{t+1}\approx-J_t is the gate-level image of the known EoS oscillation (Cohen et al., 2021; Damian et al., 2022), not a new property of gating. It also requires no EoS crossing, consistent with the deep case.
Questions for authors
  1. (Could change my assessment.) Please report the rank trajectory against a rate-preserving null (e.g., within-row permutation) at every observation, not three snapshots. Does any "return" remain once the rates are controlled?
  2. (Could change my assessment.) Do condensation, the learning-rate–dependent return and the period-two current hold across ≥5 seeds, several data subsets and cross-entropy loss? Please report them with intervals.
  3. (Clarification.) Can you show that the period-two current is more than the projection of the top-eigenvector oscillation? For example, regress JtJ_t on the sign of ⟨θt−θˉ,v1⟩\langle\theta_t-\bar\theta, v_1\rangle, or test runs with 1/2<χ<11/2<\chi<1.
  4. (Suggestion.) Running the recruitment-restriction intervention proposed in Sec. 5 would turn the hypothesis into a testable claim.
1 comment
AuthorsOpenReview ↗

Response to 5T5W: Spectral Return and Hessian-Driven Gate Motion

Thank you for the careful review. We particularly appreciated your two alternative explanations: that changing participation rates may account for much of raw spectral recovery, and that period-two current may reflect familiar GD oscillations. Both helped us revisit what our observations establish and where we should narrow their interpretation.

What remains beyond participation marginals? Your Bernoulli example and the declining centered/null-normalized rank highlight an important limitation: raw-rank recovery does not by itself establish recovery of support organization beyond changing participation rates. We agree.

Building on Rebuttal I, we extended the null preserving both neuron and sample degrees to the full trajectories of four representative GD runs (Appendix E.1, Fig. 13). The nulls recover more rank than the observed supports, as you anticipated, but do not reproduce their spectral concentration or turning times. In deep FashionMNIST, for example, observed rank minimum occurs during training, whereas both null minima occur at initialization. Rebuttal III reveals a complementary behavior: under higher-rate cross-entropy and in the tested ReLU-ViT, the spectral deficit relative to neuron-degree nulls instead narrows during return. These distinctions characterize the learned paths without establishing a causal mechanism.

Are these findings reproducible? Your concerns about initialization variability and the limited training settings were important. Rebuttal II established the shallow learning-rate contrast across five GD seed pairs, including at matched training loss. Deep return precedes renewed switching by more than 2,000 epochs in every high-rate seed, and alternating-current episodes recur with varying persistence. Appendix C reports the individual seed trajectories and across-seed variability. Rebuttal III extends the findings across training subsets, cross-entropy, and initial CNN/ReLU-ViT experiments, while noting the more limited replication of these extensions in Appendix G.

Is period-two current simply a Hessian-mode oscillation? Your observation that alternation need not require an EoS crossing was especially useful. We tested the connection directly at the level of individual gates on the validated original deep high-rate trajectory (Appendix F.1). The leading Hessian mode reproduces only 17.36% of complete same-gate reversals, whereas the stable leading two-dimensional subspace reproduces 96.61%; removing it suppresses 96.85%. This strongly supports your explanation while refining the relevant geometry from one mode to two.

We have accordingly de-emphasized period-two current as a standalone contribution and connected it to known GD dynamics. Its role is to characterize late gate motion, not to explain the much earlier onset of spectral return.

Your suggested recruitment-restriction intervention is also well taken. Neither the spectral controls nor the Hessian analysis establishes structural slack as a causal mechanism; we retain it as a hypothesis about which participation reorganizations are accessible.

We hope these additions clarify both what established optimization effects explain and what remains distinctive about the spectral return paths we seek to understand. We would particularly welcome your perspective on whether the gate-level analysis clarifies the distinction you raised between Hessian-mode oscillations and gate dynamics.

Reviewer_nfjiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the evolution of binary neuron-sample activation supports in ReLU MLPs trained with MSE on FashionMNIST and CIFAR-10. It combines support density and entropy effective rank with gate turnover, signed activation-deactivation current, participation histories, and active-core connectivity. The main observation is an initial decrease in raw-support effective rank, often alongside increasing density, followed by returns whose extent and timing differ across learning rates, depths, datasets, and GD/SGD settings. High-rate GD examples exhibit approximately period-two signed current. A deep-network example shows density-rank return substantially before the later increase in gate turnover. These observations motivate a structural-slack hypothesis, which the authors explicitly distinguish from an established causal explanation or a claim about useful predictive capacity.

Critical strengths
  1. A useful and carefully measured distinction between support amount and dynamics. The paper shows that nearly stationary density can accompany either very few transitions or substantial balanced exchange. The period-two analysis uses unsmoothed traces, one-step reversal and two-step recurrence errors, and current amplitudes, rather than relying only on negative lag-one correlation. Table 4 includes two shallow initializations and informative early/late and learning rate contrasts.
  2. Careful treatment of counterexamples and interpretive limits. The deep example separates density-rank return from the later switching increase. The authors report centered/permutation controls that do not reproduce the raw-rank rebound, distinguish current recurrence from support recurrence, and avoid equating participation or connectivity with predictive contribution. Appendix F explicitly checks sharpness estimates numerically while retaining the possibility of crossings between observations. These qualifications make the descriptive evidence more credible.
Critical weaknesses
  1. The structural information in raw spectral return remains insufficiently quantified. The spectrum of an uncentered support matrix contains both a component determined by neuron participation rates and centered support structure. Appendix E appropriately investigates this issue, but the reported centered rank divided by its row-permutation baseline decreases from approximately 0.857 to 0.823 during shallow high-rate raw rank recovery. The control is evaluated at three snapshots and preserves neuron degrees, but not sample degrees. Thus, it remains unclear how much of the headline condensation or return is explained by changing participation marginals and what additional organization is revealed. A time resolved comparison of the raw spectrum against suitable degree preserving nulls would clarify this. I am not requiring the centered statistic to rebound; the concern is identifying what the raw trajectory adds. The broad structural slack hypothesis is appropriately presented as conjectural, but currently provides limited additional discrimination between these explanations.
  2. The principal regime contrasts need stronger replication and comparable observation scales. Table 2 uses one initialization for the deep timing result and each main SGD example; the shallow GD comparisons use two initializations on one fixed data subset. The deep return before switching ordering is interesting, but its consistency is not established. Likewise, the lower-rate shallow trajectory is nearly stalled within the chosen horizon; extending it to comparable training loss or optimization progress would help distinguish a different path from delayed traversal. For GD versus SGD, turnover spans one update in GD but 40 updates per epoch in the main SGD setting. Although the manuscript explicitly acknowledges this endpoint limitation, temporal aggregation and different optimization progress remain intertwined with the optimizer comparison. Targeted replication and matched interval measurements would make the characterization more compelling.
Questions for authors
  1. How does the raw-rank trajectory compare with a null preserving neuron participation rates across training? A control retaining both neuron and sample degrees would further separate marginal effects from support arrangement. Can these analyses quantify which part of condensation and return is not explained by those marginals?
  2. Does the deep return before switching ordering persist across additional initializations? Does the lower-rate shallow trajectory remain qualitatively different when extended to comparable training loss or optimization progress, rather than compared only at similar epoch endpoints?
  3. Do the main GD/SGD switching contrasts persist when flows are measured over the same number of parameter updates? Per-update SGD measurements in representative windows, or a matched-update-interval GD control, would help determine how much of the difference is due to temporal aggregation.
1 comment
AuthorsOpenReview ↗

Response to nfji: Interpreting Spectral Return and Its Distinct Paths

Thank you for the thoughtful review. We appreciate your focus on what learned spectral trajectories reveal beyond participation marginals when compared with degree-preserving nulls. Our expanded controls allow us to examine this distinction more directly.

What remains beyond participation marginals? Building on Rebuttal I and a related question from mBwc, we extended the null preserving both neuron and sample degrees from three checkpoints to the full trajectories of four representative GD runs (Appendix E.1, Fig. 13).

The results reveal an important distinction. In high-rate FashionMNIST, raw rank recovers by about 22 units while the degree-preserving null recovers by about 40. Thus raw spectral return can coexist with a widening spectral deficit relative to randomized supports with identical marginals. In shallow FashionMNIST, the observed rank minimum also occurs later than the null's; In deep FashionMNIST, the learned support's minimum occurs during training, while both nulls attain their minima at initialization. What the raw trajectory adds, then, is a view of how learned participation evolves spectrally alongside its changing marginals, including differences in concentration and timing that the marginals alone do not determine.

Interestingly, Rebuttal III reveals the complementary possibility: under higher-rate cross-entropy and in the tested ReLU-ViT, the deficit instead narrows during return relative to neuron-degree nulls (Appendix G). Across these settings, rank recovery can accompany either increasing or decreasing density, while the spectral deficit relative to participation-preserving nulls can widen or narrow. These contrasting behaviors help sharpen the structural-slack question: does the participation geometry present before return constrain the direction and persistence of subsequent spectral reorganization?

Are these genuinely distinct paths? Your concerns about initialization variability and delayed optimization were important here. Rebuttal II established the shallow learning-rate contrast across all five GD seed pairs, including a persistent higher-rate rank advantage at matched training loss. Deep return also precedes renewed switching by more than 2,000 epochs in every high-rate seed (Appendix C). Rebuttal III further extends the findings across training subsets, objectives, and architectures, with the replication scope of those extensions stated in Appendix G. Together, these results make the distinctions harder to attribute to initialization or slower optimization alone.

What survives observation-scale differences? Your GD–SGD concern was well founded: epoch endpoints conceal approximately 97% of late SGD flip events. We therefore examined switching on a common fixed probe, comparing per-update turnover, accumulated flips, 40-update signed current, and turnover at one-data-pass and common 100-update intervals (Appendix E.4).

The finer measurements retain different temporal histories. SGD maintains gate activity from early training, whereas low-rate GD approaches near-freezing and high-rate GD can exhibit a pronounced later switching renewal. SGD also reaches its saved rank minimum and partial recovery earlier per data pass, though not per optimizer update, and without a uniform rank advantage at matched training loss. This suggests different routes through participation reorganization, without isolating their underlying mechanism.

We hope these additions make clearer both what the spectral description contributes and which alternative interpretations the evidence can now distinguish. We would particularly welcome your perspective on whether the degree-conditioned trajectories address the structural distinction you had in mind, and what further comparisons might sharpen it.

Reviewer_joRGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Summary. The paper studies how ReLU gating, the binary neuron–sample support Ais(t)=1{zi(xs;t)>0}A_{is}(t)=\mathbb{1}\{z_i(x_s;t)>0\}, evolves during training and relates it to the Edge of Stability. In small MSE-trained MLPs on FashionMNIST and CIFAR-10, support density first rises while effective rank falls (spectral condensation), after which rank recovers and density decreases (return), along paths that depend on learning rate, depth, and dataset. The authors hypothesize that participation structure shapes the available paths of return, while gate switching influences how they are traversed.

Contributions.

  1. Measurement framework. Density ρt\rho_t, effective rank reffr_{\mathrm{eff}}, and gate flow split into turnover TtT_t and net current JtJ_t (with ρt+1−ρt=−Jt\rho_{t+1}-\rho_t=-J_t), plus neuron-level participation and connectivity measures.
  2. Condensation and regime-dependent return. Shallow FashionMNIST recovers rank substantially at η=.04\eta=.04 but stalls at .02.02, and CIFAR-10 overshoots its initial rank. In the deep network, return begins about 2000 epochs before gate switching rises, and the routes differ in participation (a pool of rarely active neurons in deep FashionMNIST, almost none in CIFAR-10). Condensation and return also appear under SGD.
  3. Balanced exchange near EoS. In high-rate GD, switching resumes near EoS with little density drift, and the net current shows approximate period-two recurrence (Jt+1≈−JtJ_{t+1}\approx -J_t, Jt+2≈JtJ_{t+2}\approx J_t).
Critical strengths

1. A measurement framework that exposes gating dynamics invisible to sparsity. Treating ReLU gating as a binary neuron–sample matrix, the paper separates how much is active (density ρ\rho), how it is arranged (effective rank), and how much it changes (turnover TT vs. current JJ, with ρt+1−ρt=−Jt\rho_{t+1}-\rho_t=-J_t). These measures really do diverge in the experiments: density rises while rank falls, runs with similar density differ sharply in rank recovery (η=.04\eta=.04 vs. .02.02), and large turnover coexists with near-zero density drift. This directly bears on work linking large learning rates to sparsity, where density is the usual summary.

This is the paper's main lasting contribution. The measures are simple, cheap, and architecture-agnostic, and they stay useful even if the paper's hypothesis does not hold up. Its weight is somewhat limited by how close it is to existing activation-pattern statistics (e.g., Hartmann et al., 2021). The distinction the paper draws between these is valid, but it could be shown more directly.

2. A concrete, quantified gating signature at the Edge of Stability. In high-rate GD, gate switching resumes near EoS as balanced exchange, and the net current shows approximate period-two recurrence (Jt+1≈−JtJ_{t+1}\approx -J_t, Jt+2≈JtJ_{t+2}\approx J_t). This is quantified, not just shown visually. The normalized recurrence errors are small late in high-rate training (E1≈.10E_1\approx .10, E2≈.09E_2\approx .09 shallow; .08.08, .02.02 deep) and large in low-rate runs (E1≈.84E_1\approx .84–1.961.96) and early training (E1≳1.4E_1\gtrsim 1.4). The authors also correctly note that negative lag-one correlation alone could come from differenced noise.

EoS is usually described through the loss and parameters. This result shows a matching oscillation in which neurons are active, linking an optimization phenomenon to a representation-level quantity in a form other architectures can test. It is the finding most likely to prompt follow-up work, but small models, few seeds, and no mechanism make it a promising observation rather than an established result.

Critical weaknesses

1. The central hypothesis is untested, and the main observable is hard to interpret. The "structural slack" hypothesis rests only on correlations across a few runs. The authors describe the experiment that would test it (restricting recruitment of rarely active neurons) but do not run it. Raw effective rank also mixes changes in participation rates with changes in arrangement. In the paper's own permutation-normalized control (Appendix E), the high-rate shallow normalized ratio does not recover (the ratio falls from .857.857 to .823.823), so the "return" may largely reflect rate shifts rather than reorganization. None of the measures is connected to generalization or prediction.

This is the main reason I lean toward rejection. Without an intervention or a functional link, the paper documents patterns without showing what causes them or why they matter. The rank ambiguity also weakens the headline phenomenon itself. Running the proposed intervention and reporting a rate-controlled rank would address most of this concern.

2. The empirical scope and statistical support are too narrow for the claims about regimes. All experiments use small MSE-trained MLPs on 10k-example subsets. Each regime comparison rests on one or two runs: the deep FashionMNIST and SGD results use a single seed, and no seed variability is reported. Temporal resolution is coarse (supports every 100 epochs), and the deep network never shows a measured EoS crossing (max S≈.987S\approx .987).

The paper argues that regimes take "distinct paths of return." With so few runs and no variance, regime effects can't be told apart from run-to-run variation, especially for orderings resolved only to 100 epochs. Whether the phenomena hold for cross-entropy, CNNs or transformers, or full datasets is also unknown. More seeds and one more standard setting would substantially strengthen the claims.

Questions for authors
  1. The paper proposes restricting recruitment of rarely active neurons as a test of the structural-slack hypothesis. Can you run this, even at small scale? Does it selectively reduce rank recovery?

  2. In Appendix E, the permutation-normalized centered rank ratio falls (.857→.823.857 \to .823) while raw rank recovers. How much of the raw-rank return is explained by changing participation rates alone, e.g., compared with a row-degree-preserving randomized support tracked over training?

  3. Can you report seed variability for the deep FashionMNIST timing (return at epoch ≈700\approx 700 vs. switching onset at ≈3138\approx 3138) and for E1,E2E_1, E_2?

  4. Do any of the gating measures (turnover freezing, RR, marginal fraction) correlate with generalization across your runs?

2 comments
AuthorsOpenReview ↗

Response to joRG: Structural Slack, Spectral Return, and Empirical Significance

Thank you for the thoughtful review. We particularly appreciated your recognition of the measurement framework and the significance of the gate-current observations. Your questions about causal interpretation, participation marginals, reproducibility, and functional relevance helped us sharpen the empirical picture.

Does structural slack causally shape spectral return? We examined whether shallow low-rate rank stalling simply reflects frozen support or a lack of nearby higher-rank alternatives (Appendix E.5, Fig. 20).

Across five paired FashionMNIST seeds, roughly 5% of low-rate support entries change between epochs 1500 and 3300 despite almost no rank gain. At matched 4% displacement, higher-rate runs gain 5.28–6.29 rank units, versus −1.07 to +0.88 at the lower rate. Degree-preserving rewiring of just 1% of entries also produces higher-rank configurations near the stalled supports.

Thus spectral stalling is neither literal freezing nor the absence of combinatorially accessible alternatives. Whether these alternatives are accessible along GD trajectories remains open.

Do the gating observables relate to generalization? Our measurements characterize participation reorganization rather than directly predicting test performance. Replication and matched-training-loss comparisons establish distinct spectral trajectories at comparable optimization progress, but not a relationship with generalization. They provide a reproducible basis for investigating the functional consequences of these differences.

How much of spectral return is explained by participation frequencies? We extended neuron-degree- and both-degree-preserving nulls to every saved checkpoint of four representative GD runs (Appendix E.1, Figs. 12–13).

Changing marginals can generate substantial raw-rank recovery: in high-rate shallow FashionMNIST, observed rank recovers 21.96 units versus 40.16 under the both-degree null. Yet the nulls do not reproduce the observed spectral concentration or turning times. In deep FashionMNIST, the observed minimum occurs during training, while both null minima occur at initialization. Higher-rate cross-entropy and the tested ReLU-ViT also show that the spectral deficit can narrow during return (Appendix G). These comparisons distinguish raw recovery from its evolution relative to participation-preserving baselines.

Are the return paths and gate dynamics reproducible? We expanded five GD settings to five seeds each (Appendix C). All 25 runs condense. The shallow rate contrast persists across seeds, with recovery of 20.42 ± 1.23 versus 0.67 ± 0.51 rank units, including a persistent difference at matched training loss.

Deep return precedes renewed switching by 2,053–3,008 epochs across all five high-rate seeds. Alternating-current episodes recur, with their variability quantified by the recurrence errors in Appendix C. We also connected late deep gate reversals to a stable two-dimensional Hessian subspace in the original trajectory (Appendix F.1). Training-subset, cross-entropy, CNN, and controlled ReLU-ViT experiments further extend the findings (Appendix G), with architectural replication limits stated explicitly.

We hope these additions clarify the empirical contribution independently of the proposed structural-slack mechanism. We would welcome your perspective on whether the expanded controls and the distinction between support motion and spectral recovery address your central concerns.

Reviewer_joRGOpenReview ↗

Response to authors: addressed and remaining concerns

Thanks to the authors for the thorough responses and all the extra experiments in a short window.

The reproducibility concerns are resolved. With five seeds per setting, the shallow learning-rate contrast holds up, and deep return precedes renewed switching in every seed. The CE, CNN, and ViT runs are reasonable first steps for generality. The Hessian-subspace connection in F.1 also makes the period-two gating result near EoS more convincing.

My remaining concern is about what “return” means after the new controls. In the MSE MLPs, the participation-preserving nulls recover more rank than the actual supports, and centered rank relative to the null keeps falling during return. E.5 also attributes most of the learning-rate gap to the marginal component. So in the main setting, the raw-rank recovery is more than explained by participation rates, and the learned arrangement actually moves further from its baseline during return. The shape and timing differences relative to the nulls are real, but they don’t support the “distinct paths of return” story as it’s currently told. The structural-slack hypothesis is also still untested. The rewiring analysis shows nearby higher-rank supports exist, but not that GD can reach them.

I agree with the authors that the contribution should be judged apart from the mechanism. Judged that way, I see a useful set of measurements and an interesting EoS observation, but no link yet to what the network learns or how it generalizes. That makes the paper informative but limited in impact. The null results that complicate the return story currently sit mostly in the appendix. If accepted, I’d ask that the final version bring null-relative rank into the main figures and frame the participation-structure hypothesis explicitly as a hypothesis, leading instead with the measurement framework and the EoS result.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Rebuttal I: Can Participation Frequencies Alone Explain Spectral Return?

We thank the reviewers for their thoughtful feedback. Their suggestions have helped us sharpen our understanding of spectral return and its possible connection to structural slack.

A shared concern was whether changes in neuron participation frequencies alone could explain raw-rank recovery. To test this, we extended the return-path analysis in Section 4.1 (Figure 2), comparing raw and centered effective rank against 32 participation-preserving randomizations at every saved checkpoint.

(The results are included in the updated Appendix E of the revised manuscript.)

Two observations stand out.

  • Quantitatively, the nulls show substantial rank recovery, as the reviewers suggested. But they recover considerably more rank than the observed FashionMNIST supports. In CIFAR-10, the amount of recovery is much closer.
  • Qualitatively, the nulls follow broadly similar rank–density return paths across settings. The observed supports, especially in FashionMNIST, trace different shapes, with distinct slopes and directions. Each null preserves the exact participation frequencies and density at every checkpoint, so these differences point to how activations are organized across examples.

This suggests that learned participation relationships may shape the path of return itself, as hypothesized in Section 4.4. Recent work by Liang et al. (2026) shows that data geometry can influence the sharing of ReLU features, raising the possibility that geometry also contributes to these distinct paths.

This does not yet establish the mechanism, but it does give a clearer picture of the phenomenon. We're continuing experiments across more seeds and settings. We'll also follow up with individual responses to the remaining questions from each reviewer as results become available.

We welcome any further thoughts or suggestions in the meantime.

Liang et al. (2026): Generalization Below the Edge of Stability: The Role of Data Geometry https://arxiv.org/abs/2510.18120

AuthorsOpenReview ↗

Rebuttal II: Are Distinct Return Paths Reproducible?

A shared concern was whether the observed return paths and gate dynamics are consistent across initializations.

To test this, we expanded five GD settings to five seeds each (25 runs, including 17 new runs), keeping each setting's architecture, data subset, and training protocol fixed across seeds. The results are reported in the updated Appendix C (Figures 7–10, Table 4).

Three findings stand out.

Spectral return. Condensation occurs in all 25 runs. The shallow FashionMNIST learning-rate contrast holds across all five seed pairs: rank recovery is 20.42 ± 1.23 at η=.04, compared with 0.67 ± 0.51 at η=.02. This difference also persists at matched training loss. Deep FashionMNIST and CIFAR-10 reproduce their characteristic return paths.

Gate dynamics. Deep FashionMNIST return precedes renewed switching in every high-rate seed, by 2053–3008 epochs. Alternating-current episodes also recur across all five seeds in each high-rate setting, although their persistence varies. Notably, none of the deep high-rate runs exhibits a sampled EoS crossing.

Spectral controls. The participation-preserving null results from Rebuttal I also replicate. Null rank recovers more than observed rank across all nonzero-length recovery intervals, while centered rank relative to its null continues to decline.

Together, these results strengthen the evidence that the distinct return paths and gate dynamics are reproducible within the studied training regimes.

We are separately testing robustness across data subsets and training objectives, and will share those results as they become available.

AuthorsOpenReview ↗

Rebuttal III: How General Is Spectral Return?

A shared concern (joRG, 5T5W) was whether spectral return extends beyond the original MSE-trained MLPs. We tested three extensions, reported in the new Appendix G.

Training subsets. Across three FashionMNIST subsets, the shallow learning-rate contrast persists. High-rate rank recovery remains around 24–25 units, while low-rate recovery stays below 1.

Training objective. With cross-entropy, condensation and return appear across both learning rates and two initializations. The higher rate consistently recovers more rank, and its raw rank remains higher at matched training loss.

Architectures. Both a ReLU CNN and a ReLU ViT on CIFAR-10 exhibit condensation and return. Their return paths and gate-current dynamics differ, showing that the phenomenon extends beyond MLPs without following a single trajectory. Across architectures, we retain the same binary-gating definition, measuring hidden neurons in MLPs, channel-location pairs in CNNs, and CLS feedforward units in the ReLU ViT.

Density paths also differ. In the original MLP/MSE experiments, rank recovery accompanies density decrease. In the CE MLPs, rank recovers while density continues rising; in the ReLU ViT, density first falls and then rises. We therefore distinguish spectral return (rank recovery) from its accompanying density trajectory.

The null comparisons reveal a further distinction. From each observed rank minimum to epoch 4000, participation-frequency-preserving nulls recover more rank than observed supports in the MSE-trained MLPs and CNNs. In both higher-rate CE runs and both ViT runs, observed supports recover more than their nulls. Observed ranks remain below their null baselines throughout, but the spectral deficit can widen or narrow depending on the setting.

These results extend spectral condensation and return across the tested settings, while revealing differences in how support organization evolves during return.

The architectural results use one initialization each, and the ReLU ViT is a controlled variant. We therefore treat these as initial architectural extensions.

AuthorsOpenReview ↗

Rebuttal IV: Structural Slack

We thank the reviewers for helping us examine how much of spectral return can be explained by participation frequencies and how reliably the different paths recur. Changing participation frequencies can produce a substantial rank rebound. What remains striking is the variety of return paths across architectures (shallow and deep MLPs; later a CNN, and a ReLU ViT) and datasets (FashionMNIST and CIFAR-10). Viewed alongside the participation-preserving nulls, these patterns come into sharper focus and suggest that the shape and extent of return depend on how optimization reorganizes which neurons share which examples.

What depth suggested. On the same FashionMNIST data, the shallow return climbs or stalls with learning rate, while the deep network traces a tight hairpin as density and rank turn together (Figure 2). The layerwise histories offer a clue. Near the input, almost every neuron continued to participate on at least some examples. In the deeper layers, a wider set of neurons fell silent, and some later re-entered (Appendix D). This contrast led us to ask how much room an existing participation pattern leaves for reorganization. We call that room structural slack.

What the learning rate suggested. Across all five shallow FashionMNIST seed pairs, the higher-rate return arm climbs while the lower-rate arm stays nearly flat, even as its support changes (Appendices C and E.5). At the higher shallow rate, switching resumes near the edge of stability and may help the network rearrange which examples contribute to each neuron’s updates (cf. Nguyen et al., 2026). In the deep runs, rank begins to recover while switching is still sparse (Appendix C).

Why structural slack? The FashionMNIST return paths depart from their participation-preserving nulls in distinct ways. CIFAR-10, on the other hand, follows the upward bend of its null much more closely, even with very few rarely active neurons (Figure 12). On CIFAR-10, the link between class and which examples share neurons is also weaker than on FashionMNIST (Section 4.4), possibly reflecting differences in data geometry (Liang et al., 2026). This gives us one way to read the contrast. On CIFAR-10, the existing pattern of sharing may already leave room for return. On FashionMNIST, return appears more constrained, so the room we see in deeper layers may matter more, and in the shallow network oscillatory gate motion may help open up a path.


Nguyen et al. (2026): A Minimal ReLU Classifier on Non-Linearly Separable Data: Convergence and Implicit Bias at and beyond the Edge of Stability. https://arxiv.org/abs/2610.23007

Liang et al. (2026): Generalization Below the Edge of Stability: The Role of Data Geometry. https://arxiv.org/abs/2510.18120

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.