- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper tracks binary neuron-sample activation patterns in ReLU MLPs using support density, effective rank, gate turnover, and net activation current. FashionMNIST and CIFAR-10 experiments show initial spectral condensation followed by different return trajectories across depths, learning rates, and optimizers. In the deep example, density-rank return precedes a late switching surge; selected GD runs show approximately period-two current. Participation analyses track previously isolated neurons, while centered and permutation controls show that raw-rank recovery need not imply recovery of more specific support organization. The contribution is descriptive, motivating an open hypothesis about how participation structure shapes reorganization.
Critical strengths
The paper offers an accessible set of tools for describing neuron participation, with explicit definitions of density, effective rank, turnover, and current. Distinguishing total switching from net activation change makes clear why nearly constant density can conceal substantial exchange. The organization into state, flow, and structure helps readers follow the measurements and provides a useful basis for further investigation.
The study combines spectral trajectories, recurrence tests, and individual neuron histories across depth, learning rate, and dataset settings. In the deep example, rank return near epoch 700 precedes the switching surge near 3138, constraining explanations based solely on that later event. Distinguishing periodic current from recurring support also adds descriptive value, although the broader structural interpretation remains unresolved.
Critical weaknesses
-
The added value of the spectral description remains unclear. The paper acknowledges that raw-rank recovery can reflect changing participation rates. However, Appendix E’s null preserves neuron degrees while changing sample degrees. It therefore does not establish which differences between return paths require support arrangement beyond these marginals. This limits the interpretation of recovery, although the raw-rank observations remain valid. The distinction between raw-rank recovery and centered, null-normalized behavior requires cross-referencing Figure 2, Section 4.1, and Appendix E. Although the caveat is stated, presenting these results together would make the meaning and limits of “return” easier to follow.
-
The temporal comparison remains limited by observation resolution. Epoch endpoints span one GD update versus approximately 40 SGD updates, roughly matching dataset passes. As the paper acknowledges, switches can reverse between observations. The comparison therefore leaves unresolved how much of the contrast reflects persistent gate changes versus hidden switching.
Questions for authors
-
For representative Figure 2 trajectories, compare raw and centered ranks against randomized supports preserving both neuron and sample degrees. Use initialization, the raw-rank minimum, and the endpoint, and report variability across randomizations.
-
Which differences between return paths remain beyond this null? If participation marginals explain the recovery, what additional insight does the spectral description provide?
-
Could you examine short early and late GD/SGD windows on the same fixed probe set, recording supports at every optimizer update? Indicate training location in both updates and examples processed.
-
In those windows, compare turnover and current at one-update and common 40-update intervals. Also report accumulated switches per gate to quantify switching missed by endpoint measurements. Which temporal contrasts persist across observation scales?
Response to mBwc: Spectral Paths and Switching Resolution
Thank you for the careful review and follow-up.
Rebuttal I introduced time-resolved row-permutation controls and addressed your request for a both-degree-preserving null at three checkpoints; Rebuttal II established the five-seed and matched-loss results you acknowledged; and Rebuttal III extended the findings across training subsets, objectives, and architectures.
We have since examined your two remaining concerns.
What remains beyond participation marginals? We extended your requested both-degree control from three checkpoints to the full trajectories of the four representative GD runs (Appendix E.1, Fig. 13). The two nulls closely agree in raw and centered rank, but neither reproduces the observed spectral paths. In shallow FashionMNIST, observed rank reaches its minimum at epoch 400 versus 200 under both nulls; in deep FashionMNIST, the observed minimum is interior while the null minima occur at initialization. The nulls recover more rank, and centered rank relative to its null declines, as you noted. Thus changing participation marginals can reproduce a substantial rebound. What remains distinctive is the observed spectral deficit, the timing of condensation and return, and their rank–density geometry, even when both degree sequences are preserved.
What survives matched observation scales? We included SGD to examine spectral return under continuing gate activity, rather than GD's pronounced freezing and subsequent reactivation. Your sampling concern was justified: epoch endpoints conceal 96.9% and 97.7% of late SGD flip events in FashionMNIST and CIFAR-10. On the same fixed probe, we now compare per-update turnover, accumulated switching, exact 40-update signed current, and turnover at one-data-pass and common 100-update intervals; the distinct switching histories persist (Appendix E.4). SGD reaches its saved rank minimum and partial recovery earlier per data pass, suggesting that training regimes may differ in how readily they access paths of participation reorganization, although the extent of return varies and there is no uniform matched-loss rank advantage.
Together with the reproducibility and generality results, these controls strengthen the evidence for spectral condensation and distinct paths of return, while clarifying their interpretation beyond participation marginals and observation-scale effects.