- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates symmetry-charge drift in forward-only continual learning using node perturbation. The authors theoretically characterize how perturbation noise induces charge diffusion along loss-invariant rescaling symmetries, potentially affecting long-term learning. To mitigate this issue, they propose Gauge-Damped Node Perturbation (GD-NP), which periodically applies function-preserving weight rescaling to restore the symmetry charge toward its initial value.
Critical strengths
None
Critical weaknesses
-
Insufficient connection between symmetry drift and loss of plasticity. The paper provides a theoretical characterization of symmetry-charge diffusion under node perturbation and proposes GD-NP to suppress it. However, it remains unclear to what extent this drift contributes to the observed loss of plasticity. In particular, exact gradient-based optimization can also suffer from plasticity loss despite preserving the symmetry charge under gradient flow. Therefore, restoring charge conservation alone does not establish improved plasticity, and the paper lacks sufficient evidence isolating the contribution of symmetry drift to long-term learning degradation.
-
Questionable justification for restoring the initial symmetry charge. GD-NP explicitly drives the symmetry charge toward its initial value . However, preserving the initial charge does not necessarily yield balanced parameter scales or favorable optimization conditioning, let alone optimal plasticity. Moreover, deviations from are not inherently detrimental and may even lead to better-balanced parameterizations. The paper does not sufficiently justify why restoring is preferable to directly controlling parameter imbalance or effective update scales.
Questions for authors
-
Could the authors provide more direct evidence that symmetry-charge drift is a significant contributor to plasticity loss under node perturbation, rather than merely a correlated phenomenon? In particular, how much of the observed late-task degradation can be attributed to charge drift?
-
Why does GD-NP restore the initial charge rather than targeting balanced parameter norms or directly controlling relative update scales? Could the authors compare these alternative correction objectives in terms of late-task accuracy?