- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes a function-preserving network-expansion method that expands the network for continual training by considering useful directions. The main experiments are conducted in a very small-scale task setting using 6 images from BSDS500.
Critical strengths
None
Critical weaknesses
- The paper is poorly written and very hard to follow the flow as a reader. Some examples such as:
- The abbreviation OST is used throughout the paper and i could not find anywhere that explains what OST stands for. To the best of my knowledge, it is not a common term in the field that does not need any explanation.
- The variable N, P, and q is undefined in the introduction.
- In Line 71, it says "SKETCH-SCHUR-OST uses the same Schur complement for all three decisions." And it is very unclear that what "three decisions" here refer to.
- Putting the overview figure on page 6 and isn't mentioned in the main text anywhere in the paper.
-
The motivation for growing network without changing its prediction is very weak and unjustified. The authors simply state that "restarting with a stronger architecture is not an available intervention", but never explicitly show the downstream use cases, and does not discuss why common alternatives such as efficient finetuning or distillation isn't a good choice.
-
The experiments are poorly designed and far from realistic to support any reasonable contribution.
- The main experiment is using a model with only 697 to 745 parameters, and it is only using six images from a small and outdated dataset (Line 304-307).
- The baselines in the main exp results are all self-constructed. And closely related baselines such as [1, 2] are not compared.
- Training loss is reported as one of the main metrics for comparison, which I believe have little to no value in this kind of limited data setting.
[1] Verbockhaven et al., "Growing Tiny Networks: Spotting Expressivity Bottlenecks and Fixing Them Optimally." TMLR 2024.
[2] Wu et al., "Firefly Neural Architecture Descent: a General Approach for Growing Neural Networks." NeurIPS 2020.
Questions for authors
See weaknesses.