- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
In this paper, the authors study the representation of neural networks using quanta, which is a description between two extreme descriptions of learning dynamics: the parameter trajectory (microscopic) and the scalar loss (macroscopic). They introduce the demand, the conditional complexity, and the acquisition priority to measure which quantum is acquired earlier. They empirically study how acquisitions compose, why their aggregate looks smooth, and how macroscopic regularities arise. They also empirically investigate a task of recovering quanta.
Critical strengths
This paper tries to clarify an important problem: how the models can be represented using reusable and composable elements. The topic is very interesting. They try to understand the phenomena that a quantum tends to be acquired earlier when it is useful to more examples and when it is simpler given the currently available computations by introducing the notion of demand and conditional complexity. The idea is interesting.
Critical weaknesses
My major concern is about the conditional complexity that the authors introduced. How to compute this value in practice is unclear for me. d(I) is not available before the training. In addition, is it reasonable to assume that there is a single direction in parameter space that builds the quantum q? Since in the overparameterized regeme, the same function can be represented even with different parameters. Thus, I am not sure if we have one single direction.
Another concern is about the presentation. Section 3 is too dense, and it was difficult for me to clarify which parts and formulas are important in this work. I think adding figures that describe the concept schematically would help readers to understand. Also, some displayed formulas should be added to show important definitions and statements.
Questions for authors
Please see the weakness section.
2 comments
Thank you for the clarification. The two proposed methods for estimating the complexity still seem to need separate runs, and my concern has been only partially resolved. I would like to keep my score unchanged.
Dear reviewer J7bq,
We wanted to politely follow up on whether our rebuttal has addressed your concerns. Since the time for author-reviewer engagement is soon coming to a close, we would appreciate the opportunity to engage further if any of your concerns remain unresolved or if additional work would help improve your assessment of our paper.
Otherwise, if you feel that our responses have satisfactorily addressed your concerns, we would be grateful if you would consider increasing your score.