- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposed a metric Task Entropy to evaluate the task-side representational demand. For an N-way identification task, the authors showed that the number of classes that can be separated with a fixed margin is determined by packing capacity of the representation space. They then use the concept to compare the euclidean and hyperbolic spaces and empirically supported their results on WordNet data.
Critical strengths
-
The perspective of using task demand to re-visit the representation space capacity is interesting.
-
The main capacity results are clean and easy to interpret.
-
The authors are careful about the interpretation of the empirical results.
Critical weaknesses
-
The main theorem rely on the assumption that each task state has its own score coordinate and that success can be formulated as N-way margin identification. It is unclear how relevant the real representation learning objectives satisfy this.
-
The WordNet results do not clearly show that higher task demand gives a larger hyperbolic advantage. The hierarchy-versus-coarse comparison is seed-sensitive and reverses under the K=22 task definition, and the experiment uses metric stress rather than the identification objective assumed by the theorem.
-
The task metric, the resolution, and the scale parameter can all change the packing rate. It is unclear to what extent task entropy is an intrinsic property of the task itself.
Overall, these concerns limit the generality and empirical support of the proposed framework, which is why I lean toward weak rejection despite finding the core idea interesting.
Questions for authors
-
How sensitive is task entropy to the choice and normalization of the task pseudometric?
-
What role does the tree structure actually play in Theorem 9 beyond providing exponentially many leaves?