- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper tries to enumerate the necessary conditions for additional input representations to be beneficial from the perspective of loss reduction. The authors focus on a situation in which there exist two lossy representations of the input. They derive a decomposition of the log risk given one representation and compare it to the risk if the model used both representations. From the theoretical analysis, the authors conclude that adding input representations is not necessarily useful if it leads to higher finite-model approximation errors. They then develop a selection pipeline to select which features to use at test time on an input by input basis and demonstrate that this leads to better performance.
Critical strengths
This paper asks an interesting and relevant question: What is the value of including more features in a sequence-prediction task. The authors make an effort to formalize this question theoretically. The finding that more features (or employing a dual strategy to use the terms of the paper) is not universally optimal is conceptually interesting and potentially useful for practitioners.
Critical weaknesses
My primary concerns with this paper are clarity and mismatch between theory and experiment– namely that the experiments do not directly test the theory.
-
Clarity: I found the writing in this manuscript very hard to follow. The wave/particle analogy is not necessary and really only obfuscates the point of the work. These are defined in the paper as two arbitrary lossy representations of the input sequence (Line 48), which implies that there is no real reason to label them as Wave and Particle. While the authors discuss these representations as fundamental objects, it is my opinion that the entire paper could be written without using this language, suggesting that they do little work in proving the claims of the paper. In lines 234-247, the authors even acknowledge that this terminology could be confusing given its ubiquity in quantum mechanics, but they don’t make a convincing case for why it should remain as the primary framing of the work. In addition, there is a general tendency toward jargon and away from clearly defined terms throughout. For example, the terms “deployment cost” , “task-risk”, and “task native loss” are used interchangeably, but none of these are clearly defined in the paper. The two core objects of the paper, W and P are only defined abstractly, and it is never mentioned what their functional form is. Even if they are allowed to be arbitrary functions, which functions were used in experiments is unclear. The selector experiment uses “spectral concentration, normalized spectral entropy, detrended concentration, local difference-energy entropy and peak concentration, effective event support, and relative shock magnitude” (line 275), but none of these are defined, making it difficult to reproduce the results.
-
Theory experiment mismatch: The theory is straightforward and simply relies mostly on the chain rule for mutual information. The takeaways from the theory are clear, but they demonstrate the relatively obvious result that more representations of the input can be detrimental if it leads to worse approximation error. Furthermore, the experiments themselves do not actually use the log risk and so they do not explicitly test the theory. The details of the experiments (how exactly is data split, what is the Base model, etc.) are not readily available in the paper.
Questions for authors
How does C_h factor into your experiments? All theoretical results are conditioned on this quantity, but it is never defined for your experimental setting.