- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
I am unable to summarize the contributions of the paper because, despite my best attempts, i largely do not understand it. (see below)
Critical strengths
The authors have benchmarked five realistic deep learning models on a large benchmark comprising over one hundred tasks.
Critical weaknesses
(1) The problem statement and contribution of the paper are unclear. Quote:
The question is underdetermined as posed, because it locates phase sensitivity in the model
I do not see why the question is underdetermined as posed: all it takes is to check whether, for some model h and any two signals x and y with different phase spectra, h(x)≠h(y). Perhaps this is the wrong question to ask in an applied, but it is certainly not underdetermined. Then, the authors claim that there is a need to "reframe" the question by ascribing phase sensitivity to be a property of the task. I do not see how this is a kind of "reframing": rather, it is a whole other question. The authors are presenting the reader with a false dichotomy (either a property of the model or of the task) whereas it seems to me that both can coexist. The authors give an argument in Figure 1 that, i must confess, i have not understood after reading it multiple times:
Whether phase awareness can help is bounded by a logically prior quantity: how much of the task’s label information depends on phase at all—a property of the joint distribution of signals and labels, measurable without committing to an architecture. Once measured it varies so widely (zero to fifty-plus accuracy points) that averaging a model comparison over an arbitrary basket must produce unstable conclusions—the signature of a task property treated as a model property. I do not know what "varies so widely" nor over what domain. I do not know what "an arbitrary basket" or "unstable conclusions" are. It seems to me that the authors are trying to state a problem that does not really exist in the signal processing community, and then proceed to solve it in a way that is inadequate and inconclusive. The only theoretical result in the article, Proposition 1, is obvious and presented with excessive jargon. There are two many intermediate notations: what is the point of Z=amp(X) and eta_y(x)=P[X=x|Y=y]? What is sigma? But the bigger problem is that this Proposition has no practical utility since, by the authors' own admission, it concerns abstract quantities ("Bayes estimands"). The empirical version of \Delta\phi "has no guaranteed sign". Likewise, decomposition (3) is obvious and does not bring insight. It is purely a matter writing a difference of two terms as a sum of three differences of two terms by introducing telescoping.
(2) I have found the whole paper very hard to follow, and at parts inscrutable. As a specialist of machine learning for signal processing, i am puzzled by terms such as
"the empirical teeth"
"orbit-level disagreement is a sufficient witness"
"benchmark-level ordering is measured, per-task phase dependence only screened"
"the complex result binds one trunk"
"the readout matters more than the arithmetic"
"two points of a common fibre carry different labels with positive probability" (N.B. probability is defined over Borel sets, not points),
"Position” and “relative phase” are shorthand for these group- and repertoire-relative quantities, not identified information-theoretic ones; since the two sides also differ in candidates and budget, both are pipeline-specific performance difference"
and many more. The paper is poorly organized, with numerous repetitions of almost exactly the same claim: see for example "single-dataset use is a screen" or "input and readout, not arithmetic". Sections 3 through 7 are particularly challenging to read: more than once, i have felt like the paper might have been completed in a rush, with little attention to clarity.
Questions for authors
- Are you able to explain, in simple terms, what i should be seeing in Figure 2 and Table 6? This figure is not directly referenced in the text.
- Did this huge benchmark allow you to come up with practical recommendations for researchers and engineers working on neural network design? If so, how confident are you with your recommendations?
- Your paper does not discuss architecture design at all. Most of the models you benchmark have already been published: InceptionTime, MiniRocket, etc. Are you able to explain your results, at least tentatively, by linking specific design choices (in terms of architecture, initialization, choice of nonlinearity, loss function, optimizer, etc.) to Delta Phi?