- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper looks at the idea of using local information in neural network encodings. The idea is that deep learning encodings are often pooled in a global fashion before being used in the final classification layer, which throws away information. This paper instead proposes that local instances can augment the global information and improve performance. They propose to do the instance selection via reinforcement learning. The authors compared their method against several standard benchmarks on the UEA multivariate time series datasets, and provided some synthetic experiments. Performance seems competitive with standard benchmarks such as MINIROCKET in accuracy, though I could not find any timing information.
Critical strengths
-
The basic idea is interesting: to exploit the local information in encodings that gets thrown away after global pooling. The basic algorithm is actually quite simple, which should be read as a positive here, and as such should be adaptable to other models.
-
The results look competitive with other benchmarks, though as usual, how much tweaking and tuning was done to achieve them is difficult to assess. The synthetic data example is nice and Figure 3 is great, and really helps sell the idea to the reader.
Critical weaknesses
-
Unfortunately, I found this paper very difficult to read and parse, especially as the underlying idea is not immediately obvious. It took me some time to realise that the local features are being selected per instance, by the RL mechanism, and then given the same coefficients regardless of which local feature (i.e., what ) they are. This is because these ideas are buried in the text, when it should be presented up-front. For this idea to make sense it relies on the local features somehow having the same semantic meaning regardless of which local instance they are. That might be perfectly valid for the architecture they have chosen but is not discussed at all, which is strange as this seems to be the key idea underlying the paper. It is not immediately obvious to me why different features from the encoder should work well with the same logistic regression coefficients. Some discussion of this topic is crucial.
-
There are also issues with notation that make what is ultimately a reasonably simple idea, at least mathematically, initially quite difficult to understand. Symbols are introduced without sizes (e.g., , -- these are matrices), and poor notation obscures what is actually going on. To wit, eq (8) defines as the classifier, but then the text before eq (10) says the classifier and encoder and trained using (10). It was not immediately obvious to me whether only the weights are updated, or whether the encoder parameters are updated as well as doesn't show any dependence on .
-
A presentational issue is the presence of what seems to me to be irrelevant Propositions. I do not really see what use they serve the paper or exposition? Proposition 1 seems to link mutual information to Bayes log-risk, and Proposition 2 seems to be a restatement of the data processing inequality. These would appear to be standard results, but even beyond that, neither of these results appear to be invoked anywhere in the paper other than as high level justifications of the general idea of using local lists; there doesn't really appear to be much use specifically for the authors architecture. They seem to add clutter to the paper without seeming to provide any payoff.
-
Finally, the paper also introduces a very elaborate mechanism for training via reinforcement learning. Eq (7) appears to be a sparse linear model -- albeit one in which the features are adaptively chosen per instance. It is not clear to me that reinforcement learning is necessary to achieve this. There is a small ablation against an attention alternative which attains poorer results, but overall it seems possible that the level of complexity around using RL is not necessary and the paper does not provide any compelling arguments, beyond a single empirical ablation study (presented without any confidence intervals), to justify its use.
-
Finally -- it seems to me that what the authors are proposing is really just a sparse, adaptive form of global pooling. This is because the selected s enter the model in the order of selection, not in any temporal order. Model (7) could be thought of as giving additional weight only to the selected representations, beyond their contribution to the global average, and then pooling them by a weighted linear average. In fact, from (1), if denote the indices of the selected instances, then you can rewrite (7) as
So this appears to be an adaptive (matrix weighted) pooling mechanism rather than augmenting global with local features, per se.
Questions for authors
-
What value do the Propositions bring to the paper? Where and how are they used to justify your particular choices beyond high level justifications.
-
To follow up -- why do we need Proposition 1 to justify why we need a Proposition to know that local information may exist? It seems that it would be trivial to propose time series classification problems where the local position of a simple temporal feature would be the prime (only) discriminator between two classes.
-
What is "sg"?
-
Where are the timing results? There are some ms / instance results reported in Table 3 but I think that timing results for training and at prediction time, vis a vis the other benchmarks, are necessary.
-
Before Eq (10) it is said you update the encodings and classifier, but (10) shows no dependency on the encoding. Is this a typo or are you also updating the encoding?
-
Why is it that different local instances can use the same weights effectively in the classifier? If these were just standard features to a classifier, then this would obviously not make any sense; I couldn't use height for one individual and weight for another as my first feature with the same coefficient, which, if I understood, is essentially what you are doing. So what is the fundamental aspect of the model that makes it work for these encodings? This is important to understanding the idea.
-
Please comment on the connection to adaptive global pooling.
3 comments
The authors have addressed a number of my questions. I acknowledge that the presentation and experimental evaluation has been substantially improved. However, some of my fundamental concerns remain. To wit:
-
Regardless of how the authors wish to interpret equation (7) it remains a form of (adaptive) weighted global pooling over the features. In fact, concatenation followed by a linear classifier is algebraically equivalent to weighted additive aggregation over that concatenated representation. I do not believe that the distinction that the authors draw between their approach, and pooling, is particularly compelling.
-
I maintain that Propositions 1 and 2 do not add anything substantive to the paper. Specifically, Proposition 1 appears to be a standard relationship between conditional mutual information and Bayes log-risk. Proposition 2 appears to be the data-processing inequality. Beyond being well known results, neither provide any specific insight into the proposed algorithm; in fact, they are sufficiently general that they could be invoked to justify an enormous range of methods. I do not see that they contribute meaningfully to understand the specific idiosyncracies of LIST.
-
Finally, I remain unconvinced that reinforcement learning is necessary, or even particularly well justified for this problem. In fact, in the authors expanded experimental comparisons (which are appreciated), the results indicate that LISTS does not significantly outperform either Attention-3 or Top-3 under the authors own statistical testing procedure. Both are simpler alternatives with lower computational costs and are more readily understood. As such, the results reinforce my original concern than the addition complexity (computationally, and methodologically) of reinforcement learning is inadequately justified.
As a final note, the paper combines together a number of established machine learning techniques, without, in my view, providing any compelling theoretical or methodological argument for why this specific combination is preferable to some other combination, or a simple method. The provided theory is too general to answer these questions. I acknowledge the empirical results are promising, but without strong methodogical justification it is difficult to know whether these will generalize beyond the repositories on which the method was developed.
I appreciate the authors' efforts in revising the paper -- and do not dispute that the method achieves competitive performance on the evaluated data sets. However, I believe that the paper is not of the standard required for ICLR and am happy to retain my original score.
Thank you for your follow-up response. We appreciate the time you have taken to review our revised manuscript. However, we believe that several of your remaining concerns shift the discussion away from the actual contributions of LISTS. We respectfully disagree that these concerns undermine the methodological contribution or novelty of our work. Below, we address each point directly.
1. Algebraic Equivalence Does Not Challenge Our Core Contribution
We agree that Eq. (7) can be algebraically rewritten as a form of adaptive weighted pooling. However, this algebraic equivalence does not undermine any of the methodological contribution or novelty of LISTS.
Our claimed novelty has never been the invention of a new algebraic aggregation operator. Rather, it lies in the problem formulation and the mechanism for selecting complementary local information beyond an existing global representation.
Eq. (7) only describes how the selected representations contribute to the final prediction. It does not characterize how these representations are selected or how the selection policy is optimized. Therefore, we believe that repeatedly focusing on the algebraic form of Eq. (7) overlooks the actual contribution claimed in our paper.
2. Propositions 1 and 2 Directly Motivate Our Key Design Choices
We have already clearly explained how Propositions 1 and 2 directly motivate our key design choices in both the methodology section of our paper and our rebuttal:
Proposition 1 formally quantifies the complementary class information provided by local instances beyond the global context. This directly motivates our objective of selecting local instances that provide complementary information.
Proposition 2 establishes that compressing global and local representations may discard class-relevant information. This directly motivates our design choice to preserve the selected local evidence alongside the global representation through concatenation.
Furthermore, regarding your concerns that these propositions are "too general" or "could be invoked to justify an enormous range of methods," we would like to clarify that the generality of a theoretical result does not diminish its relevance to a specific methodological design. On the contrary, many foundational results in machine learning are broadly applicable, and this generality is precisely what makes them valuable, as they can provide theoretical principles and motivation for a wide range of methods.
3. Reinforcement Learning Is a Justified Design Choice, Not a Claim of Necessity
We have never claimed that reinforcement learning itself is our novelty, main contribution or the only possible optimization method. REINFORCE is a design choice for optimizing our sequential discrete selection policy.
Every machine learning framework necessarily involves design choices, and a design choice does not need to be the only viable option, or significantly outperform every alternative, to be justified. The relevant question is whether the choice is well motivated and supported by empirical evidence.
In our case, REINFORCE provides a practical mechanism for optimizing sequential discrete selection, with relatively modest computational overhead. Regarding performance, LISTS-3 achieves the highest average accuracy among the compared selection strategies on the 30 UEA datasets. Although not all of these differences are statistically significant after Holm correction, this does not imply that REINFORCE is unjustified. Furthermore, on the synthetic Sleep benchmark, LISTS-3 achieves substantially stronger localization performance, with All-Hit@3 reaching 86.00%, compared with only 38.00% for Attention-3.
These results provide concrete empirical support for our choice of REINFORCE, demonstrating favorable performance in both classification and localization with relatively modest computational overhead.
4. Response to the Reviewer's Final Note
We respectfully disagree that using established machine learning techniques undermines methodological justification. Most deep learning frameworks build upon existing components, and methodological justification does not require inventing every component from scratch.
In LISTS, our theoretical analysis provides a clear rationale for the proposed framework. Our experiments, especially the ablation studies, also provide empirical justification for our key design choices.
Finally, we hope that our clarifications address your remaining concerns. Thank you again for your time and consideration of our work.
Thank you again for your suggestions! We have updated the manuscript with additional experiments, computational efficiency analyses, and improvements to the presentation of our method. All revisions are highlighted in blue in the updated PDF. We have also summarized the main changes in our general comment. We hope these revisions address your concerns.