acceptodds
Under review as a conference paper at ICLR 2027

WHEN IS A SECOND REPRESENTATION WORTH ACTIVATING? INFORMATION WAVE–PARTICLE DUALITY FOR SEQUENTIAL LEARNING

Abstract

Modern sequence models increasingly combine spectral, temporal, local, and global views, implicitly treating more representation as better. We show why that assumption can fail under finite model capacity and ask a more basic question: when is a second representation worth activating? We introduce Information Wave–Particle Duality (IWPD), where a distributed spectral Wave view and a localized Packet view are alternative lossy representations of the same observed history. Under Bayes log loss, the context-specific value of adding one view after the other is exactly a directional conditional mutual information. For restricted predictors, we derive a finite-capacity decomposition showing that positive population information can still be offset by approximation burden; adding representation cost yields an explicit activation law. We instantiate the Wave branch with a constrained maximum-entropy spectral prior and deploy the theory through history-only prediction of the task risk of Base, Wave, Packet, and Dual actions. Experiments across five datasets reject universal Dual superiority. Most importantly, under an identical-input, identical-split four-action trajectory protocol, the history-only selector attains lower five-seed mean ADE than the condition-wise best fixed action in all 12 highD/NGSIM horizon conditions. Failures outside this setting are retained explicitly. The resulting principle is not “fuse everything,” but activate a representation only when its conditional value survives model and cost constraints.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
information-theoretic representation learning, sequential learning, representation selection, conditional computation, conditional mutual information, spectral representation learning, maximum entropy, time-series forecasting

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_u66nOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper tries to enumerate the necessary conditions for additional input representations to be beneficial from the perspective of loss reduction. The authors focus on a situation in which there exist two lossy representations of the input. They derive a decomposition of the log risk given one representation and compare it to the risk if the model used both representations. From the theoretical analysis, the authors conclude that adding input representations is not necessarily useful if it leads to higher finite-model approximation errors. They then develop a selection pipeline to select which features to use at test time on an input by input basis and demonstrate that this leads to better performance.

Critical strengths

This paper asks an interesting and relevant question: What is the value of including more features in a sequence-prediction task. The authors make an effort to formalize this question theoretically. The finding that more features (or employing a dual strategy to use the terms of the paper) is not universally optimal is conceptually interesting and potentially useful for practitioners.

Critical weaknesses

My primary concerns with this paper are clarity and mismatch between theory and experiment– namely that the experiments do not directly test the theory.

  1. Clarity: I found the writing in this manuscript very hard to follow. The wave/particle analogy is not necessary and really only obfuscates the point of the work. These are defined in the paper as two arbitrary lossy representations of the input sequence (Line 48), which implies that there is no real reason to label them as Wave and Particle. While the authors discuss these representations as fundamental objects, it is my opinion that the entire paper could be written without using this language, suggesting that they do little work in proving the claims of the paper. In lines 234-247, the authors even acknowledge that this terminology could be confusing given its ubiquity in quantum mechanics, but they don’t make a convincing case for why it should remain as the primary framing of the work. In addition, there is a general tendency toward jargon and away from clearly defined terms throughout. For example, the terms “deployment cost” , “task-risk”, and “task native loss” are used interchangeably, but none of these are clearly defined in the paper. The two core objects of the paper, W and P are only defined abstractly, and it is never mentioned what their functional form is. Even if they are allowed to be arbitrary functions, which functions were used in experiments is unclear. The selector experiment uses “spectral concentration, normalized spectral entropy, detrended concentration, local difference-energy entropy and peak concentration, effective event support, and relative shock magnitude” (line 275), but none of these are defined, making it difficult to reproduce the results.

  2. Theory experiment mismatch: The theory is straightforward and simply relies mostly on the chain rule for mutual information. The takeaways from the theory are clear, but they demonstrate the relatively obvious result that more representations of the input can be detrimental if it leads to worse approximation error. Furthermore, the experiments themselves do not actually use the log risk and so they do not explicitly test the theory. The details of the experiments (how exactly is data split, what is the Base model, etc.) are not readily available in the paper.

Questions for authors

How does C_h factor into your experiments? All theoretical results are conditioned on this quantity, but it is never defined for your experimental setting.

Reviewer_3c2jOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a criterion for deciding whether including a second representation of a time series in a sequence model is worth it. The criterion compares a conditional mutual information term, representing the information gained by the additional view, with the sum of a finite-model approximation term and a deployment-cost term. Finally, the authors propose a practical surrogate for inference time, when future history is not available and validate their methods on trajectory (highD, NGSIM), finance and weather datasets.

Critical strengths
  • Understanding which representation is the most relevant in forecasting is an important problem with a lot of applications
  • The proposed criterion is very comprehensive as it encompasses both information present in the data and the model capacity
Critical weaknesses
  • The related work section is incomplete: the authors skip all the literature on generative mutual information estimators such as [1,2,3]; this is important because the authors motivate the fact that they don't estimate mutual information because estimators are unreliable. Likewise, the authors ignore a vast transfer entropy literature [4,5]. Crucially, the quantities in Eq. (2) are conditional mutual informations of the same form as transfer entropy, which is recovered when the conditioning view is the target’s own past; this connection should be discussed.

  • The terms in Eq. (9) are never measured: deployment costs are a structural count proxy rather than measured compute, and the approximation gaps εF\varepsilon_F are never estimated, nor is model capacity varied. It is therefore difficult to say what characteristics of the model and of the representations make one view better than the other. This is important because as of now the paper introduces definitions and standard identities without deeper insights into the central research question of the paper.

  • The authors should be more specific about the setting in Section 4: the main text never defines ϕw\phi_w, ϕp\phi_p, the fusion mechanism, or how the maximum-entropy prior enters the Wave branch, and these should be presented rigorously.

[1] Foresti et al. ; Information estimation with discrete diffusion

[2] Franzese et al. ; Minde: Mutual information neural diffusion estimation

[3] Butakov et al. ; Infobridge: Mutual information estimation via bridge matching

[4] Bossomaier et al. ; Transfer entropy

[5] Munoz et al. ; TENDE: Transfer entropy neural diffusion estimation

Questions for authors
  • Can the authors elaborate on how they quantify the approxumation gaps and the model capacity? What kind of insights are we gaining when we instantiate them explicitly?
  • Why are some gains in table 2 negative when the theory implies GW(c)≥0G_W(c)\geq 0?
Reviewer_qPXZOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies when global spectral and localized temporal representations should be used separately or together. Their conclusion is that more representation is not always monotonically better, but is rather dependent on direction, horizon, predictor class, and deployment objectives.

Critical strengths

The paper is honest about its scope and expeirment conclusions by including several negative and mixed results.

Critical weaknesses

Writing quality can be improved significantly. The intro is quite dense for someone without sufficient backgrounds. Several expressions like "This section turns the paper’s central question into mathematics." and "The empirical results sharpen the theory rather than simply confirming it" (there are many others) read very AI-heavy and can use some human tuning.

Questions for authors

None.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.