acceptodds
Withdrawn from ICLR 2027

Sequential Disentangled Representation Learning by Dual-Transformer Subtraction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Sequential disentanglement, which decomposes temporal observations into time-invariant (static) and time-varying (dynamic) factors, is fundamental to interpretable representation learning. We present DTS (Disentanglement by Dual-Transformer Subtraction), a simple framework that combines an architectural inductive bias for static–dynamic separation. DTS employs a hierarchical U-Net of dual-stream Transformers: at each level, a static stream without positional encoding runs in parallel with a position-encoded full stream, and their difference is a dynamic residual feeding the next level. We show that each level's static code is permutation-invariant by construction and that, under a weak-dependence assumption, its mean-pooled estimate concentrates at rate . With this architecture, a compact objective suffices: reconstruction with an optional second pass and at most one covariance regularizer. On six video, speech, and time-series benchmarks, DTS achieves the competitive results with the strongest baselines, fully unsupervised and modality-agnostic.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Feature Disentanglement, Static and Dynamic Factors, Unsupervised Feature Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_fbtjOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes Disentanglement by Dual-Transformer Subtraction (DTS), a sequential disentanglement framework built on a hierarchical U-Net of dual-stream Transformers. A static stream without positional encoding produces a permutation-invariant, mean-pooled representation, which is subtracted from the full stream to separate time-invariant from time-varying factors. Two additional losses support the separation: a covariance regularization loss (CovReg) and a dynamic-only reconstruction loss. The authors also provide a theoretical result (Lemma 1) showing that the static code converges to its conditional mean given the static factor at rate O(1/T)O(1/\sqrt{T}), where TT is the input sequence length. The method is evaluated on video, audio, and time-series benchmarks.

Critical strengths
  1. State-of-the-art results on audio and time series. DTS achieves state-of-the-art results on both modalities, which is the main empirical evidence supporting the effectiveness of the proposed approach.

  2. A new modeling direction for sequential disentanglement. Instead of the VAE/GAN formulations and standard bottlenecks used by most prior work, DTS separates static and dynamic factors through architectural subtraction between two Transformer streams. This is a distinct design choice that could open a new line of work in the field.

Critical weaknesses
  1. Limited and saturated video evaluation. Lack of quality is a known limitation of many sequential disentanglement methods, as noted in [1] and acknowledged in the introduction. However, the video evaluation relies on Sprites, which is saturated (Table 1) and no longer discriminative. The paper does not evaluate on MUG, a standard benchmark in related work it cites [1, 2, 3], nor on CelebV, VoxCeleb1, and TaiChi, which [1] introduced as harder, more complex video benchmarks for both quality and disentanglement. Without these, it is unclear whether DTS also suffers from limited quality in the video modality. At minimum, MUG should replace or complement Sprites.

  2. Claims that overstate the advantage over prior work. The introduction motivates DTS by the large number of loss hyperparameters in prior methods, yet DTS itself has three loss terms: more than DiffSDA [1] and the same as DBSE [2]. Section 3.4 contrasts DTS with the "five interdependent weights of prior methods (Zhu et al., 2020; Han et al., 2021; Bai et al., 2021)" while omitting these newer methods, which is partially misleading. Similarly, Section 4.2 states that DTS learns "without task-label supervision during representation learning", but all compared methods are unsupervised, so this is not a distinguishing property. A table comparing the loss terms and loss weights of DTS against all compared methods should be added, and these claims should be rephrased in the proper context.

  3. Unclear significance of Lemma 1. The paper states that "the lemma justifies the static stream as a stable estimator of what is constant over the sequence." However, showing that s(l)s^{(l)} converges to s∗s^{\ast} does not imply that s∗s^{\ast} is itself a good representation of the static factors. The authors should justify this link more concretely, either theoretically or empirically (see Questions 6–7).

[1] DiffSDA: Unsupervised Diffusion Sequential Disentanglement Across Modalities. Zisling et al., 2026. [2] Sequential Disentanglement by Extracting Static Information From A Single Sequence Element. Berman et al., 2024. [3] Sample and Predict Your Latent: Modality-free Sequential Disentanglement via Contrastive Estimation. Naiman et al., 2023.

Questions for authors

Questions

  1. Can you provide a detailed architecture description, including what the modality encoders and decoders are, where the down- and up-convolutions are located (as expected in a U-Net), and how the static stream, full stream, and decoder are built? An additional figure showing the internal structure of the blocks would help.
  2. How compact are the static and dynamic codes? Do their shapes differ across modalities?
  3. Can the dynamic-only reconstruction loss cause information collapse in the full stream?
  4. Is the ss used for static swapping taken before or after the projection?
  5. Did you need different architectures for video, audio, and time series?
  6. s∗s^{\ast} is used without a clear definition. Can you define it and explain how it relates to the static factors of the data?
  7. Do you have any analysis showing that approaching s∗s^{\ast} yields a better static representation, i.e., that longer input sequences lead to better static representations?
  8. Do you have any analysis of how well the static and dynamic codes are separated at each level of the U-Net hierarchy?

Notes / Suggestions

  1. Can you add reconstruction and swapping examples for video and audio as qualitative results?
  2. The audio experiments lack quality metrics. Can you report MSE for reconstruction quality and DNSMOS for audio quality?
  3. DTS cannot sample new data, which is a disadvantage relative to existing generative methods. This limitation should be stated in the main paper.
  4. Related work on unsupervised representation learning should be added to place this work within the broader field.
  5. If the dual-stream Transformer is an established architecture, can you cite the relevant paper? Citations for U-Net and "Attention Is All You Need" are also missing.
  6. Can you state in the abstract and introduction what converges at rate O(1/T)O(1/\sqrt{T})? Also, TT is used before it is defined; please define it as the input sequence length.
  7. TF\mathrm{TF} is used in Section 3.1 without a clear definition.
  8. s(l)s^{(l)} has several different definitions across the paper, sometimes within the same subsection (e.g., Section 3.5), which makes it hard to follow. In the unnumbered equation of Section 3.5, s(l)s^{(l)} appears as a function, while elsewhere it denotes a latent representation. Should this be TFstatic\mathrm{TF}_{\text{static}}?
  9. Tables 3–6 are not ordered correctly.
Reviewer_U7a9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes an unsupervised framework called DTS for separating static and dynamic components in sequential data. Its hierarchical architecture pairs a Transformer without positional encoding with a position-aware Transformer to obtain dynamic residuals by subtracting pooled static codes. The training process combines reconstruction, covariance regularization, and an optional purely dynamic reconstruction objective function. This paper presents a theoretical analysis of the architectural features and evaluates DTS on six video, speech, and time-series benchmark datasets, reporting competitive results and ablation experiments on key components.

Critical strengths
  1. DTS proposes a coherent architectural scheme for static-dynamic separation by combining an out-of-order static data stream, a position-aware complete data stream, and a learning-based subtraction technique. Estimating static information from multiple observations provides a reasonable alternative to relying on a single sequence element. The matching capacity control results in Table 9 further corroborate this contribution, demonstrating that both the learning-based static estimator and the subtraction technique improve the evaluation metrics.
  2. This paper presents experiments on six benchmark tests in the fields of video, speech, and time series, achieving significant improvements over powerful baseline models. DTS reduced ETTh1’s mean squared error from 9.89 to 8.267 and increased the accuracy of air quality predictions from 69.2% to 81.25%, while also reducing speaker verification error on both speech datasets. A paired-swap experiment conducted on the Sprites dataset further confirmed the effectiveness of factor separation by simultaneously validating appearance transfer and action preservation.
  3. The theoretical discussion provides valuable insights into this architecture by formalizing permutation invariance and establishing a relationship between estimation accuracy and sequence length and time dependence.
  4. This paper is written clearly and is well-organized. The introduction to the research motivation, network architecture, and training objectives is clear, and Figure 1 and the relevant equations make the process easy to understand. The ablation experiments are also well-structured, helping readers connect specific design choices with the reported results.
Critical weaknesses
  1. I am not convinced by the explanation for dynamic-only reconstruction. In Section 3.3 (lines 219–232), all static codes are set to zero, while the dynamic code is required to reconstruct the complete input. The authors argue that excluding static information from the dynamic code is the responsibility of subtraction and CovReg. However, reconstructing the complete input still encourages the dynamic code to retain static information. The decoder may prioritize the static codes when they are available, but this does not mean that the dynamic code no longer contains this information. Table 8 shows improvements on the PhysioNet prediction tasks but does not directly examine this trade-off. Appendix A.5 acknowledges this risk, which is helpful, but the issue remains unquantified. For configurations where this loss is enabled, I would like to see whether it reduces dynamic information in the static codes without increasing static information in the dynamic code.
  2. The window-agreement term does not appear sufficient to establish static-factor consistency across different temporal windows. Section 3.2 (lines 184–192) constructs the two representations by dividing the Transformer outputs into temporal halves and pooling each half separately. However, the Transformer has already processed the entire sequence, so both halves can contain information from the same observations. They may therefore agree by sharing a global summary, even when that summary contains dynamic information. The authors acknowledge in Section 3.3 that order-free statistics of the dynamics can satisfy this constraint without incurring a penalty. My concern is that the experiments do not directly establish whether the learned static codes avoid this behavior. A comparison with independently encoded temporal windows, together with tests of which dynamic attributes can be predicted from the static codes, would provide stronger evidence for the proposed mechanism.
  3. Experimental results do not fully demonstrate better disentanglement than existing methods. On Sprites, several baselines in Table 1 already achieve 100% action accuracy, while DTS and SKD obtain IS scores of 9.000 and 8.999, respectively. The authors acknowledge the limitations of these metrics and introduce a more informative paired-swapping evaluation in Table 2. However, this table reports only DTS, so it remains unclear whether its factor separation is better than that of the strongest baselines. For speech, Appendix A.2 specifies that speaker verification uses cosine similarity between mean-pooled utterance-level embeddings. While this provides useful evidence, it does not rule out identity information in the full dynamic representations or demonstrate that useful speech content is preserved. I would like to see baseline comparisons under the Table 2 protocol, stronger identity probes on the dynamic representations, and a speech-content evaluation such as phoneme recognition.
  4. The claim of a nonzero, T-independent error floor in Proposition 3 does not hold as stated. Following Equation (A.6), the proof in Appendix A.1 states that ∥uˉ∥22\|\bar{u}\|_2^2 does not depend on the sequence length T. However, uˉ=T−1∑t=1Tut\bar{u}=T^{-1}\sum_{t=1}^{T}u_t itself depends on TT. For example, if the dynamic components are independent and identically distributed with zero mean and covariance Σu\Sigma_u, then E∥uˉ∥22=tr⁡(Σu)/T\mathbb{E}\|\bar{u}\|_2^2=\operatorname{tr}(\Sigma_u)/T, which converges to zero. When the representation error ξt\xi_t is zero, the temporal-mean baseline can therefore also have vanishing expected error. Without additional assumptions, the decomposition in Equation (A.6) does not establish a nonzero error floor independent of T. This weakens the theoretical argument for using a separate static stream rather than temporal averaging. The authors should revise this conclusion or explicitly state the conditions under which it holds.
Questions for authors
  1. When “Dynamic-Only” reconstruction is enabled, how does this affect bidirectional information leakage? Table 8 reports an improvement in downstream performance, but can the authors demonstrate whether this loss can reduce dynamic information in the static encoding without increasing static information in the dynamic encoding?

  2. Since the two temporal subséquences are merged after full-sequence attention, what evidence rules out the possibility of reaching agreement through a shared global summary that includes dynamic information? Is it the independently encoded windows or the dynamic attributes predicted directly from the static encoding that support the argument for this expected interpretation of window consistency?

  3. Under the paired-swap protocol in Table 2, how does DTS perform compared to the strongest baseline methods? For speech data, is there evidence that dynamic representations not only make speaker identification difficult through mean-pooled embeddings and cosine similarity but also preserve speech content?

  4. When uˉ=T−1∑t=1Tut\bar{u}=T^{-1}\sum_{t=1}^{T}u_t depends on TT, how is the TT-independent error lower bound in Proposition 3 justified? For independent, zero-mean dynamic components with covariance Σu\Sigma_u, E∥uˉ∥22=tr⁡(Σu)/T\mathbb{E}\|\bar{u}\|_2^2=\operatorname{tr}(\Sigma_u)/T tends to zero. Is there an additional assumption missing, or should the comparison with the time-averaged theory be qualified?

Reviewer_HeLZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes DTS, a method for separating what stays constant in a sequence, such as a speaker’s identity, from what changes, such as the words being spoken. It uses two Transformer streams: one without positional information estimates the static content, while the other represents the full sequence. Subtracting the static estimate produces the dynamic representation. The model is trained using reconstruction and covariance regularization, with an optional reconstruction pass that uses only the dynamic code.

The results across video, speech, and time-series benchmarks are encouraging. My main concern is whether the method achieves the claimed separation of information. Several results establish useful representations, but the explanation of why those representations are disentangled is less convincing.

The paper presents a clear architectural idea and encouraging results across several modalities. However, I am not yet convinced that the objective and evaluation establish the claimed separation of static and dynamic information. I lean toward rejection, but would reconsider if the authors resolve the reconstruction-objective concern, correct the theoretical overclaim, and provide direct evidence of information leakage and content preservation.

Critical strengths

The architectural idea is clear and broadly applicable. Using separate streams to estimate static content and temporal information is easy to understand. Applying the same core design across three modalities makes the evaluation more informative than a demonstration on a single dataset.

Several reported improvements are substantial. For example, Air Quality accuracy improves from 69.2% for DiffSDA to 81.25%, while ETTh1 error decreases from 9.89 to 8.267. The speaker-verification results are also strong, particularly the static representation on TIMIT.

The paper includes useful ablations and factor-swapping experiments. Tables 8–9 examine the contributions of the losses and subtraction mechanism. The paired swaps in Table 2 also strengthen the video evaluation by checking whether appearance and action follow their intended codes.

Critical weaknesses

The dynamic-only reconstruction objective appears to work against the stated goal. Section 3.3 asks the dynamic code to reconstruct the entire input without access to the static code. To do this accurately, it would need to retain information about static attributes as well. The authors acknowledge this tension, but the explanation does not establish why subtraction and covariance regularization would resolve it. This needs direct evidence showing what information remains in each representation.

The architecture and regularizer do not necessarily enforce the intended separation. Removing positional information prevents sensitivity to ordering, but a representation can still capture an action or other dynamic content through the collection of observed frames. Similarly, low covariance does not establish independence. The window-agreement loss is also less restrictive than it initially appears: both halves are pooled after processing the full sequence, so they already share information.

The evaluation sometimes supports a stronger conclusion than the measurements justify. Poor speaker recognition from the dynamic code does not establish that it preserves phonetic content. Likewise, better temperature regression or month classification shows that a representation is useful, but does not directly demonstrate separation of static and dynamic factors. The swapping experiments help, although they reveal what the decoder uses more directly than everything the latent codes contain.

The theoretical comparison with simpler alternatives is overstated. Proposition 3 claims that temporal-mean subtraction has an error floor that does not disappear with longer sequences. This is not generally true: for independent, zero-mean dynamics, the average itself becomes more accurate as the sequence grows. The paper needs to narrow this claim and distinguish assumptions about the desired representation from properties actually enforced by training.

The practical simplicity and efficiency claims need stronger support. Having fewer loss weights is useful, but it does not establish lower computational cost or easier tuning overall. DTS uses multiple Transformer streams and potentially an additional decoder pass. Runtime, memory, parameter counts, and tuning budgets would help assess the practical advantage.

Nits:

  • The conclusion’s claim of state-of-the-art results on all six benchmarks is too broad. On PhysioNet, the default model matches DiffSDA’s reported AUPRC but trails its AUROC and ICU-type accuracy.
  • Please explain why action accuracy is 100% in Table 1 but approximately 97.5% in Table 2. Different protocols could explain this, but the distinction should be explicit.
  • Table 8’s dashes need clearer interpretation. Was dynamic-only reconstruction already disabled for those datasets, or were those ablations not run?
  • State the number of runs and meaning of the reported uncertainty, and clarify how the “best” depths used in the ablations were selected.
  • Small wording fixes include “the two passes gives” and “achieves the competitive results.”
Questions for authors
  1. On which datasets is dynamic-only reconstruction enabled, and how does increasing its weight affect static-information leakage into the dynamic code?
  2. What happens when the two temporal windows are encoded independently before applying the agreement loss?
  3. Can you directly evaluate preservation of speech content, alongside stronger speaker-information probes on the complete dynamic representation?
  4. What additional assumptions are needed for Proposition 3’s claimed error floor, and are those assumptions supported by the datasets?
  5. What are the training time, inference time, and memory costs relative to the strongest baselines under comparable settings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.