- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ReWeave, an augmentation framework for contrastive learning on multimodal wearable data. ReWeave keeps a learned subset of each modality's latent dimensions and synthesizes the rest from the other modalities with temporal cross-attention. A sparsity loss penalizes kept fractions above , and a gap loss encourages the view to match the original better than a control built with another sample's attention keys. Across four downstream tasks, two fusion architectures, and two self-supervised objectives, ReWeave outperforms eight baselines on most metrics.
Critical strengths
- The paper addresses a relevant gap. Most time-series augmentations ignore cross-modal temporal relations, and these relations carry physiological meaning in wearable data.
- The matched-versus-shuffled gap loss checks whether the view uses the sample's own correspondence between modalities. Its shuffled view takes helper keys from another sample but keeps the original helper values, so it changes only where attention looks.
- The method is modular. The authors apply it to early- and late-fusion encoders with both InfoNCE and Barlow Twins objectives, and also to the FOCAL framework.
- The evaluation protocol avoids leakage. The pretraining and downstream datasets share no subjects, and the downstream evaluation uses subject-wise cross-validation.
Critical weaknesses
- The losses only loosely follow the theory. is meant to enforce , but it measures correlation with the original input latent rather than mutual information with . It therefore raises the similarity to that aims to limit.
- The kept fraction may not reflect retained information. The sparsity bound assumes weakly redundant latent dimensions (). All modalities share the same latent size despite very different sampling rates, so low-rate modalities likely have highly redundant dimensions.
- The method has many hyperparameters. The paper does not state how or the AE loss weights were chosen, or whether the sensitivity sweeps use a validation split separate from the evaluation folds. The best lies at opposite ends of the tested range for WESAD () and DaLiA (), which makes tuning on new datasets a concern.
- Several margins are small relative to the reported standard deviations. The ablations in Figure 4 also lack error bars.
- The paper does not measure how much the views differ from the originals. The temporal error in Section 3.3 gives zero error to an identity augmentation.
Questions for authors
- The paper explains , , and . How did the authors choose and the AE loss weights, and did they use a validation split separate from the evaluation folds? How did they tune the baselines?
- The Reproducibility Statement says Appendix F covers the model architecture, but Table 6 describes only the AE and the augmentation module. What is the architecture of the multimodal encoder in each setting, and how are negatives constructed for InfoNCE?
- rewards views whose latents correlate with the original latent. Beyond the Barlow Twins analogy, why does this measure information about , as requires? How does it fit with , which asks views to share less with the original?
- Does the kept fraction reflect retained information in each modality, for example when reconstructing a modality from its kept dimensions alone? What do the different settled kept fractions across modalities in Figure 8 indicate?
- Can the authors report how much the augmented views differ from the originals, for both ReWeave and the baselines? How does this difference change as the kept fraction decreases during training?
5 comments
Author Rebuttal to Reviewer JhFQ [Part 2/3]
II. Kept Fractions and Measure of Retained Information [Weakness 2; Question 4]
Thank you for this insightful comment. We indeed make the assumption of weakly redundant latent dimensions and explicitly enforce this property during Stage I autoencoder pretraining via variance–covariance regularization.
While directly quantifying information is difficult, we designed an experiment to test whether low-rate modalities suffer from high dimensional redundancy: Because emotion recognition on WESAD is predominantly driven by the low-rate modality EDA(), we trained a WESAD classification head on top of the frozen EDA encoder from our pretrained VC-Reg AE. To reflects the total information retained in the latent space rather than generalization variance, we trained and evaluated the classifier on the full WESAD dataset. At evaluation time, we randomly masked , , and of the latent dimensions and measured the resulting performance drop, comparing our VC-Reg AE against Ablation A1 (the same EDA autoencoder pretrained without VC-Reg):
Table 1. Classification performance on WESAD using the frozen EDA encoder under different random latent-dimension masking ratios.
| Method | 25% Masked | 50% Masked | 75% Masked |
|---|---|---|---|
| Ours (AE w/ VC-Reg) | 0.499 | 0.473 | 0.424 |
| Ablation A1 (AE w/o VC-Reg) | 0.478 | 0.447 | 0.443 |
As shown in Table 1, for the low-rate EDA signal, Our latent space (w/ VC-Reg) exhibits a obvious performance drop as the masking ratio increases, indicating that task-relevant information is distributed relatively evenly across latent dimensions. In contrast, Ablation A1 (w/o VC-Reg) yields nearly identical performance when masking and of the dimensions, showing that an unregularized latent space contains substantial inter-dimensional redundancy where masked information remains recoverable from surviving dimensions. These results demonstrate that VC-Reg effectively mitigates dimensional redundancy even for low-sampling-rate modalities, making the kept fraction a reliable proxy for retained information.
The kept fraction represents the ratio of retained dimensions , with the remaining dimensions replaced by cross-modally generated . Because reconstruction error alone does not directly measure information preservation, we design a probing experiment to verify that retains less replaceable information than . We train an independent predictor to regress the target modality's frozen VC-Reg AE latent embedding () from the three helper modalities. During testing, dimensions routed to by our learned gate incur an average prediction error of 0.314, versus just 0.139 for . This demonstrates that isolates features that are difficult to infer from helper modalities, while captures easily recoverable information.
Although we set , it only penalizes modalities exceeding this kept fraction rather than strictly constraining their kept dimensions (-dim). Therefore, Figure 8 plots the actual kept fraction () for each modality across training epochs. These fractions describe the learned retention policy; they are not direct measurements of physiological information.
III. Hyperparameter Selection and Validation Protocol [Weakness 3; Question 1]
We thank the reviewers for raising these reproducibility concerns. First, the value of was selected based on the validation split of the pretraining dataset, independently of the evaluation folds. Neither our pretraining process nor that of the baselines involved the use of any evaluation data. As discussed in Appendix C, we acknowledge that the optimal value of may vary across datasets. Nevertheless, our results demonstrate that downstream performance is not particularly sensitive to the choice of . For new pretraining datasets, we therefore recommend selecting within the range of 0.35–0.5.
IV. Statistical Reliability and Missing Error Bars [Weakness 4]
We acknowledge that ReWeave's advantages are less pronounced in certain comparisons than in others. As discussed in Appendix C, this is likely because different downstream tasks have different preferences for augmentation strength. We consider addressing this task-dependent behavior an important direction for future work. We thank the reviewer for raising this insightful and important point. Regarding the error bars, we have added them to the revised manuscript.
Author Rebuttal to Reviewer JhFQ [Part 3/3]
V. Augmentation Diversity and Its Evolution During Training [Weakness 5; Question 5]
To more directly measure how much the augmented views differ from the original samples, we report the MAE between the original and augmented samples generated by ReWeave and the baseline methods on the validation set, as shown in Table 2. Additionally, Table 3 reports how the MAE evolves over training epochs. However, we acknowledge that, similar to lag error, identity augmentation also yields an MAE of zero. This is because the magnitude of the difference does not necessarily reflect the quality of the augmentation. Therefore, augmentation quality should be evaluated based on its performance on downstream tasks.
Table 2. Comparison of signal-space view diversity (MAE between original and augmented samples) across ReWeave and baseline augmentation methods. and denote the 50th and 75th percentiles, respectively.
| Signal Difference (MAE) | |||
|---|---|---|---|
| Method | Mean | ||
| Ours | 0.255 | 0.247 | 0.283 |
| Traditional | 0.334 | 0.314 | 0.411 |
| Chaos | 0.088 | 0.081 | 0.102 |
| AutoTCL | 0.393 | 0.372 | 0.489 |
| TF-C | 0.182 | 0.164 | 0.244 |
| xMAE | 0.732 | 0.685 | 0.850 |
| REBAR | 0.320 | 0.311 | 0.415 |
| TS2Vec | 0.524 | 0.505 | 0.618 |
| TNC | 0.329 | 0.318 | 0.423 |
Table 3. Evolution of signal-space view diversity (Mean MAE between original and augmented samples) across training epochs.
| Epoch No. | 9 | 19 | 29 | 39 | 49 | 59 | 69 | 79 | 89 | 99 | 109 | 119 | 129 | 139 | 149 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Mean MAE | 0.022 | 0.022 | 0.056 | 0.105 | 0.169 | 0.224 | 0.242 | 0.249 | 0.251 | 0.252 | 0.253 | 0.254 | 0.254 | 0.255 | 0.255 |
VI. Multimodal Encoder Architecture and InfoNCE Negatives [Question 2]
-
We thank the reviewer for the question and will add the full encoder specifications to Appendix F (Table 6). For a fair comparison, all three encoders are matched in model capacity (~3.7M–3.8M parameters) and produce a 256-D representation for downstream linear probing.
Table 4. Architectural specifications of the multimodal encoders across different fusion and contrastive settings.
Component Early Fusion Encoder Late Fusion Encoder FOCAL Encoder Input Stem Modality linear proj. (-D) + Sinusoidal PE Linear proj. (-D) + Sinusoidal PE Complex FFT + (2D ConvStem + 3 ResBlocks, , ) Sequence Backbone 1 Shared 4-Layer Transformer () 4 Indep. 4-Layer Transformers () 4 Indep. 3-Layer Bi-GRUs (, bidirectional -D) Fusion & Pooling Time-interleaved 40 tokens + [CLS]token poolingPatch-wise MLP () + Mean pooling across 10 patches Per-branch Mean pooling + LayerNorm + Concat (-D) Projector Head 3-Layer MLP () 3-Layer MLP () 2-Layer MLP (; -D shared + -D private) Output Dim 256-D 256-D 256-D () -
In InfoNCE, negative samples are the other samples in the batch.
If additional analyses/clarifications are helpful, we are happy to provide them during the discussion. Thank you for your time!
Author Rebuttal to Reviewer JhFQ [Part 1/3]
I. Alignment Between the Gap Loss and the Information-Theoretic Objective [Weakness 1; Question 3]
We sincerely thank the reviewer for pointing out the leap from the theoretical formulation to the practical implementation of Lgap, as well as the potential confusion regarding our objective formulation.
Why diagonal cross-correlation serves as a proxy for mutual information:
Complementary Roles of I(V+;X)≤ϵ and Lgap in InfoMin Trade-off:
In InfoMin, the constraint I(V+;X)≤ϵ serves to reduce the mutual information regarding task-irrelevant noise between views while preserving task semantics. We achieve this reduction via cross-modal replacement: replacing the n dimensions of the target modality with n^ synthesized from companion modalities substitutes original target-modality noise with cross-modal variations. While this replacement process may impair cross-modal temporal relationships, we leveraged Lgap to protect them. Therefore, the two terms work collaboratively toward InfoMin's goal of introducing meaningful variation while preserving task-relevant semantics.