acceptodds
Under review as a conference paper at ICLR 2027

What to Keep, What to Replace: Cross-Modal Dependency-Aware Augmentation for Wearable Contrastive Learning

Abstract

Contrastive learning from multimodal wearable data requires augmented views that introduce sufficient variation while preserving both intra-modal semantics and cross-modal relationships. Cross-modal generation offers a way to exploit these relationships, but faithful reconstruction may provide little variation, while generation from other sensors alone can discard information specific to the target-modality signal. We introduce ReWeave, a plug-and-play augmentation framework that learns which target-modal-specific information to retain and which to replace using synchronized companion modalities. A learned gate preserves selected components in a frozen, variance–covariance-regularized latent space, while temporal cross-attention synthesizes the residual from helper signals. A sparsity penalty limits direct target retention, and the self-supervised objective encourages agreement between original and augmented views. To encourage preserving the sample-wise cross-modal relationships, we compare matched attention with a control that substitutes helper keys from another instance while retaining the original helper values. Decoding the edited embeddings produces signal-space augmentations compatible with different multimodal encoder architectures and self-supervised objectives. We pretrain on 3,565 hours of wearable recordings and evaluate frozen representations on four disjoint downstream datasets spanning emotion recognition, sleep staging, activity recognition, and oxygen-consumption estimation. Across two fusion architectures and two self-supervised objectives, ReWeave generally improves performance over alternative positive-view construction baselines, with gains of up to 9.7 macro-F1 points and an 11.4% reduction in oxygen-consumption MAE over the strongest baselines in the corresponding settings.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multimodal Wearable Sensing, Contrastive Learning, Learnable Data Augmentation, Self-Supervised Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_JhFQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ReWeave, an augmentation framework for contrastive learning on multimodal wearable data. ReWeave keeps a learned subset of each modality's latent dimensions and synthesizes the rest from the other modalities with temporal cross-attention. A sparsity loss penalizes kept fractions above ρ\rho, and a gap loss encourages the view to match the original better than a control built with another sample's attention keys. Across four downstream tasks, two fusion architectures, and two self-supervised objectives, ReWeave outperforms eight baselines on most metrics.

Critical strengths
  1. The paper addresses a relevant gap. Most time-series augmentations ignore cross-modal temporal relations, and these relations carry physiological meaning in wearable data.
  2. The matched-versus-shuffled gap loss checks whether the view uses the sample's own correspondence between modalities. Its shuffled view takes helper keys from another sample but keeps the original helper values, so it changes only where attention looks.
  3. The method is modular. The authors apply it to early- and late-fusion encoders with both InfoNCE and Barlow Twins objectives, and also to the FOCAL framework.
  4. The evaluation protocol avoids leakage. The pretraining and downstream datasets share no subjects, and the downstream evaluation uses subject-wise cross-validation.
Critical weaknesses
  1. The losses only loosely follow the theory. Lgap\mathcal{L}_{\text{gap}} is meant to enforce I(Y;n^∣p)≥δI(Y;\hat{n}\mid p)\ge\delta, but it measures correlation with the original input latent rather than mutual information with YY. It therefore raises the similarity to XX that I(V+;X)≤ϵI(V^{+};X)\le\epsilon aims to limit.
  2. The kept fraction may not reflect retained information. The sparsity bound assumes weakly redundant latent dimensions (I(p;X)≈c⋅dpI(p;X)\approx c\cdot d_p). All modalities share the same latent size despite very different sampling rates, so low-rate modalities likely have highly redundant dimensions.
  3. The method has many hyperparameters. The paper does not state how ρ\rho or the AE loss weights were chosen, or whether the sensitivity sweeps use a validation split separate from the evaluation folds. The best ρ\rho lies at opposite ends of the tested range for WESAD (0.20.2) and DaLiA (0.80.8), which makes tuning on new datasets a concern.
  4. Several margins are small relative to the reported standard deviations. The ablations in Figure 4 also lack error bars.
  5. The paper does not measure how much the views differ from the originals. The temporal error in Section 3.3 gives zero error to an identity augmentation.
Questions for authors
  1. The paper explains α\alpha, β\beta, and δ′\delta'. How did the authors choose ρ\rho and the AE loss weights, and did they use a validation split separate from the evaluation folds? How did they tune the baselines?
  2. The Reproducibility Statement says Appendix F covers the model architecture, but Table 6 describes only the AE and the augmentation module. What is the architecture of the multimodal encoder in each setting, and how are negatives constructed for InfoNCE?
  3. Lgap\mathcal{L}_{\text{gap}} rewards views whose latents correlate with the original latent. Beyond the Barlow Twins analogy, why does this measure information about YY, as I(Y;n^∣p)≥δI(Y;\hat{n}\mid p)\ge\delta requires? How does it fit with I(V+;X)≤ϵI(V^{+};X)\le\epsilon, which asks views to share less with the original?
  4. Does the kept fraction reflect retained information in each modality, for example when reconstructing a modality from its kept dimensions alone? What do the different settled kept fractions across modalities in Figure 8 indicate?
  5. Can the authors report how much the augmented views differ from the originals, for both ReWeave and the baselines? How does this difference change as the kept fraction decreases during training?
5 comments
AuthorsOpenReview ↗

Author Rebuttal to Reviewer JhFQ [Part 1/3]

I. Alignment Between the Gap Loss and the Information-Theoretic Objective [Weakness 1; Question 3]

We sincerely thank the reviewer for pointing out the leap from the theoretical formulation to the practical implementation of Lgap\mathcal{L}_{\text{gap}}, as well as the potential confusion regarding our objective formulation.

  1. Why diagonal cross-correlation serves as a proxy for mutual information:

    • MI proxy in a VC-regularized space: In Barlow Twins (Zbontar et al.,2021), mutual information maximization is operationalized by combining an invariance term (diagonal correlation) with a redundancy-reduction term (off-diagonal decorrelation). Because our Stage-I autoencoder is already pretrained and frozen with variance–covariance regularization which serves as the redundancy-reduction term. Thus, diagonal cross-correlation in Eq. 9 directly serves as a lightweight proxy for mutual information information in this fixed space.
    • Pretrained AE latent. Since ground-truth task labels YY are unavailable during self-supervised pretraining, using a pretrained autoencoder's embedding as a proxy for underlying semantics is a standard practice in self-supervised learning and time-series augmentation (e.g., MAE He et al., 2022; Finding Order in Chaos Demirel & Holz, 2023).
  2. Complementary Roles of I(V+;X)≤ϵI(V^+; X) \le \epsilon and Lgap\mathcal{L}_{\text{gap}} in InfoMin Trade-off:

    In InfoMin, the constraint I(V+;X)≤ϵI(V^+; X) \le \epsilon serves to reduce the mutual information regarding task-irrelevant noise between views while preserving task semantics. We achieve this reduction via cross-modal replacement: replacing the nn dimensions of the target modality with n^\hat{n} synthesized from companion modalities substitutes original target-modality noise with cross-modal variations. While this replacement process may impair cross-modal temporal relationships, we leveraged LgapL_{gap} to protect them. Therefore, the two terms work collaboratively toward InfoMin's goal of introducing meaningful variation while preserving task-relevant semantics.

AuthorsOpenReview ↗

Author Rebuttal to Reviewer JhFQ [Part 2/3]

II. Kept Fractions and Measure of Retained Information [Weakness 2; Question 4]

Thank you for this insightful comment. We indeed make the assumption of weakly redundant latent dimensions and explicitly enforce this property during Stage I autoencoder pretraining via variance–covariance regularization.

While directly quantifying information is difficult, we designed an experiment to test whether low-rate modalities suffer from high dimensional redundancy: Because emotion recognition on WESAD is predominantly driven by the low-rate modality EDA(4 Hz4\text{ Hz}), we trained a WESAD classification head on top of the frozen EDA encoder from our pretrained VC-Reg AE. To reflects the total information retained in the latent space rather than generalization variance, we trained and evaluated the classifier on the full WESAD dataset. At evaluation time, we randomly masked 2525\\%, 5050\\%, and 7575\\% of the latent dimensions and measured the resulting performance drop, comparing our VC-Reg AE against Ablation A1 (the same EDA autoencoder pretrained without VC-Reg):

Table 1. Classification performance on WESAD using the frozen EDA encoder under different random latent-dimension masking ratios.

Method25% Masked50% Masked75% Masked
Ours (AE w/ VC-Reg)0.4990.4730.424
Ablation A1 (AE w/o VC-Reg)0.4780.4470.443

As shown in Table 1, for the low-rate EDA signal, Our latent space (w/ VC-Reg) exhibits a obvious performance drop as the masking ratio increases, indicating that task-relevant information is distributed relatively evenly across latent dimensions. In contrast, Ablation A1 (w/o VC-Reg) yields nearly identical performance when masking 5050\\% and 7575\\% of the dimensions, showing that an unregularized latent space contains substantial inter-dimensional redundancy where masked information remains recoverable from surviving dimensions. These results demonstrate that VC-Reg effectively mitigates dimensional redundancy even for low-sampling-rate modalities, making the kept fraction a reliable proxy for retained information.

The kept fraction represents the ratio of retained dimensions pp, with the remaining dimensions replaced by cross-modally generated n^\hat{n}. Because reconstruction error alone does not directly measure information preservation, we design a probing experiment to verify that pp retains less replaceable information than nn. We train an independent predictor to regress the target modality's frozen VC-Reg AE latent embedding (p+np + n) from the three helper modalities. During testing, dimensions routed to pp by our learned gate incur an average prediction error of 0.314, versus just 0.139 for nn. This demonstrates that pp isolates features that are difficult to infer from helper modalities, while nn captures easily recoverable information.

Although we set ρ=0.35\rho = 0.35, it only penalizes modalities exceeding this kept fraction rather than strictly constraining their kept dimensions (pp-dim). Therefore, Figure 8 plots the actual kept fraction (p-dim/256p\text{-dim} / 256) for each modality across training epochs. These fractions describe the learned retention policy; they are not direct measurements of physiological information.


III. Hyperparameter Selection and Validation Protocol [Weakness 3; Question 1]

We thank the reviewers for raising these reproducibility concerns. First, the value of ρ\rho was selected based on the validation split of the pretraining dataset, independently of the evaluation folds. Neither our pretraining process nor that of the baselines involved the use of any evaluation data. As discussed in Appendix C, we acknowledge that the optimal value of ρ\rho may vary across datasets. Nevertheless, our results demonstrate that downstream performance is not particularly sensitive to the choice of ρ\rho. For new pretraining datasets, we therefore recommend selecting ρ\rho within the range of 0.35–0.5.


IV. Statistical Reliability and Missing Error Bars [Weakness 4]

We acknowledge that ReWeave's advantages are less pronounced in certain comparisons than in others. As discussed in Appendix C, this is likely because different downstream tasks have different preferences for augmentation strength. We consider addressing this task-dependent behavior an important direction for future work. We thank the reviewer for raising this insightful and important point. Regarding the error bars, we have added them to the revised manuscript.

AuthorsOpenReview ↗

Author Rebuttal to Reviewer JhFQ [Part 3/3]

V. Augmentation Diversity and Its Evolution During Training [Weakness 5; Question 5]

To more directly measure how much the augmented views differ from the original samples, we report the MAE between the original and augmented samples generated by ReWeave and the baseline methods on the validation set, as shown in Table 2. Additionally, Table 3 reports how the MAE evolves over training epochs. However, we acknowledge that, similar to lag error, identity augmentation also yields an MAE of zero. This is because the magnitude of the difference does not necessarily reflect the quality of the augmentation. Therefore, augmentation quality should be evaluated based on its performance on downstream tasks.

Table 2. Comparison of signal-space view diversity (MAE between original and augmented samples) across ReWeave and baseline augmentation methods. F.50F_{.50} and F.75F_{.75} denote the 50th and 75th percentiles, respectively.

Signal Difference (MAE)
MethodMeanF.50\boldsymbol{F_{.50}}F.75\boldsymbol{F_{.75}}
Ours0.2550.2470.283
Traditional0.3340.3140.411
Chaos0.0880.0810.102
AutoTCL0.3930.3720.489
TF-C0.1820.1640.244
xMAE0.7320.6850.850
REBAR0.3200.3110.415
TS2Vec0.5240.5050.618
TNC0.3290.3180.423

Table 3. Evolution of signal-space view diversity (Mean MAE between original and augmented samples) across training epochs.

Epoch No.9192939495969798999109119129139149
Mean MAE0.0220.0220.0560.1050.1690.2240.2420.2490.2510.2520.2530.2540.2540.2550.255

VI. Multimodal Encoder Architecture and InfoNCE Negatives [Question 2]

  1. We thank the reviewer for the question and will add the full encoder specifications to Appendix F (Table 6). For a fair comparison, all three encoders are matched in model capacity (~3.7M–3.8M parameters) and produce a 256-D representation for downstream linear probing.

    Table 4. Architectural specifications of the multimodal encoders across different fusion and contrastive settings.

    ComponentEarly Fusion EncoderLate Fusion EncoderFOCAL Encoder
    Input StemModality linear proj. (256256-D) + Sinusoidal PE4×4 \times Linear proj. (128128-D) + Sinusoidal PEComplex FFT + 4×4 \times (2D ConvStem + 3 ResBlocks, C=192C=192, K=3K=3)
    Sequence Backbone1 Shared 4-Layer Transformer (d=256,H=4,dff=1024d=256, H=4, d_{\text{ff}}=1024)4 Indep. 4-Layer Transformers (d=128,H=4,dff=512d=128, H=4, d_{\text{ff}}=512)4 Indep. 3-Layer Bi-GRUs (hidden=32\text{hidden}=32, bidirectional =64=64-D)
    Fusion & PoolingTime-interleaved 40 tokens + [CLS] token poolingPatch-wise MLP (512→256512 \to 256) + Mean pooling across 10 patchesPer-branch Mean pooling + LayerNorm + Concat (4×64=2564 \times 64 = 256-D)
    Projector Head3-Layer MLP (256→512→512→256256 \to 512 \to 512 \to 256)3-Layer MLP (256→512→512→256256 \to 512 \to 512 \to 256)4×4 \times 2-Layer MLP (64→256→6464 \to 256 \to 64; 3232-D shared + 3232-D private)
    Output Dim256-D256-D256-D (64-D×464\text{-D} \times 4)
  2. In InfoNCE, negative samples are the other samples in the batch.


If additional analyses/clarifications are helpful, we are happy to provide them during the discussion. Thank you for your time!

Reviewer_mGMdOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces ReWeave, a method that helps AI learn from multiple wearable sensors without needing large amounts of labeled data. Instead of randomly altering sensor readings during training, ReWeave learns which parts of each signal should remain unchanged and which can be realistically replaced using information from the other synchronized sensors, preserving meaningful relationships such as the delay between movement and a rise in heart rate. Tested on emotion, sleep, activity, and oxygen-consumption tasks, it generally outperformed existing methods across several AI setups.

Critical strengths
  1. ReWeave addresses a practical problem of standard wearable-data augmentation, namely changing each sensor independently can break meaningful relationships between sensors. The proposed learned gate offers a sensible balance by retaining information unique to the target sensor while replacing other information using synchronized companion sensors.

  2. The proposed method was evaluated on four different health-related tasks, two sensor-fusion architectures, and two self-supervised objectives, while using the same encoders and evaluation protocol for the main baselines. ReWeave performs well in most settings, and the ablation studies support the claim that selective retention and cross-modal temporal modeling both contribute to its performance.

Critical weaknesses
  1. The paper measures preservation mainly through changes in the single time lag that maximizes Pearson correlation between two signals. This is easy to understand, but wearable-sensor relationships can be nonlinear, change within a window, or involve several delays. Therefore, the analysis supports the claim but does not fully establish that ReWeave preserves complex physiological relationships.
  2. Cross-modal predictability may not always reflect meaningful information. Shared patterns can include noise or sensor artifacts, while useful information may be unique to one modality. The paper could better justify why its objectives choose predictable cross-sensor patterns instead of meaningful physiological and behavioral structure.
Questions for authors
  1. How well does the maximum Pearson-correlation lag capture the cross-modal relationships that ReWeave is intended to preserve? Especially in nonlinear, time-varying, or multiple temporal dependencies within the same window?

  2. How does ReWeave distinguish meaningful cross-modal dependencies from shared noise or sensor artifacts?

5 comments
AuthorsOpenReview ↗

Author Rebuttal to Reviewer mGMd [Part 1/2]

I. Adequacy of the Pearson-Correlation Lag Metric [Weakness 1; Question 1]

We thank the reviewer for this insightful comment, which helps us characterize temporal dependency preservation more comprehensively. As described in Section 3.3, our evaluation pipeline for Figure 3 shifts each modality pair in one-bin increments to obtain a full correlation sequence across candidate lags before selecting the single lag with the largest absolute correlation. We agree that when wearable relationships involve multiple delays or vary within a window, reducing this entire sequence to a single peak lag is insufficient. We therefore introduce two additional experiments:

  • Lag-Distribution Preservation (Pearson-JSD): We compare distributions derived from the original and augmented correlation profiles across all candidate lags. Lower JSD indicates better preservation of relative correlation weights, including multiple lag peaks.
  • Nonlinear Dependency Preservation (dCor): We measure changes in distance correlation, computed from double-centered pairwise distance matrices across time points, to assess preservation of nonlinear dependencies.

Table 1. Evaluation of complex temporal dependency preservation across candidate lags using Lag-Distribution Jensen–Shannon Divergence (Pearson-JSD\text{Pearson-JSD}) and Nonlinear Distance-Correlation MAE (dCor-MAE\text{dCor-MAE}). Best results are marked in bold. Second-best results are underlined‾\underline{\text{underlined}}.

↓Pearson-JSD (Multi-Delay & Time-Varying)↓dCor (Nonlinear Dependency)
Method↓P25↓P50↓P75↓P25↓P50↓P75
Ours (ReWeave)0.0080‾\underline{0.0080}0.02120.05370.02180.03800.0613
Traditional0.13290.14920.16540.11630.13930.1639
Chaos0.00790.0247‾\underline{0.0247}0.0643‾\underline{0.0643}0.0243‾\underline{0.0243}0.0410‾\underline{0.0410}0.0738‾\underline{0.0738}
AutoTCL0.13470.16040.18600.17290.19690.2221
TF-C0.05530.08180.10920.06940.09760.1283
xMAE0.14250.16150.18700.18530.20980.2425
REBAR0.10360.11970.13940.12000.14010.1599
TS2Vec0.13270.15260.17420.18560.20940.2309
TNC0.11060.13000.15010.14620.16810.1896

Table 1 reports the 25th, 50th, and 75th percentiles across samples. ReWeave has the lowest error in five of six reported statistics and is second in the remaining one (Pearson-JSD P25: 0.0080 vs. 0.0079 for Chaos). Note that the error of Chaos (Demirel & Holz, 2023) is also very low because its MixUp strategy takes into account the similarity between the two samples. We think these results extend the evidence to multiple-delay, time-varying, and nonlinear structures.

AuthorsOpenReview ↗

Author Rebuttal to Reviewer mGMd [Part 2/2]

II. Meaningful Cross-Modal Dependencies Versus Shared Noise [Weakness 2; Question 2]

Thank you for this insightful question. Because physiological and behavioral structures are complex and diverse, predefined rule-based metrics struggle to capture their full scope; we therefore adopt data-driven cross-modal synthesis to explore broader inter-modal dependencies. For useful information unique to a single modality, our learnable p–np\text{--}n gate provides a direct pathway for preservation. We acknowledge that ReWeave does not explicitly separate shared noises/artifacts from meaningful cross-modal structure, and we will state this limitation. In the meantime, we would like to note that ReWeave introduces an inductive bias toward stable and meaningful structures that encourage robust correspondence via three jointly enforced constraints.

  • Our cross-modal prediction is performed in the latent space of the Stage 1 AE. Through VC-Reg and Lrobust\mathcal{L}_{\text{robust}}, the AE encoder can filter out noise to some extent, making the latent space less susceptible to shared noise than the raw input space.
  • Decorrelation across latent dimensions limits the number of dimensions occupied by simple noise patterns. As the latent dimensionality nn increases, the model is increasingly encouraged to learn meaningful cross-modal relationships. Furthermore, Lgap\mathcal{L}_{\text{gap}} is applied independently to each dimension with an upper bound, thereby preventing simple noise patterns from dominating the learning process.
  • Joint LSSL\mathcal{L}_{\text{SSL}} optimization: Noise components that cannot be stably synthesized across views degrade original–augmented alignment; joint optimization with LSSL\mathcal{L}_{\text{SSL}} therefore suppresses reliance on these unstable components.

Empirical evidence further demonstrates that ReWeave effectively preserves inter-modal lags and achieves strong performance on downstream tasks, suggesting that shared noise does not dominate the learning of cross-modal relationships.


If additional analyses/clarifications are helpful, we are happy to provide them during the discussion. Thank you for your time!

I appreciate the clarification on the Pearson-lag metric which addressed the first concern. As for the cross-modal dependency, how about stable noise or artifacts shared across sensors? Do VC regularization and/or decorrelation always remove noises?

Reviewer_i6XVOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes an augmentation framework for contrastive learning that aims to generate more data from wearable sensing modalities. This work explores the tradeoff between keeping information from a target modality or introducing more variation through other modalities, which would add more variety but could lose the main characteristics of the target modality. To address this, the work proposes an architecture roughly consisting of modality-specific encoders that extract latent representations of each modality, a binary gate that selects specific dimensions of the modality to retain, and a cross-modal attention mechanism for synthesizing the remaining dimensions that are not kept. The framework is evaluated on a variety of downstream datasets, across many users, and evaluates a wide range of ablations and settings.

Critical strengths
  1. The overall approach and design principle of balancing how much of the real data you use vs. how much you synthesize or add noise to generate more data is reasonable and makes sense. This is essentially the core problem of all data augmentation schemes.

  2. The evaluation is fairly broad and the gains are fairly good. There were four different tasks evaluated, two fusion strategies, 56 users, over 3000 hours of data, and more, which are all good. There were also a whole laundry list of ablations performed, ranging from varying the gating mechanism, to changing the augmentation method, and more.

Critical weaknesses
  1. The novelty seems limited and incremental. Many of the approaches used in this work have been established in existing works. For example, encoder-agnostic augmentation (e.g., AutoTCL), latent-guided augmentation for time series (e.g., Finding Order in Chaos), multimodal representation learning (e.g., FOCAL), and cross-modal signal reconstruction (e.g., xMAE). It seems this work combines many established principles into a heavily engineered system that performs better since it has more bells and whistles.

  2. Evaluation does not seem to support the central thesis of this work, which is to resolve "how much to keep". The ablations show that the proposed mechanisms can improve performance, but that's not the same as supporting the central thesis or establishing that these mechanisms aid in learning temporal or physiological dependencies. Perhaps it would be useful to compare the samples generated by different approaches, and expand the description of the design decisions (e.g., why adding variance and covariance regularization when designing the objective and expanding the information theoretic design of the augmentation module). Moreover, while the results are strong in some areas, they are very heterogeneous, with some cases seeing limited performance improvements.

Questions for authors
  1. Please elaborate on the core novelty of the approach compared to existing work, and provide experiments to support (beyond just a combination of existing techniques).

  2. Please motivate the information theoretic approach and elaborate on which are guarantees provided by the formulation and which are just heuristics.

7 comments
AuthorsOpenReview ↗

Author Rebuttal to Reviewer i6XV [Part 1/3]

I. Core Novelty Beyond Existing Techniques [Weakness 1; Question 1]

We thank the reviewer for the opportunity to clarify our core contributions.

1. Key Problem Formulation: We first identify a critical limitation in existing time-series data augmentations: they do not explicitly preserve cross-modal temporal dependencies. While xMAE shows that cross-modal reconstruction can capture such dependencies, it faces two fundamental limitations when applied as an augmentation for wearable sensing: (1) pure reconstruction introduces insufficient view variation, and (2) its two modalities (ECG and PPG) observe the same underlying cardiac process, which does not generalize to multimodal wearables where ACC, PPG, EDA, and TEMP monitor distinct physiological processes (motion, cardiovascular activity, sympathetic arousal, and thermoregulation) and thus contain irreplaceable modality-specific information.

2. Core Insight: We propose a novel insight: by performing selective cross-modal editing in the latent space, where only irreplaceable target components are retained and the rest are synthesized from companion modalities, we directly turn the multimodal heterogeneity limitation (Aforementioned Point 2) into the solution for the view-variation limitation (Aforementioned Point 1). This makes it possible to generate diverse positive views while preserving cross-modal temporal dependencies in wearable contrastive learning, as formulated in Eq. (1).

3. Methodological Novelty: Our technical novelty lies in how we translate the theoretical formulation in Eq. (1) into tractable, differentiable objectives rather than stacking heuristics:

  • Novel Information Retention Control: We introduce a VC-Reg AE specifically to encourage modality information evenly and with low redundancy across latent dimensions. Only within this regularized space can our subsequent learnable dimension-wise gate serve as a valid proxy for bounding retained information (I(pm;Xm)≈c⋅dpI(p^m; X^m) \approx c \cdot d_p in Eq. (6)). Thus, the VC-Reg AE and the learnable gate form a tightly coupled design rather than module stacking.

  • Novel Cross-Modal Temporal Dependency Learning: Intuitively, given the target-preserved information pp, the more complementary information the synthesized component n^\hat{n} provides about the semantic representation YY (quantified by I(Y;n^∣p)I(Y; \hat{n} \mid p)), the better the model captures true sample-wise temporal alignment across modalities. Because directly maximizing I(Y;n^∣p)I(Y; \hat{n} \mid p) is intractable, a principled strategy in SSL is to raise its lower bound. To this end, we construct a novel counterfactual shuffled view using helper keys from another sample to serve as a dynamic lower bound that tightens as cross-modal relationships are learned. Furthermore, since conditional mutual information lacks effective empirical proxies, we reformulate the comparison of two conditional mutual information terms into the difference between two unconditional mutual information terms in Eqs. (7) and (8), which can then be practically optimized via standard contrastive or correlation proxies.

Novelty Compared to existing work:

  • xMAE. xMAE is a masked representation pretraining paradigm that performs full, unselective cross-modal reconstruction between homogeneous cardiac signals (ECG and PPG). It lacks any mechanism to partition irreplaceable modality-specific features (pp) from cross-modally replaceable features (nn), nor does it enforce sample-wise interaction gaps (Lgap\mathcal{L}_{\text{gap}}). When repurposed for augmentation across heterogeneous sensors (ACC, PPG, EDA, TEMP), pure reconstruction either yields trivial view variation or erases modality-private semantics

  • AutoTCL and Finding Order in Chaos. Both are strictly intra-modal perturbation methods: AutoTCL learns parametric point-wise masks/scaling within a single series, and Chaos performs phase–amplitude mixup within individual quasi-periodic signals. Neither models cross-modal interactions nor synthesizes one modality from another, leaving cross-sensor physiological latencies unaddressed.

  • FOCAL (multimodal representation learning). FOCAL is an encoder-side contrastive objective that factorizes final representations into orthogonal shared and private subspaces using standard hand-crafted augmentations; it is not a data augmentation framework. In fact, as shown in Appendix E (Table 5), ReWeave serves as an orthogonal, plug-and-play view generator that directly boosts FOCAL's performance.

AuthorsOpenReview ↗

Author Rebuttal to Reviewer i6XV [Part 2/3]

II. Evidence for Selective Retention and Design Choices [Weakness 2]

Thank you for this important question. We added two new analyses that directly examine what the gate retains and how retention balances variation with dependency preservation.

  1. Cross-Modal Predictability of pp vs. nn: We trained an independent predictor using the three helper modalities to predict the target modality's frozen VC-Reg AE latent embedding (p+n). During testing, dimensions selected as pp by our learned p-n gate show an average prediction error of 0.314, whereas nn dimensions have an error of only 0.139. This suggests that pp captures features that are more difficult to infer by helper modalities, while nn captures recoverable information.

  2. Sample Variation (MAE) vs. Temporal Dependency Error (Lag) Across Ablations (A0–A11):

    • Retention Trade-off: Increasing fixed retention (A3 →\to A4 →\to A5 →\to A11) reduces both signal MAE and lag error. A5/A11 preserve timing with less variation; A11 is the AE-only reconstruction reference. Our design produces more variation than 50% retention (MAE 0.255 vs. 0.218) with lower lag error (617.33 vs. 915.68 ms). Random gating and 25% retention produce much larger signal differences and lag errors. And our design outperformed all of them in the downstream tasks.

    • Why VC-Reg is Required (A1): Without VC-Reg, latent dimensions are highly correlated; even after heavy replacement, the remaining subset leaks nearly the entire original signal, causing A1 to collapse to near-identity reconstruction (MAE matching A11) and confirming the need for variance–covariance regularization.

    • Temporal Dependency Mechanisms (A6–A10): Under the same sparsity-budget setting, removing cross-attention (A6), removing Lgap\mathcal L_{\mathrm{gap}} (A7), or shuffling both keys and values (A10) yields comparable MAE (0.221–0.257 vs. 0.255) but larger lag error (897.69–1021.75 vs. 617.33 ms), verifying that these components specifically preserve cross-modal temporal relationships.

Table 1. Comparison of signal-space view diversity (MAE between original and augmented samples), cross-modal temporal dependency preservation error (Lag Error in ms), and downstream task performance across the full ReWeave model (A0) and ablation variants (A1–A11). F.50F_{.50} and F.75F_{.75} denote the 50th (median) and 75th percentiles, respectively. Downstream task performance is reported as mean values only.

\begin{array}{lccc} \hline \begin{array}{c} \vphantom{\underline{\vphantom{\textbf{Ag}}}} \\\\ \begin{array}{ll} \textbf{ID}\vphantom{F_{.75}} & \textbf{Ablation Setting} \\\\ \hline \textbf{A0}&\text{Ours}\\\\ \text{A1}&\text{AE w/o VC-Reg}\\\\ \text{A2}&\text{Random Gate}\\\\ \text{A3}&\text{Fixed Gate Ratio (25\\%)}\\\\ \text{A4}&\text{Fixed Gate Ratio (50\\%)}\\\\ \text{A5}&\text{Fixed Gate Ratio (75\\%)}\\\\ \text{A11}&\text{AE-only Augmentation (100\\%)}\\\\ \text{A6}&\text{w/o Cross-Attention}\\\\ \text{A7}&\text{w/o Gap Loss}\\ \smash{(\mathcal{L}_{\mathrm{gap}})}\\\\ \text{A8}&\text{w/o Helper Scorer}\\\\ \text{A9}&\text{w/o Translator}\\\\ \text{A10}&\text{KV-Shuffled Sample} \end{array} \end{array} & \begin{array}{c} \underline{\vphantom{\textbf{Ag}}\textbf{Signal Difference (MAE)}} \\\\ \begin{array}{rrr} \textbf{Mean}\vphantom{F_{.75}} & F_{.50} & F_{.75} \\\\ \hline 0.255&0.247&0.283\\\\ 0.021&0.014&0.023\\\\ 0.796&0.763&0.936\\\\ 0.520&0.485&0.584\\\\ 0.218&0.206&0.260\\\\ 0.088&0.082&0.103\\\\ 0.022&0.021&0.027\\\\ 0.257&0.221&0.303\\\\ 0.221&0.205&0.268\\\\ 0.203&0.186&0.241\\\\ 0.169&0.145&0.203\\\\ 0.241&0.215&0.285 \end{array} \end{array} & \begin{array}{c} \underline{\vphantom{\textbf{Ag}}\textbf{Temporal Lag Error (ms)}} \\\\ \begin{array}{rrr} \textbf{Mean}\vphantom{F_{.75}} & F_{.50} & F_{.75} \\\\ \hline 617.33&479.17&963.54\\\\ 422.58&208.33&708.33\\\\ 1141.76&968.75&1490.62\\\\ 1108.41&927.08&1448.96\\\\ 915.68&734.38&1308.33\\\\ 519.98&307.29&912.50\\\\ 370.63&276.04&661.46\\\\ 1021.75&833.33&1365.62\\\\ 973.45&766.67&1335.42\\\\ 878.58&697.92&1256.25\\\\ 786.09&625.00&1183.33\\\\ 897.69&718.75&1282.29 \end{array} \end{array} & \begin{array}{c} \underline{\vphantom{\textbf{Ag}}\textbf{Downstream Task Performance}} \\\\ \begin{array}{rrrr} \textbf{WESAD (F1)}\vphantom{F_{.75}} & \textbf{AAUWSS (F1)} & \textbf{DALIA (F1)} & \textbf{WEEE (MAE)} \\\\ \hline 0.652&0.301&0.720&3.270\\\\ 0.491&0.222&0.703&3.694\\\\ 0.565&0.292&0.684&3.469\\\\ 0.549&0.293&0.664&3.280\\\\ 0.614&0.299&0.683&3.322\\\\ 0.512&0.238&0.718&3.673\\\\ 0.539&0.260&0.698&4.016\\\\ 0.467&0.275&0.712&3.745\\\\ 0.554&0.277&0.703&3.629\\\\ 0.564&0.269&0.706&3.635\\\\ 0.541&0.277&0.710&3.702\\\\ 0.565&0.276&0.707&3.587 \end{array} \end{array} \\\\ \hline \end{array}$$
AuthorsOpenReview ↗

Author Rebuttal to Reviewer i6XV [Part 3/3]

III. Information-Theoretic Motivation and Guarantees [Question 2]

Thank you for this question. We distinguish exact information-theoretic relationships, modeling assumptions, and practical optimization surrogates below.

1. Motivation: make cross-modal synthesis contribute beyond direct retention.

The InfoMin-inspired objective balances semantic retention against excessive input retention. However, I(Y;V+)=I(Y;p)+I(Y;n^∣p)I(Y;V^+) = I(Y;p)+I(Y;\hat n\mid p) shows that semantic agreement can be achieved through the retained component pp alone. If synthesis simply reproduces a function of pp, then I(Y;n^∣p)=0I(Y;\hat n\mid p)=0. The third condition in Eq.1 therefore requires the synthesized component to provide additional information beyond what is already retained.

2. Exact information bound and the role of VC regularization.

Let HH denote all helper modalities. Since p=f(Xm)p=f(X^m) and n^=G(p,H)\hat n=G(p,H), the chain rule and conditional data-processing inequality give

I(V_m; X^m) &= I(p, \hat{n}; X^m) = I(p; X^m) + I(\hat{n}; X^m \mid p) \le I(p; X^m) + I(H; X^m \mid p) \le I(p; X^m) + I(H; X^m). \end{aligned}$$ This bound is exact. We specifically introduce VC regularization to support a dimensionality-based approximation of the target-retention term $I(p;X^m)$. The variance term discourages inactive dimensions, while the covariance term penalizes linear redundancy. Together, these terms aim to distribute information more evenly across latent dimensions and reduce duplicated information, making the number of retained dimensions a more meaningful proxy for retained information. Under the simplifying assumption that active dimensions carry roughly comparable amounts of weakly redundant information, we therefore use $$I(p;X^m)\approx c\cdot d_p,$$ where $c$ represents the approximate information contribution per active dimension in the frozen latent space. Thus, VC regularization is explicitly motivated by the dimensionality-based information surrogate, and gate sparsity provides a tractable way to control this surrogate. **3. Conditional information gain: exact relationships and practical surrogate.** *Target and shuffled reference.* The third condition in Eq. (1) requires synthesis to contribute information beyond direct retention: $A_i=I(Y_i;\hat n_{i\leftarrow i}\mid p_i)\ge\delta$. Since this conditional MI lacks a straightforward empirical surrogate in our unlabeled setting, we construct a shuffled reference $B_i=I(Y_i;\hat n_{i\leftarrow j}\mid p_i)$ in order to consruct a lower bound for A. Because both views share the same $p_i$, the chain rule cancels $I(Y_i;p_i)$ exactly: $$\begin{aligned} G_{\mathrm{MI}} &= I(Y_i;\hat n_{i\leftarrow i}\mid p_i)-I(Y_i;\hat n_{i\leftarrow j}\mid p_i) = \bigl[I(Y_i;V_i^+)-I(Y_i;p_i)\bigr]-\bigl[I(Y_i;V_i^{\mathrm{shuf}})-I(Y_i;p_i)\bigr] = I(Y_i;V_i^+)-I(Y_i;V_i^{\mathrm{shuf}}). \end{aligned}$$ Optimizing this gap is therefore a lower-bound optimization strategy for the third condition. *Practical correlation surrogate.* Using the original frozen AE latent as an unsupervised semantic reference, we operationalize this unconditional MI gap through diagonal cross-correlation. VC regularization has already encouraged active, less redundant latent coordinates, motivating our focus on corresponding-coordinate agreement. We therefore adapt the diagonal statistic underlying Barlow Twins' invariance term into the margin-based gap loss in Eq. (9): $$L_{\mathrm{gap}} = \frac{1}{|\mathcal M|} \sum_{m\in\mathcal M}\frac{1}{d_m} \sum_{k=1}^{d_m} \left[ \delta' - \left( [C^{m,+}]_{kk} - \operatorname{sg}([C^{m,\mathrm{shuf}}]_{kk}) \right) \right]_+ .$$ Eq. (9) is a correlation-based optimization surrogate; the information identities and lower-bound relation above are exact. --- **If additional analyses/clarifications are helpful, we are happy to provide them during the discussion. Thank you for your time!**

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to All Reviewers

We thank the reviewers for their thoughtful feedback. In our rebuttals, we provided additional experiments and clarifications to address the main concerns raised across the reviews.

  • We clarified ReWeave’s contribution relative to prior augmentation and reconstruction methods and explained how the information-theoretic formulation motivates selective retention and the matched-versus-shuffled gap loss. We also identified the modeling assumptions and practical proxies used to implement the objective.
  • We added a cross-modal prediction probe showing that the dimensions retained by the learned gate were harder to predict from helper sensors than the dimensions selected for replacement. We also tested latent redundancy in low-rate EDA signals and reported how augmentation diversity varied across baselines, ablations, and training epochs.
  • To address the concern that relationships between wearable sensors can be nonlinear, vary within a window, or involve multiple delays, we expanded our evaluation to compare full lag profiles across candidate delays and used distance correlation to assess nonlinear dependencies. ReWeave achieved the lowest error in five of six reported statistics. We also discussed the limitations of cross-modal predictability in the presence of shared sensor artifacts.
  • We clarified how the retention hyperparameter was selected on a separate pretraining validation split, provided the multimodal encoder specifications, and explained how InfoNCE negatives were sampled. We also added error bars to Figure 4.

We believe these analyses and clarifications address most of the concerns raised. Please see our individual responses for details. We have not received follow-up feedback from the reviewers, so we are unsure whether any concerns remain. We would be happy to address further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.