What to Keep, What to Replace: Cross-Modal Dependency-Aware Augmentation for Wearable Contrastive Learning

Contrastive learning from multimodal wearable data requires augmented views that introduce sufficient variation while preserving both intra-modal semantics and cross-modal relationships. Cross-modal generation offers a way to exploit these relationships, but faithful reconstruction may provide little variation, while generation from other sensors alone can discard information specific to the target-modality signal. We introduce ReWeave, a plug-and-play augmentation framework that learns which target-modal-specific information to retain and which to replace using synchronized companion modalities. A learned gate preserves selected components in a frozen, variance–covariance-regularized latent space, while temporal cross-attention synthesizes the residual from helper signals. A sparsity penalty limits direct target retention, and the self-supervised objective encourages agreement between original and augmented views. To encourage preserving the sample-wise cross-modal relationships, we compare matched attention with a control that substitutes helper keys from another instance while retaining the original helper values. Decoding the edited embeddings produces signal-space augmentations compatible with different multimodal encoder architectures and self-supervised objectives. We pretrain on 3,565 hours of wearable recordings and evaluate frozen representations on four disjoint downstream datasets spanning emotion recognition, sleep staging, activity recognition, and oxygen-consumption estimation. Across two fusion architectures and two self-supervised objectives, ReWeave generally improves performance over alternative positive-view construction baselines, with gains of up to 9.7 macro-F1 points and an 11.4% reduction in oxygen-consumption MAE over the strongest baselines in the corresponding settings.