- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes CARAT, a test-time adaptation method for multimodal wearable time series under sensor corruption. During source training, an asymmetric modality-dropout curriculum adjusts each modality's dropout probability using windowed input-projection gradient norms; the terminal dropout rates are then frozen as a backbone-specific reliance profile . At test time, a one-class detector flags suspect modalities using reconstruction and spectral-distance features relative to clean source statistics. If the unflagged modalities retain at least a fraction of total reliance mass, suspect streams are replaced with the backbone's trained missingness symbol. Otherwise, their latent representations are attenuated before fusion. Adaptation updates only normalization and attenuation parameters using a weighted entropy objective. The evaluation covers four wearable datasets, three missingness-resilient backbones, five synthetic corruption types, and eight TTA baselines. All TTA methods use the same source checkpoint in the main comparison. CARAT achieves the best overall macro-F1 and mean rank across the 12 dataset backbone settings, with the largest improvements occurring on ShaSpec. The paper also includes ablations on detector quality, routing choices, reliance-prior perturbations, alternative source-training regimes, and computational cost.
Critical strengths
-
The paper addresses a useful deployment problem that identifying a corrupted modality is not sufficient to determine whether it should be discarded. CARAT separates current sensor abnormality from the model's dependence on that sensor and uses the two signals to choose between omission and attenuation. The unconditional-drop experiments support this motivation; for example, on PAMAP2--MAESTRO, dropping every flagged modality reduces F1 from about to , while CARAT maintains approximately .
-
The benchmark covers multiple sensing regimes, backbones, corruption families, and TTA baselines, while evaluating all methods on a common source checkpoint and corrupted test batches. The appendices also provide condition-level results and explicitly qualify the dependence structure of the statistical comparisons rather than treating all corruption conditions as independent replications.
-
The paper includes useful component analyses. It evaluates always-drop and always-attenuate policies, detector removal/noise, frozen attenuation, alternative source-training curricula, prior perturbations, and sensitivity to the routing threshold. These experiments help separate the contributions of source preparation, detection, routing, and online adaptation.
Critical weaknesses
-
The incremental benefit of the source-learned reliance prior is not clearly established. On the main DaliaHAR--ShaSpec ablation, always attenuation matches CARAT on pooled Div. F1 (), and a random prior also matches CARAT on Div. and reversing the prior reduces performance to . These results suggest that the coarse routing policy matters, but do not yet show that the particular learned reliance profile is a major source of the overall gains. The key prior ablations are also concentrated on the setting where CARAT performs best.
-
The headline improvement is strongly backbone-dependent. CARAT improves over EATA by F1 points on average for ShaSpec, but only on MAESTRO and on Flex-MoE. Several condition-level comparisons on MAESTRO and Flex-MoE favor other methods. This makes the broader practical default framing stronger than the current evidence supports.
-
The statistical analysis does not capture source-training uncertainty in the main TTA comparison. The condition-level intervals reuse the same checkpoint and overlapping examples, and the five-corruption-redraw experiment also keeps the source checkpoint fixed. This is particularly relevant for the small margins on MAESTRO and Flex-MoE.
-
The evaluation is limited to synthetic corruption families. The detector uses reconstruction and spectral features that are well suited to several of these corruptions, but it remains unclear how well the detector and frozen reliance prior transfer to natural sensor failures, long-term drift, or device/subject shifts. Although the paper itself notes that the prior may become stale and lists natural failures as future work.
Questions for authors
-
Can the authors report the always-drop, always-attenuate, random-prior, and reversed-prior ablations across all 12 dataset--backbone settings? If always attenuation or a random prior performs similarly to CARAT beyond DaliaHAR--ShaSpec, what incremental benefit comes from the source-learned reliance profile itself?
-
Can the authors report the main TTA results across multiple independently trained source checkpoints, with uncertainty at the dataset--backbone level? The current condition-level intervals and corruption-redraw experiment keep the source checkpoint fixed, which does not capture training-seed variability.
-
Can the authors compare against a simpler control that gives TENT or EATA access to the same suspect-modality detector, for example by masking or down-weighting flagged streams? This would help separate the value of the detector from the value of reliance-guided routing between omission and attenuation.
4 comments
Response to Reviewer wvLs (2/3)
Q2 / W3 — Can you report TTA results across independently trained source checkpoints, with uncertainty at the dataset–backbone level?
As requested, we have added a new source-checkpoint experiment: on DaliaHAR, we evaluate three independently trained checkpoints per backbone (source seeds 2711, 2712, and 2713), each with three TTA seeds (2711, 2712, and 2713), across all three backbones and ten methods (Appendix Q, Table 40). Every method uses the same source checkpoints and matched corrupted inputs within each evaluation. We first average the 50 conditions within each TTA run, then the three TTA runs within each checkpoint, and report mean ± sample SD across the three source-checkpoint means (n=3).
The closest competing methods are shown below; Table 40 gives all ten methods.
| Method | MAESTRO | ShaSpec | Flex-MoE | Mean F1 | Mean rank ↓ |
|---|---|---|---|---|---|
| TENT | 0.76 ± 0.0281 | 0.64 ± 0.0074 | 0.68 ± 0.0023 | 0.69 ± 0.0061 | 2.33 |
| EATA | 0.76 ± 0.0287 | 0.64 ± 0.0065 | 0.68 ± 0.0025 | 0.69 ± 0.0066 | 2.00 |
| CARAT | 0.76 ± 0.0335 | 0.66 ± 0.0063 | 0.68 ± 0.0157 | 0.70 ± 0.0038 | 1.67 |
Approximately, for each dataset it takes ~11 H100 or ~15 GB200 hours per checkpoint and one TTA seed, leading to ~300 or ~405 GPU-hours, respectively, for the full 3 × 3 study across three backbones, excluding training and setup. Given this cost, we currently report DaliaHAR (Table 40), with additional source checkpoint variations on request during rebuttal and full coverage planned for the final revision. Separately, Tables 13–24 now report three TTA evaluations at fixed checkpoints across all 12 settings, directly quantifying adaptation variability and complementing source-checkpoint uncertainty, consistent with general benchmarking suggestions (Bouthillier et al., MLSys 2021).
Q3 — Can TENT/EATA use the same suspect-modality detector with simple masking or downweighting to separate detector benefit from reliance-guided routing?
Yes as suggested, we have added a new detector-matched controls (Appendix R, Tables 41–43) equip TENT and EATA with CARAT’s detector on DaliaHAR across all three backbones and all 50 corruption conditions. Within each backbone, the controller variants share the source checkpoint, corrupted inputs, and detector flags. They either drop every flagged stream or attenuate it by α=0.5, retaining each baseline’s adaptation objective and update scope as given below :
| Method | Controller | MAESTRO | ShaSpec | Flex-MoE |
|---|---|---|---|---|
| TENT | original | 72.00 | 60.00 | 67.00 |
| Always drop flagged | 57.181 | 59.069 | 51.460 | |
| Always attenuate, α=0.5 | 72.860 | 63.043 | 68.246 | |
| EATA | original | 72.00 | 60.00 | 67.00 |
| Always drop flagged | 57.247 | 58.786 | 51.550 | |
| Always attenuate, α=0.5 | 72.868 | 62.912 | 68.246 | |
| CARAT | original | 76.00 | 66.00 | 68.00 |
Always dropping performs substantially worse than fixed attenuation for both baselines across all three backbones. CARAT’s reported reference is higher on MAESTRO and ShaSpec, while fixed attenuation is slightly higher on Flex-MoE. Thus, simple detector-assisted attenuation is competitive, and intervention choice matters. These comparisons assess the combined detector, intervention, and adaptation design; Tables 2 and 4; and the new policy/prior experiment across all 12 settings (Appendix S.1, Table 44) discussed in Q1, more directly examine the overall reliance guided routing within CARAT.
W2 — How should backbone-dependent gains and condition-level losses qualify the headline improvement and “practical default” claim?
The evidence supports CARAT as a practical TTA option with backbone-dependent benefits. Updated Table 1 (based on 3 re-runs of each TTA method acros 50 scenarios) reports a mean gain of 2.43 F1 points over EATA and mean rank 2.33. While the gains remain uneven: 5.60 points on ShaSpec, 1.21 on MAESTRO, and 0.48 on Flex-MoE, as reported in Section 4.1 (p. 7) using means before rounding, CARAT ranks first/ties in 10/12 settings, including 6/8 non-ShaSpec settings. Further our new headroom and corruption-family analyses (Appendix P, Table 39; Figure 14). The frozen-model diagnostic across all 12 settings finds greater clean-to-corrupted loss for ShaSpec on DSADS (0.33, versus 0.17/0.21 for MAESTRO/Flex-MoE) and DaliaHAR (0.14, versus 0.05/0.13). These separate diagnostic cohorts suggest more opportunity for recovery. Within DSADS–ShaSpec, the two families with the largest losses also yield the largest CARAT gains.
The updated manuscript uses “practical TTA option” in the abstract (p. 1) and Section 4.1 (p. 8), where the claim is conditional on adaptation FLOPs and updated-parameter count being the primary constraints. Section 5 (p. 10) describes CARAT as “a practical option in the evaluated setting.”
Response to Reviewer wvLs (3/3)
W4 — How well do the detector and frozen reliance prior transfer to natural failures, long-term drift, and device/subject shifts, including when the prior becomes stale?
Our 720-condition benchmark uses real wearable recordings and five synthetic profiles motivated by observation loss, waveform artifacts, saturation/compression, frequency-response degradation, and decreasing within-window SNR. In a new unmodified-input and false-positive sensitivity experiment (Appendix D.1, Table 8) on DaliaHAR–ShaSpec windows, using one fixed checkpoint and three paired evaluation repeats, native CARAT remains 0.88 F1 points above Static. Native detector false positives incur a 0.22% relative F1 decrease compared with CARAT with all detector flags suppressed, which still adapts; independently injecting false positives at a 40% probability per modality–window pair yields a 1.19% relative decrease.
Our test sets already use non-overlapping subject splits (Appendix B) and we explicitly acknowledge the stale-prior limitation in Section 5, which states that “The frozen, parameterization-dependent prior may become stale” and identifies slow-timescale prior refresh and “validation under natural failures across broader tasks and architectures” as future work.
I thank the authors for their detailed response and additional experiments as well as the effort in addressing several of my concerns. I particularly appreciate the new ablations and source checkpoint analyses. While I do still have some reservations about the incremental benefit of the learned reliance prior over simpler alternatives, I still appreciate the authors’ efforts and therefore I will maintain my original positive recommendation.
Response to Reviewer wvLs (1/3)
Thank you for your time and constructive feedback. We appreciate your recognition that our work identifies a useful gap in multimodal TTA and offers an intuitive method, supported by extensive evaluations and comprehensive ablations. Below, we address your concerns and questions point by point.
Based on your suggestion we have added a new policy/prior experiment across all 12 settings (Appendix S.1, Table 44). Always drop/attenuate force the intervention on flagged streams while retaining CARAT’s other components as given below :
CARAT is higher/tied/lower than the random prior in 8/3/1 settings and the reversed prior in 6/4/2. On DaliaHAR–ShaSpec, CARAT scores 0.66 and the reversed prior scores 0.61. Always attenuating is higher than CARAT in 2/12 settings and tied in 6/12. We acknowledge that the learned profile has setting-dependent benefits, and always attenuating could be a strong alternative. However, in the existing ablations (Tables 2 and 4) and a new sensor-pair omission experiment (Appendix S.2, Table 45). we observe that removing corrupted EDA + temperature improves F1 from 71.21 to 73.62, whereas removing corrupted wrist + chest acceleration reduces it from 48.14 to 22.95; CARAT achieves 55.86 on the latter. The acceleration pair also receives more prior mass (51.9% vs. 22.2%). This supports the motivation for reliance-guided interventions.