- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces STRUM, a synchronized dyadic multimodal physiological dataset with 66 participants recorded during cooperative tasks using eleven signal families, including EEG, EMG, ECG, EOG, EDA, respiration, eye tracking, motion capture, force plate, controller, and avatar signals. The paper evaluates two continuous regression tasks—phasic electrodermal arousal and mean heart rate—and compares classical, from-scratch, and pretrained models under leave-dyad-out evaluation. Its main conclusion is that modality choice affects decoding performance more than model choice, with additional analyses on multimodal fusion, personalization, modality necessity, and inter-partner physiological coupling.
Critical strengths
1.The scope and idea itself is unique interesting
2.The synchronized dyadic acquisition across many physiological and behavioral modalities is unusual and potentially useful.
3.The paper pays careful attention to synchronization, subject-level splitting, target circularity, and statistical testing.
Critical weaknesses
-
Many things feels over claimed and lack of proof (1)The central modality-versus-model claim is not based on a controlled modality × architecture comparison. Different modalities use different encoders, pretraining sources, input structures, and in some cases different window sets. (2) Modality usefulness is inherently model-dependent. Poor decoding performance may reflect an unsuitable or insufficient model rather than limited information in the modality itself. (3) The benchmark contains only two closely related autonomic targets. It is unclear whether the observed modality ranking would generalize to cognitive, behavioral, motor, or clinical prediction tasks. (4) The cohort is small and demographically narrow for the breadth of the generalization claims: 66 participants, ages 18–27, with strong imbalance toward male participants. (5) The paper’s own personalization results suggest strong subject-specific physiological structure, which raises further concerns about drawing broad modality-level conclusions from this limited population. (6) The multimodal evaluation does not fully exploit the dataset’s synchronization. The main fusion approach relies largely on separate encoders followed by feature concatenation. (7) Several mechanistic interpretations appear stronger than the evidence supports. Masking, lag analysis, and leave-one-modality-out experiments establish association or redundancy, not unique physiological attribution.
-
The practical framing is somewhat narrow. In many medical applications, available modalities are fixed by devices, protocols, cost, or invasiveness, making model improvement the more relevant problem.
-
The presentation is difficult to follow. Several tables and figures are dense, and some layouts visually encourage comparisons that the evaluation protocol does not fully support.
Questions for authors
see weaknesses
4 comments
Response to Reviewer gZJn: revised paper and new experiments
Thank you for your detailed feedback, which helped us improve the quality and soundness of the paper. Based on the reviews, we ran substantially more experiments, which changed our results section, while its main messages held. We would welcome your view on whether the revision now meets your concerns. Dark blue marks new or changed text, and green italics report fine-tuning runs that are still finishing, so those values may change slightly.
Thank you for raising the need for a controlled comparison of modality and model. We built a grid that runs four models, ridge, a CNN trained from scratch, SleepFM, and MOMENT, on every modality, scored on the same windows and participants, with all 205 EEG channels. Modality choice still matters more than model choice, by on heart rate and on electrodermal arousal. A permutation test shows that this gap does not come from how the grid is built.
As you commented, a modality's usefulness partly depends on the model. We now report the modality–model interaction, which carries 27% and 17% of the systematic variance, while modality keeps the larger effect overall. To test whether weaker models hide information, we fine-tuned every pretrained encoder on STRUM, fully or with LoRA. Fine-tuning gives slight gains, and motion capture remains the strongest modality with and without it. Ridge and the CNN, which are trained on STRUM alone, also rank it first.
To avoid over-claiming, we scoped the claims to the two targets and the models we tested, and the abstract now states that motion capture carries most of the modality gap. On electrodermal arousal, the gap still holds without motion capture once ECG is included (). We reworded the masking, lag, leave-one-out, and coupling analyses to describe association rather than attribution, and we state the limitations more clearly, including the size and composition of the cohort.
On the subject-specific structure suggested by our personalization results, each participant's own model does not beat a single model trained on all other participants on heart rate, and does worse on electrodermal arousal. EMG also beats EEG in 31 of 39 participants on heart rate and 30 of 39 on electrodermal arousal, so the group-level conclusions are not driven by a few people.
As you noted, adding non-autonomic targets would be very insightful, and we are working on it as a next step. Our current compute does not yet support running the full model grid and its controls for each new target, so we consider this out of scope for this paper and part of our future work. We believe it does not diminish STRUM's value as a ready-to-use multimodal dyadic resource for the many tasks it supports, cooperative or individual, autonomic or not.
We also implemented a time-aligned fusion of modalities, which outperforms every single modality given to the same model, by on heart rate and on electrodermal arousal. A single encoder pretrained jointly on several STRUM modalities is the natural next test, and we are currently exploring it.
We now address the case of fixed devices in the Discussion. Thank you also for your comments on presentation and readability, which we have worked to improve throughout the paper.
Thank you again, and we look forward to any further thoughts you may have.
Thank you for the additional experiments and clarifications. They address several of my concerns, particularly the controlled modality–model comparison and the more cautious interpretation of physiological attribution.
However, I remain concerned that the evidence does not support the paper’s central claims at the broader level implied. The benchmark evaluates only two autonomic targets in a small, demographically narrow cohort. Although the revised experiments provide a more controlled comparison, the observed modality advantage remains specific to the tested models and evaluation setting; it does not establish broader guidance about modality choice. The dataset may enable valuable work on other tasks, but that potential remains prospective in this paper.
I am also concerned that the cohort is too small to support a substantial deep-learning benchmark contribution. With 66 participants in 33 dyads, the effective sample size for evaluating generalization to unseen participants is limited, regardless of the total recording hours. This raises concerns about how robust the deep-learning results are to the particular cohort and evaluation split. The evidence presented does not, in my view, sufficiently rule out overfitting or establish broad conclusions about deep-learning methods.
Taken together, these limitations leave me unconvinced that the paper’s overall contribution is sufficiently supported for acceptance. I recognize the potential value of the dataset and the improvements made in the revision, but I maintain my weak-reject recommendation.
Thank you for your careful reading of the revision, and for acknowledging the improvements we made during the rebuttal period.
We have tried to address your concerns about scope and cohort size directly. On scope, we do not aim in this paper to give general guidance on modality choice: our claims are stated for the two autonomic targets, this cohort, and the tested models. We leave other targets explicitly untested, and present STRUM as a dataset and benchmark that could support that broader question in future work.
On cohort size, every test treats the dyad as the unit, so the sample size is already reflected in every interval. To check that our results do not depend on the particular cohort or evaluation split, we have added several analyses, which are included in the updated version (Appendix A.4). The modality-over-model gap remains significant when any single dyad is removed and in nearly all random subsets of half the dyads. It also holds across different dyad splits, scoring rules, and additional model classes, and, with ridge and the pretrained encoders, it does not change detectably when training is reduced from 23 to 8 dyads. Since every score comes from dyads the model never saw, and models trained on permuted targets score near zero, these results argue against overfitting.
We understand that you may still have concerns about the number of targets and the size of the cohort, but we respectfully disagree that these limit the soundness of our results as stated or of the contribution of this paper as a whole. We believe STRUM remains a valuable asset for the community, as it opens the door to comparing modalities on the same people at the same moment, measuring leakage between sensors directly, and studying how two people's physiology couples during a shared task.
Thank you again for your feedback and thoughtful review.
Thank you — initial response to Reviewer gZJn
Many thanks for your review; your comments have been very helpful and will make this paper better. To make the modality versus model comparison controlled, we now run every model on every sensor with the same windows and participants. Preliminary results point the same way, with some interaction between sensor and model that we now report in the paper. Since, as you point out, a weak model could hide useful information, we are fine-tuning every pretrained encoder. We also added time-aligned fusion between modalities, and we reword the masking and lag results as shared information rather than attribution. We now also report results person by person, simplify the tables, and discuss how the corpus can guide which sensors to record when devices are not yet fixed. You are right that other targets and broader cohorts would be very valuable. The two autonomic targets we report are one example of many this dataset supports, including cognitive, behavioral and emotional targets, in cooperative or individual settings. We are actively working on several of these in follow-up studies beyond this paper, and we are also excited about the use cases the community will pursue with this dataset. Please let us know if there is anything else you would like us to address. We will share the updated draft by this weekend.