acceptodds
Under review as a conference paper at ICLR 2027

STRUM: A Dyadic Multimodal Benchmark for Comparison of Physiological Modalities

Abstract

Representation learning for physiological signals has focused on scale, prioritizing longer recordings and larger cohorts. How the signals compare to one another is harder to establish: cross-modality comparisons are typically drawn from separate studies, each with its own cohort and instrumentation, so any apparent difference carries everything else that varied along with it. Isolating it requires recording every candidate modality from the same participants simultaneously, which no EEG benchmark provides. We present STRUM (Small Team Reconnaissance Urban Missions), a corpus of 66 people recorded as 33 pairs playing a cooperative mission game, with both partners instrumented at once across eleven stream families on a single clock: 205-channel scalp EEG each, muscle, cardiac, ocular, electrodermal and respiratory channels, eye tracking, motion capture, the in-game avatar, a force plate and the controller, over roughly 87 h, with the two participants' neural recordings aligned to a median of 0.208 ms. We define two continuous, event-free regression tasks on the corpus, predicting a person's phasic electrodermal arousal and mean heart rate from the remaining simultaneously acquired channels, evaluated under leave-one-dyad-out cross-validation, released with fixed splits and a reference loader. Across both targets, modality choice affects decoding more than model choice: the neck muscle band alone reaches on heart rate, ahead of every other modality in the same comparison, and pretrained encoders on EEG do not surpass a small network trained directly on muscle on matched windows. Multimodal fusion improves further even with encoders reused off the shelf: the full suite reaches against the best single modality's , a gain of . Simultaneous acquisition separates an apparent neural result into the muscular, ocular and cardiac signal inside it, and recording both partners makes pair-level targets measurable. For continuous autonomic decoding, the instrumentation chosen at acquisition constrains performance more than the encoder chosen at training.

Primary area:
datasets and benchmarks
Keywords:
multimodal fusion, time series representation, collaborative gaming, multimodal biosignals, benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_gZJnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces STRUM, a synchronized dyadic multimodal physiological dataset with 66 participants recorded during cooperative tasks using eleven signal families, including EEG, EMG, ECG, EOG, EDA, respiration, eye tracking, motion capture, force plate, controller, and avatar signals. The paper evaluates two continuous regression tasks—phasic electrodermal arousal and mean heart rate—and compares classical, from-scratch, and pretrained models under leave-dyad-out evaluation. Its main conclusion is that modality choice affects decoding performance more than model choice, with additional analyses on multimodal fusion, personalization, modality necessity, and inter-partner physiological coupling.

Critical strengths

1.The scope and idea itself is unique interesting

2.The synchronized dyadic acquisition across many physiological and behavioral modalities is unusual and potentially useful.

3.The paper pays careful attention to synchronization, subject-level splitting, target circularity, and statistical testing.

Critical weaknesses
  1. Many things feels over claimed and lack of proof (1)The central modality-versus-model claim is not based on a controlled modality × architecture comparison. Different modalities use different encoders, pretraining sources, input structures, and in some cases different window sets. (2) Modality usefulness is inherently model-dependent. Poor decoding performance may reflect an unsuitable or insufficient model rather than limited information in the modality itself. (3) The benchmark contains only two closely related autonomic targets. It is unclear whether the observed modality ranking would generalize to cognitive, behavioral, motor, or clinical prediction tasks. (4) The cohort is small and demographically narrow for the breadth of the generalization claims: 66 participants, ages 18–27, with strong imbalance toward male participants. (5) The paper’s own personalization results suggest strong subject-specific physiological structure, which raises further concerns about drawing broad modality-level conclusions from this limited population. (6) The multimodal evaluation does not fully exploit the dataset’s synchronization. The main fusion approach relies largely on separate encoders followed by feature concatenation. (7) Several mechanistic interpretations appear stronger than the evidence supports. Masking, lag analysis, and leave-one-modality-out experiments establish association or redundancy, not unique physiological attribution.

  2. The practical framing is somewhat narrow. In many medical applications, available modalities are fixed by devices, protocols, cost, or invasiveness, making model improvement the more relevant problem.

  3. The presentation is difficult to follow. Several tables and figures are dense, and some layouts visually encourage comparisons that the evaluation protocol does not fully support.

Questions for authors

see weaknesses

4 comments
AuthorsOpenReview ↗

Thank you — initial response to Reviewer gZJn

Many thanks for your review; your comments have been very helpful and will make this paper better. To make the modality versus model comparison controlled, we now run every model on every sensor with the same windows and participants. Preliminary results point the same way, with some interaction between sensor and model that we now report in the paper. Since, as you point out, a weak model could hide useful information, we are fine-tuning every pretrained encoder. We also added time-aligned fusion between modalities, and we reword the masking and lag results as shared information rather than attribution. We now also report results person by person, simplify the tables, and discuss how the corpus can guide which sensors to record when devices are not yet fixed. You are right that other targets and broader cohorts would be very valuable. The two autonomic targets we report are one example of many this dataset supports, including cognitive, behavioral and emotional targets, in cooperative or individual settings. We are actively working on several of these in follow-up studies beyond this paper, and we are also excited about the use cases the community will pursue with this dataset. Please let us know if there is anything else you would like us to address. We will share the updated draft by this weekend.

AuthorsOpenReview ↗

Response to Reviewer gZJn: revised paper and new experiments

Thank you for your detailed feedback, which helped us improve the quality and soundness of the paper. Based on the reviews, we ran substantially more experiments, which changed our results section, while its main messages held. We would welcome your view on whether the revision now meets your concerns. Dark blue marks new or changed text, and green italics report fine-tuning runs that are still finishing, so those values may change slightly.

Thank you for raising the need for a controlled comparison of modality and model. We built a grid that runs four models, ridge, a CNN trained from scratch, SleepFM, and MOMENT, on every modality, scored on the same windows and participants, with all 205 EEG channels. Modality choice still matters more than model choice, by 0.047 [0.031,0.063]0.047\ [0.031, 0.063] on heart rate and 0.071 [0.050,0.092]0.071\ [0.050, 0.092] on electrodermal arousal. A permutation test shows that this gap does not come from how the grid is built.

As you commented, a modality's usefulness partly depends on the model. We now report the modality–model interaction, which carries 27% and 17% of the systematic variance, while modality keeps the larger effect overall. To test whether weaker models hide information, we fine-tuned every pretrained encoder on STRUM, fully or with LoRA. Fine-tuning gives slight gains, and motion capture remains the strongest modality with and without it. Ridge and the CNN, which are trained on STRUM alone, also rank it first.

To avoid over-claiming, we scoped the claims to the two targets and the models we tested, and the abstract now states that motion capture carries most of the modality gap. On electrodermal arousal, the gap still holds without motion capture once ECG is included (+0.043 [+0.027,+0.061]+0.043\ [+0.027, +0.061]). We reworded the masking, lag, leave-one-out, and coupling analyses to describe association rather than attribution, and we state the limitations more clearly, including the size and composition of the cohort.

On the subject-specific structure suggested by our personalization results, each participant's own model does not beat a single model trained on all other participants on heart rate, and does worse on electrodermal arousal. EMG also beats EEG in 31 of 39 participants on heart rate and 30 of 39 on electrodermal arousal, so the group-level conclusions are not driven by a few people.

As you noted, adding non-autonomic targets would be very insightful, and we are working on it as a next step. Our current compute does not yet support running the full model grid and its controls for each new target, so we consider this out of scope for this paper and part of our future work. We believe it does not diminish STRUM's value as a ready-to-use multimodal dyadic resource for the many tasks it supports, cooperative or individual, autonomic or not.

We also implemented a time-aligned fusion of modalities, which outperforms every single modality given to the same model, by +0.107 [+0.056,+0.160]+0.107\ [+0.056, +0.160] on heart rate and +0.056 [+0.022,+0.089]+0.056\ [+0.022, +0.089] on electrodermal arousal. A single encoder pretrained jointly on several STRUM modalities is the natural next test, and we are currently exploring it.

We now address the case of fixed devices in the Discussion. Thank you also for your comments on presentation and readability, which we have worked to improve throughout the paper.

Thank you again, and we look forward to any further thoughts you may have.

Thank you for the additional experiments and clarifications. They address several of my concerns, particularly the controlled modality–model comparison and the more cautious interpretation of physiological attribution.

However, I remain concerned that the evidence does not support the paper’s central claims at the broader level implied. The benchmark evaluates only two autonomic targets in a small, demographically narrow cohort. Although the revised experiments provide a more controlled comparison, the observed modality advantage remains specific to the tested models and evaluation setting; it does not establish broader guidance about modality choice. The dataset may enable valuable work on other tasks, but that potential remains prospective in this paper.

I am also concerned that the cohort is too small to support a substantial deep-learning benchmark contribution. With 66 participants in 33 dyads, the effective sample size for evaluating generalization to unseen participants is limited, regardless of the total recording hours. This raises concerns about how robust the deep-learning results are to the particular cohort and evaluation split. The evidence presented does not, in my view, sufficiently rule out overfitting or establish broad conclusions about deep-learning methods.

Taken together, these limitations leave me unconvinced that the paper’s overall contribution is sufficiently supported for acceptance. I recognize the potential value of the dataset and the improvements made in the revision, but I maintain my weak-reject recommendation.

Thank you for your careful reading of the revision, and for acknowledging the improvements we made during the rebuttal period.

We have tried to address your concerns about scope and cohort size directly. On scope, we do not aim in this paper to give general guidance on modality choice: our claims are stated for the two autonomic targets, this cohort, and the tested models. We leave other targets explicitly untested, and present STRUM as a dataset and benchmark that could support that broader question in future work.

On cohort size, every test treats the dyad as the unit, so the sample size is already reflected in every interval. To check that our results do not depend on the particular cohort or evaluation split, we have added several analyses, which are included in the updated version (Appendix A.4). The modality-over-model gap remains significant when any single dyad is removed and in nearly all random subsets of half the dyads. It also holds across different dyad splits, scoring rules, and additional model classes, and, with ridge and the pretrained encoders, it does not change detectably when training is reduced from 23 to 8 dyads. Since every score comes from dyads the model never saw, and models trained on permuted targets score near zero, these results argue against overfitting.

We understand that you may still have concerns about the number of targets and the size of the cohort, but we respectfully disagree that these limit the soundness of our results as stated or of the contribution of this paper as a whole. We believe STRUM remains a valuable asset for the community, as it opens the door to comparing modalities on the same people at the same moment, measuring leakage between sensors directly, and studying how two people's physiology couples during a shared task.

Thank you again for your feedback and thoughtful review.

Reviewer_G2mDOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

STRUM argues that comparisons between physiological modalities are usually drawn across separate studies, so differences in cohort and equipment are mixed into any apparent modality effect. It addresses this with a corpus in which every candidate signal is recorded from the same people at the same moments, and with both members of a pair instrumented at once.

Contributions. Dataset: 66 participants in 33 pairs playing a cooperative game, 87 h in total eleven stream families on one LSL clock: 205-channel EEG, neck/face EMG, ECG, EOG, EDA, respiration, eye tracking, motion capture, force plate, controller and avatar pose released in BIDS format with event annotations, fixed splits and a loader Benchmark: predicting phasic EDA and mean heart rate from the other signals, scored by per-participant Pearson r under leave-one-dyad-out. Protocol: an enforced circularity ban (ECG may not predict heart rate), removal of the fixed mission-schedule confound, paired tests with FDR correction, and minimum detectable effects for null results. Findings: sensor choice matters more than model choice for these targets fusion beats any single modality apparent neural signal is partly muscle and eye activity partners’ heart signals covary during cooperation

Validation. Leave-one-modality-out necessity over 26 dyads: the full suite reaches r = 0.507 on heart rate and 0.400 on EDA. Neck EMG is the best single modality for heart rate (0.445) and motion capture for EDA (0.376). Model comparison: ridge vs. small CNN vs. pretrained encoders (CBraMod, BIOT, SleepFM, PhysioWave, ECGFounder, MOMENT). Additional analyses: synchronization quality (median 0.208 ms between the two EEG streams) leakage and split-inflation controls masking decomposition of the neural signal personalization codes partner coupling against surrogate pairs

Critical strengths

A genuinely rare resource. Recording both partners at once, on one clock, with dense EEG, EMG, motion capture and peripheral physiology together, makes possible something separate-cohort studies can’t do: removing one modality while holding everything else fixed. Even if the benchmark tasks need work, the corpus itself is likely to be valuable for hyperscanning, multimodal fusion and physiological representation learning. Rigorous and unusually honest evaluation. The paper: holds out partners together because they share a task timeline quantifies how much random splits inflate results (+0.09 to +0.46) enforces the ECG circularity ban in code removes the schedule confound explicitly reports minimum detectable effects before calling anything a null persists per-fold predictions so every statistic can be traced This level of care is well above typical benchmark papers and makes the reported numbers trustworthy as far as they go.

Critical weaknesses

Both benchmark targets are probably contaminated by artifacts from nearby sensors. Heart rate: neck EMG is the best single predictor, but surface electrodes on the neck sit close to the heart and carotid arteries and pick up the ECG itself. The authors acknowledge the EMG passband retains cardiac content. EDA: the uncalibrated channel has no artifact control, and its best predictor, motion capture, peaks at zero lag rather than the 1–3 s delay an autonomic response would show. The authors themselves decline to call this autonomic coupling. Without ECG-artifact removal from the EMG and motion-artifact screening of the EDA, a leaderboard on these targets risks rewarding detection of heartbeat leakage and electrode movement rather than autonomic state. The headline “modality matters more than model” claim reaches beyond the design. Every pretrained encoder was trained on another corpus, and none on movement-time EMG or motion capture, so the model side of the comparison is handicapped from the start. The heart-rate null for EEG uses only 8 of 205 channels through a sleep encoder, yet a simple band-power pipeline keeps improving up to 192 channels without plateauing. That suggests the neural representation, not the modality, is the bottleneck. With two autonomic targets that are physically close to their best predictors, the conclusion should be scoped to this setting rather than stated as a general lesson about instrumentation versus encoders.

Questions for authors

How much of neck EMG’s heart-rate performance survives ECG-artifact removal (e.g., template subtraction), or is time-locked to R-peaks? What fraction of phasic-EDA windows are motion-contaminated, and do the modality rankings change on clean windows? What does the heart-rate necessity analysis show for the neural block with the full EEG montage through CBraMod or REVE?

4 comments
AuthorsOpenReview ↗

Thank you — initial response to Reviewer G2mD

Thank you very much for your careful review; it will clearly make the paper stronger. Thanks for pointing out that both targets could reward leakage. We now subtract the heartbeat from the neck EMG and screen the EDA signal for movement artifacts, and the benchmark now includes both, with results reported on the clean windows. Preliminary results suggest that part of the EMG advantage on heart rate was the heartbeat, while the EDA ranking seems to hold on clean windows. To test EEG properly, every model is now also tested with all 205 EEG channels, including CBraMod and REVE, and so far EEG still adds nothing detectable. Because every encoder was pretrained elsewhere, we are also fine-tuning all of them. Finally, as you suggest, we now scope the modality-versus-model result to these autonomic targets, as a finding about these targets rather than a limit on what the dataset supports. Is there anything else you would like us to address during the rebuttal? The updated draft will follow by this weekend.

AuthorsOpenReview ↗

Response to Reviewer G2mD: revised paper and new experiments

Thank you for your review and for the detailed pointers for improvement you provided. Following the reviews, we ran substantially more experiments, so our results section changed, but its main messages held. We have also worked throughout to make the paper clearer and more consistent. We outline below how we addressed each concern and would value your view on whether the changes resolve them. New or changed text appears in dark blue, and fine-tuning results still running appear in green italics and may change slightly.

We have addressed your concern that both targets could reward leakage from nearby sensors, which has helped us make the benchmark more reliable. As you suggested, we implemented template subtraction of the ECG-locked heartbeat from each neck EMG channel, with the same subtraction at shifted beat times as a control. We use this cleaned EMG throughout the paper and include it in the release. The subtraction lowers the EMG scores of frozen MOMENT and SleepFM by 0.220.22–0.240.24 relative to the shifted control, and cleaned EMG now scores 0.320.32–0.370.37 across the four grid models. Removing EMG from fusion now costs +0.005+0.005 instead of +0.123+0.123, and motion capture leads on heart rate for every model.

On electrodermal arousal, we implemented a motion screen that flags about one third of windows as electrode movement, and we release the remaining windows as a motion-screened split. Every model keeps its modality ranking on this split, and modality choice still outweighs model choice. We also introduced a lag rule that formalizes your point about the zero-lag peak. It marks part of motion capture's score as movement, and the benchmark treats motion capture as the movement reference. ECG leads by about 1.91.9 s, like an autonomic response.

We now give all grid models the full 205-channel montage. EEG still ranks below the body-derived signals on average across models, which we now report as a main finding. With CBraMod or REVE reading all channels, removing EEG from fusion has no detectable cost on heart rate. The changes are −0.002 [−0.025,+0.018]-0.002\ [-0.025, +0.018] and +0.004 [−0.020,+0.028]+0.004\ [-0.020, +0.028]. We agree that the band-power trend shows room for better EEG representations, and the paper now says so.

Thank you for pointing out that encoders pretrained on other corpora start at a disadvantage. We have since fine-tuned each of them on STRUM, fully or with LoRA, with slight gains. Motion capture remains the strongest modality with and without fine-tuning, including for ridge and the CNN, which see only STRUM. As you recommended, we now scope the modality-over-model claim to the two targets and the models we tested, and we report the modality–model interaction. We also added time-aligned fusion of modalities, which outperforms every single modality given to the same model on both targets.

We would welcome any further comments and are happy to clarify anything that remains open.

Reviewer_G2mDOpenReview ↗

Thank you for the revision and new experiments

Thank you for the detailed response and the substantial additional work. My three questions are addressed:

  • The ECG template subtraction, with the shifted-beat control, shows that the neck-EMG advantage on heart rate was largely cardiac. Using the cleaned EMG throughout and releasing it makes the benchmark more trustworthy.
  • The motion screen and the lag rule deal with my concern about the electrodermal target, and it is reassuring that the modality ranking holds on the screened split.
  • With the full 205-channel montage through CBraMod and REVE, the EEG result on heart rate no longer depends on a small channel subset, and I appreciate the note that better EEG representations may still help.

Scoping the modality-over-model claim to these two targets and the tested models, and fine-tuning the encoders on STRUM, also resolve my second weakness.

One small suggestion: please replace the fine-tuning values still marked as running with final numbers in the camera-ready version. I have no further questions and will take the revision into account in my final assessment.

Thank you for the positive assessment and for your careful reading of the revision. We are glad the heartbeat subtraction, the motion screen, and the full-montage EEG results address all your questions and concerns. Following your suggestion, all fine-tuning values in the updated version are now final, and no values remain marked as running.

Reviewer_g24ROpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces STRUM, a multimodal dataset collected from 66 participants in 33 pairs performing cooperative tasks. The dataset includes synchronized EEG, EMG, ECG, EOG, motion capture, and other signals, totaling 87.4 hours of recordings (174.8 subject-hours). The authors evaluate the dataset on heart-rate and electrodermal activity prediction, comparing different modalities, pretrained models, and fusion strategies. They also investigate cardiac synchronization between participants.

Critical strengths
  1. The dataset itself is a useful contribution. The simultaneous recording of multiple modalities from both participants is valuable, as it reduces differences in populations, tasks, and recording setups across datasets. It also allows comparisons between real and surrogate dyads.
  2. The paper carefully addresses several common evaluation issues. It uses leave-one-dyad-out evaluation to avoid data leakage, examines the effects of overlapping windows, prevents direct ECG-based heart-rate prediction, and includes controls for mission effects and elapsed time.
  3. The modality analysis provides useful insights beyond simply comparing model performance. In Table 4, the full model achieves a heart-rate correlation of 0.5072, compared with 0.4449 using neck EMG alone. Removing EMG reduces performance by 0.0973 (95% CI: [0.0299, 0.1569]). Motion capture also provides additional benefits. These results help show which modalities contribute useful information.
  4. The paper discusses its limitations and negative findings. The authors acknowledge that electrodermal prediction from motion may be affected by artifacts, that personalization results are limited to within-session settings, and that cardiac correlations between partners do not necessarily imply causal influence.
Critical weaknesses
  1. My main concern is that the strong performance of neck EMG for heart-rate prediction may come from cardiac signals picked up by the electrodes rather than actual muscle activity. The paper acknowledges this possibility on page 5. Simply excluding ECG inputs does not fully address this issue, since cardiac signals may still be present in EMG and EEG. I think the authors should analyze how much cardiac information is present in neck EMG and whether the performance remains strong after removing these signals.

  2. The EDA prediction results need further validation. Motion capture achieves the best standalone performance (r = 0.3760), and its prediction peak occurs much earlier than those of physiological signals. This raises concerns that the model may be learning movement artifacts rather than actual physiological responses. It would be helpful to provide more details about EDA signal quality, electrode placement, and artifact removal, and to evaluate performance after excluding potentially noisy segments.

  3. The conclusion that modality choice matters more than model choice is not fully supported by the current experiments. Some comparisons use different data windows and channel configurations, making it difficult to draw a fair conclusion. Also, the neural models use only a small subset of EEG/EOG channels, while the classical models continue to improve with more channels. I think a more controlled comparison using the same data, channels, and evaluation settings would make this conclusion more convincing.

  4. Some details about model adaptation and statistical analysis are unclear. The paper uses unlabeled data from held-out participants for alignment, but does not clearly explain how much this contributes to performance. It is also unclear how participant-specific representations are obtained for new subjects and whether labeled calibration data is needed. In addition, the cardiac synchronization analysis uses only seven surrogate assignments, which seems limited for drawing reliable conclusions. More details and additional experiments would help clarify these results.

Questions for authors
  1. How much does neck-EMG performance change after removing cardiac signals? Can the authors clarify whether the improvement comes from muscle activity or cardiac signals?
  2. Where are the EDA electrodes placed, and how are movement artifacts handled? Do the results remain consistent after removing noisy segments?
  3. Can the authors compare different modalities and models using the same data windows, participants, channels, and training settings?
  4. How are participant-specific codes and alignment computed for unseen participants? How much does alignment improve performance, and how is the reliability ceiling estimated?
3 comments
AuthorsOpenReview ↗

Thank you — initial response to Reviewer g24R

Thank you so much for your review; it has been very useful and will definitely help make this paper better. Thanks for pointing out that neck EMG may carry the heartbeat. We now subtract the heartbeat from the EMG, and preliminary results suggest that part of its advantage was cardiac. We are also screening the EDA signal for movement artifacts and will report results on the clean windows, where the modality ranking holds so far. For a fair comparison, we are also running every model on every sensor with the same windows, participants, and all EEG channels, and we are fine-tuning every pretrained encoder. We now test the coupling against all surrogate partners rather than seven, and we will explain the person codes and the reliability ceiling more clearly. Are there any other concerns you would like to see addressed during the rebuttal? We will share the updated draft by this weekend.

AuthorsOpenReview ↗

Response to Reviewer g24R: revised paper and new experiments

Thank you again for your review, it has made the paper substantially stronger. Our results section changed because we ran substantially more experiments based on the reviews, but the main messages did not change. We would be glad to hear whether the changes below resolve your concerns. In the paper, new or changed text is in dark blue, and green italics mark fine-tuning results that are still running and may change slightly.

Thank you for pointing out that neck EMG may carry cardiac signal. We now subtract the ECG-locked heartbeat from every EMG channel, with the same subtraction at shifted beat times as a control, and use this cleaned EMG throughout the paper. As you suspected, the EMG advantage on heart rate was largely cardiac, and removing EMG from fusion now costs +0.005+0.005 instead of +0.123+0.123. This does not affect the main findings, since motion capture leads on heart rate for every model and modality choice still matters more than model choice. The full EEG montage carries only about a third of that heartbeat, so removing it would only lower EEG's score further.

For electrodermal arousal, recorded with one uncalibrated bipolar sensor on the back of the hand, we now flag windows with electrode-movement artifacts and release the screened windows as a separate split. On these windows, the modality ranking and the modality-over-model gap both hold. Following your observation on motion capture's early peak, we introduced an explicit lag rule, which marks part of its electrodermal score as movement. ECG, by contrast, leads the target by about 1.91.9 s, as an autonomic response would, so the rule separates physiological responses from movement.

Following your suggestion of a controlled comparison, we now run every model on every modality and score it on the same windows and participants. Every model in this grid reads all 205 EEG channels, instead of the small subset you pointed out. This controlled comparison confirms that modality choice matters more than model choice, and we also report their interaction. The grid also reinforces that EEG ranks below the body-derived signals on average across models, even when the models read the full montage, which the paper states as a main finding. As you observed, a classical pipeline keeps improving up to 192 channels, so scalp channels may carry information that current encoders do not yet fully use.

To make the analysis more complete, we also fine-tuned every pretrained encoder on STRUM, fully or with low-rank adaptation, and added time-aligned fusion, which beats every single modality given to the same model on both targets. Fine-tuning gives small gains, and motion capture stays the strongest modality.

On your remaining questions, our models use no alignment step, so the results do not depend on one, and we have corrected this statement in the paper. The paper now explains how person codes are computed for unseen participants without labeled calibration. We now test coupling against every surrogate partner who held the same seat instead of seven fixed assignments, aligned mission by mission, and coupling holds in both conditions. We estimate the reliability ceiling with a split-window Spearman–Brown correction, which Appendix A.2 describes.

We appreciate any further feedback and would be glad to address any remaining concerns.

Reviewer_g24ROpenReview ↗

Thanks for clarification

Thanks for the detailed response and additional results. I have no further concerns and will maintain my positive rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

We'd like to thank all reviewers for their detailed comments, which significantly improved the quality and narrative of the paper. We summarize the main changes below.

Controlled comparison of modality and model. In response to the reviews, we built a grid in which four models run on every modality: ridge, a CNN trained from scratch, SleepFM, and MOMENT. All of them are scored on the same windows and participants, and the EEG models read all 205 channels. With this design, we found that modality choice matters more than model choice, by 0.047 [0.031, 0.063] on heart rate and 0.071 [0.050, 0.092] on electrodermal arousal. The paper now reports the interaction between modality and model as well. We found that EEG ranks below the body-derived signals on average across models, even when the models read the full montage.

Robustness to the cohort and the split. Every test treats the dyad as the unit, so each interval reflects the number of pairs rather than the number of windows. The modality-over-model gap stays significant when any single dyad is removed and in nearly all random subsets of half the dyads. It also holds across other dyad splits and scoring rules, with additional models, and after controlling for the shared task schedule. With ridge and the pretrained encoders, it shows no detectable change when training is reduced from 23 to 8 dyads. Models trained on permuted targets score near zero, which also argues against overfitting.

Leakage between sensors. Recording every sensor at once lets us measure leakage directly. Neck EMG picks up the heartbeat, so we subtract the ECG-locked heartbeat from every EMG channel, with a shifted-beat control, and use the cleaned EMG throughout. On EDA, electrode movement can mimic an electrodermal response, so we flag movement windows, release a motion-screened split, and use a lag rule to separate physiological responses from movement.

Fine-tuning and fusion. We fine-tuned the pretrained encoders on STRUM, fully or with LoRA, which gave modest gains. Motion capture remains the strongest modality with and without fine-tuning. Ridge and the CNN, which are trained only on STRUM, rank it first as well. We also added time-aligned fusion, which beats every single modality given to the same model on both targets.

Scope and interpretation. We now state our claims for the two autonomic targets, this cohort, and the tested models, and we state the limitations of the cohort explicitly. We reworded the masking, lag, and coupling analyses to describe association rather than attribution. Coupling is now tested against every surrogate partner who held the same seat. The fixed splits, the motion-screened split, the gated heart-rate target, per-seed values, and the heartbeat-subtraction code are available to reviewers on the anonymous project site (https://strum-neuro.netlify.app).

We thank the reviewers for feedback that made the paper's claims more precise, its comparisons better controlled, and its evidence more robust. We acknowledge the remaining concerns about the number of targets and the size of the cohort, and the paper states them as limitations. We consider these concerns addressed within the scope of this paper, and we strongly believe that STRUM and its benchmark are a valuable contribution to the community. They make it possible to compare modalities on the same people at the same moment, to measure leakage between sensors directly, and to study how two people's physiology couples during a shared task. We hope STRUM will be a useful resource for the community, and we are grateful to all reviewers for their time and engagement throughout the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.