- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces a unified benchmark for mobile human sensing by harmonizing seven longitudinal datasets at the structured sensing-feature level. It defines three generalization settings—within-user temporal prediction, unseen-user generalization, and cross-dataset transfer—and systematically evaluates classical machine learning, deep tabular models, DG/UDA methods, and tabular foundation models under these settings.
Critical strengths
- The benchmark is well designed and covers multiple datasets, tasks, and timescales. The S1–S3 protocol provides a clear progression from temporal shift to user shift and finally study-level distribution shift.
- The experimental scope is broad, with diverse model families evaluated under largely unified preprocessing, splitting, tuning, and evaluation protocols.
- The empirical findings are useful beyond individual model rankings. For example, the advantage of foundation models under within-user evaluation largely disappears for unseen users and independent datasets, while standard DG/UDA methods provide limited consistent gains.
- The benchmark construction, splits, and evaluation pipeline appear sufficiently detailed and reusable, giving the work potential value as a community resource.
Critical weaknesses
- The main contribution is a benchmark rather than a new learning method, so its novelty should be positioned more carefully relative to prior sensing-generalization benchmarks. GLOBEM (IMWUT 2023) already studies cross-dataset generalization in longitudinal human sensing, FEEL (NeurIPS 2025 Datasets and Benchmarks) benchmarks heterogeneous physiological sensing, and SIDA (TIST 2024) studies generalization across unseen subjects in physiological sensing. The stronger novelty here therefore lies in jointly organizing multiple timescales and temporal/user/dataset-level generalization within one evaluation framework, rather than in introducing generalizable sensing as a new problem.
- Harmonization at the structured-feature level necessarily discards fine-grained temporal information in raw or sequential signals, and the shared cross-dataset feature space contains relatively limited wearable-specific information.
- Some datasets are related in terms of collection groups or sensing ecosystems, and the participant populations are concentrated around students and young adults, which limits the breadth of the study-level generalization claims.
- Label harmonization across independently collected studies introduces additional design choices, and substantial differences in class balance raise questions about strict semantic comparability across some tasks.
Questions for authors
Please see Weaknesses
7 comments
Q2. Structured features and temporal information
Q2. Structured features and temporal information
Harmonization at the structured-feature level necessarily discards fine-grained temporal information in raw or sequential signals, and the shared cross-dataset feature space contains relatively limited wearable-specific information.
We agree that aggregation discards within-window temporal order. This comment addresses temporal information; Q3 addresses wearable coverage. We added a sequence evaluation to test an alternative representation, while acknowledging that it is not a raw-signal benchmark or a pure temporal-order ablation.
Design. The same 60 minutes preceding each prompt are represented as twelve 5-minute bins or twenty 3-minute bins, with observation masks (A.15). We evaluated 38 sequence models (B.9, Tables 32–33): recurrent, convolutional, attention-based and missingness-aware models; DG/UDA methods on a GRU; and eight pretrained time-series models. Comparisons match the setting, target, split, feature-space designation, and test rows—not the full input feature content.
Table 1. Sequence minus tabular performance on matched cells, 5-minute bins (pooled AUROC).
| Setting (cells) | Comparison | ΔAUROC [95% CI] | Holm p | Equivalent within ±0.01 |
|---|---|---|---|---|
| S1 (14) | Best vs. best | −0.101 [−0.112, −0.091] | 0.0013 | No |
| S2 (14) | Best vs. best | −0.005 [−0.015, +0.005] | 1.000 | No |
| S2 (14) | Mean vs. mean | −0.012 [−0.018, −0.006] | 0.021 | No |
| S3 (17) | Best vs. best | −0.003 [−0.011, +0.004] | 1.000 | Yes |
| S3 (17) | Mean vs. mean | −0.004 [−0.008, +0.000] | 1.000 | Yes |
“Best” selects the best model on each side over these cells; these selected-best comparisons are exploratory. “Mean” compares 22 sequence and 14 tabular models. Tests are paired Wilcoxon signed-rank tests with Holm correction; intervals and equivalence concern aggregate paired differences, not every individual cell.
Interpretation. The tested sequence representations do not yield a positive aggregate improvement in these comparisons. The best sequence model loses all 14 S1 cells to the best tabular FM. Under S2, the mean comparison is slightly negative; under S3, both reported comparisons meet the ±0.01 equivalence criterion. These results do not establish that temporal information is absent or that no individual sequence model can improve on any tabular model.
The GRU–MLP comparison, using the same training loop, differs by approximately 0.000 AUROC under S1. This compares two architectures as well as two representations; it does not hold the learner itself fixed.
Table 2. Shared inputs in the reported momentary runs. Feature dimensions below are those of the compared inputs, not a count of every feature available in the source releases.
| Target | Tabular features | Sequence channels per bin |
|---|---|---|
| Stress | 44 | 11 |
| Valence | 68 | 15 |
| Arousal | 79 | 17 |
The sequences preserve order but contain fewer features. Entropies, dispersion statistics and location-derived features are not retained in these short bins; no wearable channel is added. Therefore, differences cannot be attributed solely to retaining or discarding temporal order.
Scope and limits. These experiments cover the momentary track and three studies under S1/S2, with a fourth under S3. Inputs remain binned engineered features, not raw physiological or motion signals. GLOBEM and CES provide precomputed features for this benchmark; these experiments cannot establish what end-to-end models would recover from unavailable raw streams. They also do not establish an intrinsic ceiling on cross-study prediction.
Revisions already included: A.15 documents inputs and limitations; B.9 documents the roster and matching; C.12 reports results, including family means (Table 49), per-model results (Table 51), and detailed tables. By November 18 (AoE), we will summarize the tested sequence alternative and its restricted coverage in the main text and explicitly retain the raw-signal limitation in Section 4.5.
Q3. Wearable-specific feature coverage
Q3. Wearable-specific feature coverage
and the shared cross-dataset feature space contains relatively limited wearable-specific information.
We agree, and clarify the scope more precisely: no wearable feature is included in any Shared feature space. Every S3 result therefore uses phone-derived features. Wearable information is used only in GLOBEM's Native-space S1/S2 experiments.
Table 1. Wearable data in the source releases versus their actual benchmark use. The other four resources release no wearable data in the supplied inventory.
| Resource | Device | Released signals | Used in Native features? | Used in Shared features? |
|---|---|---|---|---|
| K-EmoPhone | Microsoft Band 2 | Heart rate, electrodermal activity, skin temperature, acceleration, steps | No | No |
| EmoPhone | Fitbit Inspire HR | Heart rate, steps, calories, distance | No | No |
| GLOBEM | Fitbit | Steps and sleep | Yes: 56 of 201 weekly features | No |
GLOBEM's retained wearable features comprise 34 sleep and 22 step features. The availability of wearable streams in a source release must not be confused with their inclusion in the evaluated feature matrices.
Why the Shared space excludes them. A Shared schema is the intersection of compatible features across the resources serving a target (A.12, Table 15). The stress and valence rosters include resources without wearable data, such as DeepStress, StudentLife, and DiversityOne. The existing K-EmoPhone and EmoPhone feature tables also exclude their wearable streams, so their presence in the arousal roster does not create wearable Shared features. GLOBEM has no evaluated S3 transfer partner.
Where they contribute. GLOBEM's Fitbit-derived features are retained in its Native representation (A.11, Table 14) and used under S1 and S2. Conversely, the wearable streams of K-EmoPhone and EmoPhone were not integrated into their benchmark Native matrices. The sequence extension adds no wearable channels (A.15).
What follows—and what does not. This is a substantive coverage limitation, not merely a naming issue. The S3 results characterize transfer of phone-derived behavioral/contextual representations. They do not establish that physiological wearable signals or wearable-specific foundation models would transfer equally poorly.
Adding the available wearable streams to K-EmoPhone/EmoPhone Native spaces is a feasible extension, but it is not reported here. Their differing devices would require explicit alignment; a comparison of only those two related studies would also have narrower provenance coverage than the full benchmark. Shared institutional origins do not by themselves preclude a cross-study comparison.
Revisions. A.14 already identifies where wearable information is retained and excluded, with sensing coverage in A.3/Table 7 and the Native/Shared definitions in A.11–A.12. We will correct A.3's source-release count to three, not four, and distinguish source-level device availability from model-input coverage throughout the dataset description. Section 4.5 will explicitly state that S3 is phone-derived and cannot support general claims about wearable physiological transfer. These remaining changes will be integrated by November 18 (AoE).
Q1. Novelty and positioning
Q1. Novelty and positioning
We agree. Our contribution is the unified evaluation and its empirical findings, not introducing sensing generalization or a new architecture.
Table 1. Positioning relative to existing sensing-generalization research. GLOBEM refers to the journal benchmark, not its separate four-year dataset release.
*Wearable features are retained only in GLOBEM's native space; S3 uses phone-derived features (Q3). Coverage varies by target and timescale. SIDA is a method contribution, not a general-purpose benchmark.
We acknowledge GLOBEM's cross-institute/year transfer and FEEL's heterogeneous multi-dataset evaluation. Our distinction is the joint organization of behavioral/contextual sensing, multiple targets, three timescales, and temporal/user/study-level evaluation. Shared research ecosystems do not imply identical protocols, but neither do seven collections establish complete independence (Q4).
Table 2. Model-family means on the shared affective benchmark (pooled AUROC).
Three findings go beyond repeating that transfer is difficult:
Revisions. We will refine the Introduction, Related Work, comparison table, and Discussion to acknowledge these precedents and emphasize unified evaluation and setting-dependent findings. We will describe GLOBEM's cross-institute/year evaluation explicitly rather than imply it lacks cross-dataset transfer. These remaining main-text changes will be integrated by November 18 (AoE).