acceptodds
Under review as a conference paper at ICLR 2027

Mobile-Human-Bench: A Unified Benchmark for Generalizable Mobile Human Sensing

Abstract

Mobile sensing enables modeling of human states and behaviors, yet progress is difficult to compare across studies because datasets differ in sensing modalities, targets, temporal resolution, preprocessing, and evaluation. We introduce Mobile-Human-Bench, a unified benchmark for evaluating generalization across time, users, and independently collected studies. It harmonizes seven longitudinal studies spanning 1,751 participants across nine countries and yields 295,162 labeled feature rows after benchmark construction. The benchmark operates at the structured sensing-feature level, supports momentary, daily, and weekly prediction, and covers affective, mental-health, and behavioral targets. We evaluate classical machine learning, deep tabular models, domain generalization, unsupervised domain adaptation, and tabular foundation models under within-user temporal, unseen-user, and cross-dataset evaluation. Results show that model rankings and apparent gains change substantially across generalization settings, targets, and timescales. Tabular foundation models are strong in within-user settings but do not retain a systematic advantage for unseen users or independently collected studies, while standard domain generalization and adaptation provide limited gains over matched controls. Mobile-Human-Bench provides a reproducible testbed for identifying which modeling advances remain robust across users, studies, outcomes, and timescales. Resources are available at https://anonymous.4open.science/r/Mobile-Human-Benchmark-B3C5.

Primary area:
datasets and benchmarks
Keywords:
mobile sensing, human sensing, benchmark, distribution shift, cross-user generalization, cross-dataset generalization, domain generalization, unsupervised domain adaptation, tabular foundation models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DXzqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a unified benchmark for mobile human sensing by harmonizing seven longitudinal datasets at the structured sensing-feature level. It defines three generalization settings—within-user temporal prediction, unseen-user generalization, and cross-dataset transfer—and systematically evaluates classical machine learning, deep tabular models, DG/UDA methods, and tabular foundation models under these settings.

Critical strengths
  1. The benchmark is well designed and covers multiple datasets, tasks, and timescales. The S1–S3 protocol provides a clear progression from temporal shift to user shift and finally study-level distribution shift.
  2. The experimental scope is broad, with diverse model families evaluated under largely unified preprocessing, splitting, tuning, and evaluation protocols.
  3. The empirical findings are useful beyond individual model rankings. For example, the advantage of foundation models under within-user evaluation largely disappears for unseen users and independent datasets, while standard DG/UDA methods provide limited consistent gains.
  4. The benchmark construction, splits, and evaluation pipeline appear sufficiently detailed and reusable, giving the work potential value as a community resource.
Critical weaknesses
  1. The main contribution is a benchmark rather than a new learning method, so its novelty should be positioned more carefully relative to prior sensing-generalization benchmarks. GLOBEM (IMWUT 2023) already studies cross-dataset generalization in longitudinal human sensing, FEEL (NeurIPS 2025 Datasets and Benchmarks) benchmarks heterogeneous physiological sensing, and SIDA (TIST 2024) studies generalization across unseen subjects in physiological sensing. The stronger novelty here therefore lies in jointly organizing multiple timescales and temporal/user/dataset-level generalization within one evaluation framework, rather than in introducing generalizable sensing as a new problem.
  2. Harmonization at the structured-feature level necessarily discards fine-grained temporal information in raw or sequential signals, and the shared cross-dataset feature space contains relatively limited wearable-specific information.
  3. Some datasets are related in terms of collection groups or sensing ecosystems, and the participant populations are concentrated around students and young adults, which limits the breadth of the study-level generalization claims.
  4. Label harmonization across independently collected studies introduces additional design choices, and substantial differences in class balance raise questions about strict semantic comparability across some tasks.
Questions for authors

Please see Weaknesses

7 comments
AuthorsOpenReview ↗

Q1. Novelty and positioning

Q1. Novelty and positioning

The main contribution is a benchmark rather than a new learning method, so its novelty should be positioned more carefully relative to prior sensing-generalization benchmarks. GLOBEM (IMWUT 2023) already studies cross-dataset generalization in longitudinal human sensing, FEEL (NeurIPS 2025 Datasets and Benchmarks) benchmarks heterogeneous physiological sensing, and SIDA (TIST 2024) studies generalization across unseen subjects in physiological sensing. The stronger novelty here therefore lies in jointly organizing multiple timescales and temporal/user/dataset-level generalization within one evaluation framework, rather than in introducing generalizable sensing as a new problem.

We agree. Our contribution is the unified evaluation and its empirical findings, not introducing sensing generalization or a new architecture.

Table 1. Positioning relative to existing sensing-generalization research. GLOBEM refers to the journal benchmark, not its separate four-year dataset release.

DimensionGLOBEM (Xu et al.)FEEL (Singh et al.)SIDA (Ma et al.)Mobile-Human-Bench
ScopeLongitudinal mobile sensingPhysiological emotion recognitionCross-subject sleep stagingLongitudinal mobile human sensing
Coverage4 coordinated institute-year datasets19 heterogeneous datasets3 sleep-staging datasets7 studies, multiple research programs
InputsPhone/wearable behavioral featuresEDA/PPG signalsPhysiological sleep signalsPhone/wearable behavioral/contextual features*
TargetsDepressionArousal/valenceSleep stagesAffect, mental health, behavior
TimescalesPrimarily weeklyDataset-specific segmentsSleep epochs/sequencesMomentary, daily, weekly
GeneralizationTemporal, cross-year, cross-institute, LODOWithin-/cross-datasetCross-subject DGS1 temporal, S2 unseen-user, S3 cross-study
ModelsDepression models, DG, ReorderTraditional/deep/pretrained physiological modelsProposed SIDA methodBaselines, deep tabular, tabular FMs/ICL, DG, UDA

*Wearable features are retained only in GLOBEM's native space; S3 uses phone-derived features (Q3). Coverage varies by target and timescale. SIDA is a method contribution, not a general-purpose benchmark.

We acknowledge GLOBEM's cross-institute/year transfer and FEEL's heterogeneous multi-dataset evaluation. Our distinction is the joint organization of behavioral/contextual sensing, multiple targets, three timescales, and temporal/user/study-level evaluation. Shared research ecosystems do not imply identical protocols, but neither do seven collections establish complete independence (Q4).

Table 2. Model-family means on the shared affective benchmark (pooled AUROC).

FamilyS1S2S3
Baseline: trees and generic neural networks0.60170.52470.5084
Deep tabular0.54570.52760.5101
Tabular foundation models0.63980.52470.5015
DG—0.52920.5106
UDA—0.53310.5115

Three findings go beyond repeating that transfer is difficult:

  • Setting-dependent FM advantages: +0.0381 over the baseline family in pooled S1, with 44/54 cell wins, but no retained advantage in S2/S3. Family spreads are 0.0084 and 0.0100. Pooled S1 combines within- and between-person discrimination (C.1).
  • Matched objective comparisons: 0/24 DG/UDA method–setting comparisons has a significant positive gain after Holm correction; 21/24 meet practical equivalence within ±0.01 AUROC. This concerns the evaluated objectives, not untested Reorder or all DG/UDA methods.
  • Outcome/timescale dependence: screen-on S3 AUROC is 0.667 (momentary), 0.507 (daily), and 0.604 (weekly). Different coverage and label construction limit comparison with affect; persistence contributes to this behavioral signal (C.3).

Revisions. We will refine the Introduction, Related Work, comparison table, and Discussion to acknowledge these precedents and emphasize unified evaluation and setting-dependent findings. We will describe GLOBEM's cross-institute/year evaluation explicitly rather than imply it lacks cross-dataset transfer. These remaining main-text changes will be integrated by November 18 (AoE).

AuthorsOpenReview ↗

Q2. Structured features and temporal information

Q2. Structured features and temporal information

Harmonization at the structured-feature level necessarily discards fine-grained temporal information in raw or sequential signals, and the shared cross-dataset feature space contains relatively limited wearable-specific information.

We agree that aggregation discards within-window temporal order. This comment addresses temporal information; Q3 addresses wearable coverage. We added a sequence evaluation to test an alternative representation, while acknowledging that it is not a raw-signal benchmark or a pure temporal-order ablation.

Design. The same 60 minutes preceding each prompt are represented as twelve 5-minute bins or twenty 3-minute bins, with observation masks (A.15). We evaluated 38 sequence models (B.9, Tables 32–33): recurrent, convolutional, attention-based and missingness-aware models; DG/UDA methods on a GRU; and eight pretrained time-series models. Comparisons match the setting, target, split, feature-space designation, and test rows—not the full input feature content.

Table 1. Sequence minus tabular performance on matched cells, 5-minute bins (pooled AUROC).

Setting (cells)ComparisonΔAUROC [95% CI]Holm pEquivalent within ±0.01
S1 (14)Best vs. best−0.101 [−0.112, −0.091]0.0013No
S2 (14)Best vs. best−0.005 [−0.015, +0.005]1.000No
S2 (14)Mean vs. mean−0.012 [−0.018, −0.006]0.021No
S3 (17)Best vs. best−0.003 [−0.011, +0.004]1.000Yes
S3 (17)Mean vs. mean−0.004 [−0.008, +0.000]1.000Yes

“Best” selects the best model on each side over these cells; these selected-best comparisons are exploratory. “Mean” compares 22 sequence and 14 tabular models. Tests are paired Wilcoxon signed-rank tests with Holm correction; intervals and equivalence concern aggregate paired differences, not every individual cell.

Interpretation. The tested sequence representations do not yield a positive aggregate improvement in these comparisons. The best sequence model loses all 14 S1 cells to the best tabular FM. Under S2, the mean comparison is slightly negative; under S3, both reported comparisons meet the ±0.01 equivalence criterion. These results do not establish that temporal information is absent or that no individual sequence model can improve on any tabular model.

The GRU–MLP comparison, using the same training loop, differs by approximately 0.000 AUROC under S1. This compares two architectures as well as two representations; it does not hold the learner itself fixed.

Table 2. Shared inputs in the reported momentary runs. Feature dimensions below are those of the compared inputs, not a count of every feature available in the source releases.

TargetTabular featuresSequence channels per bin
Stress4411
Valence6815
Arousal7917

The sequences preserve order but contain fewer features. Entropies, dispersion statistics and location-derived features are not retained in these short bins; no wearable channel is added. Therefore, differences cannot be attributed solely to retaining or discarding temporal order.

Scope and limits. These experiments cover the momentary track and three studies under S1/S2, with a fourth under S3. Inputs remain binned engineered features, not raw physiological or motion signals. GLOBEM and CES provide precomputed features for this benchmark; these experiments cannot establish what end-to-end models would recover from unavailable raw streams. They also do not establish an intrinsic ceiling on cross-study prediction.

Revisions already included: A.15 documents inputs and limitations; B.9 documents the roster and matching; C.12 reports results, including family means (Table 49), per-model results (Table 51), and detailed tables. By November 18 (AoE), we will summarize the tested sequence alternative and its restricted coverage in the main text and explicitly retain the raw-signal limitation in Section 4.5.

AuthorsOpenReview ↗

Q3. Wearable-specific feature coverage

Q3. Wearable-specific feature coverage

and the shared cross-dataset feature space contains relatively limited wearable-specific information.

We agree, and clarify the scope more precisely: no wearable feature is included in any Shared feature space. Every S3 result therefore uses phone-derived features. Wearable information is used only in GLOBEM's Native-space S1/S2 experiments.

Table 1. Wearable data in the source releases versus their actual benchmark use. The other four resources release no wearable data in the supplied inventory.

ResourceDeviceReleased signalsUsed in Native features?Used in Shared features?
K-EmoPhoneMicrosoft Band 2Heart rate, electrodermal activity, skin temperature, acceleration, stepsNoNo
EmoPhoneFitbit Inspire HRHeart rate, steps, calories, distanceNoNo
GLOBEMFitbitSteps and sleepYes: 56 of 201 weekly featuresNo

GLOBEM's retained wearable features comprise 34 sleep and 22 step features. The availability of wearable streams in a source release must not be confused with their inclusion in the evaluated feature matrices.

Why the Shared space excludes them. A Shared schema is the intersection of compatible features across the resources serving a target (A.12, Table 15). The stress and valence rosters include resources without wearable data, such as DeepStress, StudentLife, and DiversityOne. The existing K-EmoPhone and EmoPhone feature tables also exclude their wearable streams, so their presence in the arousal roster does not create wearable Shared features. GLOBEM has no evaluated S3 transfer partner.

Where they contribute. GLOBEM's Fitbit-derived features are retained in its Native representation (A.11, Table 14) and used under S1 and S2. Conversely, the wearable streams of K-EmoPhone and EmoPhone were not integrated into their benchmark Native matrices. The sequence extension adds no wearable channels (A.15).

What follows—and what does not. This is a substantive coverage limitation, not merely a naming issue. The S3 results characterize transfer of phone-derived behavioral/contextual representations. They do not establish that physiological wearable signals or wearable-specific foundation models would transfer equally poorly.

Adding the available wearable streams to K-EmoPhone/EmoPhone Native spaces is a feasible extension, but it is not reported here. Their differing devices would require explicit alignment; a comparison of only those two related studies would also have narrower provenance coverage than the full benchmark. Shared institutional origins do not by themselves preclude a cross-study comparison.

Revisions. A.14 already identifies where wearable information is retained and excluded, with sensing coverage in A.3/Table 7 and the Native/Shared definitions in A.11–A.12. We will correct A.3's source-release count to three, not four, and distinguish source-level device availability from model-input coverage throughout the dataset description. Section 4.5 will explicitly state that S3 is phone-derived and cannot support general claims about wearable physiological transfer. These remaining changes will be integrated by November 18 (AoE).

Reviewer_pebVOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2(+1)
Presentation
3
Soundness
3
Read review
Paper summary

The manuscript presents a unified benchmark platform for mobile human sensing. In particular, the authors gathered existing large-scale datasets, containing data from smartphones and wearable devices alike, and use these data to compare a plethora of existing ML approaches. In particular, they compare classical ML methods with Deep Learning as well as tabular foundation models. They also setup three different evaluation modalities, aimed at replicating different real-world scenarios.

Overall, the setup and benchmarks do not present significant methodological issues. However, they also lack novelty in the finding or in providing a benchmark that significantly goes beyond existing approaches

Critical strengths

The manuscript aggregates collections which are large by themselves and, when taken together, they further provide a large scale benchmarking setup. Additionally, the authors present a validation setup that goes beyond a simple traditional leave-one-participant-out (LOPO) cross validation approach

Critical weaknesses

There are significant weaknesses in the work that, while partially interesting, I think do not warrant publication. In particular, the benchmarking does not meaningfully extend beyond existing setups. For example, most of the results are consistent with what can already be obtained using the GLOBEM datasets. Moreover, while I appreciate the fact the authors tried to come up with a novel validation protocol, the leave-one-dataset-out setting does not really provide more actionable insight than a traditional LOPO. Indeed, what we see is that AUROC in this setup is hardly above random chance for every model family, so it cannot distinguish between methods, reflecting how leaving out an entire dataset is difficult. However, I question whether the difficulty stems from their methodological choices, i.e., the models they are using, more than the actual setup.

I think specialized venues might be better suited for this type of work. However, I also suggest the authors to present a novel contribution by showing, for example, how to address validation on a novel dataset. Other gaps in this area are probably some of the following:

  1. more data, especially diverse (the DiversityOne tried to tackle that, but it's still a limited dataset). In particular, more fine-grained information. Recently, a new dataset called PAAWS went into this direction, but with slightly different data types (accelerometers mostly). Now this is difficult, I know, but really needed.
  2. Develop models that actually tackle the "everything is simply not good enough" problem. For example, a foundation model trained directly on the data the authors gathered. The problem is that the proposed benchmark is not novel in itself, since the results are not that different from the individual datasets anyway.
  3. Not relying on handcrafted features only, which is what the authors do in their work. Indeed, now that I think about it, this is also a major issue: in a benchmark, I would expect to handle raw data directly, but the authors pre-compute features and then use models on top of those features
Questions for authors

I do not have specific questions to the authors -- again, the writing is very clear and the work well described.

UPDATE After the rebuttal, I increased the "informative" score from 1 to 2 ("I learned something interesting or useful") but I still feel the contribution is too marginal. However, the authors' claim that their Benchmark "provides a common evaluation foundation" is nice enough that I consider raising my score after all (from "1 strong reject" to "2 weak reject")

13 comments
AuthorsOpenReview ↗

Q1. Novelty beyond GLOBEM and existing benchmarks

Q1. Novelty beyond GLOBEM and existing benchmarks

the benchmarking does not meaningfully extend beyond existing setups. For example, most of the results are consistent with what can already be obtained using the GLOBEM datasets.

Thank you for recognizing the benchmark's scale, clarity, and technical soundness. Our contribution is not introducing cross-dataset evaluation, but testing whether model advantages persist across heterogeneous studies, targets, timescales, and learning paradigms. We address each concern separately; new evidence is in the revised appendix, with main-text integration by November 18 (AoE).

1. Study-design heterogeneity, not simply more datasets. GLOBEM's journal study describes similar collection models: university students, approximately ten-week studies in the same academic quarter, and consistent feature extraction. Its apps and some instruments differ; we fully acknowledge its cross-institute and cross-dataset evaluation. Our separately designed collections vary more broadly in duration, sensing, and target construction, although some share research groups or logger lineages.

Table 1. Comparison with GLOBEM, PACM IMWUT 6(4) (Xu et al., 2023).

DimensionGLOBEMMobile-Human-Bench
Organization/design4 institute-year datasets; 2 groups/universities; broadly coordinated protocols7 collections from multiple research programs; separately designed protocols
Participants618 participant-years; 534 unique people1,751 participant records, not necessarily unique people
Geography/durationUSA; about 10 weeks per dataset9 countries; about 1 week to several years
SensingTwo apps; similar core functionality; common feature extractionMultiple ecosystems; heterogeneous modality/feature coverage
TargetsDepressionAffect, mental health, sensor-derived behavior
TimescalesPrimarily weekly; end-term within datasetsMomentary, daily, weekly
SettingsWithin-dataset temporal, LODO, cross-institute/year, overlapping-userS1 temporal, S2 unseen-user, S3 cross-study across compatible tasks
Models9 depression algorithms + 8 DG + Clustering/Reorder = 1927 across Baseline, Deep Tabular, Tabular FM, DG, UDA
Tabular FMs/ICLNot evaluatedTabPFN, TabFM-Torch, TabICL, TabDPT
Transductive UDANo separate family; DANN evaluated as DGUnlabeled target features; matched source-only controls

Our characterization finds 14 sensing families, none in all seven resources (maximum 6/7); momentary stress HIGH prevalence ranges 32.5%–73.8%. Valence is 96.5% in DiversityOne, versus 77.8% EmoPhone/79.7% K-EmoPhone. This documents heterogeneity, not isolated causes of transfer loss (A.14/B.10/C.13).

2. What new questions does this coverage answer? Mean pooled AUROC on the shared affective benchmark:

Model familyS1S2S3
Baseline: trees, MLP, ResNet.6017.5247.5084
Deep Tabular.5457.5276.5101
Tabular FM.6398.5247.5015
DG—.5292.5106
UDA—.5331.5115

Do pretrained advantages transfer? Tabular FMs exceed Baseline by .0381 in pooled S1, winning 44/54 cells, but lose this advantage in S2/S3; family spreads are .0084/.0100. C.1 qualifies pooled S1 as within- and between-person discrimination, not pure within-person tracking.

Do shift-aware objectives consistently help? Of 24 matched DG/UDA comparisons, 0 show a Holm-corrected significant gain and 21 satisfy equivalence within ±.01. We extend the scope of this assessment, not refute GLOBEM's Reorder, which we did not test.

Does transfer always approach chance? Screen-on S3 is .667 momentary, .507 daily, .604 weekly. Different target construction/coverage and persistence qualify this contrast (C.3); weekly results cover two directions. The benchmark identifies conditional effectiveness, not a universal model ordering.

3. Other precedents and revisions. FEEL harmonizes 19 EDA/PPG emotion datasets; SIDA studies cross-subject sleep staging. These complement our behavioral/contextual mobile-sensing focus. We will clarify this positioning in Introduction/Related Work, distinguish GLOBEM's two resources and coordinated collection designs, and replace ambiguous binary coverage indicators. The sequence and adaptation evidence directly addressing modeling choices is in Q4/Q6/Q7.

AuthorsOpenReview ↗

Q2. Scientific value of leave-one-dataset-out evaluation

Q2. Scientific value of leave-one-dataset-out evaluation

the leave-one-dataset-out setting does not really provide more actionable insight than a traditional LOPO.

S2 and S3 address distinct deployment questions. S2 uses participant-grouped five-fold evaluation, rather than literal leave-one-person-out, but tests the same within-study unseen-person condition. S3 includes pairwise transfers and pooled leave-one-dataset-out transfers to a separately collected study.

DimensionS2: unseen participantsS3: unseen study
Test participantsUnseenUnseen
Collection protocolSame studySeparately designed study
SensingSame study ecosystemMay differ
LabelsSame study definitionsHarmonized across studies
FeaturesNative/shared; shared in matched analysesShared cross-study representation
ShiftPrimarily participant-levelCompound study-level heterogeneity
Deployment questionNew people under an existing protocolNew study with changed measurement/collection conditions

Shared groups/loggers do not make separate campaigns equivalent; different protocols do not establish statistical independence or isolate a causal shift.

1. Matched benchmark evidence. Across 364 matched affective configurations, pooled AUROC decreases .5278→.5086 from S2 to S3:

S3–S2 difference: −0.0192 AUROC (95% CI: [−0.0226, −0.0158]).

Target, evaluated dataset, timescale, feature-space condition, and model are matched. Training/evaluation distributions are not identical. The interval summarizes configurations that share participants/studies, not 364 independent study replications. The modest gap is not an isolated causal protocol effect or a universal decline; it shows that within-study and cross-study results are not interchangeable.

2. Participant-count controls. We compare LightGBM transfer with training on the same number of other participants drawn repeatedly from the target study, using identical held-out test rows.

Target: source → test studySource participantsTransferSame-study controlΔ
Stress: DeepStress → EmoPhone24.511.527−.016
Stress: K-EmoPhone → EmoPhone77.508.545−.037
Stress: StudentLife → EmoPhone35.524.533−.009
Valence: K-EmoPhone → DiversityOne77.487.517−.030
Valence: K-EmoPhone → EmoPhone77.525.554−.030

Differences use unrounded values. The gaps of .009–.037 AUROC are descriptive, not five statistically established penalties. Matching participant counts does not equalize training-row counts, response density, collection duration, or populations. This covers one model, five momentary directions, and two target studies. It shows that matching source population size alone does not necessarily eliminate the observed difference, not which remaining factor causes it.

3. Heterogeneity after harmonization.

DiagnosticEvidence and scope
Sensing availability14 families; none present in all 7 resources
Study-membership classificationAUROC .90–1.00 for examined momentary pairs
Same-study referenceAUROC .33–.61 for random participant halves of one study
Momentary stress prevalenceHIGH labels 32.5%–73.8%
Related logger schemaK-EmoPhone/EmoPhone remain separable after participant standardization: .95 stress, .92 valence, .96 arousal

A common schema enables comparison without erasing study differences. Separability does not prove causal independence, demographic representativeness, or that discriminating features cause transfer failure. These diagnostics cover five momentary resources, not all seven studies at every timescale (A.14/B.10/C.13).

4. Actionable implications. S3 tests whether same-study model evidence carries over to another collection pipeline, so deployment claims should report it separately from S2. Source composition and measurement conditions need documentation, and methods should be compared against matched controls under the intended deployment setting. A benchmark can expose a common limitation without producing a reliable leaderboard; Q3 quantifies the ranking uncertainty.

S3 is also outcome-dependent: screen-on averages .667 momentary/.604 weekly, though its different labels, narrower roster, and persistence require qualification (C.3). This is not a causal control for affective prediction.

Revisions. A.14/B.10/C.13 already document study differences. We will add the source-count controls to C.3 and expand the main setting motivation by November 18 (AoE). The conclusion will remain bounded: cross-study evaluation adds a deployment condition and measurable empirical differences, without identifying the individual causal contribution of each shift.

AuthorsOpenReview ↗

Q3. Near-chance performance and model discrimination

Q3. Near-chance performance and model discrimination

AUROC in this setup is hardly above random chance for every model family, so it cannot distinguish between methods, reflecting how leaving out an entire dataset is difficult.

We agree that affective S3 does not support a reliable fine-grained leaderboard. We quantify its limits and assess a separate target-label adaptation setting.

1. Zero-label model discrimination. Shared affective roster; model means use all cells per setting, not the 364 configurations matched across settings.

QuantityS1S2S3
Cells / models54 / 1454 / 2781 / 27
Family-mean AUROC range.5457–.6398.5247–.5331.5015–.5115
Model-mean AUROC range (spread).526–.673 (.147).512–.538 (.026).498–.515 (.017)
Spread 95% CI[.121, .174][.021, .034][.012, .027]
Median model-mean 95% CI width.033.014.012
Split-half rank ρ: 14 supervised models.96.80.64
Split-half rank ρ: all models—.75.41
Models below observed best (Holm-corrected)13/1310/264/26
Model-mean 95% CIs above chance (.5)14/1427/2721/27

CIs use 2,000 cell bootstraps; rank ρ uses 1,000 split-half draws. Observed-best comparisons use paired Wilcoxon tests with Holm correction. Overlapping tasks make these exploratory.

Under S3, model means span only .017 AUROC (95% CI [.012,.027]), close to median CI width .012. Rank reliability falls to ρ=.41; within-cell model SD (.020) is near seed SD (.012). Only 4/26 models (TabPFN, TabICL, LightGBM, TabNet) are below the observed best (MLP) after correction; this is not evidence of a stable winner. 21/27 model-mean CIs exceed chance, by at most .015.

2. What S3 still establishes. Across 24 matched DG/UDA method–setting comparisons, 0/24 show a Holm-corrected positive gain and 21/24 are equivalent within ±.01 AUROC. Thus, S3 supports matched effect and family-level conclusions, not exact leaderboard positions. The same protocol retains signal for screen-on (S3 AUROC .667 momentary, .604 weekly), although persistence contributes. Alternative representations are discussed in Q4.

3. Separate 50% target-support evaluation (B.1/C.8). Each held-out participant's first chronological half provides labeled support; source-only and adapted models are scored on the identical second half. 27 models × (70 S2 + 101 S3 configurations) = 4,617 pairs, one seed, main-run hyperparameters. Trees refit with source + support (weight 20); four FMs use frozen ICL; 20 neural/DG/UDA models fine-tune (AdamW, 10⁻⁴, ten epochs).

Table. Source-only versus adapted pooled AUROC on identical test rows. Differences use unrounded values; adaptation mechanisms differ by model family, so their group means are not causal ablations.

SettingAdaptation (models)PairsSource-onlyAdaptedΔAUROC
S3-PTree refitting (3)303.521.656+.135
S3-PICL (4)404.524.659+.135
S3-PGradient fine-tuning (20)2,020.515.585+.070
S3-PAll (27)2,727.517.604+.087
S2-PTree refitting (3)210.540.667+.127
S2-PICL (4)280.546.653+.107
S2-PGradient fine-tuning (20)1,400.542.612+.070
S2-PAll (27)1,890.542.624+.082

4. Model discrimination on the same test halves. Across the 27 model means:

QuantityS3-PS2-P
Source-only pooled AUROC range (spread).506–.533 (.027).530–.553 (.023)
Source-only split-half rank ρ.40.23
Adapted pooled AUROC range (spread).521–.682 (.161).549–.690 (.141)
Adapted split-half rank ρ.91.92

In S3-P, spread increases .027→.161, and split-half ρ .40→.91. This supports comparing target-supervised adaptation, not a claim that zero-label S3 has become discriminative. Reliability concerns splits of these tasks, not new studies or seeds. A 50% budget does not establish few-shot label efficiency.

Revisions. B.1/C.8 already document personalization. We will add uncertainty/rank-reliability diagnostics to C.9 and qualify Section 4.3 by November 18 (AoE): zero-label S3 supports matched comparisons, while target-supervised adaptation is a separate condition.

Reviewer_ruDeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Mobile-Human-Bench, which is a unified benchmark for evaluating generalization across time, users, and independently collected studies. Existing benchmarks do not span independently collected behavioral and contextual mobile sensing studies across multiple targets, prediction time scales, and generalization settings. This paper proposes a unified benchmark for behavioral and contextual mobile sensing to bridge this gap. Empirical evaluations have been made across diverse baseline models to show performance under distribution shift.

Critical strengths

The paper addresses a practical problem by aggregate sparse mobile sensing datasets to evaluate generalization across temporal, users, and independent studies.

The benchmark harmonizes seven longitudinal studies spanning 1,751 participants across nine countries and yields 295,162 labeled feature rows after benchmark construction.

Critical weaknesses

The paper has limited novelty in the main finding because decline from within-user to cross-user to cross-dataset performance has already been reported.

For the affective targets, S3 performance is close to random, but the paper doesn't analyze why? Computing pooled AUROC across users in S1 will cause a collapse by learning each person’s label base rate instead of true within-person sensing dynamics.

Questions for authors

Several targets are highly imbalanced (e.g., DiversityOne valence is ~97% positive), can you add more matrix like balanced accuracy?

Low S2/S3 scores (0.509–0.528 AUROC) could motivate stronger future methods, but how can you tell this ~0.02 model spread reflects room for algorithmic improvement rather than an intrinsic information ceiling?

5 comments
AuthorsOpenReview ↗

Q1. Novelty

Q1. Novelty

The paper has limited novelty in the main finding because decline from within-user to cross-user to cross-dataset performance has already been reported.

We agree that this decline is established, including in GLOBEM. Our contribution is testing whether findings about model effectiveness persist across studies, targets, timescales, and modeling paradigms—not introducing sensing generalization as a new problem.

Cross-study scope. We harmonize seven collections from multiple research programs. Some share research groups or logging infrastructure, but differ in collection and measurement protocols. The additional characterization quantifies these differences:

DimensionEvidence
Coverage7 studies; 1,751 participant records; 9 countries
Logging ecosystemsStudentLife, ABC Logger, iLog, AWARE-based and study-specific applications
Sensing14 families; maximum coverage 6/7 studies
Momentary stress HIGH prevalenceDeepStress 32.5%; StudentLife 73.8%
Valence HIGH prevalenceDiversityOne 96.5%; EmoPhone 77.8%; K-EmoPhone 79.7%
PHQ-4 threshold prevalenceCES 11.4%; GLOBEM 17.5% of weeks

Participant records are not necessarily unique individuals. These statistics characterize heterogeneity, not its causal contribution to transfer loss.

Timescale and target dependence. Mean pooled AUROC:

Target / timescaleMatched nS1S2S3
Affective, overall3640.5930.5280.509
Affective, momentary1260.6310.5430.515
Affective, daily1260.5850.5320.510
Affective, weekly1120.5590.5060.499
Screen-on, momentary—0.7210.7150.667
Screen-on, daily—0.5760.5630.507
Screen-on, weekly—0.6720.6270.604

Affective configurations are matched across settings. Screen-on uses the common 14-model roster but different coverage and target construction; weekly S3 has only two directions. The affective decline recurs at all three timescales, whereas behavioral transfer is outcome-dependent. Persistence contributes to screen-on predictability (Appendix C.3); this contrast does not establish that label subjectivity causes affective transfer failure.

Do foundation-model advantages transfer? Family means on the shared affective benchmark:

FamilyS1S2S3
Baseline: trees and generic neural networks0.60170.52470.5084
Deep tabular0.54570.52760.5101
Tabular foundation models0.63980.52470.5015
DG—0.52920.5106
UDA—0.53310.5115

Tabular FMs, evaluated through in-context inference, exceed the baseline family by 0.0381 in pooled S1 and win 44/54 cells. This advantage disappears in S2/S3: family spreads are 0.0084 and 0.0100. Thus, a model family's advantage under pooled within-user evaluation does not imply stronger unseen-user or cross-study performance. Q3 clarifies the interpretation of pooled S1 scoring.

Do shift-aware objectives consistently help? We compare DG against ERM and UDA against a source-only MLP, using matched backbones, splits, preprocessing, and training budgets.

Finding across 24 method–setting comparisonsCount
Significant positive gains, Holm-corrected0/24
Practical equivalence within ±0.01 AUROC21/24
Significant decreases, Holm-corrected3/24

Categories overlap: DANN-S3 is both significantly negative and practically equivalent; VREx-S2 meets neither criterion. Decreases occur for IRM-S2, JAN-S2, and DANN-S3. This extends, rather than overturns, GLOBEM's limited gains from standard DG methods; Reorder is not in our roster.

Representation sensitivity (C.12). We tested 38 sequence models, including temporal networks, GRU-based DG/UDA, and pretrained time-series models, on ordered bins of the same 60-minute pre-prompt window. Comparisons use matched cells and test rows with 5-minute bins; ΔAUROC is sequence minus tabular.

SettingSequence vs. tabularCellsΔAUROCHolm p
S1SeFT vs. TabFM-Torch14−0.10130.0013
S2GRU-D vs. TabTransformer14−0.00541.000
S3BRITS vs. TabTransformer17−0.00271.000

Sequence modeling provides no significant S2/S3 advantage here, while the selected tabular model leads in S1. Thus, the evaluated generalization limitations persist beyond tabular-only modeling.

Best models were selected on these cells, so tests are exploratory. Binned features retain fewer channels than tabular inputs and are not raw signals: this is not a pure temporal-order ablation and does not rule out gains from other representations.

Revisions. Dataset diagnostics are in A.14/B.10/C.13; sequence inputs, methods, and results are in A.15/B.9/C.12. Main-text revisions will emphasize these conditional findings rather than the degradation pattern alone.

AuthorsOpenReview ↗

Q2. Understanding near-chance cross-dataset performance

Q2. Understanding near-chance cross-dataset performance

For the affective targets, S3 performance is close to random, but the paper doesn't analyze why?

We agree that the original explanation was insufficient. Our additional analyses distinguish two observations: unseen-user prediction is already weak within a study, and transfer to another study adds an uneven loss. The diagnostics below cover the momentary track and its five studies.

Table 1. Same-study unseen-user prediction versus single-source cross-study transfer. Shared features; mean pooled AUROC over 27 models. S3 averages the single-source directions into each test study.

TargetTest studyS2S3S2 − S3
StressEmoPhone0.5530.516+0.037
StressK-EmoPhone0.5160.507+0.009
StressDeepStress0.5410.523+0.018
StressStudentLife0.5340.513+0.021
ValenceEmoPhone0.5500.505+0.045
ValenceK-EmoPhone0.5210.502+0.020
ValenceDiversityOne0.5460.506+0.040
ArousalEmoPhone0.5710.537+0.035
ArousalK-EmoPhone0.5570.546+0.011
Mean0.5430.517+0.026

S2 is approximately 0.52–0.57, leaving relatively little performance above chance before study-level shift. The additional loss is larger for EmoPhone and DiversityOne, approximately 0.035–0.045, than for the three smaller test studies (0.009–0.021). It is therefore not a uniform collapse across every source–target configuration.

Table 2. Cross-study transfer versus a participant-count-matched same-study source. LightGBM; repeated random participant draws for the same-study control.

Target: source → test studySource participantsTransferSame-size sourceDifference
Stress: K-EmoPhone → EmoPhone770.5080.545−0.037
Stress: StudentLife → EmoPhone350.5240.533−0.009
Stress: DeepStress → EmoPhone240.5110.527−0.016
Valence: K-EmoPhone → EmoPhone770.5250.554−0.030
Valence: K-EmoPhone → DiversityOne770.4870.517−0.030

Each control trains on the same number of other participants from the target study and uses the same held-out target test rows. Differences use unrounded scores. Matching participant counts does not equalize training-row counts, collection duration, or label density.

Cross-study transfer is 0.009–0.037 AUROC below the control mean. These are descriptive robustness comparisons, not five independently significant transfer penalties. They show that matching source participant counts alone does not necessarily eliminate the observed difference, without isolating why it occurs.

Additional characterization already included in the appendix:

  • Sensing and features (A.14/C.13). None of 14 sensing families is available in all seven studies. On the examined momentary pairs, dataset-membership classification reaches AUROC 0.90–1.00, compared with 0.33–0.61 for random participant halves of one study. Distinguishability persists after within-participant standardization.
  • Label distributions (A.5). HIGH-label prevalence differs by target and study; momentary stress ranges from 32.5% to 73.8%. This documents a difference, not a sufficient explanation of transfer error.
  • Label–feature relationships (B.10/C.13). Permutation tests detect differences for the two valence pairs involving DiversityOne: p=0.002, Holm-adjusted p=0.02 over ten pairs. Remaining comparisons are not significant after correction; limited power for smaller studies prevents interpreting nondetection as equivalence.
  • Representation sensitivity (C.12). In the exploratory best-model comparisons, sequence-minus-tabular differences are −0.0054 in S2 and −0.0027 in S3, both Holm p=1.000. Near-chance transfer therefore also occurs with the evaluated binned-feature sequences, not only tabular models.

Scope. These analyses do not causally apportion the loss among sensing, population, label, and source-size differences. The participant-count controls use one model and two target studies; the separability and relationship tests exclude daily/weekly tracks, GLOBEM, and CES. Nor do the results establish an intrinsic information ceiling.

Revisions. A.14, B.10, and C.13 already contain the dataset characterization; C.12 contains the sequence evidence. We will add Tables 1–2 above to C.3 and integrate a short, bounded explanation into Section 4.3 by November 18 (AoE).

AuthorsOpenReview ↗

Q3. Pooled AUROC and evaluation protocol

Q3. Pooled AUROC and evaluation protocol

Computing pooled AUROC across users in S1 will cause a collapse by learning each person’s label base rate instead of true within-person sensing dynamics.

We agree that pooled S1 AUROC combines within- and between-person discrimination. We retain it for benchmark coverage and add participant-level sensitivity analysis.

1. Computation. Models in each cell share splits, features, labels, preprocessing, and test rows. S1 pools the final 20% of each participant's chronological 60/20/20 split (momentary histories capped at 30 days). S2 pools held-out participants within each of five participant-disjoint folds and then averages fold scores. S3 pools all test rows of the held-out study. Seeds are scored separately and averaged before folds. Reportable splits require both classes and ≥30 test rows; <200 is flagged as underpowered. Participant identifiers are not model inputs.

2. Participant-level coverage. To assess within-person discrimination, we additionally compute AUROC for each participant with ≥20 test rows containing both classes, then take a test-row-weighted mean over eligible participants.

TimescaleCells with participant-level AUROC
Momentary18
Daily1
Weekly11
Total30/54 (55.6%)

For 24/54 cells, no participant qualifies: n/a means undefined, not an unfinished experiment. Daily datasets other than CES and weekly GLOBEM/K-EmoPhone lack sufficient test observations. Even at the momentary timescale, only 5/74 K-EmoPhone and 1/32 StudentLife participants qualify in the reported examples. Using participant-level AUROC as the primary metric would exclude whole task/timescale combinations and change the evaluated populations.

3. Sensitivity results. The best model is selected independently in each column.

Target / dataset / featuresBest pooled AUROC (model)Best participant AUROC (model)
Stress / DeepStress / native0.776 (TabFM-Torch)0.599 (RF)
Stress / K-EmoPhone / shared0.711 (TabFM-Torch)0.679 (RF)
Valence / DiversityOne / shared0.850 (TabFM-Torch)0.615 (TabFM-Torch)
Valence / EmoPhone / native0.729 (TabFM-Torch)0.558 (TabFM-Torch)
Valence / K-EmoPhone / native0.686 (TabFM-Torch)0.702 (SAINT)
Arousal / EmoPhone / shared0.745 (TabFM-Torch)0.586 (TabFM-Torch)

Performance estimates and model rankings can change with the metric. On DiversityOne valence (shared features), the same TabFM-Torch model scores 0.850 pooled versus 0.615 participant-level AUROC. On DeepStress stress, TabFM-Torch leads pooled evaluation while RF leads participant-level evaluation. However, participant-level AUROC uses only eligible participants, and the metrics weight different observation pairs. Their difference does not directly decompose within- and between-person effects; participant-level AUROC is not universally lower.

4. Interpretation across settings.

SettingParticipants seen in training?Their labels available?Pooled AUROC reflects
S1YesEarlier training labelsWithin- and between-person discrimination
S2NoNoDiscrimination among observations of unseen participants
S3No, nor their studyNoAs in S2, under study-level shift

In S1, models may exploit stable participant-specific differences in label prevalence. Removing identifiers and standardizing each participant using their training slice limits feature offsets but cannot remove all person-specific information (e.g., sensing availability or missingness). Participants with more test rows also carry more weight. S1→S2 must therefore not be interpreted solely as lost within-person predictability.

In S2/S3, held-out participants have no labels available for fitting or selection, so their base rates are not directly learned from labels. Pooled AUROC nevertheless still includes between-person observation pairs, potentially ranked using sensing features. The near-chance S3 and matched DG/UDA conclusions are comparisons within participant-disjoint settings; they do not depend on interpreting S1 as purely within-person prediction.

Revisions. Appendix C.1 already documents pooled S1 scoring, participant-level coverage, and this sensitivity analysis. By November 18 (AoE), we will clarify aggregation in Section 4.1, state the limitation in Section 4.5, and qualify the foundation-model advantage in the Abstract, Section 4.3, and Conclusion as a pooled-S1 finding.

In summary, participant-level sensitivity confirms that pooled S1 AUROC does not isolate within-person dynamics. We retain pooled AUROC as the primary metric because it is consistently defined across the benchmark, whereas participant-level AUROC is unavailable for 24/54 S1 cells under our eligibility criterion.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Author update: Revised manuscript and additional analyses (all reviewers)

We thank all reviewers for their feedback. Our revised manuscript includes new experiments and appendix analyses. Individual replies address each concern; we will integrate the main findings into the main text by November 18 (AoE).

1. Sequence modeling beyond aggregated tabular inputs (A.15/B.9/C.12). We constructed 5- and 3-minute sequences within the same 60-minute pre-prompt window, preserving missingness masks, labels, splits, and test rows. We evaluated 38 sequence models in five groups (3,200 completed model–cell runs):

GroupModelsMethods evaluated
Temporal architectures14GRU, LSTM, GRU-D, BRITS, SeFT, TCN, ResNet1D, InceptionTime, MLSTM-FCN, TST, iTransformer, ConvTran, S4D, xLSTM
Sequence DG8ERM, IRM, VREx, GroupDRO, CSD, MixStyle, Fish, Diversify; GRU-based objectives and controls
Sequence UDA8DANN, CDAN, DAN, JAN, CORAL, MCC, CoTMix, RAINCOAT; GRU-based objectives and controls
Pretrained time-series encoders / ICL6Chronos-2, Chronos-Bolt, TiRex, Toto-2, TSPulse, TimEE
Pretrained forecasting models2TabPFN-TS, TimesFM-3

With 5-minute bins, matched sequence-minus-tabular mean AUROC is −0.081 (S1; 14 cells, Holm p=0.001), −0.012 (S2; 14, p=0.021), and −0.004 (S3; 17, p=1.00). Results are similar with 3-minute bins. GRU-based DG/UDA objectives also show no significant positive gain over matched controls. Thus, the tested sequence learners do not recover cross-study affective performance. These sequences contain fewer engineered channels than the tabular inputs; they are neither raw signals nor a pure temporal-order ablation; we do not claim a fundamental information ceiling.

2. Target-support adaptation across 27 models (B.1/C.8). We added S2-P/S3-P, in which the first 50% of each held-out participant's chronologically ordered labeled observations provide support and both conditions are scored on the same remaining 50%. Across 171 configurations (70 S2 + 101 S3; 4,617 model–configuration pairs):

SettingSource onlyAdaptedΔAUROC
S2-P, all 27 models0.5420.624+0.082
S3-P, all 27 models0.5170.604+0.087

Mechanisms include tree refitting, frozen-FM in-context learning, and neural/DG/UDA fine-tuning. These results require labeled target support and do not establish that zero-target-label S2/S3 has become easy or that a small label budget suffices.

3. Controlled foundation-model fine-tuning (B.8/C.11). In a separate momentary stress experiment (4 S2 cells; 16 S3 directions), we compare frozen TabFM-Torch against source-training-only LoRA adaptation, with identical inference contexts and target-support/test observations. Adding 50% target support to frozen ICL improves pooled AUROC by +0.176 (S2) and +0.217 (S3). LoRA then changes these results by −0.007 and −0.011, respectively. This one-model, one-target, one-seed analysis tests source-domain weight updates, not pretraining a new model or fine-tuning on target-support labels.

4. Dataset and label diagnostics (A.14/B.10/C.13; A.4–A.5). We added protocol/logger provenance for seven studies (1,751 participant records; nine countries), sensing inventories, feature-distribution comparisons, and label–feature analyses. None of 14 sensing families is present in all seven studies. A dataset-membership classifier distinguishes the examined momentary study pairs from shared features with AUROC 0.90–1.00; tests detect label–feature relationship differences for the two valence pairs involving DiversityOne (Holm-adjusted p=0.02). These observations demonstrate heterogeneity, not causal attribution or population representativeness. We document label instruments, recoding and midpoint rules. Participant-count-matched LightGBM controls are presented in the individual replies.

5. Metric sensitivity and limitations (A.5/C.1). We added validation-thresholded balanced accuracy for S1 to address class imbalance (13/31 dataset–target–timescale combinations fall outside 30–70% HIGH); the model-mean ranking is nearly unchanged from AUROC (Spearman ρ=0.996). We also clarify the limits of student-heavy cohorts, shared loggers, phone-only cross-study features, and near-chance S3 ranking.

Revision status. These appendix additions are already included in the revised PDF. Main-text integration is ongoing. We welcome questions on the protocols or results.

Dear Reviewers and Area Chair,

We have uploaded our revised manuscript, incorporating the key findings from the additional experiments and analyses into the main text.

The revisions strengthen the presentation of our cross-study benchmark contribution, sequence-modeling evaluation, target-user personalization, and foundation-model adaptation results. We have also expanded the discussion of benchmark utility, dataset heterogeneity, and limitations.

We sincerely appreciate your feedback and consideration throughout the discussion.

Best,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.