Let the Query Lead: Factorized Reference Conditioning for Wearable Recognition

Wearable recognition models must generalize to unseen wearers whose sensor patterns can differ from those observed during training. Unlabeled recordings from a new wearer can provide useful context, but their statistics mix persistent wearer- or acquisition-dependent variation with the activities or states that happen to be recorded. Such context can be informative without specifying how an individual prediction should change. We introduce Query–Subject Factorization (QSF), a query-led interface for reference-conditioned wearable recognition. Starting from a frozen population recognizer, QSF learns a query-only correction that activates shared correction modes. References supply gains that act only through these query-dependent activations. Activity-relative residual aggregation with bounded per-class evidence yields a compact reference state that is reusable across queries and requires no target labels or gradient updates at deployment. The interface exactly recovers the reference-free predictor when references are absent and provides an explicit bound tying reference-induced changes to query activation strength. Across eight activity, sleep, and affect recognition datasets, the complete system ranks first in accuracy on all six activity-recognition tasks and among the top three on every task. With a common frozen upstream predictor, QSF achieves higher mean weighted F1 than a compact joint-conditioning baseline on all eight tasks without a larger conditioning head. Targeted decision-level analyses link this advantage to fewer harmful changes to correct query-only predictions, while controlled activation interventions show that the query-activated mode mixture affects recognition. These findings support treating reference context as a modulator of query-led corrections rather than an independent source of prediction changes.