acceptodds
Under review as a conference paper at ICLR 2027

Let the Query Lead: Factorized Reference Conditioning for Wearable Recognition

Abstract

Wearable recognition models must generalize to unseen wearers whose sensor patterns can differ from those observed during training. Unlabeled recordings from a new wearer can provide useful context, but their statistics mix persistent wearer- or acquisition-dependent variation with the activities or states that happen to be recorded. Such context can be informative without specifying how an individual prediction should change. We introduce Query–Subject Factorization (QSF), a query-led interface for reference-conditioned wearable recognition. Starting from a frozen population recognizer, QSF learns a query-only correction that activates shared correction modes. References supply gains that act only through these query-dependent activations. Activity-relative residual aggregation with bounded per-class evidence yields a compact reference state that is reusable across queries and requires no target labels or gradient updates at deployment. The interface exactly recovers the reference-free predictor when references are absent and provides an explicit bound tying reference-induced changes to query activation strength. Across eight activity, sleep, and affect recognition datasets, the complete system ranks first in accuracy on all six activity-recognition tasks and among the top three on every task. With a common frozen upstream predictor, QSF achieves higher mean weighted F1 than a compact joint-conditioning baseline on all eight tasks without a larger conditioning head. Targeted decision-level analyses link this advantage to fewer harmful changes to correct query-only predictions, while controlled activation interventions show that the query-activated mode mixture affects recognition. These findings support treating reference context as a modulator of query-led corrections rather than an independent source of prediction changes.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Wearable Recognition,Reference Conditioning,Query--Subject Factorization,Cross-User Generalization, Personalization

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_vXyJOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how unlabelled recordings from a new wearer can improve wearable recognition. Specifically, the paper suggests that such reference context mixes wearer characteristics with the activities recorded, so on its own it does not tell the model how a prediction should change. They propose Query–Subject Factorization (QSF), where a query-only correction on a frozen population recognizer activates shared correction modes, and a compact reference state supplies gains that act only through those activations. This needs no target labels and no gradient updates at deployment. Across eight activity, sleep, and affect datasets, the full system ranks first on all six activity tasks and beats a joint-conditioning baseline on all eight. The authors conclude that reference context should modulate query-led corrections rather than change predictions on its own.

Critical strengths
  1. The authors put real effort into analysing the method, and the paper includes a wide set of controls and experiments.
  2. The method looks efficient, adding little inference cost and needing no target labels or gradient updates at deployment.
Critical weaknesses
  1. It is unclear how much QSF contributes to the Table 1 ranking. Table 1 compares the authors’ own backbone and training recipe plus QSF against other backbones trained without it, so the effect of the conditioning cannot be separated from the effect of the backbone. Reporting each baseline (for example CNN, Residual, and InvConvNet) with and without QSF under the same training recipe would isolate what QSF actually adds.
  2. The gain from QSF looks small. In Table 2, adding QSF to InvConvNet and DeepConvLSTM changes accuracy by only 0.00 to 0.14 percentage points, with standard deviations often larger than the mean, and in Table 11 QSF does slightly worse than DIRECT on HAPT and MotionSense. On this evidence, the conditioning seems to offer little over using no references at all.
  3. The experimental design is hard to follow. The paper uses several evaluation protocols, but how they connect to the main claim about QSF is not made clear, and it is often hard to tell which experiment supports which conclusion. Stating the main hypothesis and the experiment that tests it would help a lot.
Questions for authors

See Weakness

Reviewer_DhyHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Query–Subject Factorization (QSF) for wearable recognition of unseen wearers. A query-only correction activates shared modes, while unlabeled references modulate their gains. The interface bounds reference influence and recovers DIRECT when references are absent. Experiments cover eight datasets, with matched conditioning controls and decision-level analyses.

Critical strengths
  1. Explicit control of reference influence. References affect predictions through query-activated modes, with exact recovery of DIRECT and explicit sensitivity bounds. Reference inference requires no target labels or deployment-time parameter updates.
  2. Informative comparisons and diagnostics. QSF exceeds compact Joint in mean W-F1 across eight tasks. Capacity controls, objective ablations, and repair–harm analyses provide useful evidence that the interface reduces harmful prediction changes.
Critical weaknesses

The applicability of correction-based gating under cross-user shifts remains insufficiently characterized: QSF ties reference influence to query activations derived from a source-trained correction, thereby restricting reference-induced adjustments when these activations are weak. Figure 5(a) and Appendix C.10 verify this sensitivity relationship, while the activation interventions demonstrate that the mode mixture affects recognition. However, these analyses do not directly establish whether the restriction aligns with the adjustments needed by unseen wearers under distribution shift. In particular, the errors and repairability of low-activation queries remain insufficiently characterized, leaving unclear when the gating mechanism is beneficial and when it may be overly conservative.

Questions for authors

Could you extend the activation-bin analysis in Figure 5(a) to report DIRECT’s error rate, the fraction of DIRECT errors repaired by QSF, and the fraction of correct DIRECT predictions harmed by QSF? Could a comparison with Logit on the same query groups help clarify under what conditions source-trained correction activations provide an appropriate basis for reference modulation?

Reviewer_av9FOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The model is highly unclear, with many things left undefined and very little explanation provided. I’m going to start by going through my understanding of the model, so the authors can correct any misunderstandings I have made. As best as I can tell:

Stage I

  • You pretrain a population detector. In the paper, this claims to have an output h, z_0 where h \in \mathbb{R}^D (D unspecified) and z_0 \in \mathbb{R}^K(K=number of classes). I assume that h is feature projection layer and z_0 the logits (with D=256), but that is never stated.
  • You manually convert h and z_0 into some derived vector q.
  • This is then passed through some function rθr_\theta to get a correction d(x)d(x), which is never specified by name but which I assume is the “LayerNorm, a linear layer, GELU, dropout, and a K-dimensional output layer” described in B.2. This is trained to predict the class (via a correction to z0z_0. It’s not at all clear to me why you take a trained predictor of class, and use it to train another predictor of class, presumably using the same data? Particularly since there doesn’t seem to be any statistically significant advantage to doing so from Table 11?

Stage II

  • We get a user-specific standardized state sus_u and attenuation coefficient ρu\rho_u
  • We then learn a function mu(x)m_u(x) which is a function of the correction d(x)d(x), the attenuation ρu\rho_u, and the user-specific state sus_u. It’s not clear why this would depend on the correction rather than x, making me think I have missed what granularity the correction is applied at. This function is added to the logits from stage I

Calculating the user-specific state

  • We start off with a prototype μk\mu_k and a fixed residual dictionary Dk\mathbf{D}_k for each class. μk\mu_k is just the mean embedding for that class, and Dk\mathbf{D}_k (loosely) captures the directions of per-user variability around that mean (calculated via SVD)
  • We use the stage-I predictors (with some sharpening) to get a per-segment/class score wm,kw_{m,k}
  • We get a user-specific vector sus_u that models the residuals (in the space spanned by Dk\mathbf{D}_k) of the user’s representation vs the class prototypes

In short - a lot of stages, a lot of unexplained design decisions, a lot to sift through.

At this point, I am interested to see whether all these stages and design decisions actually matter. But, while there are a lot of experiments, there is very little of the ablation that I actually want to see: Do all these steps actually improve performance?4.3 shows that the authors’ method outperforms the same method with a different way of incorporating personalization vectors, which would be a reasonable thing to explore once you have shown that any personalization method outperforms the population method. But this initial question is never directly considered, and the indirect evidence suggests that the personalization methods considered do not improve significantly on the population model.

Several tables do compare the “Direct” model (either with the authors’ population model, or a different backbone model) with the full QSF approach (using the corresponding backbone), but these typically show no significant benefit to QSF. In Table 2 (comparing other backbones with direct, to other backbones with QSF), most confidence intervals include zero. Table 11 (comparing different stages with the author’s population model) shows that nothing improves significantly over the population model.

While there are results in Table 1 showing good performance against various baselines, I believe the results in Tables 2 and 11 suggest that this performance is entirely, or almost entirely, due to the population model.

Overall, I found this paper dense to read, and lacking intuition, and in the end the personalization approaches do not seem to give any meaningful improvement over a population level classifier.

Critical strengths
  • Method outperforms several baselines
Critical weaknesses
  • Paper is very hard to parse
  • Little to no intuition or explanation of design decisions
  • Complex multi-stage methodology that does not seem to improve upon the first stage
Questions for authors

Can you better explain your design decisions, and why such a complex approach is necessary when the population-level baseline seems to work well in all scenarios tested?

Reviewer_Z6GzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes Query–Subject Factorization for reference-conditioned wearable recognition. A frozen population model is augmented with a source-trained query-only correction, and an unlabeled same-wearer reference state modulates shared modes of that correction.The complete system leads six activity-recognition datasets and ranks in the top three on the two remaining sleep/affect datasets.

Critical strengths

Substantial controlled analysis. Shared upstream predictors, five-seed paired differences, widened heads, loss controls and temporal/reference interventions make the narrower comparison against joint conditioning credible. The negative or mixed findings are disclosed.

Critical weaknesses
  1. Overly complicated presentation and insufficient clarity. The paper addresses a relatively straightforward problem: leveraging unlabeled historical sensor data from the same user to improve recognition through personalization. However, the presentation is unnecessarily complex, making it difficult to identify the core intuition and technical contributions. The extensive mathematical formulation and numerous design components obscure the central idea of query-guided reference conditioning. A clearer explanation of the motivation, the overall framework, and why each component is necessary would substantially improve the paper's readability and accessibility.

  2. Marginal improvements and limited evidence of effective personalization. Although the authors conduct extensive experiments, many of the reported improvements are relatively small, raising questions about the practical significance of the proposed method. More importantly, the results do not consistently demonstrate that incorporating historical reference data improves performance over the reference-free DIRECT baseline. For instance, in the matched comparisons (Table 11), QSF performs slightly worse than DIRECT on HAPT and MotionSense, while achieving only a modest improvement on WESAD. Much of the experimental evidence therefore demonstrates that QSF is less harmful than alternative conditioning methods, rather than establishing a clear benefit from personalization itself. Stronger evidence of consistent and meaningful gains over the reference-free baseline would be needed to justify the additional complexity of the proposed framework.

Questions for authors

Showing more statistically significant improvements over the baselines.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.