acceptodds
Under review as a conference paper at ICLR 2027

More Correct Labels Can Hurt: Delayed Feedback in Streaming Activity Recognition

Abstract

In streaming activity recognition, a requested label may arrive after the activity has changed. Unsupervised test-time adaptation updates batch statistics (BN), minimizes entropy (Tent), or filters predictions (OATTA), while delayed replay (IWMS) learns from historical labels. These approaches leave open how a returned label should enter the current activity state and influence subsequent predictions. By analyzing errors at label arrival, we find that directly injecting a correct historical label can overwrite a correct current prediction. Local loss analysis further shows that even a useful correction can become harmful when applied too strongly. We therefore propose Gated Arrival, which separates historical learning, state injection, and output mixing. A calibrator learns from returned labels and their saved origin predictions. Two gates fitted offline use label age and current predictions to set the weight of state injection and its later contribution to the output, with the recognition backbone frozen. Experiments on five activity datasets under simulated response delays show that additional correct labels can reduce recognition and that delay can reverse feedback-rule rankings. Gated Arrival reduces errors caused by labels that no longer match the current activity. Component comparisons show how historical calibration and state correction affect recognition and probability estimates.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Delayed feedback, Streaming activity recognition, Online adaptation, Temporal state correction, Probability calibration

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_StisOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces Gated Arrival, a method for handling delayed labels in streaming activity recognition. The main motivation is that a correct historical label may no longer represent the current activity and could negatively affect the model's predictions. To address this, the authors separate historical calibration from current-state correction. They use returned labels and their saved original predictions to update a lightweight calibrator while keeping the backbone frozen. They also introduce two gates: an injection gate that controls how much a delayed label influences the current state based on factors such as label age and current evidence, and an output gate that controls the influence of the temporal state on subsequent predictions.

The authors evaluate their method on five activity recognition datasets under different feedback delays and label budgets. Their experiments show that receiving more correct labels does not always improve performance, and that the effectiveness of feedback methods depends on the delay. They also provide component comparisons to study the effects of historical calibration and state correction.

Critical strengths
  1. The proposed Gated Arrival method improves recognition performance over the Base model across different feedback delay settings.
  2. The method does not require updating the pretrained recognition backbone during test-time adaptation. Instead, it uses a lightweight historical calibrator with only 2K trainable parameters, while the two gates are fitted offline.
  3. The paper addresses an interesting problem in streaming activity recognition: a correct historical label may negatively affect current predictions when it arrives after the activity has changed.
  4. The authors evaluate their method on five activity recognition datasets with different feedback delays and label budgets. The experiments also include component comparisons to investigate the effects of historical calibration and state correction.
Critical weaknesses
  1. I found the paper difficult to follow, especially the figures. For example, Figure 3 is supposed to explain the main idea of the proposed method, but I found it confusing. The authors should provide a simpler illustration or a step-by-step example to help readers understand how the historical calibrator and the two gates work together.
  2. Although Gated Arrival improves over the Base model, Table 2 shows almost identical Macro-F1 performance to the No history (refit) variant. At fixed delays of 30 and 120, the improvement is only 0.02 percentage points. This raises questions about whether the additional complexity of the full method is justified. While the historical calibration component improves NLL, its practical importance for activity recognition needs clearer discussion.
  3. Table 3 shows that state injection provides substantial improvements on CAPTURE-24 but can reduce performance on UCI HAR and PAMAP2 under certain delays. I would like the authors to explain more clearly why these differences occur and under which conditions each component is expected to help.
  4. The method uses dataset-specific development information for some components, such as the transition matrix and class prior. It is unclear how well the approach would generalize to a new dataset or activity distribution without this information. Additional cross-dataset experiments or discussion would strengthen the paper.
  5. In Equation (3), the authors combine the original and calibrated predictions using fixed weights of 1/4 and 3/4. I would like to understand why these weights were selected and whether the performance is sensitive to this choice.
Questions for authors
  1. Although the gates are trained using other datasets, how well do they generalize to datasets with substantially different activity durations, transition frequencies, and prediction error patterns? Could the authors provide further analysis of the gates' robustness under these changes?
  2. The injection gate uses 12 features, while the output gate uses 14. Although the paper includes a comparison with a simpler age/persistence gate, could the authors provide a more detailed feature ablation to show which additional features contribute most to performance?
  3. I am also curious whether the authors have considered comparing Gated Arrival with a recurrent model, such as an LSTM or GRU, that explicitly receives delayed feedback. Such a comparison could help clarify whether the proposed transition filter and two-gate design offer advantages over a learned temporal model.
2 comments
AuthorsOpenReview ↗

Component Evidence, Generalization, and Presentation

Thank you for identifying the presentation problem and for asking how much the historical and state-correction components contribute in practice. We agree that the figures and the interpretation should be more explicit.

Figure 3 and a step-by-step example. We will redraw the method as a four-step timeline: (1) at time t0, the frozen backbone emits p0 and a label is requested; (2) when y0 returns at t1, the calibrator learns from (p0, y0); (3) the injection gate controls how much the historical label updates the current state using its age and current evidence; and (4) the output gate controls how much that state affects the current prediction. We will also reorganize Fig. 6 by historical learning, state injection, reset, and output mixing, with a concise definition for each variant.

No history (refit) and practical Macro-F1 contribution. We agree that the aggregate Macro-F1 difference is nearly zero: Full exceeds No history (refit) by +0.024 pp at fixed d=30 and +0.016 pp at d=120. We will not present the calibrator's probability-quality improvement as a recognition-accuracy gain. We will explain that its value is for downstream use of probabilities, while its independent aggregate Macro-F1 contribution is negligible in these conditions.

Why injection helps on some datasets and hurts on others. Table 3 shows that removing injection from CAPTURE-24 reduces Macro-F1 by 10.63/9.10/5.46/2.59 pp at d=0/30/120/300, whereas removing it on UCI HAR improves Macro-F1 by 0.73 pp at d=30. Table 22 offers a descriptive alignment clue at d=120: CAPTURE-24's historical labels match the current activity 94.62% of the time while Base current accuracy is 78.77%; for HARTH the corresponding values are 88.21% and 94.14%. We will present this as a dataset-dependent alignment pattern, not a causal explanation or universal rule.

Cross-dataset generalization. Gate coefficients are evaluated with leave-one-dataset-out fitting, but transition statistics, class priors, and fallback selection use labeled development data from the target dataset, excluding test participants. Thus the evaluation is held-out-dataset for the gates, but the complete system is not zero-shot on a new dataset. We will state that deployment without target-dataset development information remains untested.

The fixed 1/4–3/4 mixture. The weight on the original posterior is a conservative anchor: because p_mix(y) >= p_orig(y)/4, the per-example NLL increase is bounded above by log(4). We also scanned original-prediction weights lambda in {0, 1/8, 1/4, 1/2, 1} on 126 streams and four delays, using the paper's equal-dataset aggregation. Relative to lambda=0, lambda=1/4 changed overall Macro-F1 by +0.106, +0.100, and +0.047 pp at d=30/120/300; all paired intervals included zero. It improved NLL at those positive delays by 0.00443, 0.00493, and 0.00339, with intervals excluding zero. Dataset-level directions differed, so these results do not establish an empirically optimal universal weight.

Which gate features contribute. We completed a one-feature-at-a-time deletion study: full-control refit plus 12 injection-feature and 14 output-feature deletions, across five LODO folds, 42 participants, three source seeds, three paired query seeds, and d=30/120. Removing output log_returns lowered Macro-F1 by 0.262/0.256 pp and worsened NLL by 0.00714/0.00668 at d=30/120; all paired 95% participant-bootstrap intervals excluded zero. Removing injection log_age_steps lowered Macro-F1 by 0.136/0.033 pp. However, deleting output source_prior_entropy increased aggregate Macro-F1 by 0.130/0.151 pp and improved NLL, with effects varying across datasets. The 52 feature-by-delay contrasts are unadjusted and exploratory. We will report the complete analysis, avoid selecting features on this target panel, and not claim that every input is necessary.

Recurrent baseline. We did not evaluate a delayed-feedback LSTM/GRU. The datasets have different label spaces (4, 6, 12, and 18 classes), and the current protocol does not specify a cross-dataset output head or a participant-level validation and model-selection scheme for such a baseline. We agree this comparison would be useful, but it requires a separately specified protocol; we will list it as future work rather than imply that it has been tested.

Thank you to the authors for the detailed response and additional experiments. I appreciate the effort to clarify the methodology and address my concerns.

However, I still have concerns regarding the complexity and clarity of the proposed framework. The response confirms that some components provide limited improvements in recognition performance, which raises further questions about the necessity of the full method.

One question remains: Given that the No history (refit) variant achieves nearly identical Macro-F1 performance to the full method, and that state injection can negatively affect performance on some datasets, what is the main practical advantage of using the complete Gated Arrival framework over a simpler alternative?

I also believe the paper would benefit from substantial rewriting and clearer illustrations to make the proposed method easier to understand. Therefore, I maintain my original assessment.

Reviewer_KacmOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies delayed label feedback in streaming activity recognition, distinguishing a label’s correctness for a historical sample from its relevance to the current activity. The proposed Gated Arrival method combines historical calibration, gated state injection, and output mixing while keeping the backbone frozen. Experiments on five datasets examine how feedback delay and label budget affect recognition.

Critical strengths
  1. The distinction between historical correctness and current relevance is useful for understanding how delayed feedback affects streaming predictions.

  2. The evaluation covers five datasets, multiple delay distributions, and different feedback budgets, including conditions where simpler methods outperform the proposed approach.

  3. The method keeps the backbone frozen and updates only a small calibrator online, limiting its additional computational and storage costs.

Critical weaknesses
  1. I have concerns about the practical relevance of the setting. The paper assumes that users can accurately label the activity at the time of the original request after a delay, but provides no real interaction study or measured response delays to support this assumption. Why not cancel outdated requests or ask again about the current activity? These straightforward alternatives are not compared, leaving the practical need for this setting unclear.

  2. Historical calibration essentially trains a small calibrator on past prediction–label pairs and applies it to current predictions. This assumes that classwise prediction biases remain stable over time. Correctly pairing historical predictions and labels does not establish that the learned calibration remains useful now, particularly with sparse feedback, class imbalance, or changing activity distributions.

  3. The paper argues that historical labels may no longer describe the current activity, yet still mixes them with current predictions to reset the state, relying on a gate to control the weight. Why is this preferable to historical learning alone or re-filtering forward from the label’s original timestamp? The comparisons do not establish a consistent advantage.

  4. Even when label injection is suppressed, the method resets the state to the current source prediction rather than retaining the existing filtered state. Some gains may therefore come from clearing accumulated state rather than using the returned label. A reset-only baseline is needed to separate these effects.

  5. The need for separate injection and output gates is not sufficiently established. The paper reports that simpler rules outperform the full method in some conditions. It should clarify when the two gates provide reliable gains that justify the added complexity.

  6. The method also uses labeled offline data to train the gates, estimate transitions, and select the fallback rule. Although these resources are disclosed, stronger comparisons with simple methods given the same development data and tuning effort are needed to establish the value of the proposed design.

Questions for authors

Please check the weaknesses. If the author can address my questions, I will raise my score.

3 comments
AuthorsOpenReview ↗

Practical Feedback Assumptions and Conditional Gains

Thank you for the detailed questions. We agree that controlled delay experiments do not establish the behavior of every real interaction policy, and we will make that boundary explicit.

Real response times and cancellation/re-query. We did not collect user response-time measurements or conduct a human interaction study. We therefore cannot claim that our simulated delays match real users. Cancelling stale requests or asking again about the current activity changes the feedback process and was not evaluated. Our additional geometric/long-tail analysis is a synthetic robustness test under a shared query schedule. At matched mean delay 120, the long-tail condition reduced Output-only Macro-F1 by 1.192 pp, while Full exceeded Output-only by 1.373 pp (95% CI [1.104, 1.644]). This supports robustness to the tested synthetic delay shape only; it does not validate real user behavior or cancellation policies.

Calibration under sparse feedback, imbalance, and drift. The experiments vary feedback budgets and delay schedules across five datasets, but do not test arbitrary class imbalance or temporal/distribution shift. Correct pairing of an origin prediction with its historical label is necessary for learning from that pair, but does not guarantee that the learned mapping remains useful under changing classwise bias. We will distinguish the tested delay/budget conditions from untested drift and imbalance regimes.

Historical learning, forward re-filtering, and state reset. The submitted Direct variant starts from the most recent label's origin time and filters forward to the current time. In the additional synthetic delay analysis, Full exceeded Direct on Macro-F1 by +0.514 to +0.790 pp and improved NLL by 0.01982 to 0.02338 across the four tested conditions; all paired 95% intervals excluded zero. We will make the Direct comparison and its assumptions easier to find, while limiting the conclusion to these conditions. We also clarify the reset control: No injection (refit) sets the injection weight to zero, retains the return-triggered reset (q_t=p_t), and refits the output gate. It therefore controls for the reset without using the returned label to inject into the state. We will label this operation explicitly in the text and figure.

Whether both gates justify their complexity. The main evaluation supports a conditional answer. In the six positive-delay Full-versus-Output-only cells, five paired participant-bootstrap intervals for Macro-F1 exclude zero; the remaining LN30 interval is uncertain (+0.165 pp, 95% CI [−0.034, +0.344]). At zero delay, Output-only is 0.28 pp higher. Full exceeds No history (refit) by only 0.024 pp at fixed d=30 and 0.016 pp at d=120 in overall Macro-F1, so we will not present historical calibration as a meaningful aggregate F1 gain. The synthetic stress test adds evidence of Full's advantage over Output-only under the four tested positive-delay conditions. We will report the probability-quality results separately from recognition accuracy and acknowledge settings in which simpler methods are preferable.

Development-data fairness. Gate coefficients are fitted using leave-one-dataset-out development streams. Other components, including transition statistics, class priors, and fallback selection, use labeled development participants from the target dataset while excluding test participants. The paper compares against simpler development-fitted rules under the disclosed protocol, but this is not an exhaustive equal-effort search over every possible gate. We will state both the shared protocol and this remaining limitation directly.

Reviewer_KacmOpenReview ↗

Response

Thank you for the response. After carefully reviewing the replies to each concern, I find that concerns 3 and 4 have been addressed. However, concerns 1 and 2 remain unresolved.

For concern 1, the authors explicitly acknowledge that they have neither real interaction data nor measured response times, and have not evaluated cancellation or re-querying strategies. The additional experiments only demonstrate robustness under a particular simulated delay setting.

For concern 2, the authors acknowledge that they have not tested changing classwise calibration biases or class imbalance. Revising the wording does not replace empirical validation.

Overall, I still do not consider the paper sufficiently developed for acceptance, and I maintain my score of 1.

Reviewer_XLd2OpenReview ↗

Response

Thank you for your response. I maintain my original score. I believe the paper requires a major revision. The authors have also confirmed several points I posted. These points are quire relevant and shall not be postponed as the future work and only kept as limitation in the current manuscript.

Reviewer_XLd2OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigated test time adaption in streaming human activity recognition tasks. It discussed the impact of delayed label which is no longer accurate and also introduced the gating mechanism to make adaptation more resilient to label noisy and misalignment with the sensor measurements.

Critical strengths
  1. the streaming task setup is quite practically relevant. The authors also simulated different delay models.
  2. the separation of learning and tracking makes sense.
Critical weaknesses
  1. the paper is very hard to read. a lot of numbers is embedded in text, and for instance Fig 6, very dense, and each variant has subtle difference to be inspected.
  2. the target task is very specific, would need to show the kind of approach for general streaming tasks.
  3. the gating architecture is conceptual sound and simple, but in general very heuristics. That is fine if it is shown to work in varying settings. Also, in general, there are very classic tracking algorithms like Hidden Markov model filtering or Bayesian modeling, would they be useful to track the sensor signal changes, e.g., from the raw sensor measurements, one can also derive if there is significant change, if it is consistent with label prediction. And state change is a label conditioned distribution, and the constraint to be exploited for updating the model is some level of smoothness.
Questions for authors
  1. why should a historical label update the belief of the current state?
  2. why not store old observations that generated the historical label and do the adaptation.
  3. The gating design tries to smoothen in the label space. Does it perform better than maintaining a probabilistic model that jointly models the sensor observation and label?
  4. given the gating depends on the frozen backbone (to extract features) and the classification head is only trainable, is it fair to say the approach only corrects some systematic probability bias, as the features dont change? The gating decision also includes other information source, therefore it helps to mitigate the bias for each individual case.
1 comment
AuthorsOpenReview ↗

Historical Labels, State Tracking, and Scope

Thank you for recognizing the relevance of delayed feedback and the separation between historical learning and current-state tracking. We agree that the distinction and the algorithmic sequence need to be clearer.

Historical correctness versus current relevance. A returned label remains correct for the sample that was queried. What may change is its relevance to the activity at the time it arrives. Our temporal filter represents a belief about the current activity; the transition model and injection gate determine whether a historical label should update that belief, using its age and current predictions. We will replace wording that suggests the label itself becomes inaccurate and add a short timeline example.

Why retain the origin prediction instead of the raw observation? We store the prediction made when the label was requested and pair it with the returned label to update a lightweight calibrator. This avoids retaining the raw sensor stream and separates learning from the historical prediction–label pair from correcting the current state. It is a design choice in this study, not evidence that raw-observation replay is ineffective. We have not compared against raw-observation replay and will state this limitation.

Relation to HMM/Bayesian filtering. Our filter is discriminative: it combines the classifier posterior with an estimated label-transition model. The posterior is not a generative likelihood for the raw sensor observation, so Eq. (4) should not be interpreted as a joint HMM over sensor measurements and labels. A raw-sensor model would require specifying and evaluating an observation likelihood, which we have not done. We will narrow the description and identify this comparison as future work.

What frozen-backbone adaptation can address. The frozen backbone cannot learn new features or repair arbitrary representation shift. The online calibrator addresses prediction-probability bias, while the two gates control state correction and output mixing using current per-instance evidence. We will describe the method as output-level calibration and temporal-state control, not feature adaptation, and clarify that substantial sensor or concept drift may remain unaddressed.

Evidence across settings and scope. Our evaluation remains within five human-activity datasets; it does not establish generality to all streaming tasks. In a post-submission synthetic stress test on 126 streams, we held the query schedule fixed and compared geometric and two-point long-tail delays with matched means of 30 and 120 seconds. Full exceeded Output-only in Macro-F1 in all four conditions, by +0.216 to +1.373 percentage points (all paired 95% participant-bootstrap intervals excluded zero). Full also exceeded Direct on Macro-F1 in all four conditions. These results support a qualified advantage for the tested delay patterns and task, not a general claim for arbitrary streaming applications or real user behavior.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.