- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper introduces Gated Arrival, a method for handling delayed labels in streaming activity recognition. The main motivation is that a correct historical label may no longer represent the current activity and could negatively affect the model's predictions. To address this, the authors separate historical calibration from current-state correction. They use returned labels and their saved original predictions to update a lightweight calibrator while keeping the backbone frozen. They also introduce two gates: an injection gate that controls how much a delayed label influences the current state based on factors such as label age and current evidence, and an output gate that controls the influence of the temporal state on subsequent predictions.
The authors evaluate their method on five activity recognition datasets under different feedback delays and label budgets. Their experiments show that receiving more correct labels does not always improve performance, and that the effectiveness of feedback methods depends on the delay. They also provide component comparisons to study the effects of historical calibration and state correction.
Critical strengths
- The proposed Gated Arrival method improves recognition performance over the Base model across different feedback delay settings.
- The method does not require updating the pretrained recognition backbone during test-time adaptation. Instead, it uses a lightweight historical calibrator with only 2K trainable parameters, while the two gates are fitted offline.
- The paper addresses an interesting problem in streaming activity recognition: a correct historical label may negatively affect current predictions when it arrives after the activity has changed.
- The authors evaluate their method on five activity recognition datasets with different feedback delays and label budgets. The experiments also include component comparisons to investigate the effects of historical calibration and state correction.
Critical weaknesses
- I found the paper difficult to follow, especially the figures. For example, Figure 3 is supposed to explain the main idea of the proposed method, but I found it confusing. The authors should provide a simpler illustration or a step-by-step example to help readers understand how the historical calibrator and the two gates work together.
- Although Gated Arrival improves over the Base model, Table 2 shows almost identical Macro-F1 performance to the No history (refit) variant. At fixed delays of 30 and 120, the improvement is only 0.02 percentage points. This raises questions about whether the additional complexity of the full method is justified. While the historical calibration component improves NLL, its practical importance for activity recognition needs clearer discussion.
- Table 3 shows that state injection provides substantial improvements on CAPTURE-24 but can reduce performance on UCI HAR and PAMAP2 under certain delays. I would like the authors to explain more clearly why these differences occur and under which conditions each component is expected to help.
- The method uses dataset-specific development information for some components, such as the transition matrix and class prior. It is unclear how well the approach would generalize to a new dataset or activity distribution without this information. Additional cross-dataset experiments or discussion would strengthen the paper.
- In Equation (3), the authors combine the original and calibrated predictions using fixed weights of 1/4 and 3/4. I would like to understand why these weights were selected and whether the performance is sensitive to this choice.
Questions for authors
- Although the gates are trained using other datasets, how well do they generalize to datasets with substantially different activity durations, transition frequencies, and prediction error patterns? Could the authors provide further analysis of the gates' robustness under these changes?
- The injection gate uses 12 features, while the output gate uses 14. Although the paper includes a comparison with a simpler age/persistence gate, could the authors provide a more detailed feature ablation to show which additional features contribute most to performance?
- I am also curious whether the authors have considered comparing Gated Arrival with a recurrent model, such as an LSTM or GRU, that explicitly receives delayed feedback. Such a comparison could help clarify whether the proposed transition filter and two-gate design offer advantages over a learned temporal model.
2 comments
Thank you to the authors for the detailed response and additional experiments. I appreciate the effort to clarify the methodology and address my concerns.
However, I still have concerns regarding the complexity and clarity of the proposed framework. The response confirms that some components provide limited improvements in recognition performance, which raises further questions about the necessity of the full method.
One question remains: Given that the No history (refit) variant achieves nearly identical Macro-F1 performance to the full method, and that state injection can negatively affect performance on some datasets, what is the main practical advantage of using the complete Gated Arrival framework over a simpler alternative?
I also believe the paper would benefit from substantial rewriting and clearer illustrations to make the proposed method easier to understand. Therefore, I maintain my original assessment.
Component Evidence, Generalization, and Presentation
Thank you for identifying the presentation problem and for asking how much the historical and state-correction components contribute in practice. We agree that the figures and the interpretation should be more explicit.
Figure 3 and a step-by-step example. We will redraw the method as a four-step timeline: (1) at time
t0, the frozen backbone emitsp0and a label is requested; (2) wheny0returns att1, the calibrator learns from(p0, y0); (3) the injection gate controls how much the historical label updates the current state using its age and current evidence; and (4) the output gate controls how much that state affects the current prediction. We will also reorganize Fig. 6 by historical learning, state injection, reset, and output mixing, with a concise definition for each variant.No history (refit) and practical Macro-F1 contribution. We agree that the aggregate Macro-F1 difference is nearly zero: Full exceeds No history (refit) by +0.024 pp at fixed d=30 and +0.016 pp at d=120. We will not present the calibrator's probability-quality improvement as a recognition-accuracy gain. We will explain that its value is for downstream use of probabilities, while its independent aggregate Macro-F1 contribution is negligible in these conditions.
Why injection helps on some datasets and hurts on others. Table 3 shows that removing injection from CAPTURE-24 reduces Macro-F1 by 10.63/9.10/5.46/2.59 pp at d=0/30/120/300, whereas removing it on UCI HAR improves Macro-F1 by 0.73 pp at d=30. Table 22 offers a descriptive alignment clue at d=120: CAPTURE-24's historical labels match the current activity 94.62% of the time while Base current accuracy is 78.77%; for HARTH the corresponding values are 88.21% and 94.14%. We will present this as a dataset-dependent alignment pattern, not a causal explanation or universal rule.
Cross-dataset generalization. Gate coefficients are evaluated with leave-one-dataset-out fitting, but transition statistics, class priors, and fallback selection use labeled development data from the target dataset, excluding test participants. Thus the evaluation is held-out-dataset for the gates, but the complete system is not zero-shot on a new dataset. We will state that deployment without target-dataset development information remains untested.
The fixed 1/4–3/4 mixture. The weight on the original posterior is a conservative anchor: because
p_mix(y) >= p_orig(y)/4, the per-example NLL increase is bounded above bylog(4). We also scanned original-prediction weightslambda in {0, 1/8, 1/4, 1/2, 1}on 126 streams and four delays, using the paper's equal-dataset aggregation. Relative tolambda=0,lambda=1/4changed overall Macro-F1 by +0.106, +0.100, and +0.047 pp at d=30/120/300; all paired intervals included zero. It improved NLL at those positive delays by 0.00443, 0.00493, and 0.00339, with intervals excluding zero. Dataset-level directions differed, so these results do not establish an empirically optimal universal weight.Which gate features contribute. We completed a one-feature-at-a-time deletion study: full-control refit plus 12 injection-feature and 14 output-feature deletions, across five LODO folds, 42 participants, three source seeds, three paired query seeds, and d=30/120. Removing output
log_returnslowered Macro-F1 by 0.262/0.256 pp and worsened NLL by 0.00714/0.00668 at d=30/120; all paired 95% participant-bootstrap intervals excluded zero. Removing injectionlog_age_stepslowered Macro-F1 by 0.136/0.033 pp. However, deleting outputsource_prior_entropyincreased aggregate Macro-F1 by 0.130/0.151 pp and improved NLL, with effects varying across datasets. The 52 feature-by-delay contrasts are unadjusted and exploratory. We will report the complete analysis, avoid selecting features on this target panel, and not claim that every input is necessary.Recurrent baseline. We did not evaluate a delayed-feedback LSTM/GRU. The datasets have different label spaces (4, 6, 12, and 18 classes), and the current protocol does not specify a cross-dataset output head or a participant-level validation and model-selection scheme for such a baseline. We agree this comparison would be useful, but it requires a separately specified protocol; we will list it as future work rather than imply that it has been tested.