- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies temporal localization in time series under multi-label bag-level supervision, where the classes present in a sequence are known but their temporal locations are not. The proposed StepMIL framework combines class-wise sparse temporal pooling with Local Soft-Target Propagation (LSTP). Class-wise pooling selects informative timesteps separately for each class, while LSTP provides bag-constrained soft supervision to unselected instances using neighboring predictions, source reliability, and feature affinity. Experiments on six datasets spanning human activity recognition, sleep staging, and ECG show that StepMIL achieves the highest instance macro-F1 and Cohen's kappa among the evaluated MIL baselines on all six datasets, as well as the highest bag mAP on five of the six datasets.
Critical strengths
1.The paper addresses an important distinction between bag-level class recognition and instance-level temporal localization. Its two design principles are clearly motivated, and the combination of class-wise sparse evidence selection and bag-constrained local supervision forms a coherent methodological framework. The analysis of the limitations of bag-level supervision further strengthens the problem motivation.
2.The empirical evaluation is comprehensive and carefully controlled. The method is evaluated across six diverse time-series datasets using shared encoder architectures, training budgets, and checkpoint-selection protocols. It consistently improves instance macro-F1 and Cohen's kappa over the evaluated MIL baselines. Additional comparisons with adapted weakly supervised temporal action localization methods, component ablations, and single-class versus multi-class bag analyses provide further support for the proposed approach.
Critical weaknesses
1.Insufficient comparison with closely related weakly supervised temporal segmentation approaches. Although the paper includes extensive comparisons with MIL methods and adapted weakly supervised temporal action localization approaches, Appendix G acknowledges that set-supervised action segmentation uses the same form of class-presence supervision but does not evaluate these methods. This limits the assessment of StepMIL's contribution relative to closely related temporal localization frameworks. Moreover, class-wise sparse pooling and local consistency-based supervision build on established ideas, making it important to clarify the methodological advances beyond their combination. Given the differences between video and time-series representations, a representative adapted set-supervised segmentation baseline or a more detailed analysis of the methodological distinctions would strengthen the novelty and empirical claims.
2.Potential foreground-biased supervision in mixed foreground–background bags. LSTP constructs soft targets only over classes present in the bag, without explicitly modeling background. In a bag containing one foreground class and background timesteps, Equation (7) assigns that foreground class a one-hot target at every timestep, while the entropy-based reliability factors in Equations (8) and (11) both equal one. Although edge affinity can modulate the supervision weights, it cannot introduce a background target. Consequently, LSTP may reinforce foreground predictions at background timesteps. The paper uses a separate background decision rule at inference, but the effect of this training-time bias is not sufficiently quantified, particularly on datasets with substantial background activity.
Questions for authors
1.Appendix G acknowledges that set-supervised action segmentation methods use the same form of supervision as StepMIL. Could the authors provide a representative comparison adapted to time-series data, or explain more concretely which methodological capabilities distinguish StepMIL from these closely related approaches?
2.For bags containing one foreground class together with background timesteps, how does LSTP avoid reinforcing foreground predictions in background regions? Could the authors report background precision, recall, or F1, together with an analysis of the LSTP supervision weights on background timesteps?
2 comments
Official Comment to Reviewer fkQP (2/2)
Q2, W2) LSTP Supervision on Background Timesteps
Table 1. Background Precision, Recall, and F1
| Method | OPP | HAPT | ||||
|---|---|---|---|---|---|---|
| precision | recall | F1 | precision | recall | F1 | |
| MI-Net | .928±.066 | .184±.146 | .280±.187 | .446±.056 | .026±.010 | .049±.018 |
| ABMIL | .904±.055 | .182±.110 | .287±.155 | .358±.037 | .025±.006 | .047±.009 |
| Additive MIL | .912±.066 | .186±.106 | .293±.150 | .329±.039 | .021±.005 | .039±.009 |
| MILLET | .924±.052 | .181±.105 | .287±.148 | .330±.048 | .026±.004 | .049±.007 |
| TimeMIL | .809±.100 | .355±.375 | .368±.355 | .327±.081 | .486±.256 | .376±.104 |
| StepMIL (Ours) | .817±.043 | .784±.123 | .793±.051 | .483±.048 | .520±.149 | .498±.092 |
Table 2. Average of LSTP supervision weight ω
| Dataset | split | background | foreground |
|---|---|---|---|
| OPP | train | 1.87 | 2.07 |
| test | 1.21 | 1.35 | |
| HAPT | train | 2.34 | 2.6 |
| test | 2.16 | 2.55 |
LSTP is designed to pass targets between adjacent timesteps, restricted to the classes present in the bag, so that the timesteps not selected by sparse pooling also receive supervision. Background was not explicitly modeled in this design. We therefore agree that, as the reviewer points out, LSTP can provide foreground supervision to background timesteps in some bags that contain a single foreground class on datasets with background. Across the whole datasets, however, the source reliability of LSTP assigns lower supervision weights to background timesteps, which partly mitigates such incorrect propagation. In addition, unlike other MIL methods, StepMIL uses class-wise sparse pooling so that the supervision from the bag loss reaches only the instances that are discriminative for each class. Background timesteps therefore receive little supervision from the bag loss overall. As a result, StepMIL achieves a higher background F1 than the other MIL baselines (please refer to Figure 9 of the revised version). We also agree that an MIL propagation scheme that separates background more explicitly is an important direction for future work. We thank the reviewer for this constructive comment.
Official Comment to Reviewer fkQP (1/2)
We would like to thank the reviewer for the in-depth review and the valuable suggestions. We are glad that the reviewer found the problem formulation and the two design principles of StepMIL clearly connected, and regarded our experimental evaluation as comprehensive and fair. Below, we respond to each of the reviewer's questions.
Q1, W1) Comparison with Set-Supervised Action Segmentation
Set-supervised action segmentation and StepMIL share a similar learning objective, since both learn timestep-wise class assignment from only the unordered set of classes present in a sequence. For this reason, Appendix G of the paper discusses the relation between the two approaches.
The two approaches differ first in the domain they target. Video methods take pretrained snippet features as input, whereas StepMIL takes the sampling points of multivariate sensor and physiological time series as its input unit, with up to 90,000 timesteps per bag. In addition, SCT and Action Sets explicitly model the length and structure of segments. SCT predicts the class and length of each region, and Action Sets searches over possible segmentations with a length model and a grammar. In contrast, StepMIL has no such structure and directly learns the timestep evidence produced by the encoder through class-wise sparse pooling and LSTP.
Nevertheless, we agree that a direct comparison with representative set-supervised action segmentation methods is needed. We ported two methods already cited in the paper, Action Sets (CVPR 2018) and SCT (CVPR 2020), onto the shared InceptionTime encoder. Training settings follow the public code wherever possible, and the rest of the protocol is the same as in Table 1.
The results are shown in the table above. Both methods have lower instance macro-F1 and κ than StepMIL on all six datasets. SCT has κ below 0.1 on four datasets and assigns one class to more than 97% of timesteps on DOD-H, HMC, and MIT-BIH. SCT is higher only in MIT-BIH accuracy, which results from predicting the majority class (N) at most timesteps. We have added this comparison to Appendix G (Table 7) of the revised version. We thank the reviewer for this suggestion, which allowed us to show the differences from set-supervised action segmentation methods more explicitly and to strengthen the case for our method.