acceptodds
Under review as a conference paper at ICLR 2027

From Which to When: Multiple Instance Learning for Temporal Localization in Time Series

Abstract

Multiple instance learning (MIL) enables time series classification from annotations that list the classes in a sequence without their temporal locations. We extend MIL for time series from sequence classification to temporal localization, which requires locating each class at the timesteps where it occurs. Knowing which classes occur does not determine when they occur, and we therefore identify two design principles for recovering the latter. Specifically, evidence should be selected sparsely and separately for each class, and instances outside the selected evidence should receive supervision constrained by the bag label. We propose StepMIL, which combines class-wise sparse temporal pooling with Local Soft-Target Propagation (LSTP). The pooling bases each class prediction on a few timesteps selected for that class. LSTP propagates bag-constrained soft targets between neighboring timesteps, weighted by source reliability and edge affinity, and is used only during training. We evaluate StepMIL on six datasets from human activity recognition, sleep staging and electrocardiography, with bags of up to 90,000 instances, against five MIL methods for bag classification. StepMIL improves temporal localization and sequence classification together, attaining the highest instance macro-F1 and Cohen's on every dataset and the highest bag mAP on five of the six datasets.

Primary area:
learning on time series and dynamical systems
Keywords:
Multiple Instance Learning, Time Series Classification

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_fkQPOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies temporal localization in time series under multi-label bag-level supervision, where the classes present in a sequence are known but their temporal locations are not. The proposed StepMIL framework combines class-wise sparse temporal pooling with Local Soft-Target Propagation (LSTP). Class-wise pooling selects informative timesteps separately for each class, while LSTP provides bag-constrained soft supervision to unselected instances using neighboring predictions, source reliability, and feature affinity. Experiments on six datasets spanning human activity recognition, sleep staging, and ECG show that StepMIL achieves the highest instance macro-F1 and Cohen's kappa among the evaluated MIL baselines on all six datasets, as well as the highest bag mAP on five of the six datasets.

Critical strengths

1.The paper addresses an important distinction between bag-level class recognition and instance-level temporal localization. Its two design principles are clearly motivated, and the combination of class-wise sparse evidence selection and bag-constrained local supervision forms a coherent methodological framework. The analysis of the limitations of bag-level supervision further strengthens the problem motivation.

2.The empirical evaluation is comprehensive and carefully controlled. The method is evaluated across six diverse time-series datasets using shared encoder architectures, training budgets, and checkpoint-selection protocols. It consistently improves instance macro-F1 and Cohen's kappa over the evaluated MIL baselines. Additional comparisons with adapted weakly supervised temporal action localization methods, component ablations, and single-class versus multi-class bag analyses provide further support for the proposed approach.

Critical weaknesses

1.Insufficient comparison with closely related weakly supervised temporal segmentation approaches. Although the paper includes extensive comparisons with MIL methods and adapted weakly supervised temporal action localization approaches, Appendix G acknowledges that set-supervised action segmentation uses the same form of class-presence supervision but does not evaluate these methods. This limits the assessment of StepMIL's contribution relative to closely related temporal localization frameworks. Moreover, class-wise sparse pooling and local consistency-based supervision build on established ideas, making it important to clarify the methodological advances beyond their combination. Given the differences between video and time-series representations, a representative adapted set-supervised segmentation baseline or a more detailed analysis of the methodological distinctions would strengthen the novelty and empirical claims.

2.Potential foreground-biased supervision in mixed foreground–background bags. LSTP constructs soft targets only over classes present in the bag, without explicitly modeling background. In a bag containing one foreground class and background timesteps, Equation (7) assigns that foreground class a one-hot target at every timestep, while the entropy-based reliability factors in Equations (8) and (11) both equal one. Although edge affinity can modulate the supervision weights, it cannot introduce a background target. Consequently, LSTP may reinforce foreground predictions at background timesteps. The paper uses a separate background decision rule at inference, but the effect of this training-time bias is not sufficiently quantified, particularly on datasets with substantial background activity.

Questions for authors

1.Appendix G acknowledges that set-supervised action segmentation methods use the same form of supervision as StepMIL. Could the authors provide a representative comparison adapted to time-series data, or explain more concretely which methodological capabilities distinguish StepMIL from these closely related approaches?

2.For bags containing one foreground class together with background timesteps, how does LSTP avoid reinforcing foreground predictions in background regions? Could the authors report background precision, recall, or F1, together with an analysis of the LSTP supervision weights on background timesteps?

2 comments
AuthorsOpenReview ↗

Official Comment to Reviewer fkQP (1/2)

We would like to thank the reviewer for the in-depth review and the valuable suggestions. We are glad that the reviewer found the problem formulation and the two design principles of StepMIL clearly connected, and regarded our experimental evaluation as comprehensive and fair. Below, we respond to each of the reviewer's questions.


Q1, W1) Comparison with Set-Supervised Action Segmentation

OPPDOD-HHARTH
Methodbag mAPaccmF1κbag mAPaccmF1κbag mAPaccmF1κ
Action Sets (CVPR 2018).554±.121.121±.052.067±.012.062±.041.842±.004.376±.024.336±.017.247±.020.726±.055.710±.018.404±.011.618±.021
SCT (CVPR 2020).416±.035.165±.048.036±.007.091±.036.715±.015.197±.026.067±.004.005±.009.684±.033.716±.037.379±.007.610±.040
StepMIL (Ours).707±.084.349±.075.213±.055.297±.071.924±.006.573±.013.368±.011.318±.014.852±.072.899±.010.611±.026.856±.032
HAPTHMCMIT-BIH
Methodbag mAPaccmF1κbag mAPaccmF1κbag mAPaccmF1κ
Action Sets (CVPR 2018).926±.016.605±.011.474±.002.555±.011.859±.048.405±.050.353±.012.274±.061.428±.003.417±.186.183±.061.038±.015
SCT (CVPR 2020).897±.018.524±.163.268±.087.448±.184.714±.048.267±.122.083±.033.000±.000.453±.017.885±.012.248±.008.015±.012
StepMIL (Ours).955±.010.880±.016.655±.022.860±.019.916±.024.494±.033.449±.079.327±.057.497±.034.864±.045.362±.023.269±.020

Set-supervised action segmentation and StepMIL share a similar learning objective, since both learn timestep-wise class assignment from only the unordered set of classes present in a sequence. For this reason, Appendix G of the paper discusses the relation between the two approaches.

The two approaches differ first in the domain they target. Video methods take pretrained snippet features as input, whereas StepMIL takes the sampling points of multivariate sensor and physiological time series as its input unit, with up to 90,000 timesteps per bag. In addition, SCT and Action Sets explicitly model the length and structure of segments. SCT predicts the class and length of each region, and Action Sets searches over possible segmentations with a length model and a grammar. In contrast, StepMIL has no such structure and directly learns the timestep evidence produced by the encoder through class-wise sparse pooling and LSTP.

Nevertheless, we agree that a direct comparison with representative set-supervised action segmentation methods is needed. We ported two methods already cited in the paper, Action Sets (CVPR 2018) and SCT (CVPR 2020), onto the shared InceptionTime encoder. Training settings follow the public code wherever possible, and the rest of the protocol is the same as in Table 1.

The results are shown in the table above. Both methods have lower instance macro-F1 and κ than StepMIL on all six datasets. SCT has κ below 0.1 on four datasets and assigns one class to more than 97% of timesteps on DOD-H, HMC, and MIT-BIH. SCT is higher only in MIT-BIH accuracy, which results from predicting the majority class (N) at most timesteps. We have added this comparison to Appendix G (Table 7) of the revised version. We thank the reviewer for this suggestion, which allowed us to show the differences from set-supervised action segmentation methods more explicitly and to strengthen the case for our method.

AuthorsOpenReview ↗

Official Comment to Reviewer fkQP (2/2)

Q2, W2) LSTP Supervision on Background Timesteps

Table 1. Background Precision, Recall, and F1

MethodOPPHAPT
precisionrecallF1precisionrecallF1
MI-Net.928±.066.184±.146.280±.187.446±.056.026±.010.049±.018
ABMIL.904±.055.182±.110.287±.155.358±.037.025±.006.047±.009
Additive MIL.912±.066.186±.106.293±.150.329±.039.021±.005.039±.009
MILLET.924±.052.181±.105.287±.148.330±.048.026±.004.049±.007
TimeMIL.809±.100.355±.375.368±.355.327±.081.486±.256.376±.104
StepMIL (Ours).817±.043.784±.123.793±.051.483±.048.520±.149.498±.092

Table 2. Average of LSTP supervision weight ω

Datasetsplitbackgroundforeground
OPPtrain1.872.07
test1.211.35
HAPTtrain2.342.6
test2.162.55

LSTP is designed to pass targets between adjacent timesteps, restricted to the classes present in the bag, so that the timesteps not selected by sparse pooling also receive supervision. Background was not explicitly modeled in this design. We therefore agree that, as the reviewer points out, LSTP can provide foreground supervision to background timesteps in some bags that contain a single foreground class on datasets with background. Across the whole datasets, however, the source reliability of LSTP assigns lower supervision weights to background timesteps, which partly mitigates such incorrect propagation. In addition, unlike other MIL methods, StepMIL uses class-wise sparse pooling so that the supervision from the bag loss reaches only the instances that are discriminative for each class. Background timesteps therefore receive little supervision from the bag loss overall. As a result, StepMIL achieves a higher background F1 than the other MIL baselines (please refer to Figure 9 of the revised version). We also agree that an MIL propagation scheme that separates background more explicitly is an important direction for future work. We thank the reviewer for this constructive comment.

Reviewer_hqWpOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses time series classification from a multiple instance learning (MIL) perspective, where labels are provided only at the bag level, while the goal is to produce instance-level predictions and identify the temporal locations of different classes. The authors propose StepMIL, which combines class-wise sparse temporal pooling to select informative timesteps for each class and Local Soft-Target Propagation (LSTP) to supervise unselected instances using neighboring predictions.

While the paper addresses a relevant problem and reports encouraging experimental results, my main concern is the limited methodological novelty, as the proposed framework largely combines ideas already explored in MIL and weakly supervised temporal localization. In addition, the experimental comparisons and ablation studies could be strengthened, and some design choices require better justification. That was the ground for my vote: weak reject.

Critical strengths

• The paper addresses a relevant practical problem, as obtaining dense timestep-level annotations can be expensive, especially in biomedical applications. Learning temporal localization from bag-level labels is therefore a meaningful research direction.

• The reported results across six datasets from different domains are encouraging, because the proposed method consistently outperforms the competitors in instance-level classification. This is promising because achieved at no substantial extra computation cost. • The idea of incorporating local neighborhood information through soft-target propagation is intriguing, particularly for time series where temporal continuity is important. The use of reliability and feature similarity to regulate propagation is also reasonable, as it attempts to avoid propagating unreliable predictions across class boundaries.

Critical weaknesses

• The authors present shifting the focus from time series classification to temporal localization under bag-level supervision, through class-wise sparse selection and bag-constrained supervision, as their key contribution. However, some aspects of this method have already been used. For instance, localization is nothing but identifying instance-level prediction, which is an integral part of MIL. Similarly, emphasizing a subset of informative instances is not new. For example, attention-based MIL (ABMIL) assigns different importance weights to instances, while existing weakly supervised temporal localization methods have explored top-k evidence selection and pseudo-label-based supervision. Although StepMIL combines these ideas in a time series setting, the authors should better clarify what is fundamentally new beyond their integration.

• The authors compare against several established MIL methods, which is useful, but the main baselines were published in 2024 or earlier. I suggest the author compare against some newer published method to better position the proposed approach. Examples include Hi-Time (ICML 2026), FIC-TSC (ICML 2025), Learning Soft Sparse Shapes for Efficient Time-Series Classification (ICML 2025), Spectral-Aware Reservoir Computing for Fast and Accurate Time Series Classification (ICML 2025), Adaptive Time Encoding for Irregular Multivariate Time-Series Classification (NeurIPS 2025), and SGN (NeurIPS 2025). I understand that these methods may not directly address the same weakly supervised setting, but comparisons or discussion of their relevance would strengthen the paper. More importantly, stronger comparisons with recent weakly supervised temporal localization and segmentation methods would be valuable. Finally, I am not asking to compare against all of these methods, rather your own selection of a few recently published methods in credible venues.

• The paper assumes a single class label for each timestep (noted by ltl_t in ection 3.1) . While this formulation is convenient, its practical interpretation deserves more discussion. For example, in ECG or EEG biomedical signals, meaningful patterns are often characterized by local morphology spanning multiple consecutive samples, rather than an isolated time point. The same applies to trends in stock market. Although the encoder produces contextualized representations, the authors should clarify how this formulation captures such temporal patterns and whether predicting labels at individual sampling points is always meaningful. I understand that the users use such temporal relations in their second module (through the local neighborhood propagation in Section 3.5), but does that mean that the first method of explicitly selecting top k instances is irrelevant. This requires further justification.

• Although the paper includes component-level ablations, several important design choices remain insufficiently investigated. In particular, the authors fix the selection budget at (k=0.125T) without analyzing its sensitivity to the actual duration of class evidence ((n_c)). It would be useful to study different selection ratios, especially when the true class duration is much shorter or longer than k. The comparison with and without LSTP is helpful, but a more detailed per-dataset analysis would strengthen the conclusions. Furthermore, the paper only considers immediate neighboring timesteps for propagation. An analysis of different propagation depths or neighborhood sizes could reveal the tradeoff between localization accuracy, temporal smoothing, and computational cost. Needless to say that transformer-based methods can capture short—term and long-term relations.

• Minor comment: Some methodological discussions are unnecessarily complicated and difficult to follow. For example, the class-specific evidence allocation and gradient analysis in Eqs. 4–5 are not well explained.

Questions for authors
  1. What is the key methodological novelty of StepMIL compared with existing MIL approaches using attention, sparse instance selection, and pseudo-label propagation?
  2. How sensitive is the performance to the selection budget k.k. How does it relate to n_c? Do we know what is the average ratio of TS (n_c/T) that contributes to the TS classification, for different applications. In other words, how to tune kk for optimal performance. I am particularly interested to see how the model performs for extreme cases k<<nck<<n_c and k>>nck >> n_c.
  3. How does the performance change when increasing the propagation neighborhood beyond adjacent timesteps? Is there a tradeoff between improved temporal consistency and inaccurate propagation across class boundaries?
  4. Given that many meaningful time series patterns span multiple timesteps, what is the justification for the per-timestep labeling formulation, and how does the model preserve local temporal morphology during sparse evidence selection?
6 comments
AuthorsOpenReview ↗

Official Comment to Reviewer hqWp (1/4)

We thank the reviewer for the detailed review and the thoughtful questions. We appreciate that the reviewer found the problem practically important, recognized the consistent instance-level improvements across domains without substantial additional computation, and considered the design of LSTP reasonable. We address each comment below.


Q1, W1) Methodological Novelty of StepMIL

How Our Goal Differs from Existing MIL. We agree that instance-level outputs are an important aspect of MIL and that revealing important instances has long been one of its goals. MILLET and TimeMIL also address this for time series. Their timestep outputs, however, mainly serve to interpret sequence-level predictions, and these outputs have been examined through perturbation analyses, synthetic data, and attention visualizations. To the best of our knowledge, only a few works have explicitly evaluated timestep-wise class assignment against dense annotations and made it the training objective, and they did so within a single domain. Figure 3 further shows that accurate timestep-level prediction does not follow as a by-product of bag-level training in existing MIL methods. Models with nearly identical bag mAP differ by up to .59 in timestep macro-F1. StepMIL therefore extends the training objective of MIL to timestep-level localization, and we verify that it reaches this objective on six datasets from three domains (human activity, sleep, and ECG).

Methodological Novelty of StepMIL. The methodological novelty of StepMIL lies in building two properties of multivariate time series, temporal continuity and class boundaries, directly into a training objective that supervises every timestep from bag labels alone. We do not claim novelty for sparse selection or pseudo-labeling on their own.

In time series, real-world dynamics evolve continuously, so adjacent timesteps usually share a class, except at class boundaries. LSTP follows this property by passing targets only between adjacent timesteps. The targets are restricted to the classes present in the bag, since the bag label rules out all other classes. Two weights control this propagation. The source reliability indicates how much the target at a timestep can be trusted. The edge affinity indicates whether two adjacent timesteps may share a target, which suppresses propagation across class boundaries.

These weights play a different role from the attention weights of attention-based MIL. An attention weight indicates how important an instance is for the bag prediction and is used for aggregation. The LSTP weights, in contrast, are used for supervision. Neither the source reliability nor the edge affinity is a learnable parameter, and both are used only during training.

In addition, class-wise sparse pooling reflects that not every timestep serves as evidence for the bag-level prediction of a time series, and that a small portion of discriminative timesteps gives a more accurate prediction. LSTP fills the gap left by this sparsity, since it supervises every timestep, including those that the pooling does not select. With this combination, StepMIL attains the best value in 23 of the 24 bag-level and instance-level cells of Table 1 against the MIL baselines, on six datasets from three domains.

Transferring Weakly Supervised Localization Methods from Video. We understand that the question also concerns whether StepMIL mainly brings localization ideas from other domains to time series. To the best of our knowledge, no prior MIL work combines sparse instance selection with pseudo-label propagation for multivariate time series, and therefore no such method is available for a direct comparison. The closest methods come from weakly supervised temporal action localization (WS-TAL) in video, and Appendix F of the paper already shows experimentally that transferring them to time series is not sufficient. We transferred five WS-TAL methods, including W-TALC and UM, which use top-k selection, and ISSF, which uses pseudo-labels. We kept the architectures and losses of their public code and used the same encoder and protocol. All five have instance macro-F1 lower than StepMIL by .105–.279. These methods assume a few actions separated by background, so their video-level objective can be satisfied by class scores that barely change over time. In that case, every timestep of a bag is assigned the same class. The failures therefore concentrate on OPPORTUNITY, DOD-H, and HMC, where a bag contains several classes. BaS-Net predicts a single class at every timestep on DOD-H and HMC (κ = 0), and UM treats some timesteps as background even on datasets without background. These results indicate that multivariate time series that contain multiple classes in a single bag call for a design suited to their properties rather than a transfer of existing methods.

AuthorsOpenReview ↗

Official Comment to Reviewer hqWp (2/4)

Q2, W4) Sensitivity to the Selection Budget k

Datasetk = 1T/32T/16T/8 (paper)T/4T/2k = T
OPP.135±.055.194±.072.200±.070.213±.055.174±.061.153±.056.127±.031
DOD-H.219±.009.286±.020.337±.030.305±.013.327±.024.356±.097.279±.048
HARTH.507±.011.584±.008.580±.041.611±.026.615±.013.604±.043.608±.033
HAPT.539±.036.613±.023.631±.013.655±.022.633±.014.612±.018.590±.010
HMC.249±.036.388±.068.424±.010.449±.079.438±.084.404±.063.460±.033
MIT-BIH.277±.011.353±.008.350±.020.362±.023.359±.009.356±.011.342±.011
Average.321.403.420.432.424.414.401

Following the reviewer's suggestion, we varied the selection budget over seven values, from k = 1 (class-wise max pooling) to k = T (class-wise mean pooling). The paper uses a single value, k = ⌈T/8⌉, for all datasets. The six-dataset average of instance macro-F1 is highest at T/8 (.432), and all values from T/16 to T/4 are within .012 of it. In contrast, performance drops sharply on every dataset at k = 1, which shows the limitation of max pooling.

Performance at k = T differs across datasets depending on their average class length. On HARTH and MIT-BIH, where classes are relatively long, performance barely changes from T/32 to k = T. This is because the pooled score depends less on the class length when k is smaller than the evidence length. On OPPORTUNITY and HAPT, where classes are short, performance gradually decreases as k grows beyond T/8. At k = T, which corresponds to mean pooling, macro-F1 drops sharply from .213 to .127 on OPPORTUNITY and from .655 to .590 on HAPT. This is consistent with our analysis in Section 3.3 that sparse pooling reduces, though does not eliminate, the dilution.

DatasetTmedian n_c/T10%90%n_c < T/32n_c < T/16n_c < T/8n_c < T/4n_c < T/2
OPP1,800.045.029.086.13.80.95.981.0
DOD-H90,000.17.017.82.11.27.44.56.76
HARTH1,500.40.0391.0.08.16.27.42.55
HAPT750.34.083.66.03.06.23.40.79
HMC57,600.22.033.72.06.17.33.53.76
MIT-BIH3,600.91.0941.0.01.04.13.21.30

We also measured the length that each class occupies in a bag. This length is an upper bound of n_c, because n_c in the paper is not the length of a class instance but a hypothetical number of the instances that actually determine the class. The median length ranges from .045T on OPPORTUNITY to .91T on MIT-BIH, and it also varies widely across classes and bags within a dataset (the 10–90% range on DOD-H is .017T–.82T). Hence there is no single k that fits a dataset, and n_c cannot be known without precise instance annotations. T/8 is not perfect optimal on every dataset. DOD-H scores higher at T/2 and HARTH at T/4, and on HMC the difference from k = T is within the seed-to-seed variation. Our additional experiments show that T/16–T/4 is a stable range that can be used without per-dataset tuning. We have added this experiment to Appendix H of the revised version. Please refer to Figure 6 there for the results on each dataset and to Table 8 for the class lengths.


Q3, W4) Wider Propagation Neighborhood

Datasett±1 (Ours)[t−5, t+5][t−25, t+25]
OPP.213±.055.204±.070.197±.083
DOD-H.305±.013.319±.039.310±.016
HARTH.611±.026.598±.025.594±.034
HAPT.655±.022.648±.006.615±.024
HMC.449±.079.407±.050.403±.048
MIT-BIH.362±.023.352±.014.322±.043
Average.432.421.407

Widening the propagation neighborhood did not improve performance. To examine this, we changed LSTP to propagate targets among all timesteps within [t−5, t+5] or [t−25, t+25] instead of between the adjacent timesteps t±1, and kept every other setting. The average instance macro-F1 over the six datasets was .432 with t±1, .421 with [t−5, t+5], and .407 with [t−25, t+25]. Both wider neighborhoods were lower than t±1 on five of the six datasets.

The receptive field of the encoder is the range over which features are shared, whereas the neighborhood of LSTP is the range over which targets are propagated. The initial targets of adjacent timesteps are already similar, so a wider neighborhood adds little new information. A wider neighborhood instead makes it more likely that incorrect targets are propagated across class boundaries. For this reason, StepMIL propagates targets only between adjacent timesteps. We have added this experiment to Appendix H of the revised version. Please refer to Figure 7 there for the results on each dataset.

AuthorsOpenReview ↗

Official Comment to Reviewer hqWp (3/4)

Q4, W3) Per-Timestep Labeling and Local Morphology

Per-timestep prediction does not mean that a class is decided from an individual sampling point alone. It means that the model predicts which state or class a timestep belongs to based on its surrounding temporal context. That meaningful patterns in ECG and EEG span multiple timesteps is compatible with assigning to each timestep the class predicted from those patterns.

In StepMIL, the temporal encoder is responsible for representing local temporal morphology. The convolutions of InceptionTime aggregate information over temporal windows and produce a contextualized representation z_t = f_θ(X)t for each timestep. The evidence head maps it to class scores s{t,c}, and class-wise top-k pooling selects among these scores. Sparse selection does not discard input sampling points beforehand. It selects context-aware class evidence after the whole input has been encoded. The evidence at a selected timestep therefore already reflects the waveform and temporal pattern around that position.

The encoder, sparse pooling, and LSTP play different roles. The encoder represents the temporal morphology of the input. Sparse pooling selects the evidence that supports the bag-level prediction of each class. LSTP uses targets built from this evidence to provide additional supervision for timestep prediction. The fact that LSTP exploits local relationships therefore does not replace the morphology modeling of the encoder or the need for sparse selection.

We adopt per-timestep prediction as a common unit of prediction and evaluation so that localization can be compared across domains under controlled conditions. Evaluating class assignments aligned with the time axis of the input measures the quality of temporal predictions, which bag-level recognition alone cannot reveal. We do not claim that individual samples are independent semantic units in every domain, or that sample-level resolution is always optimal.


W5) Clarity of Eqs. 4–5

The passage with Eqs. 4–5, which the reviewer mentions, analyzes a limitation of existing MIL methods to motivate our class-wise evidence selection. It compares how the gradient of the bag loss is distributed over the class scores at each timestep. When all classes share one temporal weight a_t, as in attention-based MIL, the gradient received by the score of class c at timestep t is a_t δ_c (Eq. 4, along the direct pathway with the weights held fixed). Classes that co-occur in a bag are therefore supervised at the same timesteps in the same proportions. With class-wise top-k pooling, the gradient reaches only the timesteps selected for class c (Eq. 5). Supervision is thus allocated to different timesteps for different classes.

We aimed to state our argument precisely through these equations, but some parts may not have been clear to the reviewer. If the reviewer could specifically point out which parts were unclear, we would be glad to explain further during the discussion and to reflect it in the revision.

Reviewer_8kUCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper extends multiple instance learning for time series from bag classification to temporal localization, where training sees bag labels and evaluation asks for a class per timestep. StepMIL is built from class-wise sparse evidence selection and bag-constrained supervision of unselected instances, the principles the paper names. The method averages the highest pooling scores of each class over a fixed fraction of timesteps, then propagates soft targets between neighbors through two gates at training time only. Experiments cover six datasets against MIL baselines and report bag mAP, accuracy, macro-F1 and Cohen's κ\kappa. The motivating question, which label versus when, becomes a measurable statement about trained models, and the macro-F1 gains are largest on OPPORTUNITY and HAPT.

Critical strengths
  1. The Framing Is Made Measurable: Section 4.3 and Figure 3 turn the which versus when question into a statement about trained models. Among model pairs whose bag mAP is nearly tied, the instance macro-F1 gap per dataset reaches 0.59, a spread that bag mAP alone does not reveal.
  2. Inference-Neutral Method: One cosine head serves the bag objective, the soft targets and the instance predictor, and the propagation term enters only through the training loss. Inference is therefore a per-timestep arg⁡max⁡\arg\max with no extra parameters, no extra passes and no change to the encoder, so the reported gains cost nothing at test time.
  3. Controlled and Broad Evaluation: All methods share the encoder, the splits, the training budget and the checkpoint rule on validation bag mAP. Coverage spans six benchmarks, from short beat-level recordings to long multi-sensor sessions, with the inter-patient split on MIT-BIH and a held-out scorer protocol on DOD-H.
Critical weaknesses
  1. The Formal Claim Establishes Non-Identifiability, Not Model Quality: Proposition 1 shows that for every instance prediction some consistent labeling agrees on at most m timesteps. Appendix A then chooses that labeling after seeing the prediction, so the construction also succeeds when the prediction is perfect. The result is invariant to model quality and cannot show that a model with good bag scores lands on a poor labeling. Narrowly read, it says the bag label does not single out one labeling, which is what Sections 3.4 and 4.3 need, while Figure 3 supplies the performance claim.
  2. The Sweep Rests on Point Estimates With Overlapping Intervals: Section 4.2 reports a near sweep of the comparison table, supported only by point estimates. The plus or minus one standard deviation intervals overlap the best baseline in 5 of the 12 macro-F1 and κ\kappa cells and in 5 of 6 bag mAP cells. Overlap is descriptive rather than a significance test, and no paired test across seeds or folds appears in the paper. Tables 2 and 3 also omit standard deviations, so the ablation premiums carry no uncertainty estimate at all.
  3. The Reference Labels Do Not Support the Stated Resolution: Appendix B advertises millisecond resolution, but DOD-H and HMC label one sleep stage per 30 s scoring epoch. Those two of the six targets therefore carry 60 reference decisions per bag rather than thousands. Where predictions stay constant inside an epoch, sample-level macro-F1 and κ\kappa equal the epoch-level values exactly, while within-epoch variation breaks that equality. OPPORTUNITY exposes a second gap: Section 3.1 assumes one class per timestep, and Appendix B never states whether two gestures can occupy one.
  4. Appendix D Omits the Baseline-Specific Settings, So the Completeness Claim Fails: The paper says Appendix D lists every hyperparameter, yet Table 4 is titled "Hyperparameters shared by all datasets". Several of its rows are StepMIL-specific, including the evidence budget, the two temperatures and the propagation weight. Table 5 adds per-dataset windows and strides, but no attention dimension, hidden width, pooling temperature or tuning rule appears for any MIL baseline. TimeMIL's warm-up appears only in Appendix E, so that schedule detail lies outside the appendix the paper calls complete, while capacity and tuning remain unverifiable.
Questions for authors
  1. Can the authors train InceptionTime with dense labels on the same splits, reporting κ\kappa and macro-F1, to show whether the remaining gap is annotation-limited?
  2. Suggestion: Proposition 1 could be restated as a lower bound on the consistent-labeling set, with Figure 3 cited for the performance claim.
  3. Could Table 5 state each dataset's effective label resolution, and could Appendix B say whether two OPPORTUNITY gestures can share one timestep?
  4. Could the authors list the baseline-specific hyperparameters, including TimeMIL's warm-up, and say whether any baseline was tuned on validation bag mAP?
4 comments
AuthorsOpenReview ↗

Official Comment to Reviewer 8kUC (1/2)

We sincerely thank the reviewer for the careful review and the constructive suggestions. We appreciate that the reviewer recognized the gap between bag-level and instance-level performance shown in Figure 3, the absence of additional inference cost, and our broad and controlled experiments. We address each comment below.


Q1) Gap to a Dense-Label Upper Bound

DatasetModelbag mAPaccmF1κ
OPPMILLET.666±.119.286±.046.112±.028.228±.047
Ours.707±.084.349±.075.213±.055.297±.071
dense.760±.106.519±.100.310±.140.471±.113
DOD-HMILLET.894±.006.335±.016.287±.008.148±.011
Ours.924±.006.573±.013.368±.011.318±.014
dense.898±.006.702±.016.577±.032.549±.029
HARTHMILLET.799±.049.876±.011.586±.050.827±.027
Ours.852±.072.899±.010.611±.026.856±.032
dense.867±.061.896±.004.635±.031.854±.014
HAPTMILLET.961±.012.802±.018.554±.004.770±.021
Ours.955±.010.880±.016.655±.022.860±.019
dense.919±.012.906±.011.817±.014.890±.013
HMCMILLET.876±.028.402±.054.378±.033.245±.059
Ours.916±.024.494±.033.449±.079.327±.057
dense.933±.035.604±.023.547±.030.488±.035
MIT-BIHMILLET.468±.036.450±.047.222±.014.042±.004
Ours.497±.034.864±.045.362±.023.269±.020
dense.560±.016.821±.039.382±.019.303±.049

Following the reviewer's suggestion, we trained the same InceptionTime encoder with a per-timestep classifier on dense labels. The splits, batch size, number of epochs, and checkpoint selection are identical to those in Table 1. The results are shown in the table above. On most datasets, StepMIL does not yet reach the upper bound obtained with dense supervision. Since the encoder and training conditions are identical, the remaining gap can be attributed to the limitation of the annotation. Compared with the MIL baselines, however, StepMIL substantially narrows this gap. We believe that the rest of the gap can be further reduced in future work. The gap also widens with the number of classes that co-occur in a bag, since the bag label constrains the temporal location of each class less tightly when more classes share the bag. The κ gap is below .04 on HARTH, HAPT, and MIT-BIH, whose bags contain 1.4–2.3 classes on average, and at least .16 on OPPORTUNITY, DOD-H, and HMC, whose bags contain 3.1–4.8. We have added this comparison to Appendix J of the revised version. Please refer to Figure 8 of the revised version, which shows for each dataset the gap that StepMIL closes relative to MILLET and the gap that remains.


Q2, W1) Scope of Proposition 1

Proposition 1 was intended from the outset as a worst-case non-identifiability result. It states a theoretical limit, namely that bag annotations alone cannot identify instance labels. Accordingly, both the main text (Section 3.4) and Appendix A state that bag supervision alone "cannot guarantee" instance accuracy. The proposition does not prove that the bag-level and instance-level performance of real models are unrelated. That real models with similar bag performance differ widely in localization is shown separately, as an empirical observation, in Section 4.3 and Figure 3. If any phrase in the paper could be read otherwise, we would be grateful if the reviewer could let us know during discussion phase. We would be glad to clarify it.


Q3, W3) Label Resolution and OPPORTUNITY Annotation

We thank the reviewer for this careful observation. We agree that the prediction resolution and the resolution of the reference labels should be stated separately. We have added a column to Table 5 that gives the resolution of the reference annotation for each dataset. For OPPORTUNITY, we have added a clarification to Appendix B. The gesture labels are read from a single mid-level annotation track (ML_Both_Arms). This is the label set of the gesture recognition task in the OPPORTUNITY challenge (Chavarriaga et al., 2013), and prior gesture recognition work on this dataset, such as DeepConvLSTM (Ordóñez and Roggen, 2016) and Hammerla et al. (2016), uses the same track. This track stores exactly one label per sample, either one of the 17 gestures or NULL. Two gestures therefore never share a timestep. We believe these changes make the evaluation setting clearer, and we appreciate the reviewer's attention to this detail.

AuthorsOpenReview ↗

Official Comment to Reviewer 8kUC (2/2)

Q4, W4) Baseline Hyperparameters and Tuning

The baseline heads keep the architectures of their public code. All methods are trained with the shared settings in Table 4 of the revised version, except the WS-TAL methods, which keep their released optimizers (Appendix F). No hyperparameter of any baseline was tuned, and validation bag mAP is used only for checkpoint selection, in the same way for every method. For TimeMIL, we used a warm-up of 20 epochs. We thank the reviewer for pointing out these missing details, which are now stated in Appendix D of the revised version.


W2) Paired Statistical Analysis

baselinemF1 diff. [95% CI]Holm pκ diff. [95% CI]Holm p
W-TALC+.105 [+.063, +.150]7.6e-5+.155 [+.093, +.226]7.6e-5
BaS-Net+.231 [+.134, +.321]7.6e-5+.294 [+.181, +.404]7.6e-5
UM+.279 [+.189, +.376]7.6e-5+.391 [+.246, +.545]7.6e-5
ACM-Net+.246 [+.174, +.321]7.6e-5+.238 [+.180, +.294]7.6e-5
ISSF+.147 [+.115, +.178]3.9e-3+.165 [+.139, +.190]3.9e-3
MI-Net+.083 [+.060, +.105]7.6e-5+.117 [+.075, +.162]7.6e-5
ABMIL+.088 [+.046, +.129]2.3e-4+.114 [+.056, +.176]1.5e-4
Additive MIL+.086 [+.046, +.122]2.3e-4+.116 [+.058, +.177]2.3e-4
MILLET+.086 [+.051, +.118]1.9e-4+.111 [+.058, +.170]2.3e-4
TimeMIL+.322 [+.217, +.438]7.6e-5+.417 [+.283, +.585]7.6e-5

We thank the reviewer for this helpful suggestion and agree that whether mean ± std intervals overlap cannot by itself tell whether two methods differ. All methods in our paper are trained with the same seeds and splits, so the results are paired by design. We have added a paired analysis that uses this pairing.

Because OPPORTUNITY folds and DOD-H scorers are not independent, we first average them within each seed. On the resulting 18 (dataset, seed) units, we apply a one-sided paired Wilcoxon signed-rank test to the differences between StepMIL and each baseline. For ISSF, we use the 9 units from the three datasets on which it was run. We apply the Holm correction across the 20 comparisons (10 baselines × 2 metrics, macro-F1 and κ).

As a result, the mean differences in both macro-F1 and κ are positive against all ten baselines (macro-F1 +.083 to +.322, κ +.111 to +.417), and every 95% confidence interval lies above zero. All 20 comparisons give p ≤ .004 after correction. This statistical test thus confirms that StepMIL outperforms the baselines.

The intervals in the tables overlap because the variation across folds, subjects, and seeds is shared by all methods. Taking the difference between two methods on the same unit cancels this shared variation. This is why the paired differences are consistently positive even where the intervals overlap. We have added this analysis and the table to Appendix M of the revised version.

Thank you for the revision. Appendix D now states the warm-up and the no-tuning rule, Appendix B settles the overlap question, Table 5 carries the reference resolution, and Appendix M adds a corrected paired test over dataset and seed units. Those are the artifacts I asked for.

What is settled

  • Baseline settings and the tuning rule are disclosed (Appendix D).
  • The OPPORTUNITY overlap is resolved by construction (Appendix B).
  • Reference resolution per dataset is tabulated (Table 5).
  • The paired test is corrected and its arithmetic checks out (Appendix M).
  • Single-class bags are separated in the dense comparison (Table 10).

What needs action

  1. Proposition 1 still carries two claims. The proposition is unchanged, and it says what I read it as saying, but the contributions bullet on page 2 presents it as formal and empirical at once, and the conclusion still draws the localization claim from it. Please state the formal result and the empirical one separately, and attribute the second to Section 4.3 and Figure 3.

  2. The dense reference is not an upper bound. It is behind the method in several cells of Table 11, so the label in Appendix J, in the caption and in the figure legend overstates it. Please describe it as a reference at another operating point, or say what the band denotes.

  3. The resolution sentence contradicts the table. The contributions bullet and Appendix B still advertise instance resolutions at a millisecond scale, while Table 5 reports annotation resolutions in seconds. That is true of the sampling rate and false of the annotation. Please call it the sampling period in both places.

  4. Background is a class in one appendix and not in the other, and the two datasets where that happens carry the largest gains. Please say which governs the denominator.

  5. The paired test does not support the per-dataset sentence. It pools units across datasets and supports a positive average, while Section 4.2 claims the method exceeds every baseline on every dataset and Appendix I shows a case where it does not. Please narrow the sentence or add a per-dataset statement.

  6. The new evidence is not cited from the body. No sentence in the first nine pages points to the new appendices, tables or figures, so a reader sees the round-one paper. Please add the cross-references and reconcile the two six-dataset averages that differ by a rounding of the same quantity.

  7. Uncertainty and Table 4. Tables 2 and 3 report no dispersion, and Table 4 is still titled as shared by all datasets while several rows apply to the method alone and no per-head dimensions are given. Please add the dispersion, retitle the table, and give the dimensions or a narrower phrasing.

Where this leaves me

I will keep my evaluation for now, but happy to raise it once the issues are addressed. The paired test could have gone the other way and did not, and the dense comparison shows that the six-dataset average was hiding a spread rather than inventing a gain. For revision, please attribute the performance claim to Section 4.3 and Figure 3 and drop the upper bound label from Appendix J, Table 11 and Figure 8. If both stay, the paper carries two claims its own appendices contradict.

We thank the reviewer for the detailed follow-up. We have revised the paper according to each point. The changes of this round are marked in purple in the revised version.

  1. Proposition 1. The contributions and the conclusion now state the formal result and the empirical result in separate sentences. The empirical claim is attributed to Section 4.3 and Figure 3.

  2. Dense reference. We have removed the upper-bound label from Appendix J, Table 11 and Figure 8 and now refer to a dense reference. Appendix J reports the cells in which StepMIL is higher, and the caption of Figure 8 states what each band spans.

  3. Sampling period. The contributions and Appendix B now refer to sampling periods. Table 5 lists the resolution of the reference annotation.

  4. Background and the denominator. On OPPORTUNITY and HAPT, macro-F1 is computed over all test timesteps and counts background as one additional category. Its denominator is therefore C+1, which is 18 and 13. The foreground accuracy and κ exclude background. Appendices B and C now state this consistently.

  5. Paired test. Section 4.2 now states that the mean instance macro-F1 and the mean κ of StepMIL are higher than those of all five baselines on every dataset, and it cites the paired test as support over the six datasets. Appendix M states that the test pools the datasets and adds that the mean difference is positive on each dataset against every baseline.

  6. Cross-references. Sections 4.1, 4.2 and 4.5 now refer to Table 5 and to Appendices F, G, H, J and M. The two averages differ only through DOD-H, which is evaluated with one scorer in the selection-budget analysis and with five scorers in Tables 2 and 3. Appendix H now explains this difference.

  7. Dispersion and Table 4. Tables 2 and 3 now report the standard deviation over three seeds. Table 4 separates the settings shared by StepMIL and the MIL baselines from those of StepMIL alone. Appendices D and F give the dimensions of the baseline heads.

The reviewer's comments were constructive and have made our paper more solid. We sincerely thank the reviewer for them.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Revised Version Uploaded

We thank all reviewers for their careful reviews and constructive comments. We have uploaded a revised version of the paper, and all revised parts are marked in blue. We would be grateful if the reviewers could check the revised version. The main revisions are the label resolution of each dataset in Table 5 and the annotation track of OPPORTUNITY in Appendix B (Reviewer 8kUC), the training settings of the baselines and the absence of tuning in Appendix D (Reviewer 8kUC), a comparison with set-supervised action segmentation methods in Appendix G (Reviewers fkQP and hqWp), analyses of the selection budget k and of the propagation neighborhood in Appendix H (Reviewer hqWp), a comparison with a dense upper bound trained on timestep labels in Appendix J (Reviewer 8kUC), an analysis of background timesteps in Appendix K (Reviewer fkQP), and a paired statistical comparison with every baseline in Appendix M (Reviewer 8kUC).

AuthorsOpenReview ↗

Discussion Phase Summary

Dear Reviewers,

We sincerely thank all reviewers for their careful evaluation. We summarize below the recognized strengths and the concerns addressed during the discussion phase. All revisions are marked in blue and purple.

Acknowledged Strengths

  • From recognition to temporal localization (hqWp, fkQP). StepMIL extends time-series MIL from bag classification to temporal localization, locating each class in time from class-set labels alone. Reviewer fkQP recognized this distinction as important, and Reviewer hqWp found the setting practical since dense timestep-level annotation is costly.
  • Coherent design principles (hqWp, fkQP). StepMIL selects evidence sparsely and separately for each class, and supervises the unselected timesteps with targets constrained by the bag label. Reviewer fkQP found these two principles clearly motivated, and Reviewer hqWp found the reliability and affinity weights of LSTP reasonable.
  • No additional inference cost (8kUC, hqWp). LSTP is used only during training, and inference is a per-timestep argmax of the same head. The improvements are therefore achieved without introducing additional parameters or forward passes at inference.
  • Comprehensive and controlled evaluation (8kUC, hqWp, fkQP). All methods share the encoder, splits, training budget and checkpoint rule on six datasets from three domains. StepMIL consistently improves instance macro-F1 and κ.
  • Measurable gap between bag- and instance-level performance (8kUC). Section 4.3 and Figure 3 show that models with nearly tied bag mAP differ widely in instance macro-F1.

Summary of Rebuttal

  • Claims and statistics (8kUC). To clarify that Proposition 1 shows only that bag labels do not determine instance labels, we state this formal result separately from the empirical gap shown in Section 4.3 and Figure 3. A paired test supports the improvement against all ten baselines (Appendix M), and Tables 2 and 3 report standard deviations.
  • Evaluation and implementation details (8kUC). We added a dense reference (Appendix J), the annotation resolution and background definition (Table 5, Appendices B and C), and the baseline settings and head dimensions (Appendix D, Table 4).
  • Novelty (hqWp). We clarified that the novelty lies in building temporal continuity and class boundaries into a training objective that supervises every timestep from bag labels alone. LSTP weights serve supervision, not aggregation as in attention-based MIL.
  • Design choices (hqWp). Performance is stable for selection budgets from T/16 to T/4, and adjacent-timestep propagation outperforms wider neighborhoods (Appendix H). We also clarified that the encoder captures local morphology before sparse selection.
  • Additional baselines (hqWp, fkQP). Two set-supervised action segmentation methods, Action Sets and SCT, are below StepMIL in instance macro-F1 and κ on all six datasets (Appendix G). StepMIL also outperforms SoftShape and MambaSL on all instance-level entries.
  • Background timesteps (fkQP). StepMIL attains the highest background F1 on OPPORTUNITY and HAPT, and LSTP assigns lower weights to background timesteps (Appendix K).

Discussion Status

Reviewer hqWp acknowledged our response, in particular the rationale and new results on how local temporal patterns are retained in our method, and raised the score. Reviewer 8kUC indicated that the evaluation would be raised once the remaining points were addressed, and we have addressed all seven points in the second revision. Although we have not received further feedback, we hope that the revision resolves the remaining concerns. For Reviewer fkQP, we carefully responded to both concerns with new comparisons and analyses, which are now included in the revised version, although there was no opportunity to discuss them further.

We are sincerely grateful to all reviewers for their constructive comments, which have made the paper more solid.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.