acceptodds
Withdrawn from ICLR 2027

Joint Class–Time Learning for Video Classification with Multi-Instance Partial-Label Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multi-instance partial-label learning (MIPL) addresses inexact supervision in both the instance and label spaces, which can be applied to video classification. However, bag-level labels do not explicitly supervise the correspondence between candidate classes and temporal evidence. We propose PIVOTMIPL, which couples label disambiguation with temporal evidence allocation through a joint class–time assignment. Occupancy-regularized spherical matching associates contextualized video features while learning nonuniform temporal mass and discouraging excessive concentration. During training, candidate-restricted inference recomputes the assignment within the candidate label set. A dual-marginal KL projection then constructs a structured teacher that incorporates momentum-refined class beliefs while preserving the proposal's temporal occupancy. A single plan-level KL objective aligns the full-space predictor with this teacher. Our analysis characterizes when candidate re-solving differs from masking and shows that, under the stated construction, the joint objective decomposes into class-marginal and class-conditional temporal supervision. We construct VCMIPL benchmarks from Breakfast, DoTA, and FineAction using model-generated candidate labels and evaluate the method across four feature representations. Extensive experimental results demonstrate that PIVOTMIPL outperforms existing MIPL algorithms in both effectiveness and efficiency. Code is available at https://anonymous.4open.science/r/PIVOTMIPL-202B/.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
multi-instance partial-label learning, label disambiguation, weakly supervised learning, video classification

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_D4VEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies video classification under multi-instance partial-label supervision, where each video is associated with a candidate label set containing one true class, without temporal annotations. The proposed PIVOTMIPL constructs a class–prototype–time assignment using contextualized video features and occupancy-regularized spherical matching. During training, candidate-restricted inference produces a proposal, whose class marginal is refined through momentum updates. A dual-marginal KL projection constructs a teacher that combines these refined class beliefs with the proposal’s temporal occupancy. The full-space predictor is trained through a plan-level KL objective.

The paper analyzes the relationship between candidate re-solving and masking, the concentration properties of occupancy regularization, and the decomposition of the joint objective. Validation covers three constructed video benchmarks with four feature representations, seven MIPL baselines, component ablations, an encoder-matched comparison, and efficiency analyses. The reported results achieve the highest mean ACC and BACC in 10 of 12 configurations.

Critical strengths
  1. Coherent structured supervision mechanism. The candidate-restricted proposal, momentum-refined class beliefs, and dual-marginal teacher form a well-connected learning procedure. In particular, the teacher construction provides a concrete way to combine historical class information with the current temporal allocation. This is a technically meaningful contribution beyond simply attaching an additional temporal encoder to a bag-level classifier.

  2. Careful analysis supported by relevant empirical controls. The paper precisely characterizes when re-solving differs from masking and establishes the class–time decomposition and current-point gradient equivalence of the objective. The appendix also appropriately distinguishes concentration control from correct localization. Empirically, the joint-loss and teacher-projection ablations show consistent directional improvements across the twelve tested single-fold configurations, while the encoder-matched comparison provides useful evidence that the encoder alone does not explain all reported gains.

Critical weaknesses
  1. The candidate-generation protocol appears to permit indirect use of downstream test labels. Sections 4.1 and B.3 describe generating out-of-fold candidate labels using the same five-fold partition subsequently used for downstream evaluation. Under this procedure, when fold F1 is the downstream test set, candidates for training samples in F2 are generated by a recognizer trained on F1, F3, F4, and F5, including the exact labels of F1. Thus, being out-of-fold for each individual sample does not ensure isolation from the downstream test labels. The supplementary code loads precomputed candidates and reuses them across downstream folds, but does not include the candidate-generation pipeline needed to resolve this concern. If the protocol operates as described, the reported results do not establish performance under a strictly held-out evaluation. Sharing these candidates across methods does not eliminate the issue or establish that relative comparisons are unaffected.

  2. The experiments do not sufficiently isolate the value of the central class–time mechanism over simpler alternatives. The classification results and ablations are encouraging, but the theoretical distinction between candidate re-solving and masking is not accompanied by a direct empirical comparison. A simple class-specific temporal aggregation model with a PLL objective and the same encoder is also missing. Moreover, the evidence for the proposed temporal-allocation explanation is indirect: improved ACC/BACC does not establish that the learned allocation better identifies supporting evidence or reduces reliance on background. This matters because improved class–time association is central to the motivation. The key ablations are limited to one fold, with some differences being small, leaving the reliability of those component-level conclusions uncertain.

Questions for authors
  1. Protocol clarification: For each downstream test fold, were all candidate generators used to construct the downstream training candidates trained without any examples or exact labels from that test fold? Please provide the relevant generation code or explicit split mapping. If candidates were generated once using the shared five-fold procedure described in B.3, can you provide results with candidate generation nested entirely within each outer training split?

  2. Evidence needed to assess the central mechanism: Do you have a controlled comparison between candidate-restricted re-solving and global masking followed by renormalization, with the remaining teacher construction and training procedure unchanged? How does the method compare with a simpler class-specific temporal aggregation model using the same encoder and a PLL objective?

  3. Reliability and interpretation of the ablations: Are multi-fold or repeated-seed results available for A3 and A4? Is there direct evidence, such as temporal-annotation-based analysis or controlled snippet/background perturbations, that the joint objective improves the quality of supporting temporal evidence rather than only the final classification accuracy?

  4. Configuration clarification: Table 5 specifies four prototypes per class for Breakfast, whereas all four supplied Breakfast scripts use --num_prototypes 6. Which setting produced the main results and the full-model reference in the ablations?

Reviewer_KgCoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies video classification under multi-instance partial-label learning (MIPL), where both the true class within a candidate label set and the supporting temporal evidence are ambiguous. The proposed PIVOTMIPL framework models a joint class–prototype–time assignment using occupancy-regularized spherical transport. During training, it re-solves the assignment within the candidate label set, refines class beliefs through momentum updates, and constructs a dual-marginal teacher that preserves both refined class marginals and temporal occupancy. A plan-level KL objective is then used to supervise the full-space predictor. The paper also provides theoretical analyses of candidate re-solving and the resulting supervision objective. Experiments are conducted on three constructed VCMIPL benchmarks based on Breakfast, DoTA, and FineAction with four feature representations. PIVOTMIPL achieves the best mean ACC and BACC in 10 of 12 configurations.

Critical strengths
  1. Coherent structured formulation of label and temporal ambiguity. The paper goes beyond bag-level label disambiguation by explicitly modeling class–time associations through a joint assignment. The candidate-restricted re-solving, dual-marginal teacher construction, and plan-level KL objective form a relatively coherent methodological framework rather than a collection of unrelated modules.

  2. Broad empirical evaluation with useful controls. The method is evaluated across three datasets and four feature representations, and achieves the best mean ACC/BACC in 10 of 12 configurations. Importantly, the authors also augment the strongest baseline with the same temporal encoder and still retain an average advantage, which helps separate the proposed learning mechanism from improvements due solely to a stronger temporal backbone.

Critical weaknesses
  1. The central class–time claim is not directly validated. The main motivation of the paper is that explicit class–time modeling better associates candidate classes with their supporting temporal evidence. However, the main experiments evaluate only downstream classification metrics (ACC and BACC). Thus, the current evidence demonstrates that the proposed structured latent supervision improves classification, but does not directly show that the inferred temporal evidence itself is more accurate or meaningful. This is consequential because temporal evidence allocation is the central distinction between PIVOTMIPL and conventional bag-level MIPL methods. A direct temporal localization/evidence-alignment evaluation would substantially strengthen the main claim.

  2. The realism of the VCMIPL benchmarks is unclear. All three benchmarks are constructed from fully labeled video datasets using predictions from a trained VideoMAEv2 recognizer. Moreover, if the ground-truth class is absent from the generated candidate set, it is explicitly inserted to guarantee Yi \in S_i. This is mathematically consistent with the standard MIPL assumption, but it is unclear how closely this construction reflects realistic weak-label annotation pipelines. The paper would benefit from stronger justification of the intended real-world setting and robustness experiments under alternative candidate-set generation processes or noise models.

Questions for authors
  1. Can the authors directly evaluate the quality of the learned temporal evidence? For example, if temporal annotations or action-localization information are available in any of the source datasets, could the inferred class-conditional temporal assignments be compared against them? Such results would materially affect my assessment of the paper’s central class–time learning claim.

  2. How representative is the proposed VCMIPL benchmark construction of a realistic application? In particular, what practical annotation or pseudo-labeling process would produce candidate sets with the same assumptions, including guaranteed inclusion of the true label? Results under alternative candidate-set generation mechanisms or stronger candidate-label noise would help assess whether the gains are specific to the current construction.

  3. Can the authors provide stronger statistical validation for the key ablations? The main results are averaged over five folds, whereas the ablations are evaluated only on Fold 1 at epoch 40. Since the joint KL objective and dual-marginal teacher are central to the paper’s contribution, multi-fold or multi-seed results for these key ablations would materially strengthen the attribution of the gains.

Reviewer_gCMMOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces Video Classification with Multi-Instance Partial-Label Learning (VCMIPL), a weakly supervised video-classification setting where the exact video label is not provided during training. Instead, each training video is associated with a set of candidate labels, with the assumption that one of them is correct. In addition, the model does not know which temporal parts of the video provide evidence for the correct class.

To address these two sources of ambiguity, the paper proposes PIVOTMIPL, which jointly performs candidate-label disambiguation and temporal evidence allocation through a structured class–prototype–time assignment. The learned temporal information is used to improve video classification, while the final inference output remains a single video-level class.

Critical strengths

Strengths:

  • Introducing a new video classification problem Video Classification with Multi-Instance Partial-Label Learning (VCMIPL)
  • Proposing PIVOTMIPL to solve the VCMIPL
  • They create several benchmarks for the VCMIPL using the exsiting datasets
Critical weaknesses
  1. Poor paper presentation. The main motivation and target problem are not clearly explained.
    • The paper is difficult to follow because the main motivation, target problem, and contribution are not clearly established early in the paper.
    • It is unclear whether the main contribution is a new weakly supervised video-classification setting (VCMIPL), a new benchmark protocol, or a new MIPL method for video classification. The title and introduction do not make this distinction sufficiently clear. For example, the introduction starts discussing “existing methods” for learning from inexact supervision without clearly defining which methods or research setting it refers to.
  • More importantly, the paper does not sufficiently explain why VCMIPL is needed, what real-world applications naturally provide candidate-label supervision, or why this supervision setting is preferable to existing weakly supervised video-learning paradigms.
  • Figure 1 mainly presents the PIVOTMIPL workflow without explaining how it works, what each part in the figure do.
  • Similarly, the methodology introduces several technical components before clearly motivating why each component is necessary for solving the proposed problem. For example, it is unclear why components such as FlowFast net and bidirectional modeling are needed. The motivation behind these technical design choices is not sufficiently explained.
  1. Missing validation of temporal evidence. The paper claims that temporal information helps label disambiguation, but does not directly evaluate whether the learned temporal assignments are correct.

  2. Unclear practical need for VCMIPL with the existence of large VLM for data annotations. I assume that the VCMIPL will help for video annotaion to decrease the need for manual annotaions. Howvere, with the large MLLM model do we still need these models for annotations. A relevant baseline would provide an open-weight VLM such as Qwen with each training video and its candidate-action set, ask the model to select the most likely action, and use the resulting predictions or soft scores as pseudo-labels to train a standard strong video classifier (current SOTA) and compare it with PIVOTMIPL? the same for closed source models (e.g., gpt etc) to see how PIVOTMIPL's perfromance is different from a trained model on the annotation optained from a closed-source model?

Questions for authors

please see the weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.