acceptodds
Under review as a conference paper at ICLR 2027

Recursive Predictive Abstraction for Unsupervised Temporal Partonomy Induction

Abstract

We present , an unsupervised framework for inducing temporal partonomies through recursive predictive abstraction. RePart couples a shared causal predictor with Bayesian composition evidence over current-level representations and prediction residuals, recursively merging adjacent units into increasingly coarse events without predefined hierarchy levels. The hierarchy is induced through discrete Bayesian composition, while predictive and composition-aware objectives shape the representation. On Breakfast Actions, 50Salads, and Assembly101, RePart achieves the best unsupervised hierarchical F1 on all three benchmarks and improves over PARSE on every TED-Sim/hF1 comparison, while remaining competitive on fine- and coarse-grained segmentation. Ablations show that recursive Bayesian composition provides a strong structural inductive bias, with joint state–residual evidence and representation adaptation further refining higher-level organization. In controlled Assembly101 action anticipation experiments, partonomy-conditioned models outperform fixed-scale models, with RePart remaining strong across increasing prediction horizons. These results show that recursively induced predictive structure provides coherent part–whole organization and useful context for downstream temporal reasoning. Code is in the supplementary material and will be released publicly after review.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Temporal partonomy induction, Hierarchical event understanding, Predictive representation learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_rJ3aOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies unsupervised temporal partonomy induction: discovering part–whole relationships between fine-grained actions and higher-level events in videos without temporal annotations. The authors propose RePart, which uses current-level representations and prediction residuals to compute Bayesian composition evidence for merging adjacent temporal units. The same predictor is then reapplied to the resulting coarser sequence, recursively building a variable-depth hierarchy. On Breakfast-Part, 50Salads-Part, and Assembly101-Part, RePart achieves the highest hierarchical F1 (hF1) among the unsupervised methods compared. Action anticipation experiments on Assembly101 further show that the induced partonomies provide useful context for downstream prediction.

Critical strengths

1. The paper goes beyond detecting action boundaries to study how fine-grained actions compose into higher-level events. This part–whole structure is useful for understanding how activities unfold in long videos, giving the work a clear and practically relevant goal.

2. The method is well motivated, fits the problem, and is easy to follow. The experiments cover comparisons with multiple baselines across different datasets, and the appendix includes additional hyperparameter sensitivity analyses.

3. The paper is clearly organized and readable. Figure 1 provides an intuitive overview of RePart, making it easier to understand how predictive contextualization and recursive composition work together.

Critical weaknesses

1. Some training components provide only modest gains in hierarchical performance. In Table 2, using only the prediction loss already achieves an hF1 of 31.66, compared with 32.02 for full RePart. In Table 3, the variant without predictive adaptation still achieves an hF1 of 31.76.

2. RePart requires 25 epochs of sequence-specific unsupervised adaptation for each video, which could become costly for long videos or large video collections. Reporting processing times across different video lengths would help readers assess the method’s computational efficiency.

Questions for authors

1. The limitations discussion notes that hard recursive composition is irreversible: once adjacent temporal units are merged, they cannot be split at later levels, so early grouping errors can propagate through the hierarchy. Do the authors have plans to mitigate this issue, for example through recoverable composition?

2. On Breakfast-Part and 50Salads-Part, RePart performs worse than PARSE on fine-grained segmentation. This does not undermine its value for temporal partonomy induction, but it suggests a trade-off between hierarchical agreement and fine-scale boundary accuracy. Further analysis of this relationship would be helpful.

3. Appendix C examines sensitivity to the objective weights and coherence temperature, but not to the composition threshold τ\tau, which directly determines whether adjacent temporal units are merged. Please include a sensitivity analysis showing how performance changes with this threshold.

Reviewer_7bNXOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes RePart, an unsupervised method for discovering hierarchical event structures in videos. It recursively merges adjacent temporal units using Bayesian evidence derived from contextual representations and prediction residuals. Unlike PARSE, which operates over predefined hierarchy levels, RePart repeatedly reconstructs the temporal units and applies a shared predictor at each level. Experiments on Breakfast Actions, 50Salads, and Assembly101 show improvements in hierarchical F1 and tree-structure similarity.

Critical strengths
  1. Its recursive design constitutes a valuable extension to earlier predictive‑hierarchy approaches including PARSE and STREAMER.
  2. The Bayesian composition criterion offers clear rules to build temporal part‑whole relations.
  3. Structural ablation studies nicely disentangle the individual impacts brought by recursion, predictive residuals, and representation adaptation.
  4. Action‑anticipation experiments offer solid evidence showing the inferred hierarchy brings benefits for downstream temporal reasoning.
Critical weaknesses
  1. The Bayesian criterion lacks sufficient justification. Merging is driven by standardized evidence with a hard‑coded threshold set to 0.5. It is still an open question how well this setup approximates proper calibrated Bayesian model selection.
  2. When comparing against PARSE, adaptation cost has not been properly controlled. RePart carries out 25 epochs of optimization for each individual sequence. It would be helpful to see overhead and performance numbers when adaptation budgets are kept identical for both methods.
  3. Evaluation for the variable‑depth property is incomplete. Only three representative hierarchy levels are exported for benchmarking, so we cannot tell how well the method recovers more complex hierarchical structures.
  4. Performance on fine‑grained segmentation is mixed. PARSE achieves better results than RePart on Breakfast‑Part and 50Salads‑Part. One trade‑off here is that gains in hierarchical consistency sacrifice some precision on local event boundaries.
  5. Another limitation comes from irreversible hard recursive merging. Errors made at early merging steps get passed onto higher levels. The work could be improved by including comparisons to soft or revisable hierarchical grouping alternatives.
Questions for authors
  1. The composition threshold is fixed at τ = 0.5, corresponding to the mean standardized evidence within each level. How sensitive are the results to this threshold? Does the resulting composition score have a meaningful probabilistic interpretation?
  2. RePart requires 25 epochs of optimization for each video. However, Table 3 shows only minor changes in TED-Sim and hF1 when predictive adaptation is removed. What is the computational cost of this optimization, and how does it compare with PARSE under a comparable computational budget?
  3. Appendix A explains how variable-depth hierarchies are reduced to three representative levels. How sensitive are the reported improvements to this selection procedure? Could the authors provide statistics on the actual hierarchy depths and clarify whether alternative level-selection rules affect the results?
  4. Table 4 shows that RePart performs worse than PARSE on fine-grained segmentation for Breakfast-Part and 50Salads-Part. Is this mainly caused by irreversible early merges or by the selection of hierarchy levels for evaluation? Can the authors provide evidence to distinguish these effects?
2 comments

Thanks for the extra results. They help answer some of my questions. I'm still a bit concerned about how sensitive the method is to the threshold, and the adaptation seems quite expensive for a fairly small gain. I'm also not fully convinced by the hierarchy depth results. Still, I think the main idea is interesting, so I'll keep my rating.

AuthorsOpenReview ↗

Thank you for the acknowledgement and a clarification on Threshold Sensitivity and Future Directions

Thank you for reviewing the additional results and for your constructive feedback! We agree that jointly adapting composition granularity and recursion depth is an important direction for future work. We also acknowledge the computational cost of adaptation and the challenges of evaluating variable-depth hierarchies using benchmarks with only two annotated semantic levels.

We would like to clarify one point regarding threshold sensitivity: τ\tau controls the rate of recursive abstraction. Lower thresholds produce coarser partitions more rapidly, while higher thresholds require more recursive levels to reach comparable granularity. Consequently, performance differences under a fixed recursion budget reflect both composition decisions and the temporal scales reached, rather than composition quality alone. Our depth-constrained experiments illustrate this interaction, with τ=0.25\tau=0.25 recovering competitive hierarchical agreement using only three levels.

We have explicitly discussed these limitations and corresponding future directions in the revised manuscript. Thank you again for your thoughtful assessment.

Reviewer_dQM7OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces RePart, an unsupervised method for inducing nested temporal partonomies from video. Starting from frame-level visual features, a shared causal Transformer predicts the next representation at the current abstraction level. RePart then computes local Bayesian composition evidence from each representation and its prediction residual, standardizes this evidence within the level, and merges adjacent units whose evidence exceeds a threshold. Mean-pooled groups form the next level, and the same predictor is recursively reapplied to the newly composed sequence. Sequence-specific unsupervised adaptation couples predictive loss with evidence-regression, within-event coherence, and boundary-separation objectives; final hard composition is then rerun with frozen adapted parameters.

Evaluation uses the Breakfast-Part, 50Salads-Part, and Assembly101-Part benchmarks. RePart obtains the best unsupervised hierarchical F1 on all three datasets and improves over PARSE on both hierarchical metrics in all six dataset/metric comparisons, although hierarchical clustering retains the best unsupervised TED-Sim on 50Salads. Fine-grained segmentation is mixed, while coarse segmentation generally improves. Ablations indicate that recursion is the dominant ingredient and that state plus residual evidence is stronger than either alone. In a controlled Assembly101 anticipation experiment, RePart's induced hierarchy gives the best result at four of five horizons. The paper identifies irreversible early merges and resulting over-segmentation as the main failure mode.

Critical strengths
  1. RePart offers a coherent and technically well-motivated alternative to fixed-scale hierarchy induction. Recursively redefining the temporal units and reapplying a shared predictive model is a meaningful architectural contribution, and the explicit parent-child containment matches the partonomy objective. The use of both contextualized state and prediction residuals for Bayesian composition is intuitive and well supported by the structural ablations.

  2. The validation is broad and mostly well controlled. RePart improves hierarchical agreement over the closest unsupervised predictive baseline across three datasets, provides fine/coarse segmentation breakdowns rather than only hierarchy metrics, and evaluates downstream utility while holding the TempAgg architecture fixed and varying only the hierarchy. The paper is also candid that gains concentrate at coarser scales and that hard recursive decisions can propagate errors.

Critical weaknesses
  1. Statistical reliability is underreported. The main tables present point estimates without run-to-run uncertainty or significance tests, even though each video is independently initialized and adapted and the method includes hard thresholded decisions. It is therefore difficult to determine whether smaller differences, including several anticipation and segmentation gains, are robust. This is particularly important because the method performs test-video-specific optimization for 25 epochs.

  2. The evaluation protocol obscures the practical and methodological cost of sequence-specific adaptation. RePart fits PCA independently per video, randomly initializes, and adapts the predictive model on the same unlabeled sequence later evaluated, but the paper does not report compute/runtime, convergence variability, or a comparison under matched per-video test-time optimization budgets. The ablation without predictive adaptation is close on TED-Sim and hF1, which raises the question of how much the adaptation machinery contributes relative to the analytic recursive grouping and whether comparisons to baselines are resource-matched.

Questions for authors
  1. Please report variability across independent initializations for the main partonomy and anticipation results. Which improvements over PARSE and hierarchical clustering remain statistically reliable?

  2. What is the per-video runtime and compute cost of PCA, 25-epoch adaptation, and recursive inference, and how does this compare with the baselines? Are the main conclusions unchanged if all methods are compared under a matched test-time compute budget?

  3. Since “no predictive adaptation” retains 43.04 TED-Sim and 31.76 hF1 versus 43.31 and 32.02 for full RePart on Breakfast-Part, can you report this ablation across all three datasets and in downstream anticipation? This would materially clarify whether the key contribution is recursive Bayesian composition or the proposed joint adaptation objective.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.