Recursive Predictive Abstraction for Unsupervised Temporal Partonomy Induction

We present $\text{RePart}$, an unsupervised framework for inducing temporal partonomies through recursive predictive abstraction. RePart couples a shared causal predictor with Bayesian composition evidence over current-level representations and prediction residuals, recursively merging adjacent units into increasingly coarse events without predefined hierarchy levels. The hierarchy is induced through discrete Bayesian composition, while predictive and composition-aware objectives shape the representation. On Breakfast Actions, 50Salads, and Assembly101, RePart achieves the best unsupervised hierarchical F1 on all three benchmarks and improves over PARSE on every TED-Sim/hF1 comparison, while remaining competitive on fine- and coarse-grained segmentation. Ablations show that recursive Bayesian composition provides a strong structural inductive bias, with joint state–residual evidence and representation adaptation further refining higher-level organization. In controlled Assembly101 action anticipation experiments, partonomy-conditioned models outperform fixed-scale models, with RePart remaining strong across increasing prediction horizons. These results show that recursively induced predictive structure provides coherent part–whole organization and useful context for downstream temporal reasoning. Code is in the supplementary material and will be released publicly after review.