acceptodds
Under review as a conference paper at ICLR 2027

Identifiability of Temporal Action Structure under Incomplete Observations

Abstract

We study when temporal action structure is identifiable from incomplete observations that can make missing evidence resemble genuine action change. We represent this structure by temporal boundaries and a cross-time category equivalence relation, distinguishing unique recovery from boundary bands and ambiguous relations. Our geometric analysis decomposes normalized masked affine-subspace distances into retained residual energy and coefficient-refitting leakage. Explicit conditioning and coherence assumptions yield residual-order guarantees; temporal margins then characterize when neighboring evidence helps and when boundary contamination reverses its benefit. Deterministic perturbation bounds connect ideal candidates to noisy, estimated models, while indistinguishable observations expose algorithm-independent limitations. TSCC implements the resulting masked-distance objective with adaptive temporal neighborhoods and safeguarded alternating updates. Existing experiments on five motion benchmarks support temporal aggregation under a supplied category count. The guarantees are conditional: neither objective descent nor a single predicted segmentation establishes unique recovery under arbitrary missingness.

Primary area:
learning on time series and dynamical systems
Keywords:
Temporal action segmentation, incomplete observations, masked subspace geometry, structural identifiability

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_5xDkOpenReview ↗
Rating
1(−1)
Confidence
3(+1)
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies the identifiability of temporal action structure under incomplete observations. The proposed method, Temporal Segmentation with Contextual Consistency (TSCC), combines masked subspace clustering with adaptive temporal neighborhood aggregation. Experiments on five motion datasets report improvements over various baselines under different missing-data settings.

Critical strengths
  1. The paper works on a relevant problem of temporal action segmentation under incomplete oobservations.
  2. The experiments cover multiple datasets and missingness patterns, with several baseline comparisons and ablation studies.
Critical weaknesses
  1. The proposed method largely combines established techniques in subspace clustering, missing-data handling, and temporal aggregation. The key technical innovation over existing approaches is not sufficiently clear.
  2. The extensive identifiability analysis is not directly reflected in the proposed method or experimentally validated through specific metrics.
  3. The paper is heavily dominated by mathematical derivations, without any figures in the main text. The overall methodology and its distinctive contributions are difficult to follow.
  4. The main text does not adequately explain the evaluation protocol and the interpretation of the reported scores. Some results require further clarification.
Questions for authors
  1. Could the authors clarify the key technical contributions beyond existing subspace clustering and temporal segmentation methods, particularly how the structural identifiability analysis contributes to the proposed TSCC algorithm?
  2. Could the authors explain why TSCC achieves higher performance (89.4% Acc) at 30% missingness than at 10% missingness (83.8% Acc) under the same coordinate MAR setting?
Reviewer_9dUQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the recovery of temporal action structure under incomplete observations, framing it as the identifiability of a boundary set B and an action equivalence relation Π. Using the set of admissible explanations and its structural projection F, the authors define complete, partial, and impossible identifiability. On the theoretical side, they derive a masked-distance identity that decomposes the normalized observed residual into retained energy and refitting leakage, and they prove masked residual preservation under uniform coordinate sampling. They also give temporal margin conditions specifying when temporal context helps and when boundary contamination hurts.The proposed method, TSCC, combines masked residuals with adaptive temporal neighborhoods. Experiments on Keck, Weiz, MAD, UT, and Mocap under coordinate, frame, joint, and tracking-failure missingness report consistent gains over existing methods.

Critical strengths
  1. The research perspective is valuable. The paper distinguishes between the “unique optimal solution to the objective function” and the “interpretation uniquely supported by the observed evidence,” and points out that missing data itself may masquerade as a transition between actions.
  2. The masked distance identity clearly reveals the source of error. Equation (4) attributes the bias that persists even after scaling by the observation ratio to refitting leakage, illustrating that the observation ratio alone is insufficient to characterize reliability.
  3. The temporal margin analysis is insightful. Equations (11) and (12) provide the conditions under which temporal context enhances the relative margin, as well as the tolerance threshold for cross-boundary contamination, explaining why the same temporal weight can stabilize frames within a segment while disrupting boundary frames.
Critical weaknesses
  1. Inconsistent experimental results. With 30% coordinate missingness, all five datasets in Table 6 outperform their complete-data results in Table 10. This contradicts the claim in Section 6.3 that performance degrades monotonically with missingness. Moreover, Appendix S6.2 states that Figure 3 and Table 6 report the same experiment, yet Keck at 30% is about 85 in Figure 3(a) but 92.1 in Table 6. The number of runs is also inconsistent: five in the main text versus twenty in Appendix S43. Finally, some rows of Table 1(b) are single-dataset results rather than macro-averages.
  2. The identifiability analysis is disconnected from the method. Theorems 1 and 2 only address the stability of nearest-subspace assignment for fixed candidate subspaces. They give neither conditions for |F| = 1 nor bounds on the width of uncertainty bands. TSCC itself outputs a single labeling and never computes F.3.
  3. Potential assignment bias from differing subspace dimensions. Subspace dimensions are selected independently by the eigengap rule. Per Eq. (4), higher-dimensional subspaces fit away more residual under missingness, while D/m_i corrects only energy, not degrees of freedom. This may bias assignments toward higher-dimensional subspaces. Have the authors considered this effect?
  4. The imputation baselines are too weak. All complete-data baselines rely only on linear interpolation. To support the claim that direct masked fitting outperforms impute-then-segment pipelines, the authors should compare against low-rank matrix completion or subspace-tracking imputation followed by temporal clustering.
  5. The theory does not cover the actual algorithm. Section 3.3 requires candidate subspaces to be independent of the masks, yet TSCC learns its subspaces from the same masked data. Could the authors report quantities such as restricted Gram eigenvalues and coherence on real data, to connect theory and experiments?
Questions for authors

Please refer to the weakness part.

Reviewer_iWHYOpenReview ↗
Rating
1(−1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
1(−1)
Read review
Paper summary

This paper studies temporal action segmentation under incomplete observations. It formalizes when the true segmentation is partially identifiable from masked data, providing impossibility results under mask collisions and sufficient conditions under uniform missingness, plus an energy decomposition identity that diagnoses imputation induced leakage. Based on this, the authors propose TSCC, an imputation free alternating optimization method based on masked residuals and neighborhood weighted feature updates. Validation is on five standard benchmarks.

Critical strengths
  1. Novel and well executed identifiability theory, including impossibility results, sufficient conditions, and a certification relation, which formalizes the critique of the impute then segment paradigm.
  2. Exceptional statistical rigor and scoping honesty, far above the norms of this field.
Critical weaknesses
  1. TSCC is a masked residual variant of TSC style alternating optimization, and the certificate machinery is never operationalized or empirically evaluated despite the abstract's claim that the method is guided by these conditions.
  2. Dated benchmarks, consisting of datasets from 2007 to 2014, archived hand crafted features, which limits ecological validity
Questions for authors
  1. Can the certificates of Section 4.5 be implemented as algorithm outputs, with the relation between certification coverage and actual error rate reported?
  2. Is the masked residual ranking stable with modern deep features?
  3. Has this paper been previously submitted to ICLR 2026 or another venue?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.