acceptodds
Under review as a conference paper at ICLR 2027

AutoClAM: End-to-End Differentiable Deep Clustering for Unsupervised Action Localization

Abstract

Unsupervised Temporal Action Localization (UTAL) aims to discover and segment action instances within untrimmed videos without ground-truth annotations. Existing clustering-based pipelines typically rely on disjoint two-stage frameworks that apply discrete, non-differentiable clustering (e.g., k-means) to frozen representations. Such discrete assignments inherently sever the computational graph, preventing gradient propagation from the localization objective back to the feature encoder. To overcome this limitation, we propose a novel end-to-end differentiable deep clustering framework for skeleton-based UTAL that seamlessly unifies representation learning and temporal boundary discovery. We formulate temporal clustering as a continuous energy-minimization process by coupling a Spatio-Temporal Graph Encoder with Continuous Local Associative Memories (ClAM). During training, our model jointly optimizes the feature manifold and continuous action prototypes via a composite differentiable objective combining masked pattern reconstruction with continuous soft-relaxations for intra-cluster compactness and inter-cluster separation. Furthermore, to eliminate the field's rigid reliance on pre-defined cluster counts, we introduce the Automated Continuous Local Associative Memory (AutoClAM) inference module. Operating strictly at inference time, AutoClAM employs recursive splitting guided by the Bayesian Information Criterion (BIC) to dynamically discover the optimal number of distinct underlying motions within each unseen sequence. Extensive evaluations across three complex skeleton benchmarks (HuGaDB, BABEL, and LARa) demonstrate that our framework consistently outperforms state-of-the-art methods, including recent Optimal Transport baselines, especially at strict overlap thresholds (F1@50). We further show that our continuous formulation exhibits linear computational complexity O(T * k * D) and empirically generalizes to dense RGB video features. Code will be released upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Unsupervised Learning, Deep Clustering, Video Understanding, Temporal Action Localization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TyTmOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper works with unsupervised skeletion-based action localization. The authors puts a graph encoder in front of Clam so that it can run on skeletion sequences, trains on a subset of dataset with differenetiable clustering on frames to obtain an enhanced graph encoder . During testing, only the graph encoder is used for clustering per video with a BIC-driven splitting loop, which is termed as AutoClam. They report gains of a couple of points over SMQ and CLOT on BABEL, HuGaDB and LARa.

Critical strengths

The motivation for solving the problem of determining the number of cluster is meaningful, as it is a long standing problem in clustering. The authors tried out on the skeletion-based action benchmarks and show the effectiveness.

The ablation is comprehensive, in terms of loss terms, backbone, β\beta, on the k-choices (fixed, oracle) are good. It would be better if the oracle k comparsion can be put in the main paper.

Critical weaknesses

The major weaknesses is the technical novelty. It oversells what is new. The energy function, the attractor update, the softmax affinities, the masked pattern-completion loss, soft WCSS are not significantly different. The “skeleton-based” part is incidental, and the encoder is a standard GCN+MLP mixer. Compared to SMQ or LAC, nothing here exploits skeletal structure of the data itself.

The second is the framing and evalutaion. The task is temporal action segmentation, not localization. Every frame gets exactly one label, and metrics are standard in TAS rather than TAL. The comparison itself is also a biggest concern. As far as I know, the standard unsupervised TAS setting is transductive: train on all sequences and evaluation on the same. Yet this paper adopts the training/test splits, can the authors explain any specific reason why this is chosen? How would the approach perform on the entire dataset without splitting training and testing data?

Questions for authors
  1. Please emphasize your technical contributions and its connections to the skeletion structure of the data.

  2. Please explain why framing this as action localization despite the metric are segmentation metrics, as well as rationale for not following standard dataset level transductive paradigm.

  3. Sec 3.5 seems out of context by introducing confidence estimation, yet nowhere else in paper refers to this.

1 comment

Thank you for reviewing our paper thoroughly and providing constructive feedback — Q2's second half identified the most consequential problem in the submission.

Status: all three points resolved. Q1 contributions and skeleton dependence (§1–2, App S.3/R) · Q2 framing, retitled, and the protocol re-run, now primary (title, §4.1/4.3, Tab. 1–2) · Q3 §3.5, now evaluated (App Y).

Q1. Technical contributions and their connection to the skeleton structure.

Contribution: the clustering interface, not the backbone. We concede that the STMLP-Graph encoder is taken unchanged from Hassan & Hamza (2023) — uncited in the submission, now cited — and is not claimed as a contribution. Ours: (i) cluster assignment as continuous settling in a jointly trained associative memory, giving the encoder an affinity-weighted gradient over all prototypes, not just the nearest; (ii) a separation objective provably equal to confidence × relative margin, so it optimizes the quantity governing contiguity (Prop. 1); (iii) AutoClAM, resolving k per sequence under a localized BIC.

Skeleton connection — tested, not asserted (F1@50):

AdjacencyMaskingS-1LARa
Identity (no spatial graph)coord-wise18.413.7
Complete graphcoord-wise23.216.5
Fully learnablecoord-wise25.519.0
Anatomicalcoord-wise27.320.8
Anatomicalpart-structured27.120.6

Removing the spatial graph costs 8.9 points, so body structure is not incidental. But a fully learnable adjacency recovers all but 1.8 — the anatomical prior is useful yet largely learnable, consistent with HuGaDB having no skeleton yet still working. We also tried part-structured masking over kinematic limbs, a genuinely skeleton-aware pretext task: it did not help (27.1 vs 27.3). We report it because it bounds how much of our performance is skeleton-specific. The paper now concludes the method is modality-general (IMU, mocap, RGB), not skeleton-specific.

→ §1–2, §4.5; App. S.3, R.

Q2. Why frame as localization when the metrics are segmentation metrics? And why not dataset-level transductive?

(a) Framing — conceded. Retitled to "…for Unsupervised Temporal Action Segmentation", UTAS throughout, and §4.1 opens with an explicit task definition: every frame receives one label from a discovered vocabulary, scored by MoF, Edit and F1@α — as distinct from localization, which emits possibly overlapping proposals scored by mAP@tIoU. Metrics and framing now agree; §3.5 is the bridge to that protocol and is evaluated there (Q3).

(b) Protocol — accepted, and the most consequential point raised. Baseline numbers were taken from SMQ (transductive, dataset-level Hungarian) while ours used an inductive split with per-video Hungarian, so the two were not directly comparable. We re-ran everything under the baselines' protocol; it is now primary. The full matrix, since both axes matter and interact (F1@50):

F1@50 as (dataset-level / per-video) — HuGaDB: transductive 30.2 / 34.5, inductive 26.8 / 27.7. BABEL S-1: transductive 29.4 / 32.6, inductive 25.6 / 27.3.

Per-video matching is worth +4.3 transductively on HuGaDB but only +0.9 inductively; the transductive regime +3.4 at dataset level and +6.8 per-video — so a comparison leaving either unstated is uninterpretable at our margins; every number now states both.

In the baselines' own cell AutoClAM reaches 30.2 F1@50 on HuGaDB (+5.9 over SMQ, +4.1 over CLOT, +1.9 over HiST-VQ) and 29.4 on S-1, exceeding every published method there by ≥4.0. The protocol-matched values sit a uniform +2.1 to +2.7 above the inductive ones on all five benchmarks — a protocol change, not a model change. Inductive rows are retained only as a capability result (frozen-encoder transfer to unseen subjects, unreported by prior work), never as a comparison. Provenance markers († copied, ‡ reproduced) make this visible per row.

→ Title, §4.1, §4.3, Tab. 1–2; App. H, K.

Q3. §3.5 is out of context and referenced nowhere else.

Agreed — it was. It is now load-bearing: C(Sc)\mathcal C(S_c) is evaluated as a proposal score under the localization protocol, against uniform and duration-proportional baselines:

BenchmarkScore[email protected][email protected]ECE↓
HuGaDBuniform21.07.322.0
duration22.78.418.0
affinity (ours)27.913.38.7
S-1uniform23.810.220.7
duration25.611.916.9
affinity (ours)31.215.87.2

It improves mAP at every threshold on both benchmarks and more than halves calibration error (22.0→8.7, 20.7→7.2). §3.5 is thus the bridge to the localization protocol of Q2, which is what makes it worth keeping. The unsupported phrase "intrinsically calibrated" is replaced by the measured ECE.

→ §3.5, §4.1; App. Y.

Reviewer_V5NSOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes AutoClAM, an unsupervised temporal action localization framework that combines a spatiotemporal graph encoder with a continuous associative memory module. Experiments on BABEL, HuGaDB, and LARa show improvements over several unsupervised action localization baselines. The paper also reports ablation studies on the clustering mechanism, loss functions, encoder backbone, and computational efficiency.

Critical strengths

The framework jointly optimizes feature representations and clustering rather than applying a separate clustering algorithm to frozen features. Experiments on BABEL, HuGaDB, and LARa provide some evidence that the method is not limited to a single benchmark. The paper also examines the choice of encoder and sensitivity to selected hyperparameters.

Critical weaknesses
  1. The paper presents continuous associative memory as the central mechanism, but the relationship between its update process, temporal boundary estimation, and the observed performance gains is not sufficiently established. The comparisons with k-means and soft k-means are useful, but they do not fully isolate the contribution of recurrent memory updates from the effects of the proposed loss functions and learned representations.

  2. The theoretical argument for temporal continuity appears insufficient. Equation (2) uses a Lipschitz continuity argument to motivate the behavior of adjacent frames in the learned feature space. However, a bound on changes in the embeddings does not guarantee that adjacent frames receive the same cluster assignment. Small changes in the feature space may still cross a decision boundary. The paper needs a more direct justification of how the proposed formulation reduces temporal fragmentation.

  3. The adaptive cluster discovery procedure needs stronger validation. AutoClAM uses recursive BIC-guided splitting to estimate the number of action clusters. The paper includes additional cluster-count analyses, but it remains important to establish how reliably this procedure works across sequences with different lengths, action distributions, and degrees of temporal ambiguity. In particular, improvements in localization scores do not by themselves establish that the discovered cluster counts are accurate.

  4. The related work cites the 2026 CVPR paper Unsupervised Skeleton-Based Action Segmentation via Hierarchical Spatiotemporal Vector Quantization, but this method does not appear in the main comparison tables. Its omission makes it difficult to assess the proposed approach against recent skeleton-based unsupervised segmentation methods.

Questions for authors
  1. In Equation (9), how is the prototype index μ∗\mu^* selected, and how are gradients handled through this selection? Please specify which parts of the proposed clustering objective are differentiable in the implemented training procedure.

  2. What conditions are required for the Lipschitz argument in Equation (2) to support temporal consistency in cluster assignments? Can the authors provide a more direct analysis or quantitative measure of temporal fragmentation?

  3. Why is the 2026 CVPR method Unsupervised Skeleton-Based Action Segmentation via Hierarchical Spatiotemporal Vector Quantization not included in the main comparisons? Can the authors provide results under a consistent evaluation protocol

  4. The paper claims generalization to dense RGB video features. Since the RGB pathway uses a different feature-processing design from the skeleton pathway, what evidence demonstrates that the proposed clustering mechanism generalizes across modalities, rather than that the framework can be adapted to a second type of input representation?

  5. Can the authors revise the method figures to show the actual tensor flow, prototype update equations, and recursive splitting decisions? The current diagrams should allow readers to trace the implementation from the input sequence to the final temporal clusters without having to infer key operations from abstract module labels.

3 comments

Thank you for reviewing our paper thoroughly and providing constructive feedback — five questions that produced concrete new experiments; Q1 found a description in our text that did not match the implementation.

Status: all points resolved, including the two weaknesses outside your questions: the NstepsN_{steps} sweep isolates recurrent settling (1 step ≡ soft k-means; F1@50 22.0→27.7, App S.7), and k reliability is stratified by length and count (MAE 0.67→0.92, 0.50→0.97; App S.5).

Q1. In Eq. 9, how is μ∗\mu^\ast selected, how are gradients handled, and what is differentiable?

μ∗\mu^\ast is an argmin: μ∗(t)=arg⁡min⁡μ∥et−ρμ∥2\mu^\ast(t)=\arg\min_\mu\Vert e_t-\rho_\mu\Vert^2. Our submitted wording described btb_t as avoiding an argmin; that was inaccurate — masking the primary logit is one — and is corrected. The implementation, as always: the index is piecewise constant and detached in the backward pass, gradients flowing through the distances, as in standard k-means.

We also made the motivation precise: we implied hard assignment is non-differentiable, whereas it is differentiable a.e.; the real issue is that its gradient sits on a single prototype, blind to the affinity landscape that sets boundaries. Logger files are available on request.

A new per-operation differentiability audit lists what carries gradients (softmax affinity, ClAM update, all four losses, LSil\mathcal L_{Sil} index detached) and what is inference-only (label argmax, BIC, subset isolation).

→ §1, §3.3; App. T, E.

Q2. What conditions would make the Eq. 2 Lipschitz argument work? A direct fragmentation measure?

No useful condition exists, hence the deletion: a uniform bound sufficient for contiguity needs δt<γt/2\delta_t\lt\gamma_t/2 at every t, which forbids any boundary — it proves too much. Prop. 1 is instead per-frame and conditional, its violating frames exactly the boundaries, and LSil\mathcal L_{Sil} is a differentiable surrogate for γt\gamma_t, so the objective acts on the inequality rather than assuming it.

Measured, encoder identical across rows:

Variant% δt<γt/2\delta_t\lt\gamma_t/2Npred/NgtN_{pred}/N_{gt}Flips/100Bnd. F1
Hard k-means51.11.750.697.5
Soft k-means61.51.520.599.0
ClAM w/o LSil\mathcal L_{Sil}67.51.360.5210.6
ClAM (full)78.31.190.4514.3

All four improve together; removing LSil\mathcal L_{Sil} degrades all four.

→ §3.1; App. D, U.

Q3. Why is HiST-VQ absent? Results under a consistent protocol?

It is now in both main tables, with MASQ; DuoCLR and Spurio et al. are cited in §2. Both rows are marked † (from the publication), so no significance is asserted against them.

On consistent protocol: primary tables are now transductive + dataset-level Hungarian, HiST-VQ's own setting, so the comparison is protocol-matched (previously not; see TyTm-Q2):

HuGaDB F1@50S-1 F1@10S-2 F1@25S-2 F1@50
Ours30.248.145.033.2
HiST-VQ28.344.541.130.4

One comparability issue: HiST-VQ evaluates the 14-participant LARa release; we use version 3 with 16 — distinct corpora, so that cell is not comparable. HiST-VQ leads S-2 Edit/F1@10 and HuGaDB frame accuracy, which we report.

→ Tab. 1–2, §2; App. K.

Q4. Does the clustering mechanism generalize, or is the framework merely adaptable?

Directly: on RGB the spatial graph module is bypassed entirely. I3D features go straight to the temporal path — no adjacency, no joint structure — so only the clustering interface transfers, exactly your distinction. One pipeline, all five methods; excerpt:

MethodBreakfast MoFF1@5050Salads MoFF1@50
CLOT43.331.852.038.7
AutoClAM48.236.957.145.9

We qualify this sharply: our reproductions of all four baselines fall 16–17 points below their published Breakfast MoF, uniformly — a systematic difference in our RGB pipeline. So we claim only what the data supports: under a controlled pipeline the interface transfers and orders above the OT baselines — not that we beat them as published.

→ App. R (incl. the RGB cross-check table).

Q5. Revise the figures to show tensor flow, prototype updates and splitting decisions.

Both are fully redrawn as vector diagrams: tensor shapes at each stage (T×J×C → T×D → T×k → T), the settling and affinity equations inlined, the prototype bank and masking as separate stages, the split→refine→BIC-test cycle explicit, and gradient-carrying paths (solid) vs inference-only (dashed).

Two real errors surfaced and are fixed: the flowchart drew the accept test inverted as BICchild>BICparentBIC_{child}\gt BIC_{parent} (Eq. 7 is a deviance form, so a split is accepted only if the criterion decreases), and the rasters misspelled the module.

→ Fig. 1–2; App. A, B.

Reviewer_V5NSOpenReview ↗

Reponse to Authors

The response is comprehensive and the revisions appear to have substantially improved the methodological clarity and experimental reliability of the paper. I have no further major concerns, provided that the revised manuscript and supplementary materials consistently incorporate the proposed corrections and additional analyses.

Thank you very much for your careful review, and for taking the time to return and confirm the revision. We can confirm that every correction and additional analysis described in our response is already incorporated into the revised manuscript and supplementary materials, and applied consistently throughout. Thank you again for the attention you have given our work.

Reviewer_EdJEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes AutoClAM, an end-to-end differentiable deep clustering framework for unsupervised temporal action localization (UTAL) based on human skeletal motion sequences. This approach aims to address two limitations inherent in existing clustering-based methods: the decoupling of representation learning from the clustering process, and the requirement to pre-specify the number of action clusters.

Critical strengths
  1. The research problem has practical significance. 2. The overall design of the methodology is relatively comprehensive and reasonably sound. 3. Performance improvements have been reported across multiple datasets.
Critical weaknesses
  1. There is a potential fundamental contradiction in the optimization objectives during the inference phase. Section 3.4 specifies that stochastic masking is disabled during the local optimization process in the inference phase, while simultaneously employing the mask-reconstruction loss (Equation 7) to optimize the split cluster prototypes. However, Equation 7 calculates the reconstruction error exclusively at the masked feature positions. If disabling masking implies retaining all features (i.e., m=1m=1), the reconstruction loss becomes identically zero, failing to provide a valid gradient for prototype optimization. This issue directly calls into question whether the AutoClAM recursive splitting algorithm can function as described in the paper.

  2. The innovation is relatively limited compared to existing differentiable clustering methods. The core concept of utilizing associative memory to achieve end-to-end differentiable clustering was previously introduced by Saha et al. (ICML 2023)—a work that the authors cite. The proposed approach primarily builds upon this existing associative memory-based clustering mechanism by incorporating a spatiotemporal graph encoder, standard clustering regularization losses, and a BIC-based model selection criterion. The paper fails to clearly disentangle the respective contributions of the learned prototypes and the frozen encoder to the final performance.

  3. The theoretical justification for temporal continuity is insufficient. Section 3.1 posits that features from adjacent frames satisfy Lipschitz continuity, which is claimed to naturally ensure that these frames are assigned to the same cluster prototype, thereby forming temporally continuous action segments. However, a small distance between adjacent features does not guarantee identical cluster assignments; even features in close proximity may lie on opposite sides of a clustering decision boundary. Consequently, Equation 2 is inadequate to substantiate the authors' claims regarding temporal continuity. Furthermore, the current clustering objective lacks an explicit temporal consistency constraint. The paper needs to further distinguish between the feature smoothness inherently provided by the spatiotemporal encoder and any actual improvements in temporal continuity derived from the associative memory-based clustering mechanism.

  4. There are several inconsistencies in the experimental results, which undermine the credibility of the main conclusions. There are numerical discrepancies in some of the experimental tables in the paper that require clarification: (1) Table 1: For CLOT on the BABEL Subset-1 dataset, the F1@10 score is 36.8, while the F1@25 score is 43.1. According to the standard Segmental F1 evaluation protocol described in the paper, the F1 score should not increase as the IoU threshold rises. Furthermore, AutoClAM’s F1@25 score is 38.7, lower than the 43.1 reported for CLOT in the same table, which contradicts the paper’s claim of achieving superior performance across the board. (2) Table 10: On the HuGaDB dataset, AutoClAM achieves an accuracy of 44.2, compared to 55.2 for DeepCluster-Inertial; on the LARa dataset, the corresponding figures are 39.6 and 50.1. Consequently, the cross-modal experiments do not support the broad conclusion that the proposed method outperforms the baselines across all metrics. (3) Table 12: The average F1@10 score on HuGaDB (based on ten random initializations) is 49.4, whereas Table 2 reports a value of 42.7, even though other F1 metrics remain largely consistent. The authors should clarify whether different experimental settings were employed in these instances.

Questions for authors
  1. How exactly are the clustering prototypes optimized when the reconstruction mask is disabled during the inference phase?
  2. What is the core algorithmic innovation of this paper compared to existing associative memory-based clustering methods and traditional model selection approaches?
  3. In the absence of explicit temporal constraints, how do the authors demonstrate that the model can ensure temporal continuity?
  4. Can the authors explain the numerical discrepancies found in the experimental tables and provide more reliable experimental results?
1 comment

Thank you for reviewing our paper thoroughly and providing constructive feedback — four questions that go to the core of the method; Q1 caught a write-up inconsistency we had missed.

Status: all four points resolved. Q1 masking (§3.4, App S.7) · Q2 novelty + disentanglement (§4.5, App S.1) · Q3 contiguity → Prop. 1 (§3.1, App D) · Q4 the three numeric errors (Tab. 1–2).

Q1. How are prototypes optimized when the mask is disabled at inference?

By the full unmasked reconstruction, now its own equation:

Lreconlocal=1∣Eμ∣D∥Eμ−ClAM(Eμ;Pμ)∥F2\mathcal L_{recon}^{local}=\frac{1}{\vert E_\mu\vert D}\Vert E_\mu-\mathrm{ClAM}(E_\mu;P_\mu)\Vert_F^2

You are right that the submitted text was contradictory: Eq. 7 is supported only on withheld coordinates, so with m ≡ 1 it is identically zero. Eq. 7 is the training objective only; omitting the separate local equation was a write-up oversight, and the implementation has always optimized the unmasked form. Masking is an anti-collapse bottleneck needed only while prototypes are learnable — at inference the encoder is frozen and the local two-prototype problem is over-determined.

Ablating all three local objectives (HuGaDB): the choice does matter, and the implemented one is best:

Local objectiveMAE(k)↓F1@50↑
Lreconlocal\mathcal L_{recon}^{local} unmasked (ours)0.727.7
Lrecon\mathcal L_{recon}, fixed deterministic mask0.925.7
LWCSS\mathcal L_{WCSS} only (≡ soft 2-means)1.124.5

→ §3.4; App. S.7.

Q2. Core innovation vs associative-memory clustering and classical model selection?

vs associative-memory clustering (Saha et al. 2023; DEC): they cluster static images at fixed k. Ours is the sequential setting — a margin account of contiguity (Prop. 1), a separation objective that is its differentiable surrogate, and k resolved per sequence at inference.

vs classical model selection (X-means, G-means, DeepDPM; now cited): these apply BIC or a Gaussianity test to fixed features, once per dataset. We apply the X-means criterion to the attractor geometry of a jointly trained memory, per sequence, in a PCA subspace of the isolated subset.

Both are measured. Disentanglement, 6 seeds, HuGaDB F1@50: joint objective +9.4; end-to-end gradient flow +2.6 (Welch's t, p = 2.0×10⁻³); per-sequence re-derivation +4.2. Factorial, identical encoder and loss, S-1 F1@50: recurrent settling +6.3 over hard assignment, BIC-selected k a further +2.5 — near-additive, so neither alone explains the gap.

→ §1–2, §4.5; App. S.1, S.4.

Q3. Without explicit temporal constraints, how is continuity demonstrated?

We replaced Eq. 2 with a provable condition. Prop. 1: with γt=d(2)(t)−d(1)(t)\gamma_t=d_{(2)}(t)-d_{(1)}(t) and δt=∥et+1−et∥\delta_t=\Vert e_{t+1}-e_t\Vert, if δt<γt/2\delta_t\lt\gamma_t/2 then μ∗(t+1)=μ∗(t)\mu^\ast(t{+}1)=\mu^\ast(t). Smoothness alone is insufficient — two points can be close yet straddle a Voronoi boundary — so contiguity needs smoothness relative to the margin. The encoder shrinks δt\delta_t; LSil\mathcal L_{Sil} controls γt\gamma_t, admitting the closed form −1T∑tAt,μ∗(1−dμ∗/bt)-\frac1T\sum_t A_{t,\mu^\ast}(1-d_{\mu^\ast}/b_t) = confidence × relative margin. Unlike Eq. 2, it permits boundaries: the frames with δt≥γt/2\delta_t\ge\gamma_t/2.

We then measure it, encoder identical across rows:

Variant% δt<γt/2\delta_t\lt\gamma_t/2Npred/NgtN_{pred}/N_{gt}Bnd. F1
Hard k-means51.11.757.5
Soft k-means61.51.529.0
ClAM w/o LSil\mathcal L_{Sil}67.51.3610.6
ClAM (full)78.31.1914.3

Removing LSil\mathcal L_{Sil} costs 10.8 points of coverage — direct evidence the separation term supplies the margin. Npred/NgtN_{pred}/N_{gt} also makes the no-post-processing claim checkable: a smoothing filter would drive it toward 1.

→ §3.1; App. D, U.

Q4. The numerical discrepancies.

All three are confirmed; in each case the cause was transcription from our logs into the tables, not the runs themselves:

  • Tab. 1, CLOT S-1: the F1@10/F1@25 entries were transposed during typesetting. The ordering is in fact a hard identity — F1@α=2TP(α)/(Npred+Ngt)F1@\alpha=2TP(\alpha)/(N_{pred}{+}N_{gt}) has an α-independent denominator and non-increasing TP, so F1@10 ≥ F1@25 ≥ F1@50 always. Now 43.1/36.8/25.4.
  • Tab. 10: a bolding error, with an "Acc" column not comparable across rows. Deleted, replaced by RGB results with standard metrics.
  • Tab. 12: the F1@10 column was read from the wrong configuration's log. Regenerated; all four means now match Tab. 2 to rounding.

Every table is now generated under automated assertions (F1 monotonicity; every bold cell is its column best), so the class cannot recur. It also caught six further write-up issues we fixed unprompted, including an inverted BIC inequality in Fig. 2. We can supply the per-seed logger files for any table, so every value can be checked directly.

→ Tab. 1–2; App. H, S.5.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear all,

Thank you for three detailed and, in our assessment, accurate reviews. We have treated them as a correction list rather than as something to argue with: where a reviewer identified a flaw, we fixed the flaw. Every point raised is now resolved. Per-reviewer replies follow; this note summarizes the revision.

Three changes we consider decisive.

1. Evaluation protocol (TyTm). Baseline numbers were taken from SMQ (transductive, dataset-level Hungarian) while ours were produced under an inductive split with per-video Hungarian, so the headline comparison was not protocol-matched. We re-ran everything in the baselines' own setting and it is now primary: AutoClAM reaches 30.2 F1@50 on HuGaDB and 29.4 on BABEL S-1 there. §4.3 reports the full {transductive, inductive} × {dataset-level, per-video} matrix, and every row now carries a provenance marker († copied, ‡ reproduced).

2. Inference objective (EdJE). Eq. 7 is supported only on withheld coordinates, so with masking disabled it is identically zero — as EdJE showed, the splitting step could not function as written. The implemented objective, the full unmasked local reconstruction, is now its own equation in §3.4 and ablated in App S.7.

3. Temporal contiguity (all three reviewers). The Eq. 2 Lipschitz argument is deleted: it proved too much, since a uniform bound forbids any boundary at all. Proposition 1 replaces it with a per-frame margin condition (proof, App D), and LSil\mathcal L_{Sil} is shown to equal confidence × relative margin, so the objective acts on the quantity governing contiguity instead of assuming it. App U measures it.

Closure of every remaining point.

Point raisedResolutionWhere
Novelty vs associative-memory clustering (EdJE, TyTm)sequential setting, margin account, per-sequence k; X-means, G-means, DeepDPM now cited§1–2
Encoder and prototypes not disentangled (EdJE)five-way disentanglement plus a factorial, encoder held fixed§4.5, App S.1
Recurrent update not isolated (V5NS)one settling step is exactly soft k-means, so the sweep isolates itApp S.7
Reliability of k (V5NS)stratified by length, action count and duration imbalance; MAE 0.72App S.5
HiST-VQ absent (V5NS)added to both main tables with MASQ; DuoCLR and Spurio et al. citedTab. 1–2, App K
What is differentiable (V5NS)per-operation audit; μ∗\mu^\ast is an argmin, index detached§3.3, App T
RGB generalization (V5NS)Breakfast and 50Salads with I3D, our reproduction gap statedApp R
Figures uninformative (V5NS)redrawn as vectors with tensor shapes and stop-gradients markedFig. 1–2
Encoder attribution (TyTm)Hassan & Hamza (2023) cited, not claimed as a contribution§3.1
Skeleton structure incidental (TyTm)adjacency ablation; part-structured masking reported as a negative resultApp S.3
Task framing (TyTm)retitled to Segmentation, UTAS throughout, task definition addedTitle, §4.1
§3.5 orphaned (TyTm)evaluated as a proposal score, with mAP@tIoU and ECEApp Y

Main paper. Retitled. §3.1 carries Proposition 1; §3.3 states the closed form and the detached index; §3.4 states the unmasked local objective and a corrected, dimensionally consistent BIC; §4.1 adds the task definition, both matching schemes and the previously missing hyperparameters; §4.3 is the protocol matrix; §4.5 is restructured around disentanglement and the factorial. §4 prose and captions were condensed to stay inside the page limit.

Appendix. New sections: proof of Proposition 1 (D), baseline comparability (K), differentiability audit (T), margin and fragmentation analysis (U), extended ablation studies (S), modality generalization including RGB (R), confidence and localization-style evaluation (Y), and Reproducibility and Ethics statements (W, X).

Corrections. All three numerical discrepancies EdJE identified are confirmed and fixed; in each case the cause was transcription from our logs into the tables rather than the runs themselves. Tab. 1's CLOT row was transposed, Tab. 10 is deleted, and Tab. 12 is regenerated and now agrees with Tab. 2. Every table is now generated under automated assertions (F1 monotonicity; every bold cell is its column best), and the same audit caught six further write-up issues we fixed unprompted, including an inverted BIC inequality in Fig. 2. We can supply the per-seed logger files for any table, so every value can be checked directly.

We are grateful for reviews this specific; the paper is materially stronger for them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.