- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper works with unsupervised skeletion-based action localization. The authors puts a graph encoder in front of Clam so that it can run on skeletion sequences, trains on a subset of dataset with differenetiable clustering on frames to obtain an enhanced graph encoder . During testing, only the graph encoder is used for clustering per video with a BIC-driven splitting loop, which is termed as AutoClam. They report gains of a couple of points over SMQ and CLOT on BABEL, HuGaDB and LARa.
Critical strengths
The motivation for solving the problem of determining the number of cluster is meaningful, as it is a long standing problem in clustering. The authors tried out on the skeletion-based action benchmarks and show the effectiveness.
The ablation is comprehensive, in terms of loss terms, backbone, , on the k-choices (fixed, oracle) are good. It would be better if the oracle k comparsion can be put in the main paper.
Critical weaknesses
The major weaknesses is the technical novelty. It oversells what is new. The energy function, the attractor update, the softmax affinities, the masked pattern-completion loss, soft WCSS are not significantly different. The “skeleton-based” part is incidental, and the encoder is a standard GCN+MLP mixer. Compared to SMQ or LAC, nothing here exploits skeletal structure of the data itself.
The second is the framing and evalutaion. The task is temporal action segmentation, not localization. Every frame gets exactly one label, and metrics are standard in TAS rather than TAL. The comparison itself is also a biggest concern. As far as I know, the standard unsupervised TAS setting is transductive: train on all sequences and evaluation on the same. Yet this paper adopts the training/test splits, can the authors explain any specific reason why this is chosen? How would the approach perform on the entire dataset without splitting training and testing data?
Questions for authors
-
Please emphasize your technical contributions and its connections to the skeletion structure of the data.
-
Please explain why framing this as action localization despite the metric are segmentation metrics, as well as rationale for not following standard dataset level transductive paradigm.
-
Sec 3.5 seems out of context by introducing confidence estimation, yet nowhere else in paper refers to this.
Thank you for reviewing our paper thoroughly and providing constructive feedback — Q2's second half identified the most consequential problem in the submission.
Status: all three points resolved. Q1 contributions and skeleton dependence (§1–2, App S.3/R) · Q2 framing, retitled, and the protocol re-run, now primary (title, §4.1/4.3, Tab. 1–2) · Q3 §3.5, now evaluated (App Y).
Q1. Technical contributions and their connection to the skeleton structure.
Contribution: the clustering interface, not the backbone. We concede that the STMLP-Graph encoder is taken unchanged from Hassan & Hamza (2023) — uncited in the submission, now cited — and is not claimed as a contribution. Ours: (i) cluster assignment as continuous settling in a jointly trained associative memory, giving the encoder an affinity-weighted gradient over all prototypes, not just the nearest; (ii) a separation objective provably equal to confidence × relative margin, so it optimizes the quantity governing contiguity (Prop. 1); (iii) AutoClAM, resolving k per sequence under a localized BIC.
Skeleton connection — tested, not asserted (F1@50):
Removing the spatial graph costs 8.9 points, so body structure is not incidental. But a fully learnable adjacency recovers all but 1.8 — the anatomical prior is useful yet largely learnable, consistent with HuGaDB having no skeleton yet still working. We also tried part-structured masking over kinematic limbs, a genuinely skeleton-aware pretext task: it did not help (27.1 vs 27.3). We report it because it bounds how much of our performance is skeleton-specific. The paper now concludes the method is modality-general (IMU, mocap, RGB), not skeleton-specific.
→ §1–2, §4.5; App. S.3, R.
Q2. Why frame as localization when the metrics are segmentation metrics? And why not dataset-level transductive?
(a) Framing — conceded. Retitled to "…for Unsupervised Temporal Action Segmentation", UTAS throughout, and §4.1 opens with an explicit task definition: every frame receives one label from a discovered vocabulary, scored by MoF, Edit and F1@α — as distinct from localization, which emits possibly overlapping proposals scored by mAP@tIoU. Metrics and framing now agree; §3.5 is the bridge to that protocol and is evaluated there (Q3).
(b) Protocol — accepted, and the most consequential point raised. Baseline numbers were taken from SMQ (transductive, dataset-level Hungarian) while ours used an inductive split with per-video Hungarian, so the two were not directly comparable. We re-ran everything under the baselines' protocol; it is now primary. The full matrix, since both axes matter and interact (F1@50):
F1@50 as (dataset-level / per-video) — HuGaDB: transductive 30.2 / 34.5, inductive 26.8 / 27.7. BABEL S-1: transductive 29.4 / 32.6, inductive 25.6 / 27.3.
Per-video matching is worth +4.3 transductively on HuGaDB but only +0.9 inductively; the transductive regime +3.4 at dataset level and +6.8 per-video — so a comparison leaving either unstated is uninterpretable at our margins; every number now states both.
In the baselines' own cell AutoClAM reaches 30.2 F1@50 on HuGaDB (+5.9 over SMQ, +4.1 over CLOT, +1.9 over HiST-VQ) and 29.4 on S-1, exceeding every published method there by ≥4.0. The protocol-matched values sit a uniform +2.1 to +2.7 above the inductive ones on all five benchmarks — a protocol change, not a model change. Inductive rows are retained only as a capability result (frozen-encoder transfer to unseen subjects, unreported by prior work), never as a comparison. Provenance markers († copied, ‡ reproduced) make this visible per row.
→ Title, §4.1, §4.3, Tab. 1–2; App. H, K.
Q3. §3.5 is out of context and referenced nowhere else.
Agreed — it was. It is now load-bearing: C(Sc) is evaluated as a proposal score under the localization protocol, against uniform and duration-proportional baselines:
It improves mAP at every threshold on both benchmarks and more than halves calibration error (22.0→8.7, 20.7→7.2). §3.5 is thus the bridge to the localization protocol of Q2, which is what makes it worth keeping. The unsupported phrase "intrinsically calibrated" is replaced by the measured ECE.
→ §3.5, §4.1; App. Y.