acceptodds
Under review as a conference paper at ICLR 2027

SCLTrack: Structured Correspondence Learning for Category-Level Garment Pose Tracking

Abstract

Tracking category-level garment pose from partial point clouds requires correspondence to remain reliable despite large non-rigid deformation, self-contact, occlusion, and continuously changing observations. A key difficulty is that dense inter-frame features alone must simultaneously encode regional structure, instantaneous geometry, and persistent identity, although these cues follow different spatial and temporal behaviors. We present SCLTrack, a Structured Correspondence Learning framework that explicitly organizes these complementary cues before canonical correspondence estimation. SCLTrack introduces a semantic branch to capture region-level structural context, a geometry branch to preserve local metric relations, and a consistency branch that retrieves temporal evidence from an ordered canonical memory. Their point-wise representations are integrated with the base inter-frame feature through adaptive gated residual fusion. The enhanced representation is further refined in NOCS space and decoded through a canonical volumetric representation and implicit warp field, which also updates the canonical memory for subsequent frames. Extensive experiments demonstrate that SCLTrack significantly enhances the accuracy of garment pose tracking by up to 14.5% over GaPT-DAR, while reducing correspondence, surface, and NOCS errors by 2.1%, 6.5%, and 8.9%, respectively.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Category-Level Garment Pose Tracking, Non-Rigid Point Cloud Tracking, Superpoint Representation, Temporal Consistency

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_cp1rOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SCLTrack is a reasonably well-motivated architectural extension to an existing canonical-space garment tracking pipeline. Its three branches have distinct intended roles, and the memory update provides a mechanism for propagating correspondence information.

My main concern is that the manuscript establishes architectural plausibility more convincingly than it establishes the technical necessity and independent value of each design. There are also reproducibility questions concerning temporal training, inference, and the memory update.

Critical strengths
  1. Regional semantics, local geometry, and temporal identity address different failure modes in non-rigid correspondence.
  2. The method enriches the inter-frame feature before NOCS prediction rather than replacing the entire pipeline.
  3. Ordered canonical anchors provide a sensible indexing mechanism for retrieving historical evidence.
  4. The original representation remains available, which may reduce the risk of harmful auxiliary features.
  5. The method combines canonical-coordinate, mesh, surface, local-geometry, correspondence, and anchor-related objectives.
Critical weaknesses
  1. The manuscript states that training constructs a fresh previous-frame cache for each pair, whereas inference passes the updated cache sequentially. If the network is not adequately trained against accumulated prediction errors and stale or corrupted memory, the claimed temporal robustness may not hold over long sequences.
  2. The memory update queries the learned warp using fixed canonical anchors, then updates their task-space positions and features. If the warp becomes incorrect, the resulting memory can reinforce erroneous associations in subsequent frames. The current experiments do not sufficiently characterize this failure mode. In particular, there is no quantitative analysis of error accumulation over time or recovery after a tracking failure.
  3. The method uses nearby cached task-space positions to identify candidate anchors. This is a reasonable efficiency strategy, but it may exclude the correct anchor when a garment undergoes rapid motion or a region reappears far from its last observed position. The paper should clarify how candidate neighborhoods are constructed, how many anchors are retrieved, and whether the correct canonical identity remains available under large displacements.
  4. The full framework combines additional representations, gated fusion, recurrent memory, auxiliary anchor supervision, NOCS refinement, and volumetric reconstruction. The component ablations are useful, but do not fully disentangle architectural benefits from changes in supervision or training behavior.
  5. The paper defines numerous objectives and refers to their weighted sum, but the main text does not provide a complete, readily reproducible list of all loss weights and every important branch-specific configuration.
Questions for authors
  1. How does training expose the consistency branch to errors accumulated over many frames?
  2. What happens when a canonical anchor's predicted task-space position becomes substantially incorrect?
  3. How sensitive is performance to the number of anchors and the size of the memory retrieval neighborhood?
  4. Does the temporal branch improve identity recovery after a long occlusion, or mainly improve short-term feature aggregation?
  5. Can the full method be reproduced from the paper without relying on undocumented configuration files?
Reviewer_T1osOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Paper Summary

This paper studies category-level garment pose tracking from partial point cloud sequences. The authors argue that existing methods rely on dense inter-frame features to jointly encode regional structure, local geometry, and temporal identity. These cues become entangled. To address this issue, SCLTrack introduces three branches after the inter-frame feature fusion module of GarmentTracking:

  • Semantic branch: Uses adaptive superpoints and soft associations based on features and coordinates.
  • Geometric branch: Uses farthest point sampling (FPS) anchors and local KNN-based relative geometry.
  • Consistency branch: Maintains an ordered anchor memory with fixed canonical coordinates and updates it frame by frame using predicted deformation fields.

The three branches are combined through gated residual fusion. The fused features are then used for NOCS refinement and voxel-based implicit deformation reconstruction.

Experiments on four garment categories and two tasks in VR-Folding show that SCLTrack outperforms GaPT-DAR across all reported metrics.

Critical strengths

1. Consistent improvements across all evaluation settings.

SCLTrack outperforms GarmentNets, GarmentTracking, and GaPT-DAR across all reported metrics in the eight category-task combinations. On the folding task, the average A3 increases from 36.5% to 41.8%. The improvements are more noticeable under stricter evaluation thresholds.

2. Comprehensive ablation studies.

Tables 2–4 evaluate individual branches, pairwise combinations, and gated fusion. The authors also examine the internal designs of the superpoint, anchor, and memory modules. These experiments include hard groups, random anchors, and frozen positions or features. The results suggest that each branch contributes to the overall performance.

Critical weaknesses

1. The fairness of the comparisons is unclear, and additional training may confound the results.

The implementation details describe a fine-tuning configuration with Adam for 12 epochs, a learning rate of 5 × 10⁻⁶, and a backbone learning rate multiplier of 0.01. This suggests that SCLTrack starts from a pretrained checkpoint, possibly GarmentTracking, and receives additional training.

I checked the baseline row in Table 2. Its A3 (34.7%) and Dcorr (7.33) exactly match the average GarmentTracking results in Table 1. This raises the possibility that the authors directly reused the original baseline results without applying the same additional 12 epochs of fine-tuning.

As a result, it is difficult to determine how much of the improvement comes from the proposed modules rather than additional training or increased model capacity.

2. The improvements are relatively small, and the paper lacks variance estimates.

Compared with GaPT-DAR, the average Dcorr decreases only from approximately 7.24 cm to 7.09 cm, a relative improvement of 2.1%. The "up to 14.5%" improvement highlighted in the abstract refers to the relative gain in A3. This presentation may overstate the overall performance improvement.

Many differences between ablation settings are only around 0.01–0.05 cm (e.g., 7.09 vs. 7.10). The authors report results from single runs without multiple random seeds or confidence intervals. Therefore, the evidence is insufficient to establish that the three branches provide reliable complementary benefits.

The methodological novelty is also somewhat incremental. Superpoints, FPS anchors, and memory banks are established techniques, and the proposed method mainly combines these components. In addition, the quantitative evaluation uses only one synthetic dataset, while the real-world evaluation provides only qualitative results.

Questions for authors
  1. Are the baseline results in Tables 1 and 2 obtained through retraining, or are they taken from the original papers? Which checkpoint does SCLTrack use for fine-tuning? Could the authors provide two additional comparisons: GarmentTracking fine-tuned for the same 12 epochs and a parameter-matched baseline?

  2. Could the authors report the mean and standard deviation across multiple random seeds? This would help determine whether the small differences in Tables 2–4 are statistically reliable.

  3. During training, the authors construct the previous-frame cache independently for each sample pair. During inference, however, the cache is updated recursively across the sequence. This creates a mismatch between training and inference. Does the tracking error accumulate over long sequences? Could the authors provide a plot showing how the error changes with sequence length?

Reviewer_VaWnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents SCLTrack, a framework for category-level garment pose tracking from partial point clouds. The method introduces three complementary branches to capture regional structure, local geometry, and temporal identity through adaptive superpoints, geometric anchors, and canonical memory, respectively. These representations are combined through gated residual fusion and used for canonical correspondence estimation and garment reconstruction. Experiments on VR-Folding show consistent improvements over existing methods across four garment categories and both folding and flattening tasks. Overall, I find the approach well motivated and the results encouraging.

Critical strengths
  • The method addresses a meaningful problem with a clear design rationale.
  • The ablation studies are detailed and help explain the design choices. Beyond removing individual branches, the authors examine superpoint association, geometric sampling, canonical identity, memory retrieval, and state updates. These experiments provide useful evidence that the components contribute complementary information.
  • The canonical memory mechanism is appealing for tracking deformable objects. Maintaining fixed canonical identities while updating their task-space states provides a natural way to carry information across frames.
  • The reported inference speed of 21 FPS also suggests potential for practical use.
Critical weaknesses
  • Quantitative evaluation is limited to VR-Folding. The real-world examples are encouraging, but quantitative results on captured sequences, or controlled tests with depth noise and missing observations, would strengthen the evidence for practical robustness.
  • The contribution of the training objectives could be clearer. The framework includes several auxiliary losses in addition to the three branches. Clarifying which objectives are shared across the Table 2 variants, and providing a small loss ablation, would help distinguish gains from structured representations and additional supervision.
Questions for authors

Nil.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.