acceptodds
Withdrawn from ICLR 2027

Dense4D: Pixel-Level 4D Human Correspondence

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reconstructing human motion in 4D requires identifying the same surface points through articulation, deformation and changes in visibility. We introduce Dense4D, to our knowledge the first feed-forward model dedicated to dense 4D correspondence across the body, clothing and hair from RGB images, without a body template. Trained specifically for human geometry, Dense4D jointly processes person-centred crops with known image-to-crop transforms to predict Human Point Maps (HPMs) and depth for each image. An HPM assigns each person pixel the 3D position of its observed surface point at a common reference time. These maps support amodal correspondence: points hidden in the reference image can be localised from observations in other frames. Matching HPM coordinates across crops establishes correspondence without predicting camera poses or large scene displacements; per-image depth then lifts the matches into 3D tracks. Experiments on real people demonstrate substantially higher tracking accuracy and closer agreement with scanned surfaces than the evaluated baselines. Points from different frames assemble into coherent human geometry in the reference pose, providing visible evidence of consistent dense correspondence through articulation and clothing deformation. Videos of dense 4D correspondence and tracking: https://anonymous.4open.science/w/research-materials-F6D0/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
4D Dense Correspondence, Human Point Maps, 4D reconstruction, dense correspondence, 4D point tracking, clothed humans, canonical representation, feed-forward 3D reconstruction, monocular video

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Jx2JOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Dense4D proposes a feed-forward model for dense correspondence and 3D tracking of humans, including their body, clothing, and hair, without fitting a parametric body template.

The main representation is a Human Point Map (HPM) where each observed person pixel in a frame t predicts the 3D position of that surface point at a common reference time (t=0). The proposed network takes cropped images of humans and produces HPMs and per-frame depth maps. Correspondences are obtained by nearest neighbour matching in reference space, then lifted into 3D using the predicted depth.

Critical strengths
  • Human tracking and dense human correspondences is a relevant and important problem. The paper proposes a method that naturally extends to loose clothing and hair.

  • The paper does extensive evaluations and the provided in the wild examples are also very promising.

Critical weaknesses
  • Dense4D adapts a large pretrained reconstruction backbone, VGGT-Omega using extensive human-specific supervision. Most comparisons are against general-purpose dynamic reconstruction or tracking systems. The improvements could therefore reflect a combination of domain-specialised training, person-centred processing, the reference-space representation, and the matching-based readout. Hence, a matching baseline also trained on human datasets would better establish what HPMs contribute. In particular using crops for human prediction tasks itself is not novel and has been used by previous human mesh recovery pipelines as well. Hence, HPM seems to be the main contribution. To validate this, finetuning VGGT-Omega on the same dataset without HPM would provide a good baseline.

  • As also noted by the authors, the reference frame selection can be critical for the overall performance of the method. More thorough evaluation about this, especially how this selection effects cases with multiple people interacting etc., will be useful.

  • While the paper does not predict human meshes, it still trains on human specific datasets. Hence, I think in addition to general purpose tracking methods, comparisons to SOTA human mesh recovery methods is needed.

My current slightly negative rating is mostly due to concerns regarding experiments to really pin-point the main contribution of the work.

Questions for authors
  1. For the 4D-Dress protocol, is it useful also measure the error in the other direction, i.e., from the reference scan to the predictions to make sure the target predictions don't localize in small regions of the body?

  2. How much of the gain remains against a baseline trained on the same human data, with the same backbone, crops, and training budget, but using direct motion prediction or a conventional dynamic-point-map formulation?

  3. What happens to point identity after extended occlusion and reappearance?

Reviewer_keLxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Dense4D addresses dense correspondence and 3D tracking of human surfaces, including clothing and hair, without a fixed body template. It predicts Human Point Maps at a selected reference time together with per-frame depth, using person-centered crops and crop geometry conditioning. Reference-space matching followed by depth unprojection yields trajectories, with substantial improvements over the evaluated baselines.

Critical strengths
  1. Strong empirical results. On Panoptic Studio Humans, Table 1 reports AJ of 38.4 versus 17.0 for the strongest baseline, alongside improvements in APD and occlusion accuracy under the same queries and evaluator. Table 2 also shows substantially better reference geometry: 0.86 cm point-to-scan distance versus 2.31 cm for V-DPM and 4.84 cm for 4RC, using identical person crops and reference-frame alignment.

  2. A coherent formulation for individualized human surfaces. Separating reference-time coordinates from current-frame depth avoids imposing a shared body template on clothing geometry. The matching and unprojection in Eqs. (6)–(7) also ensure that transferred points agree with the model's target-frame surface. Combined with direct surface correspondence supervision, this is a practical design for human-specific reconstruction and tracking.

Critical weaknesses
  1. The evidence for fine-grained surface identity remains incomplete. Table 1 provides meaningful tracking evidence, but Table 2 measures surface proximity rather than correspondence identity, as the authors acknowledge. Points can slide along clothing while remaining close to the reference scan. The tracking and synthetic pointwise results are useful, yet the evidence is not separated by reference visibility or by body, clothing, and hair. A breakdown of existing correspondence evaluations across these groups would better establish the scope of the central pixel-level correspondence claim, particularly for regions hidden in the reference image.

  2. The source of the methodological gain is not sufficiently isolated. Shared reference-time representations already appear in Dynamic Point Maps [Sucar et al., ICCV 2025] and V-DPM [Sucar et al., CVPR 2026], while Face Anything [Kocasarı et al., ECCV 2026] combines object coordinates, depth, and nearest-neighbor matching. Dense4D's extension to the full surface of an individual human is valuable, but the comparisons do not isolate its parameterization from human-specific training, backbone differences, and crop resolution. One controlled comparison using the same backbone, human data, cropped inputs, and training budget would clarify how much of the improvement comes from the HPM and depth formulation.

Questions for authors

See weaknesses.

Reviewer_3pSoOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Dense4D for dense human correspondence and 3D tracking across the body, clothing, and hair, addressing the limitations of body templates in representing detailed surface geometry and motion. It jointly processes person-centred image crops to predict Human Point Maps (HPMs) in a shared reference-time coordinate system and per-frame depth. Matching HPM coordinates establishes correspondence, while the predicted depth lifts matched pixels into 3D trajectories.

Critical strengths

The framework is well structured, using HPMs to establish correspondence in a shared reference space and per-frame depth to recover 3D trajectories. This separation provides a direct way to turn surface matches into tracks that remain consistent with the predicted geometry.

Critical weaknesses
  • The significance of the person-specific representation is not sufficiently established. Body models such as SMPL provide a shared topology and parameterization across people, supporting comparison and motion transfer despite their limited treatment of clothing and hair. Dense4D instead represents each person’s surface in a selected reference pose, without a shared correspondence space across individuals. While the reported accuracy gains are substantial, the paper provides limited evidence of what additional capabilities this person-specific representation enables beyond existing body models and general 4D reconstruction methods.

  • The paper lacks an analysis of severe occlusion, particularly when people occlude one another or large body regions remain hidden across frames. Since Dense4D relies on learned human priors and visible surface observations, it may struggle to predict reliable reference-time correspondences under these conditions.

Questions for authors
  • SMPL provides a shared representation across people, whereas Dense4D represents each person’s surface in a selected reference pose. Beyond improved tracking and surface localisation, what additional capabilities does this person-specific representation enable, and in which settings does it offer a meaningful advantage over body models or general 4D reconstruction methods?

  • Dense4D relies on human-specific learning, whereas 4RC and V-DPM model general dynamic scenes. How do these approaches compare under severe inter-person occlusion or when large body regions remain hidden across frames?

  • The paper emphasizes correspondence for clothing and hair beyond the underlying body surface. How does performance vary across clothing types and hairstyles, such as tight-fitting versus loose garments and short versus long hair, particularly under large deformations?

  • HPMs use the person’s geometry at a selected reference frame as the correspondence space. How does reference-frame selection affect tracking performance, particularly when self-contact brings different surface regions close together?

  • The current evaluation covers a limited range of benchmarks for human correspondence. An additional 2D point tracking evaluation on human query points in TAP-Vid-DAVIS, following AnthroTAP [1], would help assess the method’s performance more broadly.

My main concern is whether the paper sufficiently motivates the person-specific representation and establishes its significance beyond existing approaches. Convincing evidence on this point would lead me to consider raising my rating from clear rejection to weak rejection. If the response also adequately addresses the remaining evaluation concerns, I would consider weak acceptance.

[1] Kim et al., AnthroTAP: Learning point tracking with real-world motion, CVPR 2026

Comments on the paper

1 comment

We are withdrawing this submission and thank the reviewers for their time. We disagree with two points in the reviews and record our position here.

First, on motivation and scope: Dense4D addresses dense 4D reconstruction and correspondence across individual human surfaces. People are a central class of dynamic objects, and their surfaces and motion merit dedicated modelling. On Panoptic Studio Humans, Dense4D reaches 38.4 AJ against 17.0 for the strongest evaluated general-purpose baseline; on 4D-DRESS, its mean point-to-scan distance is 0.86 cm against 2.31 cm for V-DPM. SMPL does not explicitly represent individual clothing and hair geometry. The requested 2D tracking evaluation measures image-plane trajectories; the paper instead evaluates tracking in 3D, on the human queries of TAPVid-3D Panoptic Studio.

Second, on HPMs and human-specific training: our training renders primarily use environment-map backgrounds, which provide no translational parallax. Jointly translating the person and camera at a non-reference frame leaves the image unchanged but changes the observed-time positions in the reference camera predicted by methods such as V-DPM. A matched-data V-DPM baseline would therefore still be trained to predict a motion component that the images alone cannot determine, even if renderer annotations supply its labels. HPMs and own-camera depth remain unchanged under this transformation. Avoiding this ambiguity is an intended advantage of the formulation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.