- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Dense4D proposes a feed-forward model for dense correspondence and 3D tracking of humans, including their body, clothing, and hair, without fitting a parametric body template.
The main representation is a Human Point Map (HPM) where each observed person pixel in a frame t predicts the 3D position of that surface point at a common reference time (t=0). The proposed network takes cropped images of humans and produces HPMs and per-frame depth maps. Correspondences are obtained by nearest neighbour matching in reference space, then lifted into 3D using the predicted depth.
Critical strengths
-
Human tracking and dense human correspondences is a relevant and important problem. The paper proposes a method that naturally extends to loose clothing and hair.
-
The paper does extensive evaluations and the provided in the wild examples are also very promising.
Critical weaknesses
-
Dense4D adapts a large pretrained reconstruction backbone, VGGT-Omega using extensive human-specific supervision. Most comparisons are against general-purpose dynamic reconstruction or tracking systems. The improvements could therefore reflect a combination of domain-specialised training, person-centred processing, the reference-space representation, and the matching-based readout. Hence, a matching baseline also trained on human datasets would better establish what HPMs contribute. In particular using crops for human prediction tasks itself is not novel and has been used by previous human mesh recovery pipelines as well. Hence, HPM seems to be the main contribution. To validate this, finetuning VGGT-Omega on the same dataset without HPM would provide a good baseline.
-
As also noted by the authors, the reference frame selection can be critical for the overall performance of the method. More thorough evaluation about this, especially how this selection effects cases with multiple people interacting etc., will be useful.
-
While the paper does not predict human meshes, it still trains on human specific datasets. Hence, I think in addition to general purpose tracking methods, comparisons to SOTA human mesh recovery methods is needed.
My current slightly negative rating is mostly due to concerns regarding experiments to really pin-point the main contribution of the work.
Questions for authors
-
For the 4D-Dress protocol, is it useful also measure the error in the other direction, i.e., from the reference scan to the predictions to make sure the target predictions don't localize in small regions of the body?
-
How much of the gain remains against a baseline trained on the same human data, with the same backbone, crops, and training budget, but using direct motion prediction or a conventional dynamic-point-map formulation?
-
What happens to point identity after extended occlusion and reappearance?
We are withdrawing this submission and thank the reviewers for their time. We disagree with two points in the reviews and record our position here.
First, on motivation and scope: Dense4D addresses dense 4D reconstruction and correspondence across individual human surfaces. People are a central class of dynamic objects, and their surfaces and motion merit dedicated modelling. On Panoptic Studio Humans, Dense4D reaches 38.4 AJ against 17.0 for the strongest evaluated general-purpose baseline; on 4D-DRESS, its mean point-to-scan distance is 0.86 cm against 2.31 cm for V-DPM. SMPL does not explicitly represent individual clothing and hair geometry. The requested 2D tracking evaluation measures image-plane trajectories; the paper instead evaluates tracking in 3D, on the human queries of TAPVid-3D Panoptic Studio.
Second, on HPMs and human-specific training: our training renders primarily use environment-map backgrounds, which provide no translational parallax. Jointly translating the person and camera at a non-reference frame leaves the image unchanged but changes the observed-time positions in the reference camera predicted by methods such as V-DPM. A matched-data V-DPM baseline would therefore still be trained to predict a motion component that the images alone cannot determine, even if renderer annotations supply its labels. HPMs and own-camera depth remain unchanged under this transformation. Avoiding this ambiguity is an intended advantage of the formulation.