- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
JIGAR reconstructs a hand and a manipulated object from a single RGB image. It first adapts a pretrained image-to-3D model to jointly generate a hand-object interaction scaffold, using hand renderings and a hand/contact-weighted flow-matching objective. The scaffold is segmented, and its object component initializes object regeneration conditioned on an inpainted image. The estimated MANO hand is retained as the final hand asset; residual rigid hand registration transfers the regenerated object into that hand's frame. Experiments on SHOWMe, HOT3D, and HO-Cap evaluate both hand-aligned and independently aligned object reconstruction. The paper also includes formulation and component ablations, output-coverage and interaction metrics, and adaptation to a second 3D backbone.
Critical strengths
First, the central formulation is supported by a relevant controlled comparison.** In Table 4(a), changing the generation target from object-only to joint hand-object geometry reduces HOT3D hand-aligned CD from 50.26 to 27.39 mm under 2D-render conditioning. Joint generation also improves the result under 3D-latent conditioning. These comparisons provide evidence that generating the shared scaffold contributes beyond merely supplying hand conditions. This empirical finding is an important part of the paper's contribution.
Second, the gains extend across datasets and beyond relative placement.** JIGAR obtains the best reported hand-aligned CD and F-scores on all three datasets in Table 1. It also obtains the best independently aligned reconstruction scores in Table 2, although the size of the gain varies by dataset. The recovery ablations, separate reporting of output coverage, and matched comparison with the adapted Step1X-3D backbone strengthen the evidence for the complete pipeline.
Critical weaknesses
The alignment contribution is not yet measured independently of shape reconstruction. The hand-aligned and independently aligned results, joint-generation ablations, and drift analysis provide evidence for the proposed formulation. However, hand-aligned CD combines object-shape error with placement error, and the two evaluation protocols do not directly quantify relative translation, rotation, or scale accuracy. The residual hand registration in Section 3.3 is presented as a coordinate-frame correction rather than separate object-pose optimization, and therefore does not necessarily contradict the proposed formulation. Appendix G measures scaffold-to-regenerated-object drift of 4.79 degrees and 0.74 cm using similarity ICP; these values describe change relative to the scaffold, rather than error against ground truth, and may also reflect shape changes. Thus, the current evidence supports the overall reconstruction gains but does not fully isolate how much relative-alignment accuracy joint generation establishes and recovery preserves. A targeted direct evaluation would make the central alignment claim more precise.
Questions for authors
-
Alignment accuracy. On evaluation objects for which a reliable pose comparison is possible, could you provide a limited stage-wise analysis of relative translation, rotation, and scale error for the scaffold and final reconstruction, with the coordinate frames and treatment of object symmetries specified? The key distinction is error against ground truth versus the scaffold-to-object drift already measured in Appendix G. Evidence that the final output retains an accurate relative configuration would strengthen my assessment of the main claim.
-
Generalization. Do any physical object instances in the 20-object DexYCB refinement set also appear in the evaluation datasets? Dataset disjointness does not necessarily imply instance disjointness. If there is overlap, could the authors clarify whether the reported improvements also hold for previously unseen object instances? This would help assess the generalization of the proposed generative reconstruction pipeline.
4 comments
Response to Questions 2
Q2: Generalization and Object-Instance Overlap
We thank the reviewer for highlighting the importance of object-instance disjointness and generalization. Additional experiments show JIGAR maintains its leading performance on unseen object instances, and removing overlapping instances from training yields nearly identical results. We would also like to clarify that object-instance overlap does not imply strict training–evaluation sample overlap. The training and evaluation sets contain distinct images and hand–object interactions, while JIG learns to generate joint hand–object geometry rather than isolated object shapes.
(1) Object-instance overlap is limited and confined to DexYCB refinement.
| Evaluation dataset | Shared YCB instances | Overlapping images | Ratio |
|---|---|---|---|
| SHOWMe | 021, 024, 061 | 15/500 | 3.0% |
| HOT3D | 040 | 26/500 | 5.2% |
| HO-Cap | None | 0/500 | 0% |
| Total | 4/20 | 41/1,500 | 2.7% |
Only 2.7% of evaluation images involve shared object identities. Within our task-specific training, the identified overlap occurs only in DexYCB real-world refinement, not in the preceding ShapeNet-based ObMan synthetic adaptation.
(2) JIGAR remains the best-performing method on unseen object instances.
Using the original checkpoint, we remove overlapping evaluation instances and recompute all methods' results on the same common valid-output subsets.
SHOWMe: Hand-aligned evaluation (N = 457 → 442)
| Method | CDh ↓ (Full → Unseen) | F@5h ↑ (Full → Unseen) | F@10h ↑ (Full → Unseen) |
|---|---|---|---|
| IHOI | 40.65 → 40.62 | 0.206 → 0.207 | 0.384 → 0.385 |
| AlignSDF | 161.88 → 163.17 | 0.048 → 0.048 | 0.095 → 0.095 |
| gSDF | 60.92 → 61.24 | 0.162 → 0.163 | 0.306 → 0.309 |
| HORT | 36.52 → 36.68 | 0.204 → 0.204 | 0.380 → 0.379 |
| EasyHOI | 84.64 → 85.47 | 0.112 → 0.111 | 0.212 → 0.210 |
| FollowMyHold | 83.26 → 83.57 | 0.124 → 0.124 | 0.232 → 0.233 |
| JIGAR (ours) | 21.06 → 20.75 | 0.422 → 0.425 | 0.665 → 0.669 |
HOT3D: Hand-aligned evaluation (N = 375 → 359)
| Method | CDh ↓ (Full → Unseen) | F@5h ↑ (Full → Unseen) | F@10h ↑ (Full → Unseen) |
|---|---|---|---|
| IHOI | 58.96 → 58.61 | 0.159 → 0.161 | 0.295 → 0.298 |
| AlignSDF | 258.26 → 256.26 | 0.026 → 0.027 | 0.051 → 0.052 |
| gSDF | 123.18 → 122.22 | 0.075 → 0.078 | 0.144 → 0.148 |
| HORT | 67.66 → 69.30 | 0.166 → 0.161 | 0.303 → 0.296 |
| EasyHOI | 110.76 → 109.93 | 0.090 → 0.092 | 0.171 → 0.174 |
| FollowMyHold | 120.70 → 118.45 | 0.081 → 0.083 | 0.149 → 0.154 |
| JIGAR (ours) | 27.39 → 26.82 | 0.352 → 0.358 | 0.582 → 0.591 |
HO-Cap contains no overlapping instances so its result remains unchanged.
(3) Removing all overlapping objects from DexYCB refinement preserves performance.
We repeat the DexYCB refinement stage from the same ObMan-trained checkpoint, excluding all 952 overlapping DexYCB views (19.5%), while retaining the original three-epoch schedule and ObMan replay. Evaluation uses the original common subsets, including overlapping instances.
| Dataset | CDh ↓ (Original → Retrained) | F@5h ↑ (Original → Retrained) | F@10h ↑ (Original → Retrained) |
|---|---|---|---|
| SHOWMe | 21.06 → 21.02 | 0.422 → 0.423 | 0.665 → 0.666 |
| HOT3D | 27.39 → 27.42 | 0.352 → 0.350 | 0.582 → 0.577 |
| HO-Cap | 29.83 → 29.84 | 0.314 → 0.314 | 0.523 → 0.522 |
We will revise the manuscript to clarify dataset-level versus instance-level disjointness and incorporate these additional evaluations. We thank the reviewer again for this valuable suggestion.
I appreciate the authors’ detailed response and additional experimental results, which have satisfactorily resolved my main concerns.
Thank you for your positive feedback! We sincerely appreciate your time and are glad that our responses have addressed your main concerns.
Clarifications on Weaknesses 1 and response to Question 1
W1 & Q1: Direct Evaluation of GT-Relative Alignment Accuracy
We thank the reviewer for highlighting that hand-aligned CD conflates shape and placement errors. To directly demonstrate our method's contribution to hand–object alignment, we conduct two complementary experiments and provide the evaluation details requested by the reviewer:
(1) Stage-wise alignment preservation. We compare the scaffold and final reconstructions on all 479 valid JIGAR outputs from HOT3D. Regeneration introduces only small changes in GT-relative alignment errors, while substantially improving reconstruction quality (Table 4(b) of the paper: CDh 35.17 → 27.39 mm; F@10h 0.359 → 0.582, evaluated on the 375-case common subset).
(2) Relative alignment accuracy against HORT (the strongest baseline). We additionally compare the final reconstructions of JIGAR and HORT on the shared 375 HOT3D cases. JIGAR achieves lower GT-relative errors across all subsets. Since both JIGAR and HORT retain the same WiLoR-derived hand estimate (Appendix C), this comparison further supports the benefit of joint interaction generation for hand-relative object alignment.
(3) Evaluation protocol and ambiguity handling.
These measurements are GT-referenced alignment estimates. Since the generated shapes may differ from GT, registration-derived pose corrections can still be affected by shape discrepancies. Notably, the higher-ranked subsets (e.g., Best 20%) have lower registration losses, indicating better geometric agreement with GT and potentially reducing the influence of shape discrepancies on the estimated pose errors.
Together, the two evaluations provide direct evidence that joint generation establishes a more accurate hand-relative configuration than HORT, while subsequent object regeneration largely preserves that configuration.
We sincerely appreciate the reviewer's thoughtful feedback. We will incorporate these results and the full evaluation details into the revised manuscript.