acceptodds
Under review as a conference paper at ICLR 2027

Intrinsic Alignment by Generation: Monocular Hand–Object Reconstruction via Interaction Scaffolds

Abstract

Monocular hand–object reconstruction ultimately requires separate hand and object assets, but their relative pose and scale are inherently joint geometric quantities. This suggests the interaction is most naturally established in a shared geometry. Existing methods either incorporate hand geometry or interaction constraints into learned reconstruction while making limited use of object-centric image-to-3D priors, or exploit pretrained image-to-3D priors while deferring hand–object placement to test-time alignment optimization. JIGAR (Joint Interaction Generation and Asset Recovery) bridges this gap: its Joint Interaction Generation (JIG) stage adapts a pretrained image-to-3D model to jointly generate hand–object geometry as an interaction scaffold, which then guides separate asset recovery through in-place object regeneration and parametric hand recovery. This preserves the hand–object configuration while allowing the prior to refine object geometry. In this way, JIGAR combines pretrained generative shape priors with intrinsic hand-relative reconstruction without requiring test-time optimization of the hand–object relative placement. Evaluations on SHOWMe, HOT3D, and HO-Cap show that JIGAR outperforms all evaluated baselines in object and interaction reconstruction, reducing hand-aligned object Chamfer distance by 42.3–63.2% relative to the strongest evaluated baseline across the three datasets. The code is included in the supplementary material and will be open-sourced upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Hand–Object Reconstruction, Joint Interaction Generation, Image-to-3D Generation, Generative 3D Priors

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_EsH8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

JIGAR reconstructs a hand and a manipulated object from a single RGB image. It first adapts a pretrained image-to-3D model to jointly generate a hand-object interaction scaffold, using hand renderings and a hand/contact-weighted flow-matching objective. The scaffold is segmented, and its object component initializes object regeneration conditioned on an inpainted image. The estimated MANO hand is retained as the final hand asset; residual rigid hand registration transfers the regenerated object into that hand's frame. Experiments on SHOWMe, HOT3D, and HO-Cap evaluate both hand-aligned and independently aligned object reconstruction. The paper also includes formulation and component ablations, output-coverage and interaction metrics, and adaptation to a second 3D backbone.

Critical strengths

First, the central formulation is supported by a relevant controlled comparison.** In Table 4(a), changing the generation target from object-only to joint hand-object geometry reduces HOT3D hand-aligned CD from 50.26 to 27.39 mm under 2D-render conditioning. Joint generation also improves the result under 3D-latent conditioning. These comparisons provide evidence that generating the shared scaffold contributes beyond merely supplying hand conditions. This empirical finding is an important part of the paper's contribution.

Second, the gains extend across datasets and beyond relative placement.** JIGAR obtains the best reported hand-aligned CD and F-scores on all three datasets in Table 1. It also obtains the best independently aligned reconstruction scores in Table 2, although the size of the gain varies by dataset. The recovery ablations, separate reporting of output coverage, and matched comparison with the adapted Step1X-3D backbone strengthen the evidence for the complete pipeline.

Critical weaknesses

The alignment contribution is not yet measured independently of shape reconstruction. The hand-aligned and independently aligned results, joint-generation ablations, and drift analysis provide evidence for the proposed formulation. However, hand-aligned CD combines object-shape error with placement error, and the two evaluation protocols do not directly quantify relative translation, rotation, or scale accuracy. The residual hand registration in Section 3.3 is presented as a coordinate-frame correction rather than separate object-pose optimization, and therefore does not necessarily contradict the proposed formulation. Appendix G measures scaffold-to-regenerated-object drift of 4.79 degrees and 0.74 cm using similarity ICP; these values describe change relative to the scaffold, rather than error against ground truth, and may also reflect shape changes. Thus, the current evidence supports the overall reconstruction gains but does not fully isolate how much relative-alignment accuracy joint generation establishes and recovery preserves. A targeted direct evaluation would make the central alignment claim more precise.

Questions for authors
  1. Alignment accuracy. On evaluation objects for which a reliable pose comparison is possible, could you provide a limited stage-wise analysis of relative translation, rotation, and scale error for the scaffold and final reconstruction, with the coordinate frames and treatment of object symmetries specified? The key distinction is error against ground truth versus the scaffold-to-object drift already measured in Appendix G. Evidence that the final output retains an accurate relative configuration would strengthen my assessment of the main claim.

  2. Generalization. Do any physical object instances in the 20-object DexYCB refinement set also appear in the evaluation datasets? Dataset disjointness does not necessarily imply instance disjointness. If there is overlap, could the authors clarify whether the reported improvements also hold for previously unseen object instances? This would help assess the generalization of the proposed generative reconstruction pipeline.

4 comments
AuthorsOpenReview ↗

Clarifications on Weaknesses 1 and response to Question 1

W1 & Q1: Direct Evaluation of GT-Relative Alignment Accuracy

We thank the reviewer for highlighting that hand-aligned CD conflates shape and placement errors. To directly demonstrate our method's contribution to hand–object alignment, we conduct two complementary experiments and provide the evaluation details requested by the reviewer:

(1) Stage-wise alignment preservation. We compare the scaffold and final reconstructions on all 479 valid JIGAR outputs from HOT3D. Regeneration introduces only small changes in GT-relative alignment errors, while substantially improving reconstruction quality (Table 4(b) of the paper: CDh 35.17 → 27.39 mm; F@10h 0.359 → 0.582, evaluated on the 375-case common subset).

SubsetNScaffold: R (°) / T (mm) / S (%) ↓Final: R (°) / T (mm) / S (%) ↓
All (100%)47935.42 / 26.01 / 13.8236.86 / 26.31 / 13.57
Best 80%38333.11 / 21.82 / 10.2034.42 / 22.30 / 10.07
Best 60%28732.45 / 19.40 / 8.7834.16 / 19.94 / 8.17
Best 40%19231.46 / 18.78 / 7.3532.70 / 19.30 / 6.66
Best 20%9627.69 / 18.50 / 5.9928.20 / 18.89 / 5.26

(2) Relative alignment accuracy against HORT (the strongest baseline). We additionally compare the final reconstructions of JIGAR and HORT on the shared 375 HOT3D cases. JIGAR achieves lower GT-relative errors across all subsets. Since both JIGAR and HORT retain the same WiLoR-derived hand estimate (Appendix C), this comparison further supports the benefit of joint interaction generation for hand-relative object alignment.

Shared subsetNJIGAR (ours): R (°) / T (mm) / S (%) ↓HORT: R (°) / T (mm) / S (%) ↓
All (100%)37535.07 / 25.19 / 13.5449.66 / 54.85 / 49.08
Best 80%30034.71 / 22.12 / 11.3548.20 / 47.85 / 44.95
Best 60%22534.35 / 20.71 / 9.8947.74 / 43.15 / 43.03
Best 40%15034.31 / 19.16 / 9.3245.24 / 39.18 / 41.22
Best 20%7533.36 / 17.28 / 8.2241.16 / 29.28 / 27.77

(3) Evaluation protocol and ambiguity handling.

These measurements are GT-referenced alignment estimates. Since the generated shapes may differ from GT, registration-derived pose corrections can still be affected by shape discrepancies. Notably, the higher-ranked subsets (e.g., Best 20%) have lower registration losses, indicating better geometric agreement with GT and potentially reducing the influence of shape discrepancies on the estimated pose errors.

AspectProtocol
Coordinate framesBoth JIGAR stages are transferred into the WiLoR/MANO hand frame using the same residual hand-registration transform. All predictions undergo the same hand-derived scale normalization and rigid hand-to-GT alignment (Appendix A.4).
GT-relative errorsAn evaluation-only similarity registration to GT estimates corrective rotation R (°), translation T (mm), and scale S (%), minimizing Chamfer distance + 0.25 × normal loss + 0.25 × (1 − volumetric IoU).
Object symmetriesTo mitigate rotational ambiguity, we first identify the minimum registration loss. Among candidate solutions with losses no more than 10% above this minimum, we select the one requiring the smallest rotation correction.
Shared subsetsWithin each comparison, cases are ranked by the maximum registration loss of the two compared outputs. Thus, both outputs use exactly the same cases in every row. The two experiments use independently constructed rankings.

Together, the two evaluations provide direct evidence that joint generation establishes a more accurate hand-relative configuration than HORT, while subsequent object regeneration largely preserves that configuration.


We sincerely appreciate the reviewer's thoughtful feedback. We will incorporate these results and the full evaluation details into the revised manuscript.

AuthorsOpenReview ↗

Response to Questions 2

Q2: Generalization and Object-Instance Overlap

We thank the reviewer for highlighting the importance of object-instance disjointness and generalization. Additional experiments show JIGAR maintains its leading performance on unseen object instances, and removing overlapping instances from training yields nearly identical results. We would also like to clarify that object-instance overlap does not imply strict training–evaluation sample overlap. The training and evaluation sets contain distinct images and hand–object interactions, while JIG learns to generate joint hand–object geometry rather than isolated object shapes.

(1) Object-instance overlap is limited and confined to DexYCB refinement.

Evaluation datasetShared YCB instancesOverlapping imagesRatio
SHOWMe021, 024, 06115/5003.0%
HOT3D04026/5005.2%
HO-CapNone0/5000%
Total4/2041/1,5002.7%

Only 2.7% of evaluation images involve shared object identities. Within our task-specific training, the identified overlap occurs only in DexYCB real-world refinement, not in the preceding ShapeNet-based ObMan synthetic adaptation.

(2) JIGAR remains the best-performing method on unseen object instances.

Using the original checkpoint, we remove overlapping evaluation instances and recompute all methods' results on the same common valid-output subsets.

SHOWMe: Hand-aligned evaluation (N = 457 → 442)

MethodCDh ↓ (Full → Unseen)F@5h ↑ (Full → Unseen)F@10h ↑ (Full → Unseen)
IHOI40.65 → 40.620.206 → 0.2070.384 → 0.385
AlignSDF161.88 → 163.170.048 → 0.0480.095 → 0.095
gSDF60.92 → 61.240.162 → 0.1630.306 → 0.309
HORT36.52 → 36.680.204 → 0.2040.380 → 0.379
EasyHOI84.64 → 85.470.112 → 0.1110.212 → 0.210
FollowMyHold83.26 → 83.570.124 → 0.1240.232 → 0.233
JIGAR (ours)21.06 → 20.750.422 → 0.4250.665 → 0.669

HOT3D: Hand-aligned evaluation (N = 375 → 359)

MethodCDh ↓ (Full → Unseen)F@5h ↑ (Full → Unseen)F@10h ↑ (Full → Unseen)
IHOI58.96 → 58.610.159 → 0.1610.295 → 0.298
AlignSDF258.26 → 256.260.026 → 0.0270.051 → 0.052
gSDF123.18 → 122.220.075 → 0.0780.144 → 0.148
HORT67.66 → 69.300.166 → 0.1610.303 → 0.296
EasyHOI110.76 → 109.930.090 → 0.0920.171 → 0.174
FollowMyHold120.70 → 118.450.081 → 0.0830.149 → 0.154
JIGAR (ours)27.39 → 26.820.352 → 0.3580.582 → 0.591

HO-Cap contains no overlapping instances so its result remains unchanged.

(3) Removing all overlapping objects from DexYCB refinement preserves performance.

We repeat the DexYCB refinement stage from the same ObMan-trained checkpoint, excluding all 952 overlapping DexYCB views (19.5%), while retaining the original three-epoch schedule and ObMan replay. Evaluation uses the original common subsets, including overlapping instances.

DatasetCDh ↓ (Original → Retrained)F@5h ↑ (Original → Retrained)F@10h ↑ (Original → Retrained)
SHOWMe21.06 → 21.020.422 → 0.4230.665 → 0.666
HOT3D27.39 → 27.420.352 → 0.3500.582 → 0.577
HO-Cap29.83 → 29.840.314 → 0.3140.523 → 0.522

We will revise the manuscript to clarify dataset-level versus instance-level disjointness and incorporate these additional evaluations. We thank the reviewer again for this valuable suggestion.

I appreciate the authors’ detailed response and additional experimental results, which have satisfactorily resolved my main concerns.

Thank you for your positive feedback! We sincerely appreciate your time and are glad that our responses have addressed your main concerns.

Reviewer_UFJ1OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents JIGAR, a framework for reconstructing separate hand and object meshes from a single RGB image. It adapts a pretrained image-to-3D model to jointly generate a hand–object interaction scaffold, encoding their relative pose and scale in a shared coordinate frame. Then, the scaffold is split for subsequent object regeneration and residual registration between two hand estimates. Experiments on SHOWMe, HOT3D, and HO-Cap report improvements in object geometry and hand-relative reconstruction accuracy over existing methods, supported by formulation and component ablations.

Critical strengths
  1. Effective formulation. Joint generation offers a compelling way to combine pretrained 3D priors with hand-relative reconstruction, reducing reliance on explicit relative-pose optimization.
  2. Strong experimental results. Consistent improvements across three datasets, supported by formulation ablations and two-backbone experiments, provide evidence for the method's effectiveness.
Critical weaknesses
  1. Limited novelty relative to Follow My Hold. Generating a coarse hand–object mesh to guide subsequent asset reconstruction has already been explored in Follow My Hold. The main distinction appears to be task-specific adaptation of the I2M model to improve this intermediate mesh, making the contribution incremental.
  2. The scope of optimization-free alignment needs clarification. Although the method avoids explicit test-time optimization of hand–object relative placement, it still relies on rigid registration between the WiLoR estimate and the scaffold, which will also lead to sensitive results.
  3. Alignment preservation during asset recovery is insufficiently established. Joint generation indeed places the hand and object in a shared frame, but subsequent object regeneration and replacement of the hand can alter the object shape, relative pose, and scale. The paper should directly test whether contact and alignment are preserved after object regeneration and hand registration.
  4. Details about evaluating the physical plausibility. First, the contact metric does not establish correct interaction. CR only requires the minimum hand–object distance to be below 1 cm. This criterion is too loose, so 100% CR does not demonstrate physically plausible grasping. Second, the simulation is too short to substantiate grasp stability. SD measures displacement over only 20 steps at 1/2401/240 seconds each—around 0.083 seconds. Low displacement over this interval may reflect temporary contact rather than stable support. Longer simulations are needed to validate stability.
Questions for authors
  1. Could you please clarify the obvious novelty against Follow My Hold, and also the optimization-free claim?
  2. How can object regeneration and replacement of the hand ensure that the object shape, relative pose, and scale are correct?
  3. Could you please make the evaluation of physical plausibility more convincing by using a more rigorous setting?
5 comments
AuthorsOpenReview ↗

Clarifications on Weaknesses 1 and 2 and response to Question 1

W1 & Q1(a): Novelty relative to FollowMyHold

We thank the reviewer for highlighting this comparison. We would like to clarify that, while both methods use coarse hand–object geometry, their reconstruction paradigms differ fundamentally:

  • FollowMyHold (initialization → optimization-in-the-loop): The coarse HOI mesh mainly provides geometric initialization, while the core contribution of FollowMyHold lies in its optimization-in-the-loop framework, which guides object generation and iteratively refines hand and object transforms to determine their final configuration.
  • JIGAR (joint generation → in-place recovery): The jointly generated scaffold directly establishes the hand–object configuration, enabling subsequent asset recovery without optimizing their relative placement. Importantly, our hand-only ICP is intended to replace the generated non-standard hand mesh with a standard MANO hand mesh required by the output format, NOT to optimize hand–object relative placement. It only estimates a rigid hand-to-hand coordinate mapping, without involving object geometry or interaction constraints (see Q1(b)).

Our ablation in the original manuscript further support the importance of joint generation and scaffold-guided recovery:

ComparisonCDh (mm) ↓F@10h ↑
Object-only → Joint generation50.26 → 27.390.351 → 0.582
Without → With scaffold initialization147.25 → 27.390.136 → 0.582
Original → JIG-adapted Hunyuan3D-2.168.76 → 28.360.194 → 0.567

These results demonstrate that JIGAR goes beyond generating a better coarse mesh: it establishes the interaction through joint generation and largely preserves it through in-place asset recovery, rather than relying on FollowMyHold-style pose optimization.


W2 & Q1(b): Scope of the claim and registration sensitivity

To clarify, JIGAR avoids explicit test-time optimization of hand–object relative placement, not all forms of test-time optimization. JIG jointly generates the configuration in a shared scaffold frame. Hand-only ICP enables replacement of the generated hand with the WiLoR-estimated MANO hand and transfer of the object into the WiLoR frame. It estimates only rigid hand-to-hand rotation and translation, without using object geometry or interaction constraints or independently optimizing object pose.

On all 479 valid HOT3D outputs, our evaluation shows lightweight registration (0.23 s/case) and low initialization sensitivity (mean vertex deviations: 0.287/0.432 mm).

(1) Registration quality and efficiency. We compare ICP caps of 40 and 80 iterations with early stopping. Rotation/translation quantify corrections; F@5/10 and normal consistency measure hand–scaffold agreement.

Metric (per case)Max 40 iter. (original)Max 80 iter. (extended)
Mean iter.33.0238.86
Mean runtime (s)0.230.25
Rotation correction (°)8.308.54
Hand-centroid translation (mm)8.878.69
Hand-scaffold F@5mm / F@10mm0.8246 / 0.92360.8137 / 0.9266
Normal consistency (hand→scaffold / scaffold→hand) (mean cosine similarity, %)95.65% / 95.47%95.56% / 95.41%

The 80-iteration cap changes little in agreement, adding only 0.02 s/case.

(2) Initialization sensitivity. For each case, we perturb ICP initialization 10 times using random-axis rotations (±5°/±10°) about the hand centroid and translations within 5/10 mm-radius balls. We measure corresponding hand-vertex displacements relative to the original ICP solution (40-iteration cap).

PerturbationMean (mm) ↓Median (mm) ↓
±5°, 5 mm0.2870.015
±10°, 10 mm0.4320.032

Small mean and median deviations indicate low sensitivity under both tested perturbations.

Overall, these results support that JIGAR establishes hand–object alignment through joint generation, while the subsequent hand-only ICP provides an efficient and stable coordinate-frame correction without explicit hand–object pose optimization.


We sincerely appreciate the reviewer's thoughtful feedback, which has motivated experiments that solidify our claims. We will incorporate all results in the revised manuscript.

AuthorsOpenReview ↗

Clarifications on Weaknesses 3 and response to Question 2

W3 & Q2: Preservation of Object Shape, Relative Pose, Scale, and Contact During Asset Recovery

We thank the reviewer for raising this important question. While Asset Recovery does not guarantee 100% alignment preservation, our stage-wise evaluations below demonstrate limited pose and scale drift during regeneration and high contact proximity after hand replacement. Furthermore, our GT-based results demonstrate accurate final object geometry and hand-relative configuration.

We conduct the analysis on all 479 valid JIGAR outputs from HOT3D.

(1) Stage-wise alignment preservation. We evaluate three stages: (A) separated joint scaffold, (B) regenerated object with scaffold hand, and (C) final reconstruction with registered WiLoR hand. A→B isolates object regeneration, while B→C isolates hand replacement.

MetricA→B: RegenerationB→C: Hand Replacement
Hand-relative Rotation Drift (°) ↓4.43Rigid frame transfer only
Hand-relative Translation Drift (cm) ↓0.68Rigid frame transfer only
RMS Radius Change (%) ↓2.8140 (No object geometry change)
Sim(3) Scale Ratio0.97441 (No object scaling)

Regeneration produces relatively small changes in hand-relative pose and geometric scale. During B→C, the object undergoes only a rigid coordinate transfer, preserving its intrinsic shape and scale without additional object regeneration or independent pose optimization.

(2) Contact preservation across recovery stages. We further evaluate contact proximity, local contact-region consistency, and volumetric intersection.

MetricA: ScaffoldB: RegeneratedC: Final
CR@2mm (%) ↑100.00100.0097.91
Intersection Volume (cm³) ↓0.000†3.3763.890
Contact Region CD@5mm (mm) ↓—2.372 (A→B)3.140 (B→C)

† The scaffold's zero IV is guaranteed by the complementary watertight proxy construction, which will be detailed in the revised paper.

(3) In the manuscript, Tables 1–2 show JIGAR's superior GT-based reconstruction accuracy across all three datasets, under independent object alignment (shape) and hand-only alignment (shape and hand-relative configuration), respectively.

Together, the stage-wise drift measurements and GT-based evaluations demonstrate that Asset Recovery improves object geometry while largely preserving the interaction configuration.

We will update the manuscript with these additional experimental results and the corresponding implementation and evaluation details. We thank the reviewer again for this constructive feedback.

AuthorsOpenReview ↗

Clarifications on Weaknesses 4 and response to Question 3

W4 & Q3: Physical Plausibility Evaluation

We thank the reviewer for the constructive comments. Our stricter contact evaluation and extended simulations provide stronger evidence of physical plausibility: JIGAR achieves the highest CR at a 1 mm threshold (97.30–99.79%) and reduces SD by 46.3–70.4% relative to the strongest baselines at 240 simulation steps (1 s). However, as acknowledged in our paper, CR and SD are complementary diagnostics rather than guarantees of physically valid grasping in every case.

Our robustness evaluation builds on established metrics from prior work:

  • IV is used in EasyHOI and FollowMyHold;
  • SD follows EasyHOI's official implementation;
  • OR corresponds to Reconstruction Rate (RR) in FollowMyHold.
  • We additionally introduce CR to measure hand–object proximity, complementing IV by distinguishing cases in which the hand and object have zero IV but are not in contact.

(1) Stricter contact evaluation. To address the concern that the original contact criterion is too loose, we additionally evaluate CR at 1/2/5 mm thresholds, alongside the original 10 mm setting. Complete results and sensitivity curves are provided in the CR supplementary results.

DatasetJIGAR CR@10mm ↑JIGAR CR@1mm ↑Best baseline CR@1mm ↑
SHOWMe100.0%99.79%99.60% (HORT)
HOT3D99.6%97.70%90.06% (EasyHOI)
HO-Cap99.8%97.30%85.32% (HORT)

(2) Extended simulation evaluation. To address the concern that the original simulation is too short to assess grasp stability, we extend SD evaluation to longer horizons while keeping all other settings unchanged. The simulation protocol is adapted from the official EasyHOI repository (commit a1fd9d0). Complete results across 7 simulation horizons and detailed settings are provided in the SD supplementary results. Representative results at 240 steps (1 s) are shown below.

DatasetJIGAR SD@240 (cm) ↓Best baseline SD@240 (cm) ↓Reduction
SHOWMe59.49160.80 (HORT)63.0%
HOT3D58.14196.72 (EasyHOI)70.4%
HO-Cap98.86183.98 (HORT)46.3%

Metrics are computed on each method's own valid outputs from the fixed 500-input set per dataset. SD measures the object's net displacement under gravity with the predicted hand fixed and no ground plane.

We will incorporate the additional evaluations and clarifications into the revised manuscript. We thank the reviewer again for the valuable suggestions.

Reviewer_nTQUOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces JIGAR, a pipeline for reconstructing hand–object interactions from a single RGB image. Its central idea is to generate a joint hand–object “scaffold” that implicitly encodes the spatial relationship between the hand and object. A frozen image-to-3D backbone is adapted through trainable adapters, conditioned on the interaction image and renderings of an estimated hand, with additional supervision on hand and contact regions. The scaffold is then segmented, and the object is regenerated using the original pretrained generator, initialized from a partially noised scaffold latent and conditioned on an inpainted object image. The final hand is provided by an off-the-shelf hand estimation model, while registration with the scaffold hand transfers the regenerated object into the estimated hand’s coordinate frame. Experiments on three datasets demonstrate state-of-the-art reconstruction performance among the evaluated methods, particularly in object reconstruction quality.

Critical strengths
  1. The idea of HOI joint-generation is inspiring. Conditioning on complementary front and far renderings of the estimated hand to generate joint hand–object geometry is an interesting approach to recovering their spatial relationship.

  2. The object-regeneration strategy is technically novel. It combine interaction configuration with pretrained object-shape priors, allowing object refinement without a separate optimization of hand–object placement.

  3. The validation is solid. Consistent improvements across three datasets, together with component ablations and an alternative-backbone experiment, provide substantive evidence that the proposed design is effective and reasonably transferable.

Critical weaknesses
  1. Object regeneration is anchored through latent initialization, without explicit constraints preserving contact or relative pose. The reported drift of 4.79° and 0.74 cm shows that regeneration can alter the established configuration. Although the reconstruction results are encouraging, the permissive contact criterion and short simulation horizon provide limited evidence that the recovered interactions remain physically stable.

  2. The paper does not adequately explain the trainable adapter architecture, how correspondence between latent tokens and MANO vertices is established within the frozen ShapeVAE, or the residual hand-registration procedure. These mechanisms is important for reproducibility.

Questions for authors
  1. How much does regeneration change object scale, hand–object contact, and penetration relative to the scaffold? These measurements would help assess whether improved object geometry preserves the intended interaction.

  2. JIG is trained with ground-truth hand renderings but uses estimated hands at inference, creating a potential sensitivity to pose errors under severe occlusion. How does reconstruction quality change as hand-pose errors increase? The WiLoR-to-HaMeR comparison does not directly isolate this sensitivity. Could you provide results grouped by hand-pose error or a controlled perturbation analysis to clarify whether incorrect hand conditions propagate into the scaffold and final hand–object configuration?

4 comments
AuthorsOpenReview ↗

Clarifications on Weaknesses 1 and 2

W1: Regeneration Drift and Physical Plausibility

We thank the reviewer for highlighting the concerns regarding regeneration drift, contact preservation, the permissive contact criterion, and the short simulation horizon. Although latent initialization does not explicitly guarantee pose or contact preservation, our additional evaluations show 97.30–99.79% CR@1mm and 46.3–70.4% lower mean SD than the strongest baselines at 240 steps across three datasets.

(1) Regeneration drift and contact preservation. We acknowledge that regeneration can alter the scaffold-established configuration. However, reducing drift does not necessarily improve reconstruction accuracy. In Appendix G, increasing scaffold retention reduces rotation/translation drift from 4.79°/0.74 cm to 1.07°/0.05 cm, but decreases hand-aligned F@10 from 0.582 to 0.497, revealing a trade-off between pose preservation and geometric refinement.

We also directly quantify stage-wise contact changes on 479 valid HOT3D reconstructions, as detailed in our response to Q1 below.

(2) Stricter contact criterion and extended simulation horizon. We additionally evaluate CR at 1/2/5/10 mm and extend SD evaluation to 20/35/60/120/240/480/1,000 steps, with other simulation settings unchanged. Full supplementary results and evaluation protocols are available at: CR supplementary results and SD supplementary results.

DatasetCR@1mm ↑ (Ours / Best Baseline)SD Reduction vs Best Baseline @240 Steps ↑
SHOWMe99.79% / 99.60%63.0%
HOT3D97.70% / 90.06%70.4%
HO-Cap97.30% / 85.32%46.3%

JIGAR achieves the highest CR@1mm and lowest mean SD at every evaluated horizon among the evaluated methods. All comparisons use each method's own valid outputs.

We appreciate the reviewer's constructive suggestion and will incorporate these additional evaluations into the revised manuscript.

W2: Clarification of Implementation Details

We thank the reviewer for emphasizing reproducibility. We clarify all three mechanisms and provide two diagrams in our anonymous supplementary material.

(1) Trainable adapter architecture. JIG freezes DINOv2, ShapeVAE, and the pretrained DiT backbone, training Q/K/V/O attention LoRA, bottleneck adapters (2048→64→2048) parallel to FFN/MoE in all 21 blocks, and MoE expert LoRA in the last 6 blocks.

StageTrainable modules and updates
I: Image-onlySelf-/cross-attention LoRA and bottleneck adapters.
II: Render-conditionedMerge Stage-I self-/cross-attention LoRA into base projections and freeze the merged weights; train new cross-attention LoRA (rank 64, all 21 blocks), bottleneck adapters, MoE expert LoRA, and patch-fusion MLP with modality embeddings.
DexYCB refinementCross-attention LoRA (last 4 blocks), bottleneck adapters, MoE expert LoRA, and patch-fusion MLP with modality embeddings.

(2) Latent–MANO correspondence. The first 779 of 4,096 surface queries are anchored to the fixed vertex order of the capped ground-truth MANO mesh; the other 3,317 are sampled from the hand–object union via FPS. We replace the frozen ShapeVAE encoder's default query selection with these ordered queries, keeping its weights frozen. Latent tokens preserve query indices, enabling hand- and contact-weighted flow matching.

(3) Residual hand registration. Rigid point-to-point ICP aligns the WiLoR hand to the scaffold without independently optimizing object pose.

StepImplementation
InputSource: 778 original WiLoR/MANO vertices (excluding wrist cap); target: external surface of the segmented scaffold hand.
Rigid ICPNearest-neighbor matching with distance/normal filtering; SVD-based rigid transform. Threshold decreases from 30 to 12 mm over at most 40 iterations, with early stopping. Scale and articulation remain fixed.
Coordinate transferApply the inverse transform to the regenerated object to return it to the WiLoR frame. Retain the original WiLoR hand as the final asset.

We appreciate the reviewer's feedback and will incorporate these details into the revised manuscript.

AuthorsOpenReview ↗

Response to Questions 1 and 2

Q1: Object Regeneration and Interaction Preservation

We thank the reviewer for highlighting how regeneration changes object scale, hand–object contact, and penetration relative to the scaffold, and whether the intended interaction is preserved. Our stage-wise evaluation below on 479 valid HOT3D reconstructions shows that regeneration introduces relatively limited changes in object scale, hand–object contact, and penetration relative to the scaffold: an average scale change of 2.814%, 100% CR@2mm after regeneration, and 3.376 cm³ of proxy-based intersection volume.

We evaluate three stages on JIGAR's own valid outputs (N=479) from HOT3D: (A) Scaffold (jointly generated hand and object), (B) Regenerated (scaffold hand and regenerated object), and (C) Final (registered WiLoR hand and regenerated object).

(1) Object scale and pose changes during regeneration (A→B).

MetricA→B: Regeneration
Rotation drift (°) ↓4.43
Translation drift (cm) ↓0.68
RMS radius change (%) ↓2.814
Sim(3) scale ratio0.9744

These results show that regeneration largely maintains the scaffold's global pose and scale, with an average rotation drift of 4.43°, translation drift of 0.68 cm, and RMS radius change of 2.814%.

(2) Hand–object contact and penetration across stages.

MetricA: ScaffoldB: RegeneratedC: Final
CR@2mm (%) ↑100.00100.0097.91
Intersection volume (cm³) ↓0.000†3.3763.890
Contact-patch CD@5mm (mm) *↓—2.372 (A→B)3.140 (B→C)

† The scaffold's zero IV is guaranteed by the complementary watertight proxy construction, which will be detailed in the revised paper.

* Contact-patch CD measures changes in the 5-mm contact regions on fixed reference surfaces: the scaffold hand for A→B and the regenerated object for B→C.

Overall, the stage-wise results indicate that regeneration largely preserves the scaffold's global configuration.

We will incorporate these stage-wise measurements and provide the additional experimental details in the revised manuscript. We appreciate the reviewer's constructive suggestion, which helps us provide a more comprehensive evaluation of the proposed recovery process.

Q2: Sensitivity to Hand-Pose Errors

We thank the reviewer for highlighting the training–inference mismatch between GT hand renderings and estimated hand conditions. We provide an anonymous hand-pose perturbation experiment report with stage-wise curves, complete numerical results, and detailed experimental protocols. This controlled hand-pose perturbation experiment directly evaluates this sensitivity: at a moderate perturbation (10° RMS), final F@5h decreases by only 1.6% (0.3410 → 0.3357). At a larger perturbation (20° RMS), errors propagate significantly to both the scaffold and final reconstruction (final F@5h: 0.2843, −16.6%).

We vary only the MANO finger articulation at 0°/5°/10°/20° RMS joint rotation, keeping the remaining inference pipeline fixed:

Rotation RMSResulting Hand MPJPE (mm) ↓Scaffold CDh (mm) ↓Scaffold F@5h ↑Final CDh (mm) ↓Final F@5h ↑
0°13.3434.140.306228.360.3410
5°13.7634.390.308228.730.3420
10°15.3135.620.302129.660.3357
20°19.9440.660.254233.850.2843

We will incorporate these controlled perturbation results and experimental details into the revised manuscript. We thank the reviewer again for this constructive suggestion.

Thank the authors for their detailed responses, which have helped clarify several aspects of the paper. I appreciate their efforts in addressing my concerns and will maintain my original score of Weak Accept.

Thank you for carefully considering our responses. We sincerely appreciate your time and thoughtful feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Responses to Reviewers and Manuscript Revisions

We sincerely thank the reviewers for their valuable feedback. We summarize how we addressed the concerns and revised the manuscript. We welcome any further questions and would be happy to provide clarification or evidence as needed.

  1. Novelty relative to FollowMyHold.

    The key distinction is how the final hand–object configuration is established. FollowMyHold uses coarse hand–object geometry for initialization and then determines the configuration through an optimization-in-the-loop framework. JIGAR instead establishes it through joint interaction generation (JIG) and uses the scaffold as a spatial reference for in-place asset recovery, largely preserving the interaction without separate relative-placement optimization. Our ablation supports this anchoring role: replacing scaffold initialization with pure noise raises CDₕ from 27.39 to 147.25 mm.

    Revisions: Revised the Introduction and Section 3.3 to clarify this distinction.

  2. The role and stability of hand-only ICP.

    Hand-only ICP provides the rigid mapping needed to replace the generated hand with a MANO hand. It uses no object geometry or interaction constraints and thus does not constitute explicit optimization of hand–object relative placement. To test efficiency and stability, we measured runtime and perturbed the ICP initialization. Registration takes 0.23 s/case, and perturbations up to 10° and 10 mm yield a mean hand-vertex deviation of 0.432 mm from the original solution, supporting lightweight, stable coordinate correction.

    Revisions: Added Appendix I with registration details and evaluations.

  3. Separate evaluation of the alignment contribution.

    As suggested by the reviewer, we registered generated objects to their GT counterparts to estimate pose and scale errors separately from shape quality. Shape differences can influence this registration, so these values offer an approximate reference. Under this protocol, JIGAR’s estimated translation error is 25.19 mm, versus HORT’s 54.85 mm, with lower rotation and scale errors. The stage-wise comparison also indicates that recovery largely retains scaffold alignment.

    Revisions: Added Appendix J with the protocol and comparisons.

  4. Interaction preservation during asset recovery.

    To assess how well recovery preserves the scaffold’s configuration, we evaluated regeneration and hand replacement separately. This evaluation shows 4.43° of rotation drift, 0.68 cm of translation drift, and a 2.814% change in RMS radius during regeneration, while the final contact rate remains 97.91% at 2 mm. Hand replacement preserves object geometry and scale through rigid transfer, while intersection volume changes slightly from 3.376 to 3.890 cm³. These results indicate that recovery largely preserves the interaction.

    Revisions: Added Sections G.2–G.3 with the stage-wise evaluation.

  5. Controlled evaluation of generalization.

    To clarify, our training and evaluation sets share no hand–object interaction samples. A small number of object identities are nevertheless shared between the DexYCB refinement and the evaluation sets, but these shared instances account for only 2.7% of evaluation images. To test whether this instance overlap explains the gains, we first excluded shared instances from evaluation; JIGAR still performed best on unseen instances. Separately, we repeated refinement after removing all views of the shared objects, obtaining nearly unchanged performance. The gains therefore persist after controlling for instance overlap.

    Revisions: Added Appendix K with the overlap analysis and control tests.

  6. Stricter CR and SD evaluations.

    To test robustness under stricter settings, we reduced the contact threshold from 10 mm to 1/2/5 mm and extended simulations to 1,000 steps. Across three datasets, JIGAR retains the highest contact rates (97.30–99.79% at 1 mm) and has the lowest mean simulation displacement, 46.3–70.4% below the best baselines at 1 s. Its advantage holds across all tested thresholds and horizons, supporting relative physical robustness.

    Revisions: Added Appendix H with these tests and protocols.

  7. Sensitivity to hand-pose errors.

    As suggested by the reviewer, we perturbed MANO articulation with other inputs fixed to assess sensitivity to hand-pose errors. Final F@5ₕ decreases by 1.6% at 10° RMS and 16.6% at 20°, relative to the unperturbed estimate. These results show JIGAR’s tolerance to moderate perturbations.

    Revisions: Added Section B.2 with the protocol and stage-wise results.

  8. Details of the key mechanisms.

    As requested, we expanded Section F.2 to specify the adapter architecture and modules trained or frozen at each stage. Section F.3 now explains ordered queries, latent–MANO correspondence, and weighted supervision. We also added a Reproducibility Statement pointing to reproducibility details.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.