acceptodds
Under review as a conference paper at ICLR 2027

Touch-HOI: Active Tactile 3D Reconstruction from Hand–Object Interactions

Abstract

Humans can infer an object's shape through touch, even when hidden inside an opaque container. For robots, tactile 3D reconstruction without visual input remains challenging because sparse contacts reveal only local surface patches. Existing methods underexploit structured hand–object interaction cues and lack explicit posterior representations to reason about shape uncertainty. In this paper, we present Touch-HOI, a unified generative framework for tactile 3D reconstruction with active exploration. Specifically, we formally unify tactile sensing and proprioception as Hand-Object Interaction Evidence, and encode their spatial distribution and statistics as Hand-Object Interaction Encoding. Building upon this, we construct a flow matching model with a pretrained 3D prior, establishing an explicit generative belief state over several shape hypotheses. Therefore, belief-guided grasp selection evaluates prospective tactile feedback via Bayesian reweighting to maximize expected geometric risk reduction without training an extra exploration policy. Extensive experiments on the ABC dataset demonstrate that Touch-HOI achieves mean Chamfer Distances of with five random grasps and with active selection, which verifies the effectiveness of the proposed Touch-HOI. The code will be made publicly available upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Shape Reconstruction, Active Perception, Tactile Perception, Hand–Object Interactions

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mPmMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Touch-HOI, a framework for tactile shape reconstruction and active grasp selection. It combines contact observations, valid non-contact observations, and posed-hand geometry to condition a latent flow model. Generated shape hypotheses support reconstruction and action selection through hypothetical Bayesian reweighting and geometric disagreement reduction.

Critical strengths

The paper shows that non-contact and hand-pose information can improve tactile reconstruction, with particularly clear benefits when contacts are sparse (Table 4). This is a useful finding for reconstruction from limited observations. The proposed grasp-selection strategy consistently improves reconstruction over the compared exploration policies (Table 3), without requiring additional policy training.

Critical weaknesses

The evaluation assumes a known object frame and does not test errors in hand pose or tactile registration. Since the method gains from spatially combining contact, non-contact, and hand geometry, it remains unclear how well these gains hold under imperfect alignment. This limits the evidence for use beyond the simulated benchmark.

Questions for authors

How sensitive are the gains from non-contact and hand-pose information to small registration errors? A modest perturbation test would help establish whether these cues remain beneficial under imperfect alignment.

1 comment
AuthorsOpenReview ↗

Response to Reviewer mPmM

We sincerely thank the reviewer for recognizing that “non-contact and hand-pose information can improve tactile reconstruction” with “particularly clear benefits when contacts are sparse” and that the proposed method “consistently improves reconstruction without requiring additional policy training.” We also appreciate your constructive suggestion to assess whether these benefits persist under imperfect alignment.

W1. Robustness to imperfect hand–object alignment.

We recognize that accurate alignment cannot be taken for granted, particularly in real-world deployments. According to [1], the hand–object registration errors typically remain within 5 mm in translation and 5° in rotation, we therefore evaluate registration errors that jointly transform contact observations, non-contact observations, and posed-hand geometry. As summarized in the table below, the original model exhibits strong robustness at 1 mm/1° and relatively stable performance at 2.5 mm/2.5°. Under the extreme stress test of 5 mm/5° registration errors, a more pronounced degradation is observed. However, this degradation can be substantially mitigated through retraining with the same data augmentation. Furthermore, to account for tactile sensing imperfections, we evaluate our method's robustness by injecting 1 mm Gaussian noise into the contact positions. As shown in the table below, our method achieves a Chamfer Distance (CD) of 2.411 under 1 mm Gaussian noise (approximately 5.7% of the tactile sensor width), demonstrating robustness to potential tactile hardware noise.

Noise conditionTraining settingMean CD
Clean referenceOriginal2.399
Registration: 1 mm/1°Original2.400
Registration: 2.5 mm/2.5°Original2.577
Registration: 5 mm/5°Original3.657
Registration: 5 mm/5°Retrained with the same augmentation3.044
Gaussian noise in Tactile Sensing: 1 mmOriginal2.411

Q1. Do non-contact and hand-pose cues remain beneficial under registration errors?

As shown in the table below, our model exhibits strong resilience under common 1 mm/1° registration errors and the larger 2.5 mm/2.5° registration errors, achieving Mean CD values of 2.400 and 2.577, respectively. Both values are lower than those of the models without hand proprioception (2.618) or non-contact evidence (3.202), even when these two models are evaluated without any registration errors. These results demonstrate the effectiveness of our method under registration errors.

ModelMean CD
w/o registration errors2.399
Registration errors: 1 mm/1°2.400
Registration errors: 2.5 mm/2.5°2.577
w/o hand proprioception2.618
w/o non-contact3.202

Reference

[1] Nonnengießer et al. In-Hand Object Pose Estimation via Visual-Tactile Fusion. arXiv:2506.10787, 2025.

Reviewer_1CeZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Touch-HOI, a framework that performs 3D reconstruction of objects, using tactile inputs of a simulated allegro hand grasping the object. It also provides a grasp selection module based on multiple shapes hypothesis. The key contributions of this paper are :

  • The global framework that uses both tactile inputs for 3D reconstruction and the active exploration
  • The "Hand Object Interaction Evidence and Encoding" formulation of the inputs
  • The belief guided grasp selection strategy. The method is evaluated on ABC and ShapeNet, reporting a mean Chamfer distance of 2.399 with five random grasps and 2.093 with active selection on ABC.
Critical strengths
  • Clear motivation and problem statement.
  • Table 4 shows clearly that non-contact and proprioceptive information help especially when contacts are sparse.
  • The limitations and failure cases are also discussed honestly.
  • The explanations in the appendix are thorough and clearly describe the method.
Critical weaknesses
  • Insufficient description and ablation of the belief-guided grasp selection : The paper presents its belief guided grasp selection as one of its main contributions. However, the explanations of the method in section 3.5 are too short. The geometric contact model is not clearly explained there, and the notations are not introduced clearly. Also, the pipeline relies on a set of parameters that are not analyzed in this paper such as: N=16 planning hypotheses (vs 8 for reconstruction), the tolerance schedule (0.02,0.05 ,0.10 ,0.15), λ=0.65\lambda = 0.65 in DijD_{ij}, the tempering α=0.75\alpha = 0.75, the binning of 5 contact stages into 4 categories. Importantly, for the last two grasps the choice is made by a directional-novelty heuristic among near best candidates (B.5.4) which is not mentioned in Section 3.5. Ablation studies on the pipeline's design choices are needed to justify the design such as the number of hypotheses and also the different impact of the selection e.g. risk only ; two-step lookahead for all steps ; directional novelty on all steps. If the space is too tight, maybe the introduction could be shortened slightly.

  • Unexcepted exploration results in table 3 : Table 3 presents the mean Chamfer distance for different numbers of grasps across 4 different exploration policies. The RL baseline improves over random for K=2 to 4 but at is worse K=5 (2.605 against 2.399). This is unexpected and blur the global results. Was the RL policy retrained for Touch-HOI or transferred from Touch2Shape directly ?

  • Clarity of main contribution in the main text: The statistical and spatial interaction encoding could benefit from a clearer presentation in Section 3.3 since it is one of the main contributions. I would suggest moving the mathematical formulation that is inside the appendix to the main text in order to improve clarity.

Questions for authors
  • Can you report variance over seeds for your runs ? If you ran training using only 1 seed, I'd recommend to use at least 2 or 3 to ensure that performance gains are coming from the method.
  • How sensitive are the results to the number of integration steps ? Also, since you are doing 50 interactions x 16 (number of hypothesis) how long is one inference forward ?
  • In table 3 what is the "Even" baseline ?
  • In figure 4 why do you compare qualitatively to Active VT method and not Touch2Shape that seems the closest method. Also do you have an explanation of why the results of Active VT are so "spiky" ?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 1CeZ(Part 1/2)

We sincerely thank the reviewer for recognizing our “clear motivation and problem statement”, the evidence that “non-contact and proprioceptive information help especially when contacts are sparse” and “the explanations in the appendix are thorough and clearly”. We also sincerely appreciate your constructive suggestions and have expanded the main-text presentation and added further ablation studies to better justify the active exploration strategy.

W1. More details and ablations of belief-guided grasp selection.

We appreciate the reviewer’s detailed suggestions. With the additional page permitted for the rebuttal revision, we have revised the manuscript to provide a clearer explanation of the active exploration strategy and added more ablations. Below, we summarize these clarifications and ablations.

1. Number of hypotheses. We sample multiple shape hypotheses from the distribution learned by the generative model and use their geometric disagreement to guide next-grasp selection without separately training an RL policy. As shown in the table below, N=16N=16 achieves the lowest Mean CD, while increasing NN to 32 adds inference time without improving accuracy.

Number of hypothesesMean CD ↓Latency (s)
82.2124.895
16(default)2.0936.444
322.1279.865

2. Tolerance schedule. At each selection, we retain actions whose expected one-step risk reduction is at least 1−ϵ1-\epsilon of the maximum, then apply lookahead or directional novelty within this shortlist. Increasing ϵ\epsilon gradually relaxes the risk constraint, allowing more flexibility in later selections. Our ablation below shows relatively small differences across three different schedules. The default setting achieves the lowest Mean CD among the ABC test-set, although its optimality is not guaranteed.

Tolerance across the four stagesMean CD ↓
0 / 0 / 0 / 02.186
0.08 / 0.08 / 0.08 / 0.082.161
0.02 / 0.05 / 0.10 / 0.15(default)2.093

3. Geometric-risk weight. The geometric-risk weight λ\lambda balances two complementary measures of disagreement between shape hypotheses: occupancy sign captures whether a location is inside or outside the object, while truncated distance captures differences in proximity to the surface. As shown in the table below, λ=0.80\lambda=0.80 achieves the lowest Mean CD, while the default setting λ=0.65\lambda=0.65 performs comparably. The modest variation across the ABC test-set suggests lower sensitivity to this hyper-parameter.

Geometric-risk weight λ\lambdaMean CD ↓
0.502.162
0.65(default)2.093
0.802.091

4. Likelihood tempering. The exponent α\alpha softens the joint predictive likelihood before renormalization, reducing overconfidence in hypothetical observations. As shown in the table below, the default α=0.75\alpha=0.75 achieves lower Mean CD than both stronger softening 0.50 and no tempering 1.00, demonstrating a balance between moderating confidence and preserving informative differences.

Likelihood tempering α\alphaMean CD ↓
0.502.225
0.75(default)2.093
1.002.166

5. Contact-stage binning. We categorize contact outcomes into four stages: early, middle, late, and no contact. This representation retains coarse information on where fingers contact the object along the closing trajectory while keeping the joint outcome space tractable. As reported in the table below, reducing this to binary contact/no-contact labels increases Mean CD from 2.093 to 2.333, demonstrating the value of contact-stage cues in grasp selection.

Contact modelMean CD ↓
Contact/no-contact outcomes2.333
Multi-stage contact outcomes(default)2.093

6. Risk, lookahead, and directional novelty. Geometric risk first identifies promising action candidates, within which lookahead and directional novelty perform fine-grained ranking. Specifically, our phased strategy employs two-step lookahead for the second and third grasps to resolve global shape ambiguity early, before transitioning to directional novelty for the fourth and fifth grasps to encourage complementary observations. As shown in the table below, this staged schedule achieves a lower Mean CD than applying one-step risk, lookahead, or directional novelty uniformly across all steps, albeit with modest margins.

Selection StrategyMean CD ↓
One-step geometric risk2.180
Two-step lookahead2.148
Directional novelty2.196
Default2.093

Main-text Presentation. The ICLR page limit led us to place these implementation details in the appendix. Following your suggestion, we have revised our manuscript and added more details about active exploration strategy with one extra page for revision. Thanks for your kind suggestion.

AuthorsOpenReview ↗

Response to Reviewer 1CeZ(Part 2/2)

W2. RL baseline and its performance at five grasps.

Since the official codebase and pretrained weights of Touch2Shape were unavailable, we independently implemented and trained a Touch2Shape-inspired DQN baseline. Adapting its diffusion loss-based reward design, we formulated the reinforcement learning (RL) reward using a flow-matching loss. For reference, Touch2Shape reports that the Chamfer Distance (CD) after five RL-selected grasps drops to 6.63% of the zero-grasp baseline. Applying this ratio to our zero-grasp CD of 40.283 yields an estimated final CD of 2.671, which aligns closely with the 2.605 achieved by our RL baseline.

As you perceptively note, RL outperforms Random at K=2–4 but falls behind at K=5. This trend indicates that while the loss-based reward offers effective guidance during early interactions, but its utility saturates as observations accumulate, where further decreases in generative loss do not necessarily yield commensurate gains in Mean CD.

W3. Clarity of Statistical and Spatial interaction encoding.

We agree that these components should be more clearly formulated in the main text. Following your suggestion, we have moved their key mathematical formulations from the appendix to Section 3.3 and explicitly defined the notation alongside each equation. The revised presentation explains how statistical encoding summarizes accumulated observations and contact incidence, while spatial encoding preserves the spatial locations of tactile evidence relative to the hand geometry. We also clarify how both representations condition shape generation, making their complementary roles explicit, while keeping detailed implementation settings in the appendix. Thank you for your constructive suggestion, which has significantly improved the clarity of our manuscript.

Q1. More random seeds.

Following your suggestion, we trained 5 independent models using 5 different random seeds, which yield Mean CDs of 2.5327, 2.4199, 2.5398, 2.3990, and 2.4636, respectively. Across these five runs, our method achieves an overall Mean CD of 2.4710 ± 0.0640, indicating relatively consistent reconstruction performance across training seeds.

Q2. Ablation on Integration steps and inference time. We evaluate 25, 50, and 100 integration steps while keeping the reconstruction and planning sample counts fixed. As shown in the table below, 25 steps already achieve performance close to the default 50-step setting, particularly under active selection. Notably, performance under the Random baseline saturates at the default 50 steps, showing no gains at 100 steps and suggesting that additional integration steps do not necessarily enhance reconstruction quality.

Integration stepsRandom 5 graspsActive 5 graspsLatency (s)
252.5072.1124.357
50(default)2.3992.0936.444
1002.4772.07710.882

In terms of inference speed, our default setting takes approximately 6.444 s per step for active grasp selection and reconstruction on an RTX 4090, which effectively balances reconstruction quality and runtime.

Q3. What is the Even baseline?

Following ActiveVT[1] and Touch2Shape[2], the Even baseline selects evenly spaced actions from the grasp library. Specifically, starting from the shared initial grasp, our implementation selects every tenth index within the 50-action library using circular wraparound. While this strategy approximates uniform sampling across the action indices, it does not guarantee uniform geometric coverage in 3D space.

Q4. Qualitative comparison and ActiveVT’s spiky meshes.

The surface spikes are inherent to ActiveVT’s original chart-deformation representation, as we directly render its predicted vertices and mesh connectivity without post-processing. As noted in Sec. 6 of ActiveVT, the authors attribute this degraded mesh quality to optimizing solely with CD without explicit mesh regularization. Consistent with their analysis, severely stretched, sliver triangles, and similar artifacts are also reported in Touch2Shape[2].

As for qualitative comparisons with Touch2Shape[2], because its official codebase and pretrained weights are still unavailable, it is difficult to guarantee that a reimplementation would faithfully reproduce its visual reconstruction quality. Consequently, we primarily compare against the quantitative results reported in Touch2Shape[2], rather than qualitative results.

[1] Smith et al. Active 3D Shape Reconstruction from Vision and Touch. NeurIPS, 2021.

[2] Wang et al. Touch2Shape: Touch-Conditioned 3D Diffusion for Shape Exploration and Reconstruction. CVPR, 2025.

Reviewer_1CeZOpenReview ↗

Thank you for the clarifications

Thank you for your response and for the revisions. I will take them into account in my final assessment.

AuthorsOpenReview ↗

Thank you for the timely reply!

Dear Reviewer 1CeZ:

Thank you for your timely reply and for considering our responses and revisions in your final assessment. We greatly appreciate your time and constructive feedback. If any additional questions or concerns remain, please do not hesitate to let us know, and we will gladly provide further clarification.

Reviewer_hf7gOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents Touch-HOI, a framework for active tactile 3D reconstruction using hand–object interactions. The proposed method combines tactile contact observations, and hand proprioception to condition a flow-matching generative model built on a pretrained TRELLIS representation. The model generates multiple plausible object shapes, which are then used for belief-guided grasp selection through Bayesian reweighting and expected geometric risk reduction. The method is evaluated on simulated ABC and ShapeNet datasets, showing improvements over existing tactile reconstruction and active exploration approaches.

Critical strengths
  • Using information from the entire hand is well motivated. A grasp provides more geometric information than just the contact patches. So the belief-guided grasp selection is a sound idea. It reuses the generative model’s own sample diversity as a posterior over shapes and ranks grasps by expected reduction in hypothesis disagreement, so it requires no separately trained exploration policy, unlike the RL-based baselines.

  • The experimental results are promising. The proposed method achieves lower reconstruction errors than the compared baselines on ABC and ShapeNet. The improvements in both random and active exploration settings are encouraging.

  • The authors are upfront about its limitations: the authors clearly state that experiments are limited to simulation and that exploration selects from a discrete set of grasps rather than planning continuous motion.

Critical weaknesses
  • Limited real-world applicability and strong assumptions: The evaluation is entirely simulation-based, and several assumptions may not hold in real robotic systems. The method relies on accurately registered tactile observations, hand configurations, and non-contact information, while real sensors introduce noise and calibration errors. The exploration also selects from a predefined set of up to 50 grasps whose feasibility is already provided. In practice, generating feasible grasps for unknown geometries is itself challenging. These assumptions significantly simplify the problem, and the lack of robustness experiments makes it difficult to assess the applicability of the proposed method to physical robots.

  • Limited methodological novelty and incomplete related work: Most components build on existing techniques, including tactile representations from prior work, flow matching with a pretrained TRELLIS model, and Bayesian uncertainty reduction for active sensing. Non-contact and free-space constraints have also been explored previously. The contribution appears to be primarily the integration of these components, without sufficiently demonstrating what is fundamentally new. Moreover, the literature review overlooks relevant works including AcTExplore [1], ViTract [2], Proactive Tactile Exploration [3], and ShapeMap 3-D [4]. Discussing these methods would help clarify the novelty and advantages of the proposed approach compared with existing tactile exploration and probabilistic reconstruction techniques.

  • Insufficient justification of the exploration objective: The method evaluates candidate grasps using predicted contact stages rather than the richer tactile surface information used for reconstruction. It is unclear whether grasps that maximally distinguish hypotheses based on contact timing necessarily produce the greatest improvement in reconstruction quality. The contact model also assumes conditional independence between finger outcomes, which may not hold during physical interactions. Comparisons with simpler uncertainty-based or spatial exploration strategies would help establish which components are actually responsible for the reported improvements.

References [1] Shahidzadeh et al., AcTExplore: Active Tactile Exploration on Unknown Objects, ICRA 2024. [2] Dutta et al., ViTract: Robust Object Shape Perception via Active Visuo-Tactile Interaction, RA-L 2024. [3] Oikonomou et al., Proactive Tactile Exploration for Object-Agnostic Shape Reconstruction from Minimal Visual Priors, ICRA 2025. [4] Suresh et al., ShapeMap 3-D: Efficient Shape Mapping Through Dense Touch and Vision, ICRA 2022.

Questions for authors
  1. How sensitive is the method to noise in contact positions and hand configuration?

  2. Does the selection criterion correlate with actual CD improvement per grasp?

5 comments
AuthorsOpenReview ↗

Response to Reviewer hf7g(Part 1/3)

We sincerely thank the reviewer for recognizing that “using information from the entire hand is well motivated”, “the belief-guided grasp selection is a sound idea”, and “The experimental results are promising”. We also appreciate the assessment that “The improvements in both random and active exploration settings are encouraging”.

W1. Noise Robustness, Interaction Assumptions and Real-World Deployment.

Noise Robustness. For a calibrated hand, interaction noises primarily arise from imperfect hand–object registration. According to [1], the hand–object registration errors typically remain within 5 mm in translation and 5° in rotation, we therefore evaluate registration errors that jointly transform contact observations, non-contact observations, and posed-hand geometry. We use 1 mm/1° and 2.5 mm/2.5° as common conditions, and the 5 mm/5° registration errors as the extreme stress test. As shown in the table below, the proposed method demonstrates strong robustness to 1 mm/1° registration errors, with Mean CD changing only from 2.399 to 2.400, and maintains stable performance under the 2.5 mm/2.5° registration errors, with a CD of 2.577. Even under the extreme stress test of 5 mm/5°, the proposed method achieves a mean CD of 3.657. Importantly, retraining with the same registration augmentation substantially improves CD to 3.044, demonstrating strong robustness to large registration errors. In addition, we also inject Gaussian noise into the tactile observations to simulate contact-localization errors. As shown in the table below, our method maintains a Mean CD of 2.411 when 1 mm Gaussian noise (~5.7% of the tactile sensor width) is injected into the contact positions, underscoring its robustness to tactile hardware and localization noise.

Noise ConditionTraining SettingMean CD
Clean referenceOriginal2.399
1 mm/1°Original2.400
2.5 mm/2.5°Original2.577
5 mm/5°Original3.657
5 mm/5°Retrained with the same augmentation3.044
1 mm Gaussian noise in Tactile SensingOriginal2.411

Generating feasible grasps for unknown geometries is challenging. The “grasps” in our methodology are exploratory hand closures that require neither stable object holding nor fingertip contact at every interaction. Across the candidate library, 10.96% of interactions involve no fingertip contact, and 23.20% involve only a single contacting finger. These non-contact and sparse-contact interactions remain informative in our method because non-contacting fingers and the hand as a whole constrain where object geometry can exist.

Real-World Deployment. To demonstrate the real-world feasibility of the proposed method, we have conducted a real-world experiment using an Allegro hand equipped with Xense fingertip tactile sensors and mounted on a RealMan robotic arm. Specifically, we collected tactile observations across five interactions with a 3D-printed object selected from the ABC dataset. Using the five collected tactile interactions, the proposed method successfully reconstructed the object geometry. We kindly remind you to refer to Sec. 4.5 and Fig. 5 in our revised manuscript, which present the experimental setup, the tactile observations from the five interactions, and the reconstruction results.

W2. Novelty and Related work.

To clarify, we do not claim flow matching or pretrained TRELLIS models as our contributions. The key insights underlying our work are: (1) hand-object relationships provide valuable geometric evidence for tactile 3D reconstruction, which is captured within and across grasps via the proposed Hand-Object Interaction Evidence and Encoding; and (2) disagreement among generated shape hypotheses guides informative interactions, enabling belief driven grasp selection without requiring additional RL policy training. Building on these insights, we establish a unified framework that integrates tactile 3D reconstruction with active exploration through a generative model.

Regarding the suggested references, AcTExplore[2] learns a coverage-oriented tactile RL policy, whereas our Touch-HOI conditions shape generation on hand–object interaction evidence and reuses the resulting shape hypotheses to guide exploration without training a separate policy. ViTract[3] and Proactive[4] combine tactile exploration with visual information, while Touch-HOI reconstructs shapes from tactile and proprioceptive evidence alone. ShapeMap 3-D[5] incrementally fuses touch and depth into a Gaussian-process SDF under passive exploration, whereas Touch-HOI performs active exploration without depth fusion.

We sincerely thank the reviewer for pointing out these relevant works, and we have revised the Related Work section to clarify their distinctions from our approach.

AuthorsOpenReview ↗

Response to Reviewer hf7g(Part 2/3)

W3. Insufficient justification of the exploration objective.

(1) Contact Stages We use contact-stage to simplify contact likelihood evaluation and make multi-candidate planning computationally tractable. Our contact-stage strategy of categorizing when fingers make surface contact preserves coarse spatial information while avoiding explicit modeling and integration over high-dimensional tactile-surface outcomes.

(2) Selection Criterion. Our criterion evaluates expected reductions in geometric risk, using contact stages to model prospective contact feedback. Table 3 in the manuscript provides empirical support for this criterion: the Mean CD decreases consistently from 8.660 to 4.012, 2.945, 2.323, and 2.093 as the number of grasps K increases from 1 to 5. These results demonstrate the effectiveness of our selection criterion in improving reconstruction quality over the interaction budget. Moreover, it is worth pointing out that greedily maximizing single-step CD improvement at each interaction does not necessarily yield the optimal cumulative reconstruction after multiple steps.

(3) Finger Dependence. Finger independence assumption in here is a computational simplification for a given shape and action in active selection strategy, rather than for physical execution.

(4) Comparisons with Spatial Exploration Strategy As shown in the table below, our contact-stage strategy achieves a Mean CD of 2.093, outperforming the spatial exploration through hand-pose coverage (2.326), which demonstrate that modeling multi-stage contact timing yields superior reconstruction accuracy compared to simpler spatial exploration strategies.

Exploration configurationMean CD ↓
Ours2.093
Spatial exploration through hand-pose coverage2.326

We thank the reviewer for pointing out the insufficient justification, and we have revised our manuscript to provide more detailed explanation of the active exploration objective.

Q1. How sensitive is the method to noise in contact positions and hand configuration?

As detailed in W1, our method demonstrates strong robustness against common registration errors, achieving a Mean CD of 2.400 at 1 mm/1° and 2.577 at 2.5 mm/2.5°. Even under the extreme stress test of 5 mm/5° registration errors/noises, incorporating registration augmentation during training improves reconstruction accuracy, bringing the Mean CD down from 3.657 to 3.044 (a 16.8% reduction). In addition, our method maintains a Mean CD of 2.411 even when 1 mm Gaussian noise (~5.7% of the tactile sensor width) is injected into the contact positions, underscoring its robustness to tactile hardware and localization noise.

Noise ConditionTraining SettingMean CD ↓
Clean referenceOriginal2.399
Registration: 1 mm/1°Original2.400
Registration: 2.5 mm/2.5°Original2.577
Registration: 5 mm/5°Original3.657
Registration: 5 mm/5°Retrained with the same augmentation3.044
Gaussian noise in Tactile Sensing: 1 mmOriginal2.411

Q2. Does the selection criterion correlate with actual CD improvement per grasp?

Thank you for the question.While we have not explicitly measured the step-wise correlation with actual Chamfer Distance (CD) reduction per grasp, Table 3 in the manuscript provides strong empirical support for our criterion: the Mean CD decreases monotonically from 8.660 to 4.012, 2.945, 2.323, and 2.093 as the number of grasps K increases from 1 to 5. This consistent improvement across the interaction budget demonstrates that our selection criterion serves as an effective proxy for reconstruction quality. Moreover, greedily maximizing single-step CD improvement at each interaction does not necessarily yield the optimal cumulative reconstruction after multiple steps, as each action alters subsequent observations and future action choices.

Reference

[1] Nonnengießer et al. In-Hand Object Pose Estimation via Visual-Tactile Fusion. arXiv:2506.10787, 2025.

[2] Shahidzadeh et al. AcTExplore: Active Tactile Exploration on Unknown Objects. ICRA, 2024.

[3] Dutta et al. ViTract: Robust Object Shape Perception via Active Visuo-Tactile Interaction. IEEE Robotics and Automation Letters, 2024.

[4] Oikonomou et al. Proactive Tactile Exploration for Object-Agnostic Shape Reconstruction from Minimal Visual Priors. ICRA, 2025.

[5] Suresh et al. ShapeMap 3-D: Efficient Shape Mapping Through Dense Touch and Vision. ICRA, 2022.

AuthorsOpenReview ↗

Response to Reviewer hf7g(Part 3/3)

More Real-World Deployment Experiments.

Thank you again for your constructive suggestions regarding real-world applicability. Over the past three days, we have conducted additional real-world experiments on two ShapeNet objects: a jar and a bottle. Using the same robotic platform, Touch-HOI successfully reconstructed each object from five tactile interactions. We kindly remind you to review these additional results in Section 4.5 and the accompanying figure in our revised manuscript.

We sincerely appreciate your time and consideration, and we would be happy to provide any further clarification.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Area Chair and All Reviewers

Dear Reviewers and Area Chair,

We sincerely thank all reviewers for your careful evaluations and constructive suggestions. We appreciate your recognition of our work’s strengths, summarized as follows:

1. Clear motivation and well-motivated use of hand information (Reviewers hf7g and 1CeZ): “using information from the entire hand is well motivated” and “clear motivation and problem statement.”

2. Valuable non-contact and proprioceptive evidence (Reviewers 1CeZ and mPmM): both reviewers highlighted the benefits of these cues “when contacts are sparse.”

3. Effective exploration without additional policy training (Reviewers hf7g and mPmM): “the belief-guided grasp selection is a sound idea” and “consistently improves reconstruction without requiring additional policy training.”

To address your concerns, we have conducted additional experiments and revised the manuscript to further substantiate Touch-HOI’s robustness, demonstrate the feasibility of physical deployment, and clarify the active exploration strategy. We summarize the additional results and key revisions below:

1. Robustness to Noise and Registration Errors. To address the shared concerns of Reviewers hf7g and mPmM, we evaluated Touch-HOI under different levels of registration errors. The model maintains stable performance, with Mean CD increasing from 2.399 without errors to 2.400 at 1 mm/1° and 2.577 at 2.5 mm/2.5°. Under the strongest 5 mm/5° stress test, training with registration augmentation reduces Mean CD from 3.657 to 3.044.

2. Feasibility of Real-world Deployment. We achieved real-world deployment using an Allegro hand equipped with Xense G1-WS fingertip tactile sensors and mounted on a RealMan RM75-6F arm. Touch-HOI successfully reconstructs three objects using five tactile interactions per object. The revised manuscript presents our experimental setup, tactile observations, and reconstructions.

3. More Detailed Explanations and Ablations of Active Exploration. We revised Section 3.5 in the main text and added methodological details in the supplementary material to more clearer clarify the active exploration mechanism, along with additional ablations to justify its design.

Finally, we sincerely thank you once again for your time and constructive feedback. We believe Touch-HOI advances tactile 3D perception by demonstrating how sparse tactile and proprioceptive evidence enables robust reconstruction of complete object geometry, while generated shape hypotheses guide active exploration without a separately trained exploration policy. We hope these revisions and additions address your concerns, and we welcome any further questions or suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.