acceptodds
Under review as a conference paper at ICLR 2027

Elephant: Recognize and Reconstruct the World Blindly

12 views

Abstract

Understanding an object's geometry and semantic identity through physical contact is an important capability for robots operating with limited visual information. Yet each tactile observation reveals only a small region of the surface, leaving open how much object-level understanding can be achieved from just a few contacts. We introduce Elephant, a framework for joint object reconstruction and recognition from sparse tactile observations, without visual input at inference time. Elephant uses sensor poses to express local observations in a common spatial frame and aggregates evidence across grasps to infer global geometry and semantic category. We also introduce a real-world tactile dataset collected using UMI-based grippers equipped with optical Daimon and magnetic eFlesh sensors, pairing tactile measurements with sensor poses and object geometry. The curated Daimon subset comprises 52 objects from 3 families, while the eFlesh study covers 6 semantic categories composed of 18 instances. These real-world observations are complemented by simulated tactile data from 279 object instances spanning 43 semantic categories. On real Daimon data, Elephant achieves a CD-L1 of 6.19 mm with ten grasps. In a separate six-category eFlesh study with six held-out object instances, it achieves 94.4% recognition accuracy over 18 sampled inputs at 20 tactile episodes, alongside a reconstruction F-score of 56.2%, compared with 51.8% for the strongest evaluated reconstruction baseline. These results highlight the potential of spatially localized tactile observations for recovering global object shape and semantic identity across optical and magnetic sensing technologies.

Primary area:
applications to robotics, autonomy, planning
Keywords:
tactile perception, few-touch reconstruction, object recognition, tactile datasets, sim-to-real transfer, cross-sensor adaptation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WfdPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Elephant, which is a framework that jointly reconstructs 3D object geometry and recognizes semantic categories from a small number of localized tactile observations. Contact geometry is combined with spatial attention, and the method then conditions reconstruction on predicted category probabilities. The authors also collected datasets using optical Daimon and magnetic eFlesh sensors and evaluated the method with different numbers of tactile observations, in simulation and on real objects.

Critical strengths
  1. I find particularly interesting the joint formulation of recognition and reconstruction from few tactile contacts. The hierarchical encoding of local patches and relations between contacts is pretty elegant, and soft class conditioning provides a natural way to couple semantic predictions with geometric completion.

  2. The real tactile datasets and experiments with optical and magnetic sensors are likely one of the most valuable practical contributions. Supporting both sensing technologies makes the framework interesting beyond a single hardware setup. I believe the planned release of data and calibration information would also benefit the community.

  3. Holding out physical object instances, rather than only individual recordings, is a good choice for assessing generalization. The evaluation across touch budgets also provides useful insight into how additional observations affect recognition and reconstruction.

Critical weaknesses
  1. The supplementary material mentioned in the reproducibility statement was not provided as a file or in the pdf, so I couldn’t fully check calibration, training, sampling, and baseline configurations.

  2. The separate contributions of geometric attention, semantic conditioning, and joint training are hard to assess on their own. The eFlesh baseline adaptation is also unclear, since magnetic vectors differ from the partial point clouds expected by the comparison methods. Although TouchSDF and Touch2Shape are discussed in the introduction and Table 1, they are not included as experimental baselines; comparing against them would strengthen the evaluation, especially given that the gains on Daimon are limited.

  3. While the 94.44% eFlesh accuracy is encouraging, it comes from 18 sampled inputs across only six held out objects, with three subsets from one recording per object; this and the absence of uncertainty estimates limit the strength of the generalization claim.

  4. I would like more clarification on the scope of the blind perception and transfer claims, especially as the known object frame depends on offline visual and CAD alignment, without an evaluation of alignment sensitivity. It is also unclear how the Daimon model is trained and whether learned weights transfer to eFlesh.

  5. Simba framework in section 4.1 and Figure 7 is missing a citation, so would be good to add it in the final version.

Questions for authors
  1. Could you add ablations showing how much geometric attention, semantic conditioning, and joint training each contribute to reconstruction and recognition?

  2. What happens when the predicted category is wrong? Does semantic conditioning still help reconstruction, or does it push the shape toward the wrong category?

  3. How are the magnetic measurements given to the eFlesh baselines, and do all methods get the same information? Could you also clarify which models use simulated or real training data, and whether any weights are transferred between sensors?

  4. Are the eFlesh gains consistent across the six held out objects and different sampled inputs?

5 comments
AuthorsOpenReview ↗

Response to WfdP (1/3)

Thank you for your patience and for asking how Elephant’s components affect its predictions and how the real-robot and eFlesh results should be interpreted. We took additional time to complete our supplementary real-robot experiments after our Daimon tactile sensor failed and we sent it for repair. We report the relevant results below.

W1 — Reproducibility

Appendix A of the revised manuscript documents the original Daimon registration chain, the supplementary xArm7 acquisition and scoring settings, and the Real38 and eFlesh evaluation protocols. Figures 3 and 4 show the original registration chain, robot setup, tactile measurements, and corresponding 3D contacts.

W2.1 / Q1 — Component ablations and joint training

To quantify the contribution of the geometric and semantic components, Table R1 compares the full model with three single-component ablations, a reconstruction-only variant, and a plain Transformer.

Table R1. Real38 reconstruction ablations.

Training used simulated and real tactile data, with fixed 10,000-step checkpoints and three matched seeds. We evaluated 273 cases from seven held-out objects, with 91 cases at each of K = 1, 5, and 10. Within each seed, we averaged reconstruction scores equally across the three K values, giving equal weight to shape groups and to objects within each group. Entries report the mean ± sample SD of these aggregated scores across the three training seeds. The held-out objects lack official class labels. We also report Density-aware Chamfer Distance (DCD; Wu et al., 2021) to assess how well the local point densities of the reconstructions match those of the ground-truth point clouds.

VariantDCD ↓CD-L1 (mm) ↓F@5 mm (%) ↑F@10 mm (%) ↑
Full0.9304 ± 0.002112.41 ± 0.2135.35 ± 1.9857.54 ± 0.38
Without inter-contact geometric bias0.9318 ± 0.004012.54 ± 0.1835.10 ± 1.9857.33 ± 1.94
Without soft semantic conditioning0.9332 ± 0.002112.42 ± 0.1434.62 ± 0.8457.23 ± 0.51
Reconstruction only0.9318 ± 0.001012.62 ± 0.2034.99 ± 0.2356.52 ± 0.19
Without within-contact k-NN0.9383 ± 0.002412.86 ± 0.8830.88 ± 3.2252.18 ± 4.84
Plain Transformer0.9336 ± 0.002612.72 ± 0.9634.81 ± 0.5756.92 ± 1.27

In Table R1, removing within-contact k-NN caused the largest drops in mean F@5 and F@10. Removing geometric bias or soft semantic conditioning had smaller effects on these metrics. Table R1 measures the contribution of Daimon’s within-touch point graph. Q3.1 describes the eFlesh graph over contact tokens. In Table R1, the full model achieved better mean scores than the reconstruction-only variant on all four reconstruction metrics.

Q2 — Effect of an incorrect predicted category

To test whether a wrong category can affect reconstruction, we kept the checkpoint and tactile inputs fixed and changed only the category signal given to the decoder.

Table R5. Category conditioning on labeled Real38 validation cases. The same 117 cases come from three physical computer-mouse instances. Scores give equal weight to each instance and to K = 1, 5, and 10.

Decoder category inputCD-L1 (mm) ↓F@5 (%) ↑F@10 (%) ↑
Predicted soft probabilities6.8949.5480.49
Ground-truth category6.8348.9280.83
Forced wrong category7.8847.6179.12
Conditioning disabled7.9047.9079.38

On these three mouse instances, using the predicted soft category probabilities improved all three mean reconstruction metrics relative to disabling conditioning. CD-L1 decreased by 1.01 mm, and F@5 and F@10 increased by 1.64 and 1.11 percentage points. Providing a wrong category instead of the ground-truth category increased CD-L1 by 1.05 mm and reduced F@5 and F@10 by 1.31 and 1.71 percentage points. The scores with a forced wrong category were close to those with conditioning disabled. These results show that category conditioning contributes to reconstruction on this subset. Whether natural misclassifications shift reconstructed shapes toward an incorrect category remains open.

AuthorsOpenReview ↗

Response to WfdP (2/3)

W2.2–W2.3 / Q3.1 — eFlesh baseline inputs and comparability

Yes. All methods received the same standardized 15-dimensional magnetic measurements, contact position and rotation, contact strength, and finger identity. We pair each magnetic observation with a time-aligned tactile sensor pose and use the pose’s XYZ translation as the observation’s position in a common coordinate frame. We added a magnetic tactile input encoder of the same design to each model.

Both variants retain Elephant’s object-level design: aggregating features across touches and using the resulting features for category prediction and shape reconstruction. The local graphs operate at different levels. Daimon connects 3D points within a touch, while eFlesh connects contact tokens by sensor position. The eFlesh graph described here is documented in the archived configuration for the Table 3 Local-Graph run.

Table 3 reports aggregate results for each method. Table R6 below compares the reported means by object.

Elephant’s Daimon pathway takes registered 3D contact patches as input, whereas the TouchSDF and Touch2Shape pipelines start from tactile images. We did not run a matched comparison because adapting the pipelines would require aligning the contact and pose information available to each method. A future comparison could evaluate the methods on the same recorded touches and test objects with a common touch budget and reconstruction metric.

W3 / Q4 — Variability across eFlesh objects and inputs

Table R6 shows how Elephant’s results vary across objects at K = 20 and how they compare descriptively with eight baselines across the six objects in the table.

Table R6. Per-object eFlesh results at K = 20. The first two score columns report Elephant’s reconstruction scores. The last three columns summarize the object-level comparisons between Elephant and the eight baselines. Accuracy wins, ties, and losses count how many baselines Elephant scored above, equal to, or below. The other two comparison columns count how many baselines Elephant outperformed on CD-L1 and F@5. The final row reports Elephant’s scores across six objects and the total across 48 object–baseline comparisons.

InstanceElephant CD-L1 (mm) ↓Elephant F@5 (%) ↑Accuracy wins / ties / lossesBaselines with higher CD-L1Baselines with lower F@5
camera27.0345.820/8/06 / 88 / 8
cup29.6738.751/7/05 / 86 / 8
microphone25.7554.367/1/08 / 88 / 8
mouse27.2847.595/0/36 / 88 / 8
pineapple23.5976.437/1/08 / 88 / 8
shoe14.0274.282/6/07 / 87 / 8
Across-object summary6.22 ± 2.2756.21 ± 15.6622/23/340 / 4845 / 48

Elephant had lower CD-L1 in 40 of 48 comparisons and higher F@5 in 45 of 48. For recognition, Elephant scored higher in 22 comparisons, tied in 23, and scored lower in 3. Across all six evaluated objects, Elephant’s accuracy was 94.44 ± 13.61%. Within an object, the sample SD across the three subsets ranged from 0.20 to 1.35 mm for CD-L1 and from 0.67 to 7.56 percentage points for F@5. The comparisons use object-level means.

AuthorsOpenReview ↗

Response to WfdP (3/3)

W4 / Q3.2 — Pose assumptions and transfer

In the original robot experiments, we used RGB-D/CAD alignment offline to establish the object frame and tracked sensor poses to register the tactile observations. The prediction network receives the registered tactile observations without RGB or depth images. In a supplementary xArm7 experiment, we used measured joint states and forward kinematics to obtain sensor poses, expressed contacts in the robot base frame, and centered inputs using the tactile observations. Aligned CAD models served as scoring references.

Table R3. Exploratory reconstruction results with proprioceptive localization. All methods used the same registered tactile inputs and CAD reference for each case. We averaged scores across recordings within each object, then weighted the cylinder, camera, mouse, and dragon fruit equally. The evaluation covers 10 recordings at K = 1, 3, and 5 and nine at K = 10. We used 1,024 predicted points and 2,048 reference points for scoring.

MethodCD-L1 at K = 1 (mm) ↓K = 3 ↓K = 5 ↓K = 10 ↓F@5 at K = 10 (%) ↑Balanced accuracy at K = 10 (%) ↑
Local-Graph10.908.517.947.0845.0250.0
PCN13.4410.539.818.8938.570.0
SnowflakeNet11.299.098.707.8043.7325.0
AdaPoinTr11.709.618.807.3442.5125.0
ProtoComp25.9521.0424.5317.1519.0825.0
SymmCompletion14.2310.279.458.4839.2225.0
ESCAPE16.3413.6311.279.7830.2050.0
PointSea14.2110.8410.899.1634.930.0
Hyper-PCN12.749.408.517.8143.1050.0

The results in Table R3 show that robot proprioception can supply the poses needed to register tactile contacts for reconstruction in this setup.

With Elephant’s weights fixed, combined perturbations of approximately 10 mm translation and 10° rotation RMS increased mean CD-L1 from 12.50 to 13.28 mm and reduced F@5 by 2.1 percentage points. This measures sensitivity to local registration errors. Systematic calibration offsets and accumulated drift remain untested.

For Figure 7, we trained Elephant and all baselines from scratch on simulated tactile data. For Daimon (Table 2), we pretrained each method on simulated tactile data and then fine-tuned it on the same real tactile training data. The Real38 ablations also used both simulated and real tactile data. The eFlesh model was trained from scratch with a magnetic encoder and no Daimon weights. Q3.1 describes the object-level design reused across the two sensor variants.

W5 — Simba citation

Thank you for pointing out the missing citation. The baseline in Section 4.1 and Figure 7 is Simba, Zhang et al., AAAI 2026. We have added it into our revised version of essay. Thanks again for pointing out this mistake.

Reviewer_jaiqOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Elephant, a framework for joint 3D object reconstruction and semantic recognition from sparse tactile observations without visual input to the prediction network at inference time. The method aligns tactile measurements using known sensor poses in a common object coordinate frame and employs hierarchical geometric encoding and Transformer-based aggregation to integrate information across multiple contacts. A semantic classification branch further conditions the reconstruction decoder to predict complete object point clouds. The authors also introduce real-world tactile datasets collected using UMI-based grippers equipped with optical Daimon and magnetic eFlesh sensors. Experiments are conducted on 279 simulated objects spanning 43 categories, 52 real objects using Daimon, and 18 objects using eFlesh. Comparisons with existing point cloud completion methods evaluate reconstruction accuracy and semantic recognition under different tactile observation budgets, demonstrating competitive reconstruction performance and improved recognition accuracy in selected settings.

Critical strengths
  1. Real-world tactile data collection across different sensing modalities. The paper collects real-world tactile datasets using two distinct sensing technologies, Daimon and eFlesh, with corresponding sensor poses and object geometry. These datasets provide useful resources for studying tactile-based object understanding and extend the evaluation beyond simulation.

  2. Quantitative evaluation under different tactile observation budgets. The authors systematically evaluate reconstruction and recognition performance with varying numbers of tactile observations, comparing against multiple established point cloud completion methods in both simulation and real-world settings. The results demonstrate competitive reconstruction performance under controlled conditions and provide empirical evidence of the potential of sparse tactile sensing for object-level perception.

Critical weaknesses
  1. Overly restrictive assumptions and insufficient validation of real-world applicability. The proposed framework assumes accurately known sensor poses in a common object coordinate frame, established through offline RGB-D/CAD alignment. This substantially simplifies the tactile-only reconstruction problem and raises concerns about applicability to previously unknown objects. More importantly, the paper does not explain how 6D pose uncertainty, calibration errors, or accumulated drift are handled during real-world tactile exploration, despite accurate spatial registration being essential to the method. The manuscript also provides insufficient details on sensor installation, calibration, and the complete data collection pipeline. The lack of supplementary documentation and real-robot demonstration videos in the reviewed materials further limits the reproducibility and independent verification of the reported results.

  2. Limited methodological novelty and insufficient experimental justification. The proposed architecture primarily combines existing geometric encoding, Transformer-based feature aggregation, semantic classification, and point cloud completion techniques, without introducing a clearly distinguished methodological innovation. Essential ablation studies are missing, making it difficult to identify which components contribute to the reported improvements. Moreover, the real-world evaluation remains limited, particularly the eFlesh experiments involving only six held-out object instances. The cross-sensor experiments demonstrate architectural reuse with sensor-specific encoders rather than actual cross-sensor knowledge transfer. Consequently, the experimental evidence is insufficient to substantiate the broader claims of robust and generalizable tactile object understanding.

Questions for authors

See above.

In Figure 3, what is the meaning of "Xingying Observations"?

2 comments
AuthorsOpenReview ↗

Response to jaiq (1/2)

Thank you for your patience and for highlighting Elephant’s real-world tactile datasets and evaluation across touch budgets. We took additional time to complete our supplementary real-robot experiments after our Daimon tactile sensor failed and we sent it for repair. We address your concerns about pose assumptions and experimental evidence below and report the relevant results.

Weakness 1 — Pose assumptions and reproducibility

Object frame and model inputs

Thank you for raising the question of how tactile contacts are localized. The reconstruction network takes registered tactile contacts as input and receives no visual images. In the original experiments, offline RGB-D/CAD alignment established the object frame, and calibrated marker tracking provided the sensor poses.

To answer your question, in Figure 3, “Xingying Observations” refers to the 3D marker positions recorded by Xingying, the motion capture system used in the original experiments. We use these positions to estimate the poses of the tactile sensors.

Real-robot reconstruction with proprioceptive localization

In our supplementary experiment, robot proprioception supplies the sensor poses for registering tactile contacts in the robot base frame, and CAD alignment is used only for scoring. We collected data on four objects with an xArm7 robot and a G2 gripper. We computed sensor poses from the robot’s measured joint states using forward kinematics and centered the registered contacts using the tactile observations. This video shows how we collected tactile data on the dragon fruit. https://tactile-demo-7f4c.pages.dev/pitaya-data-collection-4x.mp4

The xArm7 and G2 gripper position the two tactile pads around the dragon fruit. Figure 4 (p. 6) shows both tactile measurements and the corresponding 3D contacts in the original setup.

Table R3. Exploratory reconstruction results with proprioceptive localization. All methods used the same registered tactile inputs and CAD reference for each case. We averaged scores across recordings within each object, then weighted the cylinder, camera, mouse, and dragon fruit equally. The evaluation covers 10 recordings at K = 1, 3, and 5 and nine at K = 10. We used 1,024 predicted points and 2,048 reference points for scoring.

MethodCD-L1 at K = 1 (mm) ↓K = 3 ↓K = 5 ↓K = 10 ↓F@5 at K = 10 (%) ↑Balanced accuracy at K = 10 (%) ↑
Local-Graph10.908.517.947.0845.0250.0
PCN13.4410.539.818.8938.570.0
SnowflakeNet11.299.098.707.8043.7325.0
AdaPoinTr11.709.618.807.3442.5125.0
ProtoComp25.9521.0424.5317.1519.0825.0
SymmCompletion14.2310.279.458.4839.2225.0
ESCAPE16.3413.6311.279.7830.2050.0
PointSea14.2110.8410.899.1634.930.0
Hyper-PCN12.749.408.517.8143.1050.0

The results in Table R3 show that robot proprioception can supply the poses needed to register tactile contacts for reconstruction in this setup.

Sensitivity to pose errors

To quantify the effect of pose errors on reconstruction, we perturbed the registered contact poses while keeping Elephant’s weights fixed. We used 273 cases from seven Real38 objects. We used one checkpoint, with three perturbation draws per nonzero setting.

Table R4. Sensitivity to local pose errors on Real38. We perturb registered contact poses for 273 cases from seven objects, using one fixed checkpoint and three draws per nonzero setting.

Translation RMS (mm)Rotation RMS (°)CD-L1 (mm) ↓F@5 (%) ↑F@10 (%) ↑
0012.5035.5657.46
01012.6435.1656.96
10013.1333.7555.27
101013.2833.4454.56

At 10 mm translation and 10° rotation RMS, mean CD-L1 increased by 0.78 mm, while F@5 and F@10 decreased by 2.12 and 2.90 percentage points, respectively. These results show modest degradation under the tested local registration perturbations. We appreciate the reviewer’s point about systematic calibration offsets and accumulated drift. Evaluating these effects separately would help characterize Elephant’s robustness during longer real-world explorations. Given the time constraints of the rebuttal period, we focused this analysis on local registration errors and leave a dedicated evaluation of systematic calibration offsets and accumulated drift to future work.

AuthorsOpenReview ↗

Response to jaiq (2/2)

Weakness 2 — Methodological contribution and real-world evidence

Architecture and ablations

Compared with the plain Transformer baseline, Elephant explicitly models the local surface within each touch using a k-NN graph before aggregating information across touches. Table R1 tests the contribution of this graph by removing within-contact k-NN from Elephant. It also reports separate ablations of inter-contact geometric bias and soft semantic conditioning, along with a reconstruction-only variant.

Table R1. Real38 reconstruction ablations.

Training used simulated and real tactile data, with fixed 10,000-step checkpoints and three matched seeds. We evaluated 273 cases from seven held-out objects, with 91 cases at each of K = 1, 5, and 10. Within each seed, we averaged reconstruction scores equally across the three K values, giving equal weight to shape groups and to objects within each group. Entries report the mean ± sample SD of these aggregated scores across the three training seeds. The held-out objects lack official class labels. We also report Density-aware Chamfer Distance (DCD; Wu et al., 2021) to assess how well the local point densities of the reconstructions match those of the ground-truth point clouds.

VariantDCD ↓CD-L1 (mm) ↓F@5 mm (%) ↑F@10 mm (%) ↑
Full0.9304 ± 0.002112.41 ± 0.2135.35 ± 1.9857.54 ± 0.38
Without inter-contact geometric bias0.9318 ± 0.004012.54 ± 0.1835.10 ± 1.9857.33 ± 1.94
Without soft semantic conditioning0.9332 ± 0.002112.42 ± 0.1434.62 ± 0.8457.23 ± 0.51
Reconstruction only0.9318 ± 0.001012.62 ± 0.2034.99 ± 0.2356.52 ± 0.19
Without within-contact k-NN0.9383 ± 0.002412.86 ± 0.8830.88 ± 3.2252.18 ± 4.84
Plain Transformer0.9336 ± 0.002612.72 ± 0.9634.81 ± 0.5756.92 ± 1.27

Removing within-contact k-NN caused the largest drops in mean F@5 and F@10 among the variants in Table R1. Removing inter-contact geometric bias or soft semantic conditioning had smaller effects on these metrics.

eFlesh evaluation and sensor adaptation

We agree that evaluating six objects limits what we can conclude about generalization. Both variants retain Elephant’s object-level design: aggregating features across touches and using the resulting features for category prediction and shape reconstruction. The local graphs operate at different levels. Daimon connects 3D points within a touch, while eFlesh connects contact tokens by sensor position. The eFlesh graph described here is documented in the archived configuration for the Table 3 Local-Graph run. We used sensor-specific input encoders for Daimon’s optical measurements and eFlesh’s magnetic measurements and trained the eFlesh model from scratch. We pair each magnetic observation with a time-aligned tactile sensor pose and use the pose’s XYZ translation as the observation’s position in a common coordinate frame.

For context, TouchSDF reported quantitative real-world reconstruction results on 4 objects with CAD references, and FingerSLAM evaluated 6 unseen physical objects in a visuotactile setting. Each additional object requires tactile recordings and an aligned reference geometry for scoring. The six-object evaluation demonstrates reuse of Elephant’s object-level design across optical and magnetic tactile sensing.

Scope and reproducibility

Appendix A of the revised manuscript documents how we build and run these experiments in details.

Our study focuses on recognizing and reconstructing objects from sparse tactile observations. An important direction for future work is to maintain reconstruction quality under systematic calibration errors and accumulated pose drift during real-world tactile exploration.

Reviewer_QP3mOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work proposes Elephant, a framework for joint 3D object shape reconstruction and semantic category recognition from sparse tactile observations without visual input. The method transforms local touches into a shared reference frame, aggregates and propagates information among touch locations based on their spatial proximity, predicts a class likelihood among a fixed set and finally reconstructs a 512 or 2048 points cloud of the whole shape. The method is tested on a 279-object (43-category) MuJoCo simulation dataset and on two real-world tactile datasets collected with UMI grippers using optical Daimon sensors (52 objects) and magnetic eFlesh sensors (18 objects across 6 categories). The author show how their method can be adapted to all the previous scenarios (2 sensors x 3 datasets). When compared to previous methods they achieve consistent improvements in Chamfer distance, F-score, and classification accuracy over point-completion baselines adapted with a classification head.

Critical strengths
  • Real-world multi-sensor tactile dataset and evaluation: Most prior tactile 3D reconstruction works rely heavily on simulation or evaluate on a single sensor modality. Collecting and releasing calibrated real-world datasets covering both optical surface-patch sensors and magnetic vector sensors is a valuable and timely contribution.

  • No extra input: Unlike prior methods that require auxiliary inputs at inference time Elephant only needs a small budget of tactile grasps to classify the object and reconstruct the full shape.

Critical weaknesses

A. Missing architectural ablations: Sections 3.3 and 3.4 introduce several specific and somewhat ad-hoc design choices such as the intra-contact k-NN layers (Eq. 3), the four-term 3D geometric relation bias r_uv in inter-contact attention (Eq. 4), and the soft category-embedding conditioning (Eq. 5). The paper provides no ablation study isolating which of these components actually drive the gains compared to simpler architectures (e.g. a pure transformer after having touches in a shared frame). B. Details on e-flesh missing: Sections 3.2 and 3.3 are written exclusively around the 3D point-patch representation of the Daimon sensor, making it unclear how Eq. 2 and the intra-contact k-NN graph in Eq. 3 operate on the eFlesh sensor where each touch is just a single 15D magnetic vector.

C. Unclear experimental setting: From the current test it is not clear whether the model has been trained once across experiments, or once per dataset and sensor. Table 1 checks "Sim-to-Real" and Section 4 asks "How well does a simulation-trained model transfer to real Daimon observations?", implying a checkpoint transfer, however all three datasets define separate training splits and class vocabularies. The experimental setting should be clarified.

D. Impact of touch distribution: With only few grasps, both recognition and reconstruction should depend heavily on how touches are distributed over the object (e.g., hitting discriminative geometry like a mug handle vs. redundant grasps on a flat wall), yet the paper does not analyze or discuss sensitivity to touch placement. Moreover, all results are reported solely on the authors' newly collected datasets rather than also evaluated in the benchmark settings of prior works (e.g., TouchSDF or Touch2Shape) for a direct comparison.

Questions for authors
  1. Training protocol across datasets: Were separate models trained from scratch on the training splits of Simulation, Daimon, and eFlesh, or was the simulation-trained backbone transferred/fine-tuned onto the real sensors?

  2. Baseline training: In Fig. 7 (and Tables 2–3), were all competitor baselines trained from scratch on the exact same tactile training data and coordinate normalization with the added classification head?

  3. Architectural ablations and eFlesh encoding: Can you provide ablations for the main components in Sections 3.3–3.4 (intra-contact k-NN, inter-contact geometric attention bias, and soft semantic conditioning), and clarify how the intra-contact k-NN graph in Eq. 3 is applied to single-vector eFlesh observations?

  4. Touch distribution: Have you explored how the spatial distribution of the K touches over the object surface influences recognition and reconstruction quality?

4 comments
AuthorsOpenReview ↗

Response to QP3m (1/2)

Thank you for your patience and for asking which parts of Elephant drive reconstruction, how eFlesh inputs are encoded, and how the evaluations were run. We took additional time to complete our supplementary real-robot experiments after our Daimon tactile sensor failed and we sent it for repair. We report the relevant results below.

W1 / Q3.1 — Architectural ablations

To isolate the contribution of each design choice, Table R1 compares the full model with three single-component ablations, a reconstruction-only variant, and a plain Transformer on the same Real38 test cases.

Table R1. Real38 reconstruction ablations.

Training used simulated and real tactile data, with fixed 10,000-step checkpoints and three matched seeds. We evaluated 273 cases from seven held-out objects, with 91 cases at each of K = 1, 5, and 10. Within each seed, we averaged reconstruction scores equally across the three K values, giving equal weight to shape groups and to objects within each group. Entries report the mean ± sample SD of these aggregated scores across the three training seeds. The held-out objects lack official class labels. We also report Density-aware Chamfer Distance (DCD; Wu et al., 2021) to assess how well the local point densities of the reconstructions match those of the ground-truth point clouds.

VariantDCD ↓CD-L1 (mm) ↓F@5 mm (%) ↑F@10 mm (%) ↑
Full0.9304 ± 0.002112.41 ± 0.2135.35 ± 1.9857.54 ± 0.38
Without inter-contact geometric bias0.9318 ± 0.004012.54 ± 0.1835.10 ± 1.9857.33 ± 1.94
Without soft semantic conditioning0.9332 ± 0.002112.42 ± 0.1434.62 ± 0.8457.23 ± 0.51
Reconstruction only0.9318 ± 0.001012.62 ± 0.2034.99 ± 0.2356.52 ± 0.19
Without within-contact k-NN0.9383 ± 0.002412.86 ± 0.8830.88 ± 3.2252.18 ± 4.84
Plain Transformer0.9336 ± 0.002612.72 ± 0.9634.81 ± 0.5756.92 ± 1.27

Removing within-contact k-NN caused the largest drops in mean F@5 and F@10. Removing geometric bias or soft semantic conditioning produced smaller, metric-dependent changes. The plain Transformer was close to the full model, with mean CD-L1 of 12.72 versus 12.41 mm and F@5 of 34.81% versus 35.35%. These results identify the clearest reconstruction benefit in the local geometry modeled within each touch.

W2 / Q3.2 — Encoding magnetic tactile observations

Both variants retain Elephant’s object-level design: aggregating features across touches and using the resulting features for category prediction and shape reconstruction. The local graphs operate at different levels. Daimon connects 3D points within a touch, while eFlesh connects contact tokens by sensor position. Equations 2–3 describe the Daimon input path. For eFlesh, we form one token for each finger at each sampled time step. We standardize the finger’s 15 magnetic readings using training-set statistics, then encode them with a 15→64→64 MLP. Separate MLPs encode the sensor’s 3D position and 6D rotation representation, and its contact-state features. We fuse the three outputs into a 256-dimensional token.

Each eFlesh magnetic observation is paired with a time-aligned pose of the corresponding tactile sensor. The translation (XYZ) of that pose gives the observation’s position in a common coordinate frame.

The eFlesh graph described here is documented in the archived configuration for the Table 3 Local-Graph run. Its candidate pool consists of valid contact tokens from both fingers at all sampled time steps.

AuthorsOpenReview ↗

Response to QP3m (2/2)

W3 / Q1 — Training across simulation, Daimon, and eFlesh

For Figure 7, we trained Elephant and all baselines from scratch on the same simulated tactile data. For Daimon (Table 2), we pretrained each method on simulated tactile data and then fine-tuned it on the same real tactile training data. The Real38 ablations also used both simulated and real tactile data. The eFlesh model used its own encoder and was trained from scratch without Daimon weights. W2 / Q3.2 describes the object-level design reused across the two sensor variants.

Q2 — Baseline training and inputs

In the simulation experiments (Figure 7), Elephant and all baselines were trained from scratch using the same tactile training data and coordinate normalization. We matched the number of tactile observations and used the same classification head and category supervision. For Daimon (Table 2), we pretrained each method on simulated tactile data and then fine-tuned it on the same real tactile training data using the same coordinate normalization. For recognition (Table 2(b)), all compared methods used the same classification head and category supervision.

For eFlesh (Table 3), all methods used the same training split, pose normalization, classification head, and magnetic tactile input encoder design. We trained all methods from scratch. The encoder takes the standardized 15-dimensional magnetic vector together with contact position, rotation, strength, and finger identity. We evaluated all methods on the same sampled touch subsets for the test instances they shared. Table 3 reports descriptive aggregates for each method’s full evaluation set.

W4.1 / Q4 — Touch placement at a fixed budget

To test whether more widely spaced touches improve reconstruction at a fixed touch budget, we compared three sampling strategies on the same 273 Real38 cases at requested budgets of 1, 5, and 10 touches. Within each repeat, all strategies used the same candidate contacts from the same recording and started with the same contact. Random sampled the remaining contacts without replacement. Spread added the contact whose nearest selected neighbor was farthest away. Clustered chose the remaining contacts closest to the starting contact.

Table R2. Touch placement on Real38. Values are mean ± sample SD over three sampling repeats with a fixed checkpoint. Coverage is the distance from the centroid of the selected contact centers to the farthest selected center, averaged at K = 5 and K = 10. Reconstruction scores are aggregated across K = 1, 5, and 10.

Sampling policyCoverage (mm) ↑CD-L1 (mm) ↓F@5 (%) ↑F@10 (%) ↑
Random55.42 ± 1.1512.33 ± 0.4136.47 ± 0.3958.25 ± 0.19
Spread57.54 ± 0.2112.33 ± 0.4535.64 ± 0.1957.43 ± 0.24
Clustered43.80 ± 1.1412.64 ± 0.5637.28 ± 0.3658.72 ± 0.48

Spread achieved the widest coverage and reduced mean CD-L1 relative to Clustered. It matched Random on mean CD-L1, and Clustered achieved the highest F@5 and F@10. Greater contact spread therefore offered no consistent reconstruction benefit across metrics in this experiment. The sampling rules vary contact spread and which features are touched together. A follow-up could vary these factors independently to test whether either improves recognition or reconstruction.

W4.2 — Comparison with prior benchmarks

We thank the reviewer for suggesting a comparison with TouchSDF and Touch2Shape. Our simulation experiments use registered 3D contact patches, whereas both published pipelines derive local surface observations from tactile images. TouchSDF’s reported ShapeNet evaluation covers six object categories. We ran a preliminary TouchSDF trial on our data and observed limited reconstruction quality in the tested setup. A systematic comparison would require validating the input adaptation and evaluating it across our test set under the same protocol as the other baselines, so we kept the pilot separate from the main benchmark.

Touch2Shape also learns a policy for selecting the next contact. Our evaluation measures reconstruction and recognition from given contacts at a fixed touch budget. In future work, we would like to train a touch-selection policy for Elephant and compare it with Touch2Shape on a real robot under a shared touch budget.

Reviewer_QP3mOpenReview ↗

Post Rebuttal Comment

Thanks for the extra experiments and for clearing up how eFlesh and the training splits were handled. The extra robot eval and the touch placement experiment are both interesting additions that should definitely go into the final version. At the same time, the ablations show that a plain transformer gets pretty close to the full model, so the extra architectural pieces don't add as much as the paper makes it sound. This shoudl also be reflected on the final version of the paper and if needed the proposed solution can be simplified. Overall I still think this work provides enough contributions to be interesting to people working in this field, so I'll keep my rating at 3.

Thank you for your prompt follow-up. We have included the additional robot evaluation in the revised manuscript. Your feedback has helped us improve the paper, and we appreciate your time and suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.