- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Elephant, which is a framework that jointly reconstructs 3D object geometry and recognizes semantic categories from a small number of localized tactile observations. Contact geometry is combined with spatial attention, and the method then conditions reconstruction on predicted category probabilities. The authors also collected datasets using optical Daimon and magnetic eFlesh sensors and evaluated the method with different numbers of tactile observations, in simulation and on real objects.
Critical strengths
-
I find particularly interesting the joint formulation of recognition and reconstruction from few tactile contacts. The hierarchical encoding of local patches and relations between contacts is pretty elegant, and soft class conditioning provides a natural way to couple semantic predictions with geometric completion.
-
The real tactile datasets and experiments with optical and magnetic sensors are likely one of the most valuable practical contributions. Supporting both sensing technologies makes the framework interesting beyond a single hardware setup. I believe the planned release of data and calibration information would also benefit the community.
-
Holding out physical object instances, rather than only individual recordings, is a good choice for assessing generalization. The evaluation across touch budgets also provides useful insight into how additional observations affect recognition and reconstruction.
Critical weaknesses
-
The supplementary material mentioned in the reproducibility statement was not provided as a file or in the pdf, so I couldn’t fully check calibration, training, sampling, and baseline configurations.
-
The separate contributions of geometric attention, semantic conditioning, and joint training are hard to assess on their own. The eFlesh baseline adaptation is also unclear, since magnetic vectors differ from the partial point clouds expected by the comparison methods. Although TouchSDF and Touch2Shape are discussed in the introduction and Table 1, they are not included as experimental baselines; comparing against them would strengthen the evaluation, especially given that the gains on Daimon are limited.
-
While the 94.44% eFlesh accuracy is encouraging, it comes from 18 sampled inputs across only six held out objects, with three subsets from one recording per object; this and the absence of uncertainty estimates limit the strength of the generalization claim.
-
I would like more clarification on the scope of the blind perception and transfer claims, especially as the known object frame depends on offline visual and CAD alignment, without an evaluation of alignment sensitivity. It is also unclear how the Daimon model is trained and whether learned weights transfer to eFlesh.
-
Simba framework in section 4.1 and Figure 7 is missing a citation, so would be good to add it in the final version.
Questions for authors
-
Could you add ablations showing how much geometric attention, semantic conditioning, and joint training each contribute to reconstruction and recognition?
-
What happens when the predicted category is wrong? Does semantic conditioning still help reconstruction, or does it push the shape toward the wrong category?
-
How are the magnetic measurements given to the eFlesh baselines, and do all methods get the same information? Could you also clarify which models use simulated or real training data, and whether any weights are transferred between sensors?
-
Are the eFlesh gains consistent across the six held out objects and different sampled inputs?
5 comments
Response to WfdP (2/3)
W2.2–W2.3 / Q3.1 — eFlesh baseline inputs and comparability
Yes. All methods received the same standardized 15-dimensional magnetic measurements, contact position and rotation, contact strength, and finger identity. We pair each magnetic observation with a time-aligned tactile sensor pose and use the pose’s XYZ translation as the observation’s position in a common coordinate frame. We added a magnetic tactile input encoder of the same design to each model.
Both variants retain Elephant’s object-level design: aggregating features across touches and using the resulting features for category prediction and shape reconstruction. The local graphs operate at different levels. Daimon connects 3D points within a touch, while eFlesh connects contact tokens by sensor position. The eFlesh graph described here is documented in the archived configuration for the Table 3 Local-Graph run.
Table 3 reports aggregate results for each method. Table R6 below compares the reported means by object.
Elephant’s Daimon pathway takes registered 3D contact patches as input, whereas the TouchSDF and Touch2Shape pipelines start from tactile images. We did not run a matched comparison because adapting the pipelines would require aligning the contact and pose information available to each method. A future comparison could evaluate the methods on the same recorded touches and test objects with a common touch budget and reconstruction metric.
W3 / Q4 — Variability across eFlesh objects and inputs
Table R6 shows how Elephant’s results vary across objects at K = 20 and how they compare descriptively with eight baselines across the six objects in the table.
Table R6. Per-object eFlesh results at K = 20. The first two score columns report Elephant’s reconstruction scores. The last three columns summarize the object-level comparisons between Elephant and the eight baselines. Accuracy wins, ties, and losses count how many baselines Elephant scored above, equal to, or below. The other two comparison columns count how many baselines Elephant outperformed on CD-L1 and F@5. The final row reports Elephant’s scores across six objects and the total across 48 object–baseline comparisons.
| Instance | Elephant CD-L1 (mm) ↓ | Elephant F@5 (%) ↑ | Accuracy wins / ties / losses | Baselines with higher CD-L1 | Baselines with lower F@5 |
|---|---|---|---|---|---|
| camera2 | 7.03 | 45.82 | 0/8/0 | 6 / 8 | 8 / 8 |
| cup2 | 9.67 | 38.75 | 1/7/0 | 5 / 8 | 6 / 8 |
| microphone2 | 5.75 | 54.36 | 7/1/0 | 8 / 8 | 8 / 8 |
| mouse2 | 7.28 | 47.59 | 5/0/3 | 6 / 8 | 8 / 8 |
| pineapple2 | 3.59 | 76.43 | 7/1/0 | 8 / 8 | 8 / 8 |
| shoe1 | 4.02 | 74.28 | 2/6/0 | 7 / 8 | 7 / 8 |
| Across-object summary | 6.22 ± 2.27 | 56.21 ± 15.66 | 22/23/3 | 40 / 48 | 45 / 48 |
Elephant had lower CD-L1 in 40 of 48 comparisons and higher F@5 in 45 of 48. For recognition, Elephant scored higher in 22 comparisons, tied in 23, and scored lower in 3. Across all six evaluated objects, Elephant’s accuracy was 94.44 ± 13.61%. Within an object, the sample SD across the three subsets ranged from 0.20 to 1.35 mm for CD-L1 and from 0.67 to 7.56 percentage points for F@5. The comparisons use object-level means.
Response to WfdP (3/3)
W4 / Q3.2 — Pose assumptions and transfer
In the original robot experiments, we used RGB-D/CAD alignment offline to establish the object frame and tracked sensor poses to register the tactile observations. The prediction network receives the registered tactile observations without RGB or depth images. In a supplementary xArm7 experiment, we used measured joint states and forward kinematics to obtain sensor poses, expressed contacts in the robot base frame, and centered inputs using the tactile observations. Aligned CAD models served as scoring references.
Table R3. Exploratory reconstruction results with proprioceptive localization. All methods used the same registered tactile inputs and CAD reference for each case. We averaged scores across recordings within each object, then weighted the cylinder, camera, mouse, and dragon fruit equally. The evaluation covers 10 recordings at K = 1, 3, and 5 and nine at K = 10. We used 1,024 predicted points and 2,048 reference points for scoring.
| Method | CD-L1 at K = 1 (mm) ↓ | K = 3 ↓ | K = 5 ↓ | K = 10 ↓ | F@5 at K = 10 (%) ↑ | Balanced accuracy at K = 10 (%) ↑ |
|---|---|---|---|---|---|---|
| Local-Graph | 10.90 | 8.51 | 7.94 | 7.08 | 45.02 | 50.0 |
| PCN | 13.44 | 10.53 | 9.81 | 8.89 | 38.57 | 0.0 |
| SnowflakeNet | 11.29 | 9.09 | 8.70 | 7.80 | 43.73 | 25.0 |
| AdaPoinTr | 11.70 | 9.61 | 8.80 | 7.34 | 42.51 | 25.0 |
| ProtoComp | 25.95 | 21.04 | 24.53 | 17.15 | 19.08 | 25.0 |
| SymmCompletion | 14.23 | 10.27 | 9.45 | 8.48 | 39.22 | 25.0 |
| ESCAPE | 16.34 | 13.63 | 11.27 | 9.78 | 30.20 | 50.0 |
| PointSea | 14.21 | 10.84 | 10.89 | 9.16 | 34.93 | 0.0 |
| Hyper-PCN | 12.74 | 9.40 | 8.51 | 7.81 | 43.10 | 50.0 |
The results in Table R3 show that robot proprioception can supply the poses needed to register tactile contacts for reconstruction in this setup.
With Elephant’s weights fixed, combined perturbations of approximately 10 mm translation and 10° rotation RMS increased mean CD-L1 from 12.50 to 13.28 mm and reduced F@5 by 2.1 percentage points. This measures sensitivity to local registration errors. Systematic calibration offsets and accumulated drift remain untested.
For Figure 7, we trained Elephant and all baselines from scratch on simulated tactile data. For Daimon (Table 2), we pretrained each method on simulated tactile data and then fine-tuned it on the same real tactile training data. The Real38 ablations also used both simulated and real tactile data. The eFlesh model was trained from scratch with a magnetic encoder and no Daimon weights. Q3.1 describes the object-level design reused across the two sensor variants.
W5 — Simba citation
Thank you for pointing out the missing citation. The baseline in Section 4.1 and Figure 7 is Simba, Zhang et al., AAAI 2026. We have added it into our revised version of essay. Thanks again for pointing out this mistake.
Response to WfdP (1/3)
Thank you for your patience and for asking how Elephant’s components affect its predictions and how the real-robot and eFlesh results should be interpreted. We took additional time to complete our supplementary real-robot experiments after our Daimon tactile sensor failed and we sent it for repair. We report the relevant results below.
W1 — Reproducibility
Appendix A of the revised manuscript documents the original Daimon registration chain, the supplementary xArm7 acquisition and scoring settings, and the Real38 and eFlesh evaluation protocols. Figures 3 and 4 show the original registration chain, robot setup, tactile measurements, and corresponding 3D contacts.
W2.1 / Q1 — Component ablations and joint training
To quantify the contribution of the geometric and semantic components, Table R1 compares the full model with three single-component ablations, a reconstruction-only variant, and a plain Transformer.
Table R1. Real38 reconstruction ablations.
Training used simulated and real tactile data, with fixed 10,000-step checkpoints and three matched seeds. We evaluated 273 cases from seven held-out objects, with 91 cases at each of K = 1, 5, and 10. Within each seed, we averaged reconstruction scores equally across the three K values, giving equal weight to shape groups and to objects within each group. Entries report the mean ± sample SD of these aggregated scores across the three training seeds. The held-out objects lack official class labels. We also report Density-aware Chamfer Distance (DCD; Wu et al., 2021) to assess how well the local point densities of the reconstructions match those of the ground-truth point clouds.
In Table R1, removing within-contact k-NN caused the largest drops in mean F@5 and F@10. Removing geometric bias or soft semantic conditioning had smaller effects on these metrics. Table R1 measures the contribution of Daimon’s within-touch point graph. Q3.1 describes the eFlesh graph over contact tokens. In Table R1, the full model achieved better mean scores than the reconstruction-only variant on all four reconstruction metrics.
Q2 — Effect of an incorrect predicted category
To test whether a wrong category can affect reconstruction, we kept the checkpoint and tactile inputs fixed and changed only the category signal given to the decoder.
Table R5. Category conditioning on labeled Real38 validation cases. The same 117 cases come from three physical computer-mouse instances. Scores give equal weight to each instance and to K = 1, 5, and 10.
On these three mouse instances, using the predicted soft category probabilities improved all three mean reconstruction metrics relative to disabling conditioning. CD-L1 decreased by 1.01 mm, and F@5 and F@10 increased by 1.64 and 1.11 percentage points. Providing a wrong category instead of the ground-truth category increased CD-L1 by 1.05 mm and reduced F@5 and F@10 by 1.31 and 1.71 percentage points. The scores with a forced wrong category were close to those with conditioning disabled. These results show that category conditioning contributes to reconstruction on this subset. Whether natural misclassifications shift reconstructed shapes toward an incorrect category remains open.