acceptodds
Under review as a conference paper at ICLR 2027

Vision Meets WiFi: Physics-Grounded Estimation of Volumetric Mechanical Properties

Abstract

Estimating volumetric mechanical properties, including Young's modulus, Poisson's ratio, and density at each voxel, is intrinsically ambiguous from vision alone, as visually similar objects may have substantially different material compositions and physical behavior. Existing approaches predict these properties independently across voxels, overlooking the piecewise-constant material structure of real objects and producing noisy or inconsistent estimates for voxels that share the same material, while lacking an explicit mechanism to resolve visual ambiguity. We introduce ViWi (Vision Meets WiFi), an object-centric framework for volumetric mechanical-property estimation. ViWi represents each object using a compact set of material slots that aggregate evidence from voxels with a shared material identity and produce coherent slot-level property predictions. To complement visual appearance, ViWi incorporates a compact RF descriptor generated through WiFi-band electromagnetic simulation using permittivity and conductivity. The RF descriptor conditions the material slots with global composition cues that may be unavailable from images, while visual features preserve voxel-level spatial localization. On GVM, ViWi improves over the prior state of the art on four of six per-voxel metrics, while its vision-only variant improves all reported mass-estimation metrics on ABO-500. These results demonstrate that combining object-centric material structure with complementary RF evidence enables more accurate and physically coherent volumetric property estimation beyond what is possible from visual appearance alone.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Object-Centric Learning, Mechanical Property Estimation, Multimodal Sensing, Physical Property Prediction, Vision-RF Fusion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_SdmkOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper presents ViWi, which is a multimodal framework that combines vision and RF signals for voxel-level estimation of 3D object mechanical properties, including Young's modulus, Poisson's ratio, and density. ViWi replaces independent voxel-wise regression with a material-slot representation that groups voxels based on visual and material-latent compatibility and predicts shared slot-level properties. The framework incorporates RF signals in the 2.4 GHz Wi-Fi band, summarized as an RF descriptor, to condition the voxel-level property estimation. Experiments on GVM demonstrate overall improvements over existing methods. The paper also provides ablation studies on the material-slot formulation and RF design, as well as downstream physics simulations using the predicted material fields.

Critical strengths

[+] The paper explores an interesting research direction. Estimating the mechanical properties of 3D objects is a challenging problem, particularly when objects of different materials with similar visual appearances. The idea of incorporating RF signals to complement visual observations is interesting and provides a new perspective on addressing this challenge.

Critical weaknesses

[-] The sensing hardware setup is unrealistic. Despite the name ViWi (Vision Meets WiFi), the proposed system relies on a sensing configuration that is substantially more complex than commodity WiFi devices, involving 16 transmitters and 32 receivers distributed on concentric spheres around the target object. I understand that the work focuses on RF sensing in the WiFi frequency band rather than necessarily using commodity WiFi devices. However, I am concerned that such a complex sensing setup would be difficult and costly to deploy in practice. Moreover, the motivation for employing such complex setup for estimating a few mechanical properties of an object is not sufficiently justified. It is unclear whether this approach offers practical advantages over existing professional material characterization or measurement instruments.

[-] The RF sensing process relies on unrealistic assumptions. The work uses ray tracing in Sionna RT to simulate RF propagation at 2.4 GHz. However, several aspects of the simulation rely on idealized assumptions. First, the simulation explicitly disables the LoS propagation paths, which does not reflect typical real-world wireless sensing conditions. Second, the proposed RF descriptors rely on path-level information, including path coefficients, propagation delays, and single-path gains, which can be directly obtained from the ray-tracing simulator but are difficult to extract from real-world wireless measurements such as WiFi CSI, RSSI, or CIR. In practice, signals from different propagation paths are superimposed at the receiver, making it challenging to isolate individual paths and obtain their properties accurately. Consequently, I am concerned that the proposed framework may face substantial challenges when deployed in real-world environments, potentially limiting its practical applicability and impact.

Questions for authors
  1. What is the impact of enabling or disabling the LoS paths? The RF simulation explicitly disables the Los paths to isolate object-dependent scattering. However, direct paths are present in practical wireless sensing scenarios. In particular, how would including the direct Tx–Rx propagation paths affect the RF descriptors and the resulting mechanical-property estimation accuracy?

  2. The proposed RF descriptors rely on path-level signal information. While these quantities can be obtained from the Sionna RT simulator, they may be difficult to estimate from real-world wireless sensing setups. Could the authors clarify how these features would be extracted from practical wireless signals? Additionally, what signal bandwidth and measurement resolution are assumed in the simulation?

  3. The simulation uses 16 Tx and 32 Rx distributed around the target object. Could the authors further justify the necessity of such a complex sensing configuration for estimating object mechanical properties? What practical advantages would this approach offer compared with existing material characterization techniques, and to what extent could the sensing configuration be simplified without substantially sacrificing estimation accuracy?

7 comments

We thank the reviewer for the expert reading from the wireless-sensing perspective. The three questions point exactly to the gap between simulation and practice. To answer them with evidence, we re-simulated the RF descriptors of all 1,560 objects under five new sensing settings and retrained the model for each. In short:

  • LoS (Q1). With the direct path present and standard empty-scene subtraction on realistic CSI, the error stays at the LoS-free level (5.95% vs. 6.02%).
  • Practical signals (Q2). Realistic 40 MHz CSI with noise costs no accuracy (5.98%), and even 1 dB RSSI works (5.93%).
  • Simpler configuration (Q3). Even RSSI only, with the direct path present and only 2 Tx + 4 Rx, clearly beats vision only (5.62% vs. 6.34%).

Protocol. Due to limited time and compute, each new model was trained on 1 GPU (batch size 16) instead of 8 (batch size 128) for the same 112k steps, i.e., with 8× fewer samples, so errors are slightly higher than in Table 1 (6.02% vs. 5.81% for the identical RF input). All new models start from the same vision-only checkpoint ViWi†, use the same hyperparameters and the same 148 test objects, so their rows are directly comparable. The italic first row of each table repeats ViWi from Table 1 (8 GPUs); the mean is the average of ALRE(E), ARE(ν) and ARE(ρ); best 1-GPU values are in bold.


Q1. Impact of enabling the line-of-sight path

Settings. We compare three versions of the RF input (Table R1):

  • (A) LoS off, ideal (paper setting): the simulator removes the direct Tx–Rx path, so only signals that touched the object remain. This is possible only in simulation.
  • (B) LoS on, ideal: the direct path is included; measurements are noise-free.
  • (C) LoS on, realistic: what a real device would do: 40 MHz CSI with thermal noise and gain errors, measured once with and once without the object and subtracted (H(obj+LoS) − H(empty)), the standard background removal in WiFi sensing.

Table R1. Effect of the line-of-sight path (16 Tx + 32 Rx; GVM test set, 148 objects, step 112k).

RF inputE ALDE↓E ALRE↓ν ADE↓ν ARE↓ρ ADE↓ρ ARE↓Mean↓
ViWi, Table 1 - main paper (8 GPUs; LoS off, ideal paths)0.17970.01960.01510.0515198.220.10345.81%
(A) LoS off, ideal paths (paper setting)0.1943 (±0.20)0.0210 (±0.025)0.0148 (±0.014)0.0508 (±0.051)200.11 (±203)0.1087 (±0.096)6.02%
(B) LoS on, ideal paths0.1962 (±0.23)0.0214 (±0.028)0.0141 (±0.014)0.0490 (±0.051)184.91 (±194)0.1036 (±0.093)5.80%
(C) LoS on, 40 MHz CSI + noise + subtraction0.1914 (±0.22)0.0205 (±0.026)0.0153 (±0.016)0.0526 (±0.056)189.32 (±236)0.1055 (±0.111)5.95%
Vision only (no RF)0.1985 (±0.22)0.0216 (±0.026)0.0158 (±0.014)0.0549 (±0.049)196.33 (±256)0.1139 (±0.112)6.34%

Why LoS does not hurt. The direct path does not touch the object, so it carries no material information, but it does not remove any either: the signal scattered by the object is simply added on top of it. Since the direct path is almost identical for all objects, it acts as a constant offset, while the object-dependent part remains. LoS even adds information: the receivers that are no longer reached show which direct links the object blocks, i.e., its size and extent. This plausibly explains why (B) gives the best Poisson's-ratio and density errors and the lowest mean error (5.80% vs. 6.02% for A).

Why (C) is slightly worse than (B). In a real measurement, the direct path is about 35 dB stronger than the signal from the object, so subtraction must cancel a very strong signal using two noisy measurements. A gain error of only 1% in amplitude and 1° in phase already leaves a residue of the direct path that is about as strong as the object's signal, and thermal noise and the 40 MHz bandwidth add further errors. (B) is free of all these errors, i.e., the ideal case. Even so, (C) stays at the level of the paper setting (5.95% vs. 6.02%) and below vision only (6.34%), with a lower error on 96 of the 148 objects. The reason is that each feature combines many subcarriers and receivers: the residue is random and averages out, while the effect of the object is consistent across all measurements.

Conclusion. Turning LoS off in the paper is not a requirement of the method. It isolates the object's signal in simulation, which empty-scene subtraction achieves in practice without loss of accuracy.

Q2. Extracting the descriptor from practical signals; bandwidth and resolution

Assumption in the paper. The descriptor in the paper uses path-level quantities from Sionna RT, i.e., an ideal measurement with unlimited bandwidth and no noise. We will state this explicitly in the revision.

Features from real signals. We now compute the descriptor only from what commodity WiFi hardware reports:

  • CSI (e.g., Intel 5300 CSI Tool, Atheros CSI Tool, Nexmon): 2.4 GHz, 40 MHz channel (the widest in this band), 114 subcarriers, with thermal noise (20 dBm transmit power, 7 dB noise figure) and a random gain error per measurement (1% amplitude, 1° phase). The four features per transmitter become: total received power; fraction of receivers above 10× the noise floor; strongest tap of the impulse response (IFFT, 25 ns taps); and RMS delay spread.
  • RSSI only (available on every WiFi device): received power per link, computed from the same noisy CSI, rounded to 1 dB and measured with and without the object; per transmitter we keep the total power change and the fraction of receivers that change by at least 1 dB.

Bandwidth and resolution. At 40 MHz the delay resolution is 25 ns (7.5 m of path length), while the delay spread caused by an object (scaled to 1 m) is only 0.35 ns on average. Fine timing is therefore not usable in practice, and the model must rely on power-based features.

Table R2. Descriptor from realistic signals (16 Tx + 32 Rx; GVM test set, 148 objects, step 112k).

RF inputE ALDE↓E ALRE↓ν ADE↓ν ARE↓ρ ADE↓ρ ARE↓Mean↓
ViWi, Table 1 - main paper (8 GPUs; LoS off, ideal paths)0.17970.01960.01510.0515198.220.10345.81%
(A) LoS off, ideal paths (paper setting)0.1943 (±0.20)0.0210 (±0.025)0.0148 (±0.014)0.0508 (±0.051)200.11 (±203)0.1087 (±0.096)6.02%
(D) LoS off, 40 MHz CSI + noise0.2009 (±0.23)0.0218 (±0.028)0.0145 (±0.014)0.0498 (±0.049)195.50 (±212)0.1077 (±0.104)5.98%
(C) LoS on, 40 MHz CSI + noise + subtraction0.1914 (±0.22)0.0205 (±0.026)0.0153 (±0.016)0.0526 (±0.056)189.32 (±236)0.1055 (±0.111)5.95%
(E) LoS on, RSSI (1 dB) + noise + subtraction0.1942 (±0.22)0.0210 (±0.026)0.0157 (±0.014)0.0540 (±0.050)190.59 (±227)0.1029 (±0.098)5.93%

Results. All three realistic settings include thermal noise and gain errors. Realistic CSI with noise costs no accuracy (D: 5.98% vs. A: 6.02%), the full practical pipeline with LoS, noise and subtraction gives 5.95% (C), and even noisy 1 dB RSSI gives 5.93% (E); all are better than vision only (6.34%, Table R1). The reason is that the material information lies in how much the object reflects, absorbs and blocks the signal on each link, i.e., in the received power that every WiFi device measures, and not in fine path details that only a simulator provides. We will add this measurement model and Table R2 to the appendix.

Q3. Complexity of the sensing configuration and practical advantages

Why 16 Tx and 32 Rx. The sphere was chosen as a controlled, orientation-independent setup to test whether RF adds information to vision, similar to dense multi-view rendering on the vision side. It is not a deployment requirement, and the links need not be measured at the same time: a single Tx–Rx pair could collect them one after another, e.g., by moving one antenna while the object rotates on a turntable.

How far it can be simplified. We tested the most practical setup (G): only 2 transmitters and 4 receivers (8× fewer of each), placed around the object with the placement rule of the paper, RSSI only (1 dB), with the direct path present and the empty scene subtracted, i.e., 4 numbers instead of 64.

Table R3. Simplified sensing configuration (GVM test set, 148 objects, step 112k).

ConfigurationE ALDE↓E ALRE↓ν ADE↓ν ARE↓ρ ADE↓ρ ARE↓Mean↓
ViWi, Table 1 - main paper (8 GPUs; 16 Tx + 32 Rx, LoS off, ideal paths)0.17970.01960.01510.0515198.220.10345.81%
(A) 16 Tx + 32 Rx, LoS off, ideal paths (paper setting)0.1943 (±0.20)0.0210 (±0.025)0.0148 (±0.014)0.0508 (±0.051)200.11 (±203)0.1087 (±0.096)6.02%
(G) 2 Tx + 4 Rx, LoS on, RSSI (1 dB) + noise + subtraction0.2088 (±0.24)0.0223 (±0.027)0.0137 (±0.014)0.0456 (±0.046)193.86 (±246)0.1008 (±0.106)5.62%
Vision only (no RF)0.1985 (±0.22)0.0216 (±0.026)0.0158 (±0.014)0.0549 (±0.049)196.33 (±256)0.1139 (±0.112)6.34%

The minimal setup (G) reaches 5.62%, clearly better than vision only (6.34%; lower error on 92 of 148 objects), with the best Poisson's-ratio and density errors, while Young's modulus is slightly worse. (G) is even below the full setup (A). We do not claim that fewer antennas are better; a likely reason is that (G) keeps the direct path, which (A) removes: as shown in Q1, the links blocked by the object add information (B: 5.80% vs. A: 6.02%). The main point is that the simple setup loses no accuracy, because the material information lies in how much the object changes the received power on each link, and a few links around the object already capture it.

Practical advantages. Existing RF material characterization (open-ended coaxial probes, waveguides, resonant cavities, free-space methods) measures εr and σ with laboratory instruments such as a vector network analyzer, and requires either a sample of known shape (e.g., cut to fit a waveguide or cavity, or a flat slab for free-space methods) or direct contact with the material (coaxial probes); it gives no mechanical properties. Mechanical tests (tensile tests, indentation, ultrasound) usually need contact, prepared specimens or specialized laboratory equipment. Our approach (i) needs no contact and no sample preparation; (ii) uses signals that commodity WiFi devices already provide, with RSSI alone being sufficient in our experiments; and (iii) estimates full volumetric fields of E, ν and ρ for the whole object, using RF as a compact cue that complements vision. It is not meant to replace precise laboratory measurements, but to give fast, whole-object estimates where such measurements are impractical.

Reviewer_DQ7COpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The proposed method combines vision for voxel localization with wifi material identification (soft slot assignment based on simulated material library) for 3D predictions of Young's modulus, Poisson's ratio, and density. Subcomponents of the method are explained in terms of their contribution, verified with ablation studies. Comparison to baselines is limited but successful.

Critical strengths

The approach is reasonable and fairly well-explained. The empirical results show metric improvements over baseline and include some ablations. Lower performance on density is explained well in the paragraph starting on line 364.

Critical weaknesses

Few baselines are considered, and the value of the improvement relative to the difficulty of collecting another sensing modality is arguable. The supplement shows many failures of material assignment that are glossed over in the main paper.

The figures are simply not good enough in their current form. Predictions are spatially uniform (suggesting that vision is hardly necessary, figs 1 and 6) and without meaningful context from ground truth (fig 1). Text is tiny throughout. Order and size of figures does not reflect their order in the paper or their relative importance. Multi-material examples, which are surely the whole point of the method, are left out of the main paper entirely except for the legs of the couch in figs 1 and 6. While additional multi-material examples appear in the supplement, they show significant errors in per-voxel assignment that raise questions the paper chooses not to answer.

Questions for authors

I was extremely confused by the design choices around embedding dimensions, which were not supported with reported ablations. For the material latent space, why project to 2 dimensions just to come back up to 3 physical parameters? Why use 64 dimensions for RF if they are mapped to 2 underlying variables (permittivity and conductivity)? How do these choices interact with each other?

Please comment on the multi-material failure cases shown in supplemental figures 7 & 8.

The claim on line 401 that reported errors from VoMP cannot be reproduced is significant. Is the identified gap in volume estimation? If so, would you rate the VoMP volume estimation method as better or worse than that used in this paper?

Can existing methods on material classification (sec 2.2) be used as baselines or swapped-out components for an interesting ablation?

Small points, no answers needed:

  • line 347: VoMP is no longer concurrent
  • Fig 5 takes up a relatively large amount of space to make a fairly minor point (unless there's something nontrivial that I'm missing), I would suggest swapping it with additional result examples.
7 comments

We thank the reviewer for the thoughtful review and the precise questions. They led us to clarify the role of each representation, to analyze the failure cases more explicitly, and to run three new experiments: a volume-estimation study on ABO-500, an oracle material-classification baseline, and an oracle material-composition input to ViWi. We answer each question below.


Q1. Embedding dimensions and how they interact

2-D material latent. The two-dimensional latent space is not a new design choice of ViWi: it is the pretrained MatVAE of VoMP, which we keep frozen so that ViWi and VoMP share exactly the same decoder and differ only in how the latent is predicted. The motivation is physical. Young's modulus, Poisson's ratio and density of real materials are strongly correlated (e.g., metals are both stiff and dense, foams and fabrics are both compliant and light), so valid (E,ν,ρ)(E,\nu,\rho) triplets occupy a low-dimensional manifold of the three-dimensional property space. The latent space parameterizes this manifold, and decoding from it always yields a physically valid triplet, whereas a free three-dimensional regression head can produce implausible combinations.

64-D RF descriptor. The RF descriptor is not a function of only two variables. An object generally contains several materials with different (εr,σ)(\varepsilon_r,\sigma), and the received signal also depends on the object's geometry, size, the spatial arrangement of its materials, and the direction from which it is observed. The 64 dimensions are therefore an angular sampling of the scattering response (four statistics for each of 16 transmitter positions), not an embedding-size choice. Fewer viewpoints carry less information: in Ablation VI, reducing the descriptor from 64 to 16 dimensions (16→416\to4 transmitters) increases the mean error from 5.87% to 6.45%.

Interaction. The two spaces are kept separate: the RF descriptor is never compressed into the 2-D latent. An MLP first encodes it into an RF embedding, which is used in two ways: (i) through FiLM, it adjusts the slot features and the slots' initial 2-D material latents, which guides which voxels are assigned to which slot; and (ii) it adds a small, bounded correction (at most c=0.3c=0.3 per component) to each slot's final material latent before decoding. RF therefore acts as global evidence that shifts each slot's material estimate within the material space, while the frozen MatVAE keeps every prediction physically valid. We will explain these choices in Section 3.


Q2. Multi-material failure cases (Figures 7 and 8)

We agree that these figures also show the limits of the method. We identify three failure modes:

  • Part boundaries. Voxel-to-slot assignments are soft, so voxels at the interface of two materials receive mixed properties. This is most visible for thin parts at the 64364^3 resolution, e.g., the spokes and tires of the tricycle in Figure 8 (slot IoU 0.69 for metal and 0.54 for rubber).
  • Spurious slots. In the tricycle, 7.1% of the voxels are assigned to a slot that matches no ground-truth material. These voxels are decoded to a physically valid but incorrect material.
  • Small components. Parts with few voxels provide weak evidence for their slot and are harder to separate; the armchair feet (1.6% of the voxels) are recovered (IoU 0.85), but this is not guaranteed for every small part.

In all cases, the errors affect only a few voxels near part boundaries and in small parts, while the main materials are recovered correctly. Every error is still a physically valid material, because each slot is decoded through MatVAE. A higher voxel resolution could reduce these errors. We will add this analysis to the appendix.

Q3. VoMP reproduction gap on ABO-500 (line 401)

The gap does not come from volume estimation. Mass is computed as m=cρˉVm=c\bar\rho V, but VoMP does not say how the volume VV is obtained. We therefore evaluated VoMP's released EMA weights and ViWi$^\dagger$ (also EMA) with four volume estimators on the same real-scale ABO models (exact mesh volume is not possible, since none of the 100 test models is watertight). Each estimator is used in two ways: nominal scale, i.e., the volume as computed (c=0.6c=0.6 for the thin shell, as in our protocol, and c=1c=1 otherwise), and best global scale, i.e., one constant cc for all objects, chosen to give the lowest ALDE on the test set itself. The second setting favors each model as much as possible.

Table D1. ABO-500 mass estimation under different volume estimators (100 test objects); both models use EMA weights. Better value per estimator and scale in bold.

(a) Nominal scale

Volume estimatorModelALDE↓ADE [kg]↓ARE↓MnRE↑
NeRF2Physics thin shell (c = 0.6)VoMP1.18731.972.9960.390
ViWi†1.11320.162.5370.414
Solid voxels (64³, c = 1)VoMP2.286370.4425.9360.206
ViWi†2.218296.5422.4730.201
Convex hull (c = 1)VoMP3.1803469.52237.6220.133
ViWi†3.0933211.17219.9740.130
Bounding box (c = 1)VoMP3.4854513.37347.9890.117
ViWi†3.3984174.51320.7540.112

(b) Best global scale c

Volume estimatorModelALDE↓ADE [kg]↓ARE↓MnRE↑
NeRF2Physics thin shellVoMP1.15620.441.8890.410
ViWi†1.09614.191.7910.427
Solid voxels (64³)VoMP1.16632.782.5840.422
ViWi†1.11724.242.2040.434
Convex hullVoMP1.424161.1311.1430.402
ViWi†1.354165.6311.4790.427
Bounding boxVoMP1.539163.5212.7350.374
ViWi†1.454159.5412.3940.401

Even with the best scale, VoMP's ALDE stays above 1.15 for every estimator, far from the reported 0.631 (and ADE ≥ 20.4 kg vs. the reported 8.43 kg). No volume choice therefore recovers the published numbers from the released weights; the gap must come from evaluation details that are not described in the VoMP paper.

Which volume method is best. VoMP's own method is not described, so we cannot rate it. Among the tested estimators, the NeRF2Physics thin shell, the published protocol of the benchmark that we adopt, is clearly the most accurate. ViWi$^\dagger$ is better than VoMP on all metrics with the thin shell, and on ALDE for every estimator and scale. The only exceptions are small: MnRE at nominal scale with the solid, hull and box volumes (differences ≤ 0.005), and ADE and ARE with the convex hull after scale fitting, where both models are far from usable (ARE > 10). The conclusion of Table 2 therefore does not depend on the volume estimator. We will add Table D1 to the appendix.

Q4. Material classification as a baseline or component

As a baseline. Table 1 already includes such pipelines: NeRF2Physics and PUGS classify materials with vision–language models and then look up property values. To test the best possible case, we used an oracle: every test voxel receives its true category (the 12 categories used for RF) and the median (E, ν, ρ) of that category over the training voxels.

Table D2. Oracle material classification vs. continuous prediction (GVM test set).

MethodE ALDE↓E ALRE↓ν ADE↓ν ARE↓ρ ADE↓ρ ARE↓Mean↓
Oracle classification + lookup0.25200.02780.01640.0552354.390.21679.99%
VoMP (Table 1)0.39520.04270.02450.0842153.060.08987.22%
ViWi (Table 1)0.17970.01960.01510.0515198.220.10345.81%

Even a perfect classifier reaches only 9.99%, worse than VoMP and ViWi, because properties vary strongly within a category (density ARE 0.217 vs. 0.103). Category labels are too coarse, which is why ViWi predicts continuous properties.

As a component. We replaced the RF descriptor by the object's true material composition (12-D: fraction of voxels in each category), i.e., a perfect object-level classifier used as input to ViWi. Due to limited compute, these models were trained on 1 GPU instead of 8 (same 112k steps), so errors are slightly higher than in Table 1 (6.02% vs. 5.81% for the same RF input), but the rows are directly comparable.

Table D3. Material composition as an input to ViWi (GVM test set, step 112k, single-GPU training).

InputE ALDE↓E ALRE↓ν ADE↓ν ARE↓ρ ADE↓ρ ARE↓Mean↓
None (vision only)0.19850.02160.01580.0549196.330.11396.34%
RF descriptor (paper setting)0.19430.02100.01480.0508200.110.10876.02%
Oracle material composition0.16600.01800.01370.0468170.930.09215.23%

As an input, perfect composition helps clearly (5.23%, all metrics improve; lower error than vision only on 100 and than RF on 103 of the 148 objects). Category information is therefore useful as evidence for a continuous prediction, not as the final answer, which is exactly how ViWi uses RF. This oracle is an upper bound that no sensor provides; RF, without any labels, already recovers about 29% of its gain over vision only. An RF-based material classifier could thus be added to ViWi as an extra input. We will add both experiments to the paper.

Reviewer_gBq8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a new framework named ViWi to estimate voxel-level Young’s modulus, Poisson’s ratio, and density. It groups visual voxel features into material slots and conditions the slots on a simulated RF. The experimental results show that the introduction of RF can improve the performance of purely visual methods.

Critical strengths
  1. Sharing material prototypes across voxels is well motivated, and the vision-only results support this choice. The material-consistency analysis in Figure 12 supports grouping by material rather than only spatial proximity.
  2. The ablation is good. The extended vision-only training control and downstream impact help assess the contribution and practical relevance of the predicted fields.
Critical weaknesses
  1. The paper does not convincingly establish that WiFi can resolve visual ambiguity in material properties. The RF input is simulated from known material labels, and unclear labels are assigned using the ground-truth mechanical properties. The reported gains may therefore depend heavily on the material relationships built into the simulation. It remains unclear whether actual WiFi measurements would provide similarly useful information.

  2. With matched additional training, RF reduces mean error, while both Poisson’s ratio metrics and density ADE become worse. It remains unclear how much the specific RF-slot design contributes beyond the material-slot model. Such additional contributions should be further explored both theoretically and experimentally.

  3. The experiment design can be further improved, including more details of baseline implementations, evaluation on K.

Questions for authors
  1. The authors are advised to explain the theoretical insight into the gain introduced by RF.

  2. How could the same RF descriptor be obtained from measured WiFi signals, particularly the path statistics computed after removing direct paths?

  3. Do the RF gains remain when electromagnetic properties vary within material categories and ambiguous material names are resolved without using the mechanical targets?

  4. The ablation results are inconsistent. Table 5 reports 6.45% to 5.87% mean error and improvement in only two metrics, whereas the text reports 6.31% to 5.81% and improvement in five.

5 comments

We thank the reviewer for the positive assessment and for the careful reading of both the main paper and the appendix. The questions on the source of the RF gain, on practical measurements, and on the construction of the electromagnetic (EM) inputs are very helpful, and we answer each of them below, with new experiments where possible.

Protocol of the new experiments. Because of the limited time and compute of the discussion period, each new model was trained on a single GPU (batch size 16) instead of 8 GPUs, for the same 112k steps as in the paper, and thus saw 8×8\times fewer training samples. Absolute errors are therefore slightly higher than in Table 1 (6.02% vs. 5.81% for the identical RF input). All new models share the same initialization (the vision-only checkpoint ViWi$^\dagger$), hyperparameters and test set (148 GVM objects), so the rows are directly comparable with each other.


Weakness: contribution of the RF-slot design

To isolate our RF-slot design, we trained the same model with the same RF input but a simple fusion: the RF features are added to every voxel feature, without FiLM on the slots and without the slot correction (Table G3).

Table G3. Contribution of the RF-slot design (same 64-D RF input; GVM test set, step 112k).

ModelEE ALDE↓EE ALRE↓vv ADE↓vv ARE↓pp ADE↓pp ARE↓Mean↓
Vision only (no RF)0.19850.02160.01580.0549196.330.11396.34%
RF, simple fusion0.18990.02050.01440.0504220.000.12386.49%
RF, RF-slot design (ours)0.19430.02100.01480.0508200.110.10876.02%

Simple fusion does not help: 6.49%, even worse than vision only (6.34%), while our RF-slot design gives 6.02%. A likely reason is that RF is a global, object-level cue: added to every voxel, it shifts all voxels in the same way and mainly hurts density; injected into the slots, it adjusts each material hypothesis, while vision decides which voxels belong to which slot.


Q1. Theoretical insight into the gain introduced by RF

Let YY denote a material property (EE, ν\nu or ρ\rho), VV the visual observation and RR the RF descriptor. For the squared loss, the Bayes risk is the expected posterior variance, and the law of total variance gives

E[Var(Y∣V)]−E[Var(Y∣V,R)]=E[Var(E[Y∣V,R]∣V)]≥0.\mathbb{E}\big[\mathrm{Var}(Y\mid V)\big]-\mathbb{E}\big[\mathrm{Var}(Y\mid V,R)\big]=\mathbb{E}\big[\mathrm{Var}\big(\mathbb{E}[Y\mid V,R]\mid V\big)\big]\ge0 .

Adding RR therefore never increases the risk of the optimal predictor, and the gain is strictly positive exactly when objects that look alike (same VV) but differ in their RF response (different RR) also differ in their expected material properties.

This condition holds physically. A camera observes only the surface, so different materials can look identical (e.g., painted metal and plastic). At 2.4 GHz (wavelength 12.5 cm), the RF response depends on the bulk complex permittivity εc=εr−j σ/(ωε0)\varepsilon_c=\varepsilon_r-j\,\sigma/(\omega\varepsilon_0), where εr\varepsilon_r is the relative permittivity, σ\sigma the conductivity and ω\omega the angular frequency, while thin dielectric layers such as paint have almost no effect. Since EM and mechanical properties are both determined by the same composition (e.g., metals are conductive, dense and stiff; foams have εr≈1\varepsilon_r\approx1 and are light and compliant), RR separates materials that VV cannot.

The theory predicts that the gain is small where vision is already unambiguous and largest where it is not. This is what we observe: the overall gain on GVM is moderate (6.10% →\to 5.81%, Ablation II), but it is larger on visually ambiguous objects (Ablation I: 6.79% →\to 6.21%) and on unseen material categories (Ablation V: 20.47% →\to 16.18%, 21% relative). We will add this analysis to Section 3.

Q2. Obtaining the descriptor from measured WiFi signals

Removing the direct path. A real device cannot separate the direct path from the other paths, but it does not need to. We use empty-scene (background) subtraction, the standard method in WiFi sensing: the channel is measured once without and once with the object, and the descriptor is computed from the difference Hobj+LoS−Hempty\mathbf{H}_{\mathrm{obj+LoS}}-\mathbf{H}_{\mathrm{empty}}. The direct path is the same in both measurements and cancels, so only the effect of the object remains.

Statistics from measured signals. Instead of ideal path-level quantities, we compute the four statistics per transmitter from the channel state information (CSI) that commodity WiFi cards report (e.g., Intel 5300 CSI Tool, Atheros CSI Tool, Nexmon): total received power, fraction of receivers above 10× the noise floor, strongest tap of the channel impulse response, and RMS delay spread. We simulated this at 40 MHz (114 subcarriers) with thermal noise and random gain errors. We also tested the simplest signal, RSSI computed from the same noisy CSI and quantized to 1 dB (change with respect to the empty scene) with only 2 transmitters and 4 receivers around the object. The model was retrained for each setting (Table G1; details in our response to Reviewer Sdmk).

Table G1. RF descriptor from realistic measurements (16 Tx + 32 Rx unless noted; GVM test set, step 112k).

RF inputEE ALDE↓EE ALRE↓vv ADE↓vv ARE↓pp ADE↓pp ARE↓Mean↓
ViWi, Table 1 - main paper (8 GPUs; ideal paths, LoS off)0.17970.01960.01510.0515198.220.10345.81%
Ideal paths, LoS off (paper setting)0.19430.02100.01480.0508200.110.10876.02%
40 MHz CSI + noise, LoS off0.20090.02180.01450.0498195.500.10775.98%
40 MHz CSI + noise, LoS on + empty-scene subtraction0.19140.02050.01530.0526189.320.10555.95%
RSSI (1 dB) + noise, LoS on + empty-scene subtraction, 2 Tx + 4 Rx0.20880.02230.01370.0456193.860.10085.62%

Results. The CSI-based descriptors are as good as the ideal descriptor of the paper (5.95–5.98% vs. 6.02%), also with the direct path present (5.95%), and both are better than vision only (6.34%, Table G2). Even the minimal setup, RSSI only with the direct path present and only 2 transmitters and 4 receivers, reaches 5.62% (better than vision only on 92 of 148 objects).

Why it works. The material information is in how much the object changes the received power on each link, by reflecting, absorbing and blocking the signal according to its permittivity and conductivity. It is not in fine path details: at 40 MHz the delay resolution is 25 ns, much coarser than the nanosecond-scale delays of a 1 m object. All realistic settings keep this power change, and averaging over many subcarriers and receivers reduces the noise. We will add this measurement model to the appendix.

Q3. Variation of EM properties and resolution of ambiguous names

EM variation within a category. Per-object measurements of εr\varepsilon_r and σ\sigma are not available for the GVM assets, so each category uses representative values from ITU-R P.2040 and the literature. Within-category variation acts as a perturbation of the RF descriptor, which Ablation IV of the paper measures directly: Gaussian noise of 0.25 and 0.5 standard deviations per dimension changes the mean error only from 5.81% to 5.92% and 6.02%, still below vision only (6.10%). The realistic measurement noise in Table G1 likewise causes no loss. ViWi therefore does not depend on exact EM values.

Ambiguous names. 18.2% of the voxels (in 30% of the objects) have uninformative names; as described in the appendix, their EM category was assigned by nearest centroid in mechanical-property space, because a real WiFi signal interacts with the material, not with its name. This step only selects one of 12 coarse categories, and the RF descriptor is a 64-D summary of the whole object, so it cannot encode per-voxel targets: even an oracle with the true category of every voxel reaches only 9.99% (please see our response to Reviewer DQ7C). To answer the question directly, we re-simulated all descriptors with names only, giving unknown voxels a generic dielectric (εr=2.9\varepsilon_r=2.9, σ=0.012\sigma=0.012 S/m), and retrained the model (Table G2).

Table G2. EM assignment for ambiguous material names (GVM test set, step 112k).

ModelEE ALDE↓EE ALRE↓ν\nu ADE↓ν\nu ARE↓ρ\rho ADE↓ρ\rho ARE↓Mean↓
Vision only (no RF)0.19850.02160.01580.0549196.330.11396.34%
RF, physically consistent EM (paper)0.19430.02100.01480.0508200.110.10876.02%
RF, names only (generic dielectric for unknown)0.19790.02160.01580.0541218.120.11326.30%

With names only, the RF gain disappears (6.30% vs. 6.34% without RF), and the density error becomes even worse than vision only (ADE 218.12 vs. 196.33). The reason is that the generic placeholder assigns physically wrong EM properties: more than a third of the unknown voxels are metal (10.3%), stone (9.1%), concrete (6.9%), organic material, glass or soil. A metal part, for example, is simulated as almost transparent instead of as a strong reflector, so the RF descriptor describes a different object than the one that is seen and evaluated. The model therefore receives misleading rather than missing information, consistent with the worse density error. This shows that RF helps when its EM inputs match the real materials, which a real measurement guarantees by construction, since the signal is produced by the physical material and not by its label. We will add this experiment to the appendix.


Q4. Inconsistent numbers for Ablation VI

Thank you for catching this typo. Table 5 is correct; the text reported numbers from other runs. We will correct the text: increasing the transmitters from 4 to 16 reduces the mean error from 6.45% to 5.87%, mainly through density.

Comments on the paper

1 comment

Dear Area Chair,

We sincerely thank you for handling our submission, and all reviewers for their time, careful reading, and constructive feedback. Below is a brief summary of the discussion and of the revised PDF (additions in blue for Reviewer DQ7C and in green for Reviewer Sdmk).

 

  1. Practicality of RF sensing (Sdmk, gBq8). New experiments show that the RF gain holds with realistic 40 MHz CSI and noise, with the direct path present and removed by empty-scene subtraction, and even with RSSI from only 2 transmitters and 4 receivers (5.62% vs. 6.34% for vision only). Appendix E now states the assumptions and application boundaries, shows the antenna layout, and discusses simulation vs. real deployment. Reviewer Sdmk raised the rating.

 

  1. Clarity and figures (DQ7C, Sdmk). Multi-material examples with the learned slots are now in the main paper (Figure 7), with a failure analysis in Appendix D. Figure 1 now includes ground-truth context, and the fonts of all figures have been enlarged.

 

  1. Design choices and baselines (DQ7C, gBq8). Appendix C adds analyses of the embedding dimensions, material classification as a baseline and as an input, and volume estimation for ABO-500 mass. In our responses, we also show that simply fusing RF features does not help (6.49%, vs. 6.02% with our RF-slot design), and that physically consistent EM properties are required for the RF gain (6.30% with material names only); both results will be added to the final version.

 

We are truly grateful to the reviewers for their thoughtful engagement throughout the discussion, which has substantially improved our paper, and to the Area Chair for their time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.