- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
To address the lack of standardized, physically acquired datasets for passive non-line-of-sight (NLOS) imaging in complex indoor environments—characterized by ambient light and reflections from furniture and walls—this paper introduces a novel physical benchmark dataset and evaluation protocol for passive NLOS imaging.
-
The benchmark comprises 13,000 "measurement-target" pairs physically captured within a controlled indoor conference room setting (covering 10 categories from the STL-10 dataset). It includes background references, a fixed data split (12,000 for training, 1,000 for testing), and comprehensive provenance information. The task is divided into two complementary reconstruction tracks: (1) A "measurement-only" track, which evaluates the ability of deterministic models (e.g., NLOSNet-T, U-Net, NAFNet, GAN, NLOS-OT) to recover hidden targets directly from weak, entangled passive observations; and (2) A "known-class-conditioned refinement" track, which utilizes a coarse initial reconstruction combined with external semantic class labels to perform generative refinement via a pre-trained latent diffusion model (Base LDM) or full U-Net supervised fine-tuning (Full-UNet SFT).
-
Experiments reveal a significant trade-off: while deterministic models excel in pixel-level and local structural fidelity (as measured by PSNR/SSIM), generative diffusion-based refinement—though enhancing perceptual naturalness and distribution-level metrics (such as LPIPS/FID)—does not guarantee improved reconstruction fidelity for specific target instances. In some cases, the introduction of class priors may even cause the reconstruction to deviate from the actual ground-truth target.
Critical strengths
-
Unlike previous benchmarks that relied heavily on purely synthetic simulations or active, time-resolved (laser-based) data, this work utilizes a dataset of 13,000 real-world paired samples collected in a physical environment featuring authentic indoor lighting (~300 lx) and complex multi-path reflections. This contribution is highly significant for advancing passive non-line-of-sight (NLOS) technologies toward practical applications, such as robotic search-and-rescue operations and blind-spot perception for autonomous vehicles.
-
By rigorously controlling experimental variables, the authors clearly demonstrate that "perceptual quality" (as measured by LPIPS/FID) and "pixel-level fidelity" (as measured by PSNR/SSIM) can diverge in underdetermined passive NLOS tasks. This sober scientific assessment highlights the risk of "hallucinations"—where diffusion models produce results that are visually realistic yet factually inconsistent with the actual scene—serving as an important cautionary note for the broader Scientific Machine Learning (SciML) community.
Critical weaknesses
-
In the main experiments presented in Table 1, deterministic models for direct reconstruction (such as NLOSNet-T and NAFNet) take only the processed passive measurement map as input. In contrast, the "Base LDM" and "Full-UNet SFT" models—used in the refinement track—leverage an additional external input: the ground-truth class label (class condition). This architectural information asymmetry gives the diffusion models an advantage in terms of LPIPS and FID scores (Table 1). This performance gain is largely attributable to the strong semantic priors provided by the class labels, rather than being an inherent benefit of the generative diffusion architecture itself. To ensure a truly fair architectural comparison, the authors should include an "unconditional LDM" in the refinement track, or alternatively, evaluate the deterministic models under the same conditional settings (e.g., by providing class labels to a U-Net).
-
As described in Section 3.2.1, because the projector casts images directly onto a hidden screen, the authors employ a "fixed image-space black mask" during preprocessing to manually suppress the direct specular highlights (hotspots) produced by the projector. In real-world passive non-line-of-sight (NLOS) scenarios—such as around a vehicle corner or within a completely occluded zone—the probe or target light sources are typically unknown and uncontrollable; therefore, such an ideal "prior mask" for direct hotspots does not exist. This preprocessing step implicitly leaks information about the projection geometry, thereby undermining the benchmark’s validity for evaluating scenarios involving unknown light sources in truly occluded environments.
-
While the paper highlights challenges posed by furniture, walls, and ambient reflections in indoor settings, the dataset description lacks ablation studies regarding factors such as object reflectivity, surface roughness, or geometric variations within the scenes. The fundamental ill-posed nature of passive NLOS imaging stems from severe information ambiguity caused by multipath interference. Since the benchmark is based exclusively on data collected in a static conference room, it lacks testing for generalization across different room layouts or dynamic background occlusions, making it difficult to substantiate the robustness of the proposed methods in general indoor environments.
-
In Section 3.3 and Appendix B, the authors state that the FID score is calculated using a full test set consisting of 1,000 predicted images and 1,000 target images. However, as a metric for measuring the distributional discrepancy between sets of images, FID typically requires a sufficiently large sample size—often on the order of tens of thousands—to ensure stable estimation of feature covariance. Given the small test set of only 1,000 samples, combined with an uneven distribution across 10 classes, calculating the FID on the entire set introduces significant statistical variance and bias (i.e., the "sample-size bias" of FID). Furthermore, the analysis lacks stratified statistics for individual classes; while the appendix notes that class-level FID was not computed, the overall reliability of FID results derived from such a small sample size remains questionable.
Questions for authors
-
To effectively decouple the respective contributions of the "diffusion generative prior" and the "known class condition," have the authors evaluated the performance of the Base LDM or SFT models—when the class label is omitted during inference (i.e., the setting)—as presented in Table 1? Specifically, if the class label is excluded and the model relies solely on the diffusion prior to refine the output of NLOSNet-T, do the LPIPS and FID scores still outperform those of NAFNet or the original NLOSNet-T?
-
To what extent does the fixed "black mask" simplify the mapping from measurements to the target scene? How significantly would the performance of NLOSNet-T and SFT degrade if this mask were removed during testing, or if the projector’s illumination position on the hidden screen were varied randomly? Does this imply that the current benchmark relies, to some degree, on the constraints of a fixed acquisition geometry?
-
Given that the diagnostic test set contains only 1,000 images (averaging just 100 images per class), how have the authors verified that the FID scores calculated at this scale (e.g., 85.49 vs. 53.05 in Table 1) are statistically significant rather than merely the result of random noise? Could the authors provide additional stability verification, such as through bootstrapping or multi-scale structural similarity (MS-SSIM) analysis?