acceptodds
Under review as a conference paper at ICLR 2027

Visual Generative Foundation Models Are Reusable Priors for Non-Visual Sensors

Abstract

Visual foundation models have acquired transferable knowledge of scene geometry, spatial structure, and semantics from internet-scale imagery, yet sensors that remain informative where cameras fail, such as radio-frequency (RF) sensors, lack a comparable data ecosystem. Existing approaches either pretrain a separate foundation model for each sensor or distill one task at a time from a visual teacher, requiring substantial new effort for every new sensor or task. We ask whether a sensor can instead access the knowledge of a pretrained visual foundation model through a lightweight interface. In this paper, we present RFception, which maps RF measurements into the latent space of a pretrained visual generative perception model. Because the pretrained model selects tasks through text prompts, a single RF representation supports multiple geometric and semantic perception tasks without task-specific RF backbones or prediction heads. Across six held-out buildings, Reception outperforms task-specific RF models in depth estimation and, without task-specific RF supervision, performs surface-normal estimation, human pose estimation, and category-conditioned instance segmentation zero-shot. Its visual outputs can further be consumed by off-the-shelf vision-language models without fine-tuning, extending RF sensing to language-level reasoning. These results suggest that the knowledge captured by visual foundation models is not inherently tied to images, and that new sensing modalities can inherit this knowledge by learning an interface to the model rather than building a foundation model of their own.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
RF sensing, multimodal learning, visual foundation models, zero-shot transfer

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_LPsGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper adapts vision image generation model, which can produce multi-task outputs such as depth and segmentation, to support radio-frequency (RF) sensor input. The main idea is to consider align the RF input embedding with latent embedding of vision image generation model such that the output capability of image generation model can be preserved. The experiment shows that the proposed method can achieve better depth estimation performance than task-specific model and demonstrate strong zero-shot performance on other tasks.

Critical strengths

The motivation of adapting the vision image generation model to support multi-task prediction of RF sensor is clear. I think it is a way to address the limited data issue of RF sensing.

Critical weaknesses

I think the main limitation of this work is that it does not clearly make ablations of the proposed alignment modules. Another thing is the paper does not demonstrate the proposed method can be useful across different vision generation models.

Questions for authors
  1. The overall method pipeline figure is not very clear to me to understand how the proposed alignment method is implemented. Could you please provide a detailed architecture illustration figure in the paper/appendix or clarify on some details for reproduction, eg, what parts of the model are frozen or trained during the proposed alignment stage?

  2. From the qualitative results of the proposed method, it looks like some of the predicted outputs have distortion effects especially if you zoom into some geometries that are flat/straight in line. Since vision image generation models typically do not use non-planar images as input, could you provide more analysis on how well the proposed adaptation method address the planar image input to non-planar image input domain gap?

2 comments
AuthorsOpenReview ↗

Response to Reviewer LPsG

We thank the reviewer for the detailed review and address the points one by one below.


Q1. Training details and ablations


The revised Figure 2(a) explicitly shows training and gradient paths. The RF adapter is a nine-residual-block ResNet mapping a 256×192×64256\times192\times64 heatmap to a 16×96×9616\times96\times96 condition latent. Representation alignment compares this with the frozen VAE encoding of the paired LiDAR crop. The RF condition tokens, noisy task-target tokens, and text embeddings are supplied to the generative backbone. The training proceeds in three parts.

  • Lrep=Lmse+Ldense\mathcal{L}_{\mathrm{rep}}=\mathcal{L}_{\mathrm{mse}}+\mathcal{L}_{\mathrm{dense}} updates the RF adapter. The VAE target encoder is frozen.
  • Lfunc\mathcal{L}_{\mathrm{func}} uses depth and entity targets. It backpropagates through the backbone to the RF adapter and rank-128 LoRA parameters on attention Q/K/V projections. The pretrained backbone weights, VAE encoder/decoder, and text encoder remain frozen.
  • Metric depth uses a separate stage. Only the eight-block metric decoder is trained and the complete generative interface is frozen (Appendix C).

Table R1 quantifies each loss and task-set variant, and Table R2 separates the RF adapter from LoRA. These controls show that functional optimization of the RF adapter enables transfer even when the entire pretrained model is fixed. Relative-depth results bypass the metric decoder to avoid conflating these stages.

Table R1. Ablation of the alignment objectives. Lmse\mathcal{L}_{\mathrm{mse}} and Ldense\mathcal{L}_{\mathrm{dense}} are the two terms of Lrep\mathcal{L}_{\mathrm{rep}} (Eq. 4). K\mathcal{K} is the functional alignment task set, with D for depth and E for entity segmentation. Relative depth is the direct output of DICEPTION.

ConfigurationLmse\mathcal{L}_{\mathrm{mse}}Ldense\mathcal{L}_{\mathrm{dense}}K\mathcal{K}LoRARel. depth AbsRel↓Normal 30°↑Pose AR↑Inst. AP↑
Representation only✓✓∅✗2.197.260.000.00
Depth only✓✓D✓0.11773.8345.3314.72
Full (paper)✓✓D, E✓0.12271.4251.1655.17
No dense contrastive✓✗D, E✓0.12470.1149.8154.23
No MSE✗✓D, E✓0.16062.9543.2848.29
Functional only✗✗D, E✓0.16162.0442.7648.03

Table R2. Separating the RF adapter from LoRA. The adapter AϕA_\phi and LoRA Δθ\Delta\theta columns give the objective each component is trained with. Normal, pose, and instance are zero-shot in all rows. Relative depth is the direct output of DICEPTION.

ConfigurationAdapter AϕA_\phiLoRA Δθ\Delta\thetaRel. depth AbsRel↓Normal 30°↑Pose AR↑Inst. AP↑
Full (paper)Lrep+Lfunc\mathcal{L}_{\mathrm{rep}}+\mathcal{L}_{\mathrm{func}}Lfunc\mathcal{L}_{\mathrm{func}}0.12271.4251.1655.17
Frozen backbone, no LoRALrep+Lfunc\mathcal{L}_{\mathrm{rep}}+\mathcal{L}_{\mathrm{func}}–0.14868.3147.9552.34
Frozen adapter, LoRA onlyLrep\mathcal{L}_{\mathrm{rep}}Lfunc\mathcal{L}_{\mathrm{func}}1.7710.130.000.00
Representation onlyLrep\mathcal{L}_{\mathrm{rep}}–2.197.260.000.00

Q2. Straight-line distortion


We analyze the distortion from three aspects.

(1) The curvature is expected. PanoRadar represents RF and LiDAR on an azimuth-elevation grid. Its wide field of view causes wide-angle distortion, so a straight 3D edge can appear curved in this spherical projection. The LiDAR ground truth shows the same curvature.

(2) Distortion also depends on RF accuracy. Since RF quality degrades with distance, we evaluate the depth error at different distances (Table R3). Depth MAE increases from 0.145 m at 0 to 3 m to 0.393 m at 6 to 10 m. This reflects a real limitation. Accuracy degrades as distance grows.

(3) Depth error reflects 3D error. Each pixel of the depth image corresponds to a ray with a known azimuth and elevation. Using the angular parameters of the LiDAR, the depth image can be projected into a 3D point cloud. The 3D error of each point then equals its depth error along the ray. Therefore, depth error reflects the geometric error in 3D space.

Table R3. Depth error of RFception by distance.

Distance0 to 3 m3 to 6 m6 to 10 m
Depth MAE (m)0.1450.2090.393
AuthorsOpenReview ↗

Kind Follow-up on Rebuttal Discussion

Dear Reviewer,

Thank you for your constructive feedback. We have carefully addressed your concerns in our rebuttal and provided additional experimental results. As the discussion period is approaching its end, we would greatly appreciate it if you could review our responses and let us know whether any concerns remain.

Thank you for your time and consideration.

Reviewer_fAwQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RFception connects RF heatmaps to a pretrained DiCeption visual generalist through a shared RF adapter and LoRA updates. It combines alignment to paired LiDAR-range-image latents with functional alignment on depth and entity segmentation, then evaluates surface normals, human pose and person-instance segmentation without corresponding task-specific RF supervision. It also feeds generated depth to frozen VLMs. The main contribution is cross-task reuse through one sensor interface, evaluated on six held-out buildings.

Critical strengths
  1. Reusing one RF interface across tasks with different outputs is a useful extension beyond task-specific RF prediction. With only depth/entity functional alignment, the model obtains 51.16 pose AR and 55.17 person-instance AP (Table 1). These are below supervised specialists, but substantiate useful transfer without the corresponding RF task labels; reaching specialist performance is not necessary for this contribution.

  2. The building-level split and retrained, field-of-view-matched PanoRadar baseline make the environmental generalization evidence meaningful. The complete system improves building-averaged depth MAE from 0.194 m to 0.170 m and delta1 from 0.953 to 0.974. Evaluation with four frozen VLMs additionally supports the practical compatibility of the visual interface.

Critical weaknesses
  1. The metric-depth gain is not cleanly attributable to the generative prior. Appendix C states that every depth result in Tables 1/4 uses a separately trained eight-block metric decoder receiving both generated relative depth and a direct RF branch. A matched RF-only decoder control is missing. Thus, the depth improvement cannot by itself establish the prior's independent contribution, and the broad claim of no task-specific prediction heads needs a metric-depth exception. Performance is also complementary: AbsRel is worse than matched PanoRadar in all six buildings, despite better RMSE.

  2. The novelty lies in shared-interface transfer rather than the individual ingredients: DiCeption supplies prompted multi-task generation, and RadarSFD already transfers pretrained visual geometric priors to radar. Figure 5 usefully compares no functional alignment, depth-only and depth-plus-entity alignment, but only qualitatively. Quantitative held-out-task results are needed to establish what each alignment objective contributes. Moreover, entity supervision already supplies object-grouping information, so person-instance transfer should be described as lacking task-specific RF labels rather than unrelated supervision.

I recommend weak rejection because the current evidence does not yet establish a sufficiently substantial contribution from the shared RF interface beyond the inherited multi-task model and the aligned geometry/grouping supervision. Quantitative transfer ablations and the matched metric-decoder control could materially strengthen this assessment.

Questions for authors
  1. Do you have a metric-depth control with the same RF branch, decoder and supervision budget, but without generated relative depth? Please use it to delimit the generative prior's contribution, and report the decoder's parameter count/training cost separately from the approximately 70M interface parameters.

  2. Can the existing Figure 5 checkpoints provide quantitative pose or person-instance results for representation-only, depth-only and depth-plus-entity alignment? Please also specify the DiCeption input used to create entity targets and the Keypoint R-CNN input/filtering used for pose evaluation. These would clarify the transfer mechanism and supervision boundary without changing the task set.

3 comments
AuthorsOpenReview ↗

Response to Reviewer fAwQ

We thank the reviewer for the valuable feedback and for specifying which evidence would change the assessment.


W1 / Q1. Attribution of the prior


We agree that the metric-depth gain cannot be fully attributed to the generative prior. To clarify the contribution of the prior, we remove the metric decoder and evaluate the relative depth output directly by DICEPTION. We further separate the RF adapter from LoRA (Table R1). With the backbone frozen and only the adapter trained, the model retains most of the full model's performance. In contrast, training only LoRA with the adapter frozen fails on every task.

Whether the output is metric or relative depth is determined by the base model. DICEPTION outputs relative depth. Without retraining the base model or injecting additional information, metric depth theoretically cannot be recovered from relative depth. Appendix C gives a detailed explanation and the derivation.

Table R1. Separating the RF adapter from LoRA. The adapter AϕA_\phi and LoRA Δθ\Delta\theta columns give the objective each component is trained with. Normal, pose, and instance are zero-shot in all rows. Relative depth is the direct output of DICEPTION.

ConfigurationAdapter AϕA_\phiLoRA Δθ\Delta\thetaRel. depth AbsRel↓Normal 30°↑Pose AR↑Inst. AP↑
Full (paper)Lrep+Lfunc\mathcal{L}_{\mathrm{rep}}+\mathcal{L}_{\mathrm{func}}Lfunc\mathcal{L}_{\mathrm{func}}0.12271.4251.1655.17
Frozen backbone, no LoRALrep+Lfunc\mathcal{L}_{\mathrm{rep}}+\mathcal{L}_{\mathrm{func}}–0.14868.3147.9552.34
Frozen adapter, LoRA onlyLrep\mathcal{L}_{\mathrm{rep}}Lfunc\mathcal{L}_{\mathrm{func}}1.7710.130.000.00
Representation onlyLrep\mathcal{L}_{\mathrm{rep}}–2.197.260.000.00

W2 / Q2. Ablation of the alignment objectives


Table R2 ablates the alignment objectives. Representation alignment alone does not yield a usable interface. Adding depth alignment enables depth, normals, and pose, but instance segmentation remains weak. Adding entity alignment restores instance segmentation and improves pose. The evaluation targets for each task are detailed in the Annotations paragraph of Appendix A.

Our method and RadarSFD focus on different problems. RadarSFD retrains a pretrained generative model on radar for a single task. Our method studies how an interface alone enables zero-shot transfer to multiple tasks, without retraining the visual foundation model.

Table R2. Ablation of the alignment objectives. Lmse\mathcal{L}_{\mathrm{mse}} and Ldense\mathcal{L}_{\mathrm{dense}} are the two terms of Lrep\mathcal{L}_{\mathrm{rep}} (Eq. 4). K\mathcal{K} is the functional alignment task set, with D for depth and E for entity segmentation. Relative depth is the direct output of DICEPTION.

ConfigurationLmse\mathcal{L}_{\mathrm{mse}}Ldense\mathcal{L}_{\mathrm{dense}}K\mathcal{K}LoRARel. depth AbsRel↓Normal 30°↑Pose AR↑Inst. AP↑
Representation only✓✓∅✗2.197.260.000.00
Depth only✓✓D✓0.11773.8345.3314.72
Full (paper)✓✓D, E✓0.12271.4251.1655.17
No dense contrastive✓✗D, E✓0.12470.1149.8154.23
No MSE✗✓D, E✓0.16062.9543.2848.29
Functional only✗✗D, E✓0.16162.0442.7648.03

Thank you for the new ablations. The frozen-backbone result addresses my concern about the contribution of the shared RF interface: the adapter retains most of the full model’s pose and instance-segmentation performance without LoRA. I am keeping my rating at 2 (weak rejection) because, while the cross-task reuse is now better established, I still find the methodological advance modest. The matched RF-only decoder control is also still missing, so the generative prior’s contribution to the reported metric-depth gains remains unclear.


Methodological contribution


To our knowledge, RFception is the first work in which one RF representation serves multiple perception tasks through a single pretrained visual foundation model. Tasks are selected by text prompts, and the foundation model is never retrained on RF. Prior RF methods train a dedicated model for each task with task-specific RF supervision. In RFception, the visual prior enables zero-shot task transfer. Surface normals, human pose, and person instances are obtained without task-specific RF labels.


Contribution of the generative prior to depth


We ablate the decoder and LoRA to isolate the contribution of the generative prior to depth estimation. Table R1 of our previous response reports relative depth with and without LoRA. We further convert the output of the decoder to relative depth and evaluate it with the same protocol (Table R3). The direct output of DICEPTION reaches an AbsRel of 0.122 without any decoder. With the backbone fully frozen, it still reaches 0.148. These two rows involve only the RF adapter and the pretrained model. The decoder further reduces AbsRel to 0.075 with its RF branch.

Table R3. Ablation of the decoder and LoRA on relative depth. Rows without the decoder use the direct output of DICEPTION.

ConfigurationMetric decoderLoRARel. depth AbsRel↓
Full, with decoder✓✓0.075
Full, direct output✗✓0.122
Frozen backbone, no LoRA✗✗0.148

We would like to clarify why the decoder cannot be ablated on metric depth. DICEPTION outputs relative depth. Its depth target is normalized in disparity space with the per-image minimum dmin⁡d_{\min} and maximum dmax⁡d_{\max} (Eq. 7 in Appendix C). Recovering metric depth requires inverting this normalization (Eq. 8),

d^i=(d~^i+12(1dmin⁡−1dmax⁡)+1dmax⁡)−1.\hat{d}_{i}=\left(\frac{\hat{\tilde{d}}_{i}+1}{2}\left(\frac{1}{d_{\min}}-\frac{1}{d_{\max}}\right)+\frac{1}{d_{\max}}\right)^{-1}.

The extrema dmin⁡d_{\min} and dmax⁡d_{\max} are unavailable at test time. Without them, the output of DICEPTION is defined only up to an unknown affine transform in disparity space and cannot be converted to metric depth. A control with and without the decoder can therefore only be compared on relative depth.

Reviewer_LW1xOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RFception adapts DICEPTION to RF inputs using latent alignment, depth/entity supervision, and LoRA. Normals, pose, and person segmentation transfer without direct task-specific RF labels. Metric depth uses an additional supervised decoder.

Critical strengths
  • Reusing an existing visual generalist across multiple RF tasks is a meaningful cross-modal transfer contribution.

  • Evaluation on six entirely held-out buildings, with a crop-matched PanoRadar baseline, provides a useful generalization test.

Critical weaknesses
  • Limited novelty. DICEPTION already supplies the prompt-driven perception interface; related work such as Vision Banana [1] and GenCeption [2] further establishes the broader generative-perception paradigm. RFception's potential novelty lies in making these capabilities accessible from RF measurements. The paper would also benefit from positioning its contribution against these related approaches.

  • Insufficient ablation study. Removing the functional objective also removes the only stated training signal to LoRA. Figure 5 therefore does not establish whether improved outputs result from better conditioning latents, backbone adaptation, or both. Quantitative comparisons with and without LoRA under the same functional objective, and functional-only versus combined alignment, would clarify the mechanism.

  • Worse performance. Normals (63.8% versus 85.3%) and pose (51.2 versus 63.8 AR) substantially trail cropped PanoRadar. Although PanoRadar receives task-specific RF supervision, claims of better fine geometry still require object-region or boundary metrics.

[1] Gabeur et al. Image Generators are Generalist Vision Learners (Vision Banana), 2026. https://arxiv.org/abs/2604.20329

[2] Wang et al. Video Generation Models are General-Purpose Vision Learners (GenCeption), 2026. https://arxiv.org/abs/2607.09024

Questions for authors

Please clarify the questions in the weakness section.

3 comments
AuthorsOpenReview ↗

Response to Reviewer LW1x

We thank the reviewer for the careful reading and specific suggestions.


W1. Novelty against Vision Banana, GenCeption, and DICEPTION


Our contribution is cross-modal access to an existing generalist, not a new generative-perception backbone. Vision Banana, GenCeption, and DICEPTION show that generative models are general vision learners for image or video inputs. Our question is complementary. Can a sensor that produces neither images nor videos access such a model, including for tasks that are not trained on that sensor?


W2. Insufficient ablation


Table R1 ablates the alignment objectives, and Table R2 separates the RF adapter from LoRA. The results show that zero-shot transfer still works without LoRA. With the backbone frozen and only the adapter trained, the model retains most of the full model's performance. In contrast, training only LoRA with the adapter frozen fails on every task. Therefore, transfer mainly comes from the adapter.

Table R1. Ablation of the alignment objectives. Lmse\mathcal{L}_{\mathrm{mse}} and Ldense\mathcal{L}_{\mathrm{dense}} are the two terms of Lrep\mathcal{L}_{\mathrm{rep}} (Eq. 4). K\mathcal{K} is the functional alignment task set, with D for depth and E for entity segmentation. Relative depth is the direct output of DICEPTION.

ConfigurationLmse\mathcal{L}_{\mathrm{mse}}Ldense\mathcal{L}_{\mathrm{dense}}K\mathcal{K}LoRARel. depth AbsRel↓Normal 30°↑Pose AR↑Inst. AP↑
Representation only✓✓∅✗2.197.260.000.00
Depth only✓✓D✓0.11773.8345.3314.72
Full (paper)✓✓D, E✓0.12271.4251.1655.17
No dense contrastive✓✗D, E✓0.12470.1149.8154.23
No MSE✗✓D, E✓0.16062.9543.2848.29
Functional only✗✗D, E✓0.16162.0442.7648.03

Table R2. Separating the RF adapter from LoRA. The adapter AϕA_\phi and LoRA Δθ\Delta\theta columns give the objective each component is trained with. Normal, pose, and instance are zero-shot in all rows. Relative depth is the direct output of DICEPTION.

ConfigurationAdapter AϕA_\phiLoRA Δθ\Delta\thetaRel. depth AbsRel↓Normal 30°↑Pose AR↑Inst. AP↑
Full (paper)Lrep+Lfunc\mathcal{L}_{\mathrm{rep}}+\mathcal{L}_{\mathrm{func}}Lfunc\mathcal{L}_{\mathrm{func}}0.12271.4251.1655.17
Frozen backbone, no LoRALrep+Lfunc\mathcal{L}_{\mathrm{rep}}+\mathcal{L}_{\mathrm{func}}–0.14868.3147.9552.34
Frozen adapter, LoRA onlyLrep\mathcal{L}_{\mathrm{rep}}Lfunc\mathcal{L}_{\mathrm{func}}1.7710.130.000.00
Representation onlyLrep\mathcal{L}_{\mathrm{rep}}–2.197.260.000.00

W3. Lower normal and pose performance


We agree that RFception underperforms PanoRadar on normal and pose. This gap is expected given their different training settings. RFception performs both tasks by zero-shot task transfer. PanoRadar is fully trained with supervision on each task. A foundation model is designed for generality and zero-shot use. On a specific task, it typically underperforms a fully supervised specialist. Vision Banana, GenCeption, and DICEPTION show the same gap (Table R3).

Table R3. Generalist models versus task-specific specialists. Results of the other generalists are taken from their papers. The specialist is the strongest task-specific model reported in the same table.

GeneralistTask (benchmark, metric)GeneralistSpecialistSpecialist
Vision BananaSemantic seg. (Cityscapes, mIoU↑)69.9SegMan-L84.2
Vision BananaReferring seg. (RefCOCOg, cIoU↑)73.8X-SAM83.8
Vision BananaInstance seg. (SA-Co/Gold, cgF1↑)47.5SAM 354.1
GenCeptionDepth (ETH3D, AbsRel↓)0.037VGGT-Ω0.016
GenCeptionCamera pose (Sintel, RPE-R↓)0.464D4RT0.126
DICEPTIONHuman pose (COCO, AP↑)57.8ViTPose78.3
DICEPTIONInstance seg. (COCO, AP↑)33.2Mask2Former50.1
RFceptionNormal (PanoRadar, 22.5°↑)63.8PanoRadar85.3
RFceptionHuman pose (PanoRadar, AR↑)51.2PanoRadar63.8
AuthorsOpenReview ↗

Kind Follow-up on Rebuttal Discussion

Dear Reviewer,

Thank you for your constructive feedback. We have carefully addressed your concerns in our rebuttal and provided additional experimental results. As the discussion period is approaching its end, we would greatly appreciate it if you could review our responses and let us know whether any concerns remain.

Thank you for your time and consideration.

I appreciate the authors for providing detailed responses and additional experiments. Most of my concerns are addressed. Therefore, I will increase my rating accordingly.

Reviewer_iSHKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes RFception, which connects RF sensing to a pretrained visual generative foundation model (DICEPTION) through a lightweight adapter. The method combines latent representation alignment and functional alignment, allowing the model to perform multiple perception tasks by changing text prompts. Experiments show improved depth estimation and zero-shot transfer to surface normals, human pose, and instance segmentation. The generated depth maps can also be used by existing VLMs for spatial reasoning. On the PanoRadar indoor dataset with a building-level split, RFception is demonstrated to keep the original model prior while outperforming existing baselines.

Critical strengths
  1. The paper presents an interesting, well-motivated direction with a convincing result. Qualitatively, it recovers doorways, furniture, thin structures, and people that the task-specific baseline blurs into surrounding surfaces.

  2. The combination of representation alignment and functional alignment is sensible, and the ablation on the alignment task set is insightful. The choice of tasks in KK determines which capabilities transfer: this finding is particular interesting to me.

  3. The task is broadly covered and the output is demonstrated to be useful for downstream VLM applications.

Critical weaknesses
  1. DICEPTION is pretrained with RGB image latents as its conditioning input, but RFception aligns the RF latent to the VAE latent of the LiDAR range image. A LiDAR depth latent lies outside DICEPTION's pretraining conditioning distribution, so it is unclear why this was chosen as the target or whether the residual to such a latent is the right quantity to minimize. The authors should compare RGB latent vs. LiDAR depth latent as the alignment target if possible. This would show whether the pretrained model can interpret such a representation at all and would provide an upper bound for what the RF adapter is being aligned toward.

  2. Functional alignment updates both the RF adapter and the LoRA parameters in the backbone, so it is unclear whether zero-shot transfer comes from the adapter learning a better interface or from the backbone being adapted to RF inputs. This distinction matters for the paper's central claim that the visual model's knowledge is "accessed through an interface" rather than partially re-learned.

  3. All experiments use a single indoor dataset from a single RF sensor. The scope of the experiments is hence limited to a constrained setup. Moreover, the sharper and richer details could be either genuine recovery from RF or hallucination from the visual prior. Some of the details seem unlikely to be resolvable from RF, so one should make sure that this is not overfitted to some particular scenarios/scenes.

Questions for authors
  1. Can the authors provide a more complete ablation that separately quantifies the contribution of MSE alignment, the dense contrastive loss, functional alignment, and LoRA, with numbers? In particular, if the pretrained backbone is fully frozen (no LoRA) and only the RF adapter is trained with functional alignment, does zero-shot transfer still work?

  2. How do the authors verify that the fine details recovered by the generative model come from the RF measurements rather than from hallucination by the visual prior, especially for people, furniture, and thin structures? Would the method work on more broad scenarios such as outdoor scenes?

2 comments
AuthorsOpenReview ↗

Response to Reviewer iSHK

We thank the reviewer for the valuable comments and the constructive suggestions.


W1. Why align to a LiDAR latent?


LiDAR is a spatially matched geometric anchor. The paired LiDAR range image shares the RF angular grid and records scene geometry without requiring the adapter to reconstruct unobserved color or texture. It therefore provides a practical target for local cross-modal correspondence. The PanoRadar dataset supplies paired RF and LiDAR but does not provide synchronized RGB, so we cannot make a paired RGB-versus-LiDAR target comparison on this benchmark.

We agree that VAE encoding alone does not remove the conditioning-distribution gap. DICEPTION provides a procedure for handling single-channel maps, and we follow it, replicating the range image to three channels and encoding it with the pretrained VAE.


W2 / Q1. Does transfer come from the adapter or the backbone? Does it work without LoRA?


Yes. Zero-shot task transfer persists with the backbone fully frozen. Table R1 ablates the alignment objectives, and Table R2 separates the RF adapter from LoRA. The results show that zero-shot transfer still works without LoRA. With the backbone frozen and only the adapter trained, the model retains most of the full model's performance. In contrast, training only LoRA with the adapter frozen fails on every task. Therefore, transfer mainly comes from the adapter.

Table R1. Ablation of the alignment objectives. Lmse\mathcal{L}_{\mathrm{mse}} and Ldense\mathcal{L}_{\mathrm{dense}} are the two terms of Lrep\mathcal{L}_{\mathrm{rep}} (Eq. 4). K\mathcal{K} is the functional alignment task set, with D for depth and E for entity segmentation. Relative depth is the direct output of DICEPTION.

ConfigurationLmse\mathcal{L}_{\mathrm{mse}}Ldense\mathcal{L}_{\mathrm{dense}}K\mathcal{K}LoRARel. depth AbsRel↓Normal 30°↑Pose AR↑Inst. AP↑
Representation only✓✓∅✗2.197.260.000.00
Depth only✓✓D✓0.11773.8345.3314.72
Full (paper)✓✓D, E✓0.12271.4251.1655.17
No dense contrastive✓✗D, E✓0.12470.1149.8154.23
No MSE✗✓D, E✓0.16062.9543.2848.29
Functional only✗✗D, E✓0.16162.0442.7648.03

Table R2. Separating the RF adapter from LoRA. The adapter AϕA_\phi and LoRA Δθ\Delta\theta columns give the objective each component is trained with. Normal, pose, and instance are zero-shot in all rows. Relative depth is the direct output of DICEPTION.

ConfigurationAdapter AϕA_\phiLoRA Δθ\Delta\thetaRel. depth AbsRel↓Normal 30°↑Pose AR↑Inst. AP↑
Full (paper)Lrep+Lfunc\mathcal{L}_{\mathrm{rep}}+\mathcal{L}_{\mathrm{func}}Lfunc\mathcal{L}_{\mathrm{func}}0.12271.4251.1655.17
Frozen backbone, no LoRALrep+Lfunc\mathcal{L}_{\mathrm{rep}}+\mathcal{L}_{\mathrm{func}}–0.14868.3147.9552.34
Frozen adapter, LoRA onlyLrep\mathcal{L}_{\mathrm{rep}}Lfunc\mathcal{L}_{\mathrm{func}}1.7710.130.000.00
Representation onlyLrep\mathcal{L}_{\mathrm{rep}}–2.197.260.000.00

W3 / Q2. Real recovery or hallucination?


We analyze hallucination from two aspects.

(1) RF quality bounds hallucination. Our output is generation controlled by RF, so its accuracy depends on how strongly the RF constrains it. PanoRadar rotates a single-chip radar to synthesize 8×12008\times1200 virtual antennas. This gives a range resolution of 3.75 cm, an azimuth resolution of 0.96∘0.96^\circ, and an elevation resolution of 14.2∘14.2^\circ. Within these sparse measurements, RF provides an effective constraint. Content outside them is completed by the prior of DICEPTION.

(2) Small objects come from the prior. The generative model may add small objects and texture details. Figures 1, 9, and 10 show examples such as ceiling lights, wall signs, and items on desks. These objects are below the angular resolution of the RF sensor. RF cannot constrain their shape, and the model generates one plausible instance from its prior.

Diverse held-out scenes. We split the PanoRadar dataset by building. The PanoRadar buildings differ in materials, layout, and design. The test scenes cover offices, classrooms, lecture halls, lounges, hallways, and stairwells. Table 1 of the paper reports each building separately. This indicates that the gains do not come from overfitting to particular scenes. We have not evaluated outdoor scenes.

AuthorsOpenReview ↗

Kind Follow-up on Rebuttal Discussion

Dear Reviewer,

Thank you for your constructive feedback. We have carefully addressed your concerns in our rebuttal and provided additional experimental results. As the discussion period is approaching its end, we would greatly appreciate it if you could review our responses and let us know whether any concerns remain.

Thank you for your time and consideration.

Comments on the paper

1 comment

Dear Reviewers,

The authors have submitted their rebuttal to address your concerns. As the discussion deadline approaches, please review their response and share any remaining concerns or questions soon so the authors have enough time to respond.

Thank you, -AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.