acceptodds
Under review as a conference paper at ICLR 2027

Bifröst: Multi-Sensor Diffusion Model for Cross-View Panoramic Synthesis

Abstract

Synthesizing ground-level 360° panoramas from overhead imagery typically assumes access to sub-meter aerial photography (0.3m/px). In practice, this restricts synthesis to densely surveyed metropolitan regions. Moving to globally available 10m/px Sentinel-2 multi-spectral data avoids this coverage bottleneck, but the low-resolution conditioning yields severe structural ambiguity. To overcome this, we introduce Bifröst, a diffusion framework designed to run inference strictly on open-access satellite data (10m/px Sentinel-2 and AlphaEarth embeddings) while retaining high-resolution geometric detail. Rather than feeding high-resolution imagery into the generator, we use it solely as training-time supervision via a frozen Perspective-Inversion Critic (PIC) that enforces structural consistency. This gives us the novel ability to generate street-view using only highly available low-resolution information. To handle cross-view spatial transformation, we align overhead polar coordinates directly with equirectangular spherical grids inside cross-attention layers. On a benchmark of 560k spatially disjoint locations, Bifröst eliminates the blur typical of low-resolution baselines, cutting relative depth error (AbsRel) from 0.138 to 0.106, matching models conditioned on true sub-meter aerial inputs (AbsRel 0.101) while improving FID by 77.86% over prior methods. On the public CVUSA dataset, Bifröst achieves an improvement of 68.4% FID over prior methods. These results demonstrate that the proposed approach effectively overcomes the need for high resolution information for cross-view generation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
cross-view diffusion, satellite to street-view, cycle consistency

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_QtLgOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper generates ground-level panoramas from low-resolution Sentinel-2 observations, AlphaEarth features, and geographic and temporal metadata. It introduces directional conditioning and a frozen Perspective-Inversion Critic (PIC), which maps generated panoramas back to high-resolution overhead imagery to provide structural supervision during training. Evaluation includes a geographically partitioned dataset, CVUSA, and component ablations. The internal results suggest that PIC improves perceptual and predicted-depth consistency. However, I have serious concerns about the numerical credibility and comparability of the reported FID advantage. Unresolved memorization concerns, incomplete panoramic calibration, and insufficient comparisons with simpler uses of high-resolution supervision further limit the evidence for the claimed contribution.

Critical strengths
  1. The PIC ablation provides direct evidence that the added supervision improves the current generator. With the same Sentinel-2 and AlphaEarth inputs, predicted-depth AbsRel decreases from 0.1385 to 0.1057 and LPIPS from 0.5362 to 0.5038. These controlled results are more informative about PIC than the overall comparison with earlier systems and support the usefulness of its training signal.

  2. Directional conditioning has measurable benefits in a component comparison. In Table 6, removing coordinate encoding from the Sentinel-2 plus high-resolution configuration increases AbsRel from 0.1007 to 0.1126 and LPIPS from 0.4912 to 0.5059. This provides empirical support for incorporating directional information when connecting overhead observations to panoramic outputs.

Critical weaknesses
  1. The central performance and generalization claims are not supported by sufficiently verifiable evidence. CVUSA FID is reported as 6.73, compared with 21.30 for CS2S. Page 22 states that baseline FID, PSNR, and SSIM values were taken from other publications, but the paper does not establish equivalent evaluation implementations and preprocessing. The cited SatDreamer360 reports FID 17.00 for single-image generation on CVUSA and is absent from the comparison; Bifröst reports worse LPIPS, PSNR, and SSIM despite its much lower FID. Different metric rankings are possible, but this unusually large distributional advantage requires a reproducible, common evaluation. The reported panorama resolutions also differ, making the crop and resize pipeline consequential. I have serious concerns about accepting these FID values as evidence of the claimed improvement without that verification. The stated inability to release the dataset and code makes a reproducible evaluation package particularly important. Figures 14–15 additionally show striking similarities between generated images and their targets in capture-vehicle appearance, with similar cloud structure in Figure 14(g). Such details are not determined by the coarse overhead observations. These examples raise unresolved overfitting or memorization concerns that require sample provenance, training-neighbor searches, and capture-sequence deduplication before the generalization claim is convincing.

  2. The specific methodological advance and geometric validity remain insufficiently established. The key question is whether PIC transfers high-resolution structural supervision more effectively than simpler alternatives. The paper lacks comparisons with a teacher-distillation or auxiliary overhead-reconstruction objective using the same backbone, supervision, and training budget. On the internal benchmark, the model without PIC already obtains FID 4.4003, while PIC improves it to 3.8707; most of the gap to earlier systems therefore predates the proposed critic. The geometric evidence also relies primarily on predicted depth and segmentation, leaving the accuracy of location-specific structure uncertain. Moreover, CVUSA panoramas are vertically cropped, while the method uses elevation-angle encoding. The vertical field of view, crop bounds, and pixel-to-ray calibration are needed to assess whether these directions are correct. Please report this calibration and validate perspective reprojection within the retained viewing range. A cropped panorama should not be treated as a complete spherical observation. These omissions affect both the interpretation of the geometric gains and the usability of the generated output.

Questions for authors
  1. FID verification: What implementation, weights, feature layer, sample set, crop, resize, and sampling procedure produced the reported FID values? Under a common CVUSA evaluation pipeline, how do Bifröst, CS2S, and SatDreamer360 compare, with their input differences documented? Recomputable results would directly affect my assessment of the main performance claim.

  2. Generalization verification: Which splits contain the examples in Figures 14–15, and what inputs and initialization were used to generate them? Do training-neighbor searches, capture-sequence deduplication, and evaluation with the capture vehicle masked support the interpretation that these results generalize to unseen locations?

8 comments
AuthorsOpenReview ↗

Response 1

Weakness 1

  1. CVUSA Experiment Settings and Evaluation Protocols: We thank the reviewer for highlighting the protocol for FID calculation. To this end, we retrained the baselines on the CVUSA dataset, we focused on S2S, Sat2Density and Sat2Density++ as suggested by the reviewer. The results are in the table below (Table 1 in updated paper), with details of the experiment settings and evals described below. We are in the process of rerunning CrossViewDiff, CS2S and SatDreamer360 on CVUSA and will report the results as soon as it’s done.
MethodFID ↓LPIPS ↓PSNR ↑SSIM ↑
Re-evaluated (HR input)Sat2StrPanorama57.570.5413.800.40
Sat2Density45.630.5313.440.36
Sat2Density++ [This is still running]69.230.6312.470.40
Reported in prior work (HR input)ControlNet22.550.5112.830.32
CS2S21.300.4413.670.38
CrossViewDiff23.670.4412.000.37
Bifröst (LR input)S w/ cyclic7.150.6211.690.32
A w/ cyclic6.630.5912.140.33
S+A w/ cyclic6.730.5912.170.33
  1. CVUSA Experiment Settings (Appendix E.1 Table 6 in updated paper): | | Sat2StrPanorama | Sat2Density | Sat2Density++ | Bifröst (S / A / S+A) | |---|---|---|---|---| | Test-time input | RGB aerial 256², zoom 19 | RGB aerial 256², zoom 19 | RGB aerial 256², zoom 19 | Sentinel-2 and/or AlphaEarth (no HR) | | Model | 64 height planes, concentric-cylinder projection | 65 density planes, 100 ray samples | 256²×96 tri-plane, 64×256 volume rendering, 4× StyleGAN2 SR | Pixel-space U-ViT; frozen critic in Stage 2 | | Losses | PatchGAN + VGG perceptual | Multi-scale panorama + perspective discriminators | Street + satellite discriminators, R1 | Sigmoid-weighted diffusion loss (+ PIC loss in Stage 2) | | Optimizer, LR | Adam, 2e-4 | Adam, 5e-5 | Adam, 2.5e-3 (generator) | AdamW, 1e-4 / 1e-5 | | Batch size | 16 | 16 | 32 | 512 / 256 | | Training | 35 epochs (77.7k steps) | 30 epochs (66.6k steps) | 19k [Still Running] | 41k + 14k steps |

  2. FID & LPIPS Calculation Protocol (Appendix E.1 in updated paper).

We compute FID using the penultimate layer activations of a pre-trained Inception-v3 to quantify global visual manifold coverage. All methods use the same implementation over the full evaluation split, never per batch: RGB panoramas in [0, 1] are scaled to [0, 255], bilinearly resized from 256 × 1024 to 299 × 299 (TF- 1.x convention, align corners=False), normalized as (x − 128)/128, and passed through the standard inception-2015-12-05 weights to accumulate 2048-d pool3 statistics (μ, Σ), from which we compute the exact Frechet distance for the test sets. To assess local pairwise perceptual fidelity robust to spatial phase stochasticity, we compute LPIPS using a pre-trained VGG backbone (for our 556k benchmark) and AlexNet backbone for CVUSA.

  1. Concerns on reproducibility (Appendix D, E, F):

Thank you for highlighting this. Since we are unable to release the proprietary data, we have instead included evals and ablations on the public CVUSA dataset, by retraining the baselines and our model on the publicly available data splits. In addition, we have included detailed experimental settings to reproduce the implementation as well, in Appendix sections D, E and F.

AuthorsOpenReview ↗

Response 1 - Continued

  1. Checking for Overfitting and Memorization (Appendix Section G in updated paper):

Thank you for highlighting the specific example of cloud structure similarities. We would like to draw the re- viewer’s attention to images on Fig 14 (a-e) where the model does not replicate the cloud structure. We believe that the model is most likely combining cloud structures it has learnt over the dataset, correlating it to the low resolution Sentinel-2 images and AlphaEarth embeddings where possible. Since AlphaEarth embeddings are trained on Sentinel-1 (which is good at penetrating through clouds) and Sentinel-2 (which has explicit bands for cloud cover identification), it is possible that the model learns to generate cloud structures.

To directly test for training-set memorization and capture-sequence leakage on CVUSA (35,532 training / 8,884 held-out validation pairs), we conducted an exhaustive nearest-neighbor audit across all three cyclic LR→SV conditioning configurations, EFM, Sentinel-2 multispectral, and EFM + multispectral, evaluating 16 uniformly stratified validation generations per model (48 queries to- tal) against all 35,532 training panoramas using L2-normalized 2048-D InceptionV3 embeddings. Visual inspection of Generated / Closest Training Neighbor / Validation GT triplets, including the global highest-similarity pairs across all 48 evaluations confirms zero near-duplicate copying: road curvature, lane markings, buildings, and tree/terrain silhouettes differ substantially from every retrieved training panorama. If predictions relied on memorizing adjacent frames from overlapping Street View drive sequences, nearest-neighbor retrieval over all 35,532 training panoramas would surface co-located sequence neighbors. Instead, retrieved neighbors are geographically dispersed across the continental US, and for identical validation queries evaluated across the three models, each model retrieves a different training panorama separated by 198–3,793 km across different states, confirming conditioning-driven scene synthesis rather than capture-sequence memorization. We have included this analysis in the updated draft with example visualizations (Figure 27-29).

AuthorsOpenReview ↗

Response 2

Weakness 2

We agree that most of the gap to earlier systems comes before the critic: our low-resolution model without PIC already reaches FID 4.40, and our HR-conditioned model reaches 3.93. The revised Section 4.2 states this. The contribution of PIC is measured by the matched comparison in Table 2 (S+A: FID 4.40 → 3.87, AbsRel 0.1385 → 0.1057).

To test simpler uses of the supervision, we ran two controls on CVUSA from the same Stage-1 checkpoint and budget (100 epochs, 13.8k steps, BS 256, LR 1e-5).

Table R2. Critic controls on CVUSA (paper: Table 9). Each cell lists S / A / S+A.

VariantFID ↓LPIPS ↓PSNR ↑SSIM ↑
Matched-compute panorama LPIPS (no critic)13.14 / 12.61 / 12.100.61 / 0.59 / 0.5911.84 / 12.26 / 12.220.34 / 0.35 / 0.35
Blurred-target critic (σ = 8, ~3 m/px)7.78 / 7.32 / 7.480.61 / 0.60 / 0.6011.91 / 12.29 / 12.330.32 / 0.34 / 0.34
Sharp-target critic (Bifröst)7.15 / 6.63 / 6.730.62 / 0.59 / 0.5911.69 / 12.14 / 12.170.32 / 0.33 / 0.33

Under these controls, the sharp-target critic gives the lowest FID for every input; pairwise metrics are similar, and PSNR and SSIM slightly favour the controls. We have not yet compared against teacher distillation or an auxiliary overhead-reconstruction head with the same backbone and budget, and we list this explicitly as future work (App. H.5).

Geometric evidence. Depth and segmentation are foundation-model proxies, so we present these results as improved structural consistency rather than verified location-specific accuracy (App. H.4).

Panorama calibration (Sec. 3.3, App. B.2)

Neither dataset is modelled as a full sphere. Both use a 90° vertical field of view (Shi et al., 2022; Dai et al., 2023), and perspective views are reprojected only within this retained band.

QuantityValue
Vertical cropCentral 50% of 512×1024 (rows [128, 384))
Vertical field of view90°, elevation φ ∈ [−45°, +45°]
Azimuth of column cθ(c) = 2π(c − W/2)/W ∈ [−π, π), north-centred
Elevation of row rφ(r) = π/4 − (π/2)/(H − 1) · r
Perspective reprojectionv = ½(1 − 4φ/π)H − ½, with 360° azimuthal wrap-around

References.

  • Ming Dai, Enhui Zheng, Zhenhua Feng, Lei Qi, Jiedong Zhuang, and Wankou Yang. Vision-based uav self-positioning in low-altitude urban environments. IEEE Transactions on Image Processing, 33:493–508, 2023

  • Y. Shi, D. Campbell, X. Yu, and H. Li. Geometry-guided street-view panorama synthesis from satellite imagery. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 44 (12):10009–10022, 2022

Question 1

FID Implementation Details and Baselines (Appendix section E.1 and E.2 in paper): Thank you for raising this. We have retrained our model and the baselines on CVUSA and reported the updated numbers in Table 1. Please refer to our detailed response to W1 above on the implementation and settings.

Question 2

Neighbourhood searches and concerns on overfitting (Appendix Section G): Thank you for raising this. We have examined the potential for overfitting and elaborated on the experiment settings and the evidence against it in the response to W1 above

We hope these changes address the reviewer's concerns, and we would be grateful to know whether any remain.

Reviewer_6PHAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces Bifröst, a multi-sensor diffusion framework for synthesizing north-aligned ground-level panoramas from low-resolution satellite observations. At inference, the model uses Sentinel-2 multispectral imagery, AlphaEarth embeddings, and geo-temporal metadata. During training, a frozen Perspective-Inversion Critic (PIC) predicts high-resolution aerial imagery from a noisy aerial input, conditioned on the generated panorama and low-resolution context. Its prediction supplies a cyclic loss combining weighted L1 and MS-SSIM against the aerial target. The generator also uses coordinate-aware cross-attention to connect overhead polar coordinates with panoramic viewing directions, together with dual-stream FiLM conditioning. The main benchmark contains 556k paired instances, split into 456k training, 50k validation, and 50k test pairs, with geographically separated S2 cells. The paper also evaluates on CVUSA. In Table 2, adding cyclic supervision to the S+A configuration reduces AbsRel from 0.1385 to 0.1057 and FID from 4.4003 to 3.8707. These results support the usefulness of training-time high-resolution supervision under low-resolution inference constraints. The appendices provide theoretical arguments, implementation details, additional ablations, semantic evaluation, and qualitative failure cases, while acknowledging temporal mismatch and dependence on foundation-model evaluation proxies.

Critical strengths

1. A meaningful inference-time constraint

The paper addresses a practically relevant limitation of cross-view synthesis by separating the information available during training from the inputs required at deployment. Sections 3.1 and 3.4 clearly specify that high-resolution aerial imagery supervises training but is not supplied to the generator at inference. This makes the operational contribution concrete. The agricultural and rural focus in Appendix F.1 also provides a coherent setting for studying this constraint, where the paper seeks to generate useful ground-level scene hypotheses from small low-resolution overhead patches.

2. A well-motivated geometric conditioning design

The coordinate-aware attention mechanism has an intuitive connection to the task geometry. Section 3.3 augments overhead tokens with radial distance and azimuthal harmonics, and panorama queries with angular coordinates. The shared sine and cosine representation provides a direct way to express relative viewing direction. Appendix B makes this motivation more explicit through its relative-phase calculation. Table 6 also provides empirical support within the S+H setting, where removing coordinates increases AbsRel from 0.1007 to 0.1126. This is useful evidence for the coordinate representation as an inductive bias.

3. Consistent improvements from cyclic supervision

Table 2 includes low-resolution configurations both with and without cyclic supervision, allowing a more informative comparison than the headline gains over older methods alone. For S+A, cyclic training improves AbsRel, LPIPS, SSIM, and both GLCM contrast and entropy errors. The S-only and A-only configurations also improve in FID and AbsRel. The resulting S+A AbsRel of 0.1057 approaches the S+H reference value of 0.1007 while retaining low-resolution inference inputs. This is the paper's strongest direct evidence that the additional training supervision improves the proposed system.

4. Useful data controls and explicit discussion of limitations

Section 4.1 and Appendix F.1 describe geographic splitting by S2 Level-8 cells, a minimum separation between capture locations, and a seven-day acquisition window for the main dataset's aerial, Sentinel-2, and street-view observations. The evaluation combines distributional, pairwise, texture, depth-proxy, and semantic measures. Appendices F.2 and H explicitly discuss CVUSA's temporal mismatch, unobservable vertical structures, and biases in the evaluation models. The qualitative successes and failure cases in Figures 8 to 10 further help readers distinguish plausible synthesis from faithful recovery of every local scene detail.

Critical weaknesses

1. The link between the cyclic loss and the variational interpretation needs further explanation

Section 3.5 interprets the PIC loss as a Gibbs-based surrogate for information maximization. Appendix A acknowledges that MS-SSIM is not strictly shift-invariant, describes local contrast preservation as an asymptotic property, and invokes critic gating to motivate its practical use during training. However, the step from matched local contrast to a residual-only energy and an approximately constant partition function in Eqs. 9 and 10 still needs clarification. Matching contrast for a target-prediction pair does not directly establish the same simplification throughout the normalization integral. Moreover, the gate in Appendix D.4 selects diffusion noise levels, which does not by itself establish convergence of the generator. The authors should clarify the assumptions required for this approximation and explain how gating supports them during training.

2. Direct evidence for location-specific geometric fidelity is limited

The paper evaluates both visual realism and structural consistency, and Table 2 shows that S+A with cyclic supervision reaches an AbsRel of 0.1057, close to the S+H reference value of 0.1007. Appendix H.3 identifies the DepthAnythingV2 and Mask2Former measurements as relative proxies rather than absolute physical reconstruction accuracy. Table 5 adds a semantic check on 500 validation images (Appendix E.2), and Figure 8 illustrates recovered road layouts alongside local failures. These results support structural improvement, but do not establish accurate recovery of fine geometry at individual locations. The conclusion nevertheless states that the work proves high-frequency terrestrial geometry can be recovered from low-resolution inputs. The authors should align the conclusion with Appendix H.3 and distinguish plausible fine-detail synthesis from verified location-specific recovery.

3. The cyclic-supervision implementation and ablation need clearer reporting

Table 2 compares low-resolution models with and without cyclic supervision. The main text and appendices describe this supervision inconsistently. The text following Eq. 5 defines a target-derived Sobel mask, whereas Appendix D.4 describes weighting pixels within eight pixels of the aerial tile perimeter. Appendix I also describes a PIC-removal change from AbsRel 0.101 to 0.138, while the explicit S+A comparison in Table 2 is 0.1057 versus 0.1385, and Table 6 uses S+H as its architectural reference. These discrepancies make the loss implementation and the configurations compared in the ablation unclear. The authors should specify the masking rule used in the experiments and identify the configurations behind the Appendix I comparison so that the implementation and reported evidence can be followed consistently.

Questions for authors
  1. Appendix A discusses asymptotic contrast preservation and critic gating. Under what additional assumptions does contrast matching justify the residual-only energy and approximately constant partition function in Eqs. 9 and 10? How does the diffusion-timestep gate in Appendix D.4 support those assumptions before generator convergence?

  2. How should the conclusion’s claim of recovering high-frequency terrestrial geometry be interpreted in light of the relative-proxy qualification in Appendix H.3? Please clarify which aspects are supported by Table 2, Table 5, and the qualitative examples, and whether the intended claim is plausible structural synthesis or accurate recovery of fine details at individual locations.

  3. How are the target-derived Sobel mask after Eq. 5 and the perimeter weighting in Appendix D.4 related, and what exact weighting rule was used for the cyclic loss? Which configurations produce the 0.101-to-0.138 AbsRel comparison in Appendix I, and how does that comparison relate to the S+A results in Table 2 and the S+H reference in Table 6?

4 comments
AuthorsOpenReview ↗

Response 1

Weakness 1 and Question 1

We agree that matching contrast for a target–prediction pair does not establish a prediction-independent partition function for our edge-weighted L1 + MS-SSIM objective: the Sobel weights depend on the target, and MS-SSIM depends on local image statistics. We have removed the approximate partition-function and mutual-information-bound claims (former Eqs. 9–10) from Section 3.5 and Appendix A.

Appendix A is now a short section, "Remarks on the Structural Consistency Objective". It states that the loss is chosen for its practical behaviour rather than derived from a probabilistic model. With fixed weights, the L1 term alone would correspond to maximum likelihood under a Laplace model; because our weights depend on the target and MS-SSIM depends on local image statistics, we treat the full objective as a structural similarity measure, not a likelihood. The revised paper presents PIC as a frozen inverse-view consistency objective. L1 + MS-SSIM is a standard image-restoration choice that preserves edges and local contrast better than L2, and its value is supported by the matched with/without-PIC comparison in Table 2 and the CVUSA critic controls (paper Table 9).

The timestep gate (App. D.4) excludes the highest-noise generator estimates throughout training. It does not establish convergence or justify the former theoretical approximation; the revised Appendix D.4 states this explicitly.

AuthorsOpenReview ↗

Response 2

Weakness 2 and Question 2

We agree that "proving" recovery of high-frequency geometry exceeded the evidence. Our central claim is narrower: PIC uses high-resolution aerial supervision during training to improve the structural consistency of panoramas generated from low-resolution inputs, without high-resolution imagery at inference.

Matched PIC ablation (paper: Table 2). The two S+A rows share backbone, inputs and inference cost; only the cyclic loss differs. S+H receives 0.3 m/px imagery at inference and is a reference, not an ablation.

ConfigurationFID ↓LPIPS ↓SSIM ↑GLCM contrast ↓AbsRel ↓δ1 ↑
S+A, no PIC4.40030.53620.43300.25140.13850.8830
S+A, with PIC (Bifröst)3.87070.50380.44780.20690.10570.9086
S+H (HR at inference; reference)3.92740.49120.45690.18800.10070.9079

Table R2. Semantic agreement between Mask2Former segmentations of generated and real panoramas (paper: Table 7; 500 validation images).

MethodmIoU95% CISkyRoadCropsFoliage
Bifröst (S+A), LR input0.480±0.0100.8140.5350.5380.288
Bifröst (A), LR input0.458±0.0100.8020.4860.5250.274
Bifröst (S), LR input0.445±0.0100.7930.4910.4870.252
ControlNet, HR input0.453±0.0130.8080.5020.5030.309
CrossViewDiff, HR input0.360±0.0120.7020.3400.4270.150
  • Table 2: adding PIC to S+A reduces predicted-depth AbsRel from 0.1385 to 0.1057 (23.7%), closing about 87% of the gap to S+H: (0.1385 − 0.1057) / (0.1385 − 0.1007) = 86.8%. GLCM contrast error falls from 0.2514 to 0.2069.
  • Table 7: mIoU 0.480 (roads 0.535, crops 0.538). This measures agreement between predicted layouts, not independently annotated geometry.
  • Fig. 8: correspondence in road layouts and field boundaries, alongside failures in vegetation and foreground detail. These results support improved reference-based structural consistency and plausible detail. They do not establish independently verified physical accuracy of fine details at individual locations: the low-resolution observations constrain the layout but do not determine every detail. The intended claim is therefore plausible structural synthesis, and the Abstract, contributions and Conclusion now say so, consistent with the proxy-metric qualification in Appendix H.4.
AuthorsOpenReview ↗

Response 3

Weakness 3 and Question 3

Thank you for identifying these reporting inconsistencies. We have corrected the main text and appendix to use one definition consistently.

Edge weighting. The weights depend only on target-image gradients, not on pixel coordinates. From the unblurred ground-truth aerial image, a depthwise Sobel filter produces a per-image min–max normalized edge mask M ∈ [0, 1], which the hyperparameter w_edge = 2 maps affinely to the spatial weight Ω = 1 + (w_edge − 1)M, so Ω ∈ [1, 2]. Ω multiplies only the pixel-wise L1 error between the critic's prediction and the target; this term is then blended with the unweighted MS-SSIM dissimilarity using weights w_ℓ1 = w_ssim = 0.5. The perimeter weighting described in the original Appendix D.4 was incorrect and has been replaced by this definition in Section 3.5 and Appendix D.4. The critic loss is scaled by the timestep-gated weight λ_critic, and only the generator's parameters are updated; the critic remains frozen.

PIC ablation. The comparison in Appendix I incorrectly combined a high-resolution reference with a low-resolution model without PIC. The correct matched comparison is in Table 2 (Table R1):

  • S+A without PIC: AbsRel 0.1385, FID 4.4003
  • S+A with PIC: AbsRel 0.1057, FID 3.8707 Removing PIC from the S+A configuration therefore increases AbsRel from 0.1057 to 0.1385. The S+H configuration (AbsRel 0.1007, FID 3.9274) is a separate reference that receives high-resolution imagery at inference. Table 8 (formerly Table 6) evaluates architectural components within the S+H configuration and does not report PIC removal from S+A. We have corrected Appendix I to separate the matched low-resolution PIC ablation from the high-resolution architectural ablations. The evidence for PIC's contribution is the comparison between S+A with and without PIC in Table 2.

Cyclic loss as used in all experiments

ComponentDefinition / value
Edge mask MPer-image min–max normalized magnitude of a depthwise Sobel filter on the unblurred target; M ∈ [0, 1]
Weight map ΩΩ = 1 + (w_edge − 1)M with w_edge = 2, so Ω ∈ [1, 2]; multiplies the L1 term only
Critic loss0.5 · mean(Ω ⊙ |ĉ − c|) + 0.5 · (1 − MS-SSIM(ĉ, c)); MS-SSIM is unweighted
Total objectiveL_total = L_G + λ_critic(t) · L_critic, with λ_critic = 0.1 if the generator batch timestep ≤ 0.6 and 0 otherwise
Critic timestept_critic = clip(t̄_gen + U(−0.1, 0.1), 0.2, 0.9)
Parameters updatedGenerator only; the critic is frozen

We hope these changes address the reviewer's concerns, and we would be grateful to know whether any remain.

We believe we have thoroughly addressed all the points raised. Should the reviewer find our explanations satisfactory, we kindly ask that they consider increasing their score.

Reviewer_di3vOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper generates ground-level 360° panoramas from overhead data without using high-resolution aerial imagery at test time. The generator is a U-ViT diffusion model conditioned only on a small Sentinel-2 patch, AlphaEarth embeddings, and location and date metadata. Inside cross-attention, overhead tokens carry polar coordinates and panorama queries carry spherical coordinates, so that rotation in the overhead view lines up with horizontal shifts in the panorama. High-resolution aerial imagery is used only during training, through a frozen Perspective-Inversion Critic. This critic is a diffusion model that maps a panorama back to the aerial view. Its reconstruction error against the real aerial tile is added to the generator's objective. Experiments use a private set of Street View panoramas from agricultural regions and the public CVUSA dataset. The model is compared against cross-view diffusion baselines and against the same architecture with and without high-resolution input. The authors report that the critic closes much of the gap to the high-resolution models on a depth-based proxy and gives the best FID.

Critical strengths
  1. The problem setting is practical, and the training-only use of high-resolution data is a clean idea. Sub-meter aerial imagery is unavailable in much of the world, so a model that runs on open Sentinel-2 and AlphaEarth inputs has real value. Using the high-resolution tiles only through a frozen inverse critic keeps inference cheap and avoids the instability of adversarial training.

  2. The critic gives consistent gains over matched low-resolution baselines. Adding the critic improves the depth proxy, texture statistics and perceptual metrics for all three low-resolution input combinations. With the critic, the full model comes close to the high-resolution version on depth. The dataset is large and split by disjoint S2 cells, which makes the comparison between these internal variants fairly convincing.

Critical weaknesses
  1. The backbone and training setup are not specified, so the fairness of the comparison is hard to judge. The paper says Bifröst follows the U-ViT architecture of simple diffusion. It does not report the model size, whether the model is trained from scratch or initialized from a pretrained checkpoint, or where that checkpoint comes from. Appendix D.2 mentions fine-tuning a backbone pretrained on 256 by 256 images, with no further detail. The baselines are retrained from a Stable Diffusion checkpoint, but the version, resolution, number of training steps and batch size are not given. This matters because the authors' own model with high-resolution input reaches an FID around 4, while CrossViewDiff with the same input is above 17. Most of the gap over prior work therefore seems to come from the backbone and training budget rather than from the low-resolution setting. Please state the base model, parameter count, initialization and full training configuration for Bifröst and for every retrained baseline. Without this, I find it hard to accept that the comparison is fair. On CVUSA, the baseline numbers are copied from earlier papers, so the FID protocol may differ. Bifröst is also worse than the best baseline on PSNR, SSIM and LPIPS there, and no high-resolution or no-critic version of Bifröst is reported.

  2. The set of compared methods is too narrow. Only ControlNet, CS2S and CrossViewDiff are evaluated. The related work discusses several directly relevant methods with public code that are not compared, including Sat2Density, Sat2Density++ and the geometry-guided method of Shi et al. These methods address the same satellite-to-panorama task with explicit geometry. The paper argues that explicit geometry breaks down in rural scenes, but never tests this claim against them. For the low-resolution setting, a simple two-stage baseline is also missing: super-resolve the Sentinel-2 input and feed it to the high-resolution model. At least some of these should be included, on a public dataset such as CVUSA if not on the private benchmark.

  3. It is not shown that the gain comes from high-resolution information. The critic may simply help as an extra structural regularizer. Several controls are missing: a critic trained to predict a blurred or low-resolution aerial target, a critic that does not see the panorama, or a generic perceptual loss with matched compute. The authors themselves list this comparison as future work. It also seems that the critic receives a noised version of the real aerial tile as its diffusion input. If so, at low noise it can recover the target largely without the panorama. It is then unclear how much useful gradient flows back to the generator, especially since the critic's own reconstruction quality in Table 3 is modest. This is my main concern.

  4. The ablations do not cover the main model. The coordinate and metadata ablations in Table 6 are run on the high-resolution S+H model, not on the low-resolution model that the paper is about. The critic's design choices are not ablated: the energy function, the timestep coupling, the gating threshold and the loss weight. Appendix I also compares the AbsRel of the S+H model with the low-resolution model without the critic, which mixes two different settings. All results come from a single run.

  5. It is unclear how much of each panorama is tied to its location. The Sentinel-2 input is an 8 by 8 patch covering about 80 meters, while a panorama shows much farther than that. Pairwise metrics are low for every model. On a fairly homogeneous agricultural domain, FID mostly rewards typical-looking scenes. Two simple baselines would help: a model conditioned only on location and date, and nearest-neighbor retrieval of a training panorama by Sentinel-2 or AlphaEarth similarity. Without these, I cannot tell whether the model depicts the specific place or a plausible scene for the region. If the full model clearly beats both baselines, this concern would be largely resolved.

  6. The scope and claims are stated too broadly. The title and abstract present the method as global, but the data comes from cropland-dominated cells in North America and Europe. The paper says the model matches high-resolution models, yet LPIPS, SSIM and texture errors are still behind. The conclusion says the paper proves that high-frequency geometry can be recovered from low-resolution input, while what the model learns is a prior over plausible structure. The motivation around crop mapping and monitoring also needs care, since synthesized views cannot add observations beyond the inputs.

  7. The theory adds little. The Gibbs formulation amounts to an L1 plus MS-SSIM loss. Observation 1 relies on an informal contrast-matching assumption, so the link to a mutual information bound is not established. Proposition 1 only shows that a suitable parameterization exists for a single attention layer, and rotating a square grid by an arbitrary angle is not a permutation of tokens. I would shorten these parts and present the method as a well-designed auxiliary loss.

  8. The main results cannot be reproduced. The 556k dataset and the code will not be released. Most of the evidence comes from this private benchmark, and the public CVUSA comparison has the protocol issues described above.

Minor. Eq. 5 builds the edge weight from a Sobel mask with the factor one minus λedge\lambda_{edge}, which can give negative weights for the reported λedge\lambda_{edge} of 4. Appendix D.4 instead describes a weight on pixels near the tile border. The dataset size is given as both 560k and 556k, and the final AbsRel as both 0.106 and 0.105. Several terms, such as Variational Cyclic Internalization, are used without being defined in the method section.

References

Questions for authors
  1. Can you add controls that isolate the role of high-resolution information in the critic under the same training budget? For example, a critic trained on blurred or low-resolution aerial targets, or a critic without the panorama input. This is the comparison that would most change my view.
  2. Can you report a metadata-only model and a nearest-neighbor retrieval baseline on the main benchmark? This would show how much the satellite inputs contribute beyond regional priors.
  3. During generator training, what is the critic's diffusion input ztcz_{t_c}? If it is the noised ground-truth aerial tile, how large is the gradient with respect to the panorama at the noise levels used? How does the critic perform in Table 3 when it is given only the low-resolution context?

I am willing to raise my score if the authors can address my questions listed in Weakness and Questions sections.

9 comments
AuthorsOpenReview ↗

Response 1

We thank the reviewer for the constructive comments and suggestions, which has helped us improve our paper considerably. We have carefully answered each comment below.

Weakness 1

  1. Backbone and Training Configurations (Table 4 and 5 in updated paper): The details of the base model and training configurations are provided in the table below Bifrost uses ∼ 2× fewer total parameters than every retrained baseline (632.5M vs. 1.31B–1.68B) and models trained until eval metrics saturate. While baselines like ControlNet, CrossViewDiff and CS2S compress the panoramas by 8× using a VAE, Bifrost works directly in the pixel space, with spherical-polar coordinate cross-attention.
MethodBackbone (space)Params (total / trained)InitializationBSOverhead inputSteps, optimizer, LR
Bifröst Stage 1 (no critic)5-stage U-ViT (pixel, no VAE)632.5M / 632.5MSimple Diffusion, ImageNet-1k 256²512S2 8², A 8² (HR variants: 256²)300k, AdamW, 1e-4
Bifröst Stage 2 (cyclic, LR only)U-ViT + frozen critic632.5M / 632.5MStage-1 generator + critic256S2, A+178.3k, AdamW, 1e-5
ControlNet (dual-branch)LDM UNet + KL-f8 VAE1.68B / 1.59BStable Diffusion 1.5512HR 256² + S2 8²87.9k, AdamW, 1e-4
CrossViewDiffLDM UNet + VAE + CVTM1.32B / 1.23BStable Diffusion 1.5512HR 256² + 32×128 structure map87.9k, AdamW, 1e-4
CS2SLDM UNet + VAE + GCA1.32B / 1.23BStable Diffusion 1.5256HR 256²500k, AdamW, 1e-5
  1. CVUSA Experiment Settings and Evaluation Protocols: We thank the reviewer for highlighting the protocol for FID calculation. To this end, we retrained the baselines on the CVUSA dataset, we focused on S2S, Sat2Density and Sat2Density++ as suggested by the reviewer. The results are in the table below (Table 1 in updated paper), with details of the experiment settings and evals described below. We are in the process of rerunning CrossViewDiff, CS2S and SatDreamer360 on CVUSA and will report the results as soon as it’s done.
MethodFID ↓LPIPS ↓PSNR ↑SSIM ↑
Re-evaluated (HR input)Sat2StrPanorama57.570.5413.800.40
Sat2Density45.630.5313.440.36
Sat2Density++ [This is still running]69.230.6312.470.40
Reported in prior work (HR input)ControlNet22.550.5112.830.32
CS2S21.300.4413.670.38
CrossViewDiff23.670.4412.000.37
Bifröst (LR input)S w/ cyclic7.150.6211.690.32
A w/ cyclic6.630.5912.140.33
S+A w/ cyclic6.730.5912.170.33
  1. CVUSA Experiment Settings (Appendix E.1 Table 6 in updated paper): | | Sat2StrPanorama | Sat2Density | Sat2Density++ | Bifröst (S / A / S+A) | |---|---|---|---|---| | Test-time input | RGB aerial 256², zoom 19 | RGB aerial 256², zoom 19 | RGB aerial 256², zoom 19 | Sentinel-2 and/or AlphaEarth (no HR) | | Model | 64 height planes, concentric-cylinder projection | 65 density planes, 100 ray samples | 256²×96 tri-plane, 64×256 volume rendering, 4× StyleGAN2 SR | Pixel-space U-ViT; frozen critic in Stage 2 | | Losses | PatchGAN + VGG perceptual | Multi-scale panorama + perspective discriminators | Street + satellite discriminators, R1 | Sigmoid-weighted diffusion loss (+ PIC loss in Stage 2) | | Optimizer, LR | Adam, 2e-4 | Adam, 5e-5 | Adam, 2.5e-3 (generator) | AdamW, 1e-4 / 1e-5 | | Batch size | 16 | 16 | 32 | 512 / 256 | | Training | 35 epochs (77.7k steps) | 30 epochs (66.6k steps) | 19k [Still Running] | 41k + 14k steps |

  2. FID & LPIPS Calculation Protocol (Appendix E.1 in updated paper). We compute FID using the penultimate layer activations of a pre-trained Inception-v3 to quantify global visual manifold coverage. All methods use the same implementation over the full evaluation split, never per batch: RGB panoramas in [0, 1] are scaled to [0, 255], bilinearly resized from 256 × 1024 to 299 × 299 (TF- 1.x convention, align corners=False), normalized as (x − 128)/128, and passed through the standard inception-2015-12-05 weights to accumulate 2048-d pool3 statistics (μ, Σ), from which we compute the exact Frechet distance for the test sets. To assess local pairwise perceptual fidelity robust to spatial phase stochasticity, we compute LPIPS using a pre-trained VGG backbone (for our 556k benchmark) and AlexNet backbone for CVUSA.

AuthorsOpenReview ↗

Response 2

Weakness 2

  1. Comparison against other baselines:

We thank the reviewer for raising this. We have trained the listed baselines on the CVUSA dataset. The results are in the table given in response 1.

  1. Disadvantages of explicit geometry (Figure 13 in paper):

We observe that explicit geometry indeed breaks down in rural scenes, and highlight this qualitatively. For example, in Figure 13(d) of the paper, illustrating generations from CrossViewDiff (uses depth maps from DepthAnythingV2), we see that there is no clear artifact showing depth. The model ends up creating artifacts on the right end of the panorama. Likewise in Fig. 13(e) although the depth map identifies the bus on the road, the model ends up associating it with a building when none exists in the ground truth. We will examine this further in other baseline methods as well.

  1. Baseline: Super-resolved Sentinel2 input (Appendix I in updated paper):

We trained a separate diffusion super-resolver (same U-ViT backbone and schedule, 300k steps) that predicts the 256×256 aerial image from Sentinel-2, AlphaEarth and location/date only; street view is excluded to rule out target leakage. Its predictions reach 16.86 dB PSNR against the real aerial image. We then replace the real aerial image with the super-resolved one in the converged S+H+A and S+H generators, without retraining, on the same 5,120 validation panoramas with identical sampling noise.

GeneratorAerial input at test timeFID ↓LPIPS ↓SSIM ↑PSNR ↑
S+H+AReal 0.3 m/px image6.160.4740.46714.77
S+H+ASuper-resolved from S2 + A7.350.4960.45514.27
S+HReal 0.3 m/px image6.130.4750.46614.73
S+HSuper-resolved from S2 + A7.320.4980.45514.22
Metadata only, 150k stepsNone (lat/lon + date)8.740.5810.40412.22

Substituting the super-resolved image for the real one degrades every metric within the same panorama model; this paired comparison directly measures the effect of the aerial input. The super-resolved variant still has better FID and LPIPS than the lat/lon + date baseline, closing about 54% of the FID gap and 79% of the LPIPS gap to the real-HR reference.

AuthorsOpenReview ↗

Response 3

Weakness 3

  1. Gains from high resolution images (Appendix Table 9 in updated paper):

We ran both requested controls on CVUSA from the identical Stage-1 checkpoint, with the same 100-epoch schedule (13.8k steps, BS 256, LR 1e-5) and loss curriculum, following table shows CVUSA critic control ablations metrics reported as S / A / S + A

VariantFID ↓LPIPS ↓PSNR ↑SSIM ↑
Matched-compute panorama LPIPS (no critic)13.14 / 12.61 / 12.100.61 / 0.59 / 0.5911.84 / 12.26 / 12.220.34 / 0.35 / 0.35
Blurred-target critic (σ = 8, ~3 m/px)7.78 / 7.32 / 7.480.61 / 0.60 / 0.6011.91 / 12.29 / 12.330.32 / 0.34 / 0.34
Sharp-target critic (Bifröst)7.15 / 6.63 / 6.730.62 / 0.59 / 0.5911.69 / 12.14 / 12.170.32 / 0.33 / 0.33
  • Matched-compute perceptual loss: Replacing the critic with a VGG-LPIPS loss between the generator's clean estimate and the ground-truth panorama raises FID by 5.4–6.0 for every input.
  • Blurred-target critic: Training the critic and cyclic loss on Gaussian-blurred aerial targets (no exposure to sharp HR) raises FID by 0.63–0.75 for every input. (kernel 49×49, σ=8.0)

Under these controls, sharp high-resolution targets improve FID for every input. Pairwise metrics are similar across the three variants, and PSNR and SSIM slightly favour the controls, so the benefit of sharp targets shows mainly in distributional realism.

  1. Noised aerial input

Low critic noise levels are excluded by coupling the critic timestep to the generator's batch timestep, tcritic=clip(tˉgen+U(−0.1,0.1),0.2,0.9)t_{\text{critic}} = \text{clip}(\bar t_{\text{gen}} + \mathcal{U}(-0.1, 0.1), 0.2, 0.9) (Eq. 3), together with the gate that applies the critic only when tˉgen≤0.6\bar t_{\text{gen}} \le 0.6 (App. D.4). The critic therefore operates in tcritic∈[0.2,0.7]t_{\text{critic}} \in [0.2, 0.7]. On the critic's own 256×256 schedule, the noise level is σ≥0.31\sigma \ge 0.31 even at the 0.2 floor, and σ∈[0.59,0.81]\sigma \in [0.59, 0.81] for tcritic∈[0.4,0.6]t_{\text{critic}} \in [0.4, 0.6], around the typical batch timestep. In this range the fine aerial edges in the noised tile are heavily corrupted, which encourages the critic to use the generated panorama. Because the PIC loss reaches the generator only through the panorama x^\hat x, every effect of PIC on the generator is mediated by the panorama: adding PIC improves every low-resolution model (Table 2), and sharp targets give lower FID than blurred targets for every input as given in table above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.