acceptodds
Withdrawn from ICLR 2027

SatLen: Evaluating Semantic-to-Satellite Translation Across Nested Scales

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Image generators are increasingly asked to render the world from a map. We ask whether they respect a map's physical scale. We introduce SatLen, a benchmark that renders the same 100 scenes from 45 countries and territories as land-cover maps at four ground footprints, from to  km per side, at a fixed input resolution. Across 21 generator configurations and 8,400 generations, we find a coverage paradox: as the footprint widens, every configuration becomes more similar to the real image under CLIP while matching the requested layout less closely. After removing the overlap that correct class proportions earn by chance, spatial placement falls by , and three configurations fall to near-chance placement at  km. Pixel-level diagnostics that bypass the segmenter show where the failure lies. Generated edges still follow the map's boundaries, and do so more sharply at wide coverage, yet the land-cover identity of the enclosed regions is lost. A segmenter-free nearest-class-mean classifier confirms that this semantic loss is not a segmenter artifact. Models also treat scale as a texture setting: output textures are too smooth close up and gain fine detail far faster than real imagery as coverage grows. A prompt ablation without the extent sentence shows the same ramp, indicating that models read scale mainly from the map. Finally, a model's close-up and wide views of the same location rarely depict the same place. Current generators keep the lines but lose the land.

Primary area:
datasets and benchmarks
Keywords:
Datasets and benchmarks, Generative model evaluation, Remote sensing, Semantic-to-image generation, Multi-scale evaluation, Spatial controllability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_A5sYOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SatLen, a benchmark for evaluating the ability of image generation models to translate semantic land-cover maps into realistic satellite imagery across varying physical scales. The benchmark consists of 100 geographically diverse scenes from 45 countries and territories, evaluated at four spatial footprints ranging from 0.625 to 5 km using 21 open- and closed-weight model configurations. The authors introduce a comprehensive evaluation framework combining perceptual similarity, chance-corrected spatial placement, semantic fidelity, texture analysis, and cross-scale consistency. Their experiments reveal a coverage paradox, where increasing spatial coverage improves perceptual similarity to real satellite imagery while significantly degrading semantic layout fidelity, with chance-corrected spatial placement declining by 56%.

Critical strengths
  1. The problem formulation is meaning. Evaluating scale-dependent spatial fidelity is important for satellite image generation, particularly when using synthetic imagery for downstream geospatial applications.
  2. The divergence between visual similarity and layout fidelity across scales is compelling. It illustrates why perceptual metrics alone are insufficient for evaluating spatially controlled generation.
Critical weaknesses
  1. The paper only compares general purpose image generation and editing models. I think this is not appropriate for the benchmark. The paper should compare with spatially-conditioned models such as those using Control-Nets. This way the evaluation would be fairer since segmentation masks can be provided to the models for spatial conditioning. The paper should include models like GLIGEN and InstanceDiffusion that can explicitly provide spatial prompts to condition the generation.
  2. The benchmark should compare domain specific satellite image generative models. Currently the models compared in the paper are general purpose image generative models. Furthermore, the paper should atleast try to finetune a control-net based model and see if it can close the gap of performance on the benchmark dataset.
  3. The authors evaluate cross-scale consistency using low-resolution correlation between independently generated images. However, two semantically correct generations can have different textures, colors, and local appearance. The authors should use some sort of geometry and/or object aware correspondence scores as well.
  4. The authors provide semantic maps alongside detailed text descriptions, but do not systematically investigate their relative contributions to generation quality and spatial fidelity. The authors should compare map-only vs map+text conditioning of the models can compare how much text contributes to the task.
Questions for authors
  1. How does SegEarth-OV-3 perform on the real reference images at each evaluation resolution? How does the quality of the generated segmentation masks and the choice of the model affect the benchmark?
Reviewer_kiGjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SatLen, a benchmark for testing whether map-conditioned satellite image generation actually understands physical scale. Its core observation is that the same location viewed at 0.625 km and at 5 km must remain the same place; therefore, reporting image realism at a single scale cannot reveal whether a model is genuinely placing land cover according to the map.

Critical strengths
  1. The paper contributes a systematic dataset and benchmark to the community.
  2. The "coverage paradox" finding is interesting and consequential. Widening the map footprint makes the generated images score better on CLIP-I / PSNR / FSIM and look more like real satellite imagery, while the model in fact places land cover less faithfully — so an evaluator relying on appearance scores alone would conclude that the model has improved when it has not.
Critical weaknesses
  1. Remote-sensing-specific generative baselines are missing, which weakens the extrapolation of the domain-specific claim about map-conditioned satellite generation. All 21 configurations in Table 1 are general-purpose multimodal generation/editing models; none of the remote-sensing-specific generators discussed in §2.2 (DiffusionSat, GeoSynth, CRS-Diff, and map-conditioned synthesis) appear in the evaluation.
  2. Sampling stochasticity is not quantified. Only one image is generated per (scene, scale, model) cell. Different prompts also produce different results, and this variation is likewise not quantified in the main text.
  3. The comparison across coverage levels is not entirely fair, because task difficulty differs. As coverage widens, many fine details stop mattering. For a fuzzy generation task this may itself explain the improvement in CLIP: viewed up close, a tree has leaves and other fine structure, whereas from afar it is just a green dot — and predicting the distribution of leaves is of course harder than predicting a green dot. Under this reading, the reported degradation of layout fidelity with coverage cannot be attributed entirely to the model.
  4. As a dataset paper, it provides no dataset release statement.
Questions for authors
  1. Why were remote-sensing-specific generative models such as DiffusionSat, GeoSynth, and CRS-Diff not included in the evaluation set?
  2. Only one image is generated per (scene, scale, model). Could you run a repeated-generation experiment on a subset — for example three samples per cell — and report the within-instance standard deviation of CLIP-I and FG-mIoU? Could you also vary the prompt?
  3. Can you separate task difficulty from model capability in the coverage paradox? For instance, is there a control that holds difficulty fixed while changing coverage, or a chance-level reference that itself shifts with coverage?
  4. Could you state the plan for releasing the dataset?
Reviewer_GjWwOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a benchmark for generating satellite imagery from semantic land-cover maps at different physical scales. It constructs four coverage levels for 100 scenes and evaluates 21 model configurations. the motivation is that existing model mostly conditions on map related input, but few papaer actually tests whether the generation model understand the semantic, beyond the appearance. The main finding is that larger coverage improves CLIP similarity to real imagery, but at the same time it finds reducing agreement with the conditioning map. Additional analyses examine spatial placement, boundary alignment, texture frequency, and consistency between independently generated views of the same location.

Critical strengths
  1. The problem itself is one interesting quesiton with good motivation. A satellite generator should preserve the requested geography across different ground footprints. Using crops from the same source mosaic helps control geographic and acquisition differences.

  2. The breadth of the model in test is comprehensive, which seems to cover the majority of the SOTA model available in closed and open weights. The experiment protocol is simple but straight-forward to follow.

Critical weaknesses
  1. I am confused on why we would expect the model to work across scales. One probable cause is perhaps those model are not using multi-scale training recipe to train the model. If we take one open model and compare training at a single footprint against training across multiple footprints, keeping the data budget comparable and testing on held-out geographic regions. If multi-scale training largely fixes the problem, the result points toward missing training coverage or objectives, instead of the current conclusion of the paper. If substantial failures remain, the paper has a stronger basis for investigating representation or conditioning limitations.

  2. Another concern is that perhaps the resolution of image condition plays a stronger role for the generated image quality. An informative ablation would be to keep the physical footprint fixed and vary the conditioning resolution, with suitable output resolution.

Questions for authors

as weakness above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.