acceptodds
Withdrawn from ICLR 2027

Seeing the Whole Scene: Scaling Vision-Language Models for Ultra-High-Resolution Earth Observation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Ultra-high-resolution (UHR) Earth observation imagery enables fine-grained reasoning over large areas, but rapidly growing visual sequences make full-attention remote sensing vision-language models (RS-VLMs) difficult to scale. Existing UHR RS-VLMs typically control cost through multiscale tiling, adaptive region selection, or token pruning and merging. These strategies bound computation by reducing the visual sequence, requiring increasingly aggressive visual compression as image size grows and potentially discarding fine spatial detail or query-relevant information. We introduce SourceScale-VL, a post-training-converted RS-VLM that preserves the complete post-encoder visual sequence and scales it through ring-style sequence parallelism, compressing visual interactions rather than visual content. SourceScale-VL replaces quadratic visual propagation with linear attention and introduces a magnitude-aware Referential Spectrum Residual to selectively retain Exact text-to-visual interactions where linear attention lacks sufficient source-response capacity. To preserve source-specific behavior during conversion, we further introduce Source-Factored On-Policy Distillation, which aligns source-wise output contributions on student-generated states. On the held-out conversion-evaluation split, SourceScale-VL reaches 57.46% accuracy, outperforming the same-budget RSR-Hybrid by 0.43 percentage points while reducing SCE from 0.063 to 0.041. On XLRS-Bench-lite and LRS-VQA, it achieves 54.64% and 33.58% aggregate accuracy, slightly surpassing Teacher (Exact) by 0.14 and 0.18 percentage points, respectively. With four A800 GPUs, SourceScale-VL supports longer visual sequences than sequence-parallel Exact attention while reducing peak memory by 13.8% and TTFT by 15.5%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
vision-language models, remote sensing, ultra-high-resolution vision, long-context multimodal modeling, sequence parallelism

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VCiLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on the problem of how to enable VLMs to efficiently process very high resolution remote sensing images, while existing methods reduce computational costs by discarding or merging visual tokens, or by selecting only specific image regions which often lead to the loss of important details. The proposed SourceScale-VL applies the linear attention and sequence parallelism associate with RSR method to evaluates the low-rank approximation error within different attention groups to determine which components require exact attention and which can be converted to linear attention. In addition with Source-Factored On-Policy Distillation, where the student model learns not only from the teacher’s final output but also from the relative contributions of different image regions to that output, thereby minimizing performance degradation resulting from the model transformation.

Critical strengths
  1. This paper addressed an important problem for remote sensing tasks that require the identification of small objects and the understanding of complex spatial relationships, which is well motivated.
  2. The proposed method shows a comprehensive design, integrating linear attention, RSR-guided selection between exact and linear attention, and source-factored on-policy distillation. The experimental results achieved certain performance on the target to reduce the cost while maintaining the performance from the teacher model.
Critical weaknesses
  1. The novelty of this paper needs more explaination to prove, current theoretical analysis and experiments are not sufficient to fully support the idea.
  2. The paper claimed the method can support 2M or longer visual tokens with different GPU settings, however, the experiment only proved the efficiency on certain tokens which fails to demonstrate how consumption and inference time evolve as sequence lengths scale from hundreds of thousands to millions of tokens.
Questions for authors
  1. The paper applies the low-rank residual of the attention matrix to determine which retrieval groups require the retention of exact attention. However, does a larger low-rank residual necessarily imply that linear attention fails to accurately preserve information in these regions? RSR primarily focuses on the structure of attention weights, without directly accounting for the value information or its impact on the final output.
  2. The results claimed that the method can support 5M tokens, however, the experiment only conduct the results on 98K tokens. Have the authors tested the actual consumption and inference time for longer tokens? As the sequence length increases, does the efficiency advantage of SourceScale-VL over Sequence Exact still hold?
  3. This paper set the dimension for Linear Attention to r=32r=32 and keep 25% of the Exact Retrieval Groups fixed. Why were these two parameters chosen? Can RSR consistently outperform KL-based Selection across different computational budgets?
  4. The current experiments mainly compare the proposed approach against constructed baseline variants. Can we use the same backbone and computational budget to conduct the direct comparison?
Reviewer_Q3wUOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SourceScale-VL, a VLM conversion framework based on hybrid attention and knowledge distillation, to address the computational cost and visual information loss in ultra-high-resolution remote sensing image understanding. The method replaces attention among visual tokens with linear attention, uses Referential Spectrum Residual(RSR) to selectively retain exact attention, and introduces Source-Factored On-Policy Distillation to reduce the performance loss caused by attention conversion. Experiments on XLRS-Bench-lite and LRS-VQA show that the proposed method slightly outperforms the exact-attention teacher in accuracy. Controlled attention conversion experiments further demonstrate the effectiveness of RSR-based selection and Source-Factored On-Policy Distillation. Compared with sequence-parallel exact attention, the method also supports longer visual sequences while reducing peak memory usage and time to first token.

Critical strengths
  1. The proposed method is reasonably novel and well-motivated. It provides a practical approach to improving computational efficiency while preserving visual information in ultra-high-resolution image understanding.
  2. The experiments are relatively comprehensive. This work includes comparisons and ablation studies to evaluate the effectiveness of different components, as well as evaluations of model performance and computational efficiency.
Critical weaknesses
  1. Lack of fair comparisons with visual token compression methods. Although preserving the full visual sequence is a key motivation of this work, this work does not directly compare the two approaches under the same backbone and computational budget, making it difficult to fully assess the benefits of the proposed design.
  2. Limited efficiency analysis. More experiments across different sequence lengths are recommended to better assess the efficiency and scalability of the proposed method.
Questions for authors

See weaknesses.

Reviewer_tMWUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that UHR remote-sensing VLMs control cost by reducing visual content (tiling, region selection, token pruning), which couples scalability to compression and discards query-relevant detail. It proposes to compress visual interactions instead, keeping the complete post-encoder visual sequence. A pretrained RS-adapted LongVILA-R1-7B is converted post hoc into a hybrid attention model: visual-prefix propagation becomes causal linear attention with ring-style sequence parallelism, while language-to-visual retrieval is kept Exact only for a subset of layer/KV-group "retrieval atoms". Which atoms stay Exact is decided by a Referential Spectrum Residual (RSR): the centred teacher attention mass over image source cells is decomposed and the rank-r residual beyond the linear branch's capacity is computed per atom (Eckart–Young), and the top-K_exact atoms (25% of 112) are kept Exact. Recovery uses Source-Factored On-Policy Distillation, which aligns source-wise contributions (not just aggregated hidden states) on student-generated states, together with token KD, operator KD and CE. Evaluation is on XLRS-Bench-lite, LRS-VQA, and a held-out 2K-question conversion split from the GeoLLaVA-8K corpus, plus memory/TTFT/sequence-length profiling on up to 12 A800 GPUs.

Critical strengths

S1. A principled and validated selection criterion. RSR is not a heuristic score: it upper-bounds the residual of the best rank-r approximation of the teacher's source-response matrix, and the paper checks it against an oracle ranking obtained by individually restoring atoms (ρ 0.63 vs 0.44 for KL-based selection; Recall@28 67.9% vs 46.4%). This is the kind of analysis that makes a conversion method interpretable rather than tuned, and it would be useful even if the final system were simplified.

S2. The conversion works end-to-end on a real 7B RS model and at multiple GPU counts. The full 1.99M-token visual sequence is retained versus 1.77M for sequence-parallel Exact attention, with −13.8% peak memory and −15.5% TTFT at a fixed 98K input (Table 2), and the system scales to 12 GPUs through compact-state propagation. Achieving this without discarding visual tokens is a meaningful systems result.

Critical weaknesses

W1. The full method is worse than its own simplest ablation on both reported fidelity metrics, and the paper never says so. Table 3 reports Propagation-only — linearize visual-prefix propagation, keep every retrieval atom Exact — at 57.61% accuracy and SCE 0.030, versus 57.46% and SCE 0.041 for the complete SourceScale-VL. In other words, removing both headline contributions (RSR allocation and Factor-OPD) improves the held-out conversion accuracy and reduces source-contribution error. The text discusses Propagation-only in one sentence ("approaches the Exact teacher") and then reports the gain over RSR-Hybrid (+0.43) as the headline, which is the weaker comparison. The only possible defence is cost: Propagation-only keeps all 112 atoms Exact, so it is presumably slower, but its cost is never reported alongside its accuracy. Until an accuracy-vs-cost Pareto curve including Propagation-only is given, the reader must conclude that the proposed machinery adds complexity without adding fidelity. W2. The headline improvements are numerically negligible and statistically unsupported. The abstract's "slightly surpassing Teacher" is +0.14 points on XLRS-Bench-lite (54.64 vs 54.50) and +0.18 on LRS-VQA (33.58 vs 33.40), from a single run with no seeds, no confidence intervals and no per-task breakdown. With 13 macro-averaged XLRS tasks, a 0.14-point difference is far inside the range produced by decoding or prompt variation. The efficiency claims are also modest (1.12× sequence length, −13.8% memory, −15.5% TTFT), and the paper's own Table 2 shows All-Linear dominating on all three efficiency axes (2212K tokens, 13.92 GiB, 5.31 ms) at a cost of 1.2 accuracy points. The genuine contribution is therefore a point on a Pareto frontier that is never plotted; without it, neither "scales" nor "preserves fidelity" is quantified in a way a reader can use.

W3. Control and comparability gaps in the main table. Table 1 compares the converted 7B LongVILA against Qwen2.5-VL/Qwen3-VL, a Qwen3-VL+RS-SFT baseline of a different backbone, and active-search methods (GeoEyes, DeepEyes) that use different inference budgets; "highest aggregate among non-reference models" is therefore a cross-backbone statement, not a controlled one. Conversely, the conversions the paper actually competes with (LoLCATs, RADLADS and the hybrid conversion of Li et al. 2025, all cited) are never run. The teacher itself is only LoRA-adapted on the training split, so part of the "student matches teacher" result may reflect a weak teacher rather than a faithful conversion.

Questions for authors
  1. What are the TTFT, memory and maximum sequence length of Propagation-only? Please give an accuracy-vs-cost Pareto plot containing Teacher, Propagation-only, All-Linear, KL-Hybrid, RSR-Hybrid and SourceScale-VL. If Propagation-only is dominated, the paper's claims are supported; if not, RSR and Factor-OPD are not justified.
  2. Why does the full method fall below Propagation-only on both accuracy and SCE? Is the difference attributable to the 25% Exact budget rather than to RSR?
  3. Please report at least three seeds or bootstrap confidence intervals for 54.64 vs 54.50 and 33.58 vs 33.40, and remove the "surpassing Teacher" claim if it is not significant.
  4. Were r, K_exact, the λ weights and the reduction layers selected on the reported conversion split? Please confirm the calibration set, oracle set and evaluation split are disjoint in images.
  5. Table 1 compares backbones with different pretraining and data; can you add the same-backbone non-converted baseline (LongVILA-R1-7B + RS-SFT without conversion) and one external conversion baseline (LoLCATs or RADLADS) under your protocol?
  6. Please complete the appendix (conversion schedule, layer/atom map, prompts, calibration details), fix the "Figure ??" reference, and state where the code will be available.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.