- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focuses on the problem of how to enable VLMs to efficiently process very high resolution remote sensing images, while existing methods reduce computational costs by discarding or merging visual tokens, or by selecting only specific image regions which often lead to the loss of important details. The proposed SourceScale-VL applies the linear attention and sequence parallelism associate with RSR method to evaluates the low-rank approximation error within different attention groups to determine which components require exact attention and which can be converted to linear attention. In addition with Source-Factored On-Policy Distillation, where the student model learns not only from the teacher’s final output but also from the relative contributions of different image regions to that output, thereby minimizing performance degradation resulting from the model transformation.
Critical strengths
- This paper addressed an important problem for remote sensing tasks that require the identification of small objects and the understanding of complex spatial relationships, which is well motivated.
- The proposed method shows a comprehensive design, integrating linear attention, RSR-guided selection between exact and linear attention, and source-factored on-policy distillation. The experimental results achieved certain performance on the target to reduce the cost while maintaining the performance from the teacher model.
Critical weaknesses
- The novelty of this paper needs more explaination to prove, current theoretical analysis and experiments are not sufficient to fully support the idea.
- The paper claimed the method can support 2M or longer visual tokens with different GPU settings, however, the experiment only proved the efficiency on certain tokens which fails to demonstrate how consumption and inference time evolve as sequence lengths scale from hundreds of thousands to millions of tokens.
Questions for authors
- The paper applies the low-rank residual of the attention matrix to determine which retrieval groups require the retention of exact attention. However, does a larger low-rank residual necessarily imply that linear attention fails to accurately preserve information in these regions? RSR primarily focuses on the structure of attention weights, without directly accounting for the value information or its impact on the final output.
- The results claimed that the method can support 5M tokens, however, the experiment only conduct the results on 98K tokens. Have the authors tested the actual consumption and inference time for longer tokens? As the sequence length increases, does the efficiency advantage of SourceScale-VL over Sequence Exact still hold?
- This paper set the dimension for Linear Attention to and keep 25% of the Exact Retrieval Groups fixed. Why were these two parameters chosen? Can RSR consistently outperform KL-based Selection across different computational budgets?
- The current experiments mainly compare the proposed approach against constructed baseline variants. Can we use the same backbone and computational budget to conduct the direct comparison?