- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Virtual Target Calibration (VTC), a task-aware extrapolation method for Vision Rotary Position Embedding (Vision RoPE). It aims to improve the generalization ability of vision models on high-resolution images and long-video scenarios. To address the lack of downstream task supervision in existing approaches, the authors construct a virtual target state using supervision signals from the source resolution. They simulate attention interactions at the target resolution via coordinate expansion and virtual tokens, and optimize RoPE parameters with task-aware Fisher weights and response calibration mechanisms. This method requires no additional training data or updates to backbone network parameters; adaptation can be completed with only a small number of calibration steps. Experiments cover ImageNet-1K image classification, high-resolution image generation, and long video generation tasks. The results demonstrate that VTC can effectively boost resolution extrapolation performance, image generation quality, and video temporal consistency.
Critical strengths
- The research problem bears practical significance. This work focuses on the position embedding extrapolation problem of vision transformers for high-resolution images and long-video scenarios. As the number of tokens processed by vision models keeps growing, achieving effective extrapolation without retraining the model represents a research direction with practical application value.
- A novel task-aware extrapolation framework is proposed. Unlike conventional methods that mainly rely on position interpolation or frequency adjustment, this paper constructs a Virtual Target State and leverages supervision signals from the source resolution to guide position embedding calibration at the target resolution, offering a new research paradigm for Vision RoPE extrapolation.
- The method design is relatively complete with certain theoretical support. VTC combines task-aware Fisher weights, response calibration, and source-scale anchoring mechanisms to optimize position embeddings while preserving the original attention structure as much as possible. The paper also provides corresponding theoretical analysis to strengthen the rationality of the method.
- Experiments cover a variety of vision tasks. The authors validate the effectiveness of the method on three tasks: ImageNet-1K image classification, high-resolution image generation, and long video generation. Comparisons with multiple existing extrapolation methods are conducted, demonstrating cross-task applicability.
- Low deployment cost. VTC requires no additional training data at the target resolution and does not update backbone network parameters. Adaptation can be accomplished with only a small number of calibration steps, endowing the method with good practical value.
Critical weaknesses
- The novelty of the proposed method has certain limitations. While constructing task-aware supervision signals using virtual tokens is innovative, the overall framework still largely builds upon existing technical ideas including RoPE frequency calibration, Fisher information weighting, and attention response matching. Whether the combination of these components constitutes sufficiently prominent core methodological novelty remains open for further discussion.
- The theoretical analysis relies on certain approximation assumptions. This work approximates attention interactions at the target resolution via virtual tokens, yet its effectiveness hinges on local approximations and the response relationship between the source scale and target scale. Sufficient validation is missing to verify whether these assumptions still hold when the extrapolation ratio is large or the distribution of visual content changes drastically.
- The experiments cover a limited range of model sizes and architectures. Although three categories of vision tasks are included in experiments, the image generation and video generation evaluations are mainly conducted on FiT-v2-XL and CogVideoX-5B respectively. Validations over more model architectures and diverse parameter scales are absent. In addition, performance gains observed in some experiments are relatively marginal, which cannot fully demonstrate the broad generalization ability of the method.
- The AI Use Statement does not fully comply with the official ICLR template requirements. Although the manuscript includes an AI Use Statement section, its content is not written in full accordance with the official ICLR template requirements. In particular, the disclosure of generative AI usage is non-standard. The authors are advised to carefully revise and complete this section by referring to the official template to satisfy the conference formatting and disclosure requirements.
Questions for authors
- Regarding methodological novelty: Compared with existing RoPE frequency adjustment and attention calibration methods, does the most critical innovation of VTC stem from the construction of the virtual target state, or from the task-aware Fisher weights and response calibration? Could the authors further clarify the relationship among these components and their essential differences from prior techniques?
- Regarding the validity of theoretical approximations: VTC uses virtual tokens at the source scale to approximate real attention interactions at the target scale. Does the error of this approximation accumulate significantly as the extrapolation ratio grows? Can the applicable range be verified via theoretical analysis or experiments?
- Regarding cross-model generalization ability: Current image and video generation experiments are mainly limited to a small set of models. Can VTC be directly applied to other Vision Transformers, DiTs, or video generation architectures? Do different model sizes and position encoding designs affect the calibration performance?
3 comments
Response to authors
Thank you for your reply, most of my concerns have been addressed and I would like to keep my positive rating. Good luck!
Response to Reviewer TGnU
Thank you very much for your careful reading and constructive feedback. We are grateful that our responses addressed most of your concerns, and we sincerely appreciate your continued positive assessment.
Response to Reviewer TGnU
Response to Reviewer
We thank the reviewer for the careful assessment. We address the three technical questions and will revise the manuscript to clarify the approximation scope, cross-model applicability, and AI-use statement.
1. Methodological novelty. The virtual target state and the task-aware calibration are not competing candidates for the main contribution; they are inseparable parts of one formulation. The virtual target state defines the target-conditioned behavior that the RoPE-ViT should approach at an enlarged grid by representing its positional span and key population using only source features. Task-aware Fisher weighting and response calibration provide the optimization mechanism that transfers source-task supervision toward this state; neither is intended as an independent contribution (see sec. 5.1). The core novelty is therefore their coupling into a source-only, target-aware calibration objective, which enables a frozen RoPE-ViT to learn high-resolution extrapolation behavior from low-resolution images without target-scale inputs or labels and with only a tiny controller update. Frequency calibration, Fisher weighting, and attention-response matching each have precedents, but to our knowledge VTC is the first to explicitly connect a virtual source-constructed target state to the corresponding enlarged-grid state: the virtual state defines the target-scale behavior to be matched, while task-aware Fisher calibration provides the source-task signal to optimize toward it for extrapolation. The improvement across various methods and tasks confirms the effectiveness of our method.
2. Approximation and extrapolation range.
We agree that VTC is not an unlimited or lossless extrapolator. Appendix A makes the approximation error explicit. For a target micro-key q represented by a virtual key j, the logit error is bounded by
\leq \frac{1}{\sqrt{d_h}}\sum_m \left(|\delta c_{r q m}|+ |c^V_{r j m}|\min\{2,|\delta\phi_{r q m}|\}\right).$$ The two terms correspond to content drift and within-group phase drift. As the target/source ratio increases, high-frequency phase differences and within-cell content variation can become larger. Since the induced softmax change satisfies $(\Delta p\approx\beta(\mathrm{diag}(p)-pp^\top)\Delta a)$, the error is more consequential when many comparable keys compete in the softmax. The multiplicity approximation has a complementary bound. If $(|a^T_{r q}-a^V_{r j}|\leq\epsilon_a)$, then $$\left|\log\frac{M^T_{rj}}{m_{rj}}\right| \leq 2\beta\epsilon_a.$$ It is exact only under exchangeable target micro-keys; real images and videos violate this assumption because neighboring keys may have different content and phases. The Fisher transfer is also local: if the target and virtual logit gradients differ by at most $(\epsilon_g)$, then $$|I^T_{rj}-\widetilde I_{rj}| \leq \mu_j\epsilon_g(2|\widetilde g_{rj}|+\epsilon_g).$$ These bounds explain why feature drift, phase drift, non-exchangeability, and downstream-score drift can accumulate as the extrapolation ratio grows. VTC is designed to reduce these errors, rather than eliminate all target-side changes or remain exact under arbitrary distribution shifts. Despite this limitation, VTC consistently improves over Vision-YaRN across tasks and backbones. Table~1 reports gains across five RoPE backbones, Table~4 shows improved FiT-v2-XL image generation, and Table~5 shows better Dynamic Degree and Image Quality for CogVideoX-5B. These results show that the VTC useful even though their extrapolation cannot be error-free. **3. Cross-model and cross-architecture generalization.** We agree that both model scale and positional design may affect calibration. We therefore added two complementary evaluations: | Model | 512 | 768 | 1024 | |---|---:|---:|---:| | Mix-RoPE Base | 82.89→84.35 (+1.46) | 72.70→80.73 (+8.03) | 41.78→71.72 (+29.94) | | Swin-Mix-RoPE Tiny | 73.06→75.54 (+2.48) | 44.87→58.62 (+13.75) | 24.96→36.75 (+11.79) | The Mix-RoPE Base results show that VTC remains effective at a different model scale. The 768-pixel result is also obtained by interpolation between calibrated anchors, rather than by direct calibration. The Swin-Mix-RoPE Tiny experiment extends the evaluation beyond the plain ViT setting: Swin uses a hierarchical multi-scale representation and local windows, yet VTC still produces positive gains at all three resolutions. Together with the ViT-5 result already reported in Table 1 (**41.84→55.02** at 1024 pixels), the results demonstrate the generalization of VTC across different model scales and different model frameworks. As for the generation task, extending VTC to more image-generation and video-generation architectures is an important direction for our future work as computational resources permit. **4. AI-use statement.** We will revise the AI-use statement to follow the official ICLR format at next version of our paper.