acceptodds
Under review as a conference paper at ICLR 2027

Virtual Tokens, Real Supervision: Task-Aware Extrapolation for Vision RoPE

Abstract

To improve generalization beyond the training scale, recent methods introduce Vision RoPE into vision models. However, at substantially larger spatial or spatiotemporal resolutions, existing extrapolation methods typically optimize from only one of two signals: source-scale task supervision or target-grid geometry. Source-supervised methods do not observe the target scale, whereas target-aware post-hoc corrections usually rely on geometric or statistical heuristics without target-conditioned task supervision. Their inability to use both signals simultaneously limits the effectiveness of extrapolation optimization, especially at large resolutions. We present Virtual Target Calibration, a source-only post-hoc method that constructs a target-conditioned surrogate from source-scale inputs. VTC stretches RoPE coordinates and assigns analytic multiplicities to visual keys to model target-scale phase changes and enlarged softmax competition. Source-task Fisher weights then guide the joint calibration of a block-unitary frequency mixer and a positive attention inverse-temperature factor. Calibration is performed at a small set of target anchors and interpolated to unseen resolutions while the backbone remains frozen; classification calibration takes about five minutes on RTX 4090 hardware, with at most 200 updates per anchor. Across image classification, high-resolution image generation, and long-video generation, VTC improves extrapolation over native inference and is competitive with or better than heuristic post-hoc corrections, without target-scale inputs or labels.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision RoPE, Vision Transformer, Positional Embedding, Attention

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_TGnUOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Virtual Target Calibration (VTC), a task-aware extrapolation method for Vision Rotary Position Embedding (Vision RoPE). It aims to improve the generalization ability of vision models on high-resolution images and long-video scenarios. To address the lack of downstream task supervision in existing approaches, the authors construct a virtual target state using supervision signals from the source resolution. They simulate attention interactions at the target resolution via coordinate expansion and virtual tokens, and optimize RoPE parameters with task-aware Fisher weights and response calibration mechanisms. This method requires no additional training data or updates to backbone network parameters; adaptation can be completed with only a small number of calibration steps. Experiments cover ImageNet-1K image classification, high-resolution image generation, and long video generation tasks. The results demonstrate that VTC can effectively boost resolution extrapolation performance, image generation quality, and video temporal consistency.

Critical strengths
  1. The research problem bears practical significance. This work focuses on the position embedding extrapolation problem of vision transformers for high-resolution images and long-video scenarios. As the number of tokens processed by vision models keeps growing, achieving effective extrapolation without retraining the model represents a research direction with practical application value.
  2. A novel task-aware extrapolation framework is proposed. Unlike conventional methods that mainly rely on position interpolation or frequency adjustment, this paper constructs a Virtual Target State and leverages supervision signals from the source resolution to guide position embedding calibration at the target resolution, offering a new research paradigm for Vision RoPE extrapolation.
  3. The method design is relatively complete with certain theoretical support. VTC combines task-aware Fisher weights, response calibration, and source-scale anchoring mechanisms to optimize position embeddings while preserving the original attention structure as much as possible. The paper also provides corresponding theoretical analysis to strengthen the rationality of the method.
  4. Experiments cover a variety of vision tasks. The authors validate the effectiveness of the method on three tasks: ImageNet-1K image classification, high-resolution image generation, and long video generation. Comparisons with multiple existing extrapolation methods are conducted, demonstrating cross-task applicability.
  5. Low deployment cost. VTC requires no additional training data at the target resolution and does not update backbone network parameters. Adaptation can be accomplished with only a small number of calibration steps, endowing the method with good practical value.
Critical weaknesses
  1. The novelty of the proposed method has certain limitations. While constructing task-aware supervision signals using virtual tokens is innovative, the overall framework still largely builds upon existing technical ideas including RoPE frequency calibration, Fisher information weighting, and attention response matching. Whether the combination of these components constitutes sufficiently prominent core methodological novelty remains open for further discussion.
  2. The theoretical analysis relies on certain approximation assumptions. This work approximates attention interactions at the target resolution via virtual tokens, yet its effectiveness hinges on local approximations and the response relationship between the source scale and target scale. Sufficient validation is missing to verify whether these assumptions still hold when the extrapolation ratio is large or the distribution of visual content changes drastically.
  3. The experiments cover a limited range of model sizes and architectures. Although three categories of vision tasks are included in experiments, the image generation and video generation evaluations are mainly conducted on FiT-v2-XL and CogVideoX-5B respectively. Validations over more model architectures and diverse parameter scales are absent. In addition, performance gains observed in some experiments are relatively marginal, which cannot fully demonstrate the broad generalization ability of the method.
  4. The AI Use Statement does not fully comply with the official ICLR template requirements. Although the manuscript includes an AI Use Statement section, its content is not written in full accordance with the official ICLR template requirements. In particular, the disclosure of generative AI usage is non-standard. The authors are advised to carefully revise and complete this section by referring to the official template to satisfy the conference formatting and disclosure requirements.
Questions for authors
  1. Regarding methodological novelty: Compared with existing RoPE frequency adjustment and attention calibration methods, does the most critical innovation of VTC stem from the construction of the virtual target state, or from the task-aware Fisher weights and response calibration? Could the authors further clarify the relationship among these components and their essential differences from prior techniques?
  2. Regarding the validity of theoretical approximations: VTC uses virtual tokens at the source scale to approximate real attention interactions at the target scale. Does the error of this approximation accumulate significantly as the extrapolation ratio grows? Can the applicable range be verified via theoretical analysis or experiments?
  3. Regarding cross-model generalization ability: Current image and video generation experiments are mainly limited to a small set of models. Can VTC be directly applied to other Vision Transformers, DiTs, or video generation architectures? Do different model sizes and position encoding designs affect the calibration performance?
3 comments
AuthorsOpenReview ↗

Response to Reviewer TGnU

Response to Reviewer

We thank the reviewer for the careful assessment. We address the three technical questions and will revise the manuscript to clarify the approximation scope, cross-model applicability, and AI-use statement.

1. Methodological novelty. The virtual target state and the task-aware calibration are not competing candidates for the main contribution; they are inseparable parts of one formulation. The virtual target state defines the target-conditioned behavior that the RoPE-ViT should approach at an enlarged grid by representing its positional span and key population using only source features. Task-aware Fisher weighting and response calibration provide the optimization mechanism that transfers source-task supervision toward this state; neither is intended as an independent contribution (see sec. 5.1). The core novelty is therefore their coupling into a source-only, target-aware calibration objective, which enables a frozen RoPE-ViT to learn high-resolution extrapolation behavior from low-resolution images without target-scale inputs or labels and with only a tiny controller update. Frequency calibration, Fisher weighting, and attention-response matching each have precedents, but to our knowledge VTC is the first to explicitly connect a virtual source-constructed target state to the corresponding enlarged-grid state: the virtual state defines the target-scale behavior to be matched, while task-aware Fisher calibration provides the source-task signal to optimize toward it for extrapolation. The improvement across various methods and tasks confirms the effectiveness of our method.

2. Approximation and extrapolation range.

We agree that VTC is not an unlimited or lossless extrapolator. Appendix A makes the approximation error explicit. For a target micro-key qq represented by a virtual key jj, the logit error is bounded by

\leq \frac{1}{\sqrt{d_h}}\sum_m \left(|\delta c_{r q m}|+ |c^V_{r j m}|\min\{2,|\delta\phi_{r q m}|\}\right).$$ The two terms correspond to content drift and within-group phase drift. As the target/source ratio increases, high-frequency phase differences and within-cell content variation can become larger. Since the induced softmax change satisfies $(\Delta p\approx\beta(\mathrm{diag}(p)-pp^\top)\Delta a)$, the error is more consequential when many comparable keys compete in the softmax. The multiplicity approximation has a complementary bound. If $(|a^T_{r q}-a^V_{r j}|\leq\epsilon_a)$, then $$\left|\log\frac{M^T_{rj}}{m_{rj}}\right| \leq 2\beta\epsilon_a.$$ It is exact only under exchangeable target micro-keys; real images and videos violate this assumption because neighboring keys may have different content and phases. The Fisher transfer is also local: if the target and virtual logit gradients differ by at most $(\epsilon_g)$, then $$|I^T_{rj}-\widetilde I_{rj}| \leq \mu_j\epsilon_g(2|\widetilde g_{rj}|+\epsilon_g).$$ These bounds explain why feature drift, phase drift, non-exchangeability, and downstream-score drift can accumulate as the extrapolation ratio grows. VTC is designed to reduce these errors, rather than eliminate all target-side changes or remain exact under arbitrary distribution shifts. Despite this limitation, VTC consistently improves over Vision-YaRN across tasks and backbones. Table~1 reports gains across five RoPE backbones, Table~4 shows improved FiT-v2-XL image generation, and Table~5 shows better Dynamic Degree and Image Quality for CogVideoX-5B. These results show that the VTC useful even though their extrapolation cannot be error-free. **3. Cross-model and cross-architecture generalization.** We agree that both model scale and positional design may affect calibration. We therefore added two complementary evaluations: | Model | 512 | 768 | 1024 | |---|---:|---:|---:| | Mix-RoPE Base | 82.89→84.35 (+1.46) | 72.70→80.73 (+8.03) | 41.78→71.72 (+29.94) | | Swin-Mix-RoPE Tiny | 73.06→75.54 (+2.48) | 44.87→58.62 (+13.75) | 24.96→36.75 (+11.79) | The Mix-RoPE Base results show that VTC remains effective at a different model scale. The 768-pixel result is also obtained by interpolation between calibrated anchors, rather than by direct calibration. The Swin-Mix-RoPE Tiny experiment extends the evaluation beyond the plain ViT setting: Swin uses a hierarchical multi-scale representation and local windows, yet VTC still produces positive gains at all three resolutions. Together with the ViT-5 result already reported in Table 1 (**41.84→55.02** at 1024 pixels), the results demonstrate the generalization of VTC across different model scales and different model frameworks. As for the generation task, extending VTC to more image-generation and video-generation architectures is an important direction for our future work as computational resources permit. **4. AI-use statement.** We will revise the AI-use statement to follow the official ICLR format at next version of our paper.
Reviewer_TGnUOpenReview ↗

Response to authors

Thank you for your reply, most of my concerns have been addressed and I would like to keep my positive rating. Good luck!

AuthorsOpenReview ↗

Response to Reviewer TGnU

Thank you very much for your careful reading and constructive feedback. We are grateful that our responses addressed most of your concerns, and we sincerely appreciate your continued positive assessment.

Reviewer_fGRTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper introduces a method that calibrates vision transformers to better extrapolate to larger images at test time. The calibration is performed after the original training at the original image size and before deploying the calibrated model on larger images. The calibration procedure does not give the model the deployment-resolution images, rather, it simulates these larger images and trains some parameters to adjust to the simulation. Simulation involves stretching token positions to the larger grid and using each token several times in attention. It then trains per-head adjustments via attention distillation, where the original model at the original resolution is the teacher. Distillation is weighted by the original task’s gradients.

The paper shows gains over Vision Yarn at the ViT-S size on ImageNet-1K. VTC also helps image generation and video recognition.

Critical strengths
  • the paper clearly improves over Vision Yarn and tackles an important problem
Critical weaknesses
  • the method itself is quite complex
  • it beats Vision Yarn but requires a calibration / post-training phase that Vision Yarn does not require
  • I am skeptical that the method is useful in practise because there may be simpler ways of "post-training" ViTs to prepare it for larger images at deployment. E.g. simply training at the higher-resolution for a short period of time. I understand that higher-resolution post-training is expensive, but this should be shown empirically. If it is too expensive, can we do other simple things like train at the higher resolution but use token dropping/masking in order to reduce compute? Or can we post-train at varied resolution to limit the extra costs? Or maybe only every Nth step at the higher resolution?
  • the ablations are minimal (only showing frequency vs response-only)
Questions for authors

please clarify the number of parameters that are introduced and trained during calibration and how many of them are used/kept at deployment and what their cost is in terms of FLOPs/memory/throughput.

1 comment
AuthorsOpenReview ↗

Response to Reviewer fGRT

We appreciate the reviewer’s comment. We agree that our exposition may have made VTC appear more complicated than its implementation. VTC is a frozen ViT with analytic target-grid corrections and two small controller families; it is not a second network or a full-model fine-tuning procedure, and the response term is a loss for the same controllers.

Practical cost and comparison with Vision-YaRN

For Compass-RoPE ViTs on image classification, calibrating one anchor with a batch size of 16 takes only about 5 minutes using micro-batch 2 with gradient accumulation, while directly training with batch size 16 takes less than 1 minute, demonstrating the low calibration overhead (see table in Direct high-resolution fine-tuning). This makes the practical implication clear: for classification, the controller-only adaptation can be completed in under one minute with physical batch 16 and 3.37 GiB peak memory; the four anchors can also run in parallel. The cost is paid once, while the same controllers are reused across deployment inputs and interpolated resolutions.

This small calibration is accompanied by broad, consistent gains rather than a single isolated improvement. In Table 1, VTC outperforms Vision-YaRN in all 15 backbone-resolution entries, with mean gains of 1.55, 3.75, and 3.48 points at 512, 768, and 1024 pixels. Large margins occur at difficult extrapolation ratios, for example Compass-RoPE at 1024 (65.17 vs. 60.12) and Spiral-RoPE at 1024 (59.54 vs. 52.31). Table 4 shows the same pattern for generation: at 384, VTC changes IS/FID from 157.93/23.23 for Vision-YaRN to 221.72/14.20, while at 512 it is best on all five reported metrics. Thus, a sub-minute classification calibration and a few-minutes-per-anchor (see section 5.5) image-generation calibration yield systematic improvements over Vision-YaRN across tasks and backbones. Overall, VTC introduces only a marginal one-time calibration cost over Vision-YaRN, yet delivers broader and more consistent improvements across backbones, resolutions, and generation tasks.

Direct high-resolution fine-tuning.

We thank the reviewer for suggesting simpler alternatives to VTC, particularly direct high-resolution fine-tuning and compute-saving strategies such as token dropping and mixed-resolution training. To empirically examine whether direct fine-tuning can achieve comparable extrapolation performance, we added an experiment using the same 224-trained Compass-S checkpoint. We compare VTC against direct fine-tuning at 1024 resolution, with both methods trained for 200 updates under approximately matched wall-clock budgets.

Accuracy comparison. As shown below, VTC consistently outperforms direct fine-tuning at all three evaluation resolutions, with gains of 2.93, 6.50, and 5.62 points at 512, 768, and 1024, respectively.

ResolutionDirect FTVTC
51279.2682.19
76870.9577.45
102459.5665.18

Resource comparison. This improvement does not come at a higher computational cost. VTC trains only 18,444 parameters, compared with 21,988,840 for direct fine-tuning (approximately 0.084% as many), while achieving nearly identical step time and substantially lower memory consumption.

QuantityDirect FTVTC
Batch size216
Step time0.26 s0.24 s
Peak memory17.53 GiB3.37 GiB
Trainable parameters21,988,84018,444

These results suggest that VTC offers a more effective alternative to direct high-resolution fine-tuning: it achieves better extrapolation accuracy under a comparable training budget while requiring substantially fewer trainable parameters and less memory.

We also agree that token dropping, masking, mixed-resolution training, and intermittent high-resolution updates are promising directions. They share the same underlying motivation as VTC: improving resolution extrapolation with minimal additional training cost. VTC provides one approach to this broader goal, and we see these complementary strategies as valuable directions for further investigation.

Ablation evidence. Our additional ablations further support the design choices in VTC:

AblationTop-1
Full VTC (720 px)79.05
Frequency-only78.52
Response-only56.27
Shuffled Fisher75.58
Uniform Fisher36.49
Temperature-only (1024 px)11.81
Multiplicity-aware response-only (1024 px)62.85

These results show that task-aware Fisher weighting and multiplicity-aware target-state modeling both play important roles: removing task-sensitive weighting degrades performance, while temperature scaling alone cannot replace multiplicity. Together with the direct fine-tuning comparison, these experiments demonstrate that VTC achieves effective resolution adaptation through a lightweight yet carefully structured calibration. The detailed ablation analysis can be seen in our response to reviewer MRJC.

Reviewer_MRJCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Virtual Target Calibration (VTC) for spatial and temporal extrapolation in frozen Vision RoPE models. The method constructs a virtual target state from source-scale inputs using stretched positional coordinates and analytic key multiplicities. Source-task Fisher weights guide the joint calibration of a frequency mixer and attention inverse temperature, with parameters learned at several anchors and interpolated to intermediate resolutions. Experiments cover image classification, image generation, and video generation, showing improvements over native extrapolation and Vision-YaRN in the evaluated image settings. The approach is promising, but reproducibility and the specific contribution of task-aware calibration require stronger evidence.

Critical strengths

1, The paper addresses a practical deployment problem by combining target-grid information with source-task supervision while keeping the backbone frozen.

2, Coordinate stretching and key multiplicities provide interpretable approximations of positional phase changes and enlarged attention support.

3, The classification gains are substantial. At 1024 pixels, Compass-RoPE improves from 34.22% to 65.17%, compared with 60.12% for Vision-YaRN.

4, Anchor interpolation supports additional resolutions without separate optimization, and the evaluation extends across classification, image generation, and video generation.

Critical weaknesses

1, The calibration procedure is insufficiently specified for reproduction. The operational formula connecting the frequency mixer to RoPE, its block structure, and its parameter count remain unclear. Important optimization settings, calibration data, and task losses for generation also require explicit descriptions.

2, The contribution of Fisher weighting is not isolated by the current component ablations. Uniform or shuffled weights and temperature-only calibration are important controls. Moreover, uniform duplication can leave grouped attention unchanged while rescaling the summed micro-key Fisher energy, requiring further justification of the response-matching objective.

3, Validation of the virtual target proxy remains limited. The aggregate real-model diagnostic covers 100 images from one RoPE backbone, while Fisher agreement is mainly illustrated through a representative example. Broader agreement statistics and their relationship to downstream improvements would better validate the central motivation.

4, Competitive comparisons and uncertainty reporting are incomplete. The video experiment compares only the native model and VTC, despite discussing several relevant temporal extrapolation methods. Repeated calibration runs and variability across evaluation prompts or seeds are needed to assess the reliability of the improvements.

5, The presentation needs greater precision. Figure 1 contains small text, and a single video frame cannot demonstrate temporal consistency. The five-minute calibration claim should identify the four-GPU setup. Appendix C also describes intermediate Compass baseline results as unavailable, whereas Table 6 reports corresponding values.

Questions for authors

see weaknesses

3 comments
AuthorsOpenReview ↗

The response to Reviewer MRJC

We agree that the first version under-specified implementation details and did not sufficiently isolate Fisher weighting, proxy accuracy, and uncertainty. We added the following analyses and will revise the manuscript accordingly.

1. Reproducibility. VTC maps coordinates by p~d=pdns,d−1n0,d−1\tilde p_d=p_d\frac{n_{s,d}-1}{n_{0,d}-1}; visual keys use μj=Nsvis/N0vis\mu_j=N_s^{\mathrm{vis}}/N_0^{\mathrm{vis}}, other keys μj=1\mu_j=1, and log⁡μj\log\mu_j is added inside softmax. It trains U=exp⁡(S)U=\exp(S) (skew-Hermitian, block size 4) and β=exp⁡(b)>0\beta=\exp(b)>0, initialized at U=I,β=1U=I,\beta=1, with the backbone frozen. Compass-S trains 18,444 parameters. Per anchor: 3,200 ImageNet images, 200 AdamW steps, effective batch 16, seed 1234; Fisher is recomputed per microbatch, normalized, and capped at 20. The loss is Lphase+0.25Lresponse+10−3LtemperatureL_{\rm phase}+0.25L_{\rm response}+10^{-3}L_{\rm temperature}. We use 4 4090 GPUs to train 4 anchors with one model on each GPU. Exact precision, learning rates, clipping, and generation-loss/noise settings will be added to the appendix.

2. Fisher and component ablation. The reviewer asks whether gains come from task-aware Fisher weighting and whether temperature can replace multiplicity. We added two controls.

Fisher weighting. At 720 px on Compass-S, all other settings fixed:

ModeFisher-weightedTop-1
Fullfrequency + response79.052
Frequency-onlyfrequency78.522
Shuffledshuffled weights75.582
Response-onlyresponse56.270
Uniformneither36.488

Full Fisher is strongest. Frequency-only retains most of the gain, while response-only cannot replace phase correction. Shuffling or uniform weighting removes task-sensitive assignment and causes large drops. Here response-only retains multiplicity and changes only response-loss weighting.

Temperature-only. We separately trained a Compass-RoPE temperature-only VTC with multiplicity disabled:

ResolutionNaturalTemperature-only
51280.3477.30
76865.8843.44
102434.2211.81

The role of multiplicity follows directly from the target-key population change. Each visual source key represents μj\mu_j target micro-keys, so the virtual attention probability and grouped mass are

p~j∝exp⁡(βaj+log⁡μj),mj=μjp~j\tilde p_j\propto\exp(\beta a_j+\log\mu_j), \qquad m_j=\mu_j\tilde p_j,

Thus, μj\mu_j changes the softmax partition, the relative mass assigned to visual groups, and the centered Fisher transmission, whereas β\beta only changes the sharpness of existing logit differences. A temperature parameter cannot recover the missing competition from the additional target keys. The response objective therefore matches grouped Fisher transmission after multiplicity-aware normalization, rather than an unnormalized sum of duplicated gradients.

The separate temperature-only model, which disables multiplicity and learns only β\beta, obtains 11.81% at 1024 px, which is even worse than baseline method. In contrast, the multiplicity-aware response-only VTC branch, which retains μj\mu_j and learns β\beta, obtains 62.85% at the same resolution, compared with 34.22% for natural extrapolation. The 51.04-point gap between response-only and temperature-only directly supports the theoretical role of multiplicity: the response improvement cannot be attributed to temperature scaling alone. Overall, the first table isolates the contribution of task-sensitive Fisher weighting, while the second highlights the necessity of multiplicity-aware target-state modeling.

3. Virtual-target validation. We expanded the real-model diagnostic from 100 to 10,000 images:

StateGrouped TV ↓L2 ↓
Source-native0.64380.6762
Coordinate-only0.55490.6298
Multiplicity-only0.59130.6163
Full VTC0.51320.5764

Full VTC is best on both metrics; coordinate and multiplicity corrections improve different aspects and are complementary. The ordering matches the current 100-image Table 3 diagnostic. We will replace Table 3 with the 10,000-image results.

4. Comparisons and uncertainty. For CogVideoX-5B 49→97 frames, natural and VTC were evaluated on the same 300 prompt-seed pairs (100 prompts × 3 seeds). Dynamic degree improves 0.5575→0.6063 (+0.0488; paired-bootstrap 95% CI [0.007, 0.091]). Imaging quality improves 53.176→58.331 (+5.155); 225/300 pairs improve, with CI [4.24, 6.12]. The three seed-wise imaging gains are +4.97, +4.78, and +5.71. Positive paired-bootstrap intervals and consistent gains across most prompt-seed pairs and all three seeds confirm robust improvements. Due to limited computational resources, we could not apply VTC to additional video-generation methods during rebuttal; broader video validation is our future work.

5. Presentation. We will enlarge Figure 1, replace the single video frame with a multi-frame strip, specify the four-RTX-4090 setup (see Reproducibility), correct Appendix C, and fix the noted typos in the revised version.

The author has addressed my concerns, I will keep the score.

AuthorsOpenReview ↗

The response to Reviewer MRJC

Thank you for taking the time to reconsider our responses. We are glad that the clarifications have addressed your concerns. We fully respect your decision to keep the current score. If, after considering these clarifications and additional results, you feel that they improve your overall assessment, we would be grateful if you could consider whether an upward adjustment to the score is warranted. We fully respect your final judgment either way.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.