acceptodds
Under review as a conference paper at ICLR 2027

When Visual Prompts Rotate: Spectral Guidance for Test-Time In-Context Adaptation

Abstract

Visual in-context learning (VICL) predicts dense outputs from image–annotation demonstrations. Rotating a prompt image and its annotation together preserves the demonstrated task, yet can degrade predictions for an unchanged query. This prompt-only mismatch changes spatial relationships and introduces interpolation, aliasing, and padding; a matched padding control shows that boundary fill alone does not explain the degradation. We propose GIST (Graph-Informed Spectral Tuning), a test-time adaptation method that adds graph-guided attention and error weighting to VICT's support-reconstruction cycle. Spectral Relation Guidance (SRG) uses distances in a low-frequency graph subspace to bias token interactions during both query prediction and support reconstruction. Spectral Adaptation Guidance (SAG) defines a reconstruction metric on a fixed support graph and adjusts its spectral contribution using the preceding residual's frequency composition. The two components guide information exchange and parameter updates without query labels, additional trainable modules, or rotation-specific backbone pretraining. Across 15 corruptions, two prompt settings, and four depth/segmentation benchmarks, GIST reduces average Abs.Rel by 9.9% to 27.3% relative to VICT and increases average mIoU by 0.40 to 1.14 points under matched rotations. Component ablations, attention-bias controls, and geometric diagnostics support these two intervention points while showing that graph structure remains sensitive to real image transformations.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Visual In-Context Learning, Test-Time Adaptation, Rotation Robustness, Spectral Graph Methods

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_8zhXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies prompt-only rotation in visual in-context learning, where a prompt image and its annotation are rotated together while the query remains unchanged. Building on VICT’s test-time adaptation cycle for Painter, the authors propose GIST, which adds (i) Spectral Relation Guidance, a pre-softmax attention bias derived from distances in the low-frequency Laplacian eigenspace of token affinity graphs rebuilt at selected layers, and (ii) Spectral Adaptation Guidance, a reconstruction metric that emphasizes low-frequency residual components on a fixed support-image graph, with its weight determined by the previous residual’s spectral composition. The method is evaluated on NYUv2-C, ADE20K-C, Hamlyn-C, and SUIM-C under 15 corruptions, with clean (“zero-shot”) and corrupted (“one-shot”) prompts and long-tailed random rotations. It reports 9.9–27.3% lower AbsRel and 0.40–1.14 higher mIoU points than VICT under matched rotations, supported by component ablations, attention-bias controls, padding-only controls, geometric baselines, and spectral diagnostics.

Critical strengths

1. An interesting and well-defined problem setting. The paper highlights that a visual prompt both specifies the task and provides a spatial reference for query prediction. It formulates prompt-only rotation as an asymmetric mismatch that preserves the demonstrated task while changing the prompt’s geometry. This differs from standard whole-input rotation robustness, and the padding-only control shows that boundary fill alone does not explain the observed degradation. In my view, identifying and characterizing this setting is the paper’s most original contribution and could motivate broader work on the spatial robustness of visual prompts.

2. Consistent aggregate gains with informative controls. Averaged over corruptions, GIST improves over VICT under matched rotations on all four datasets and in both prompt settings, without introducing additional trainable modules and at approximately 1.4× VICT’s runtime. The attention-bias controls on NYUv2-C and ADE20K-C show that raw feature affinity and a magnitude-matched random bias both hurt performance, whereas the spectral bias helps. On the controlled geometric-baseline subsets, GIST also outperforms the tested non-oracle geometric alternatives at lower computational cost. These results support the method’s effectiveness in the evaluated setting, although the extent to which the gains specifically reflect improved robustness to rotation remains unclear.

Critical weaknesses

1. The experiments do not clearly separate general adaptation gains from gains in robustness to prompt rotation. GIST’s results on corrupted queries with unrotated prompts are not systematically tabulated across the four datasets and both prompt settings. Figure E.2 includes a 0° point for ADE20K-C, but the corresponding scores and method differences are not explicitly analyzed. Table 3 evaluates clean, unrotated inputs and therefore does not provide the missing corruption-only comparison. This comparison is important for determining whether GIST reduces the performance penalty caused by rotation or mainly improves adaptation more generally. For example, on NYUv2-C zero-shot, GIST with rotated prompts outperforms VICT with unrotated prompts (0.330 vs. 0.365, Table 1), exceeding the improvement needed to recover VICT’s rotation-induced loss. This is encouraging, but it also makes the missing comparison particularly relevant. In addition, although the paper acknowledges that synthetic rotation combines geometric changes with resampling artifacts, the relevance of this compound shift to naturally occurring orientation mismatch remains insufficiently validated.

2. The rotation-specific rationale for spectral guidance remains insufficiently established. The paper appropriately presents spectral structure as a soft cue rather than claiming exact rotation invariance. Nevertheless, the connection between its spectral diagnostics and the observed task-level gains remains unclear. In the ADE20K-C initialization diagnostics, raw-feature neighborhoods have higher retention under rotation than spectral neighborhoods (Tables 11–12). These diagnostics use near-zero-aware mode selection, rather than the implemented fixed-index policy. Higher neighborhood retention does not imply greater usefulness as an attention bias, and Table 4 does provide downstream evidence favoring the spectral construction. However, the results do not establish that its benefit comes from better preservation of relationships under rotation. Furthermore, annotation- and fused-stream SRG graphs are disconnected in approximately 90–100% of cases in the ADE20K-C initialization diagnostic (Table 9), raising questions about the downstream sensitivity of the fixed mode-selection policy. The formal analysis establishes conditional permutation equivariance under ideal node relabeling, not a guarantee for physical image rotation. Similarly, the SAG diagnostics illustrate frequency-selective weighting without establishing why rotation-induced reconstruction errors should particularly benefit from that weighting. Thus, the components have empirical support, but their specific connection to the proposed rotation mismatch needs stronger evidence.

Questions for authors
  1. Could you report GIST and VICT on corrupted queries with unrotated prompts for all four datasets and both prompt settings? Please also compare the rotation-induced performance degradation of each method relative to its own unrotated baseline. If comparable GIST–VICT gains are already present without rotation, this would weaken the evidence that GIST provides additional robustness to prompt rotation beyond a general improvement in test-time adaptation.

  2. When rotation is the only applied transformation, with no additional benchmark corruptions on either the query or prompt, how much does prompt rotation degrade VICT, and how much does GIST recover? Table 3 evaluates clean inputs without prompt rotation and therefore does not isolate this condition.

  3. The annotation- and fused-stream SRG graphs are almost always disconnected in the ADE20K-C initialization diagnostic (Table 9), which the paper attributes partly to mask/label structure and support–query grouping. Table 10 shows that skipping near-zero modes changes the initialization diagnostics only modestly, but does this also hold for downstream performance after adaptation? It would also be informative to evaluate SRG applied only to the layer-0 image stream, whose graphs are mostly connected. The layer-0 ablation in Table 8 includes both image and annotation streams and does not isolate this alternative.

  4. How is the angle inferred in the estimated-canonicalization baseline in Table 14? Please specify the estimator or search procedure and the selection criterion. It would also help to clarify which intended deployment settings lack usable relative-orientation information, and how the comparison would change when relevant orientation metadata are available.

  5. Query orientation may also vary in the motivating applications. Could you evaluate cases where the query is rotated as well, either by the same angle as the prompt or by an independently sampled angle? This would clarify how far the findings extend beyond the controlled prompt-only setting.

  6. Since neither component is specific to rotation, additional tests involving prompt–query mismatches such as scaling, translation, and flipping would help establish the method’s broader scope. Naturally acquired pairs with orientation differences, without an added synthetic rotation, would also help assess the relevance of the evaluation protocol. As a further extension, robustness to noisy prompt annotations would be informative, since both VICT and GIST use the prompt annotation as their self-supervision signal.

4 comments

We thank the reviewer for these constructive comments. We added aligned-prompt controls, downstream graph-mode and stream ablations, rotated-query and geometric-mismatch tests, and a natural-viewpoint evaluation; additions are marked in blue, with details in Appendices E, G, and J.

Do the gains reflect rotation robustness beyond general adaptation?

To distinguish the effect of prompt rotation from each method's general adaptation benefit, we compare aligned and rotated prompts on the same corrupted queries. We added the missing aligned-prompt controls to Table G.2 for four severity-5 NYUv2-C corruption representatives: Gaussian noise, motion blur, fog, and JPEG compression. We use the same query identities, checkpoint, 60-step adaptation budget, and matched rotation draws. In zero-shot, the support prompt is clean; in one-shot, it has the corresponding corruption. VICT-0/GIST-0 denote unrotated prompts, and VICT-R/GIST-R denote rotated prompts. We report each method's rotation penalty, Δ=R−0\Delta=R-0, and D=ΔVICT−ΔGISTD=\Delta_{\rm VICT}-\Delta_{\rm GIST}; positive DD means GIST loses less (or gains more) from prompt rotation than VICT.

Zero-shot (AbsRel ↓\downarrow; clean support prompt):

CorruptionVICT-0VICT-RΔVICT\Delta_{\rm VICT}GIST-0GIST-RΔGIST\Delta_{\rm GIST}DD
Gaussian noise0.1910.215+0.0240.2010.207+0.006+0.018
Motion blur0.1320.141+0.0090.1280.133+0.005+0.004
Fog0.1530.162+0.0090.1560.159+0.003+0.006
JPEG compression0.3310.342+0.0110.3100.258−0.052+0.063

One-shot (AbsRel ↓\downarrow; support prompt has the corresponding corruption):

CorruptionVICT-0VICT-RΔVICT\Delta_{\rm VICT}GIST-0GIST-RΔGIST\Delta_{\rm GIST}DD
Gaussian noise0.2410.361+0.1200.2550.243−0.012+0.132
Motion blur0.1210.122+0.0010.1240.126+0.002−0.001
Fog0.1280.143+0.0150.1630.164+0.001+0.014
JPEG compression0.1590.272+0.1130.1780.245+0.067+0.046

The new controls show some, but limited and corruption-dependent, adaptation benefit. In zero-shot, the aligned GIST-0 comparison is mixed: GIST is slightly better than VICT-0 on motion blur and JPEG, and slightly worse on Gaussian noise and fog. After prompt rotation, GIST-R is below VICT-R on all four rows, but the relative rotation effects DD are small for Gaussian noise, motion blur, and fog (+0.018, +0.004, +0.006); JPEG is the clear exception (+0.063). Importantly, against the unrotated VICT-0 baseline, GIST-R is still worse on Gaussian noise, motion blur, and fog, and better only on JPEG. Thus lower error than rotated-prompt VICT in these rows does not by itself establish a broad adaptation gain.

In one-shot, GIST-0 has higher error than VICT-0 in all four rows. GIST-R beats VICT-R only for Gaussian noise and JPEG, while it is worse for motion blur and fog; GIST-R remains worse than aligned VICT-0 in all four. The largest relative rotation benefit is Gaussian noise, followed by JPEG; fog shows a small difference, and motion blur provides no benefit. Therefore, these controls do not show a general one-shot TTA improvement or complete recovery of the rotation penalty.

The full 15-corruption NYUv2-C zero-shot results give useful context. GIST-R is lower than VICT-R in 10/15 corruption types, but lower than unrotated VICT in only 6/15. The mean AbsRel is 0.330 for GIST-R, 0.375 for VICT-R, and 0.365 for VICT-0. Defocus blur (0.672 to 0.416), glass blur (0.533 to 0.389), and JPEG (0.342 to 0.258) account for about 72% of the GIST-R--VICT-R mean gap. As a descriptive check, excluding these three gives 12-corruption means of 0.3398 for VICT-R, 0.3243 for GIST-R, and 0.3283 for VICT-0: GIST-R and unrotated VICT are close, with only a 1.2% difference, while the remaining rotated-baseline difference is 4.6%. This shows that selected corruption gains substantially shape the aggregate and should not be read as uniform adaptation improvement.

How much does prompt rotation alone degrade VICT, and how much does GIST recover?

To isolate prompt rotation from benchmark corruptions, we compare clean-input results with aligned and rotated support pairs.

Table G.1 removes all benchmark corruptions and rotates only the support pair:

Dataset / metricVICT aligned → rotated; degradationGIST aligned → rotated; degradation
NYUv2 AbsRel (↓)0.0868 → 0.1049; +0.01810.0916 → 0.0996; +0.0080
ADE20K mIoU (↑)49.66 → 48.56; −1.1049.08 → 48.71; −0.37

Prompt rotation alone causes a 56% smaller AbsRel increase on NYUv2 and a 66% smaller mIoU loss with GIST. GIST also slightly exceeds VICT in the rotated condition on both datasets, despite starting from the stronger aligned VICT score.

Do disconnected SRG graphs affect downstream mode selection and image-only guidance?

The reviewer identifies a concrete risk in the graph construction: if a graph has multiple connected components, the implemented fixed indices u2:u9u_2{:}u_9 can include extra zero modes. We examined this at initialization on an ADE20K-C one-shot subset spanning all 15 corruptions and eight signed rotations (Appendix E.3, Tables 9--10). The temporary SRG graphs and fixed SAG graph play different roles. Layer-0 image SRG graphs are disconnected in 6.7% of records, annotation graphs in 100%, and layer-16 fused graphs in 90.4% (forward) and 99.3% (reverse); the fixed support-loss graph is connected in every record. Disconnection is therefore concentrated in the temporary attention cues, not the graph defining the SAG reconstruction metric.

Painter’s stream structure explains where the disconnected graphs arise: annotation tokens encode masks and labels, while fused tokens include support–query grouping. The initialization analysis in Appendix E.5, Table 13 shows that disconnection does not eliminate cross-rotation subspace correspondence: the forward/reverse annotation graphs have affinities 0.683/0.825, while the fixed support-loss graph is connected. Across angles, affinity decreases from 0.785 at 5° to 0.524 at 120° (Table 11), and varies by corruption from 0.440 for impulse noise to 0.793 for defocus blur (Table 12). These results characterize the spectral cue as angle- and corruption-dependent.

To test whether mode selection affects adapted predictions, Appendix J adds a downstream ADE20K-C one-shot experiment (Table J.1). It evaluates Gaussian noise, defocus blur, snow, and JPEG compression, using 500 shared query identities, the same checkpoint and optimizer, and 60 adaptation updates under rotated prompts. The fixed u2:u9u_2{:}u_9 policy averages 20.381 mIoU; skipping all modes with λ≤10−5\lambda\leq10^{-5} and selecting the next eight averages 20.518, a +0.137-point difference. Per-corruption changes are +0.792, +0.090, −0.610, and +0.274. Thus the two policies have nearly equal mean downstream performance, with mixed per-corruption effects; zero-aware selection provides no consistent advantage, and the fixed policy shows no systematic loss from the extra near-zero modes.

Appendix J also tests the layer-0 image-only variant requested by the reviewer (Table J.2). With the same four corruptions, 500 queries, 60 updates, and SAG retained, removing the annotation-stream SRG bias changes mIoU by +0.005, +0.801, +0.690, and +0.075, respectively. The image-only mean is 20.894 versus 20.501 for layer-0 image-plus-annotation, a +0.393-point gain; it is also 0.513 points above the full fixed-mode configuration. On this evaluation, the connected image stream alone is sufficient for the measured SRG benefit, while adding the annotation-stream bias gives no further gain. Together, Tables J.1–J.2 show that the fixed mode policy retains its downstream performance and that image-only layer-0 guidance is an effective option in the tested rotated ADE20K-C conditions.

How is the canonicalization angle estimated, and when is metadata available?

To make the angle-estimation baseline reproducible and answer the selection-criterion question, we specify the image-processing steps used to obtain its estimate.

For the estimated-canonicalization baseline, we convert the rotated support RGB image to grayscale, compute a Canny edge map, and extract straight-line segments with a Hough line transform. We discard short segments, calculate each retained segment's orientation from its endpoints, and measure its signed deviation from the nearest horizontal or vertical axis. We use the robust length-weighted median of these deviations as the estimated tilt, then rotate the support RGB image and annotation together by the negative estimate before applying VICT. This line-based estimate recovers tilt relative to a horizontal or vertical axis, but cannot reliably resolve the full signed orientation for rotations greater than 90°. It can therefore select an incorrect rotation branch and leave the prompt incompletely canonicalized, which helps explain why this baseline underperforms the oracle (Table 14). The oracle instead uses the recorded synthetic angle directly.

On the fixed 200-sample-per-corruption subsets, the results are:

MethodNYUv2 AbsRel (↓)ADE20K mIoU (↑)Relative time
VICT-R0.395 ± 0.00532.3 ± 0.31.00×
Rotate-and-average0.369 ± 0.00632.6 ± 0.24.60×
Estimated canonicalization0.358 ± 0.00432.7 ± 0.51.95×
Oracle inverse rotation0.302 ± 0.00333.9 ± 0.31.27×
GIST-R0.330 ± 0.00333.2 ± 0.21.40×

When reliable relative-orientation metadata are available, direct inverse correction is possible. The oracle reaches 0.302 AbsRel and 33.9 mIoU, compared with 0.330 and 33.2 for GIST-R. The estimated row represents deployment without such metadata, as can occur when a reusable support and a live query come from differently oriented devices or sessions with no synchronized orientation record. Among methods without the generated angle, GIST is best on both datasets and is substantially cheaper than rotate-and-average.

What happens when the query also rotates with or independently of the prompt?

We study the orientation relation between the prompt and query, rather than assuming that queries must always be upright. To test the reviewer's setting, Appendix G.6 (Table G.8) jointly rotates each NYUv2-C query and its depth target, together with the one-shot support image and annotation. The query and support either share a sampled angle (their relative orientation is preserved) or draw independent angles from the same distribution (their relative orientation varies). We evaluate four severity-5 corruptions with 654 queries per condition, excluding padded and invalid-depth pixels from AbsRel.

CorruptionPrompt-only VICT-RBoth rotated: same angle VICT / GISTBoth rotated: independent angles VICT / GIST
Gaussian noise0.3610.4376 / 0.39420.4008 / 0.3725
Motion blur0.1220.1989 / 0.20120.1935 / 0.1991
Fog0.1430.2290 / 0.25950.2256 / 0.2475
JPEG compression0.2720.2956 / 0.29060.2898 / 0.2866

Rotating the query as well increases VICT's error relative to the prompt-only scores on all four tested corruptions, even when prompt and query share the same angle. This shows that matching their nominal angles does not remove the effects of rotating and resampling the query. GIST retains an advantage on Gaussian noise and a smaller one on JPEG compression under both angle relations; it remains worse on motion blur and fog. With independent angles, GIST's advantage over VICT shrinks from 0.0434 to 0.0283 AbsRel on Gaussian noise and from 0.0050 to 0.0032 on JPEG, consistent with a less reliable shared spatial reference. However, both methods' absolute errors are lower under independent angles in all four rows. The sampled rotations, resampling, and valid-pixel mask all change between conditions, so these scores do not establish that independent orientation is intrinsically easier or harder. The robust conclusion is that GIST's corruption-dependent ranking persists when query orientation also varies. For the same-angle condition, the VICT penalty relative to prompt-only VICT-R is +0.0766, +0.0769, +0.0860, and +0.0236 AbsRel for Gaussian noise, motion blur, fog, and JPEG, respectively. Thus the complete query--prompt rotation setting is substantially harder for VICT, while GIST still helps in the corruption categories where its spectral cue remains task-relevant.

How does GIST perform under scaling, translation, and flipping mismatches?

Appendix G.7 evaluates broader geometric mismatches by applying a horizontal flip, 1.25$\times$ center scale, or 12% horizontal translation to the one-shot support image and depth annotation while keeping the NYUv2-C query upright. VICT and GIST use the same 50 query identities for each of four severity-5 corruptions, with 10 adaptation updates per condition (Table G.9). Of the 12 paired comparisons, six 95% intervals favor VICT, one favors GIST, and five cross zero. The only clear GIST gain is Gaussian noise under scaling: VICT 1.345 versus GIST 1.218 AbsRel, a paired difference of +0.127 [0.067, 0.209]. VICT is better for all three defocus-blur transforms, with differences from -0.059 to -0.185. Rotation remains GIST's clearest demonstrated geometric benefit.

The graph measurements help explain this mixed result (Appendix G.7, Figure G.3). After undoing the grid permutation, flipping retains high feature-affinity correlation, kNN-edge Jaccard, and low-frequency projector similarity (0.995/0.759/0.956), but task-level gains are still mixed. Scaling and translation reduce those measures to 0.401/0.074/0.147 and 0.204/0.052/0.215, respectively. A graph cue must remain relevant to the corrupted query's depth evidence; graph similarity by itself does not guarantee improvement. Figure G.4 shows the paired effects and intervals for all 12 conditions.

Does the result extend to naturally varying viewpoints without synthetic rotation?

To examine the relevance of the synthetic-rotation setting to naturally acquired images, we also evaluate prompts with naturally varying viewpoints. For a natural-data complement, Appendix G.5 uses ScanNet, a dataset of real indoor RGB-D scans with naturally varying camera viewpoints, as the source of support prompts; NYUv2 supplies the depth-evaluation queries. We selected five ScanNet prompts with visually pronounced orientation differences, converted their depth values to the NYUv2 convention, and applied no synthetic image rotation. Across the five prompts, GIST-R improves on VICT-R every time; mean AbsRel falls from 0.400 to 0.381, a 4.6% relative reduction (Table G.7). The aligned VICT reference is 0.369, so the mismatch is mitigated rather than eliminated. Figure G.2 shows representative ScanNet views. Because this comparison also crosses datasets, its conclusion is narrower than a pure orientation effect: GIST improves performance with naturally oriented, cross-dataset supports, while the remaining gap may include both orientation and domain differences.

Reviewer_RASQOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes GIST, which builds on VICT’s test-time in-context adaptation and incorporates SRG and SAG modules to guide test-time weight updates, mitigating the effects of asymmetric perturbations in visual in-context learning. Extensive experiments demonstrate average improvements across four depth estimation and segmentation benchmarks, while ablation studies examine the contributions and scope of the proposed components.

Critical strengths
  1. The paper addresses a concrete and meaningful problem: failures of visual in-context learning caused by rotated demonstrations.
  2. The roles of the two modules are clearly defined, and ablation studies demonstrate their effectiveness.
Critical weaknesses
  1. The qualitative results primarily demonstrate the method’s effectiveness. The authors should also include cases where GIST fails to improve upon VICT or even degrades performance. Analyzing these cases in relation to rotation angles, corruption types, and prediction outputs would help clarify the method’s limitations.
  2. The gains in Table 1 vary across corruption types, with limited improvements or degradation in some settings. A brief discussion of these cases, including any shared image characteristics or error patterns, would improve understanding of the method’s behavior.
Questions for authors

See the weaknesses.

1 comment

We thank the reviewer for requesting failure-case and corruption-dependent analyses. We added paired predictions, regression frequencies, angle--error associations, and layer-16 spectral-bias maps; additions are marked in blue, with details in Appendix I.

What do failure cases reveal about rotation angle and prediction errors?

To examine when GIST fails rather than improves, we compare paired predictions and their errors under prompt-only rotation. Appendix I shows regressions and recoveries under prompt-only rotation. In the displayed regressions, VICT-to-GIST AbsRel changes from 0.192 to 0.213 for fog, 0.104 to 0.152 for elastic transform, and 0.309 to 0.332 for zoom blur. A second elastic-transform query improves from 0.182 to 0.172. The opposite outcomes under the same corruption show that scene structure affects whether the spectral relation remains useful.

How do gains and regressions vary across corruption types?

To assess corruption dependence beyond the displayed examples, we compare regression frequency and angle--error patterns across corruption types.

Regressions are most frequent for fog, elastic transform, and zoom blur; defocus blur, glass blur, and JPEG compression more often improve; brightness and noise are mixed. Fog reduces contrast, elastic transform displaces local boundaries, and zoom blur removes edge detail. The angle--error association varies by corruption, so rotation magnitude alone does not determine the result. For elastic transform, the layer-16 bias map overlaps the changed error regions, illustrating how a shifted spectral cue can weaken depth-relevant correspondences. Some corruptions retain moderate or high global bias similarity despite frequent regressions, showing that task-relevant boundary alignment matters more than global similarity alone.

The analysis identifies where GIST helps and where corruption-induced changes to task-relevant boundaries can make its spectral cue less useful. Further visualizations and corruption-level statistics are in Appendix I, Figures I.1--I.5.

Reviewer_gXYaOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper identify a issue in Visual in-context learning (VICL) that the performance degrades by rotating the prompt image and its annotations (e.g. a RGB image and a depth map). To address this issue, this paper proposes a test-time adaptation method called GIST (Graph-Informed Spectral Tuning) with two major components, Spectral Relation Guidance (SRG) and Adaptation Guidance (SAG). SRG improves token interactions between query prediction and support reconstruction. SAG provides weights to patches for the support reconstruction. Experiments shows in terms of depth estimation and semantic segmentation, GIST improves upon the prior method VICT with a rotation on the prompt images and annotations.

Critical strengths
  1. This paper identify prior VICL method is senetive to rotation in visual prompts

  2. The proposed method shows some improvement when the rotation is applied

Critical weaknesses
  1. The motivation of this paper is unclear. The reviewer cannot understand why we will rotate the visual prompt.

  2. This paper failed to explain how SRG and SAG solve the problems with rotation and why they are the best choice. Based on me, if a model is sensitive to rotation, we should at least try a simple data augmentation as baseline.

  3. The evaluation is not sufficient. It only consider two specific tasks, depth estimation and sematic segmentation. How about image restoration (deblur, dehazing), style transfer, image editing, etc.

  4. The writing needs to improve. It does not provide the enough contexts before making cliams or statements, e.g. SRG and SAG in Abstract (no idea about what they are), question in Line 173, question in Line 223.

Questions for authors

See Weaknesses

4 comments

We thank the reviewer for these constructive comments. We clarified the mechanism and added test-time baselines and task evaluations; additions are marked in blue, with details in Appendices E and G.

Why would the support prompt be rotated relative to the query?

In visual in-context learning, a support image and its annotation provide both the task definition and a spatial reference for transferring the demonstrated image--annotation relation to a query. A stored support can come from a camera or platform with a different orientation from a later query, for example in underwater imaging, aerial imaging, or endoscopy. Rotating the support image and annotation together preserves their task semantics, but changes the support's spatial reference while the query remains upright. The model must therefore transfer a valid demonstration across an orientation mismatch.

This is different from rotating the complete input. Our protocol keeps the query fixed and transforms only the support pair, so it directly measures the asymmetric prompt--query mismatch. The clean-query control in Table G.1 confirms that this mismatch alone raises VICT AbsRel by 0.0181 on NYUv2 and lowers VICT mIoU by 1.10 points on ADE20K. The matched padding control is less damaging than full rotation. In the feature diagnostic, matched-patch cosine is 0.7666 for rotated content versus 0.9901 for padding alone, showing that rotation changes interior features and graph relations in addition to boundary fill in the examined prompt.

What distinguishes GIST from VICT in this rotation setting?

Our contribution is threefold.

  1. A distinct evaluation setting. We formulate prompt-only orientation mismatch: the support demonstration remains semantically valid, but its spatial reference changes relative to an upright query. This isolates a failure mode that whole-input rotation robustness does not measure.
  2. Two targeted interventions inside VICT. GIST keeps VICT's label-free support-reconstruction cycle. SRG changes the forward interaction by adding low-frequency graph-subspace distances as a pre-softmax attention bias in query prediction and support reconstruction. SAG changes the adaptation signal by weighting support-reconstruction residuals with a fixed support-graph resolvent, where the added weight is determined by the preceding residual's spectral composition. SRG controls which token relations exchange information; SAG controls how the reconstruction error updates the encoder. Neither uses query labels, an additional trainable module, or rotation-specific backbone pretraining.
  3. Evidence that separates the two design choices. In the rotated one-shot NYUv2-C ablation, VICT-R gives 0.342 AbsRel, adding SRG gives 0.289, adding SAG gives 0.295, and the full method gives 0.251. With SAG disabled and only the attention bias varied, raw feature affinity and a magnitude-matched random bias worsen both tasks, whereas SRG reduces NYUv2-C AbsRel from 0.375 to 0.330 and raises ADE20K-C mIoU from 27.5 to 28.4. These controls show that the gain is not produced by an arbitrary extra logit term or by any feature similarity. The method is a test-time relational adaptation method, not a globally rotation-equivariant representation or a canonicalization procedure.

How do SRG and SAG address rotation mismatch?

The proposed components answer two different questions in VICT's cycle. SRG asks which token relations should guide each completion. It rebuilds a temporary feature-affinity graph from the current stream, computes a low-frequency Laplacian subspace, and adds pairwise distances in that subspace to the pre-softmax attention logits. It is recomputed for the forward query prediction and the reverse support reconstruction, so it changes information exchange at both points where prompt--query relations are used. SAG asks how the known support-reconstruction error should update the model. It caches a graph from the observed support image, projects the support residual onto that graph, and increases the reconstruction weight when the preceding residual contains energy in the retained nontrivial modes. SRG therefore changes the interaction pattern, while SAG changes the update metric; they intervene at different stages of the same label-free cycle.

The mechanism analysis in Section 4.6 provides the rotation-specific link. Rotation reduces raw relation-row cosine to 0.03 at 120∘120^\circ, while the low-frequency relation-row cosine remains 0.62 and matched-node subspace affinity remains 0.768, above the 0.103 grouped-shuffle null. SRG raises cross-angle attention-row cosine from 0.32 to 0.45 and reduces the fraction of queries with increased error from 63% to 28%. Appendix H reports large-angle measurements on 100 paired episodes, with detailed visualizations in Figures H.1--H.2. The equal-MSE diagnostic in Appendix E.11 (Figure E.5) makes the SAG weighting explicit: with the same ordinary MSE, a low-frequency residual receives 3.5649 mean per-node spectral addition while a high-frequency residual receives approximately zero. The component ablation and attention-bias controls provide the downstream test of this design; the diagnostic explains the update weighting.

How does training-time rotation augmentation relate to this TTA comparison?

VICT-R is our direct test-time adaptation baseline, applying VICT to rotated prompts with the same fixed checkpoint. Training-time rotation augmentation would require an additional training or fine-tuning stage, outside this fixed-checkpoint TTA comparison. We did not evaluate a training-time rotation-augmented checkpoint.

Does the method extend beyond depth and segmentation?

To examine the reviewer's task-scope concern, we added restoration controls under the same fixed-checkpoint test-time protocol (Table G.3):

TaskVICT alignedVICT-RGIST-R
SIDD-C denoising, one-shot PSNR (↑)24.5624.5924.66
Rain-C deraining, one-shot PSNR (↑)19.9219.7019.94
LOL-C low-light enhancement, one-shot PSNR (↑)16.9416.6016.65

The restoration result is task-dependent. Denoising changes little under prompt rotation (24.56 dB for aligned VICT and 24.59 dB for VICT-R), while deraining gives a 0.24 dB GIST-R gain over VICT-R and low-light enhancement recovers 0.05 dB while remaining below aligned VICT. These controls show that rotation is a weaker diagnostic for tasks whose prediction relies less on global spatial correspondence, so we prioritized depth and segmentation where the prompt's spatial reference directly structures the output. They also show that GIST's benefit outside those tasks is measurable but modest and task-dependent. Deblurring, dehazing, style transfer, and image editing were not evaluated; the demonstrated scope is therefore depth, segmentation, and the three added restoration controls.

Where was the paper's exposition revised?

We revised the exposition at the points raised by the reviewer. The abstract now defines SRG and SAG before naming their roles. The Introduction and Figure 1 add concrete orientation-mismatch scenarios and explain why the query is held fixed. Sections 3.2--3.3 now introduce the two questions each component answers before giving the equations: SRG selects graph-relative token interactions, and SAG weights the support-reconstruction update. Section 4.4 places the raw-affinity and random-bias controls immediately after the component ablation, and Section 4.6 presents the rotation-conditioned mechanism analysis. Appendices G--I provide the added baselines, angle conditions, failure examples, and visualizations referenced by these claims.

In addition, we would like to emphasize the rationale for omitting rotation augmentation from our main adaptation protocol. Our objective is to evaluate how effectively the model adapts to the observed prompt–query orientation mismatch using the available support pair and an unlabeled query. This follows VICT’s evaluation rationale: Section 4.1 deliberately excludes data augmentation during test-time adaptation to examine generalization to unseen distribution shifts. Maintaining this protocol provides a controlled comparison in which we can isolate the contribution of SRG and SAG within the same support-reconstruction cycle.

We also compare simple geometric test-time strategies in Appendix E.6, Table 14. On the same fixed one-shot subsets, rotate-and-average achieves 0.369 AbsRel / 32.6 mIoU on NYUv2-C / ADE20K-C at 4.60× runtime, while estimated canonicalization achieves 0.358 / 32.7 at 1.95×. GIST achieves 0.330 / 33.2 at 1.40×. These comparisons support the benefit of guiding adaptation under the observed mismatch beyond the tested geometric strategies.

Dear Reviewer gXYa,

With less than three hours remaining in the discussion period, could you kindly review our responses and share your updated assessment?

Our responses clarify the motivation for prompt–query orientation mismatch, explain how SRG and SAG guide adaptation, and provide the rationale for our augmentation protocol following VICT. We also report comparisons with simple geometric test-time strategies and additional restoration-task evaluations addressing your questions about baselines and task coverage.

Regarding your interest in evaluation beyond the original tasks, our response to Reviewer Qjfg additionally reports MAE-VQGAN experiments on Taskonomy surface normal estimation and DUTS foreground segmentation, extending the evaluation to additional tasks and datasets with a second VICL model.

We would appreciate your feedback on whether these clarifications and experiments address your concerns, and respectfully ask you to reconsider your rating accordingly.

Thank you for your time and consideration.

Best regards,
The Authors

Reviewer_QjfgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the performance drop in Visual In-Context Learning when visual prompts are rotated. The authors propose GIST , a test-time adaptation method combining graph-based attention guidance (SRG) and adaptive spectral reconstruction loss (SAG). Experiments across four benchmarks demonstrate improved robustness over VICT.

Critical strengths
  1. The paper presents a clear motivation by identifying an important and underexplored robustness issue in visual in-context learning.
  2. The proposed method is well-designed, combining graph-based attention guidance with adaptive spectral reconstruction loss.
  3. The paper demonstrates strong experimental results, showing consistent average improvements across multiple datasets and tasks.
Critical weaknesses

1 The authors demonstrate that GIST is effective under the tested rotation conditions, but they have yet to fully establish a causal explanation linking rotation mismatch to graph-spectral characteristics and, subsequently, to improved adaptation.

2 Generalization across prompts and backbones is not adequately established. Can authors evaluate multiple randomly selected support prompts and quantify performance variance across prompt identities

3 The instability of spectral subspaces under large rotations, with subspace affinity dropping to 0.524 at 120°, raises concerns about the effectiveness of spectral guidance.

Questions for authors

See weaknesses

6 comments

We thank the reviewer for these constructive comments. We added support-prompt identity tests and spectral mechanism analyses; additions are marked in blue, with details in Appendices E, G, and H.

How does rotation mismatch affect graph-spectral relations and adaptation?

To address the proposed link from rotation mismatch to graph-spectral relations and then to adaptation, we compare relation, attention, and error measurements under matched rotation settings. The main-text analysis examines representative NYUv2-C episodes where GIST improves over VICT, with the query upright and the support pair rotated from 0° to 120° (Section 4.6, Figures 5--6). At 120∘120^\circ, raw relation-row cosine is 0.03, compared with 0.62 for low-frequency spectral relations; matched-node subspace affinity is 0.768 against a grouped-shuffle null of 0.103. Adding SRG raises cross-angle attention-row cosine from 0.32 to 0.45, and the fraction of these analyzed episodes whose error increases falls from 63% with VICT to 28% with GIST.

We also tested whether the benefit comes from the spectral construction itself. In the controlled attention-bias comparison, all settings are held fixed while only the bias changes: raw feature affinity and a magnitude-matched random bias worsen performance (NYUv2-C AbsRel 0.392/0.393 vs. 0.375 with no bias; ADE20K-C mIoU 26.7/26.4 vs. 27.5), while SRG improves it to 0.330 AbsRel and 28.4 mIoU (Table 4). The component ablation independently shows that SRG and SAG each improve rotated-prompt adaptation, and their combination is best: VICT-R 0.342 AbsRel, +SRG 0.289, +SAG 0.295, and full GIST 0.251 (Table 2).

Together, these results link rotation-induced relation changes to SRG-guided attention and show the separate contribution of SAG to the reconstruction update. Appendix H reports the large-angle measurements over 100 paired episodes, with relation, attention, and error visualizations in Figures H.1--H.2. Appendix E.11 (Figure E.5) illustrates SAG's frequency-dependent weighting.

Does the improvement hold across randomly selected support prompts?

To test whether performance depends on support-prompt identity, we compare the same methods and queries across randomly selected prompts. We randomly selected five distinct prompts from the NYUv2 training split (P1--P5) and evaluated them in the one-shot NYUv2-C setting. These are random support-prompt samples, not a manually fixed set of representative prompts. Each sampled prompt was evaluated on 500 query identities for each of the 15 corruption types. Tables G.4--G.6 report prompt-level averages and standard deviations across all 15 types. For each prompt, its identity was held fixed across methods, along with the query set, rotation distribution, adaptation budget, and model checkpoint.

MetricVICT (aligned)VICT-R (rotated)GIST (rotated)
AbsRel (↓)0.4301 ± 0.03260.4915 ± 0.02400.4368 ± 0.0270
RMSE (↓)1.1437 ± 0.07041.2665 ± 0.05331.1564 ± 0.0587
δ1\delta_1 (↑)0.4921 ± 0.01120.4643 ± 0.01020.4932 ± 0.0122

GIST improves over VICT-R for all five prompt identities and all three metrics. The AbsRel mean decreases from 0.4915 to 0.4368, RMSE from 1.2665 to 1.1564, and δ1\delta_1 increases from 0.4643 to 0.4932. The sign of the improvement therefore does not depend on the selected support identity in this study. The spread is not reduced: GIST has larger prompt-level SD than VICT-R for AbsRel, RMSE, and δ1\delta_1. Thus the experiment establishes prompt-level consistency of the mean improvement, not variance reduction. Compared with aligned VICT, GIST's mean AbsRel and RMSE are higher by 0.0067 and 0.0127, respectively, while its mean δ1\delta_1 is higher by 0.0011.

Does spectral guidance remain effective when affinity falls to 0.524 at 120°?

The 0.524 affinity is from the ADE20K-C initialization diagnostic (Appendix E.5, Table 11). To test whether spectral guidance remains useful after this large-angle degradation, we added a downstream mechanism analysis on 100 paired NYUv2-C query–prompt episodes (Appendix H, Figures H.1–H.2): each query remains upright while its support image and annotation are rotated together from 0∘0^\circ to 120∘120^\circ, and we measure graph relations, attention, and query error. At 120∘120^\circ, raw relation-row cosine falls to 0.03, while low-frequency spectral-row cosine remains 0.62; matched-node subspace affinity is 0.768, above the grouped-shuffle null of 0.103. Raw and spectral top-16 neighbor retention are 0.75 and 0.55, respectively. After SRG is applied, cross-angle attention-row cosine rises from 0.32 with VICT to 0.45, and the fraction of queries whose error increases falls from 63% to 28%. This experiment shows that residual low-frequency structure remains distinguishable from the null and that SRG changes attention and error outcomes in the same large-angle setting.

We also report task-level performance over a deterministic ADE20K-C prompt-angle sweep from 0∘0^\circ to 360∘360^\circ (Appendix E.4, Figure E.2). GIST-R's mIoU is above VICT-R throughout the displayed range. The four 90∘90^\circ interval summaries in Appendix E.8 (Figure E.3) are 14.7 / 12.0, 13.9 / 11.7, 12.6 / 10.1, and 14.8 / 11.8 mIoU for GIST-R / VICT-R. Thus the new 100-episode analysis tests the spectral mechanism at 120∘120^\circ, while the full-angle sweep shows that GIST-R's task-level advantage persists across all four angular intervals.

Thank you for your detailed response and additional experiments. While the new results partially address my concerns, the causal mechanism and cross-backbone generalization remain insufficiently established. Therefore, I will maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.