- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies prompt-only rotation in visual in-context learning, where a prompt image and its annotation are rotated together while the query remains unchanged. Building on VICT’s test-time adaptation cycle for Painter, the authors propose GIST, which adds (i) Spectral Relation Guidance, a pre-softmax attention bias derived from distances in the low-frequency Laplacian eigenspace of token affinity graphs rebuilt at selected layers, and (ii) Spectral Adaptation Guidance, a reconstruction metric that emphasizes low-frequency residual components on a fixed support-image graph, with its weight determined by the previous residual’s spectral composition. The method is evaluated on NYUv2-C, ADE20K-C, Hamlyn-C, and SUIM-C under 15 corruptions, with clean (“zero-shot”) and corrupted (“one-shot”) prompts and long-tailed random rotations. It reports 9.9–27.3% lower AbsRel and 0.40–1.14 higher mIoU points than VICT under matched rotations, supported by component ablations, attention-bias controls, padding-only controls, geometric baselines, and spectral diagnostics.
Critical strengths
1. An interesting and well-defined problem setting. The paper highlights that a visual prompt both specifies the task and provides a spatial reference for query prediction. It formulates prompt-only rotation as an asymmetric mismatch that preserves the demonstrated task while changing the prompt’s geometry. This differs from standard whole-input rotation robustness, and the padding-only control shows that boundary fill alone does not explain the observed degradation. In my view, identifying and characterizing this setting is the paper’s most original contribution and could motivate broader work on the spatial robustness of visual prompts.
2. Consistent aggregate gains with informative controls. Averaged over corruptions, GIST improves over VICT under matched rotations on all four datasets and in both prompt settings, without introducing additional trainable modules and at approximately 1.4× VICT’s runtime. The attention-bias controls on NYUv2-C and ADE20K-C show that raw feature affinity and a magnitude-matched random bias both hurt performance, whereas the spectral bias helps. On the controlled geometric-baseline subsets, GIST also outperforms the tested non-oracle geometric alternatives at lower computational cost. These results support the method’s effectiveness in the evaluated setting, although the extent to which the gains specifically reflect improved robustness to rotation remains unclear.
Critical weaknesses
1. The experiments do not clearly separate general adaptation gains from gains in robustness to prompt rotation. GIST’s results on corrupted queries with unrotated prompts are not systematically tabulated across the four datasets and both prompt settings. Figure E.2 includes a 0° point for ADE20K-C, but the corresponding scores and method differences are not explicitly analyzed. Table 3 evaluates clean, unrotated inputs and therefore does not provide the missing corruption-only comparison. This comparison is important for determining whether GIST reduces the performance penalty caused by rotation or mainly improves adaptation more generally. For example, on NYUv2-C zero-shot, GIST with rotated prompts outperforms VICT with unrotated prompts (0.330 vs. 0.365, Table 1), exceeding the improvement needed to recover VICT’s rotation-induced loss. This is encouraging, but it also makes the missing comparison particularly relevant. In addition, although the paper acknowledges that synthetic rotation combines geometric changes with resampling artifacts, the relevance of this compound shift to naturally occurring orientation mismatch remains insufficiently validated.
2. The rotation-specific rationale for spectral guidance remains insufficiently established. The paper appropriately presents spectral structure as a soft cue rather than claiming exact rotation invariance. Nevertheless, the connection between its spectral diagnostics and the observed task-level gains remains unclear. In the ADE20K-C initialization diagnostics, raw-feature neighborhoods have higher retention under rotation than spectral neighborhoods (Tables 11–12). These diagnostics use near-zero-aware mode selection, rather than the implemented fixed-index policy. Higher neighborhood retention does not imply greater usefulness as an attention bias, and Table 4 does provide downstream evidence favoring the spectral construction. However, the results do not establish that its benefit comes from better preservation of relationships under rotation. Furthermore, annotation- and fused-stream SRG graphs are disconnected in approximately 90–100% of cases in the ADE20K-C initialization diagnostic (Table 9), raising questions about the downstream sensitivity of the fixed mode-selection policy. The formal analysis establishes conditional permutation equivariance under ideal node relabeling, not a guarantee for physical image rotation. Similarly, the SAG diagnostics illustrate frequency-selective weighting without establishing why rotation-induced reconstruction errors should particularly benefit from that weighting. Thus, the components have empirical support, but their specific connection to the proposed rotation mismatch needs stronger evidence.
Questions for authors
-
Could you report GIST and VICT on corrupted queries with unrotated prompts for all four datasets and both prompt settings? Please also compare the rotation-induced performance degradation of each method relative to its own unrotated baseline. If comparable GIST–VICT gains are already present without rotation, this would weaken the evidence that GIST provides additional robustness to prompt rotation beyond a general improvement in test-time adaptation.
-
When rotation is the only applied transformation, with no additional benchmark corruptions on either the query or prompt, how much does prompt rotation degrade VICT, and how much does GIST recover? Table 3 evaluates clean inputs without prompt rotation and therefore does not isolate this condition.
-
The annotation- and fused-stream SRG graphs are almost always disconnected in the ADE20K-C initialization diagnostic (Table 9), which the paper attributes partly to mask/label structure and support–query grouping. Table 10 shows that skipping near-zero modes changes the initialization diagnostics only modestly, but does this also hold for downstream performance after adaptation? It would also be informative to evaluate SRG applied only to the layer-0 image stream, whose graphs are mostly connected. The layer-0 ablation in Table 8 includes both image and annotation streams and does not isolate this alternative.
-
How is the angle inferred in the estimated-canonicalization baseline in Table 14? Please specify the estimator or search procedure and the selection criterion. It would also help to clarify which intended deployment settings lack usable relative-orientation information, and how the comparison would change when relevant orientation metadata are available.
-
Query orientation may also vary in the motivating applications. Could you evaluate cases where the query is rotated as well, either by the same angle as the prompt or by an independently sampled angle? This would clarify how far the findings extend beyond the controlled prompt-only setting.
-
Since neither component is specific to rotation, additional tests involving prompt–query mismatches such as scaling, translation, and flipping would help establish the method’s broader scope. Naturally acquired pairs with orientation differences, without an added synthetic rotation, would also help assess the relevance of the evaluation protocol. As a further extension, robustness to noisy prompt annotations would be informative, since both VICT and GIST use the prompt annotation as their self-supervision signal.
4 comments
Do disconnected SRG graphs affect downstream mode selection and image-only guidance?
The reviewer identifies a concrete risk in the graph construction: if a graph has multiple connected components, the implemented fixed indices can include extra zero modes. We examined this at initialization on an ADE20K-C one-shot subset spanning all 15 corruptions and eight signed rotations (Appendix E.3, Tables 9--10). The temporary SRG graphs and fixed SAG graph play different roles. Layer-0 image SRG graphs are disconnected in 6.7% of records, annotation graphs in 100%, and layer-16 fused graphs in 90.4% (forward) and 99.3% (reverse); the fixed support-loss graph is connected in every record. Disconnection is therefore concentrated in the temporary attention cues, not the graph defining the SAG reconstruction metric.
Painter’s stream structure explains where the disconnected graphs arise: annotation tokens encode masks and labels, while fused tokens include support–query grouping. The initialization analysis in Appendix E.5, Table 13 shows that disconnection does not eliminate cross-rotation subspace correspondence: the forward/reverse annotation graphs have affinities 0.683/0.825, while the fixed support-loss graph is connected. Across angles, affinity decreases from 0.785 at 5° to 0.524 at 120° (Table 11), and varies by corruption from 0.440 for impulse noise to 0.793 for defocus blur (Table 12). These results characterize the spectral cue as angle- and corruption-dependent.
To test whether mode selection affects adapted predictions, Appendix J adds a downstream ADE20K-C one-shot experiment (Table J.1). It evaluates Gaussian noise, defocus blur, snow, and JPEG compression, using 500 shared query identities, the same checkpoint and optimizer, and 60 adaptation updates under rotated prompts. The fixed policy averages 20.381 mIoU; skipping all modes with and selecting the next eight averages 20.518, a +0.137-point difference. Per-corruption changes are +0.792, +0.090, −0.610, and +0.274. Thus the two policies have nearly equal mean downstream performance, with mixed per-corruption effects; zero-aware selection provides no consistent advantage, and the fixed policy shows no systematic loss from the extra near-zero modes.
Appendix J also tests the layer-0 image-only variant requested by the reviewer (Table J.2). With the same four corruptions, 500 queries, 60 updates, and SAG retained, removing the annotation-stream SRG bias changes mIoU by +0.005, +0.801, +0.690, and +0.075, respectively. The image-only mean is 20.894 versus 20.501 for layer-0 image-plus-annotation, a +0.393-point gain; it is also 0.513 points above the full fixed-mode configuration. On this evaluation, the connected image stream alone is sufficient for the measured SRG benefit, while adding the annotation-stream bias gives no further gain. Together, Tables J.1–J.2 show that the fixed mode policy retains its downstream performance and that image-only layer-0 guidance is an effective option in the tested rotated ADE20K-C conditions.
How is the canonicalization angle estimated, and when is metadata available?
To make the angle-estimation baseline reproducible and answer the selection-criterion question, we specify the image-processing steps used to obtain its estimate.
For the estimated-canonicalization baseline, we convert the rotated support RGB image to grayscale, compute a Canny edge map, and extract straight-line segments with a Hough line transform. We discard short segments, calculate each retained segment's orientation from its endpoints, and measure its signed deviation from the nearest horizontal or vertical axis. We use the robust length-weighted median of these deviations as the estimated tilt, then rotate the support RGB image and annotation together by the negative estimate before applying VICT. This line-based estimate recovers tilt relative to a horizontal or vertical axis, but cannot reliably resolve the full signed orientation for rotations greater than 90°. It can therefore select an incorrect rotation branch and leave the prompt incompletely canonicalized, which helps explain why this baseline underperforms the oracle (Table 14). The oracle instead uses the recorded synthetic angle directly.
On the fixed 200-sample-per-corruption subsets, the results are:
| Method | NYUv2 AbsRel (↓) | ADE20K mIoU (↑) | Relative time |
|---|---|---|---|
| VICT-R | 0.395 ± 0.005 | 32.3 ± 0.3 | 1.00× |
| Rotate-and-average | 0.369 ± 0.006 | 32.6 ± 0.2 | 4.60× |
| Estimated canonicalization | 0.358 ± 0.004 | 32.7 ± 0.5 | 1.95× |
| Oracle inverse rotation | 0.302 ± 0.003 | 33.9 ± 0.3 | 1.27× |
| GIST-R | 0.330 ± 0.003 | 33.2 ± 0.2 | 1.40× |
When reliable relative-orientation metadata are available, direct inverse correction is possible. The oracle reaches 0.302 AbsRel and 33.9 mIoU, compared with 0.330 and 33.2 for GIST-R. The estimated row represents deployment without such metadata, as can occur when a reusable support and a live query come from differently oriented devices or sessions with no synchronized orientation record. Among methods without the generated angle, GIST is best on both datasets and is substantially cheaper than rotate-and-average.
What happens when the query also rotates with or independently of the prompt?
We study the orientation relation between the prompt and query, rather than assuming that queries must always be upright. To test the reviewer's setting, Appendix G.6 (Table G.8) jointly rotates each NYUv2-C query and its depth target, together with the one-shot support image and annotation. The query and support either share a sampled angle (their relative orientation is preserved) or draw independent angles from the same distribution (their relative orientation varies). We evaluate four severity-5 corruptions with 654 queries per condition, excluding padded and invalid-depth pixels from AbsRel.
| Corruption | Prompt-only VICT-R | Both rotated: same angle VICT / GIST | Both rotated: independent angles VICT / GIST |
|---|---|---|---|
| Gaussian noise | 0.361 | 0.4376 / 0.3942 | 0.4008 / 0.3725 |
| Motion blur | 0.122 | 0.1989 / 0.2012 | 0.1935 / 0.1991 |
| Fog | 0.143 | 0.2290 / 0.2595 | 0.2256 / 0.2475 |
| JPEG compression | 0.272 | 0.2956 / 0.2906 | 0.2898 / 0.2866 |
Rotating the query as well increases VICT's error relative to the prompt-only scores on all four tested corruptions, even when prompt and query share the same angle. This shows that matching their nominal angles does not remove the effects of rotating and resampling the query. GIST retains an advantage on Gaussian noise and a smaller one on JPEG compression under both angle relations; it remains worse on motion blur and fog. With independent angles, GIST's advantage over VICT shrinks from 0.0434 to 0.0283 AbsRel on Gaussian noise and from 0.0050 to 0.0032 on JPEG, consistent with a less reliable shared spatial reference. However, both methods' absolute errors are lower under independent angles in all four rows. The sampled rotations, resampling, and valid-pixel mask all change between conditions, so these scores do not establish that independent orientation is intrinsically easier or harder. The robust conclusion is that GIST's corruption-dependent ranking persists when query orientation also varies. For the same-angle condition, the VICT penalty relative to prompt-only VICT-R is +0.0766, +0.0769, +0.0860, and +0.0236 AbsRel for Gaussian noise, motion blur, fog, and JPEG, respectively. Thus the complete query--prompt rotation setting is substantially harder for VICT, while GIST still helps in the corruption categories where its spectral cue remains task-relevant.
How does GIST perform under scaling, translation, and flipping mismatches?
Appendix G.7 evaluates broader geometric mismatches by applying a horizontal flip, 1.25$\times$ center scale, or 12% horizontal translation to the one-shot support image and depth annotation while keeping the NYUv2-C query upright. VICT and GIST use the same 50 query identities for each of four severity-5 corruptions, with 10 adaptation updates per condition (Table G.9). Of the 12 paired comparisons, six 95% intervals favor VICT, one favors GIST, and five cross zero. The only clear GIST gain is Gaussian noise under scaling: VICT 1.345 versus GIST 1.218 AbsRel, a paired difference of +0.127 [0.067, 0.209]. VICT is better for all three defocus-blur transforms, with differences from -0.059 to -0.185. Rotation remains GIST's clearest demonstrated geometric benefit.
The graph measurements help explain this mixed result (Appendix G.7, Figure G.3). After undoing the grid permutation, flipping retains high feature-affinity correlation, kNN-edge Jaccard, and low-frequency projector similarity (0.995/0.759/0.956), but task-level gains are still mixed. Scaling and translation reduce those measures to 0.401/0.074/0.147 and 0.204/0.052/0.215, respectively. A graph cue must remain relevant to the corrupted query's depth evidence; graph similarity by itself does not guarantee improvement. Figure G.4 shows the paired effects and intervals for all 12 conditions.
Does the result extend to naturally varying viewpoints without synthetic rotation?
To examine the relevance of the synthetic-rotation setting to naturally acquired images, we also evaluate prompts with naturally varying viewpoints. For a natural-data complement, Appendix G.5 uses ScanNet, a dataset of real indoor RGB-D scans with naturally varying camera viewpoints, as the source of support prompts; NYUv2 supplies the depth-evaluation queries. We selected five ScanNet prompts with visually pronounced orientation differences, converted their depth values to the NYUv2 convention, and applied no synthetic image rotation. Across the five prompts, GIST-R improves on VICT-R every time; mean AbsRel falls from 0.400 to 0.381, a 4.6% relative reduction (Table G.7). The aligned VICT reference is 0.369, so the mismatch is mitigated rather than eliminated. Figure G.2 shows representative ScanNet views. Because this comparison also crosses datasets, its conclusion is narrower than a pure orientation effect: GIST improves performance with naturally oriented, cross-dataset supports, while the remaining gap may include both orientation and domain differences.
We thank the reviewer for these constructive comments. We added aligned-prompt controls, downstream graph-mode and stream ablations, rotated-query and geometric-mismatch tests, and a natural-viewpoint evaluation; additions are marked in blue, with details in Appendices E, G, and J.
Do the gains reflect rotation robustness beyond general adaptation?
To distinguish the effect of prompt rotation from each method's general adaptation benefit, we compare aligned and rotated prompts on the same corrupted queries. We added the missing aligned-prompt controls to Table G.2 for four severity-5 NYUv2-C corruption representatives: Gaussian noise, motion blur, fog, and JPEG compression. We use the same query identities, checkpoint, 60-step adaptation budget, and matched rotation draws. In zero-shot, the support prompt is clean; in one-shot, it has the corresponding corruption. VICT-0/GIST-0 denote unrotated prompts, and VICT-R/GIST-R denote rotated prompts. We report each method's rotation penalty, Δ=R−0, and D=ΔVICT−ΔGIST; positive D means GIST loses less (or gains more) from prompt rotation than VICT.
Zero-shot (AbsRel ↓; clean support prompt):
One-shot (AbsRel ↓; support prompt has the corresponding corruption):
The new controls show some, but limited and corruption-dependent, adaptation benefit. In zero-shot, the aligned GIST-0 comparison is mixed: GIST is slightly better than VICT-0 on motion blur and JPEG, and slightly worse on Gaussian noise and fog. After prompt rotation, GIST-R is below VICT-R on all four rows, but the relative rotation effects D are small for Gaussian noise, motion blur, and fog (+0.018, +0.004, +0.006); JPEG is the clear exception (+0.063). Importantly, against the unrotated VICT-0 baseline, GIST-R is still worse on Gaussian noise, motion blur, and fog, and better only on JPEG. Thus lower error than rotated-prompt VICT in these rows does not by itself establish a broad adaptation gain.
In one-shot, GIST-0 has higher error than VICT-0 in all four rows. GIST-R beats VICT-R only for Gaussian noise and JPEG, while it is worse for motion blur and fog; GIST-R remains worse than aligned VICT-0 in all four. The largest relative rotation benefit is Gaussian noise, followed by JPEG; fog shows a small difference, and motion blur provides no benefit. Therefore, these controls do not show a general one-shot TTA improvement or complete recovery of the rotation penalty.
The full 15-corruption NYUv2-C zero-shot results give useful context. GIST-R is lower than VICT-R in 10/15 corruption types, but lower than unrotated VICT in only 6/15. The mean AbsRel is 0.330 for GIST-R, 0.375 for VICT-R, and 0.365 for VICT-0. Defocus blur (0.672 to 0.416), glass blur (0.533 to 0.389), and JPEG (0.342 to 0.258) account for about 72% of the GIST-R--VICT-R mean gap. As a descriptive check, excluding these three gives 12-corruption means of 0.3398 for VICT-R, 0.3243 for GIST-R, and 0.3283 for VICT-0: GIST-R and unrotated VICT are close, with only a 1.2% difference, while the remaining rotated-baseline difference is 4.6%. This shows that selected corruption gains substantially shape the aggregate and should not be read as uniform adaptation improvement.
How much does prompt rotation alone degrade VICT, and how much does GIST recover?
To isolate prompt rotation from benchmark corruptions, we compare clean-input results with aligned and rotated support pairs.
Table G.1 removes all benchmark corruptions and rotates only the support pair:
Prompt rotation alone causes a 56% smaller AbsRel increase on NYUv2 and a 66% smaller mIoU loss with GIST. GIST also slightly exceeds VICT in the rotated condition on both datasets, despite starting from the stronger aligned VICT score.