acceptodds
Under review as a conference paper at ICLR 2027

Prior Directions: Separating Geometric Identification from Behavioral Efficacy in Visual Revision

Abstract

Vision-language models can remain anchored to stale verbal priors even when current visual evidence supports a different answer. We use this controlled revision failure to ask whether the subspace that best captures recurrent prior-induced representation change is also uniquely more behaviorally effective. Across four models, paired prior and reference states define Prior Directions, a compact low-rank geometry that, at rank 32, spans only 0.78% of the state dimensions yet captures 49.7% to 64.0% of held-out displacement energy. Matched correspondence improves held-out capture at every tested rank in all four models; across 20 permutation seeds, this advantage remains positive in all 15 available model–rank settings. Yet the geometric advantage does not translate into a comparable intervention advantage: matched and Pairing-Permuted edits restore 60/66 and 59/66 clean prior-induced failures, respectively. We trace this discrepancy to the geometry of the realized edits. Pairing-Permuted edits remain strongly aligned with the matched Prior Directions; their aligned component restores 59/66 cases, whereas an equally norm-matched orthogonal residual restores only 20/66. These results show that the construction that best identifies recurrent representation change need not be uniquely more behaviorally effective, distinguishing global subspace identification from the geometry actually used by an intervention.

Primary area:
interpretability and explainable AI
Keywords:
Vision-Language Models, Representation Geometry, Interpretability, Cross-Modal Conflict

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_GbvJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether low-rank “Prior Directions” that capture prior-induced representation changes in vision-language models are also uniquely effective at correcting failures to update stale verbal priors using visual evidence. Across four models, matched prior–reference pairings capture held-out representation changes better than permuted pairings, yet their interventions achieve nearly identical recovery rates. Further analysis shows that permuted edits remain strongly aligned with Prior Directions, demonstrating that superior subspace identification does not necessarily translate into a corresponding behavioral advantage.

Critical strengths

The visual revision testbed is tightly controlled, with the screenshot, query, and candidate answers held fixed while only the verbal context changes. This design makes it possible to test whether a stale location cue can pull an initially correct visual judgment toward the designated outdated location. Observing the resulting visual lock-in behavior across four VLM families gives the representation analysis a solid behavioral anchor. Just as valuable is the separation between geometric identification and behavioral efficacy. The matched-correspondence Prior Directions capture held-out representation changes better, yet the matched and Pairing-Permuted interventions restore 60/66 and 59/66 of the clean lock-in failures, nearly identical. The rank-matched random subspaces, norm-matched random removal, and the aligned-versus-orthogonal decomposition all point the same way, suggesting that what the Pairing-Permuted intervention retains comes mainly from the shared orientation of its edits. Having these controls in place keeps the behavioral story from resting on subspace fit quality alone.

Critical weaknesses

The behavioral conclusion rests on a narrow slice of the experimental space. The claim that better geometric identification brings no corresponding behavioral advantage is drawn from 66 clean failures at rank 16, intervention strength 0.25, a fixed layer range, one data split, and a single Pairing-Permuted construction. At this operating point both interventions already restore most failures (60/66 vs 59/66), so the evidence establishes similar restoration counts at this one setting; it does not yet tell us whether the relationship holds under other intervention settings.

Questions for authors

The geometric identification analysis sweeps several subspace ranks, but the behavioral comparison is reported only at k = 16, λ = 0.25, and L_late = {24, ..., 36}. How were these values chosen — best restoration, or representative? And does the near-equality between the two interventions persist across a few other (k, λ) combinations, especially points where restoration is not already close to ceiling?

1 comment

Thank you for asking how the operating point was selected and whether the answer counts obscure differences between the edits.

Operating point. Our experiment plan designated rank 16 for the main behavioral comparison. We carried over layers 24–36 from the geometric analysis and applied the same layer window across all models and controls. For intervention strength, we used λ=0.25\lambda=0.25, the smallest nonzero value in the configured grid of 0, 0.25, 0.5, 0.75, and 1.0. This provides a consistent intervention setting for the matched and shuffled comparisons. At this strength, each layer hook subtracts one quarter of its cached projected displacement from the final input-token activation.

Behavior beyond answer counts. InternVL provides a comparison in which both full edits have room for further recovery. At rank 16, matched pairing captures 37.1% of held-out displacement energy, compared with 27.0% for shuffled pairing. The corresponding edits recover 7 of 11 and 6 of 11 answers (Tables 5 and 2). Figure 4b provides a complementary continuous measurement: across all 66 cases, the mean improvements in the correct-over-stale logit margin are 6.49 and 6.92 for matched and shuffled removal, respectively. These measurements distinguish the geometric consistency of the learned directions from the decision changes induced by the interventions.

What carries recovery. Figure 4 further identifies the behaviorally effective component retained by shuffled pairing. When the shuffled edit is decomposed into matched-aligned and perpendicular components of equal length, the aligned component recovers 59 of 66 answers, compared with 20 of 66 for the perpendicular component. Thus, the matched directions identify the component responsible for most of the shuffled edit's recovery, providing a mechanistic explanation for the similar full-edit outcomes.

Strength and interpretation. Intervention vectors are computed from unedited states and cached before the edited forward pass. At a fixed rank and layer window, changing λ\lambda scales these vectors while preserving their directions. We will clarify this implementation and the distinction between geometric and behavioral measurements in Section 6. The rank sweep characterizes how pairing affects held-out geometric capture, while the behavioral experiments establish how these directions contribute to recovery at the reported operating point.

Reviewer_jhZBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies visual lock-in phenomena, where a vision-language model follows stale verbal information despite a current evidence supporting a different answer. The proposed method constructs 1257 four-choice GUI-grounding instances from ScreenSpot-Pro and evaluate four models while keeping visual evidence fixed and varying verbal context. It identifies compact representation subspaces associated with this behavior in four models. These subspaces generalize to unseen examples. Correctly paired examples identify the geometry better than shuffled pairs. However, both constructions are similarly effective at correcting errors. Successful interventions largely act along shared directions. Extensive experiments show that better geometric identification does not necessarily yield better behavioral control. Generally, the findings is quite useful and insightful for development of vision-language models for behavior control study.

Critical strengths
  1. This paper has got a strong motivation of running the series of models to identify the visual-lock effect.
  2. The experiment setup is designed well such that input images, query and answers are fixed, while the designated stale location makes it possible to distinguish prior-following errors from arbitrary mistakes.
  3. The result has demonstrated strong pattern. The aligned component outperforms the orthogonal component in every model.
  4. This paper is generally written well, easy to follow.
Critical weaknesses
  1. This paper contains quite a lot of complex equations but lacks of proper problem setup and explanation, which makes the content hard to read. For example, what does 'Full-set masses' mean in Eq.(1) ? And what are the purpose of LCR and DCR? For Eq.(2), what does zm,i+3lz_{m,i+3}^l mean? Why does it calaulate the difference between zm,i+3lz_{m,i+3}^l and zm,i,irrlz_{m,i, irr}^l ? These kind of ambiguity exists for all the formula proposed throughout the entire paper.
  2. Restoration is measured on 66 selected failures from one split, at rank 16 and intervention strength 0.25; Qwen and Holo already achieve perfect restoration with both constructions. Thus, 60/66 versus 59/66 establishes similar observed outcomes here, but not general behavioral equivalence or that better geometric capture is unhelpful at lower intervention strengths. The 20 permutation seeds address geometric variability, not behavioral robustness.
  3. In section 6, Figure 4, and Table 2, rhe reported intervention results focus on cases where misleading context caused an error. They do not establish whether the same correction harms answers that were already correct or suppresses useful context.
Questions for authors

More technical detail questions:

  1. Do the geometric advantage and shared-orientation effect survive held-out wording templates, answer permutations, and application-grouped splits?
  2. How does projected removal compare with equal-norm movement directly toward the same-example reference, and what happens to initially correct non-locking cases?
  3. Are intervention displacements cached from the original prior pass, or recomputed from the evolving edited pass? How were the rank, layer range, and strength selected?
1 comment

Thank you for raising the ceiling issue in Qwen and Holo. Table 2 allows us to examine the other two models directly. Across InternVL and MiniCPM, matched and shuffled edits recover 28 of 34 and 27 of 34 failures, respectively. On these same cases, the matched-aligned component of the shuffled edit recovers 27 of 34, compared with 10 of 34 for its equally long perpendicular component. This identifies the matched directions as carrying the behaviorally effective component of the shuffled edit, even when the two full edits achieve similar recovery.

Problem setup and notation. LCR measures how often an initially correct answer becomes incorrect after adding a cue. DCR measures, among these errors, how often the model selects the specific location named by the stale cue. “Full-set mass” expresses either event as a fraction of all 1,257 examples. In Eq. 2, +3 denotes the strongest wording condition. For the same screenshot, query, and answer choices, we compute the difference between the final input-token states under the stale-cue and irrelevant-cue prompts. Both prompts contain additional text; the stale-cue prompt additionally specifies the outdated location. Our geometric analysis measures how consistently a learned subspace captures this difference on held-out examples.

  1. Wording, answer order, and evaluation splits. The five-fold evaluation separates training and held-out examples. The wording scan uses three templates per certainty level, assigned before prediction. Answer choices are shuffled into A–D positions when each example is constructed and remain fixed across its cue conditions. This ensures that the paired comparisons preserve the same screenshot, query, and answer assignment. The 20 permutation seeds quantify variability in geometric capture under shuffled training pairs. These controls establish the reported comparisons at the example level while keeping the relevant input factors explicit.

  2. Reference movement and behavioral recovery. A direct move toward the reference follows the full difference between the stale-cue and irrelevant-cue states, whereas projected removal follows the component of that difference within the fitted subspace. Figure 4 isolates the directional contribution by applying equal-length aligned and perpendicular components separately. Their average improvements in the correct-over-stale logit margin are 6.93 and 0.97, respectively. Together with the 27/34 versus 10/34 recovery contrast, these results show that the matched-aligned component accounts for substantially more behavioral recovery than its perpendicular counterpart. The recovery endpoint is defined on cases that were correct under irrelevant text and switched to the designated stale location under the cue, providing a controlled measurement of reversal of cue-induced lock-in.

  3. Cached displacements and parameter choice. Intervention vectors are computed and cached before the edited forward pass. For each example and layer, we subtract the saved irrelevant-cue state from the saved, unedited stale-cue state, then project this difference onto the fitted subspace. During intervention, each layer hook subtracts its corresponding fixed vector, scaled by λ\lambda, from the current final input-token activation. Later layers therefore receive the effects of earlier interventions while using their own precomputed vectors. The main behavioral comparison uses rank 16 and layers 24–36, carried over from the geometric analysis. The reported intervention strength is λ=0.25\lambda=0.25, the smallest nonzero value in the configured grid of 0, 0.25, 0.5, 0.75, and 1.0.

Reviewer_aAnsOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates how misleading linguistic context can cause VLMs to ignore/override the current visual evidence. The authors use a GUI visual grounding task from ScreenSpot-Pro to identify changes in the models' representations that track the misleading language priors. They then explore whether specific interventions on these identified subspaces can help correct these failures. They find that their approach (using matched examples to derive the intervention) is only as effective as their baseline method (intervention derived from shuffled pairings) in correcting model errors.

Critical strengths

None

Critical weaknesses

Limited scope: Given that both the papers method and their baseline are at near-ceiling performance, this leaves me wondering whether the task setting explored in the paper is sensitive enough to distinguish between the efficacy of their method relative to their baseline. The intervention results are evaluated on only 66 selected failures from a single task, and this leaves me wondering whether an expanded set of failures (e.g., drawn from other tasks) might provide better sensitivity (and therefore potentially favor their approach). It also leaves me concerned about the generality of the papers findings, considering the limited scope of evaluation overall.

Issues with clarity: While the authors are precise when describing their contribution, the abstract, introduction, and discussion are overly compressed and difficult to follow without already being familiar with the analyses, tasks, and methods used in the paper. I found it difficult to understand the contribution of the paper despite my own research focusing on related questions in VLM interpretability. A few examples just from the abstract, though these issues are present in the introduction as well:

  • Line 14: "We use this controlled revision failure to ask whether the subspace that best captures recurrent prior-induced representation change is also uniquely more behaviorally effective." -- What is "recurrent prior-induced representation change" in the context of the experiments? What does it mean for a subspace to be "behaviorally effective"?
  • Line 19: "Matched correspondence improves held-out capture at every tested rank in all four models" -- "Matched correspondence" is undefined up to this point, and while "held-out displacement capture" is precisely defined (much) later on page 5, why this matters could be explained more clearly (and earlier). What is being measured? Why is this important to the papers conclusions? These questions should be answered.

Terminological density: Related to the point above, the paper uses technical jargon without appropriately explaining their meaning or the importance of the thing being measured/evaluated. A few examples:

  • Line 26: "These results show that the construction that best identifies recurrent representation change need not be uniquely more behaviorally effective, distinguishing global subspace identification from the geometry actually used by an intervention." -- What does it mean to distinguish global subspace identification from the geometry used by an intervention? While paper eventually explains this distinction relatively clearly in section 6, this comes too late and should be moved earlier to provide an intuition for why this is important. Explaining this clearly is critical because it is introduced in the abstract as the papers core contribution.
  • L48-L50: "At rank 32, they span 0.78% of the 4096-dimensional state and capture 49.7% to 64.0% of held-out prior-induced displacement energy (Figure 3), compactly describing recurrent change across visual instances." -- The terms "prior-induced displacement energy" and "recurrent change across visual instances" are not clearly connected to the research question this work sets out to address, making this read as overly compressed.

Together, the above mentioned issues with clarity and terminological density make it overly difficult to understanding the question the paper seeks to address and the methods it is using to do so. This affects not only the clarity of the paper, but also makes it more difficult to evaluate the significance of the papers findings.

Questions for authors

Suggestions that would improve my assessment of "Presentation":

  • The abstract and introduction are difficult to follow without already understanding the content of the paper. Defining key concepts, terminology, and analyses more intuitively (and without assuming familiarity with the papers key methods/analyses) would go long way toward improving the papers clarity.
  • Reduce the number of quantitative results presented in the abstract and introduction (and using this space to more clearly motivate the research question) may also make the paper easier to understand.

Demonstrating that your findings generalize beyond 66 failure cases may also improve my assessment of "Soundness".

1 comment

Thank you for pointing to Section 6 as the place where the distinction becomes clear. We would bring its central finding into the opening: the matched directions tell us which part of an edit learned from shuffled pairs still recovers the correct answer.

Consider the example in Figure 1. The screenshot places the “Listening Mode” control in the upper right, while a note about an earlier screen places it in the middle left. Suppose the model follows the note and an edit to its hidden state brings back the correct answer. How much does that success tell us about which directions capture the effect of the stale note across screenshots?

What shuffling tests. For each screenshot, we record the model’s state just before it answers, once with the stale note and once with unrelated text. The difference measures how its state changes on that example. We learn directions from these differences, then measure how much of the change on a new screenshot falls along them. This is held-out capture. Shuffling changes only which states are paired when learning the directions. At test time, both constructions act on the same state difference from the same screenshot. Matching improves held-out capture at every tested rank in all four models (Table 5).

What explains the recovery. The two full edits recover 60 and 59 of 66 answers. To understand how the shuffled edit succeeds, we split that edit into the part along the matched directions and the part perpendicular to them. We give each part the same length on each example and layer, then apply it separately. The aligned part recovers 59 of 66 answers. The perpendicular part recovers 20 of 66 and leaves 44 answers at the location named by the stale note. The aligned part recovers more answers in every model (Table 2). This comparison identifies what the shuffled edit retained that matters for the model’s decision.

Sensitivity and scope. InternVL provides a comparison with room for both full edits to improve. At the intervention rank, matching raises held-out capture from 27.0% to 37.1%, while the matched and shuffled edits recover 7 of 11 and 6 of 11 answers. Figure 4b also measures the score of the correct answer relative to the stale answer, which can change even after the model answers correctly. Across the 66 cases, the mean gains are 6.49 and 6.92 for matched and shuffled edits, respectively.

The geometry result uses 500 pairs per model, sampled before behavioral filtering and evaluated across five held-out folds. The 66 cases answer the more specific recovery question: on a disjoint evaluation split, these were the cases where the model answered correctly with unrelated text and then chose the designated stale location with the cue. Our behavioral conclusion concerns what carries recovery in this controlled GUI task at the tested setting. We would open the abstract and introduction with the concrete example and the two measurements, and move the detailed rank and seed counts to the results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.