- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether low-rank “Prior Directions” that capture prior-induced representation changes in vision-language models are also uniquely effective at correcting failures to update stale verbal priors using visual evidence. Across four models, matched prior–reference pairings capture held-out representation changes better than permuted pairings, yet their interventions achieve nearly identical recovery rates. Further analysis shows that permuted edits remain strongly aligned with Prior Directions, demonstrating that superior subspace identification does not necessarily translate into a corresponding behavioral advantage.
Critical strengths
The visual revision testbed is tightly controlled, with the screenshot, query, and candidate answers held fixed while only the verbal context changes. This design makes it possible to test whether a stale location cue can pull an initially correct visual judgment toward the designated outdated location. Observing the resulting visual lock-in behavior across four VLM families gives the representation analysis a solid behavioral anchor. Just as valuable is the separation between geometric identification and behavioral efficacy. The matched-correspondence Prior Directions capture held-out representation changes better, yet the matched and Pairing-Permuted interventions restore 60/66 and 59/66 of the clean lock-in failures, nearly identical. The rank-matched random subspaces, norm-matched random removal, and the aligned-versus-orthogonal decomposition all point the same way, suggesting that what the Pairing-Permuted intervention retains comes mainly from the shared orientation of its edits. Having these controls in place keeps the behavioral story from resting on subspace fit quality alone.
Critical weaknesses
The behavioral conclusion rests on a narrow slice of the experimental space. The claim that better geometric identification brings no corresponding behavioral advantage is drawn from 66 clean failures at rank 16, intervention strength 0.25, a fixed layer range, one data split, and a single Pairing-Permuted construction. At this operating point both interventions already restore most failures (60/66 vs 59/66), so the evidence establishes similar restoration counts at this one setting; it does not yet tell us whether the relationship holds under other intervention settings.
Questions for authors
The geometric identification analysis sweeps several subspace ranks, but the behavioral comparison is reported only at k = 16, λ = 0.25, and L_late = {24, ..., 36}. How were these values chosen — best restoration, or representative? And does the near-equality between the two interventions persist across a few other (k, λ) combinations, especially points where restoration is not already close to ceiling?
Thank you for asking how the operating point was selected and whether the answer counts obscure differences between the edits.
Operating point. Our experiment plan designated rank 16 for the main behavioral comparison. We carried over layers 24–36 from the geometric analysis and applied the same layer window across all models and controls. For intervention strength, we used λ=0.25, the smallest nonzero value in the configured grid of 0, 0.25, 0.5, 0.75, and 1.0. This provides a consistent intervention setting for the matched and shuffled comparisons. At this strength, each layer hook subtracts one quarter of its cached projected displacement from the final input-token activation.
Behavior beyond answer counts. InternVL provides a comparison in which both full edits have room for further recovery. At rank 16, matched pairing captures 37.1% of held-out displacement energy, compared with 27.0% for shuffled pairing. The corresponding edits recover 7 of 11 and 6 of 11 answers (Tables 5 and 2). Figure 4b provides a complementary continuous measurement: across all 66 cases, the mean improvements in the correct-over-stale logit margin are 6.49 and 6.92 for matched and shuffled removal, respectively. These measurements distinguish the geometric consistency of the learned directions from the decision changes induced by the interventions.
What carries recovery. Figure 4 further identifies the behaviorally effective component retained by shuffled pairing. When the shuffled edit is decomposed into matched-aligned and perpendicular components of equal length, the aligned component recovers 59 of 66 answers, compared with 20 of 66 for the perpendicular component. Thus, the matched directions identify the component responsible for most of the shuffled edit's recovery, providing a mechanistic explanation for the similar full-edit outcomes.
Strength and interpretation. Intervention vectors are computed from unedited states and cached before the edited forward pass. At a fixed rank and layer window, changing λ scales these vectors while preserving their directions. We will clarify this implementation and the distinction between geometric and behavioral measurements in Section 6. The rank sweep characterizes how pairing affects held-out geometric capture, while the behavioral experiments establish how these directions contribute to recovery at the reported operating point.