- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates how task-relevant representations become organized within vision-language models before answer generation. The authors report the abrupt emergence of a spectrally dominant “decision axis” at an intermediate layer, develop a spectral framework to characterize its geometry, and apply the resulting readout to visual evidence reranking.
Critical strengths
- The paper formulates decision formation as a problem of internal geometric organization and derives sufficient conditions for spectral dominance and alignment with a reference decision axis. Its theoretical analysis explicitly accounts for value, routing, and interaction contributions while retaining cross-block covariance terms.
- The experiments cover five vision-language models and provide evidence of cross-dataset transfer by applying directions estimated on CAC to Visual-RAG while keeping them fixed. Layerwise and component-level analyses, together with question-level bootstrap intervals, strengthen the empirical evidence.
Critical weaknesses
The paper shows that task-relevant information becomes easier to read out at an intermediate layer. However, further analysis is needed to show whether this improvement coincides with a sudden concentration of that information along a dominant direction. Tracking the relevant geometric measures across layers and comparing different readout methods would make this interpretation more convincing.
Questions for authors
-
Before the transition layer, can linear probes, mean-difference directions, or multiple principal components already recover task-relevant information? Do the eigengap, directional alignment, and concentration of task-relevant variance also change abruptly at this layer?
-
How would removing, amplifying, or replacing activations along the identified decision axis affect the model’s final Yes/No probabilities and answers? How do these effects compare with interventions along random directions or other principal components?
-
Under the same models and data splits, how does the decision-axis readout compare with mean-difference probes, linear probes, tuned lens, and the model’s native ? Could the authors report the full reranking results for the native-probability baseline described in the appendix?
-
CAC labels images from the target class as relevant, but these images may not show the attribute being asked about. How do the authors distinguish whether the identified direction captures evidence sufficiency or simply category matching?
5 comments
Question 1 and Weakness 1 (Part 2)
Exp.2 :Spectral and Task-Relevance Properties before the Transition Layer
We further report spectral and task-relevance properties at layers 0 and to characterize geometric changes preceding the readout transition. Specifically, at layers 0 and , we compute the normalized test-covariance eigengap (1-2)/1, the squared cosine similarity between test PC1 and the fit-derived task axis, and the fraction of squared test-set class-mean separation captured by fit-derived PC1, as reported in Table. 3 and Table. 4, respectively.
Table. 3: Spectral Geometry at Layer 0 across Five VLMs.
| quantity | Qwen3-VL-8B (0 / 36) | Qwen2.5-VL-7B (0 / 28) | InternVL2.5-8B-MPO (0 / 32) | MiniCPM-V-4.5 (0 / 36) | Ovis2.5-9B (0 / 36) | mean of the five models |
|---|---|---|---|---|---|---|
| eigengap | 0.480 | 0.490 | 0.241 | 0.303 | 0.657 | 0.434 |
| directional alignment | 0.147 | 0.126 | 0.000 | 0.126 | 0.297 | 0.139 |
| concentration of task-relevant variance | 0.005 | 0.019 | 0.021 | 0.096 | 0.024 | 0.033 |
Table. 4: Spectral Geometry at Layer k/2 across Five VLMs
| quantity | Qwen3-VL-8B (9 / 36) | Qwen2.5-VL-7B (8 / 28) | InternVL2.5-8B-MPO (7 / 32) | MiniCPM-V-4.5 (9 / 36) | Ovis2.5-9B (10 / 36) | mean of the five models |
|---|---|---|---|---|---|---|
| eigengap | 0.324 | 0.507 | 0.404 | 0.304 | 0.419 | 0.392 |
| directional alignment | 0.001 | 0.007 | 0.018 | 0.020 | 0.037 | 0.017 |
| concentration of task-relevant variance | 0.014 | 0.006 | 0.018 | 0.002 | 0.123 | 0.033 |
As shown in Table. 3 and Table. 4, appreciable eigengaps coexist with low task alignment and task-relevant variance concentration across layers, suggesting that PC1’s spectral dominance precedes its emergence as a decision-relevant direction. These results suggest that PC1’s task relevance emerges with a change in alignment at the transition layer, rather than from spectral dominance.
Question 2
Question 2: Causal Validation of Interventions along the Decision Axis
We conduct causal interventions by amplifying or decreasing activation projections along the decision-relevant direction, other principal components, and random orthogonal directions, matching perturbation magnitudes in standardized activation space and measuring changes in output performance relative to the unedited baseline. Specifically, we edit the attention output at the final prompt token of the emergence layer as where d is a unit intervention direction, s1 is the standard deviation along the decision-relevant direction, and sd contains the coordinate-wise standard deviations used for standardization. The edited activation is then propagated through the remaining network to assess changes in the final Yes/No prediction. Our results in Table. 5 demonstrate that perturbing representations along or against PC1 causally steers model predictions across intervention strengths, with substantially larger effects than magnitude-matched perturbations along other principal components or random directions.
Table. 5: Causal Effects of Representation Perturbations across Five VLMs.
| model | alpha | no edit | pushed along PC1 | pushed along other PCs | pushed along a random direction | pushed against PC1 |
|---|---|---|---|---|---|---|
| Qwen3-VL-8B | 0.5 | 0.605 (0.000) | 0.615 (+0.010) | 0.604 (-0.001) | 0.604 (-0.001) | 0.589 (-0.016) |
| Qwen2.5-VL-7B | 0.5 | 0.594 (0.000) | 0.602 (+0.008) | 0.591 (-0.003) | 0.592 (-0.002) | 0.580 (-0.014) |
| InternVL2.5-8B-MPO | 0.5 | 0.618 (0.000) | 0.638 (+0.020) | 0.615 (-0.003) | 0.619 (+0.001) | 0.591 (-0.027) |
| MiniCPM-V-4.5 | 0.5 | 0.652 (0.000) | 0.671 (+0.019) | 0.650 (-0.002) | 0.652 (0.000) | 0.642 (-0.010) |
| Ovis2.5-9B | 0.5 | 0.676 (0.000) | 0.684 (+0.008) | 0.678 (+0.002) | 0.677 (+0.001) | 0.673 (-0.003) |
| Mean | 0.5 | 0.629 (0.0000) | 0.642 (+0.0130) | 0.628 (-0.0014) | 0.629 (-0.0002) | 0.615 (-0.0140) |
| Qwen3-VL-8B | 1 | 0.605 (0.000) | 0.627 (+0.022) | 0.605 (0.000) | 0.603 (-0.002) | 0.570 (-0.035) |
| Qwen2.5-VL-7B | 1 | 0.594 (0.000) | 0.607 (+0.013) | 0.588 (-0.006) | 0.592 (-0.002) | 0.570 (-0.024) |
| InternVL2.5-8B-MPO | 1 | 0.618 (0.000) | 0.650 (+0.032) | 0.610 (-0.008) | 0.621 (+0.003) | 0.559 (-0.059) |
| MiniCPM-V-4.5 | 1 | 0.652 (0.000) | 0.680 (+0.028) | 0.649 (-0.003) | 0.651 (-0.001) | 0.633 (-0.019) |
| Ovis2.5-9B | 1 | 0.676 (0.000) | 0.690 (+0.014) | 0.678 (+0.002) | 0.676 (0.000) | 0.665 (-0.011) |
| Mean | 1 | 0.629 (0.0000) | 0.651 (+0.0218) | 0.626 (-0.0030) | 0.629 (-0.0004) | 0.599 (-0.0296) |
| Qwen3-VL-8B | 2 | 0.605 (0.000) | 0.640 (+0.035) | 0.603 (-0.002) | 0.602 (-0.003) | 0.539 (-0.066) |
| Qwen2.5-VL-7B | 2 | 0.594 (0.000) | 0.617 (+0.023) | 0.582 (-0.012) | 0.590 (-0.004) | 0.549 (-0.045) |
| InternVL2.5-8B-MPO | 2 | 0.618 (0.000) | 0.688 (+0.070) | 0.597 (-0.021) | 0.622 (+0.004) | 0.528 (-0.090) |
| MiniCPM-V-4.5 | 2 | 0.652 (0.000) | 0.685 (+0.033) | 0.647 (-0.005) | 0.650 (-0.002) | 0.603 (-0.049) |
| Ovis2.5-9B | 2 | 0.676 (0.000) | 0.696 (+0.020) | 0.676 (0.000) | 0.676 (0.000) | 0.654 (-0.022) |
| Mean | 2 | 0.629 (0.0000) | 0.665 (+0.0362) | 0.621 (-0.0080) | 0.628 (-0.0010) | 0.575 (-0.0544) |
As shown in Table 5, perturbations along PC1 consistently improve accuracy, whereas opposite perturbations reduce it, demonstrating causal steering of model predictions. Across models and intervention strengths, the mean absolute accuracy change under bidirectional PC1 interventions is approximately 6.0 and 16.3 times that of the averaged other-PC and random-direction controls, respectively, supporting PC1’s privileged causal role in answer selection. These results distinguish the identified direction from the tested alternatives through its stronger causal influence on model predictions, supporting its privileged role and providing a functional basis for early decision readout from intermediate representations. We have added the causal intervention experiments and the corresponding discussion in **Section 5.5 **of the revised manuscript to further establish the causal significance of the identified decision-relevant directions.
Question 1 and Weakness 1 (Part 1)
We further examine pre-transition readout performance and find that task-relevant information develops gradually but becomes abruptly concentrated along a spectrally dominant direction at the transition layer. Specifically, we compare readout methods at layer 0 and halfway to the transition layer in the CAC datasets (Exp. 1), with improvements in linear-probe and mean-difference performance suggesting that task-relevant information becomes progressively recoverable before the transition. Moreover, we examine PC1’s spectral and task-relevance properties, which reveal its abrupt emergence as a dominant task-relevant direction at the transition layer. Together, these results suggest that task-relevant information accumulates gradually, with its abrupt concentration along a dominant direction coinciding with the readout transition.
Exp. 1: Recoverability of Task-Relevant Information before the Transition Layer
We report AUROC on CAC for representation-level readouts at layers 0 and ⌊k/2⌋ in Table. 1 and Table. 2, respectively. Specifically, our readout projects standardized attention outputs onto PC1, with labels determining only its sign, whereas mean-difference readout uses the relevant–irrelevant mean contrast. Additionally, multiple-PC readout fits logistic regression on the leading m PC scores, with m1,2,4,8 selected by cross-validation within the fit split, while full-vector readout applies L2-regularized logistic regression to all standardized features.
Table. 1: Readout Performance at Layer 0 across Five VLMs
Table. 2: Readout Performance at Layer k/2 across Five VLMs
As shown in Table. 1, readout performance at layer 0 remains close to chance, with modest differences across methods, suggesting limited recoverable task-relevant information. While in Table. 2, full-dimensional supervised readouts, namely mean-difference and linear probe, exhibit greater task discrimination, whereas our method and the multiple-PC readout remain near chance. These results indicate that task-relevant information accumulates progressively before the transition layer but remains weakly expressed along the leading principal directions, becoming recoverable through supervised readouts.