- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates representation-behavior misalignment in modern multimodal large language models (MLLMs), where final decisions do The authors show that linear probes trained on these hidden states outperform the models’ native normal-abnormal readouts. Their analysis attributes most of this performance gap to a mismatch in readout direction, while residual-stream analysis identifies sharp increases in native-axis separability at specific attention sublayers. Based on these findings, the authors propose Representation-Behavior Alignment (RBA), a parameter-efficient method that adapts the identified attention layers using only video-level labels while updating approximately 0.012% of the backbone parameters. Experiments on three video anomaly detection benchmarks demonstrate the effectiveness of the proposed method in improving native-readout performance.
Critical strengths
- The problem and the organisation of the paper are clear
- The studied question about the representation and behaviour gap is interesting and important for the development of MLLMs. It illustrates that current models do not fully utilise their capabilities for relevant tasks, indicating room for further improvement.
- The method is simple but effective for deployment.
Critical weaknesses
-
The experimental comparisons on VAD benchmarks may be unfair. The baseline methods in Table 1 use different backbone models, which raises concerns about the fairness of the comparisons. For example, the zero-shot readout Frame-AP (72.86%) reported in Table 10 already surpasses the performance of Anom-π and PANDA. Additionally, LRPO uses InternVL3.5-8B instead of Qwen3.5-9B.
-
It is unclear whether the performance gains come from representation alignment or task-specific fine-tuning. A comparison between RBA and standard weakly supervised fine-tuning of the MLLM could help clarify this point.
-
In Eq. (3), the capacity gap is defined as the performance difference between Fisher LDA using all token representations and only the last-token representation. However, the number of accessible tokens does not directly reflect representation capacity. This comparison primarily measures the additional linearly decodable information available from other token positions. Therefore, the term “capacity” may not accurately describe what is being measured.
-
The layer-wise behaviors appear inconsistent across different backbone models in Figures 10 and 11. In Figure 10, attention contributes positively to readout alignment, whereas in Figure 11, MLP contributions become strongly positive while attention contributions turn negative in later layers. This seems inconsistent with the claim that attention is the primary source of positive alignment gains across architectures.
Questions for authors
-
The source of the frame-level annotations used to fit Fisher LDA is unclear. Given the weakly supervised setting, the authors should clarify how these annotations were obtained and whether any additional frame-level supervision was introduced.
-
Why does RBA use hard labels instead of soft labels?
I would like to thank the authors for their detailed responses and additional evaluations. The rebuttal has addressed most of my previous conceptual concerns, and I am happy to raise my score to Weak Accept. However, the comparison in Table 1 could be presented more clearly, as the methods use different backbones. I would suggest replacing the venue column with a backbone column in the final version to make these differences explicit, as the venue information don't really add to much value to the comparison.