acceptodds
Under review as a conference paper at ICLR 2027

Representation–Behavior Alignment for Explainable Weakly-Supervised Video Anomaly Detection

Abstract

Multimodal Large Language Models (MLLMs) provide a natural way to make video anomaly detection more explainable. However, their final decisions do not always fully use the discriminative information contained in their hidden states, an issue we refer to as representation–behavior misalignment. We decompose this gap into a capacity component that measures discriminative information never aggregated into the readout position, and a directional component that measures the angular mismatch between the optimal and the native normal–abnormal axis at that position. Across multiple video anomaly detection benchmarks and MLLM backbones the directional component dominates, and residual-stream tracing shows that native-axis separability rises sharply in several mid-to-late attention layers. Because both components are governed by attention rather than MLP updates, we propose Representation–Behavior Alignment (RBA), a parameter-efficient method that adapts those layers using video-level labels alone while updating about 0.012% of the backbone parameters. Experiments on three benchmarks show that RBA improves native-readout performance and better aligns the model's decision direction with discriminative representations, and it produces anomaly decisions and explanations through a single generative process.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multimodal large language models, representation-behavior misalignment, mechanistic interpretability, representation probing, directional readout alignment, parameter-efficient fine-tuning, video anomaly detection

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_YHBhOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates representation-behavior misalignment in modern multimodal large language models (MLLMs), where final decisions do The authors show that linear probes trained on these hidden states outperform the models’ native normal-abnormal readouts. Their analysis attributes most of this performance gap to a mismatch in readout direction, while residual-stream analysis identifies sharp increases in native-axis separability at specific attention sublayers. Based on these findings, the authors propose Representation-Behavior Alignment (RBA), a parameter-efficient method that adapts the identified attention layers using only video-level labels while updating approximately 0.012% of the backbone parameters. Experiments on three video anomaly detection benchmarks demonstrate the effectiveness of the proposed method in improving native-readout performance.

Critical strengths
  • The problem and the organisation of the paper are clear
  • The studied question about the representation and behaviour gap is interesting and important for the development of MLLMs. It illustrates that current models do not fully utilise their capabilities for relevant tasks, indicating room for further improvement.
  • The method is simple but effective for deployment.
Critical weaknesses
  • The experimental comparisons on VAD benchmarks may be unfair. The baseline methods in Table 1 use different backbone models, which raises concerns about the fairness of the comparisons. For example, the zero-shot readout Frame-AP (72.86%) reported in Table 10 already surpasses the performance of Anom-π and PANDA. Additionally, LRPO uses InternVL3.5-8B instead of Qwen3.5-9B.

  • It is unclear whether the performance gains come from representation alignment or task-specific fine-tuning. A comparison between RBA and standard weakly supervised fine-tuning of the MLLM could help clarify this point.

  • In Eq. (3), the capacity gap is defined as the performance difference between Fisher LDA using all token representations and only the last-token representation. However, the number of accessible tokens does not directly reflect representation capacity. This comparison primarily measures the additional linearly decodable information available from other token positions. Therefore, the term “capacity” may not accurately describe what is being measured.

  • The layer-wise behaviors appear inconsistent across different backbone models in Figures 10 and 11. In Figure 10, attention contributes positively to readout alignment, whereas in Figure 11, MLP contributions become strongly positive while attention contributions turn negative in later layers. This seems inconsistent with the claim that attention is the primary source of positive alignment gains across architectures.

Questions for authors
  • The source of the frame-level annotations used to fit Fisher LDA is unclear. Given the weakly supervised setting, the authors should clarify how these annotations were obtained and whether any additional frame-level supervision was introduced.

  • Why does RBA use hard labels instead of soft labels?

1 comment

I would like to thank the authors for their detailed responses and additional evaluations. The rebuttal has addressed most of my previous conceptual concerns, and I am happy to raise my score to Weak Accept. However, the comparison in Table 1 could be presented more clearly, as the methods use different backbones. I would suggest replacing the venue column with a backbone column in the final version to make these differences explicit, as the venue information don't really add to much value to the comparison.

Reviewer_jFS6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates representation–behavior misalignment in MLLMs for weakly supervised video anomaly detection. The authors introduce an LDA-based framework that decomposes the discrepancy between internal discriminative representations and native predictions into Capacity and Directional Gaps. Based on layer-wise analysis, they propose RBA, which combines MIL-based pseudo-labeling with selective attention-layer LoRA and margin-based optimization. Experiments on three VAD benchmarks and multiple MLLM backbones demonstrate competitive performance while updating only 0.012% of the backbone parameters.

Critical strengths
  1. The paper investigates the discrepancy between internal representations and model predictions, offering a meaningful perspective beyond conventional performance-oriented VAD methods. The diagnostic analysis also provides a coherent motivation for the proposed adaptation strategy.
  2. RBA achieves competitive results across multiple benchmarks and MLLM backbones with minimal trainable parameters and only video-level supervision.
Critical weaknesses
  1. Limited evidence for the claimed mechanism. The LDA-based gap decomposition and layer-wise separability analysis are primarily correlational. The observed Directional Gap may depend on specific readout directions, prompts, or output tokens, while the claimed role of attention layers in routing anomaly information lacks causal validation.
  2. Insufficient isolation of methodological contributions. RBA combines established components, and the experiments do not adequately isolate the benefits of diagnostic-guided adaptation from conventional fine-tuning. Parameter-matched comparisons with random-layer or uniformly distributed LoRA are needed to establish its distinct contribution.
  3. Incomplete validation of broader claims. The paper does not assess whether RBA preserves general video understanding capabilities. Moreover, decision–explanation consistency alone does not establish explanation faithfulness without interventions on the underlying visual evidence.
Questions for authors
  1. How robust are the gap decomposition and selected attention layers to different prompts, output tokens, and readout methods? Can causal interventions validate the proposed information-routing mechanism?
  2. Under identical pseudo-labels and parameter budgets, how does RBA compare with random-layer LoRA or standard parameter-efficient fine-tuning?
  3. Does RBA preserve general video understanding performance? Can visual evidence perturbations verify the faithfulness of generated explanations?
Reviewer_aH9FOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work investigates representation‑behavior misalignment arising when multimodal large language models are deployed for weakly‑supervised video anomaly detection. The authors break down performance gaps into capacity and directional components, and their empirical results demonstrate that directional misalignment is the dominant source of performance degradation. With residual‑stream tracing, they pinpoint a prominent alignment phase transition within mid‑to‑late attention sublayers. Drawing on these diagnostic findings, they propose Representation‑Behavior Alignment: adapters are introduced exclusively into those identified attention layers, accounting for roughly 0.012% of the total backbone parameters. Using Top‑K MIL-linear-probe‑derived pseudo‑labels alongside a hinge‑margin objective, RBA rotates internal model representations toward the model’s native readout axis. Evaluations are performed across three VAD benchmarks and multiple MLLM backbones including Qwen3.5‑9B, InternVL3.5‑8B and Gemma‑4‑12B.

Critical strengths

Strength 1: Geometric diagnosis and data‑informed layer attribution. Rather than merely noting that frozen MLLMs yield poor performance for VAD, the authors construct a controlled three‑tier readout chain to tease apart capacity and directional gaps. Their residual‑stream tracing further disentangles the opposing effects from attention and MLP sublayers. This delivers a concrete, measurement‑guided rule for adapter placement instead of heuristic layer picking.

Strength 2: Lightweight weakly‑supervised intervention that retains the native generative pathway. RBA is trained purely on video‑level weak labels with very small trainable parameter overhead, and no external classifier is needed at test time. Anomaly scores and natural‑language explanations come from a single shared forward pass. Ablations validate key design decisions, such as LoRA rank, target sublayers, hinge‑margin objective, and the capacity‑isolation principle.

Critical weaknesses

Weakness 1: Limited systematic assessment of pseudo‑label noise robustness. All training signals for RBA stem from pseudo‑labels produced by the offline MIL linear probe. While the paper shows several qualitative case examples, controlled perturbation experiments are absent — for instance, artificially injecting label noise into video‑level bags to observe downstream VAD performance degradation. In practical weakly‑supervised deployments, source labels are inherently noisy, and how the model’s performance falls off under such conditions remains unclear.

Weakness 2 : Absence of vanilla LoRA fine‑tuning baseline with matched parameter budget. The paper credits performance gains to both diagnosis‑driven layer placement and hinge‑margin optimization over the native readout axis. Nevertheless, it does not include a plain LoRA fine‑tuning baseline with identical parameter count and target layers, which is trained directly with video‑level bag loss and without probe‑generated pseudo‑labels. As a result, we cannot fully rule out the alternative hypothesis that most performance improvements arise purely from low‑rank parameter updates, instead of the complete RBA pipeline combining diagnosis and pseudo‑labeling.

Questions for authors

Q1: In Equation (3) of Section 3.1, the total performance gap is decomposed as Δtotal=Δcap+Δdir\Delta_{total}=\Delta_{cap}+\Delta_{dir}. The authors state this decomposition holds by construction, yet the two terms may suffer from statistical coupling. Could you clarify whether such coupling would undermine your core conclusion that the directional component dominates? Have you conducted any analysis to quantify the magnitude of this coupling effect?

Q2: In Section 4.2, you binarize clip‑level probe scores into pseudo‑labels y^i∈{+1,−1}\hat{y}_i\in\{+1,-1\} using a fixed threshold of 0.5. The paper acknowledges that the MIL linear probe acts as a noisy weakly‑supervised estimator. Why is 0.5 chosen as the default threshold?

Q3: Table 3, Table 6 and Appendix C.5 demonstrate that adapters inserted after attention blocks 20‑21 yield the best performance, while placement after MLP sublayers performs poorly. The paper reports MLP sublayers introduce net negative interference for separability along the native axis. Adapters are injected at blocks 20‑21, yet representations propagate through subsequent attention and MLP layers all the way to layer 32. How do the beneficial directional rotations introduced by LoRA persist through these later, partially interfering layers?

Q4: Appendix D defines the adversarial rationalization metric R−R^- evaluated on set D−D^-, and RBA achieves markedly lower rationalization rates. Could you disentangle whether this improvement comes from genuine representation‑behavior alignment, or merely serves as a side effect where LoRA fine‑tuning restricts the model’s original generative freedom?

Q5: In Table 1, RBA outperforms prior MLLM‑based VAD approaches, yet remains inferior to specialized non‑linguistic VAD detectors such as RefineVAD and LAVIDA. Please analyze the source of this performance gap: is it mainly bounded by inherent limitations of the MLLM backbone, or does your representation‑alignment pipeline itself carry intrinsic performance ceilings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.