acceptodds
Under review as a conference paper at ICLR 2027

RoMod: Temporal Routing Modulation via Mixture-of-Experts for Video Anomaly Detection

Abstract

Intermediate-layer features from multimodal large language models have shown strong potential for video anomaly detection (VAD), yet the origin of their discriminative power remains unclear. We study this question using sparse mixture-of-experts (MoE) models, whose explicit expert structure and sparse activation make their internal computation easier to inspect. With a fully frozen backbone and no additional training, we find that anomaly-related evidence is concentrated in a small set of experts. These experts recur across layers, spontaneously specialize in different anomaly types, and together form a dynamic routing subnetwork. We further show that the output channels most strongly influenced by these experts are also the hidden dimensions that contain the most anomaly-relevant information. Routing statistics can therefore serve as an internal anomaly cue that complements semantic features. Based on these findings, we propose RoMod, an efficient VAD framework trained with only \(5%\) of weakly labeled videos. RoMod includes a Routing-Modulated Fusion module, RoMF, and a Routing-aware Temporal Network, RoTN. RoMF uses routing signals to adaptively recalibrate hidden semantic channels. Its design also prevents the routing branch from bypassing semantic features and making predictions on its own. RoTN captures the temporal evolution of anomalies from onset to persistence and termination. Experiments on three benchmarks show that RoMod achieves state-of-the-art performance while running substantially faster than dense backbones of comparable size.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video anomaly detection, mixture-of-experts, weakly supervised learning, multimodal large language models, representation analysis

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_s3HwOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates anomaly information in the routing and hidden states of frozen multimodal MoE models. It reports concentrated anomaly-related expert activity, channel-level alignment, and targeted expert-knockout effects. RoMod combines pseudo-label-based layer selection, routing-driven multiplicative channel modulation (RoMF), and a temporal network (RoTN), trained using 5% of weakly labeled videos. On XD-Violence, UCF-Crime, and UBnormal, the offline variant reports 92.49% AP, 91.32% AUC, and 79.17% AUC, respectively. The validation includes module and fusion ablations, supervision-budget studies, alternative backbones, and a causal-attention streaming variant.

Critical strengths
  1. Routing is developed into a practically useful feature source, with controlled same-backbone evidence. RoMF raises XD-Violence AP from 84.87% to 90.95% without RoTN, and Appendix E.1 compares it with routing-only, concatenation, additive fusion, and FiLM. These comparisons support an improvement beyond merely using a stronger backbone.
  2. The diagnostic study connects expert activity to downstream representations through complementary measurements and interventions. Appendix B includes channel alignment, another architecture, and targeted versus random knockout. This makes the analysis more informative than an expert-activation visualization alone, while still leaving room for a more carefully matched intervention control.
Critical weaknesses
  1. The claimed cross-domain advantage of the fusion mechanism is not directly demonstrated. Appendix E.1 interprets a larger operator gap on UBnormal as transfer evidence, but the reported models use each dataset's own weak supervision; Appendix E.2 instead tests native zero-shot backbone outputs. Separate in-domain training on visually different datasets does not establish source-to-target transfer of RoMF or prove that concatenation transfers poorly. This weakens a central explanation of why the proposed constraint is useful.
  2. The headline superiority and label-efficiency comparison do not isolate representation quality from the proposed method. Prior baselines use different feature extractors, and the paper explicitly lacks a same-backbone hidden-only temporal baseline trained with all weak labels (Appendix G). The dense reference also changes both architecture and readout. The existing ablations establish a conditional gain on this backbone, but support a narrower conclusion than a controlled advantage over competing training regimes.
Questions for authors
  1. Do any reported fusion experiments train on one dataset and evaluate on another without target training labels or target-specific fitting? If so, please identify the source, target, selection procedure, and corresponding results; otherwise, can the cross-domain claims be restricted to the evidence actually reported?
  2. How does a hidden-only temporal model on the same frozen backbone perform when trained with 5% and 100% of weak labels under the same selection and smoothing protocol? These results would determine how much of the headline advantage belongs to routing modulation.
Reviewer_XXgDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the use of routing statistics from a frozen multimodal Mixture-of-Experts (MoE) model for video anomaly detection. RoMod uses 5% of the weakly labeled training videos and applies category-balanced MIL (C-MIL) to produce clip-level pseudo-labels. Routing-Modulated Fusion (RoMF) uses the MoE routing fingerprint to rescale hidden semantic channels, while the Routing-aware Temporal Network (RoTN) performs temporal modeling. The complete pipeline obtains competitive results on UBnormal, UCF-Crime, and XD-Violence.

Critical strengths
  1. The paper identifies a potentially useful anomaly signal in the routing statistics of a frozen MoE and connects it to hidden semantic features.

  2. The experiments cover three VAD datasets, several MoE backbones, supervision budgets, and component ablations. The complete system is competitive.

Critical weaknesses
  1. The methodological novelty is limited. RoMF uses normalized MoE activation statistics to produce channel-wise scaling, H~=(1+γ(R))⊙H‾\widetilde{H}=(1+\gamma(R))\odot\overline{H}, but both elements have close precedents: its routing fingerprint resembles prior routing signatures [Avinash, 2026], while its modulation is a scale-only FiLM operator [Perez et al., 2018] and is closely related to SteerVAD's dynamic rescaling of anomaly-sensitive MLLM channels [Cai et al., 2026]. RoTN is also a conventional temporal Transformer. The main novelty therefore appears to be the application of MoE routing statistics as a conditioning signal for VAD, rather than a new routing, fusion, or temporal modeling mechanism. This claim should be supported by matched comparisons with standard FiLM, all-layer hidden features, and SteerVAD-style modulation under the same backbone, labels, and temporal head.

  2. The comparison with weakly supervised, unsupervised, and zero-shot baselines is not fair because RoMod is trained with MLLM-generated clip-level labels that are unavailable to those methods. These labels may themselves account for a substantial part of the reported gain. The current experiments therefore cannot determine whether the improvement comes from RoMF/RoTN or from the MLLM-generated supervision. The authors should report RoMod trained without these labels and, conversely, train representative baselines with the same generated labels. Comparisons under matched backbones and supervision are necessary to isolate the architectural contribution.

  3. The cost of C-MIL and the complete training pipeline is not reported. Table 4 gives activated parameters, per-step FLOPs, module latency, and online FPS, but omits 35B-backbone feature extraction, zero-shot scoring, C-MIL, and RoMF/RoTN training. These costs matter when 5% data use is presented as a low-resource setting.

The state-of-the-art accuracy is reported for offline RoMod, whereas 51.83 FPS is reported for the less accurate online model. The paper should not combine these as properties of one configuration. The paper should report GPU-hours, peak memory, and end-to-end latency, and compare accuracy and speed with other MLLM-based VAD methods under comparable hardware and inputs.

[Avinash, 2026] M. S. R. Avinash. "Task-Conditioned Routing Signatures in Sparse Mixture-of-Experts Transformers." arXiv:2603.11114, 2026. https://arxiv.org/abs/2603.11114

[Perez et al., 2018] E. Perez et al. "FiLM: Visual Reasoning with a General Conditioning Layer." AAAI 2018. https://ojs.aaai.org/index.php/AAAI/article/view/11671

[Cai et al., 2026] Z. Cai et al. "Steering and Rectifying Latent Representation Manifolds in Frozen Multi-Modal LLMs for Video Anomaly Detection." 2026. https://arxiv.org/abs/2602.24021

Questions for authors
  1. Does RoMF still outperform standard FiLM, all-layer hidden pooling, and SteerVAD-style modulation when the backbone, pseudo-labels, and temporal network are fixed?

  2. How much performance remains when RoMod is trained without MLLM-generated clip-level labels? If representative baselines receive the same generated labels, does RoMod retain a clear advantage?

  3. What are the GPU-hours, peak memory, and runtime of MLLM feature extraction, C-MIL, and RoMF/RoTN training? Please report accuracy and end-to-end speed for the same online or offline configuration and compare with other MLLM-based VAD methods under a common protocol.

Reviewer_FevfOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates why intermediate features of multimodal LLMs work so well for VAD, using a frozen mixture-of-experts MLLM as a testbed. It finds that a small set of experts carries anomaly information, and builds a detector that uses expert-routing statistics to reweight hidden-state channels, and does so using a very small amout of the weakly labelled videos in the most common, publicly-available datasets

Contributions: A training-free mechanistic analysis showing where discriminability of anomalous events emerges, which experts carry it, and how routing links to hidden channels. RoMF, a routing-modulated fusion module, and RoTN, a causal temporal network. A low-resource pipeline in which pseudo-labels drive layer selection and training, reaching SOTA with 5% data

Evaluation:

Data: UCF-Crime, XD-Violence, UBnormal; Qwen3.6-35B-A3B backbone, plus two other MoE backbones and a dense reference. Comparison: Against full-data, low-data and training-free methods. Ablations: Modules, data budget, fusion operators, layer selection, hyperparameters, multiple seeds. Mechanistic analyses: Fisher separability by layer, per-expert discriminability, routing divergence by anomaly category, channel–expert alignment, expert knockouts.

Critical strengths

** Strengths **

  • Well written, clear and understandable with informative figures
  • Novel view of the usage of backbones in VAD, especially leveraging the routing features of the backbone
  • Good results on three benchmarks, the performance on XDV in particular.
Critical weaknesses
  • While the leveraging of routing features is novel in VAD, it is not the first paper to propose similar ideas. To my knowledge, the first is Your Mixture-of-Experts LLM Is Secretly an Embedding Model for Free, Li et al., ICLR 2025 (Oral). HeadHunt-VAD (cited in the result table, not discussed anywhere else) proposed to select expert attention heads inside a MMLLM, which seems to be a close idea to the expert selection proposed in this paper. While these do not necessarly mean that the proposed method is not novel, it does add some perspective to the novelties introduced.
  • The choice of using 5% of the training set is underjustified. The authors even acknowledge (L1850) that "a same-backbone baseline trained on 100% of the weak labels would be needed to attribute the margin over prior work between representation quality and routing modulation." This naturally raise the question of why not using the entire training set? The authors report (Table 3) that using 10% of the training set marginally improves performance on all datasets, and select 5% as the default for lower annotation costs (L417-419), but if they mean annotations as video-level labels, then those are already available for the dataset, otherwise if they mean the costs of generating pseudo-labels then the authors should report the costs of it. This point is further confused by the fact that Figure 9 and lines L1157-1159 shows that using 10% of the training set better correlates with the ground-truth. Additionally, L1180-1181 state that the reason why the authors choose 5% is because at 10% the selected layer for UCF and XDV remains the same but the layer chosen for UBNormal changes (Layer 28 -> Layer 29), which results in slightly better performance on UBNormal and corresponds to the oracle layer (Table 10).
  • Table 6 shows that the main results in Table 1 are the best seed. While the average across seeds exhibits very good performance, it does not result in sota on UBnormal (Anom-π\pi, weakly-supervised, beats it by 0.37%) and shrinks the margin on the other two to 0.25% (π\pi-VAD, weakly-supervised (contrary to what L362 claims) on UCF) and 1.09% (LAVIDA, training free, on XDV).
  • Some claims are overstated, some terms not defined and some hyperparameters not ablated. The transferability between datasets (L366-368) is not proven via experiments in which the model is trained on one dataset and tested on another, and Appendix E.2 says that zero-shot AUC on UBnormal is quite low due to the fact that language priors do not transfer to synthetic videos. I cannot find the definition of "normal semantic anchors" at L247, they are only mentioned in the pseudo-code of Algorithm 1 at page 24 as the final step of the algorithm. Table 11 shows that λreg=0\lambda_{reg}=0 results in the best AP on XDV but the authors pick λreg=0.1\lambda_{reg}=0.1 to stay within the plateau layers for UBnormal (L1208), but do not report what happens to the performance on UBnormal and UCF by setting λreg=0\lambda_{reg}=0 and if the resulting layer ends up in the plateau of those datasets or not. In general, it seems that the Fisher curves are the justification for multiple hyperparameter choices, but they are fitted on the test set temporal annotations (L1002). Other hyperparameters not ablated go against common practice in the field (i.e. the number of frames in a clip is usually 16 following the original protocol from Sultani et al. in 2018, the authors uniformly sample 4 frames in 24-frames non-overlapping windows). I do not consider this to be a major weakness, but the paper would benefit by an ablation or explanation.
Questions for authors

See weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.