acceptodds
Under review as a conference paper at ICLR 2027

AnomMerge: Spatially Bounded Token Merging for Video Anomaly Understanding

Abstract

Video anomaly understanding (VAU) with multimodal large language models (MLLMs) aims to recognize, temporally localize, and explain abnormal events. Under limited visual token budgets, existing VAU systems often trade temporal coverage for spatial detail, potentially removing evidence needed for precise event boundaries. Separately, we observe that current VAU models can produce overly broad and low-dispersion anomaly intervals, while standard overlap metrics may not fully expose this degeneration. To address these challenges, we propose AnomMerge, a learnable token compression framework for VAU that reduces spatial redundancy while preserving observed temporal units and their positional correspondence. We further develop adaptive spatial merging with unit-specific similarity thresholds and spatial and feature consistency constraints to preserve fine-grained temporal evidence for anomaly localization. To better evaluate localization behavior, we complement standard overlap metrics with coverage-aware temporal IoU (CA-tIoU) and interval dispersion contraction (IDC), which measure excess temporal coverage and contraction of predicted interval dispersion, respectively. On Vad-Reasoning, AnomMerge reduces IDC by 44.3% relative to VAU-R1, while improving video-level anomaly recognition and retaining only 36.59% of the original visual tokens. The code will be publicly released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Anomaly Understanding, Multimodal Large Language Models, Visual Token Compression, Temporal Localization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_C297OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses video anomaly understanding with MLLMs under a fixed visual-token budget. It keeps all temporal units and compresses only spatial redundancy within each unit, and introduces two metrics for interval coverage and dispersion. The core idea "never sacrificing temporal resolution" is sound and the module is lightweight, but the method is an incremental combination of existing token-merging and Mamba aggregation, and the experiments do not support the localization claims.

Critical strengths
  1. The efficiency problem is real, and the temporal-coverage vs. spatial-detail trade-off is a sensible framing of it.
  2. This paper points out that for generative video anomaly understanding, models suffer from interval collapse in their predicted anomaly temporal intervals
Critical weaknesses
  1. This paper shows limited innovation and mainly assembles existing techniques. Specifically, AnomMerge repurposes established modules including STAC‑derived Mamba aggregation, token-merging-style spatial merging, statistical‑driven MLP thresholds, and the straight‑through estimator. It adapts these off‑the‑shelf components for VAU via engineering tweaks rather than proposing fundamentally new algorithmic or modeling paradigms.

  2. The paper repeatedly claims temporal merging corrupts localization evidence. Yet it provides no quantitative statistics identifying which anomaly categories fail under temporal merging baselines, such as short‑duration anomalies or spatially tiny anomalies. Sample‑level statistical support is absent.

  3. The same configuration gives different numbers in different tables. For example, Table 1(a) ("no merging", 100% retention, Qwen2.5-VL-3B, 1 FPS) reports Acc 0.938 / F1 0.933, while Table 6's "Qwen2.5-VL-3B (SFT), 1 FPS, 100%" reports Acc 0.879 / F1 0.856. The 0.059–0.077 gap is unexplained, and which row is the true uncompressed reference.

  4. IDC is not a localization metric, yet it carries the headline result in this paper. It compares dataset-level endpoint dispersion, so it is silent on whether any individual interval is correct. In Table 6, models with mIoU 0.253 and 0.270 have IDC = 0.000. It is also computed over a model-dependent subset V, so abstaining shifts it, and as a single scalar it cannot support a significance test.

  5. This paper lacks sufficient ablation for \lambda involved in CA‑tIoU. It does not analyse metric behaviours under different \lambda values or discuss \lambda’s generalisability across datasets and videos with varying anomaly ratios. Correlation between CA‑tIoU scores and human annotations remains unexamined. It is unclear whether CA‑tIoU aligns with human‑perceived localisation quality.

  6. IDC measures distributional shrinkage of predicted interval endpoints and only captures dataset‑level statistics rather than direct localisation accuracy. Though the paper acknowledges this limitation, reduced IDC is heavily promoted as a key result. IDC can yield favourable values even for fully incorrect predictions.

  7. The proposed framework heavily relies on Qwen‑VL backbones. Experiments are conducted only on Qwen‑family models. No transfer evaluations are performed on other mainstream multimodal large models such as Intern‑VL and LLaVA‑NeXT‑Video. This makes it difficult to demonstrate the broad generalisability of the compression framework.

Questions for authors
  1. The experimental design has notable flaws with unfair and insufficient comparisons. For example,
  1. Baselines like FastVID, VidCom2 and FlashVID are training‑free, while AnomMerge requires joint fine‑tuning of context modules, threshold heads and LoRA adapters. Direct comparison mixes trainable and training‑free setups, leaving performance gains ambiguous. Control experiments with LoRA applied to baselines are missing.
  2. The other question is that AnomMerge degrades heavily on real‑world surveillance subsets of VANE‑Bench versus full‑token VAU‑R1. The authors only blame hard surveillance scenes, without diagnosing whether spatial merging destroys small anomaly cues or adaptive thresholds fail. Error and failure‑case analyses are absent.
  3. The temporal‑merging ablation uses merely one hyperparameter configuration. It cannot prove temporal merging is universally worse than intra‑frame spatial merging. Temporal merging may deliver competitive results under suitable token budgets.
  1. See the above weakness of other questions.
5 comments
AuthorsOpenReview ↗

Response to Reviewer C297

W1. Contribution

As discussed in General response G1, we identify interval collapse and introduce CA-tIoU/IDC to complement overlap metrics. Reviewer HQ4S also recognizes the importance of balancing temporal coverage and spatial detail. We address this issue with learned thresholds and constrained spatial grouping, preserving sampled temporal units and their positions while reducing spatial redundancy.

Mamba, LoRA and surrogate gradients are established tools. With similar retention, our merging achieves F1 0.8455 versus 0.8274 for pooling without Mamba.

W2. Temporal merging

Thank you for this suggestion. We added category, duration and case analyses of temporal versus spatial merging in Appendix C.3.

Table 14. Recognition and localization by anomaly category.

CategoryCountCA-tIoU (Temporal)CA-tIoU (Spatial)Recall (Temporal, %)Recall (Spatial, %)
Traffic accidents170.15350.209888.2100.0
Riot80.50090.528675.087.5
Shooting80.07690.177587.5100.0
Wrong-way driving70.22930.427671.4100.0
Others1640.40900.401788.489.6

For the tested configurations, Table 14 shows higher CA-tIoU and recall with spatial merging for traffic accidents, shooting, wrong-way driving and riot. Improvements cover both short and long events, but short falling-object events have lower CA-tIoU. Thus, the results show no consistent duration pattern. The annotations lack spatial anomaly regions, so we cannot quantify performance by spatial anomaly size.

Case analysis. Figure 8 shows that temporal merging misses the four-second event in RoadAccidents091 and predicts an overly broad interval in ECVA/713; spatial merging produces an interval closer to GT in the latter case.

W3. Reference rows

Table 1(a) is the uncompressed reference for AnomMerge's merging ablation: it retains spatiotemporal Mamba and LoRA with all merging disabled. The SFT row in Table 7 is the plain LoRA-finetuned backbone without Mamba. Both retain all visual tokens. We have clarified this SFT configuration in blue in the revised paper.

The reported gains of 0.059 in Acc and 0.077 in F1 therefore come from adding spatiotemporal Mamba during task fine-tuning.

W4. IDC subsets and uncertainty

Thank you for these comments. General response G3 clarifies IDC's role in diagnosing interval collapse.

(1) In Table 7, IDC=0 indicates no endpoint-dispersion contraction; varied but inaccurate boundaries can still yield low mIoU.

(2) Model-dependent subsets affect comparisons, but IDC still diagnoses collapse within each evaluated subset. A single IDC value does not establish statistical significance.

W5. CA-tIoU penalty sensitivity

Thank you for this suggestion. We added sensitivity experiments at λ=0, 1, 5, 10 (Appendix C.2). By source, Vad-Reasoning comprises UCF-Crime, XD-Violence, TAD, ShanghaiTech, UBnormal, ECVA and additional Internet videos (NEW).

(1) Across sources, increasing λ changes the AnomMerge/VAD-R1 ranking on UCF-Crime, XD-Violence and ECVA, while AnomMerge remains higher on TAD and NEW. Thus, the effect of penalizing excess coverage varies across sources.

(2) Across anomaly fractions, VAD-R1 shows a larger proportional score decline at low occupancy. AnomMerge remains higher in the low- and middle-occupancy groups, while the high-occupancy ranking reverses at λ=10. We retain λ=1 as an interpretable unit-weight penalty and report mIoU on abnormal videos alongside CA-tIoU.

(3) Regarding human annotations, CA-tIoU directly compares predictions with reference temporal boundaries through overlap and excess coverage. This assesses agreement with annotated intervals; correlation with independent human ratings of localization quality has not been measured.

W6. IDC scope

Thank you for highlighting this distinction. We will report IDC as a supplementary diagnostic alongside recognition and localization metrics, and will not use its reduction alone to claim improved VAU performance.

AuthorsOpenReview ↗

Response to Reviewer C297 (2)

W7. Transfer across model families

Thank you for highlighting this limitation. We added InternVL3.5-2B and GLM-4.1V-9B-Thinking to Table 5, providing additional evidence of AnomMerge's generalizability across model families.

Table 5. AnomMerge across model sizes and backbone families.

BackboneSizeAcc ↑F1 ↑CA-tIoU ↑IDC ↓mIoU ↑
Qwen2.5-VL3B0.9320.9250.3830.1420.722
Qwen2.5-VL7B0.9410.9350.4200.1180.748
Qwen3-VL2B0.9160.9050.4260.0560.746
Qwen3.52B0.9320.9220.4310.1170.765
InternVL3.52B0.9270.9180.4040.0990.745
GLM-4.1V-Thinking9B0.9480.9420.4270.0980.760

To accommodate InternVL's frame count within our experimental context budget, we use a 108-frame cap, compared with 768 for Qwen and GLM, with uniform sampling across the full video when the cap is exceeded.

Q1.1. Adaptation fairness

Thank you for pointing this out. FastVID, VidCom2, FlashVID and OTT-Vid are applied to Qwen2.5-VL-3B fine-tuned with LoRA, using the same data, 1 FPS sampling and LoRA settings as AnomMerge. Here, "training-free" describes the compression algorithms; the model has already been adapted to VAU. AnomMerge learns compression during task fine-tuning, and the results in Table 7 suggest that this joint adaptation can bring further gains.

Q1.2. VANE failures

Thank you for requesting a diagnosis of the Avenue/Ped1 performance drop. We evaluated AnomMerge and the independently trained no-merge model from Table 1(a) on the same 21 clips (63 questions), with Mamba and LoRA retained in both models. We compared accuracy and changes in answer correctness by activity (Table 15; Appendix D.1).

Table 15. Analysis by activity category in the Avenue and Ped1 subsets of VANE-Bench.

SourceActivityCountVAD-R1 (Acc %)VAU-R1 (Acc %)Merge (Acc %)No merge (Acc %)RecoveredLost
AvenueBag throwing18100.0094.4472.2288.8941
AvenuePaper throwing977.7888.8977.7877.7800
AvenueDancing616.6750.0033.3333.3300
Ped1Skateboarding955.5666.6733.3333.3300
Ped1Wheelchair use633.3333.3333.3366.6731
Ped1Bicycles/carts1580.0086.6780.0086.6710

Recovered counts AnomMerge errors corrected by the no-merge model; Lost counts AnomMerge-correct questions answered incorrectly by the no-merge model.

The no-merge model improves accuracy on bag throwing from 72.22% to 88.89% and wheelchair use from 33.33% to 66.67%. However, dancing and skateboarding remain at 33.33%, so merging does not explain all failures.

Case analysis. Figure 10 illustrates failures involving fine-grained posture and object-interaction cues in low-resolution surveillance frames. Figure 11 shows recovered bag-throwing and wheelchair answers, consistent with sensitivity to spatial compression. The no-merge model was trained independently, so this comparison does not isolate the merging operation from adaptive-threshold effects.

Q1.3. Temporal merging budgets

Table 2 reports the comparison with manually adjusted k and approximately matched retention. At higher budgets, temporal merging has slightly higher CA-tIoU, while spatial merging generally favors recognition; recognition is similar around 62% retention. Neither method improves monotonically with more tokens. As discussed in Reviewer HQ4S Q4, our learned spatial thresholds achieve higher F1 with fewer tokens than the tested fixed-k settings.

Thank you again for your valuable questions and comments. We have added detailed experiments and made corresponding revisions to address your concerns. As the discussion period is nearing its end, we would be happy to provide further clarification if any concerns remain.

Reviewer_tqbzOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript proposes AnomMerge, a time-preserving token compression framework for generative video anomaly understanding. The scheme of the framework decreases spatial redundancy by adaptive spatial merging while preserving observed temporal units and positional correspondence.

Critical strengths

Superficially, the proposed scheme can obtain a very promising performance increase.

Critical weaknesses

The following shortcomings consist of negative aspects of the paper. 1.Generally speaking, the proposed scheme can be viewed as an improvement to existing techniques. In this sense, there is no striking originality. 2.The foundational frameworks of Token Merging originate from ToMe, TokenLearner and STAC. The underlying idea of "similarity-based clustering merging" is not first proposed in this paper. The bidirectional temporal encoding with Mamba reuses the module from STAC, with only the causal scan modified to bidirectional scan. 3. STE straight-through estimator, LoRA fine-tuning and regularized compression loss are well-known and all well-established techniques in this field. 4.There are no sufficient experimental results and analyses. Also, the data used are limited. Therefore, it seems that the presented conclusions are convincing enough.

Questions for authors
  1. As there are no sufficient experimental results and analyses, I look forward to seeing more experimental comparison.
  2. I hope that more data can be used to test the proposed scheme.
  3. Can the originality of the paper be further interpreted?
2 comments
AuthorsOpenReview ↗

Response to Reviewer tqbz

Q1. Additional comparisons

To strengthen the empirical evidence, we conducted additional experiments and analyses in three areas.

Ablation studies. (1) Aggregation without Mamba. Without Mamba, AnomMerge improves F1 over spatial average pooling from 0.8274 to 0.8455, supporting a contribution beyond contextualization (Table 3). (2) Temporal versus spatial merging. Comparisons at approximately matched inference budgets show that temporal merging remains competitive at higher retention (Table 2). (3) Threshold coefficient ablation. The threshold ablation shows that learned thresholds achieve higher F1 with fewer tokens than the tested fixed coefficients (Table 12).

Diagnostic analyses. (1) Localization and metric sensitivity. Abnormal-video localization and CA-tIoU sensitivity analyses clarify how interval collapse, anomaly fraction and penalty weight affect localization scores (Table 13; Appendices C.1-C.2). (2) Category, duration and case analysis. Category, duration and case analyses reveal scenario-dependent gains with no consistent duration trend (Table 14; Appendix C.3). (3) Surveillance failure diagnosis. The no-merge comparison on Avenue and Ped1 recovers bag-throwing and wheelchair answers, identifying compression-sensitive failures (Table 15; Appendix D.1).

Generalization and inference efficiency. (1) Evaluation across backbones. Experiments on InternVL and GLM examine transfer across model families, using different frame caps across backbones (Table 5). (2) Evaluation on additional datasets. Evaluation on MSAD and ECVA examines transfer beyond the original test set without further tuning (Table 9; Appendix D.2). (3) End-to-end inference profiling. End-to-end profiling shows that Mamba and merging account for less than 1% of total latency, while autoregressive generation dominates (Appendix E).

Q2. Evaluation scope

Thank you for this suggestion. We added evaluation of video-level anomaly recognition and temporal localization on the MSAD and ECVA subsets of VAU-Bench without further tuning.

Table 9 shows that AnomMerge improves F1 from 0.707 to 0.842 and CA-tIoU from 0.273 to 0.342 over VAU-R1, with less interval collapse. These gains indicate improved recognition and more temporally specific predictions on the additional datasets. VAD-R1 retains higher recognition and overlap scores, but its predicted intervals nearly span the entire video.

Table 16 in Appendix D.2 shows dataset-specific differences: false alarms limit overall mIoU on MSAD, while ECVA's predominantly abnormal composition makes recognition scores mainly reflect anomaly recall. Figure 12 illustrates the localization behavior: AnomMerge excludes excess background in the MSAD assault case and covers the complete ECVA fighting event, which VAU-R1 captures only partially.

Q3. Originality

The identification of interval collapse, recognized as a strength by Reviewer C297, is part of the contribution described in General response G1. We connect this diagnosis to preserving temporal observations and merging spatial tokens within each temporal unit under explicit constraints. Established contextualization and optimization tools support the implementation.

Table 4 probes this design: removing spatial extent reduces F1 from 0.925 to 0.904; removing centroid consistency raises IDC from 0.142 to 0.235. Budgets differ, so these support the design without isolating budget-independent effects. Q1 adds a comparison without Mamba to examine spatial aggregation separately.

Thank you again for your valuable questions and comments. We have added detailed experiments and made corresponding revisions to address your concerns. As the discussion period is nearing its end, we would be happy to provide further clarification if any concerns remain.

Reviewer_HQ4SOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AnomMerge, a spatial token merging framework for video anomaly understanding. It applies spatiotemporal Mamba contextualization and adaptive spatial grouping to reduce visual token redundancy while preserving temporal units. The authors also introduce two metrics, CA-tIoU and IDC, to evaluate temporal localization behavior. Experiments on Vad-Reasoning and VANE-Bench demonstrate substantial token reduction and improvements in anomaly recognition, although the results on temporal localization are mixed.

Critical strengths
  1. The paper addresses an important trade-off between visual token efficiency, temporal coverage, and spatial detail in video anomaly understanding. Preserving temporal units while compressing spatial redundancy is a reasonable design for detecting and localizing abnormal events.

  2. The experiments include comparisons with existing VAU models and video token compression methods, component ablations, multiple backbone models, and cross-dataset evaluation. The substantial reduction in visual token usage with relatively small performance degradation compared with the uncompressed model demonstrates practical potential.

Critical weaknesses
  1. The method heavily builds on existing spatiotemporal Mamba contextualization, and the ablation results suggest that much of the performance improvement comes from this component. In Appendix Table 9, adding full Mamba contextualization improves F1 from 0.846 to 0.925. In contrast, Table 2 shows that adaptive spatial merging improves F1 by only 0.010 over simple spatial average pooling, while achieving slightly lower CA-tIoU and mIoU. Furthermore, Table 6 compares the jointly trained AnomMerge with several training-free compression methods, making it difficult to isolate the benefit of the proposed merging strategy. The effectiveness of the learned threshold and its surrogate-gradient optimization is also insufficiently validated.

  2. Although AnomMerge improves anomaly recognition and reduces IDC, its conventional temporal localization performance is worse than VAU-R1 in Table 5. Specifically, mIoU decreases from 0.739 to 0.722, and [email protected] decreases from 0.743 to 0.713, while CA-tIoU improves only marginally from 0.373 to 0.383. Moreover, IDC measures the dispersion of predicted intervals rather than their correctness, so a low IDC does not necessarily indicate accurate localization. The cross-dataset results in Table 7 also show performance degradation on six of eight VANE-Bench subsets, including substantial drops on Avenue and Ped1. These observations weaken the claims regarding precise temporal localization and generalization.

Questions for authors

Q1. The ablation results suggest that spatiotemporal Mamba contextualization contributes substantially to the overall performance. For example, Appendix Table 9 shows an F1 improvement from 0.846 to 0.925 when adding full Mamba contextualization, whereas Table 2 shows only a 0.010 F1 improvement over spatial average pooling. Could the authors clarify the independent contribution of adaptive spatial merging and provide stronger comparisons under matched token budgets and training settings?

Q2. In Table 5, AnomMerge improves anomaly recognition but performs worse than VAU-R1 on conventional localization metrics, including mIoU (0.722 vs. 0.739) and [email protected] (0.713 vs. 0.743). How do the authors reconcile these results with the claim that preserving temporal units improves anomaly localization? Could the authors provide more detailed analysis on abnormal-only localization performance and different anomaly durations?

Q3. The proposed IDC metric measures interval dispersion rather than localization correctness. For example, Table 6 reports IDC = 0 for the uncompressed Qwen2.5-VL-3B under the 128-frame protocol, despite its relatively low mIoU of 0.253. How do the authors establish that IDC provides meaningful additional evidence of localization quality? In addition, how are missing or invalid predicted intervals handled when computing IDC?

Q4.Appendix A.2 uses a surrogate-gradient formulation that adds the same threshold-dependent term to every pooled token, while the actual grouping decisions remain discrete. How does this surrogate gradient reflect the influence of merging decisions on the downstream task loss? Could the authors compare the learned thresholds with fixed or non-learnable thresholds at similar compression ratios?

Q5. Table 6 shows very large improvements under the 128-frame protocol, where F1 increases from 0.664 for the uncompressed backbone to 0.913 for AnomMerge. Could the authors clarify whether all methods use identical supervision, fine-tuning procedures, input preprocessing, and output parsing? In addition, could the authors report complete end-to-end inference latency and peak GPU memory, including the cost of spatiotemporal contextualization and token grouping?

Q6. Table 7 shows that AnomMerge performs worse than VAU-R1 on six of eight VANE-Bench subsets, with particularly large drops on Avenue (66.67 vs. 84.85) and Ped1 (56.67 vs. 70.00). Could the authors analyze how compression affects short videos, brief anomalies, and different scene types, and explain the conditions under which AnomMerge can preserve performance?

3 comments
AuthorsOpenReview ↗

Response to Reviewer HQ4S

Q1. Independent spatial contribution

Thank you for raising the question of the independent contribution of adaptive spatial merging. We add a Mamba column to revised paper Table 3 to compare the same aggregation strategies with and without Mamba under the same training settings:

No MambaRet. (%)AccF1CA-tIoUIDCmIoU
Spatial average pooling40.040.85650.82740.34180.32640.7075
STAC-S54.550.86560.84100.35560.27900.7110
AnomMerge39.370.87020.84550.35160.28520.7142

Without Mamba, our F1 gain over spatial average pooling is 0.0181, larger than 0.010 with Mamba in Table 3. This supports a contribution from spatial aggregation beyond Mamba. Compared with native STAC-S, AnomMerge achieves comparable video-level anomaly recognition and temporal localization performance with fewer visual tokens.

Q2. Collapse and anomaly duration

Thank you for this suggestion. VAU-R1 also uses 1 FPS, so comparing it with AnomMerge does not isolate the benefit of retaining additional temporal units. We therefore include VAD-R1 with uniform 16-frame sampling as a sparse-sampling reference.

For GT-abnormal videos, we provide predicted intervals alongside GT in Appendix C.1, Fig. 6. Our inspection shows that VAD-R1's valid intervals nearly all span the full video, consistent with interval collapse.

We further report results by anomaly fraction in Appendix C.1, Fig. 5. Near-full predictions can obtain higher overlap scores when the anomaly occupies more of the video; this need not indicate more accurate boundaries. When the fraction is smaller, excess background coverage reduces their overlap scores.

Q3. IDC interpretation

We agree that IDC does not measure localization accuracy. For Qwen2.5-VL-3B evaluated under the 128-frame protocol in Table 7, IDC=0 indicates that predicted endpoint dispersion is not lower than GT dispersion. Such predictions may vary across videos while remaining poorly aligned with GT intervals, resulting in low mIoU.

Missing or invalid intervals are excluded from IDC and assigned zero overlap in mIoU and CA-tIoU on abnormal videos. As discussed in General response G3, IDC diagnoses interval collapse and complements recognition and localization metrics for a more comprehensive evaluation.

Q4. Threshold learning

Thank you for raising this point. In Appendix A.2, the added term is zero in the forward pass but allows task gradients to reach the threshold predictor. Although the term is shared, pooled tokens contribute distinct gradients whose sum updates k and hence subsequent grouping. This supplies task feedback rather than an exact gradient through discrete grouping.

Table 12 in Appendix B.3 reports a coefficient ablation using the same checkpoint and inference settings. The learned threshold configuration retains 36.59% of tokens, achieving F1=0.9254 and CA-tIoU=0.3828. The tested fixed coefficients (k=0.5/0.8/1.0) retain 49.76% to 71.41% of tokens.

These results demonstrate the effectiveness of our learned threshold configuration: it achieves higher recognition F1 and CA-tIoU while retaining fewer visual tokens than the tested fixed coefficients.

Q5. Protocol consistency and efficiency

All methods share annotations, 128-frame evaluation inputs and resized grids, and output parsing; fine-tuning follows native procedures.

FlexSelect trains a selector by ranking distillation; QTSplus trains compression/re-encoding modules with answer loss and regularization. Both freeze the LM. AnomMerge jointly trains LoRA, Mamba and threshold predictors with answer loss and compression regularization.

Appendix E reports 1-FPS E2E TTFT of 2.413 s, full-answer latency of 10.794 s and peak allocated GPU memory of 12.71 GB. In separate profiling, Mamba and token grouping/pooling take 14.4 and 61.3 ms, respectively, together under 1% of profiled inference time.

AuthorsOpenReview ↗

Response to Reviewer HQ4S (2)

Q6. VANE degradation

We agree that the Avenue/Ped1 deficits require further analysis. As summarized in General response G2, Appendix D.1 adds activity-wise error analysis and a comparison with an independently trained no-merge model retaining Mamba and LoRA on the same VANE-Bench inputs.

Table 15. Analysis by activity category in the Avenue and Ped1 subsets of VANE-Bench.

SourceActivityCountVAD-R1 (Acc %)VAU-R1 (Acc %)Merge (Acc %)No merge (Acc %)RecoveredLost
AvenueBag throwing18100.0094.4472.2288.8941
AvenuePaper throwing977.7888.8977.7877.7800
AvenueDancing616.6750.0033.3333.3300
Ped1Skateboarding955.5666.6733.3333.3300
Ped1Wheelchair use633.3333.3333.3366.6731
Ped1Bicycles/carts1580.0086.6780.0086.6710

For fine-grained activities and human–object interactions in real-world surveillance, Table 15 in Appendix D.1 shows that no-merge accuracy rises from 72.22% to 88.89% for bag throwing and from 33.33% to 66.67% for wheelchair use, while dancing and skateboarding remain at 33.33%. Figure 10 illustrates confusion of bag throwing with normal walking and of skateboarding with dog walking; Figure 11 shows the no-merge model recovering correct recognition of bag throwing and wheelchair use. These results suggest that fine action and object cues can be weakened by compression, especially in low-resolution Ped1 frames.

Table 8 shows that AnomMerge performs best on VideoLCM and remains competitive on OpenSora, both generated-video subsets. In the examined VideoLCM cases, it correctly identifies hand deformation and unnatural foot transformation, illustrating preserved recognition of body-part anomalies in generated videos.

Thank you again for your valuable questions and comments. We have added detailed experiments and made corresponding revisions to address your concerns. As the discussion period is nearing its end, we would be happy to provide further clarification if any concerns remain.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the reviewers for their constructive feedback and for recognizing our motivation and experimental coverage (Reviewer HQ4S), promising performance gains (Reviewer tqbz), and our problem formulation and identification of interval collapse (Reviewer C297). We have carefully considered all comments and revised the manuscript with additional experiments, analyses and clarifications, which we summarize below before addressing each reviewer's questions individually.

G1. Contribution and motivation

We identify and analyze interval collapse in existing generative VAU models: near-full-video predictions can obtain high overlap scores for long anomalies. We therefore introduce CA-tIoU to penalize excess coverage more strongly and IDC to quantify cross-video interval dispersion contraction.

This diagnosis motivates retaining denser temporal units to preserve localization evidence. We make these inputs affordable through learnable spatial merging with connected groups, spatial extent bounds, centroid consistency and source-coordinate preservation. Together, these components form a complete framework integrating collapse analysis, temporal evidence preservation and spatial compression. Mamba, LoRA and surrogate gradients provide established implementation tools.

G2. Performance differences on VANE-Bench

We further analyze performance differences between AnomMerge and other methods on VANE-Bench (Appendix D.1). With only ten frames per video, spatial compression cannot extend temporal coverage.

AnomMerge underperforms both full-token baselines on Avenue and Ped1. The examined cases require fine-grained action, posture and object cues, particularly in low-resolution Ped1 scenes (Fig. 10). To examine merging's effect, we test the no-merge model from Table 1(a), which improves accuracy on both subsets. Figure 11 shows corrected bag-throwing and wheelchair cases, suggesting that spatial merging can weaken these cues.

G3. IDC interpretation

We introduced IDC to quantify cross-video interval collapse that overlap metrics do not directly reveal. It compares predicted and GT endpoint dispersion on each model's subset of GT-abnormal videos with valid interval predictions. IDC=0 indicates no contraction relative to GT, without implying accurate boundaries.

Recognition and output validity affect this subset, but IDC still diagnoses collapse within it. Recognition metrics assess classification errors, while mIoU and CA-tIoU on abnormal videos assign zero overlap to missing or invalid intervals. Together, these metrics provide a more comprehensive VAU evaluation and help the community study interval collapse.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.