acceptodds
Withdrawn from ICLR 2027

Towards Faster Anomaly Detection and Reasoning via KV Cache Compression

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Industrial anomaly detection (IAD) has become a crucial technology for identifying anomalies in complex real-world environments. Recently, multimodal large language models (MLLMs) have demonstrated outstanding global visual understanding and zero-shot cross-domain reasoning capabilities across general multimodal tasks. As a result, several studies have begun applying MLLMs to IAD-related tasks by integrating domain knowledge and specialized methods to enable causal reasoning about anomalies, achieving competitive results on relevant benchmarks. However, these approaches often overlook the high computational cost associated with MLLM inference and the substantial key-value (KV) cache redundancy introduced by high-resolution images. This study first investigates the attention distribution patterns of MLLMs in the IAD domain. Our observations reveal that models predominantly focus their attention on normal regions, while only a few KV pairs corresponding to anomalous areas receive effective attention—potentially leading to incomplete perception of anomalies. To address this, we propose AnomalyKV, a novel KV cache compression method that combines anomaly awareness with efficient inference. Our approach adaptively allocates KV cache budgets at a fine-grained level, enabling instance-level dynamic compression based on differences among input samples. Extensive experiments show that AnomalyKV minimizes performance degradation across multiple mainstream MLLMs while significantly improving inference efficiency and reducing inference time costs. Notably, AnomalyKV achieves 2.57× real-time acceleration in full-scenario tasks, matching the performance of models with full KV caches and substantially outperforming existing mainstream KV cache compression methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Anomaly Detection, KV Cache Compression

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zK41OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Towards the issue of high inference overhead in anomaly detection using MLLMs, this paper proposes AnomalyKV, a novel anomaly-aware KV cache compression framework. This method performs fine-grained KV cache budget allocation, enabling instance-level dynamic compression across different samples. The general idea is to first compute an anomaly map for each sample, then convert it into an importance field through boundary diffusion, and adaptively set a retain threshold to preserve the patches with the highest scores. Extensive experiments demonstrate that AnomalyKV significantly improves inference efficiency while maintaining comparable performance across multiple mainstream MLLMs.

Critical strengths
  1. The issue addressed in this paper is valuable: the inference cost of using MLLM in anomaly detection is high. Considering how to reduce the inference cost of MLLM, this paper addresses this issue from the perspective of KV cache compression.
  2. The motivation behind the entire method is clear: anomaly-aware KV cache compression, which aims to retain as much of the KV cache in anomaly-related areas as possible, thus avoiding the loss of anomaly information after compression. The entire method is designed in a fancy yet reasonable manner, revolving around this motivation. Experimental results show that it achieves an improvement in inference speed while maintaining detection performance.
Critical weaknesses
  1. The innovation is limited. First, an anomaly map is calculated, and then patches that are more likely to be abnormal areas are selected and retained based on the anomaly map. This is a fairly common compression method that can be thought of for anomaly detection.
  2. In LAMS, an anomaly map can only be obtained by comparing features with N images on the L layer. Does N refer to all the images for the class in the dataset? Then the cost of calculating the anomaly map would be very high.
  3. In Section 3.2, why do we still need to obtain the importance field through diffusion fields? Can't we directly use the anomaly map to select which tokens to retain? Why does the anomaly map need to be converted into a bounding box set first, and then into an importance field? This approach makes it more complicated. What are the advantages of doing so? There is also no ablation experiment to analyze the necessity of this module.
  4. What about the compression on normal samples? If the normal samples also save areas that are more likely to be abnormal, and the model inference only sees part of the area instead of the whole image, will it easily lead to misjudgments of normal samples?
  5. The comparison method: SnapKV, AdaKV, and PyramidKV are all methods in the LLM field, so strictly speaking, the comparison is not very fair. It is also advisable not to include all ablation experiments in the appendix. The results of the most crucial experiments should be presented in the main text.
Questions for authors
  1. Table 1 doesn’t present numerical results for inference cost, with comparisons only shown in Figure 3. How can we confirm that Table 1 and Figure 3 are corresponding? If Figure 3 only focuses on maximizing speed without considering performance, actually with loss of performance.
  2. Is this method applicable to inputting multiple images simultaneously? The mainstream paradigm of AD MLLM generally inputs one or multiple normal reference images simultaneously, which is more reasonable in anomaly detection. In this case, how should it be compressed? Are existing methods still applicable?
Reviewer_oHbBOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes an anomaly-aware KV cache compression framework for efficient industrial anomaly detection and reasoning with MLLMs. The method first employs cross-image patch similarity to identify potential anomalous regions through. It then constructs a boundary-aware importance field using a reaction–diffusion process and dynamically allocates KV cache budgets to preserve anomaly-related information. Experiments on AnomalyCoT demonstrate improved inference efficiency while maintaining competitive anomaly understanding performance.

Critical strengths
  1. The proposed anomaly-aware KV cache compression strategy achieves competitive performance across three MLLMs while substantially reducing KV cache budgets and inference time.

  2. The proposed method is training-free and can be applied to different MLLMs without modifying their model parameters, making it flexible and potentially practical for deployment.

Critical weaknesses
  1. The paper claims that MLLMs allocate insufficient attention to anomalous regions, potentially leading to incomplete anomaly perception. However, limited quantitative evidence is provided to support this claim.
  2. Normal regions may also provide essential contextual information for anomaly detection and reasoning. The paper does not sufficiently investigate the importance of normal information.
  3. The proposed LAMS module relies on cross-image patch similarity from the same category. Besides, the performance of the method may be sensitive to the composition of the reference group.
  4. The figure reference in line 418 is missing (shown as "Fig. ??").
  5. Table 1 contains several baseline configurations without corresponding AnomalyKV results. For example, maybe we only consider the results on KV Cache Size ≤ 256 and Full KV for qwenvl-2.5.
Questions for authors

Please refer to weakness section

Reviewer_chJtOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes AnomalyKV, a training-free, anomaly-guided KV-cache compression method for MLLM-based industrial anomaly reasoning. It first localizes candidate anomalous regions using mutual patch scoring over images from the same product category, then constructs a reaction-diffusion importance field and uses it to assign an instance-adaptive visual-token budget before prefill.

The paper evaluates AnomalyKV on AnomalyCoT with Qwen2.5-VL-7B-Instruct, InternVL3-8B, and Gemma4-E-4B-IT, comparing against SnapKV, AdaKV, and PyramidKV. The reported results suggest substantially better accuracy preservation than these baselines at constrained cache budgets, together with reduced end-to-end inference time.

Critical strengths

(1) The paper targets a practically meaningful problem.

MLLM-based anomaly reasoning often uses high-resolution visual inputs, and cache reduction can be valuable for industrial settings where latency and memory are genuine constraints. Framing compression around defect evidence rather than generic attention salience is a sensible domain-specific direction.

(2) The pipeline is conceptually coherent and clearly visualized.

The progression from mutual patch scoring, to an anomaly map, to a boundary-aware importance field, and finally to block-level token allocation makes the intended role of each component understandable. In particular, the distinction between high-, medium-, and low-keep blocks communicates why the method is not simply global visual token pruning.

Critical weaknesses

(1) The computational efficiency and online applicability of category-level mutual scoring remain insufficiently justified.

The method is not per-image in the usual online-inference sense. In Section 3.1, LAMS computes all ordered image-pair, patch-to-image nearest-neighbor distances, with stated additional complexity O(N2Lmax2)O(N^2L^{2}_{max}). This requires collecting multiple images in the same category and assumes that the category is available as a group before inference. This conflicts with the deployment implication of real-time industrial inspection, where a model may need to process a single item as it arrives, may not know its category reliably, or may not have access to a sufficiently large same-category reference batch.

The paper asserts at the end of Section 3.3 that planning is “well under one percent” of end-to-end wall-clock time, but provides neither a timing breakdown nor measurements as a function of NN, input resolution, LmaxL_{max}, or category size.

(2) The experimental evaluation and methodological description require further improvement.

The baselines in Section 4.1 are limited and do not adequately cover recent adaptive KV-compression methods, particularly those involving multimodal[1], task-adaptive[2], and head-aware strategies[3]. Moreover, essential implementation details, including diffusion hyperparameter settings and token allocation procedures, are missing, making reproducibility difficult. Section 4.2 also references a prompt template that is not provided. Additionally, all figures and tables from Section 4.3 are placed in the appendix, with no corresponding visual results in the main text, weakening the presentation and support of the experimental claims.

[1]: https://arxiv.org/abs/2511.16786 [2]: https://aclanthology.org/2025.findings-emnlp.426/ [3]: https://arxiv.org/abs/2504.09261

Questions for authors
  1. What is the exact deployment unit for LAMS? Does every inference require access to all images from the same category, and if so, how are category membership, group size, and reference images determined in a streaming industrial-inspection setting?

  2. Can the authors provide an end-to-end timing decomposition for each model and budget: image preprocessing, vision encoding, LAMS, box extraction, diffusion-field construction, token planning, prefill, and autoregressive decoding?

  3. How robust is the approach when a category contains many anomalous samples, multiple defect types, substantial viewpoint variation, or no reliable same-category batch? These settings challenge the core LAMS assumption that normal images dominate the peer set.

  4. Can the authors quantify their motivating observation using ground-truth boxes: for each model, what fraction of attention or attribution mass falls inside anomaly boxes before and after compression? Is this related to answer correctness, defect area, and defect type?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.