- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Encoder output cache is a method that keeps encoding of multi-modal inputs (e.g., images) and reuses it when the same multi-modal input is utilized with different prompts. For such caching, this paper proposes a new cache replacement policy for the encoder output caches in multi-modal models. Although individual methods are not new, as the paper states, the main contribution of this paper is combining ideas utilized in different or similar domains. In particular, direct consideration of recomputation cost for eviction/protection decision is an interesting approach. The combination of proposed methods (Lagrangian calibration, object-specific protection window, probabilistic protection, and value-based forced eviction) were implemented and compared against vLLM's FIFO policy.
Critical strengths
- The proposed method considers recomputation costs, along with access frequency and object size, when making cache retention and eviction decisions.
- The proposed method combines offline Lagrangian calibration with lightweight online protection decisions, avoiding costly optimization at runtime.
Critical weaknesses
-
Insufficient ablation studies: The evaluation lacks comprehensive ablation studies that clarify the individual contributions of the proposed components, including Lagrangian calibration, object-specific protection windows, probabilistic protection, and retention value-based fallback eviction. This makes it unclear which components are contributing to the reported performance improvements in which degree.
-
Limited baselines: The evaluation compares the proposed method only against vLLM (FIFO-based eviction policy). Without comparisons against more advanced cache replacement policies, such as GreedyDual-Size-Frequency and SIEVE, it is difficult to understand the efficacy of the proposed method.
-
Limited evaluation settings: The evaluation relies on a small synthetic workload with predefined popularity distributions and a limited set of serving configurations. In particular, the experiments do not adequately investigate the impact of workload diversity, inaccuracies in calibration profiles, or variations in system configurations, such as cache capacity, worker configuration, and KV prefix caching. These limitations raise concerns about the robustness, scalability, and generalizability of the proposed method.
Questions for authors
Although the approach is interesting, the evaluation is overall weak to strongly support the efficacy of proposed methods. Addressing the following questions would help strengthen the evaluation and better support the novelty and efficacy of proposed methods.
-
Can you investigate the sensitivity of the proposed method to ? Using the calibrated as the reference point, comparing results with scaled values (e.g., , , , and ) would provide meaningful insights into the sensitivity of the proposed method and potentially demonstrate the importance of calibration.
-
Can you evaluate the effectiveness of probabilistic protection by comparing the proposed method against a deterministic variant (), where protection is always granted with the same protection window ?
-
Can you include additional cache replacement baselines (e.g., LRU, GreedyDual-Size-Frequency, S3-FIFO, and SIEVE) in the evaluation? In particular, I believe GreedyDual-Size-Frequency is an important baseline, as it considers access frequency, recomputation cost, and object size, which are also key factors in the proposed method. Comparing against GreedyDual-Size-Frequency would help clarify the advantages of the proposed approach over existing cost-aware cache replacement policies
-
The evaluation relies on a relatively small pool of 20 images with a fixed Zipf distribution (α = 0.5) and a fixed novel-image probability (0.3). Can you evaluate the proposed method under more diverse workloads, varying the image pool size, access distribution, and novel-image probability? Evaluating larger-scale workloads and more realistic access patterns would help demonstrate the generalizability of the proposed method.
-
The evaluation uses predefined popularity profiles that match the workload generation settings. However, in practical deployments, access patterns may change over time, and the estimated popularity may differ from the actual distribution. Can you investigate the robustness of the proposed method to inaccurate or outdated calibration profiles? For example, evaluating performance under different degrees of mismatch between the calibrated and actual popularity distributions would help demonstrate its practicality.
-
The evaluation is primarily conducted with a single model (Qwen3.5-9B), a 1E/3PD configuration, and a fixed encoder-cache capacity, with KV prefix caching disabled. Can you evaluate the proposed method under more diverse serving configurations, particularly with different cache capacities, worker configurations, and KV prefix caching enabled? Such experiments would help demonstrate whether the reported improvements generalize to different deployment settings.
- GreedyDual-Size-Frequency (GDSF): Cherkasova, 1998