acceptodds
Under review as a conference paper at ICLR 2027

Protect Before Evicting: Lagrangian-Guided Encoder Caching for Disaggregated Multimodal LLM Serving

Abstract

For multimodal models with separate encoders, repeated inputs allow encoder outputs to be reused across requests, even when the questions differ. A bounded encoder cache must decide which outputs to retain because their sizes, recomputation costs, and reuse frequencies differ. We present Protect Before Evicting, a Lagrangian-guided cache policy for disaggregated multimodal LLM serving. Offline calibration solves a scalar problem using an object profile and a cache budget. Online decisions use its solution to grant temporary protection, while allocations may override that protection when space is needed. The implementation preserves active-reference restrictions and hard capacity checks, without running an optimization solver on each request. We integrate the policy into vLLM and evaluate controlled image workloads with fixed profiles using Qwen3.5-9B on four H800 GPUs. At 16 requests/s, mean and p99 time to first token decrease by 10.03% and 13.04%, respectively, relative to vLLM's native encoder-cache replacement in the same serving stack. These results show a practical benefit from managing encoder-output retention separately from language generation.

Primary area:
optimization
Keywords:
Multimodal LLM Serving, LLM serving caching optimization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_YWr3OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Encoder output cache is a method that keeps encoding of multi-modal inputs (e.g., images) and reuses it when the same multi-modal input is utilized with different prompts. For such caching, this paper proposes a new cache replacement policy for the encoder output caches in multi-modal models. Although individual methods are not new, as the paper states, the main contribution of this paper is combining ideas utilized in different or similar domains. In particular, direct consideration of recomputation cost for eviction/protection decision is an interesting approach. The combination of proposed methods (Lagrangian calibration, object-specific protection window, probabilistic protection, and value-based forced eviction) were implemented and compared against vLLM's FIFO policy.

Critical strengths
  • The proposed method considers recomputation costs, along with access frequency and object size, when making cache retention and eviction decisions.
  • The proposed method combines offline Lagrangian calibration with lightweight online protection decisions, avoiding costly optimization at runtime.
Critical weaknesses
  • Insufficient ablation studies: The evaluation lacks comprehensive ablation studies that clarify the individual contributions of the proposed components, including Lagrangian calibration, object-specific protection windows, probabilistic protection, and retention value-based fallback eviction. This makes it unclear which components are contributing to the reported performance improvements in which degree.

  • Limited baselines: The evaluation compares the proposed method only against vLLM (FIFO-based eviction policy). Without comparisons against more advanced cache replacement policies, such as GreedyDual-Size-Frequency and SIEVE, it is difficult to understand the efficacy of the proposed method.

  • Limited evaluation settings: The evaluation relies on a small synthetic workload with predefined popularity distributions and a limited set of serving configurations. In particular, the experiments do not adequately investigate the impact of workload diversity, inaccuracies in calibration profiles, or variations in system configurations, such as cache capacity, worker configuration, and KV prefix caching. These limitations raise concerns about the robustness, scalability, and generalizability of the proposed method.

Questions for authors

Although the approach is interesting, the evaluation is overall weak to strongly support the efficacy of proposed methods. Addressing the following questions would help strengthen the evaluation and better support the novelty and efficacy of proposed methods.

  • Can you investigate the sensitivity of the proposed method to λ\lambda? Using the calibrated λ\lambda as the reference point, comparing results with scaled values (e.g., 0.25λ0.25 \lambda, 0.5λ0.5 \lambda, 2λ2 \lambda, and 4λ4 \lambda) would provide meaningful insights into the sensitivity of the proposed method and potentially demonstrate the importance of calibration.

  • Can you evaluate the effectiveness of probabilistic protection by comparing the proposed method against a deterministic variant (ρi=1\rho_i = 1), where protection is always granted with the same protection window hih_i?

  • Can you include additional cache replacement baselines (e.g., LRU, GreedyDual-Size-Frequency, S3-FIFO, and SIEVE) in the evaluation? In particular, I believe GreedyDual-Size-Frequency is an important baseline, as it considers access frequency, recomputation cost, and object size, which are also key factors in the proposed method. Comparing against GreedyDual-Size-Frequency would help clarify the advantages of the proposed approach over existing cost-aware cache replacement policies

  • The evaluation relies on a relatively small pool of 20 images with a fixed Zipf distribution (α = 0.5) and a fixed novel-image probability (0.3). Can you evaluate the proposed method under more diverse workloads, varying the image pool size, access distribution, and novel-image probability? Evaluating larger-scale workloads and more realistic access patterns would help demonstrate the generalizability of the proposed method.

  • The evaluation uses predefined popularity profiles that match the workload generation settings. However, in practical deployments, access patterns may change over time, and the estimated popularity may differ from the actual distribution. Can you investigate the robustness of the proposed method to inaccurate or outdated calibration profiles? For example, evaluating performance under different degrees of mismatch between the calibrated and actual popularity distributions would help demonstrate its practicality.

  • The evaluation is primarily conducted with a single model (Qwen3.5-9B), a 1E/3PD configuration, and a fixed encoder-cache capacity, with KV prefix caching disabled. Can you evaluate the proposed method under more diverse serving configurations, particularly with different cache capacities, worker configurations, and KV prefix caching enabled? Such experiments would help demonstrate whether the reported improvements generalize to different deployment settings.

  • GreedyDual-Size-Frequency (GDSF): Cherkasova, 1998
Reviewer_ssDKOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses the issue of encoder output cache management in disaggregated multimodal LLM services by proposing a cache protection strategy based on Lagrangian relaxation. The authors first determine the memory coefficient via a scalar Lagrangian optimisation based on offline profiling, and then probabilistically grant protection windows to objects online. Overall, this is a study focusing primarily on cache strategies and system integration.

Critical strengths

Experimental results show that this approach has some performance gains over the traditional approach.

Critical weaknesses
  1. The experiment only compares the vLLM’s native FIFO strategy; it does not compare it with strategies such as GreedyDual-Size, TinyLFU, and S3-FIFO, which are discussed in the paper. The gains relative to FIFO may stem primarily from the introduction of known popularity and cost information, rather than necessarily from Lagrangian calibration or probabilistic protection mechanisms. A robust baseline using the same profile information should be included.
  2. Key components are missing, making it impossible to identify the sources of returns. The contributions of probability-based protection, the protection window, value reversion and cost sensitivity have not been assessed separately.
  3. The workload and profile assumptions are rather idealised; the main experiments used only a single model, and generalisation ability has not been verified.
  4. The clarity of the figures needs to be improved further.
  5. The article as a whole is rather disjointed, lacks a central theme, and is not very logical, suggest you reorganise it.
Questions for authors

The hit rate counting window is not aligned with the client’s measurement boundaries, and does not distinguish between in-flight output sharing and cache reuse following reference release. There is a lack of valid data on queues, GPU utilisation and stage-by-stage execution times. The recalculation cost also serves as a proxy for the difference in hit/miss response times. Furthermore, eviction may involve repeated scans of the queue, whilst the shared file transfer directory is not subject to cache budget constraints and may continue to grow indefinitely. These issues require further measurement to assess performance in scenarios involving large object pools, long-running tasks and real-world transfer environments.

Reviewer_FKcaOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a caching algorithm for reusing visual encodings for multimodal LLMs using Lagrangian framework. While the algorithm is interesting, the paper is missing a critical comparison with related work and shows only small improvements of 10-15%.

Critical strengths

S1. Important problem.

Critical weaknesses

Thank you for submitting ICLR. I think you have a solid amount of work, yet it is missing critical related work comparison.

W1. Weak related work comparison.

a. The main problem with the paper is the weak baseline of plain vLLM using FIFO caching algorithm. Caching is a well-studied algorithm, as the authors also recognize in their related work section. However, the proposed algorithm is not compared with these existing work. The only baseline used is plain vLLM with FIFO, which is not enough to understand the contributions of the work.

b. Following up, what is the point of using Lagrangian optimization framework, compared to existing works that use a different one? This can come out with a detailed experimental analysis showing what existing successful algorithms cannot capture, while the Lagrangian can. This is missing unfortunately.

W2. Savings are too little.

a. Furthermore, the savings are too little (10-15%), despite the baseline being weak. This suggests that the proposed algorithm isn't indeed as successful. Compared to a stronger baseline, it is possible that it will perform worse.

W3. Further improvements (methodological & writing):

a. Using TTFT is a good metric, as it gives a sense of end-to-end savings. But you could also elaborate on the reduction of only the encoding part. That could show how much your method improves in the best case. You could also use more workloads with different access ratios and load patterns. GPU resource utilization is also another metric you could analyze to strengthen your arguments.

b. "temporary protection" concept in the intro is not well-defined. All protection is temporary in a cache, by definition. The whole trick is to find out which one will be better choice for the future.

c. Offline calibration, statistics, etc. are all standard caching methods. Existing works certainly use similar materials.

d. Audio and video results would improve your paper.

e. What kind of Lagrangian function do you have eventually? Do you always get a single solution (concave)? That's interesting. Why is that so? It is a relatively complex problem? How come variables of this problem can be nicely defined by a concave function? There should be some material explanation for that. And that insight would be very valuable.

f. Cost of lookup is probably small, but worth to mention somewhere.

g. w_t is not defined (line 187).

h. p_i defined later in the text (line 189)

i. Regarding "Thus, π controls retention preferences and eviction choices, while insertion is driven by observed accesses. Its decisions may be randomized and use the current access, cache state, past information, and available calibration data, but not future accesses."

Why does pi not control everything? Isn't the point of having one single powerful equation for the caching policy? Is this the reason for your Lagrangian to be concave (i.e., simple)?

j. (Line 215) "Introduce lambda >=0" --> "We introduce lambda >=0"

k. The gap D_i remind me of classic LRU... Is that so? If so, what's the point of using Lagrangian? I thought Lagrangian will handle all the problem in one single powerful equation...

Questions for authors

Q1. How do you think your method will compare with more sophisticated caching algorithms?

Q2. Why do you think your savings are only a little? Are these your best/worst-case scenarios? What would they be?

Q3. What does Lagrangian framework provide, which the existing algorithms cannot?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.