acceptodds
Under review as a conference paper at ICLR 2027

Cooldown Landmines: Cross-Tenant Interference Attacks on LLM Gateways

Abstract

LLM gateways enforce separate tenant quotas while sharing model deployments and cooldown records that temporarily exclude failing backends. However, a tenant's request failure can update these shared records and restrict other tenants' access to serviceable deployments. We identify two attacks that exploit this gap in LiteLLM. The first uses requests rejected at the key's requests-per-minute (RPM) limit: caller-supplied identifiers for known registered deployments reach failure handling, allowing two rejected requests to redirect another tenant to fallback with zero upstream calls from those requests. The second uses admitted traffic to create cooldown records that persist after backend capacity recovers. To address these failures, we design an origin check that blocks deployment updates from key RPM rejections and tenant-scoped cooldown that preserves the triggering tenant's back-off while retaining shared records for backend faults. Experiments with authenticated proxies and self-hosted vLLM demonstrate that the attacks can force fallback or denial while deployments remain serviceable. Across five paired four-worker runs, tenant scoping reduces victim fallback after recovery from to , while increasing attempts against exhausted shared quotas. These findings show that tenant isolation must cover both request admission and the failure handling that governs shared deployment availability.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Gateways, Multi-Tenancy, Denial of Service, Circuit Breakers, Routing Security, Shared State, LiteLLM

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_f1uqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies cross-tenant interference caused by shared cooldown state in LLM gateways. In LiteLLM, an attacker can use known deployment identifiers to let requests rejected by its own quota update shared cooldown state. Admitted requests can also trigger rate limits that leave other tenants excluded after capacity recovers. The authors propose failure-origin checks and tenant-scoped cooldown, evaluated using authenticated proxies, multiple workers, and self-hosted vLLM. Across five paired runs, tenant scoping reduces victim fallback from 54/60 to 0/60, but increases upstream requests.

Critical strengths
  1. Requests rejected by a tenant’s quota can change another tenant’s routing without contacting an upstream deployment. The deployment-identifier controls and failure-origin intervention provide convincing evidence for this finding.

  2. The authors combine quota-state measurements, direct upstream probes, and alternative cooldown policies to distinguish resource exhaustion from residual cooldown exclusion. Authenticated proxy and real-inference experiments further support the mechanism.

Critical weaknesses
  1. Under continuously returned 429 errors, median upstream requests increase from 15 to 209 with 64 tenants. The authors report this cost, but recovery benefits for legitimate tenants that have already triggered their own cooldown remain unclear. This limits confidence in its practical deployment benefits.

  2. Table 2 reports 54/60 fallback requests, while Appendix B.7 reports 41/60, without clearly identifying how these experiments relate. In the four-tenant 500-error experiment, successful recovery probes decrease from 35/40 to 13/40, and the cause remains unresolved. These issues affect the assessment of the defense’s operational behavior, although they do not establish that the core mechanism is incorrect.

Questions for authors
  1. Do the 54/60 result in Table 2 and the 41/60 result in Appendix B.7 come from different experimental batches?

  2. Why does tenant scoping reduce successful recovery probes in the four-tenant 500-error experiment? Does this reflect stricter enforcement of global cooldown or additional recovery delay?

  3. If legitimate tenants also receive 429 errors before quota recovery and acquire their own cooldown records, what recovery benefits do the existing experiments demonstrate for them?

2 comments
AuthorsOpenReview ↗

Tenant-Scoped Recovery Benefits and Experimental Clarifications

Thank you for your careful review and for highlighting the recovery and cost questions.

1. Who benefits from tenant scoping, and what does it cost?

M2 protects a tenant from cooldown records created by another tenant's 429. In the 64-tenant workload, after the backend switches back to 200, a previously idle key succeeds on 40/40 immediate probes with M2, versus 0/40 with stock. To measure the benefit for a tenant that had itself received 429s, we ran three paired four-tenant repetitions with separate idle and previously affected probe keys. M2 serves 24/24 idle-key probes and 4/24 previously affected probes, versus 0/24 for both under stock. Thus, M2 restores immediate access to an unaffected tenant and improves access for a previously affected tenant while retaining its own cooldown. Allowing each tenant to try the backend also increases upstream requests during the 429 phase. In five paired 64-tenant reruns after correcting the global reader, median upstream requests are 160 [158–173] with M2 versus 12 [9–13] with stock. These replace the submitted medians of 209 with M2 and 15 with stock. We will report the beneficiary cohorts and this request cost together in Section 6.4 and Figure 4.

2. Why do Table 2 and Appendix B.7 report 54/60 and 41/60?

Table 2 reports five paired stock and M2 runs, with 12 victim requests in each run. Stock sends 54/60 requests to fallback, compared with 0/60 under M2. Appendix B.7 reports 41/60 from a later set of five stock runs with additional Redis read tracing to examine worker visibility. The figures therefore come from separate measurements with different purposes. We will identify both sets of runs in the table discussion and appendix.

3. What caused the earlier 500 recovery result?

Thank you for flagging the 35/40 versus 13/40 discrepancy. While adding tenant records, our prototype combined tenant and global reads and bypassed LiteLLM's expiry correction for global records. We corrected the reader so global records use LiteLLM's stock path and tenant records are merged only for 429 failures. In five paired reruns of the same four-tenant 500 condition, both stock and M2 serve 40/40 immediate recovery probes. Therefore, the earlier figures are superseded. The corrected result preserves shared cooldown for 500 failures and matches stock recovery in this condition. We will update Section 6.4 and Appendix B.10 accordingly.

Reviewer_f1uqOpenReview ↗

Thank you for the clarifications

Thank you for the clarifications. Your response, together with your replies to the other reviewers, has given me a clearer understanding of the paper’s contributions. I believe my current rating appropriately reflects the paper’s contributions.

Reviewer_uVUsOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors present a cross-tenant interference attack in LiteLLM and propose two mitigation strategies: source verification and tenant-level cooldown state isolation.

Critical strengths
  1. The authors demonstrate an attack that manipulates the system states of other tenants through shared model deployments and cooldown records, highlighting that request isolation alone does not inherently guarantee the isolation of shared system state.
  2. Under the specified attack constraints, the attack chain is complete and the results are reproducible. The experimental design is well-structured and sufficiently comprehensive.
Critical weaknesses
  1. Practical limitations of the attack. Obtaining a victim's deployment ID typically incurs substantial costs, thereby limiting the attack's ability to selectively target specific victims.
Questions for authors
  1. The authors demonstrate an attack method based on Key RPM rejections. Could the authors explore alternative approaches to achieving similar attack effects?
  2. Have the authors considered reporting the identified security threats to the affected LLM gateway vendors?
6 comments
AuthorsOpenReview ↗

Deployment ID Visibility, Admitted Traffic, and Private Disclosure

Thank you for your thoughtful review and constructive questions.

1. Deployment IDs and targeting.

The local-rejection mechanism affects deployments shared by a model group. A is the authenticated attacker tenant, and B is a separate authenticated tenant using that group. We measured whether A can learn the deployment IDs through ordinary authorized use. In three fresh LiteLLM proxy runs under the same key allowance as the impact experiment, the first two admitted responses revealed both primary IDs, about one minute apart. B's normal request subsequently used one of those deployments in all three runs. Thus, in the evaluated shared-group setting, A obtained the required IDs within its existing key allowance, without access to B's key or administrative state.

2. Admitted traffic.

Section 4.2 evaluates a second path through admitted traffic. These requests follow ordinary routing and contain no deployment IDs. In the authenticated two-key experiment, A stays within its key limit while B is subsequently rerouted. Causal controls show that direct backend probes return HTTP 200 after the quota window clears, while the stock gateway continues to exclude B because cooldown remains active. Direct vLLM probes also return completions after gateway refusals. Thus, admitted traffic can sustain B's rerouting after backend capacity recovers.

3. Private disclosure.

We have submitted the key-RPM finding privately to the LiteLLM maintainers through GitHub's vulnerability-reporting channel. The report has entered GitHub's Triage state. We will record the disclosure in the Ethics Statement.

Reviewer_uVUsOpenReview ↗

Discussion of the Attack’s Limitations

Thank you for your prompt response. However, my concern has not been fully addressed:

  1. Practical Limitations of the Attack. According to the authors, for an attacker A to target a victim B, both must share the same model group. However, in an LLM proxy, the model groups available to A and B may overlap without being identical. In this setting, the method proposed for obtaining deployment IDs appears to have limited applicability. Could the authors explain how deployment IDs can be obtained in this case and clarify the corresponding constraints on the proposed attack?
AuthorsOpenReview ↗

Evidence Under Partially Overlapping Model Access

Thank you for clarifying the partially overlapping access case. We ran three paired trials with LiteLLM 1.99.0. A could access a shared model group and an A-only model, while B could access the same shared group and a B-only model. Permission checks confirmed that neither tenant could call the other’s private model. Both tenants could use the same two deployments in the shared group.

With A limited to 1 RPM, two ordinary requests 62 seconds apart revealed both shared deployment IDs in A’s own response metadata. B’s baseline request used one of these deployments in every trial. A then supplied only the IDs it had observed in locally rejected requests. All six rejections made zero upstream calls, and B used the configured fallback in all three trials. In paired controls without a registered shared ID, B remained on a shared primary deployment in all three trials. Thus, A can obtain deployment IDs through its own access to the shared group and affect B’s routing when their access lists differ. We will include this experiment and its per-run records in the revision and state the shared-deployment condition explicitly in the threat model.

Reviewer_EKJaOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies cross-tenant interference in LLM gateways, using LiteLLM as the main case study. It identifies two attacks that exploit shared backend cooldown state. The first uses locally rejected requests with attacker-supplied deployment IDs to trigger cooldown without upstream calls. The second exploits rate-limit-induced cooldowns that persist after backend recovery.

The authors demonstrate denial of service, policy-violating fallback, and manipulation of multi-model voting. They propose an origin check for local quota rejections and tenant-scoped cooldown for 429 errors. Experiments with LiteLLM, Redis, mock backends, and vLLM show that both attacks work and that the mitigations reduce cross-tenant interference.

Critical strengths
  1. The paper identifies a practical cross-tenant isolation issue in LiteLLM. The local rejection attack is traced through LiteLLM's source code, and the rate-limit attack is consistent with previously reported production behavior.

  2. The paper uses direct backend probes to distinguish gateway-induced denial from actual backend unavailability. It also evaluates mitigation costs, including increased upstream requests under tenant-scoped cooldown. These measurements make the tradeoffs clearer.

Critical weaknesses
  1. Table 2 reports 54/60 victim requests falling back, while Appendix B.7 reports 41/60 for what appears to be the same setup. Table 3 also gives identical upstream request counts with and without retries. More concerningly, the scoped policy serves only 13/40 recovery probes after 500 errors, compared with 35/40 for stock, despite supposedly unchanged handling of non-429 errors. I would like the authors to clarify these results.

  2. The mitigations address local key-quota rejections and 429 errors, but other attacker-controlled failures may still affect shared cooldown state. In particular, caller-supplied timeouts can produce cooldown-eligible 408 errors in the evaluated LiteLLM version. Neither proposed defense appears to cover this path. Other admission failures, such as TPM or budget limits, are also not evaluated. The paper should clarify the scope of its security claims.

Questions for authors
  1. Why does Table 2 report 54/60 fallback requests while Appendix B.7 reports 41/60? Are these different runs?

  2. Why are the upstream request counts identical with and without retries in Table 3?

  3. What explains the 13/40 versus 35/40 recovery results after 500 errors, given that non-429 handling should remain unchanged?

  4. Where is the supporting data for the 14 cross-worker observations discussed in Section 6.3?

  5. Do the mitigations cover TPM limits, budget rejections, and caller-controlled timeouts? If not, how should the defense claims be limited?

  6. Could different tenants sharing the same key alias or default namespace end up sharing cooldown state?

1 comment
AuthorsOpenReview ↗

Clarifying Cooldown Results, Defense Coverage, and Tenant Identity

Thank you for your thoughtful review and constructive questions.

1. Fallback counts.

Table 2 evaluates mitigation effectiveness through five paired stock/scoped runs. The stock fallback counts are 11, 10, 12, 12, and 9 out of 12 requests, totaling 54/60, while scoping sends 0/60 to fallback. Appendix B.7 uses five additional stock-only diagnostic runs with Redis refresh timing instrumentation to examine cooldown visibility across workers. Its 41/60 describes those diagnostic runs. We will state each experiment's purpose explicitly in the table discussion and appendix. Thus, 54/60 versus 0/60 measures mitigation effectiveness, while 41/60 supports the analysis of worker visibility.

2. Retry counts.

Table 3 counts actual backend HTTP requests under retry limits of zero and two. The retry limit sets the maximum allowed retries. Each attempt still checks whether a deployment is eligible. Once all candidates enter cooldown, further attempts stop before reaching a backend. Both stock settings therefore produce 10 backend calls in the measured schedule. Their final request outcomes differ: one run records 8 successes, 2 upstream 429s, and 50 local denials without retries, versus 8 successes, 1 upstream 429, and 51 local denials with retries. Thus, equal backend totals follow from deployment eligibility checks, while retries still affect the final responses. We will clarify retry limits and measured backend calls in the table discussion.

3. Recovery after 500 errors.

Thank you for highlighting this discrepancy. We traced it to the prototype's reader implementation. Adding tenant records had combined tenant and global keys in a single read, changing LiteLLM's global expiry-correction path. We corrected the reader to reuse the stock global read path and merge tenant records only for 429 errors. Non-429 writes and global records continue to use shared cooldown. We then reran five paired repetitions of the four-tenant condition in question: both stock and scoping serve 40/40 recovery probes. These replace the earlier 35/40 (stock) and 13/40 (scoped) totals. Therefore, the corrected implementation matches stock recovery in the evaluated 500 condition while preserving tenant isolation for 429 records.

4. Cross-worker evidence.

The cross-worker-links.csv file is in artifact/evidence/ in the anonymous repository. The supporting per-worker traces and stock reanalysis are in artifact/evidence/supporting-evidence.zip. The CSV lists 14 request/deployment links across seven victim requests. The five runs contribute 4, 4, 0, 6, and 0 links. Each row identifies the deployment, reader and writer processes, inducing attacker request, and remaining cooldown at the victim's read. Thus, these records trace all 14 observations from attacker writes to live cooldown records read by other workers.

5. Defense coverage.

We added end-to-end controls for key TPM and budget rejections, with three repetitions per condition. In stock, two locally rejected requests create cooldown records and reroute the victim in every repetition, with zero upstream calls from those requests. The origin check (M1) prevents the cooldown writes, and callback traces confirm that the origin marker is retained and deployment failure counters remain unchanged. Tenant-scoped cooldown (M2) also keeps the independent victim on a primary in every repetition by separating its 429 cooldown view from the attacker's. M1 also retains cooldown for actual provider 429 and 408 responses.

Caller-controlled timeouts still produce shared 408 cooldown and reroute the victim under both defenses. These failures arise after admission. Thus, M1 covers the tested local quota rejections, and M2 isolates 429 records. Caller-induced timeouts require separate failure-origin handling. We will make this distinction explicit in Sections 5 and 6.

6. Tenant identity.

We strengthened the scoped implementation to use only team_id from LiteLLM's authenticated key record. It is bound to the key by the server and carried in request context for cache reads and writes. Keys assigned to the same team share that team's record, while distinct team IDs use separate records. The hook no longer falls back to a key alias, user ID, or GLOBAL. Requests without a valid team ID are rejected before cache access.

In three repeated proxy controls, LiteLLM generated distinct UUID team IDs for the attacker and victim. Duplicate alias creation returned HTTP 400, and calls using a key without a team ID returned HTTP 403 in every repetition. Cache regression tests also verify that keys assigned to the same team share a record and distinct team IDs remain isolated. Thus, cooldown follows authenticated team membership, and requests missing a team identity are rejected before cache access.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Threat Model and Evaluation Updates

Dear Area Chair,

Thank you for overseeing the review process. We also thank all reviewers for their thoughtful feedback. The discussions helped us clarify the scope of our claims and strengthen the evaluation.

In the revised manuscript, we clarify the conditions under which tenants with partially overlapping model permissions can affect one another and add a paired experiment for that setting. We also explain why the two fallback experiments report different counts, document the correction to the global cooldown reader, and report the revised HTTP 500 recovery results. Finally, we expand the evaluation of mitigation coverage, tenant recovery, and upstream request costs, and update the Ethics Statement to record the private disclosure.

We have addressed all points raised in the reviews and follow-up discussions in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.