- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies cross-tenant interference caused by shared cooldown state in LLM gateways. In LiteLLM, an attacker can use known deployment identifiers to let requests rejected by its own quota update shared cooldown state. Admitted requests can also trigger rate limits that leave other tenants excluded after capacity recovers. The authors propose failure-origin checks and tenant-scoped cooldown, evaluated using authenticated proxies, multiple workers, and self-hosted vLLM. Across five paired runs, tenant scoping reduces victim fallback from 54/60 to 0/60, but increases upstream requests.
Critical strengths
-
Requests rejected by a tenant’s quota can change another tenant’s routing without contacting an upstream deployment. The deployment-identifier controls and failure-origin intervention provide convincing evidence for this finding.
-
The authors combine quota-state measurements, direct upstream probes, and alternative cooldown policies to distinguish resource exhaustion from residual cooldown exclusion. Authenticated proxy and real-inference experiments further support the mechanism.
Critical weaknesses
-
Under continuously returned 429 errors, median upstream requests increase from 15 to 209 with 64 tenants. The authors report this cost, but recovery benefits for legitimate tenants that have already triggered their own cooldown remain unclear. This limits confidence in its practical deployment benefits.
-
Table 2 reports 54/60 fallback requests, while Appendix B.7 reports 41/60, without clearly identifying how these experiments relate. In the four-tenant 500-error experiment, successful recovery probes decrease from 35/40 to 13/40, and the cause remains unresolved. These issues affect the assessment of the defense’s operational behavior, although they do not establish that the core mechanism is incorrect.
Questions for authors
-
Do the 54/60 result in Table 2 and the 41/60 result in Appendix B.7 come from different experimental batches?
-
Why does tenant scoping reduce successful recovery probes in the four-tenant 500-error experiment? Does this reflect stricter enforcement of global cooldown or additional recovery delay?
-
If legitimate tenants also receive 429 errors before quota recovery and acquire their own cooldown records, what recovery benefits do the existing experiments demonstrate for them?
2 comments
Thank you for the clarifications
Thank you for the clarifications. Your response, together with your replies to the other reviewers, has given me a clearer understanding of the paper’s contributions. I believe my current rating appropriately reflects the paper’s contributions.
Tenant-Scoped Recovery Benefits and Experimental Clarifications
Thank you for your careful review and for highlighting the recovery and cost questions.
1. Who benefits from tenant scoping, and what does it cost?
M2 protects a tenant from cooldown records created by another tenant's 429. In the 64-tenant workload, after the backend switches back to 200, a previously idle key succeeds on 40/40 immediate probes with M2, versus 0/40 with stock. To measure the benefit for a tenant that had itself received 429s, we ran three paired four-tenant repetitions with separate idle and previously affected probe keys. M2 serves 24/24 idle-key probes and 4/24 previously affected probes, versus 0/24 for both under stock. Thus, M2 restores immediate access to an unaffected tenant and improves access for a previously affected tenant while retaining its own cooldown. Allowing each tenant to try the backend also increases upstream requests during the 429 phase. In five paired 64-tenant reruns after correcting the global reader, median upstream requests are 160 [158–173] with M2 versus 12 [9–13] with stock. These replace the submitted medians of 209 with M2 and 15 with stock. We will report the beneficiary cohorts and this request cost together in Section 6.4 and Figure 4.
2. Why do Table 2 and Appendix B.7 report 54/60 and 41/60?
Table 2 reports five paired stock and M2 runs, with 12 victim requests in each run. Stock sends 54/60 requests to fallback, compared with 0/60 under M2. Appendix B.7 reports 41/60 from a later set of five stock runs with additional Redis read tracing to examine worker visibility. The figures therefore come from separate measurements with different purposes. We will identify both sets of runs in the table discussion and appendix.
3. What caused the earlier 500 recovery result?
Thank you for flagging the 35/40 versus 13/40 discrepancy. While adding tenant records, our prototype combined tenant and global reads and bypassed LiteLLM's expiry correction for global records. We corrected the reader so global records use LiteLLM's stock path and tenant records are merged only for 429 failures. In five paired reruns of the same four-tenant 500 condition, both stock and M2 serve 40/40 immediate recovery probes. Therefore, the earlier figures are superseded. The corrected result preserves shared cooldown for 500 failures and matches stock recovery in this condition. We will update Section 6.4 and Appendix B.10 accordingly.