- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
- TAO jointly selects waiting agent sessions for admission and assigns them to inference nodes.
- It lexicographically prioritizes decayed cache credit, tool readiness, and context length, using classical dynamic programming under capacity constraints.
- Experiments cover multiple models, workloads, and concurrency levels, with task execution, trace replay, and component comparisons.
Critical strengths
- Joint admission and placement account for competition over node capacity and can exploit combinations missed by sequential placement.
- With valuation held fixed, DP improves throughput by approximately 14.9% and reduces mean TTFT by approximately 21.0% relative to BFD in the reported comparison.
- Multi-model experiments and trace replay provide complementary evidence. Component comparisons also support the usefulness of cache and readiness information.
Critical weaknesses
- The strict scoring order lacks direct validation. A small advantage in total cache credit overrides any readiness advantage. Existing ablations do not establish this ordering’s advantage over alternative priorities or weighted combinations.
- The fixed-valuation DP–BFD serving comparison covers one configuration. Broader system comparisons do not clarify which workload or capacity conditions make exact joint optimization particularly beneficial.
- Capacity quantization and search-budget settings are unspecified in the paper. Reported computational overhead excludes exhausted searches and BFD fallback, leaving the complete computational cost unclear.
I tentatively favor acceptance for the substantial implementation and reusable evaluation workflow, pending clarification of the scoring rule and performance gains.
Questions for authors
- Why should cache credit strictly precede readiness? Have alternative priority orders or weighted combinations been compared under the same solver?
- Which scheduling conditions explain DP’s gains over BFD? Do fixed-valuation comparisons in other tested configurations support these gains?
- What quantization and search-budget settings were used? What is the computational overhead including exhausted searches and fallback?
5 comments
W3 & Q3: DP and BFD computational overhead
Thank you for catching this gap in our overhead reporting. We account for DP Planner, with BFD as the final fallback(Appendix I in Revised Paper). The capacity quantum is 1,024 tokens, and the search budget is 50,000 sparse states. Budget exhaustion selects internal LP solving; LP failure or its 150 ms deadline invokes BFD. This preserves the paper's DP planner → BFD organization.
We now show the complete DP Planner cost here:
| Deployment | Workers | DP search mean (ms) | DP search P95 (ms) | DP-search fallback rate | BFD mean (ms) | BFD P95 (ms) |
|---|---|---|---|---|---|---|
| DeepSeek, 2 replicas | 64 | 12.50 | 88.84 | 5.24% | 0.023 | 0.032 |
| Qwen, 4 replicas | 160 | 1.59 | 3.68 | 8.98% | 0.022 | 0.032 |
| GLM, 1 replica | 160 | 3.85 | 14.63 | 0.00% | 0.033 | 0.050 |
Note that the DP Planner accounting includes internal LP work and exhausted DP searches in the same DP column.
For each snapshot, DP Planner consists of DP search and sparse LP stage, with BFD as the fallback. Async completion additionally retains process management and BFD execution (Tables 5 and 10 in Revised Paper).
| Replicas | Candidates | DP Planner mean (ms) | DP Planner P95 (ms) | BFD Fallback mean (ms) | Async mean (ms) | Async P95 (ms) |
|---|---|---|---|---|---|---|
| 2 | 16 | 12.08 | 19.04 | — | 15.91 | 23.04 |
| 16 | 64 | 12.40 | 13.87 | — | 14.08 | 16.03 |
| 32 | 160 | 20.53 | 22.09 | — | 22.45 | 24.23 |
| 64 | 128 | 18.99 | 20.69 | — | 20.84 | 22.73 |
| 128 | 512 | 81.94 | 95.30 | — | 84.88 | 98.25 |
Stage means include zeros for unvisited stages; “—” means BFD was not invoked. Timed-out internal LP intervals retain the 150 ms cutoff. Async completion uses actual wall time. The cutoff does not limit preceding DP work.
Asynchronous execution keeps this solving cost separate from synchronous preparation. Preparation averages 1.48 ms, 0.94 ms, and 2.18 ms for DeepSeek, Qwen, and GLM, or 0.033%, 0.044%, and 0.078% of runtime (Table 4 in Revised Paper).
Dear Reviewer ZHNv,
Thank you for your careful evaluation of our work. With the discussion window closing soon, we would welcome your thoughts on whether our responses and revisions have adequately resolved the issues you raised. Any additional comments or suggestions would be valuable in further strengthening our submission.
Best regards,
The Authors
Dear Reviewer ZHNv,
Thank you for your careful reading and constructive feedback. We appreciate your understanding and recognition of the key arguments presented in our paper. Below, we respond to your comments with additional explanations and experiments.
W1 & Q1: Lexicographic priorities and weighted alternatives
Our ordering gives cache credit, readiness, and token count distinct roles (Secs. 3.2–3.4 in Original paper):
We now compare readiness-first and tokens-first on SWE-Lite with 300 tasks per workload. Here, ki,b, ri, and ti denote these three signals; > denotes priority. TAO rows are historical paper results, while the alternatives are supplementary runs.
We also test four weighted objectives to examine numerical trade-offs. The objective is aki,b/32768+bri+cti/32768, where ri is binary and 32,768 is the configured context limit. The table reports coefficients (a,b,c) in cache–readiness–token order.
Taken together, the two tables support explicit priorities without requiring workload-specific weight calibration. The best weighted setting changes across workloads: cache weighting gives DeepSeek's lowest E2E, while readiness weighting leads on GLM. Our original deisgn is better than any other settings.
W2 & Q2: Scheduling conditions and comparisons with BFD
DP helps when limited capacity couples session selection with replica placement. Different footprints and cache locations can make a greedy placement exclude a higher-value combination (Sec. 3.5 and Appendix G in Original paper).
The table presents two more configurations using the model DeepSeek and Qwen with TAO's scoring. Each workload uses SWE-Lite with 300 tasks.
Taken together, these two configurations show lower TTFT and higher decode throughput under TAO. DeepSeek also has lower E2E latency, while Qwen has a 0.9% E2E increase.