- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses context management for long-horizon agents and argues that shortening the prompt does not by itself reduce cost, because edits to the history break prefix-cache reuse. It can even raise cost: on an HCE history, immediate replacement removes 55.93% of cumulative input tokens yet increases input cost by 3.67%. The question therefore becomes when to reduce input tokens so that the agent benefits without paying for cache reconstruction. The authors propose CADOC, a scheduler for input-token reduction whose commit rule follows from the economic-order-quantity (EOQ) model of operations research, which gives the timing and condition under which a batch of reductions should be committed. The reduction itself moves structured spans, such as file contents and tool outputs, to a working memory and leaves compact Cards in the prompt, with byte-exact retrieval on demand. Validation consists of frozen-history replays against Immediate, fixed-batch and token-threshold policies, and live runs on Terminal-Bench 2.1, LongMemEval-V2 and SWE-bench Verified. In the live runs, input cost falls by 34.12–47.13% versus full context, and task completion drops by at most two tasks per benchmark.
Critical strengths
The commit rule is derived rather than tuned, and its predictions hold. The EOQ argument gives a closed-form commit condition from observable quantities and also yields a 2-competitive online bound for constant reconstruction cost. The replays behave as the theory predicts: the best fixed batch size shifts across hot-tail settings (b = 2 at h = 2, b = 8 at h = 4, 8, 16), while CADOC adapts without manual adjustment. The rule also recovers a known special case: at w_r = 1, where cached tokens get no discount, CADOC reduces to Immediate, as Eq. 32 shows. I found no prior work that derives commit timing from observed cache cost with a competitive guarantee. This matters because it lets the method carry over to new workloads without per-setting tuning.
The replay experiment is carefully controlled. Model outputs, object versions and Card contents are frozen; only commit timing varies, and every policy sees the same histories. Under this control, CADOC has the lowest input cost in every setting, tying Immediate only where the cache discount vanishes (w_r = 1).
Critical weaknesses
The live evidence has no error bars, so randomness cannot be ruled out as the source of the cost reduction or the completion drop. Table 1 comes from one chain per arm, so the 34.12–47.13% cost reductions and the completion changes (38→36 of 89, 10→10 of 18, 31→29 of 50) have no variance estimate, and LongMemEval-V2 has only 18 tasks. Because FULL and CADOC generate separate stochastic trajectories, the cost difference can also reflect changes in model calls, output lengths, tool use and retries rather than the commit policy alone.
The reported savings are model input-cost savings, not end-to-end agent-system cost. Input tokens are a useful proxy when inference dominates and behavior is comparable, but practitioners benefit from the overall end-to-end cost, which is not measured. For example, Figure 5 shows only cumulative input cost, not end-to-end cost, latency or throughput.
Questions for authors
Could you add variance or error bars to the Table 1 cost reductions (for example, repeated chains or bootstrap over tasks), and report output tokens, tool calls and retries per arm, so that the policy effect can be separated from trajectory variation?
Could you report CADOC's end-to-end monetary cost, wall-clock latency and throughput compared with FULL and Immediate?
3 comments
Thank you for the additional experiments. The repeated runs substantially address my concern about variance: CADOC continues to show large input-cost reductions across all three benchmarks, and the frozen-action replay provides stronger evidence that the reduction comes from the context policy rather than trajectory variation. The provider-billing results also strengthen the practical cost claim, with CADOC costing less than both FULL and Immediate in every benchmark. I will therefore keep my rating at 3 (weak acceptance).
We sincerely thank the reviewer for the recognition and constructive suggestions. We are glad that our responses have addressed your concerns, and we truly appreciate your valuable feedback. We wish you all the best and continued success in your work!
We thank the reviewer for the positive assessment of the derived commit rule and the controlled replay design. We address both points below.
W1 / Q1: Variance and separating the policy effect from trajectory variation
Repeated chains. To assess run-to-run variation, we repeated the benchmark evaluations using gpt-5.6-terra, with identical task lists and configurations across runs. We conducted 3 runs per arm for Terminal-Bench 2.1 and SWE-bench , and 5 runs per arm for LongMemEval-v2, as the former two benchmarks incur substantially higher evaluation costs. We report the mean and standard deviation of solved tasks across runs.
Difference from the original results. The reruns were conducted approximately four months after the original experiments, using the same task lists, CADOC scheduling configuration, and cost accounting. However, both FULL and CADOC exhibited substantial changes in their execution trajectories, suggesting that changes in the API serving stack or underlying model backend may have contributed to the differences. For example, on LongMemEval, FULL's model requests increased about 42% on average, while its average input length decreased 38% on average. CADOC's requests decreased about 48%, with average input length decreasing about 34%. These changes in both arms suggest that the execution environment may have shifted between evaluations. A concrete example is LongMemEval task
3a7a7880. In the original CADOC run, the agent made 55 model requests before failing. In the five reruns, it averaged only 8.2 requests, but still failed the task. This single task illustrates how changes in execution trajectories can substantially affect measured costs even when task outcomes remain unchanged.With agent actions held fixed, CADOC still reduces input cost by 49.83% on Terminal-Bench, 46.75% on SWE-bench, and 75.57% on LongMemEval-V2 (Section 6.1, Figure 7).
Per-arm trajectory statistics. As suggested, we report per arm the quantities that could differ between trajectories:
Output tokens per arm are reported in the cost table under W2/Q2 below. Agent-level retries, such as re-running a tool or revising a step after an error, are already counted in the model requests and tool calls above; API-level retries mainly reflect the stability of the provider's service rather than the context policy.
W2 / Q2: End-to-end monetary cost, latency, and throughput
We agree that the overall cost is what matters most to practitioners. The table below reports the token usage and billed cost returned by the provider. Billing records cover over 90% of inference requests but are not available for every request.
Latency and throughput. Our primary objective is cost reduction rather than latency or throughput optimization. Moreover, substantial fluctuations in API serving conditions make wall-clock comparisons difficult. Nevertheless, CADOC requires fewer model requests than FULL across all three benchmarks, and fewer tool calls on Terminal-Bench and SWE-bench (Appendix C.2). These reductions suggest that CADOC may also achieve lower latency and higher throughput under stable serving conditions.
We thank the reviewer for the valuable comments and hope that our responses have addressed your concerns.