- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors introduce TACO, a blackbox RL framework to train agents in a non-intrusive way, without modifying the agent’s source code. With minimal wrapping code, TACO enables online training with an asynchronous RL framework of agents behind black-box services. The framework introduces concepts like bounded-token distance to train directly on full-agent traces rather than training on isolated parts of the trace. TACO is evaluated on a math agent, an agent that generates Triton kernels, and ZeroClaw, a Rust agent service. The authors report improved task-specific held-out performance in all three settings, with detailed efficiency comparisons conducted on the math agent.
Critical strengths
-
Training on production harnesses with no harness edits is a useful contribution. The paper is very timely as we move to having dedicated harnesses for each application and need to RL directly on production, which is valuable to minimise training-deployment gaps.
-
The experimental setup covers a large distribution of use-cases The three integrations featured in the paper provide evidence that the TACO framework works across different Agent service implementations of very different natures. TACO reports improvements on all three holdout evaluation settings.
Critical weaknesses
- The TACO framework is not discussed relative to related work, with major omissions. In particular, Polar (May 2026, https://arxiv.org/abs/2605.24220) trains unchanged agent harnesses through a model-API proxy, captures token-level trajectories, supports asynchronous rollouts, and studies prefix-based reconstruction.
- The efficiency–accuracy trade-off of approximate packing is not established. Packing changes both training sequences and number of optimizer steps. This implies the reported speedups compare different effective training setups. The mini-batch ablation recovers accuracy but does not isolate the effect of rewriting contexts. Comparing with matched optimisation budget would better analyse the cost of the approximation.
Questions for authors
- What are TACO's framework contribution relative to related work like Polar (May 2026, https://arxiv.org/abs/2605.24220) and Agent Lightning v1.0 (August 2026; https://arxiv.org/abs/2608.17528)?
- Why is approximate packing preferable to reconstructing the full text trace? Could you compare both with the original recorded calls, using matched training settings, to show how rewriting the history affects accuracy?
- What causes the observed token-prefix mismatches: intended harness behavior, the text/token round trip, or avoidable serving artifacts? Could token continuity be preserved without changing the agent’s intended behavior, instead of relying on approximate packing? Is it possible to provide illustrative examples?
9 comments
Response to Reviewer To1S (Part 2 / 4)
Continuing from the previous part
2. Asynchronous execution. Polar fully decouples training and inference; a rollout may span multiple policy versions, and version consistency is left to the downstream RL engine. Agent Lightning pauses the gateway before an update and returns 429 to new requests, relying on agent-side client retries. A black-box agent without built-in retry logic will instead enter its error-handling path, and the resulting error messages may be folded into its context and pollute subsequent turns; an exhausted retry budget can also fail the task. TACO targets non-preemptible black-box tasks: it stops admission at a synchronization point and drains admitted tasks before switching weights; timed-out tasks are marked failed and their late requests are rejected. Every model call within a rollout is thus served by the same weight version. The agent never receives abnormal returns during normal execution, and no assumption is made about its retry logic. We also compare the three packing methods within the same asynchronous TACO pipeline, holding the model, data, GPU allocation, and mini-batch size (m=72) fixed. The result is as follows:
| Packing method | End-to-end time | Speedup over strict-prefix packing |
|---|---|---|
| TACO | 9.79 h | 2.05× |
| Polar | 9.52 h | 2.11× |
| Agent Lightning | 20.11 h | 1.00× |
The comparison shows that the merging method, not the surrounding system, drives the efficiency gap: strict-prefix packing is about 2× slower because it leaves most drifted boundaries unmerged, while TACO and Polar are comparable (9.79 h vs. 9.52 h, within run-to-run variation; the primary TACO run is 8.72 h). TACO's slightly lower merge rate is the price of its stricter acceptance checks, which bound the context deviation as described above. TACO's contribution relative to Polar is thus not a higher merge rate or speed, but a bounded and quantified approximation, together with the rollout-level weight consistency described above and the deployment-regime difference discussed in point 3.
3. Black-box training design. Agent Lightning embeds the rollout ID in the proxy URL, injected by its Controller when it launches the agent process or Pod for each rollout. Polar uses the API key as the session ID, injected when it creates an isolated runtime per session. Both assume that the framework owns the agent's lifecycle and that the configuration can change per task. TACO targets a different deployment: a service that is already deployed, configured once, and long-running. Its configuration fields are fixed for the service's lifetime and cannot carry a task-varying rollout ID. TACO therefore carries only a stable worker ID in the fixed field and maintains the worker-to-rollout mapping in external Redis, updated before each task. In short, Agent Lightning and Polar black-box the agent's interaction loop, while TACO treats the deployment itself as a black box. This is what "online, black-box agent services" in our title means.
Response to Reviewer To1S (Part 3 / 4)
W2/Q2:
Our unpacked baseline trains directly on the original recorded LLM calls, keeping each prompt–response pair as a separate training sequence under its original context. We use the same model, prompt budget, and GPU allocation, and aim to keep the number of response tokens used for training comparable.
The evaluated traces contain both retokenization drift and small textual changes introduced by the chat template or tool-call parser, so our accuracy experiments cover both types of mismatch. Large-scale history rewriting requires different treatment. When summarization or deletion substantially changes the history, the shared context needed to reuse earlier computation may no longer exist. Forcing these turns into one sequence would replace the context actually seen by the later call with an outdated history. Such calls should remain separate training sequences. TACO enforces this through its anchor and edit-distance checks: a missing anchor or a difference exceeding the budget starts a new segment. Changes that destroy the shared prompt prefix also fail Polar's prefix condition and Agent Lightning's stricter history-prefix condition.
Packing itself preserves the sampled response tokens; it changes how they are grouped for training. Without packing, each turn forms one training sequence. With packing, one sequence contains responses from multiple turns, allowing their gradients to be computed in a single forward/backward pass without repeatedly processing the shared history. Each packed sequence therefore contains more response tokens. Since TACO counts mini-batches in training sequences, keeping m=72 makes each packed optimizer update use more response tokens than an unpacked optimizer update. Reducing m to 24 aims to keep the number of response tokens per optimizer update as close as possible to that of unpacked training.
Our ablation changes only the mini-batch size, keeping the packing algorithm and tolerance unchanged. Optimizer updates per actor update increase from one to three, compared with 2.80 for unpacked training. Final AIME mean@8 at step 200 improves from 0.306 to 0.331, close to the unpacked result of 0.335, while retaining a 1.93× wall-clock advantage. The recovery without changing the packing rule suggests that fewer optimizer updates, each using more response tokens, are a major contributor to the original accuracy gap.
Response to Reviewer To1S (Part 1 / 4)
We sincerely thank Reviewer To1S for the thoughtful review. We hope the expanded related-work discussion, the new comparison, and the clarifications below have addressed your concerns. We would be grateful for any further feedback, and we respectfully ask whether you would consider re-evaluating the rating in light of our response.
W1/Q1:
Concurrent work and shared ground. Agent Lightning v1.0 (arXiv, Aug 2026) and Polar (arXiv, May 2026) are concurrent work with TACO. We have carefully studied both papers and their public implementations, and conducted comparative experiments. All three systems share the same capture layer: an LLM proxy records token IDs and logprobs at the protocol level, and no agent source change is required. The differences lie in three design dimensions.
1. Prefix merging. We implemented the prefix-merging methods of Agent Lightning and Polar in our system, and compared the three methods on the ReTool-style math agent. Agent Lightning requires the previous prompt and response to appear as an exactly identical token sequence at the start of the next prompt; any mismatch splits the trajectory into more samples. Polar compares server-side canonical prompt prefixes, locates the response boundary with EOS, and backfills the raw sampled tokens, so response-side re-tokenization does not break merging. TACO locates the replay boundary with a tail anchor and allows a bounded token-level edit distance τ; a merge beyond the budget is rejected and starts a new segment. On the same 244,830 attempted boundaries, merge success rates are 11.55% for Agent Lightning, 99.9992% for Polar, and 99.35% for TACO. Agent Lightning's strict token matching thus rarely merges drifted black-box traces, while TACO and Polar both recover the redundancy.
The difference between TACO and Polar is a design choice. Polar guarantees token faithfulness. TACO treats merging as a controlled approximation: τ is a tunable knob and an explicit upper bound on the approximation, and §4.3 quantifies its effect on training accuracy. Each turn's replay can introduce small token differences. The edit distance between the accumulated sequence, which keeps the raw sampled tokens, and the full replayed history therefore grows as more turns are merged. Polar's criterion compares canonical prompt prefixes and does not measure this deviation, so it keeps merging even when the deviation is large. TACO checks the full edit distance at every merge and starts a new segment once it exceeds τ, keeping the context deviation of every trained response within τ.