acceptodds
Under review as a conference paper at ICLR 2027

TACO: A Non-Intrusive RL Framework for Online, Black-Box Agent Services

Abstract

A growing number of deployed agents operate at the service level: they run online as black boxes, managing their own loops, tools, and model providers, with fixed configurations after deployment. Existing reinforcement learning (RL) frameworks, however, generally assume the agent loop can be relocated into the RL runtime, making it difficult to improve deployed agents without rebuilding them. We present TACO, a non-intrusive RL framework that trains such agents directly in their existing execution environments. TACO requires no changes to the agent source code; instead, before the service starts, it uses only a single configurable field the agent passes to the model unchanged (e.g., the model endpoint or API key) to carry a worker identifier and track rollout identity. To improve training efficiency, TACO packs rollout turns using bounded token-level edit distance to avoid repeated context computation, and overlaps rollout with training while draining admitted tasks before weight synchronization. We evaluate TACO on three diverse agents: a ReTool-style math agent, a TritonForge-style kernel agent, and ZeroClaw, a long-running Rust service. Integrating them with TACO requires only 122–334 lines of adapter code. On the multi-turn math agent, packing reduces processed tokens by 3.0×, and combining packing with asynchrony gives a 2.51× end-to-end speedup. TACO is now available on a commercial cloud AI platform as a selectable built-in training image for service-level agentic RL.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Agentic reinforcement learning, Large language model agents, Black-box agent services, Trajectory packing, Asynchronous training, Machine learning systems

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_To1SOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors introduce TACO, a blackbox RL framework to train agents in a non-intrusive way, without modifying the agent’s source code. With minimal wrapping code, TACO enables online training with an asynchronous RL framework of agents behind black-box services. The framework introduces concepts like bounded-token distance to train directly on full-agent traces rather than training on isolated parts of the trace. TACO is evaluated on a math agent, an agent that generates Triton kernels, and ZeroClaw, a Rust agent service. The authors report improved task-specific held-out performance in all three settings, with detailed efficiency comparisons conducted on the math agent.

Critical strengths
  • Training on production harnesses with no harness edits is a useful contribution. The paper is very timely as we move to having dedicated harnesses for each application and need to RL directly on production, which is valuable to minimise training-deployment gaps.

  • The experimental setup covers a large distribution of use-cases The three integrations featured in the paper provide evidence that the TACO framework works across different Agent service implementations of very different natures. TACO reports improvements on all three holdout evaluation settings.

Critical weaknesses
  • The TACO framework is not discussed relative to related work, with major omissions. In particular, Polar (May 2026, https://arxiv.org/abs/2605.24220) trains unchanged agent harnesses through a model-API proxy, captures token-level trajectories, supports asynchronous rollouts, and studies prefix-based reconstruction.
  • The efficiency–accuracy trade-off of approximate packing is not established. Packing changes both training sequences and number of optimizer steps. This implies the reported speedups compare different effective training setups. The mini-batch ablation recovers accuracy but does not isolate the effect of rewriting contexts. Comparing with matched optimisation budget would better analyse the cost of the approximation.
Questions for authors
  • What are TACO's framework contribution relative to related work like Polar (May 2026, https://arxiv.org/abs/2605.24220) and Agent Lightning v1.0 (August 2026; https://arxiv.org/abs/2608.17528)?
  • Why is approximate packing preferable to reconstructing the full text trace? Could you compare both with the original recorded calls, using matched training settings, to show how rewriting the history affects accuracy?
  • What causes the observed token-prefix mismatches: intended harness behavior, the text/token round trip, or avoidable serving artifacts? Could token continuity be preserved without changing the agent’s intended behavior, instead of relying on approximate packing? Is it possible to provide illustrative examples?
9 comments
AuthorsOpenReview ↗

Response to Reviewer To1S (Part 1 / 4)

We sincerely thank Reviewer To1S for the thoughtful review. We hope the expanded related-work discussion, the new comparison, and the clarifications below have addressed your concerns. We would be grateful for any further feedback, and we respectfully ask whether you would consider re-evaluating the rating in light of our response.

W1/Q1:

Concurrent work and shared ground. Agent Lightning v1.0 (arXiv, Aug 2026) and Polar (arXiv, May 2026) are concurrent work with TACO. We have carefully studied both papers and their public implementations, and conducted comparative experiments. All three systems share the same capture layer: an LLM proxy records token IDs and logprobs at the protocol level, and no agent source change is required. The differences lie in three design dimensions.

1. Prefix merging. We implemented the prefix-merging methods of Agent Lightning and Polar in our system, and compared the three methods on the ReTool-style math agent. Agent Lightning requires the previous prompt and response to appear as an exactly identical token sequence at the start of the next prompt; any mismatch splits the trajectory into more samples. Polar compares server-side canonical prompt prefixes, locates the response boundary with EOS, and backfills the raw sampled tokens, so response-side re-tokenization does not break merging. TACO locates the replay boundary with a tail anchor and allows a bounded token-level edit distance τ; a merge beyond the budget is rejected and starts a new segment. On the same 244,830 attempted boundaries, merge success rates are 11.55% for Agent Lightning, 99.9992% for Polar, and 99.35% for TACO. Agent Lightning's strict token matching thus rarely merges drifted black-box traces, while TACO and Polar both recover the redundancy.

The difference between TACO and Polar is a design choice. Polar guarantees token faithfulness. TACO treats merging as a controlled approximation: τ is a tunable knob and an explicit upper bound on the approximation, and §4.3 quantifies its effect on training accuracy. Each turn's replay can introduce small token differences. The edit distance between the accumulated sequence, which keeps the raw sampled tokens, and the full replayed history therefore grows as more turns are merged. Polar's criterion compares canonical prompt prefixes and does not measure this deviation, so it keeps merging even when the deviation is large. TACO checks the full edit distance at every merge and starts a new segment once it exceeds τ, keeping the context deviation of every trained response within τ.

AuthorsOpenReview ↗

Response to Reviewer To1S (Part 2 / 4)

Continuing from the previous part

2. Asynchronous execution. Polar fully decouples training and inference; a rollout may span multiple policy versions, and version consistency is left to the downstream RL engine. Agent Lightning pauses the gateway before an update and returns 429 to new requests, relying on agent-side client retries. A black-box agent without built-in retry logic will instead enter its error-handling path, and the resulting error messages may be folded into its context and pollute subsequent turns; an exhausted retry budget can also fail the task. TACO targets non-preemptible black-box tasks: it stops admission at a synchronization point and drains admitted tasks before switching weights; timed-out tasks are marked failed and their late requests are rejected. Every model call within a rollout is thus served by the same weight version. The agent never receives abnormal returns during normal execution, and no assumption is made about its retry logic. We also compare the three packing methods within the same asynchronous TACO pipeline, holding the model, data, GPU allocation, and mini-batch size (m=72) fixed. The result is as follows:

Packing methodEnd-to-end timeSpeedup over strict-prefix packing
TACO9.79 h2.05×
Polar9.52 h2.11×
Agent Lightning20.11 h1.00×

The comparison shows that the merging method, not the surrounding system, drives the efficiency gap: strict-prefix packing is about 2× slower because it leaves most drifted boundaries unmerged, while TACO and Polar are comparable (9.79 h vs. 9.52 h, within run-to-run variation; the primary TACO run is 8.72 h). TACO's slightly lower merge rate is the price of its stricter acceptance checks, which bound the context deviation as described above. TACO's contribution relative to Polar is thus not a higher merge rate or speed, but a bounded and quantified approximation, together with the rollout-level weight consistency described above and the deployment-regime difference discussed in point 3.

3. Black-box training design. Agent Lightning embeds the rollout ID in the proxy URL, injected by its Controller when it launches the agent process or Pod for each rollout. Polar uses the API key as the session ID, injected when it creates an isolated runtime per session. Both assume that the framework owns the agent's lifecycle and that the configuration can change per task. TACO targets a different deployment: a service that is already deployed, configured once, and long-running. Its configuration fields are fixed for the service's lifetime and cannot carry a task-varying rollout ID. TACO therefore carries only a stable worker ID in the fixed field and maintains the worker-to-rollout mapping in external Redis, updated before each task. In short, Agent Lightning and Polar black-box the agent's interaction loop, while TACO treats the deployment itself as a black box. This is what "online, black-box agent services" in our title means.

AuthorsOpenReview ↗

Response to Reviewer To1S (Part 3 / 4)

W2/Q2:

Our unpacked baseline trains directly on the original recorded LLM calls, keeping each prompt–response pair as a separate training sequence under its original context. We use the same model, prompt budget, and GPU allocation, and aim to keep the number of response tokens used for training comparable.

The evaluated traces contain both retokenization drift and small textual changes introduced by the chat template or tool-call parser, so our accuracy experiments cover both types of mismatch. Large-scale history rewriting requires different treatment. When summarization or deletion substantially changes the history, the shared context needed to reuse earlier computation may no longer exist. Forcing these turns into one sequence would replace the context actually seen by the later call with an outdated history. Such calls should remain separate training sequences. TACO enforces this through its anchor and edit-distance checks: a missing anchor or a difference exceeding the budget starts a new segment. Changes that destroy the shared prompt prefix also fail Polar's prefix condition and Agent Lightning's stricter history-prefix condition.

Packing itself preserves the sampled response tokens; it changes how they are grouped for training. Without packing, each turn forms one training sequence. With packing, one sequence contains responses from multiple turns, allowing their gradients to be computed in a single forward/backward pass without repeatedly processing the shared history. Each packed sequence therefore contains more response tokens. Since TACO counts mini-batches in training sequences, keeping m=72 makes each packed optimizer update use more response tokens than an unpacked optimizer update. Reducing m to 24 aims to keep the number of response tokens per optimizer update as close as possible to that of unpacked training.

Our ablation changes only the mini-batch size, keeping the packing algorithm and tolerance unchanged. Optimizer updates per actor update increase from one to three, compared with 2.80 for unpacked training. Final AIME mean@8 at step 200 improves from 0.306 to 0.331, close to the unpacked result of 0.335, while retaining a 1.93× wall-clock advantage. The recovery without changing the packing rule suggests that fewer optimizer updates, each using more response tokens, are a major contributor to the original accuracy gap.

Reviewer_YvPqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents TACO, a non-intrusive RL framework for training existing black-box agent services without modifying their source code. The system introduces three main techniques: (1) tracking rollout identities through a fixed configuration field and a worker-to-rollout mapping, (2) drift-tolerant trajectory packing that merges turns even when their histories have small token-level differences, and (3) asynchronous rollout and training with draining before weight synchronization.

Critical strengths

The problem of training existing agent services without modifying their code is practically relevant.

The system is implemented and evaluated on three different agents.

Critical weaknesses
  1. Limited technical novelty.

I find the technical contributions quite limited. Existing frameworks such as POLAR and Agent Lightning already support non-intrusive agent RL through API proxies, trajectory reconstruction, and asynchronous execution. The proposed worker-ID-based tracking and rollout draining seem like relatively straightforward engineering choices rather than substantial new contributions.

The drift-tolerant trajectory packing is perhaps the most distinctive component, but it is essentially an approximate version of existing trajectory packing techniques. Moreover, it changes the original rollout context, potentially introducing bias into RL training, which is not sufficiently studied.

  1. Weak experimental evaluation.

I don't find the experimental results particularly informative. Most of the efficiency gains come from reducing redundant computation through trajectory packing, while the experiments do not establish any meaningful advantage over existing frameworks.

The paper also lacks comparisons with POLAR or Agent Lightning. The experiments on the kernel agent and ZeroClaw mainly show that RL training can run, without providing much insight into the proposed system's advantages or limitations.

Questions for authors

What are the fundamental differences and advantages between TACO and existing frameworks such as POLAR and Agent Lightning?

5 comments
AuthorsOpenReview ↗

Response to Reviewer YvPq (Part 1 / 2)

We sincerely thank Reviewer YvPq for the careful reading and constructive comments. We appreciate the recognition that training existing agent services without code changes is practically relevant and that the system is implemented and evaluated on three different agents. We address the concerns below, and we respectfully ask whether you would consider re-evaluating the rating in light of our response.

W1/W2/Q1:

Concurrent work and shared ground. Agent Lightning v1.0 (arXiv, Aug 2026) and Polar (arXiv, May 2026) are concurrent work with TACO. We have carefully studied both papers and their public implementations, and conducted comparative experiments. All three systems share the same capture layer: an LLM proxy records token IDs and logprobs at the protocol level, and no agent source change is required. The differences lie in three design dimensions.

1. Prefix merging. We implemented the prefix-merging methods of Agent Lightning and Polar in our system, and compared the three methods on the ReTool-style math agent. Agent Lightning requires the previous prompt and response to appear as an exactly identical token sequence at the start of the next prompt; any mismatch splits the trajectory into more samples. Polar compares server-side canonical prompt prefixes, locates the response boundary with EOS, and backfills the raw sampled tokens, so response-side re-tokenization does not break merging. TACO locates the replay boundary with a tail anchor and allows a bounded token-level edit distance τ; a merge beyond the budget is rejected and starts a new segment. On the same 244,830 attempted boundaries, merge success rates are 11.55% for Agent Lightning, 99.9992% for Polar, and 99.35% for TACO. Agent Lightning's strict token matching thus rarely merges drifted black-box traces, while TACO and Polar both recover the redundancy.

The difference between TACO and Polar is a design choice. Polar guarantees token faithfulness. TACO treats merging as a controlled approximation: τ is a tunable knob and an explicit upper bound on the approximation, and §4.3 quantifies its effect on training accuracy. Each turn's replay can introduce small token differences. The edit distance between the accumulated sequence, which keeps the raw sampled tokens, and the full replayed history therefore grows as more turns are merged. Polar's criterion compares canonical prompt prefixes and does not measure this deviation, so it keeps merging even when the deviation is large. TACO checks the full edit distance at every merge and starts a new segment once it exceeds τ, keeping the context deviation of every trained response within τ.

AuthorsOpenReview ↗

Response to Reviewer YvPq (Part 2 / 2)

Continuing from the previous part

2. Asynchronous execution. Polar fully decouples training and inference; a rollout may span multiple policy versions, and version consistency is left to the downstream RL engine. Agent Lightning pauses the gateway before an update and returns 429 to new requests, relying on agent-side client retries. A black-box agent without built-in retry logic will instead enter its error-handling path, and the resulting error messages may be folded into its context and pollute subsequent turns; an exhausted retry budget can also fail the task. TACO targets non-preemptible black-box tasks: it stops admission at a synchronization point and drains admitted tasks before switching weights; timed-out tasks are marked failed and their late requests are rejected. Every model call within a rollout is thus served by the same weight version. The agent never receives abnormal returns during normal execution, and no assumption is made about its retry logic. We also compare the three packing methods within the same asynchronous TACO pipeline, holding the model, data, GPU allocation, and mini-batch size (m=72) fixed. The result is as follows:

Packing methodEnd-to-end timeSpeedup over strict-prefix packing
TACO9.79 h2.05×
Polar9.52 h2.11×
Agent Lightning20.11 h1.00×

The comparison shows that the merging method, not the surrounding system, drives the efficiency gap: strict-prefix packing is about 2× slower because it leaves most drifted boundaries unmerged, while TACO and Polar are comparable (9.79 h vs. 9.52 h, within run-to-run variation; the primary TACO run is 8.72 h). TACO's slightly lower merge rate is the price of its stricter acceptance checks, which bound the context deviation as described above. TACO's contribution relative to Polar is thus not a higher merge rate or speed, but a bounded and quantified approximation, together with the rollout-level weight consistency described above and the deployment-regime difference discussed in point 3.

3. Black-box training design. Agent Lightning embeds the rollout ID in the proxy URL, injected by its Controller when it launches the agent process or Pod for each rollout. Polar uses the API key as the session ID, injected when it creates an isolated runtime per session. Both assume that the framework owns the agent's lifecycle and that the configuration can change per task. TACO targets a different deployment: a service that is already deployed, configured once, and long-running. Its configuration fields are fixed for the service's lifetime and cannot carry a task-varying rollout ID. TACO therefore carries only a stable worker ID in the fixed field and maintains the worker-to-rollout mapping in external Redis, updated before each task. In short, Agent Lightning and Polar black-box the agent's interaction loop, while TACO treats the deployment itself as a black box. This is what "online, black-box agent services" in our title means.

We hope these clarifications and the added comparison address your concerns and welcome any further feedback.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer,

As the discussion period draws to a close, we hope our responses and the revised manuscript have addressed your questions about TACO. If any concerns remain, we would welcome the opportunity to clarify them. If our responses have resolved your concerns, we would be grateful if you would consider raising your ratings in light of the revisions and additional results. Thank you again for the time and care you have devoted to reviewing our paper. Your questions helped us explain our design choices more clearly and strengthen the evaluation.

Reviewer_yKZNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

TACO presents a non-intrusive RL framework for black-box agent services with fixed configurations and non-preemptible tasks. It combines persistent worker identifiers and a worker-to-rollout mapping for cross-turn request attribution, drift-tolerant trajectory packing to reduce repeated history computation, and asynchronous scheduling that drains admitted tasks before weight synchronization. The framework is integrated with a math agent, a kernel-generation agent, and a Rust service. The math experiments evaluate training efficiency and learning behavior; the other integrations demonstrate feasibility across runtimes and improvements in task rewards.

Critical strengths

S1. A practically useful approach to training existing agent services. The paper addresses a relevant deployment problem: training the model behind an agent while retaining its tools, dependencies, and execution loop. The worker-to-rollout mapping provides a concrete solution when service configuration cannot be changed between tasks. The three integrations, including a Rust service, support the practical value of this abstraction across runtimes and reduce the need to reimplement agents inside a training framework.

S2. A coherent design for black-box execution. Rollout identification, trajectory packing, and task-level synchronization address complementary difficulties arising from the same setting. In particular, draining admitted tasks before synchronizing weights allows complete rollouts to remain associated with a single generation policy version without requiring control over the agent's internal execution. This is a sensible systems design for services that are difficult to interrupt.

S3. Useful evidence about the interaction between packing and training efficiency. The math experiments show practical efficiency gains over unpacked or strict-prefix-matching baselines. The analysis also goes beyond compression: packing changes the actor workload and optimization granularity, which in turn affects the benefit of asynchronous execution. The mini-batch experiment and wall-clock learning curves provide useful empirical insight into these interactions.

Critical weaknesses

W1. Insufficient comparison with existing training-acceleration methods. The experiments establish gains over unpacked training and strict-prefix matching, but do not sufficiently show how competitive TACO's acceleration is relative to other methods for reducing redundant computation in multi-turn RL. AReaL-DTA [1] is a relevant example, yet the paper provides no controlled comparison. Without comparisons that account for workload, resource budget, and learning quality, the reported speedups are insufficient to establish TACO's relative efficiency.

[1] Jiarui Zhang et al. AReaL-DTA: Dynamic Tree Attention for Efficient Reinforcement Learning of Large Language Models. ICML 2026; arXiv:2602.00482.

W2. The empirical scope of drift-tolerant packing remains limited. The paper analyzes how nonzero tolerance changes conditioning contexts and reports KL and clipping diagnostics. These analyses do not establish how learning is affected when accepted edits change the content of earlier observations. The main math agent neither trims nor summarizes its history, so the results support tolerance to retokenization drift more directly than tolerance to substantive history edits. Since token edit distance does not distinguish these cases, the practical reliability of content-changing merges remains insufficiently tested.

W3. The deployment benefits are not yet adequately weighed against their costs. The persistent-worker identification scheme uses dedicated service instances, each with one active rollout, while synchronization drains admitted tasks subject to a timeout policy. Its practical value depends on service-replication costs and the task-duration distribution. The math experiments do quantify draining delays, but comparable deployment-cost evidence is missing for the other integrations. In particular, the kernel-generation and Rust-service experiments demonstrate feasibility and reward improvement without controlled training-speed comparisons. The efficiency evidence therefore remains concentrated on the math agent.

Questions for authors

Q1. Could the authors provide a controlled comparison with relevant training-acceleration methods, such as AReaL-DTA [1], using comparable workloads and resource budgets and reporting both training time and task performance? If such a comparison is not feasible, what evidence supports the relative acceleration benefit beyond the current unpacked and strict-prefix baselines?

[1] Jiarui Zhang et al. AReaL-DTA: Dynamic Tree Attention for Efficient Reinforcement Learning of Large Language Models. ICML 2026; arXiv:2602.00482.

Q2. Among accepted merges, how often do differences reflect retokenization versus changes to observation content? Are there cases with multiple anchor matches within the search window? If so, can selecting the rightmost match remove part of a new tool observation, and how is this possibility detected or prevented?

Q3. Which source of old-policy probabilities was used in each experiment? Beyond the reported sequence and optimizer-update counts, could the authors report the response-token counts and rollout coverage retained after filtering and truncation? Are repeated runs available for the packed configurations, particularly the smaller-mini-batch setting used to recover accuracy, and what evaluation decoding settings were used?

Q4. Please specify the kernel reward definition and evaluator settings, together with the task identifiers or selection procedure for the held-out kernel and Calc-X subsets. These details are needed to interpret the reported reward improvements.

7 comments
AuthorsOpenReview ↗

Response to Reviewer yKZN (Part 1 / 4)

We sincerely thank Reviewer yKZN for the detailed and constructive review. We are encouraged by the positive assessment of the practical value of training deployed agents in place (S1), the coherence of the design for black-box execution (S2), and the empirical insight into the interaction between packing and training efficiency (S3).

We believe our response has addressed most of your concerns and we respectfully ask if you're comfortable to re-consider the rating in light of our efforts and dedication. Below we respond to each question in detail. If there are any additional points we may have missed, we would be most grateful for further feedback.

W1/Q1:

We thank the reviewer for pointing out AReaL-DTA. TACO and AReaL-DTA are two different approaches to reducing redundant computation in multi-turn RL. AReaL-DTA requires strict token-level consistency: it organizes trajectories that share exact token prefixes into a tree and computes each shared prefix only once via tree attention. This requires deep modifications to the training engine, including tree-structured forward and backward attention kernels. TACO instead merges rollout turns at the data level into standard packed sequences, and it can merge turns whose histories differ within a bounded token-level edit distance. The training engine is left unchanged, so TACO is engine-agnostic.

We did not conduct a controlled comparison with AReal-DTA mainly because the experiments use different stacks. AReaL-DTA is evaluated on SGLang + Megatron, while TACO builds on vLLM + FSDP (via VeRL). Porting either method to the other's stack would change the engine or the attention kernels. These confounding variables would make it difficult to attribute any measured difference to the merging method itself. Instead, we implemented the trajectory-merging methods of two concurrent systems, Polar (https://arxiv.org/abs/2605.24220) and Agent Lightning v1.0 (https://arxiv.org/abs/2608.17528), within TACO. Agent Lightning requires the recorded prompt and response to match the beginning of the next prompt exactly at the token level; a mismatch starts a new segment. Polar compares server-side canonical prompt prefixes, locates the response boundary with EOS, and backfills the raw sampled tokens, allowing response-side retokenization. TACO locates the replay boundary with a tail anchor and accepts a merge only within the token edit-distance budget τ.

We compare merge success on the same 244,830 adjacent-call boundaries, and measure end-to-end time in separate runs of the same asynchronous TACO pipeline. The model, data, GPU allocation, and mini-batch size (m=72) are held fixed.

Packing methodMerge success rateEnd-to-end timeSpeedup over strict-prefix packing
TACO99.35%9.79 h2.05×
Polar99.9992%9.52 h2.11×
Agent Lightning11.55%20.11 h1.00×

This comparison isolates the effect of the merging method within a common training and inference stack. Agent Lightning's strict-prefix rule leaves most drifted boundaries unmerged and is about 2× slower end to end. TACO and Polar both recover this redundancy and perform comparably: the gap in merge rate (99.35% vs. 99.9992%) and wall-clock time (9.79 h vs. 9.52 h) is small relative to run-to-run variation — the primary TACO run takes 8.72 h, and the repeated run used in this comparison takes 9.79 h. We do not claim a speed advantage over Polar. TACO's slightly lower merge rate reflects its stricter acceptance checks (anchor window and edit-distance budget), which are what bound the context deviation of every trained response; Polar's canonical-prefix criterion does not measure this deviation. The differences between TACO and Polar therefore lie in the guarantees they provide — a bounded, quantified approximation (§4.3), rollout-level weight consistency (§3.3), and support for services whose configuration is fixed after deployment (§3.1) — rather than in merging throughput.

AuthorsOpenReview ↗

Response to Reviewer yKZN (Part 2 / 4)

W2/Q2:

We thank the reviewer for this careful question. We address the merge statistics and the anchor-matching concern below.

(1) Retokenization vs. content changes. Across all 244,830 adjacent-call boundaries in the additional math-agent diagnostic run, 1,683 (0.687%) show pure retokenization drift: the decoded history remains a text prefix of the next prompt, but the token IDs differ. Another 214,872 (87.764%) show decoded-text changes, classified as content or template changes, while 28,275 (11.549%) match exactly at the token level. Most exact-match failures originate from the chat template: when the agent constructs the next prompt, it re-renders earlier responses using the chat template, which may change the text.

For instance, we find a rollout where the response from the previous round ended with:

token_ids = [18481, 334, 17, 488, 19, 18481, 488, 23, 16, 1699, 82, 20201, 284, 11625, 2026, 446, 367, 11, 856, 10699, 77, 1350, 1141, 20201, 12954, 220, 456, 532, 151658, 151645]

text: 'x**2 +4x +81\nsolutions = solve(equation, x)\nprint(solutions)"\n }\n}\n</tool_call><|im_end|>'

The content corresponding to the prompt of the next round is as follows:

token_ids = [77, 1350, 1141, 20201, 8, 95642, 151658, 151645, 198, 151644, 872, 198, 27, 14172, 9655, 397, 13608, 2829, 1210, 364, 7188, 516, 364, 1994, 1210, 8981, 364, 20433, 5287, 1210]

text: 'nprint(solutions)"}}\n</tool_call><|im_end|>\n<|im_start|>user\n<tool_response>\n{'status': 'Success', 'message': '', 'compile_result':'

It can be observed that print(solutions)"\n } has become print(solutions)"}, which indicates that the tool call parser or the chat template has applied some formatting changes.

These results cover both retokenization drift and small textual rewrites introduced by the chat template or tool-call parser, as illustrated above. Our existing accuracy ablation therefore evaluates packing on traces with both kinds of difference. Large-scale history rewriting is a different case: when trimming or summarization removes the shared history, the turns should remain separate training sequences. TACO rejects merges that fail the anchor or edit-distance checks; a rewrite that destroys the canonical prompt prefix also fails Polar's prefix condition, and Agent Lightning requires an even stricter prefix match. The purpose of drift tolerance is to recover redundant computation across largely preserved histories, rather than to force unrelated contexts into one sequence.

(2) Multiple anchor matches and the rightmost rule. We did not observe any case with multiple anchor matches within the search window in our math-agent measurements. The anchor consists of the last a=10 tokens of the accumulated sequence, which ends with the EOS token of the previous response. In the next prompt's token sequence, we search around the position given by the accumulated sequence length. This is the expected end of the agent-replayed history, followed by the new tool output. EOS almost never appears at the beginning of tool output, and the search window is small (h=32 tokens around the expected position). Together, these make the anchor match unique in practice, and the "rightmost match" rule is stated for formal completeness rather than to resolve frequent ambiguity.

AuthorsOpenReview ↗

Response to Reviewer yKZN (Part 3 / 4)

Q3:

Old-policy probabilities. Asynchronous runs use recorded rollout log probabilities, whereas synchronous runs recompute old-policy log probabilities on the trainer before optimization. Appendix C explains the available denominator choices and their implications for the conditioning context and training–inference consistency.

Response tokens and rollout coverage. In the 200-step diagnostic run using TACO packing, merging preserves all 222,403,174 sampled response tokens, while reducing 316,875 per-call sequences to 129,922 packed sequences. Prompt-length filtering removes no sequences; 189 sequences undergo response-length truncation, which shortens them without reducing their count. PPO mini-batch alignment then removes 14,722 sequences, leaving 115,200 training sequences. These cover at least 113,278 of the 128,000 valid rollouts (88.50%): packing produces only 1,922 additional segments beyond one sequence per valid rollout, giving this conservative coverage bound.

Repeated runs and evaluation. We have added a repeat of asynchronous TACO packing at m=72 and m=24, using the same model, data, optimizer, batch sizes, and GPU allocation as the primary run. The repetition study in the appendix also reports three asynchronous unpacked runs. Math evaluation uses eight rollouts per problem and reports AIME mean@8, with temperature 1.0, top-p 1.0, no top-k restriction, at most 4,096 generated tokens per model call.

The tables below follow the appendix's repetition summary. Tokens/step and response tokens/turn are averaged over steps 1–200; accuracy is AIME mean@8 at step 200, consistent with the paper.

Asynchronous TACO packing, m=72.

RunStepsWall-clockTokens/stepResponse/turnAIME mean@8
Primary2008.72 h1.41M5470.306
Repeat 12009.79 h1.39M6010.319

Asynchronous TACO packing, m=24.

RunStepsWall-clockTokens/stepResponse/turnAIME mean@8
Primary2009.94 h1.56M5300.331
Repeat 120010.57 h1.68M5950.344

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank Reviewers yKZN, YvPq, and To1S for their careful reading and constructive feedback. We have revised the manuscript and addressed the comments in our individual responses.


Core contributions of TACO

  1. Training deployed black-box services. TACO trains agents in their existing environments without source changes. A stable worker identifier supports rollout tracking without per-task service reconfiguration.
  2. Drift-tolerant trajectory packing. TACO merges histories within a token edit-distance budget into ordinary training sequences. Our mini-batch ablation shows why optimizer update counts matter for packing accuracy.
  3. Asynchrony for non-preemptible agent tasks. TACO drains admitted tasks before weight synchronization, keeping each successful rollout on one parameter version without requiring retry support inside the agent.

Together, packing and asynchrony deliver a 2.51× end-to-end speedup on the math agent.


Main updates in the revised manuscript

  1. Comparisons with concurrent work (§5). We expanded the discussion of Polar, Agent Lightning v1.0, and AReaL-DTA. We compared the packing rules of TACO, Polar, and Agent Lightning within the same TACO pipeline, measuring merge success rates and end-to-end training time (Reviewers yKZN, YvPq, and To1S).
  2. Packing scope and diagnostics (§§1 and 3.2; Appendix B). We report measured frequencies of retokenization drift and textual changes, clarify anchor matching, and explain why large history changes should remain unmerged (Reviewers yKZN and To1S).
  3. Packing and optimization (§§2 and 4.3; Appendices E.1 and F). We clarify how packing changes training-sequence counts and optimizer updates, and add retention diagnostics and repeated packed-training runs (Reviewers yKZN and To1S).
  4. Agent experiments (§§4.1–4.2; Appendix A). We clarify rewards, held-out task selection, and evaluation settings, and include the kernel-agent and ZeroClaw validation curves in the main text (Reviewer yKZN).
  5. Preserving rollout behavior (§§3.2 and 5, Appendix H). We explain why TACO packs recorded calls after generation, compare exact and tolerant packing, and clarify how accepting a small intentional edit can change the context used to train a later turn (Reviewer To1S).

Our manuscript was updated on November 18, 2026 (AoE). Major additions and revisions are highlighted in blue-colored text for ease of reference. Thank you again for your feedback; we welcome further questions and discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.