- Rating
- 2
- Confidence
- 1
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether reduced successor workload after an agent handoff reflects useful progress, correct task completion, or unfinished work hidden by incorrect closures. The authors propose a paired accounting method that runs successor agents from the original state and intermediate checkpoints. This separates the different sources of apparent workload savings.
The authors evaluate the method on τ-bench and AppWorld, with 1,062 base runs and 9,398 takeovers. They find that successor workload generally decreases, but total system actions can still increase. Paired comparisons also reveal biases in conventional workload estimates. Preserving interpreter state reduces successor work by 3.44 actions, although its effect on task quality remains uncertain. On 126 held-out AppWorld task variants, the authors find that proportional allocation often underestimates work hidden by incorrect closures. Their correction improves accuracy in two secondary settings, but the primary Gemma test remains inconclusive.
Critical strengths
The paper provides a clear way to distinguish work saved by correct closures from work hidden by incorrect ones. By considering all incoming requests, the method offers insights that conventional handoff evaluations may miss. The AppWorld results illustrate why reduced successor workload does not necessarily mean better overall performance.
The authors conduct extensive experiments on τ-bench and AppWorld, with 1,062 base runs and 9,398 takeovers. The interleaved interface experiment is thoughtfully designed, and the authors are transparent about negative and inconclusive results. This adds credibility to their findings while acknowledging the limitations of the study.
Critical weaknesses
The paired accounting method helps explain where successor workload savings come from, but it is less clear how these additional measurements can be used in practice. Although the method identifies biases in conventional workload estimates, the proposed correction only reduces allocation error by 0.04 actions per request in the primary experiment, which remains inconclusive. The DOVE controller also does not reliably outperform fixed-depth baselines.
I had some difficulty following parts of the paper. I think some important concepts should be define in main text. Also the plot looks like generated by Ai. The terminology also differs between some figures and equations, which makes it harder to connect the methods with the reported results. I think there are a lot of results do not have clear interpretation. I feel the empirical diagnosis is less innovative than other papers that propose new framework.
I am also not familiar with this field.
Questions for authors
Base logs alone do not identify the three terms. Table 1 separates complete policy evaluation from recovery attribution. The closed-work identity is a population relation; independent cohorts need not balance exactly in finite samples. If original-state runs already exist on the same closed requests and retain request IDs, level 3 requires only a join, with no new model calls. Ordinary singlepath deployment does not jointly observe these branches
these sentences are hard to follow. what is recovery attribution? what is closed-work identity? what does this means: Identification implies no predictive or deployment advantage?
I feel i may not fully understand
4 comments
(2/2): terminology, what each result means, practical use, and figures
One name per term. You are right that Eq. 2 and Figure 1 use different names for the same quantities. We will use the Figure 1 names everywhere:
| Eq. 2 (current) | Name we will use | Plain meaning |
|---|---|---|
| inflation | inherited-state change | how starting from the base agent's state changes the successor's work (request A) |
| skim | correct-closure savings | work the successor no longer does because the base agent finished correctly (B) |
| false savings | wrong-closure apparent savings | work that disappears only because the base agent closed the request wrongly; a failure, not a saving (C) |
What each main result means, in one sentence.
- Total work rises in all 4 primary τ-bench pairings although the successor's work falls (Table 3): a cheaper job for the successor does not mean a cheaper system.
- With a weak base agent, successor work falls from 14.3 to 1.0 actions while failure rises from 11% to 71% (§5.2): most of that "saving" is wrong closures.
- Keeping the base agent's Python variables saves the successor 3.44 actions (Table 27): how the handoff is done is a cost you can fix.
- Comparing handed-off requests against all requests understates the effect (§5.3): you must compare the same requests.
- A ratio learned on old tasks improves predictions on new tasks only when Eq. 5 holds (§5.4): the paper gives the condition for when this works.
We will add each sentence after the corresponding result.
How the measurements are used in practice. A concrete case from the paper (Table 29b). With the gpt-oss base agent at cap 24, the log says autonomy is 1.48 actions cheaper per request than handing off immediately, and failure is 50 points higher. A team that cares mostly about cost could accept this. Our accounting shows that the whole saving comes from wrong closures. If those requests are reopened and redone, the policy is 9.98 actions more expensive. We then ran that repair: it measured +10.6 [+8.9, +12.4]. So the accounting changes the decision from "accept a cheaper policy" to "reject it", without needing any new model calls.
The 0.04 result. That test asks a harder, separate question: can a ratio learned on one set of tasks predict the split on new tasks without running them? Eq. 5 says when this works. Checking all six settings after the fact, Eq. 5 gives the right sign in all six; the primary setting sits almost exactly on its boundary (0.02 away), so a near-zero gain is what Eq. 5 predicts. Details are in our response to Reviewer PjAw (Q2).
DOVE. We agree, and we report it as a negative result. It is not the contribution of the paper. We will move the controller to the appendix so the main text focuses on the accounting.
Novelty. The new framework is the all-request paired protocol. Earlier handoff studies (the Handoff Tax and Handoff Debt, §2) measure cost at the moment of handoff or interruption. Neither runs the successor from the original state on the requests the base agent closes, and neither links that work to whether each closure was correct. That is where wrong-closure work hides, and Appendix A shows that ordinary logs cannot recover it.
Figures. Every data figure is drawn by a plotting script in the released code directly from the analysis outputs, and every plotted value also appears in a table (for example, Figure 2 ↔ Tables 3–4, Figure 4 ↔ Table 28). Figure 1 is a schematic; we will simplify it to the three-request example above.
We hope this makes the paper easy to follow. If these explanations resolve your concerns, we would be grateful if you would consider updating your score, and we are happy to explain any part further.
thanks for updating but i will still keep 2 as some parts are hard to follow and I am not fully understand
Thank you for reading our responses and for the time you have given the paper. If you have any further concrete questions, we would be glad to discuss them :)
(1/2): the idea in one example, and your three questions
Thank you for the review. We are glad you found that the paper clearly separates work saved by correct closures from work hidden by incorrect ones, that looking at all incoming requests reveals what usual handoff evaluations miss, that the interleaved interface experiment is well designed, and that the negative results are reported openly. Thank you also for quoting the exact sentences you found hard to follow. That paragraph was written too densely, and the explanation below will replace it in §3.
The whole idea in one example. A base agent works first; a stronger successor takes over if needed. Consider three requests:
The log shows the successor's work falling from 19 to 4, so it looks like 15 actions were saved. In fact 10 of the 15 come from request C, a failure. To know the "from scratch" column for B and C, we must run the successor from the original state even on requests the base agent closed. Normal deployment never does this. That is the whole method (Eq. 2 is this table written as an average).
Your three questions.
What is recovery attribution? It means splitting the change in the successor's work into its three sources: the state left by the base agent (A), correct closures (B) and wrong closures (C).
What is the closed-work identity? For requests the base agent closes, the successor work that disappears from the log equals the part on correct closures plus the part on wrong closures (3 + 10 = 13 above). This holds on average over the population. Two separately sampled groups of requests will not match it exactly, which is why we measure both parts on the same requests.
What does "Identification implies no predictive or deployment advantage" mean? Being able to measure the three parts after the fact does not, by itself, let you predict them for a new agent or pick a better policy. We test those two questions separately (§5.4 and Appendices C–D). The sentence limits our claim; it is not a result.
The quoted paragraph (lines 156–161):