acceptodds
Under review as a conference paper at ICLR 2027

WHAT HANDOFF LOGS HIDE: PAIRED ACCOUNTING FOR LLM-AGENT WORK

Abstract

Higher success rates. Fewer handoffs. More work left behind. Task success is the dominant metric for tool-using language agents, yet it can misstate the value of autonomy in systems where failed or incomplete work is handed to a stronger agent or human. An additional action can improve the acting agent’s chance of completion while simultaneously making an eventual handoff harder. We call this quantity downstream work. We introduce a paired replay protocol that forks the same executable state into immediate- handoff and continue-then-handoff branches, separating action-induced work from selection of intrinsically difficult tasks. This protocol supports Dove, a joint distributional model of task outcome and successor work, combined with a resource-aware stopping rule. As a preliminary measurement, we audit 26 public τ -bench result sets comprising 228 tasks, four agent models, and 110,068 trajectory prefixes. Under an evaluator-derived residual-action proxy, 37.3% of trajectories contain at least one work-amplifying action; the task-clustered rate is 29.2% (95% bootstrap CI: 25.8–32.5%). This proxy result motivates, but does not replace, paired successor execution. Our central proposal is that agent evaluation and delegation should optimize correct completed work per system budget rather than autonomous completion alone

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Agent Evaluation, Tool-Using Agents, Agent Handoffs Benchmarking and Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8x2vOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether reduced successor workload after an agent handoff reflects useful progress, correct task completion, or unfinished work hidden by incorrect closures. The authors propose a paired accounting method that runs successor agents from the original state and intermediate checkpoints. This separates the different sources of apparent workload savings.

The authors evaluate the method on τ-bench and AppWorld, with 1,062 base runs and 9,398 takeovers. They find that successor workload generally decreases, but total system actions can still increase. Paired comparisons also reveal biases in conventional workload estimates. Preserving interpreter state reduces successor work by 3.44 actions, although its effect on task quality remains uncertain. On 126 held-out AppWorld task variants, the authors find that proportional allocation often underestimates work hidden by incorrect closures. Their correction improves accuracy in two secondary settings, but the primary Gemma test remains inconclusive.

Critical strengths

The paper provides a clear way to distinguish work saved by correct closures from work hidden by incorrect ones. By considering all incoming requests, the method offers insights that conventional handoff evaluations may miss. The AppWorld results illustrate why reduced successor workload does not necessarily mean better overall performance.

The authors conduct extensive experiments on τ-bench and AppWorld, with 1,062 base runs and 9,398 takeovers. The interleaved interface experiment is thoughtfully designed, and the authors are transparent about negative and inconclusive results. This adds credibility to their findings while acknowledging the limitations of the study.

Critical weaknesses

The paired accounting method helps explain where successor workload savings come from, but it is less clear how these additional measurements can be used in practice. Although the method identifies biases in conventional workload estimates, the proposed correction only reduces allocation error by 0.04 actions per request in the primary experiment, which remains inconclusive. The DOVE controller also does not reliably outperform fixed-depth baselines.

I had some difficulty following parts of the paper. I think some important concepts should be define in main text. Also the plot looks like generated by Ai. The terminology also differs between some figures and equations, which makes it harder to connect the methods with the reported results. I think there are a lot of results do not have clear interpretation. I feel the empirical diagnosis is less innovative than other papers that propose new framework.

I am also not familiar with this field.

Questions for authors

Base logs alone do not identify the three terms. Table 1 separates complete policy evaluation from recovery attribution. The closed-work identity is a population relation; independent cohorts need not balance exactly in finite samples. If original-state runs already exist on the same closed requests and retain request IDs, level 3 requires only a join, with no new model calls. Ordinary singlepath deployment does not jointly observe these branches

these sentences are hard to follow. what is recovery attribution? what is closed-work identity? what does this means: Identification implies no predictive or deployment advantage?

I feel i may not fully understand

4 comments
AuthorsOpenReview ↗

(1/2): the idea in one example, and your three questions

Thank you for the review. We are glad you found that the paper clearly separates work saved by correct closures from work hidden by incorrect ones, that looking at all incoming requests reveals what usual handoff evaluations miss, that the interleaved interface experiment is well designed, and that the negative results are reported openly. Thank you also for quoting the exact sentences you found hard to follow. That paragraph was written too densely, and the explanation below will replace it in §3.

The whole idea in one example. A base agent works first; a stronger successor takes over if needed. Consider three requests:

RequestWhat happenedSuccessor actionsSuccessor actions if started from scratchMeaning
Abase did part of the work, then handed off462 saved by the inherited state (real saving)
Bbase closed it correctly033 saved by correct closure (real saving)
Cbase closed it wrongly01010 "saved" by a wrong closure (actually a failure)

The log shows the successor's work falling from 19 to 4, so it looks like 15 actions were saved. In fact 10 of the 15 come from request C, a failure. To know the "from scratch" column for B and C, we must run the successor from the original state even on requests the base agent closed. Normal deployment never does this. That is the whole method (Eq. 2 is this table written as an average).

Your three questions.

What is recovery attribution? It means splitting the change in the successor's work into its three sources: the state left by the base agent (A), correct closures (B) and wrong closures (C).

What is the closed-work identity? For requests the base agent closes, the successor work that disappears from the log equals the part on correct closures plus the part on wrong closures (3 + 10 = 13 above). This holds on average over the population. Two separately sampled groups of requests will not match it exactly, which is why we measure both parts on the same requests.

What does "Identification implies no predictive or deployment advantage" mean? Being able to measure the three parts after the fact does not, by itself, let you predict them for a new agent or pick a better policy. We test those two questions separately (§5.4 and Appendices C–D). The sentence limits our claim; it is not a result.

The quoted paragraph (lines 156–161):

  1. "Base logs alone do not identify the three terms" → the usual log (base agent only) cannot tell A, B and C apart.
  2. "Table 1 separates complete policy evaluation from recovery attribution" → Table 1 lists what each amount of data lets you measure. Running each policy on its own requests gives total cost and failure. Adding from-scratch runs on handed-off requests gives A. Adding from-scratch runs on closed requests, plus whether each closure was correct, separates B from C.
  3. "Independent cohorts need not balance exactly in finite samples" → two different random samples will not give exactly equal totals, so we pair on the same requests.
  4. "Level 3 requires only a join, with no new model calls" → if from-scratch runs already exist, you only match records by request ID.
  5. "Ordinary single-path deployment does not jointly observe these branches" → in normal use each request is run only one way, so you never see both versions of the same request; we run both on purpose.
AuthorsOpenReview ↗

(2/2): terminology, what each result means, practical use, and figures

One name per term. You are right that Eq. 2 and Figure 1 use different names for the same quantities. We will use the Figure 1 names everywhere:

Eq. 2 (current)Name we will usePlain meaning
inflationinherited-state changehow starting from the base agent's state changes the successor's work (request A)
skimcorrect-closure savingswork the successor no longer does because the base agent finished correctly (B)
false savingswrong-closure apparent savingswork that disappears only because the base agent closed the request wrongly; a failure, not a saving (C)

What each main result means, in one sentence.

  1. Total work rises in all 4 primary τ-bench pairings although the successor's work falls (Table 3): a cheaper job for the successor does not mean a cheaper system.
  2. With a weak base agent, successor work falls from 14.3 to 1.0 actions while failure rises from 11% to 71% (§5.2): most of that "saving" is wrong closures.
  3. Keeping the base agent's Python variables saves the successor 3.44 actions (Table 27): how the handoff is done is a cost you can fix.
  4. Comparing handed-off requests against all requests understates the effect (§5.3): you must compare the same requests.
  5. A ratio learned on old tasks improves predictions on new tasks only when Eq. 5 holds (§5.4): the paper gives the condition for when this works.

We will add each sentence after the corresponding result.

How the measurements are used in practice. A concrete case from the paper (Table 29b). With the gpt-oss base agent at cap 24, the log says autonomy is 1.48 actions cheaper per request than handing off immediately, and failure is 50 points higher. A team that cares mostly about cost could accept this. Our accounting shows that the whole saving comes from wrong closures. If those requests are reopened and redone, the policy is 9.98 actions more expensive. We then ran that repair: it measured +10.6 [+8.9, +12.4]. So the accounting changes the decision from "accept a cheaper policy" to "reject it", without needing any new model calls.

The 0.04 result. That test asks a harder, separate question: can a ratio learned on one set of tasks predict the split on new tasks without running them? Eq. 5 says when this works. Checking all six settings after the fact, Eq. 5 gives the right sign in all six; the primary setting sits almost exactly on its boundary (0.02 away), so a near-zero gain is what Eq. 5 predicts. Details are in our response to Reviewer PjAw (Q2).

DOVE. We agree, and we report it as a negative result. It is not the contribution of the paper. We will move the controller to the appendix so the main text focuses on the accounting.

Novelty. The new framework is the all-request paired protocol. Earlier handoff studies (the Handoff Tax and Handoff Debt, §2) measure cost at the moment of handoff or interruption. Neither runs the successor from the original state on the requests the base agent closes, and neither links that work to whether each closure was correct. That is where wrong-closure work hides, and Appendix A shows that ordinary logs cannot recover it.

Figures. Every data figure is drawn by a plotting script in the released code directly from the analysis outputs, and every plotted value also appears in a table (for example, Figure 2 ↔ Tables 3–4, Figure 4 ↔ Table 28). Figure 1 is a schematic; we will simplify it to the three-request example above.

We hope this makes the paper easy to follow. If these explanations resolve your concerns, we would be grateful if you would consider updating your score, and we are happy to explain any part further.

thanks for updating but i will still keep 2 as some parts are hard to follow and I am not fully understand

Thank you for reading our responses and for the time you have given the paper. If you have any further concrete questions, we would be glad to discuss them :)

Reviewer_fRTyOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary
  1. This paper studies why “less work for the successor” during agent handoffs does not necessarily mean that the entire system is more efficient. The authors execute the same request separately from the original state and intermediate checkpoints, breaking down changes in work into the effects of inherited state, savings from correct closures, and work hidden by incorrect closures.
  2. Experiments find that although successor work usually decreases, the total number of system actions instead increases in the four primary τ-bench pairings.
  3. The paper’s main value is improving measurement and attribution; cross-task transfer, actual error-repair costs, and whether deployment policies can be improved based on this still need to be verified.
Critical strengths
  1. Identifies a blind spot: Avoids mistakenly counting fewer calls caused by incorrect closures as an improvement in efficiency.
  2. Rigorous comparison design: Pairing the same requests makes it possible to distinguish state changes from task-selection bias.
Critical weaknesses
  1. Incomplete cost measurement: Mainly counts model actions subject to a cap, without measuring actual repair costs and human effort.
  2. Limited evidence of generalization: The primary transfer test remains inconclusive, and known task families and gaps in model-version records limit extrapolation.
Questions for authors

None

2 comments
AuthorsOpenReview ↗

(1/2): actual repair cost, and what is new

Thank you for the review. We are glad you found that the paper exposes a blind spot, where fewer calls caused by wrong closures are mistaken for efficiency, and that the paired design separates state effects from task-selection bias. Thank you also for the clear summary. Its third point names three things that still needed verification: actual error-repair cost, cross-task transfer, and whether deployment decisions can be improved. We address each below with new runs and with evidence already in the paper.

W1. Incomplete cost measurement: actual repair cost.

We have now measured actual repair cost. It is higher than the hidden-work estimate, so the paper's "false savings" are a lower bound on what wrong closures cost.

  • What and where: the 126-task held-out all-request study (Appendix I), at cap 24. An independent check reopens every wrong closure (82 for gpt-oss, 35 for Gemma), and Luna repairs it from the state the base agent left, with the same 40-action cap and interleaved logging.
  • Result: repair takes 18.6 (gpt-oss) and 21.0 (Gemma) actions per wrong closure, vs. 17.6 and 19.5 to solve the same request from its original state; repair success is 72% and 77%.
  • Conclusion: including repair, gpt-oss/24 costs +10.6 [+8.9, +12.4] actions per request more than immediate handoff, although its log shows it 1.48 cheaper. Gemma/24 costs +6.1 [+4.6, +7.6] instead of the logged +0.23. Repairing a wrong state costs more than restarting, so the logged saving is not merely smaller than it looks; it reverses.

Is the cap truncating the measurement? The quantity that carries the hidden-work estimate, original-state work, hit the 40-action cap in 1 of 126 tasks (Table 31), so the cap does not truncate it.

Beyond actions. The paper also reports tokens and dollars (Table 5c, Appendix E): under the status quo, tokens rise by 26% to 115% in the four primary Luna pairings, the same direction as actions.

Human effort. We do not measure it and do not claim to. What the accounting provides is where human effort lands: at gpt-oss/24, 82 of 126 requests (65%) are recorded in the log as completed but are wrong. A log-only analysis counts them as automation. We will state this scope explicitly in §3 and §6.

What is new relative to existing handoff studies.

The closest work, the Handoff Tax and Handoff Debt (§2), measures handoff cost on requests that are handed off or interrupted. Neither runs a successor from the original state on the requests the first agent closes, and that is exactly where wrong-closure work hides. Appendix A shows that base-only logs cannot identify this term. That design difference produces three findings those studies could not see:

  1. The successor's job gets cheaper while the system gets more expensive. Successor work falls in 165 of 178 contrasts, yet total work rises in all 4 primary τ-bench pairings (+0.52 to +4.02 actions per request, Table 3).
  2. A 93% cut in successor work can be a failure. With a weak base, successor work falls from 14.3 to 1.0 actions while failure rises from 11% to 71% (§5.2).
  3. The standard survivor comparison hides real effects. At AppWorld gpt-oss/16, the paired inherited-state effect is −6.45 [−11.01, −1.89] actions, while the survivor comparison is inconclusive, −2.25 [−5.72, +1.22] (Table 20).
AuthorsOpenReview ↗

(2/2): cross-task transfer, model versions, and deployment decisions

W2. Limited generalization.

Why the primary transfer test was inconclusive. The paper's own transfer condition (Eq. 5) explains it. Plugging each cell's observed values into Eq. 5 gives the largest source ratio κ for which the correction helps:

Cellκ limitfrozen κEq. 5 saysobserved G
Gemma/8, /161.26, 1.541.58hurts−0.07, −0.05
Gemma/24 (primary)1.601.58helps (barely)+0.04
gpt-oss/8, /16, /243.01, 2.39, 2.111.24helps+0.16, +0.31, +0.41

All 6 signs agree. The primary cell sits 0.02 below its limit, so a near-zero result is what the condition predicts. This is a post hoc check using the target's own values, not a forecast. The test also had 80% power only if at least 92% of the source correction transferred; 51% did.

Known task families. Relative to the cohorts that produced the frozen coefficients, most target families are new. The released config/ files show zero task overlap, and 86/126 target families (Gemma) and 70/126 (gpt-oss) never appear in the source cohort. On these unseen families, the gpt-oss gains hold: M = +0.37 [+0.03, +0.71] at depth 16 and +0.44 [+0.04, +0.84] at depth 24.

Model-version gaps. The fully logged, interleaved study (Appendix I) re-executes the AppWorld endpoints with served identifiers, UTC times and raw responses, and reproduces them. Successor work falls by 15.48 / 14.55 actions per request (sequential collection: 14.38 / 13.32), and gpt-oss failure rises by 50 [35, 65] pp (sequential: 60 [54, 67]). Removing the unidentified Llama endpoint leaves every conclusion unchanged. We also re-ran the three identified τ-bench primary pairings interleaved with full logging: +0.61, +2.29 and +3.88 actions per request, the same signs as Table 3.

Can deployment decisions be improved? Two decisions guided by the accounting, both checked by execution.

  1. Reject an apparent saving. A cost-focused team reading the log would adopt gpt-oss autonomy at cap 24 (1.48 actions cheaper). The accounting predicts +9.98 once wrong closures are redone, with no new model calls; executing the repair gives +10.6 (W1).
  2. Fix the handoff interface. The inherited-state term located a 6.11-action excess at the handoff. Preserving the interpreter cut successor work by 3.44 [2.80, 4.08] actions and the excess to 2.67 (Table 27). A new 2×2 ablation shows the saving comes from the preserved state (−3.1 [−3.8, −2.4]), not the instruction (−0.3 [−0.8, +0.2]). Failure is non-inferior at a 10 pp margin fixed before collection (+1.1 pp; one-sided 95% upper bound +6.4).

We do not claim a better general controller. We claim that the accounting tells a practitioner which of these decisions to make, and that both decisions held when executed. Details are in our response to Reviewer PjAw.

To summarize the three open points in your review: actual repair cost is now measured (it exceeds our estimate); the transfer result is explained in all six cells and holds on unseen families; and two deployment decisions guided by the accounting were verified by execution. If this addresses your concerns, we would be grateful if you would consider raising your score, and we would welcome any further questions.

Reviewer_PjAwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the issue of cost attribution during multi-agent handoffs. A reduction in the actions performed by a successor agent could stem from the predecessor agent having made progress or correctly completing a task—or, conversely, from the predecessor incorrectly closing a task, causing unfinished work to vanish from the logs.

The authors propose a "paired execution" protocol that covers all requests. For any given request, they execute the successor agent starting from the original state, while also initiating "takeover branches" from multiple checkpoints along the predecessor’s trajectory. By appending independently evaluated "closure labels," they decompose changes in workload into three components: the impact of the inherited state, savings from correct task closure, and "apparent" savings resulting from incorrect task closure.

The study reports on 1,062 baseline runs and 9,398 takeover attempts. Key findings indicate that a reduction in a successor’s actions does not necessarily equate to a decrease in the total number of actions performed by the system. Furthermore, preserving the interpreter state—along with its associated context—can help reduce the number of actions required by the successor. The analysis reveals a bias in workload attribution associated with incorrect task closures; however, the primary migration test—which employs a "frozen source domain" correction—remains inconclusive. The paper’s primary contribution lies in its methodology for measurement and diagnosis, rather than in demonstrating a superior deployment strategy.

Critical strengths
  • Clear distinction between easily conflated sources of cost. The breakdown on pages 3–4 separates state inheritance, correct termination, and incorrect termination—explicitly noting that incorrect termination is still counted as a failure. This prevents the scenario where a task simply "ceasing to be processed" is misconstrued as an improvement in efficiency.
  • Well-defined paired design and information requirements. Table 1 and Appendix G.1 clarify what can be identified through full-policy evaluation, the pairing of takeover requests, and the association of termination labels. Specifically, it is noted that if an original execution of the same request already exists, the additional information required may simply involve linking records, rather than necessitating further model calls.
  • Robust disclosure of negative results and analytical limitations. The authors report that the primary migration test was inconclusive, that DOVE did not consistently outperform fixed-depth approaches, and that predictive constraints failed. They also document post-hoc analyses, protocol modifications, and missing records. This level of transparency enhances the interpretability of the results.
  • Specific experimental findings regarding interpreter intervention. Across 93 valid paired scenarios, the inclusion of an interpreter and its accompanying instructions reduced subsequent actions by 3.44 (95% confidence interval: [2.80, 4.08]). This trend remains consistent even when scenarios involving restarts are excluded (pages 36–37).
Critical weaknesses
  • The practical gains from the added attribution information remain insufficient. Core decomposition is, in itself, an algebraic identity; its value depends primarily on the execution protocol and the specific problems that the additional information can solve. Since existing, comprehensive policy evaluation methods can already measure total costs and failure rates, the paper has not demonstrated how the added attribution leads to consistent improvements in actual decision-making. Furthermore, because attribution can often be reconstructed from existing data via labels, the scope and limitations of this new approach need to be more clearly defined.
  • The primary transfer test is not supported, and secondary gains are marginal. For the Gemma/24 model, the primary test metric is 0.0194, with a 95% confidence interval of [−0.4214, 0.4800]—representing a reduction in error of approximately 0.04 actions per request. In two secondary settings for "gpt-oss," improvements of 0.31 and 0.41 actions per request were observed; however, these reflect improvements in the prediction of action allocation rather than actual savings in execution costs (pp. 39–40). These results are insufficient to substantiate a claim of broad transfer capability.
  • Insufficient documentation of model identity and timestamps in early experiments hinders causal interpretation and reproducibility. As noted on page 16, some experiments lack identifiers for the specific models used, absolute timestamps, and raw response data; the underlying model weights for the Llama endpoints are also unspecified. Consequently, even when requests are paired, it is impossible to fully rule out the influence of changes in service configuration over time. While subsequent "interleaved" execution experiments address some of these issues, they cannot retroactively rectify the limitations of the earlier records.
  • Experiments involving "interpreters" fail to disentangle the effects of available state information from those of the accompanying prompts, nor do they demonstrate that performance quality is maintained. Because the interventions simultaneously alter both the state accessible to the interpreter and the corresponding instructions, it is impossible to determine which component drives the observed savings. The difference in failure rates is +2.15 percentage points (95% CI: [−4.42, 8.45]). Moreover, when accounting for preceding actions, the use of an interpreter still results in 2.67 more actions compared to executing directly from the raw state (Table 27). Thus, while these results may support localized interface improvements, they are insufficient to justify the benefits of a broader deployment.
  • Evidence for extrapolation and reproducibility remains limited. The 126 tasks consist of held-out variants from known task families, rather than entirely new, unseen families. Some analyses were modified after observing the results, and the intervals for the frozen source coefficients do not account for the estimation uncertainty derived from the source samples. Page 10 also indicates that response-level playback relies on separately stored scientific data packages; the accessibility of these packages for peer review requires clarification.
Questions for authors
  • Given that comprehensive strategy evaluation and request-level logging are already in place, what specific decision can be altered by the additional information introduced in this protocol? Could you provide a concrete validation example, while also accounting for the costs associated with data collection?
  • Following the resolution of pending checks for the Gemma/24 migration, which findings are still considered definitive? What is the statistical power of the current experiments to detect meaningful improvements in error rates?
  • Could you provide an additional, independent ablation study on state retention and handoff mechanisms, while pre-specifying a non-inferiority margin for quality?
  • Is it possible to replicate the key endpoint results—ensuring consistent logging of model identifiers and invocation timestamps, and employing interleaved execution—to assess potential confounding by temporal factors?
  • Does the codebase include all the frozen data required for response-level replay? Which results can currently only be recalculated statistically, rather than having the experimental process itself replayed?
  • Could you incorporate an analysis involving entirely new task families and an assessment of uncertainty regarding source coefficients, to verify the robustness of the migration conclusions?
3 comments
AuthorsOpenReview ↗

(1/3): practical value (Q1/W1) and what is definitive (Q2/W2)

Thank you for the careful review. We are glad you found that the paper keeps state inheritance, correct closure and incorrect closure apart, that Table 1 and Appendix G.1 make the information requirements explicit, that negative results are reported openly, and that the interpreter intervention has a clear effect. Thank you also for the six concrete questions; they showed us exactly where the paper failed to make its value concrete. We agree with your summary that the contribution is measurement and diagnosis, not a better deployment policy. What we did not show is that this measurement changes decisions that total cost and failure cannot. Q1 shows this with a concrete case.

Q1/W1. Which decision does the added information change?

Whether an apparent saving survives fixing the failures that produce it. Complete evaluation (Table 1, level 1) prices the policy as deployed. The three terms also price each fix, without running it.

Concrete case (existing data, Table 29b). AppWorld, gpt-oss base at cap 24 vs. immediate handoff, 126 held-out tasks:

  • Level 1: total work −1.48 actions per request (cheaper), failure +50 pp [35, 65]. A failure price tells a team this policy is worse, but it still reads as "cheaper, less reliable."
  • Decomposition: the whole saving is wrong-closure work. Wrong closures hide 11.46 [8.72, 14.21] actions of original-state work.
  • If each wrong closure is reopened and redone from its original state, the same policy costs +9.98 actions per request more than immediate handoff. There is no trade-off: the saving exists only because of the failures.

Validation (new run). We ran that fix: an independent check reopens each of the 82 wrong gpt-oss/24 closures, and Luna repairs it from the state the base agent left. Measured: +10.6 [+8.9, +12.4] actions per request, vs. the +9.98 predicted with zero new model calls; repair success 72%. Repair from a wrong state costs slightly more than a fresh start (18.6 vs. 17.6 actions per wrong closure), so the prediction is a lower bound.

Each term points to a different lever, which totals cannot tell apart:

Dominant termLeverEvidence
inherited-state changehandoff interfaceexcess 6.11 → 2.67 with preserved interpreter (Table 27)
correct-closure savings < base actions (r* < 1)cheaper base or smaller budgetr* = 0.30–0.94, all 4 primary pairings (Table 4)
wrong-closure "savings"closure check + repair budgetcase above

Collection cost. If original-state runs already exist with request IDs, level 3 is a join with no new calls. Otherwise it costs one original-state run per closed request: the whole 126-task, two-base all-request study cost $18.98 (Appendix I). Scope: the method needs an original-state run on every closed request. It measures bounded model actions, not human repair effort, and we now state this in §3.

Q2/W2. What is definitive, and what power did the transfer test have?

Five accounting findings are definitive. The transfer test is the one unresolved claim, and it was underpowered. We agree that G measures allocation prediction, not execution savings. Definitive (intervals exclude zero; none depends on the pending checks):

  1. Successor work falls but total work rises in all 4 primary τ-bench pairings, +0.52 to +4.02 actions per request (Table 3). With Opus as a second successor family it rises for 2 of 3 bases (+0.99, +2.99), and all three signs were forecast before collection (Table 13).
  2. Wrong closures hide 11.46 [8.72, 14.21] actions per request (gpt-oss/24, Table 29).
  3. Interface: −3.44 [−4.08, −2.80] actions; −3.52 without restarted tasks (Table 27).
  4. Selection: at AppWorld gpt-oss/16 the paired effect is −6.45 [−11.01, −1.89], while the survivor comparison is inconclusive (global simultaneous bands, Table 20).
  5. Proportional allocation underestimates wrong-closure work in 5/6 cells (joint bands, Table 28b).
AuthorsOpenReview ↗

(2/3): transfer power (Q2, cont.), state vs. instructions (Q3/W4), model identity (Q4/W3)

Q2 (cont.). Power. From the reported interval, the Gemma/24 test has SE ≈ 0.23, so 80% power needs M ≥ 0.64. M reaches 0.76 only if the target reproduces the full source correction, so the test had 80% power only if at least 92% of the source correction transferred. The target realized 51%.

Why Gemma/24 failed is what the transfer condition (Eq. 5) predicts. We plugged each cell's observed q_c and ρ (Table 31) into Eq. 5, giving the largest source ratio κ for which the correction helps, and compared it with the frozen ratio:

Cellκ limit from Eq. 5frozen κEq. 5 saysobserved G
Gemma/81.261.58hurts−0.07
Gemma/161.541.58hurts−0.05
Gemma/241.601.58helps (barely)+0.04
gpt-oss/83.011.24helps+0.16
gpt-oss/162.391.24helps+0.31
gpt-oss/242.111.24helps+0.41

6/6 signs agree, and the primary cell sits 0.02 below its limit, hence G ≈ 0. This is a post hoc check that uses the target's own ρ, not a forecast. It also shows the association itself is present in every cell (ρ = 1.13–1.56): wrong closures always carry more original work. Only its magnitude failed to transfer, so §5.4's contribution is the condition, not the Gemma/24 point estimate.

Q3/W4. Separating state from instructions; quality.

The two arms' instructions differ by one sentence, which truthfully reports the state. We now also separate the two in a 2×2 ablation with a pre-specified margin. In the released code (aw_branch.py, aw_controls.py) the notes are identical except: fresh, "You are in a fresh Python REPL: the variables defined above no longer exist"; preserved, "You are in the same Python REPL: the variables defined above still exist." Holding this sentence fixed would tell one arm something false.

New ablation. Interpreter {fresh, preserved} × note {with this sentence, neutral note without it}, same 93 scenarios, interleaved as in Appendix H. Pre-specified before collection: failure non-inferiority margin 10 pp (the ten-point tolerance already used in the second-family replication, Appendix E), one-sided 95%. Results: state effect −3.1 [−3.8, −2.4] actions, note effect −0.3 [−0.8, +0.2], interaction −0.2 [−0.9, +0.5]. Failure, preserved − fresh: +1.1 pp, one-sided 95% upper bound +6.4 < 10. The saving comes from the available state, and the preserved interface is non-inferior at the pre-specified margin. In the original study, the +2.15 pp came from 2 discordant pairs out of 93, both against the preserved interpreter.

The residual +2.67. We agree it remains, and we do not claim that handing off at depth 8 beats starting fresh. The interface removes 56% of the handoff excess (6.11 → 2.67, Table 27). The remainder is predecessor work that depth 8 has not yet repaid, which is what the accounting is built to expose.

Q4/W3. Model identity and temporal confounding.

The AppWorld endpoints were already re-executed contemporaneously with complete logging, and they reproduce. The all-request study (Appendix I) runs both bases and the original-state and continuation branches interleaved action by action, logging requested and served identifiers, UTC times and sanitized raw responses. Any provider drift therefore hits all arms in the same round. On 126 held-out tasks, its cap-24 cells reproduce the sequential endpoints:

  • successor work −15.48 / −14.55 actions per request (sequential −14.38 / −13.32, Table 4);
  • gpt-oss failure +50 [35, 65] pp (sequential +60 [54, 67]);
  • 11.46 actions of wrong-closure work.

The Opus successor calls carry served identifiers (Table 2), and their signs were forecast before collection (Table 13). For the Llama endpoint, whose weights are unknown, we will mark its rows "endpoint identity unverified." No conclusion depends on them: without Llama, all three remaining primary pairings still rise (+0.52, +2.36, +4.02). We also re-ran these three τ-bench endpoint pairings interleaved, with full logging: +0.61 [+0.30, +0.93], +2.29 [+1.90, +2.70] and +3.88 [+3.40, +4.37] actions per request, the same signs as Table 3.

AuthorsOpenReview ↗

(3/3): replay data (Q5), new families and source uncertainty (Q6/W5), summary

Q5. What can be replayed?

You are right that the package did not match our statement. The zip contains code, frozen protocols, splits and synthetic tests, but not the collected records; our reproducibility statement wrongly said it included figure inputs. We have posted an anonymous link, visible to reviewers and the AC only, containing request-level branch tables for every reported cell, the logs of both interleaved studies with sanitized raw responses, and the cost ledgers. With these:

  • Response-level replay: both interleaved studies (Appendices H and I) and the new runs above.
  • Statistical recomputation only: the sequential τ-bench and AppWorld tables. Trajectories and per-branch rows are retained, but raw provider payloads were not (Appendix B).

We will correct the reproducibility statement to say exactly this.

Q6/W5. New task families, and source-coefficient uncertainty.

Most target families are new to the source that produced the frozen coefficient, and the gpt-oss gains hold on them. The released config/controlled_cohort.json and config/source_cohorts.json show zero task overlap between source and target. Moreover, 86/126 target families never appear in Gemma's source cohort and 70/126 never appear in gpt-oss's. These 70 are AppWorld test-challenge families, which are built around apps not used in the normal split.

  • Families unseen by the source: gpt-oss/16 M = +0.37 [+0.03, +0.71] and gpt-oss/24 M = +0.44 [+0.04, +0.84] (G = +0.30 and +0.42 actions per request); Gemma/24 remains unresolved, as Eq. 5 predicts.
  • Source uncertainty: we now resample the source cohort jointly with the target in the same 20,000 draws. gpt-oss/16 M = +0.35 [+0.03, +0.67] and gpt-oss/24 M = +0.39 [+0.02, +0.76] still exclude zero.
  • Post hoc analyses: the confirmatory set is fixed in Appendix B (lines 780–784): endpoint contrasts, prespecified work-model tests, prespecified controller rows and declared prospective hypotheses. Every other analysis, including the new ones above, is post hoc. We will move this list into §5.

Revisions we will make. (1) The Q1 case, its validation run and the lever table, in §1 and §5.2; (2) the definitive/unresolved list, the power calculation and the six-cell transfer check, in §5.4; (3) the 2×2 state/instruction ablation; (4) the logging and replication results and the Llama flag; (5) the corrected reproducibility statement and data link; (6) the analyses on unseen families and source uncertainty.

The two reasons you gave for the rating were the practical value of the added information and the unsupported transfer test. We hope Q1 (a saving that turns into a +9.98-action cost, confirmed by running the fix) and Q2 (five definitive findings, plus a transfer condition that explains all six cells) resolve both. If they do, we would be grateful if you would consider raising your score. We are glad to answer any follow-up question before the discussion closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.