acceptodds
Under review as a conference paper at ICLR 2027

When Does Continuation-Conditioned Supervision Improve Tool-Using Agents?

Abstract

Counterfactual supervision values an action by executing the remainder of a task with a chosen controller. Even an exact return can favor an action that is poorly suited to a different continuation. We study this effect in a controlled tool-use simulator, holding checkpoints, candidate actions, and the current-action risk rule fixed. We derive sufficient conditions under which continuation changes preserve the learner-visible optimal action, accounting for fitting error. Fixed-checkpoint diagnostics show that reference-conditioned selection is strictly suboptimal under a frozen deployment continuation at 16.8% of 800 checkpoints. We then evaluate newly fitted policies on held-out complete tasks, keeping training states and fitting procedures fixed across targets. Relative to reference supervision, fully deployment-conditioned targets yield mean paired utility gains of 0.076 for a cost-sensitive classifier and 0.099 for a multilayer perceptron. Both learner classes gain 3.0 success percentage points on average over three seeds, with improvements in utility and success in all six matched comparisons. These findings show that continuation choice can affect both the actions favored by supervision and the complete-task performance of policies learned from it.

Primary area:
foundation or frontier models, including LLMs
Keywords:
tool-using agents, counterfactual supervision, imitation learning, learning to search, sequential decision making

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cQTZOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Counterfactual supervision values an action by executing the remainder of a task with a chosen controller. Even an exact return can favor an action that is poorly suited to a different continuation. This paper uses a simulated environment to investigate how changes in the strategy for completing the remainder of a task affect the selection of the optimal action at the current step. Fixed-checkpoint experiments show that reference-conditioned selection is strictly suboptimal under a frozen deployment continuation at 16.8% of 800 checkpoints. Experiments evaluate policies trained with different continuation targets on held-out complete tasks. Both learner classes gain 3.0 success percentage points on average over three seeds, with improvements in utility and success in all six matched comparisons. These findings show that continuation choice can affect both the actions favored by supervision and the complete-task performance of policies learned from it.

Critical strengths
  • Clear experiment design with controlled variables.
  • Transparent reporting of negative results and the limited scope of the simulated environments.
Critical weaknesses
  • Limited scope of the simulated environments. The connection between this work and real-world scenarios is weak. All experiments are conducted within a simulator with a set of five discrete actions, and decision processes up to 6 steps. No explanation about how this simulator map to real-world tasks.
  • The contribution over existing works is incremental. Not proposing any new guarantees for policy improvement.
  • Limited experiment scale with only three training seeds, with 100 test tasks per seed.
Questions for authors
  • Could you provide a concrete example of a real-world task scenario to explain how does your simulation map to a real-world scenario involving the use of tools?
  • Compared to existing work such as LOLS and AggreVaTe, what are the key findings/contributions of this paper that cannot be directly derived from prior works?
  • Do your findings still hold when increasing the number of seeds and expanding the test set? Could you validate the findings in at least one specific real world environment involving the use of tools?
2 comments

Thank you for asking how the simulator maps to an actual tool-use decision. We can now point to a scored telecom case and show how often the underlying effect appears on a larger test set.

Q1 · What does this look like in a tool-using agent? In a native τ²-Bench task, a simulated customer abroad had a mobile-data speed of 0.05 Mbps. At the same point in the conversation, the selector trained on reference labels transferred to human support. The selector trained on deployment labels and the assistant's first proposal both asked the customer to check the VPN. Their shared continuation confirmed the VPN was off, switched the handset's network preference to 4G/5G, and measured 275 Mbps. The official score was 0 for transfer and 1 for the branch that resolved the problem.

The simulator lets us measure this kind of choice systematically. At one recorded state, the reference chose Act. With the same deployment controller finishing both branches, checking first (Verify) raised utility from 2.918 to 2.939, with both branches succeeding. We will use the telecom example to explain what a checkpoint, a candidate action, and a continuation correspond to in a tool-using conversation.

Q2 · What does this add to AggreVaTe and LOLS? AggreVaTe values actions with the expert's cost to finish the task. LOLS studies how to choose the policies that visit states and finish from them. We take the same state and eligible actions, then score each action under both ways of finishing. This exposes which reference choices are costly under deployment and gives a rule for finding decisions worth checking. When the estimated between-controller shift is large relative to the gap between the top two reference actions, the reference choice deserves a closer look. On new tasks, that choice was strictly suboptimal under deployment at 25.4% of states flagged by the fitted test versus 4.5% of the rest. We will position this as a way to diagnose action choices under a change of continuation.

Q3 · What held up on a larger test set? Across 10 training runs and 1,000 shared new simulator tasks, the reference choice was strictly worse under deployment at 16.61% of checkpoints, again about one in six. Choosing the best first action by exact returns, with deployment finishing both branches, raised mean utility by 0.041 and success by 0.94 percentage points across all checkpoints with equal task weights. For both learners fitted on deployment-collected states, reference-only labels led full-task success by less than one percentage point.

In the native 12-task telecom comparison, the two CSOAA selectors and the first-proposal control shared candidates and the same Terra-low continuation. The label source changed the action on seven of the eight tasks reaching the checkpoint. Official successes were 3/8 with reference labels, 4/8 with deployment labels, and 5/8 with the first proposal. The slow-data case in Q1 accounts for the one success difference between the two selectors.

Since our earlier reply, we completed a separate evaluation on 150 held-out τ²-Bench telecom tasks, equally split between MMS and mobile data. The joint state–action selector could choose a different action at up to four points in each conversation. All three arms used the same Terra-low model for the assistant and simulated customer, and they shared candidate actions whenever they reached the same conversation state.

The joint selector succeeded on 105/150 tasks, compared with 75/150 for the previous CCR scorer and 65/150 for the first-proposal control. Against first proposal, it recorded 44 paired wins and 4 losses. In 38 of those 44 wins, the first action difference came when the assistant proposed transferring the customer to human support.

One MMS task shows what the selector changed. After looking up the account, the first-proposal branch transferred the customer. The joint selector instead asked the customer to check whether other apps could use mobile data, then had the customer check the SIM. The assistant could continue through device checks the customer was able to perform. The resulting conversation passed both official task checks, for sending MMS and refilling mobile data.

Across complete conversations, the joint selector used 37.3% more net tokens than the first-proposal control, which also generated candidates and always took its first proposal. Replaying all 450 trajectories with the original upstream scorer reproduced every official score and scoring component.

The 12-task comparison in our earlier reply shows how changing training labels changes action choice at a fixed checkpoint. The new study measures the complete selector across multiple decisions in a native tool-using conversation. We will include the MMS trace alongside the simulator example in the paper so readers can see how a checkpoint, its candidate actions, and the continuation map to a concrete benchmark task.

Reviewer_2QXXOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies counterfactual action labels. At a restored simulator checkpoint, each candidate action is forced and a chosen controller (the continuation) finishes the task, so even exact returns depend on that controller. Labels from a reference controller are compared with labels from a frozen deployment controller π0 (a reference-trained MLP), and the two are interpolated by a weight α. A sufficient condition, with and without fitting error, guarantees that the learner-visible best action is unchanged when the reference margin exceeds the spread of the actions' value shifts. In a custom five-action simulator, the reference choice is strictly worse under the frozen deployment continuation at 134 of 800 checkpoints. A matched held-out study varies only the target (CSOAA and MLP, three seeds, 100 fresh tasks per seed). There, α = 1 raises mean utility by 0.076 and 0.099 and success by 3.0 points. The effect is positive in all six learner–seed pairs, but most per-seed intervals include zero. Retrospective studies attribute larger gains to coverage and capacity, and a sampled 50–50 continuation mixture hurts.

Critical strengths
  1. Matched held-out design (Sec. 3; App. B.2–B.5). Within each learner and seed, only Tα changes. The checkpoints, the 4,200 task-balanced rows from the same 140 tasks (Eq. 11), the untuned fitting rules and the frozen risk gate (Eq. 13) are shared, and holdouts were fixed before outcomes were seen and checked for overlap (Eq. 12). Because every Tα mixes the same two recorded values (Eq. 7), the directional result is attributable to the target: all six α = 1 pairs improve utility and success (Fig. 3; Table 6). This is why I do not rate soundness lower.
  2. Candid, consistent reporting. The main text concedes that most task-level intervals include zero (p2, L90), and all 18 intervals are given (Table 6). Unfavorable results are kept: the Table 1 global blend, the +0.001-utility replay contrast in Table 2, and the Sec. 5.4 mixture loss. The evidence can therefore be taken at face value, which keeps my soundness concerns in the fixable category.
Critical weaknesses

1. The contribution is largely anticipated, and the theory does not answer the title's question.

  • LOLS [1], cited in Sec. 8, already argues that roll-outs with a reference that is suboptimal or outside the hypothesis class can hide compounding errors. It uses the same construction: force each action, complete it with a roll-out policy, and train a cost-sensitive learner on the gap to the best action. Its mixture roll-out uses the same per-roll-out coin flip as Sec. 5.4 (β = 0.5). The paper neither notes this nor discusses LOLS's finding that mixture roll-outs beat a suboptimal reference.
  • The α = 1 CSOAA arm (roll out frozen π0; cost maxb Tα(x,b) − Tα(x,a), p15, L760–761) is essentially one step of rollout-based classification policy iteration from π0 [2, 3]. That reading already explains why such targets help. For LLM step labels, Setlur et al. [4] measure progress under a "prover" policy distinct from the base policy. The paper cites none of [2–7].
  • Eq. 6 and Eq. 8 are a few lines of triangle inequality (App. A), the reasoning of action-gap analyses [7]. They only certify invariance ("Its failure leaves the ranking unresolved", p3, L147), and they are never computed ("not an estimated certificate for the fitted models", p4, after Eq. 8). The paper therefore never says when the targets help.
  • Why it matters: what remains new is a small-simulator confirmation of a known effect. Suggestion: position the work against [1–4]. Then estimate γR and ϵvis from the existing paired labels, and test whether violations of Eq. 6 predict the 134 strict flips and the per-family gains.

2. The evidence for the main claim is thin, and the comparison is under-specified.

  • Scale. The study uses one custom simulator, 140 training tasks per seed (App. B.3), three seeds and one 100-task panel per seed. At α = 1, only two of six per-seed utility intervals exclude zero, both for CSOAA (Table 6). None of the three MLP intervals at α = 1 does, though MLP has the larger mean gain (p7, L344). A gain of +3.0 success points is three tasks per panel on average.
  • Optimization. App. B.2 states "All 12 MLP fits reached this iteration limit with convergence warnings" (p15, L763–764). MLP differences across α may therefore reflect fitting speed rather than target quality.
  • Undefined baseline. The reference controller is never specified (p1, L53; Sec. 3); the only hint is a "reference-planner value gap" (p14, L738). If it is Table 1's exact-oracle planner with hidden-state access, the gain is the known privileged-expert imitation gap [5, 6]. π0 is described only in App. B.2 (p15, L756–757), and its holdout performance is not reported, so we cannot tell whether fitted policies beat their label source.
  • Coverage. The coverage regime of the training checkpoints is not stated (App. B.2). Yet Table 2 shows the target contrast falling from +0.164 utility under logged coverage to +0.001 under replay.
  • Scope. No tool-using LLM agent is evaluated (Sec. 7).
  • Why it matters: the main claim reduces to a directional effect in one toy setting, measured against an unidentified baseline, and the title's claim about tool-using agents is untested. Suggestion: define ρR, and report ρR and π0 on the holdouts. Rerun the MLP arm to convergence. Add seeds (the attribution study already uses eight) with a pre-specified test. Add one LLM-agent environment, such as τ-bench, which the paper cites.
Questions for authors
  1. What is ρR? Is it Table 1's exact-oracle planner, and does it have hidden-state access? What are the utility and success of ρR and π0 on the three holdouts, and under which coverage were the training checkpoints collected? A privileged ρR would make the result an instance of the imitation gap [5, 6]. A π0 that beats every fitted policy would weaken the practical case. The answers bear directly on my soundness and informativeness scores.
  2. Using the existing paired labels, can you estimate γR(o) and ϵvis(o) (Eq. 4–5) and report how often Eq. 6 holds? Do violations predict the 134 strict flips and the per-family closed-loop gains? A predictive result would answer the title's question and raise my informativeness score.
  3. What does the held-out study show beyond LOLS's roll-out analysis [1] and the policy-improvement reading of one rollout-classification step from π0 [2]? How does Sec. 5.4 relate to LOLS's mixture roll-out? A clear delta would change my view of the contribution.

References

[1] Kai-Wei Chang, Akshay Krishnamurthy, Alekh Agarwal, Hal Daumé III, John Langford. "Learning to Search Better Than Your Teacher." ICML, 2015.

[2] Alessandro Lazaric, Mohammad Ghavamzadeh, Rémi Munos. "Analysis of Classification-based Policy Iteration Algorithms." JMLR 17(19), 2016.

[3] Michail G. Lagoudakis, Ronald Parr. "Reinforcement Learning as Classification: Leveraging Modern Classifiers." ICML, 2003.

[4] Amrith Setlur et al. "Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning." arXiv:2410.08146, 2024.

[5] Luca Weihs et al. "Bridging the Imitation Gap by Adaptive Insubordination." NeurIPS, 2021.

[6] Andrew Warrington et al. "Robust Asymmetric Learning in POMDPs." ICML, 2021.

[7] Amir-massoud Farahmand. "Action-Gap Phenomenon in Reinforcement Learning." NIPS, 2011.

3 comments

Thank you for proposing the margin and convergence checks. We ran both and evaluated the action-label comparison in native τ²-Bench telecom.

Q1 · Which controllers and checkpoints did we use? The reference ρR\rho_R is the planner behind the Table 1 oracle and sees the full simulator state. The deployment controller π0\pi_0 is a fixed MLP trained on reference labels. It collected checkpoints on 140 training tasks per seed. We varied the label weight α\alpha from reference-only (α=0\alpha=0) to deployment-only (α=1\alpha=1) while sharing checkpoints, actions, task weights, and the risk rule. The variants within each seed share a 100-task holdout. The holdouts differ across seeds.

SeedReference plannerFixed deployment controller π0\pi_0
191.990 / 65%1.673 / 57%
291.758 / 58%1.461 / 51%
391.770 / 58%1.224 / 44%

The table gives utility and success. By utility, π0\pi_0 beat all eight fitted variants on seeds 19 and 29, and all eight beat it on seed 39. On the expanded holdout it averaged 1.567 utility and 54.29% success. We will report these baselines with the fitted policies and specify the reference's privileged access.

Q2 · Can the margin tell us where to check the reference action? On 1,000 fresh simulator tasks, the reference choice was strictly suboptimal under deployment at 25.4% of checkpoints flagged by the fitted margin test versus 4.5% of the rest. The shift-to-margin score achieved mean AUC 0.716 across 10 training runs.

We fit values for both continuations from the learner's inputs. The largest absolute value shift across eligible actions bounds the fitted ϵvis\epsilon_{\mathrm{vis}} in Eq. 5. In the fitted version of Eq. 6, the gap between the top two eligible reference scores exceeded twice this bound at 42.2% of 59,411 checkpoints, weighting tasks equally. With exact simulator values, 56.45% passed the test, and the reference choice remained optimal at every passing checkpoint. The fitted test gives a practical way to prioritize which decisions to check under the deployment continuation.

We evaluated full-task utility after refitting separately by family. In delegation, choosing the best first action by exact returns with the deployment controller fixed added 0.048 utility. After refitting, deployment-only labels gave higher utility for MLP, and reference-only labels gave higher utility for CSOAA. This family illustrates why we report the checkpoint diagnostic and the fitted-policy comparison separately.

Q3 · What does this add to rollout-based policy improvement? At α=1\alpha=1, the CSOAA arm is one rollout-classification step from π0\pi_0. The paired values let us measure the cost of using the reference's choice before any refitting. Choosing the best action by exact returns and using the same fixed controller to finish both branches gained 0.041 utility on average across all new checkpoints.

Section 5.4 uses the LOLS construction of drawing a controller once per rollout, with a 50–50 reference/current-learner mixture. LOLS reported gains from mixing on dependency parsing. Our matched target study holds the checkpoints and both continuations fixed and varies the weights of their recorded returns. We will make this relationship explicit and discuss the policy-iteration and step-supervision connections. LOLS, classification-based policy iteration, prover-based process supervision.

W2 · Optimization check. We reran the three-seed MLP comparison with higher iteration caps. At caps of 320 or above, every fit met the original stopping rule by iteration 316. In the 320-cap rerun, the reference-only MLP averaged 0.102 higher utility than the deployment-only MLP on the 1,000-task holdout. We will report the rerun alongside the original fit.

Native τ²-Bench telecom. On 12 held-out tasks, the two CSOAA selectors and the assistant's first proposal shared candidates and the same Terra-low continuation. Changing the source of the training labels changed the selected action on seven of the eight tasks reaching the checkpoint. Official success was 3/8 with reference labels, 4/8 with deployment labels, and 5/8 with the first proposal. The extra success over reference labels came on a slow-data task where the deployment-label selector continued troubleshooting and the reference-label selector transferred the customer.

Thank you authors for addressing my questions. I will keep my positive rating of this manuscript.

Thank you for taking the time to read our response. Your questions helped us clarify the paper’s contribution and strengthen the evaluation.

Reviewer_BFWXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks which controller should complete the suffix when generating counterfactual labels for tool-use policy learning. It fixes the restored checkpoint, candidate action, feasible set, and risk gate, then contrasts a reference continuation ρR\rho_R with a frozen deployment controller π0\pi_0. The authors derive the margin condition γR(o)>2ϵvis(o)\gamma_R(o) > 2 \epsilon_{vis}(o) for action-ranking invariance under target shift, and extend it to fitted scores. Experiments report reference-conditioned selection as strictly suboptimal at 16.75% of 800 anchor checkpoints, with mean ideal one-step gains of +0.068 utility and +1.88pp success. Overall, the paper gives a clean three-stage attribution of continuation choice within a controlled simulator, but the headline gains sit on a thin sample and a scope that excludes language-model agents and roll-in variation, so the contribution is a well-isolated diagnostic rather than a demonstrated agent-training recipe.

Critical strengths
  • Clean three-way attribution with pre-registered evaluation. Table 1 separates coverage, learner capacity, and target choice on 8 seeds backed by Table 5's bootstrap CIs. Figure 2(c) further localises target-choice effects to the top decile. Appendix B.4 fixes each seed's 100-task holdout before any fitted-policy outcome is inspected. Reporting hygiene is above th median.
Critical weaknesses
  • Scope restricts the finding to a controlled anchor set. The paper cites ReAct, Toolformer, Gorilla, τ-bench, ToolSandbox, WebArena, and SWE-bench but evaluate on none of them. Fixing checkpoints, risk gate, and feasible set rules out confounders cleanly, but the same choice makes the finding hard to port to the roll-in methods (DAgger / AggreVaTe / LOLS) that produce the state distributions real agents visit. Table 3's gains therefore attach to a declared continuation on fixed anchors, not to a general roll-out choice under evolving coverage.
  • Headline empirical claims outrun their statistical support. Three issues compound. First, "6/6 matched runs positive" in Table 3 is a selection on α=1: Table 6 shows two negative CSOAA cells at intermediate weights, including Seed 39 α=0.1\alpha=0.1 at −0.029, CI [−0.113, −0.0002] excluding zero, and 13 of 18 task-level intervals cover zero. Second, the whole claim rests on 3 training seeds ×\times 100 holdout tasks ×\times 2 learners with task-level bootstrap only
Questions for authors
  • With a variance-matched M50 arm, like multiple independent rollouts of the 50-50 mix averaged, matched to T0.5T_{0.5}'s zero sampling noise, does Table 7's −0.104 utility gap persist? If it closes, §5.4's "blanket mixing is not sufficient" reduces to a known lesson about single-sample label noise.
  • One real-benchmark probe (τ-bench or ToolSandbox, even at reduced N and only on restorable prefixes) comparing α=0\alpha=0 and α=1\alpha=1 under the paper's own fitting protocol: do the directions in Table 3 reproduce? Without such a probe the finding is confined to the paper's simulator.
1 comment

Thank you for proposing variance control and a native benchmark probe. We completed both.

Q1 · The mixture gap survives variance control on the original panels. In Table 7, replacing the sampled 50–50 rollout with the exact mean of reference and current-learner returns changed mixture-minus-reference utility from −0.104 to −0.119. Exact averaging removes the variance from drawing the rollout controller. On 1,000 fresh simulator tasks, the exact-mixture gap was smaller at −0.048. In a separate fixed-controller comparison, the largest sampled-versus-exact difference in mean test utility was 0.023 across both learners and 1, 8, or 32 draws. We will report the original and fresh panels together in Section 5.4.

Q2 · Changing labels changes actions in native τ²-Bench telecom. We fitted two CSOAA selectors on the same 15 development tasks with identical features, settings, and fitting seed. The reference labels used Terra-high with a troubleshooting checklist and the deployment labels used Terra-low. On 12 held-out tasks, both selectors and the assistant's first proposal shared four candidates, a Terra-low assistant and simulated user, and execution limits. We scored one rollout per distinct action.

The selectors chose different actions on seven of the eight tasks reaching the checkpoint. Official success was 3/8 with reference labels, 4/8 with deployment labels, and 5/8 with the first proposal. Four tasks succeeded before intervention, bringing the totals to 7/12, 8/12, and 9/12. Two cases explain the success differences.

  • Slow mobile data. The reference-label selector transferred after a 0.05 Mbps test. The deployment-label selector and the first proposal both chose to check the VPN. Their shared branch found it was off, switched the network preference to 4G/5G, and measured 275 Mbps, resolving the issue.
  • MMS troubleshooting. The deployment-label selector transferred immediately, and the reference-label selector transferred after checking the APN. The first proposal led to an APN reset, app discovery after reboot, permission fixes, and a successful MMS test.

Scale and training-state coverage. We ran all label weights on 10 seeds and 1,000 shared new simulator tasks. The table gives mean utility and success using deployment-collected training states.

Learner0 (reference)0.10.51 (deployment)
MLP1.380 / 49.51%1.391 / 49.88%1.367 / 48.84%1.349 / 48.56%
CSOAA1.405 / 47.98%1.399 / 47.75%1.372 / 46.91%1.375 / 47.13%

With a full-state reference, the CSOAA success difference from switching to deployment-only labels was +4.43 percentage points on reference-collected states and −0.85 points on deployment-collected states. We will tie the success claim to the training-state source and report the full sweep.

114-task system comparison. On the GPT-5.4 bridge at one seed, adding candidate generation and reranking raised official success from 56.1% to 78.1% (+21.9 percentage points), with 35 paired wins and 10 losses. Mean time rose from 277 to 881 seconds per task (+604 seconds). These numbers describe the combined system and its added inference cost.

12-task reranking pilot with common execution limits. On a prespecified panel, we compared the first proposal with reranking by the frozen CCR scorer from the 114-task study. Both arms used Terra-low for the assistant and user, shared four proposals at each of the 11 reached checkpoints, and ran one continuation per distinct action. Official success rose from 7/12 to 9/12, with 2 wins, 0 losses, and 10 ties. Counting shared calls in each arm, model calls rose from 375 to 444 (+69 during continuation) and mean time from 159.1 to 184.2 seconds per task (+25.1 seconds).

Separate 12-task reranking pilot matched on actual tokens. Both arms used Terra-low and shared proposals. Reranking used the same CCR scorer. Before scoring, we matched prefixes ending after a model action and any immediate tool reply, counting uncached input plus output tokens for both roles. For different actions, H was the minimum of 100,000 tokens and the two recorded continuation totals. Each prefix used 80–102% of H and differed by at most 2% of the smaller spend. We maximized the smaller spend, then minimized the gap. A prefix cut before termination received zero under the official timeout rule.

Both arms scored 2/11, with 11 ties. Among the three tasks whose actions changed, both branches were cut in one case (98,510 vs. 99,946 tokens). In the second, the user ended the first-proposal branch with zero reward at 33,129 tokens and the CCR branch timed out at 33,118. The third had no eligible match.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.