- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
Counterfactual supervision values an action by executing the remainder of a task with a chosen controller. Even an exact return can favor an action that is poorly suited to a different continuation. This paper uses a simulated environment to investigate how changes in the strategy for completing the remainder of a task affect the selection of the optimal action at the current step. Fixed-checkpoint experiments show that reference-conditioned selection is strictly suboptimal under a frozen deployment continuation at 16.8% of 800 checkpoints. Experiments evaluate policies trained with different continuation targets on held-out complete tasks. Both learner classes gain 3.0 success percentage points on average over three seeds, with improvements in utility and success in all six matched comparisons. These findings show that continuation choice can affect both the actions favored by supervision and the complete-task performance of policies learned from it.
Critical strengths
- Clear experiment design with controlled variables.
- Transparent reporting of negative results and the limited scope of the simulated environments.
Critical weaknesses
- Limited scope of the simulated environments. The connection between this work and real-world scenarios is weak. All experiments are conducted within a simulator with a set of five discrete actions, and decision processes up to 6 steps. No explanation about how this simulator map to real-world tasks.
- The contribution over existing works is incremental. Not proposing any new guarantees for policy improvement.
- Limited experiment scale with only three training seeds, with 100 test tasks per seed.
Questions for authors
- Could you provide a concrete example of a real-world task scenario to explain how does your simulation map to a real-world scenario involving the use of tools?
- Compared to existing work such as LOLS and AggreVaTe, what are the key findings/contributions of this paper that cannot be directly derived from prior works?
- Do your findings still hold when increasing the number of seeds and expanding the test set? Could you validate the findings in at least one specific real world environment involving the use of tools?
2 comments
Since our earlier reply, we completed a separate evaluation on 150 held-out τ²-Bench telecom tasks, equally split between MMS and mobile data. The joint state–action selector could choose a different action at up to four points in each conversation. All three arms used the same Terra-low model for the assistant and simulated customer, and they shared candidate actions whenever they reached the same conversation state.
The joint selector succeeded on 105/150 tasks, compared with 75/150 for the previous CCR scorer and 65/150 for the first-proposal control. Against first proposal, it recorded 44 paired wins and 4 losses. In 38 of those 44 wins, the first action difference came when the assistant proposed transferring the customer to human support.
One MMS task shows what the selector changed. After looking up the account, the first-proposal branch transferred the customer. The joint selector instead asked the customer to check whether other apps could use mobile data, then had the customer check the SIM. The assistant could continue through device checks the customer was able to perform. The resulting conversation passed both official task checks, for sending MMS and refilling mobile data.
Across complete conversations, the joint selector used 37.3% more net tokens than the first-proposal control, which also generated candidates and always took its first proposal. Replaying all 450 trajectories with the original upstream scorer reproduced every official score and scoring component.
The 12-task comparison in our earlier reply shows how changing training labels changes action choice at a fixed checkpoint. The new study measures the complete selector across multiple decisions in a native tool-using conversation. We will include the MMS trace alongside the simulator example in the paper so readers can see how a checkpoint, its candidate actions, and the continuation map to a concrete benchmark task.
Thank you for asking how the simulator maps to an actual tool-use decision. We can now point to a scored telecom case and show how often the underlying effect appears on a larger test set.
Q1 · What does this look like in a tool-using agent? In a native τ²-Bench task, a simulated customer abroad had a mobile-data speed of 0.05 Mbps. At the same point in the conversation, the selector trained on reference labels transferred to human support. The selector trained on deployment labels and the assistant's first proposal both asked the customer to check the VPN. Their shared continuation confirmed the VPN was off, switched the handset's network preference to 4G/5G, and measured 275 Mbps. The official score was 0 for transfer and 1 for the branch that resolved the problem.
The simulator lets us measure this kind of choice systematically. At one recorded state, the reference chose Act. With the same deployment controller finishing both branches, checking first (Verify) raised utility from 2.918 to 2.939, with both branches succeeding. We will use the telecom example to explain what a checkpoint, a candidate action, and a continuation correspond to in a tool-using conversation.
Q2 · What does this add to AggreVaTe and LOLS? AggreVaTe values actions with the expert's cost to finish the task. LOLS studies how to choose the policies that visit states and finish from them. We take the same state and eligible actions, then score each action under both ways of finishing. This exposes which reference choices are costly under deployment and gives a rule for finding decisions worth checking. When the estimated between-controller shift is large relative to the gap between the top two reference actions, the reference choice deserves a closer look. On new tasks, that choice was strictly suboptimal under deployment at 25.4% of states flagged by the fitted test versus 4.5% of the rest. We will position this as a way to diagnose action choices under a change of continuation.
Q3 · What held up on a larger test set? Across 10 training runs and 1,000 shared new simulator tasks, the reference choice was strictly worse under deployment at 16.61% of checkpoints, again about one in six. Choosing the best first action by exact returns, with deployment finishing both branches, raised mean utility by 0.041 and success by 0.94 percentage points across all checkpoints with equal task weights. For both learners fitted on deployment-collected states, reference-only labels led full-task success by less than one percentage point.
In the native 12-task telecom comparison, the two CSOAA selectors and the first-proposal control shared candidates and the same Terra-low continuation. The label source changed the action on seven of the eight tasks reaching the checkpoint. Official successes were 3/8 with reference labels, 4/8 with deployment labels, and 5/8 with the first proposal. The slow-data case in Q1 accounts for the one success difference between the two selectors.