- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Summary: This paper addresses a gap in reinforcement learning (RL) for task decomposition, a tendency toward reward hacking. The authors find that when an AI agent is rewarded simply for finding the right tools, it often learns to cheat by using simplistic strategies, like repetitive and redundant subtasks. This leads to poor performance, especially when encountering new, unseen tools OOD generalization. To solve this, the paper introduces PCTD, a training framework designed to teach agents how to create high-quality, effective subtasks.
Key contributions:
- COunter factual Reward (CFR): Instead of rewarding the agent for its absolute retrieval score, PCTD rewards it only for the improvement its decomposition provides over a simple baseline, the CFR is supposed to discourage reward hacking.
- A PRM model scores the structural quality of the decomposition, evaluating it on dimensions like logical coherence, atomicity, and non-redundancy.
- JOint optimization of structurally sound and retrieving right tools.
- Contribution of MTDtool data set : Benchmarks like ToolBench or ToolRet only evaluate the final retrieved tools. They do not contain ground-truth annotations of how a complex task should be broken down into subtasks.
Validations are done on RET and MTDTool. The later data set helps expose and penalize redundancy (reward hacking)
Critical strengths
- The paper exposes decomposition-level reward hacking, demonstrating how standard RL causes agents to generate redundant subtasks to exploit retrieval metrics.
- The framework elegantly blends a Counterfactual Reward (measuring actual value-add over the raw query) with a Preference Reward to eliminate bloated outputs.
- The authors created and released MTDTool, filling a community gap with the first multi-turn mobile interaction benchmark featuring explicit process-level annotations.
- By training the agent to master the logic of decomposition rather than memorizing specific tool lists, the model achieves highly robust transferability to entirely unseen tools.
- PCTD is backed by extensive ablation studies and consistently outperforms highly capable baselines, including state-of-the-art LLMs, across multiple benchmarks.
Critical weaknesses
- The counterfactual reward is an operational heuristic rather than a mathematically rigorous causal model. It lacks a formal structural causal framework and does not control for confounding text variables, making the claim of causality weak.
- The balance between the counterfactual and preference rewards is highly sensitive and manually tuned to 0.2. The paper lacks a principled or dynamic mechanism to automatically adapt this weight when transferring to new tasks or domains.
- Decomposition quality is evaluated using a panel of closed-source LLMs rather than objective, deterministic metrics or human annotators. This introduces potential length and stylistic biases into the primary qualitative results.
- The newly introduced MTDTool benchmark is restricted entirely to mobile phone interaction datasets. The paper fails to prove that the PCTD framework generalizes to broader, complex agent environments like software engineering or web navigation.
Questions for authors
- Since your counterfactual reward relies on a shared-environment assumption without a formal structural causal model, it seems a reward heuristic rather than true causal identification? Can you justify otherwise?
- Given that the MTDTool benchmark is restricted entirely to mobile interactions, how do you expect the PCTD framework and the specific heuristics used to train the Preference Reward Model to generalize to broader agent domains like software engineering or web navigation or some other domains/data sets?
- How do you control for known LLM-as-a-judge limitations (such as length and stylistic bias) when evaluating decomposition quality, and are there plans to validate these qualitative results using human annotators?
- Since the performance of your joint reward system is highly sensitive to the manually tuned weight (w_pref = 0.2), have you explored dynamic or adaptive methods to automatically balance these rewards during training based on task complexity?
W1 & Q1
Thank you for this valuable comment, which is important for strengthening the paper’s motivation. We have added a comparison of duplication rates for ToolQP and an absolute-NDCG reward variant. Specifically, PCTD with Absolute NDCG (w/o Preference) removes the preference reward and replaces the ranking reward max(0,Natom−Nraw) with Natom, directly rewarding the absolute retrieval performance after decomposition. The results are reported below.
The results show that ToolQP has a substantially higher duplication rate than full PCTD. Replacing absolute NDCG with the counterfactual reward reduces the duplication rate from 10.93% to 5.24%. Adding the preference reward further reduces it to 0.70%. These results provide stronger empirical support for the paper’s core motivation: directly optimizing absolute retrieval metrics can encourage repetitive decomposition and other behaviors consistent with reward hacking.
W2 & Q2
Thank you for this insightful comment, which is very helpful to us. To further validate the rationale behind our counterfactual framework, we added a control experiment that replaces the dynamic threshold in the ranking reward with a fixed global threshold of 0.6.
Under PCTD, 15.7% of training inputs have candidates on both sides of the original-query threshold, 47.1% have all candidates above it, and 37.2% have all candidates at or below it. In contrast, under the fixed threshold of 0.6, inputs with candidates straddling the threshold or entirely above it account for 89.1%, substantially higher than PCTD’s 62.8%. This indicates that PCTD’s dynamic threshold more selectively determines which inputs receive positive ranking rewards.
The performance comparison is presented below:
These results show that rewarding more inputs does not necessarily yield better performance. Compared with the fixed threshold, PCTD improves ID/OOD N@10 by 0.77/1.90 percentage points and C@10 by 1.15/1.78 percentage points, while reducing duplication rates from 4.01%/3.13% to 0.71%/1.35%, respectively. Both settings retain the same hinge-clipping form and differ in whether the threshold is based on the current query’s direct-retrieval performance. Therefore, the gains arise not only from clipping itself, but also from assigning positive ranking rewards only to decompositions that outperform direct retrieval. This directly supports the rationale behind our counterfactual framework.
W3
Thank you for the suggestion. To independently validate the annotation quality of MTDTool, we sampled 1,000 instances according to the scenario distribution. Three annotators independently checked the tool labels, task decompositions, and their consistency with user requests. Inter-annotator agreement reached 90%, and the consolidated review results showed an annotation accuracy of 95% on the sampled instances. This human validation provides independent evidence of the reliability of the automatically generated annotations. We will include the evaluation procedure and results in the revised manuscript.
W3
Thank you for the suggestion. We conducted three independent runs as requested, and the mean and standard deviation results are presented below. Table 2 (Mean ± Standard Deviation)
Table 9
The results from three independent runs further validate the robustness of the main conclusions and directly demonstrate that the primary gains are not attributable to random fluctuations in a single training run.