acceptodds
Under review as a conference paper at ICLR 2027

PCTD: Preference-Guided Counterfactual Task Decomposition for Agent Tool Retrieval

Abstract

Task decomposition aims to transform ambiguous instructions into executable atomic subtasks, thereby guiding high-precision tool retrieval. However, our analysis reveals that directly adopting tool retrieval metrics, i.e., Recall or NDCG, as rewards for task decomposition can easily induce reward hacking in reinforcement learning-based methods. Specifically, models tend to maximize retrieval matching through strategies such as repetitive decomposition. This spurious correlation between the shallow features of decomposition results and retrieval metric impairs generalization in Out-of-Domain (OOD) scenarios involving unseen tools. To address this issue, we propose PCTD, a Preference-guided Counterfactual Task Decomposition framework. PCTD quantifies the marginal causal gain of decomposition on retrieval ranking through a counterfactual reward, thereby cutting off spurious correlations at their source. Meanwhile, it introduces a preference reward to impose fine-grained structural supervision on logical coherence and atomicity, encouraging the model to generate high-quality decompositions. In addition, we construct MTDTool, the task decomposition benchmark specifically designed for mobile multi-turn interactions. Extensive experiments demonstrate that PCTD alleviates repetitive decomposition and surpasses SOTA methods in retrieval, decomposition quality, and OOD generalization.

Primary area:
reinforcement learning
Keywords:
Task decomposition, tool retrieval

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_Z3uAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary: This paper addresses a gap in reinforcement learning (RL) for task decomposition, a tendency toward reward hacking. The authors find that when an AI agent is rewarded simply for finding the right tools, it often learns to cheat by using simplistic strategies, like repetitive and redundant subtasks. This leads to poor performance, especially when encountering new, unseen tools OOD generalization. To solve this, the paper introduces PCTD, a training framework designed to teach agents how to create high-quality, effective subtasks.

Key contributions:

  1. COunter factual Reward (CFR): Instead of rewarding the agent for its absolute retrieval score, PCTD rewards it only for the improvement its decomposition provides over a simple baseline, the CFR is supposed to discourage reward hacking.
  2. A PRM model scores the structural quality of the decomposition, evaluating it on dimensions like logical coherence, atomicity, and non-redundancy.
  3. JOint optimization of structurally sound and retrieving right tools.
  4. Contribution of MTDtool data set : Benchmarks like ToolBench or ToolRet only evaluate the final retrieved tools. They do not contain ground-truth annotations of how a complex task should be broken down into subtasks.

Validations are done on RET and MTDTool. The later data set helps expose and penalize redundancy (reward hacking)

Critical strengths
  1. The paper exposes decomposition-level reward hacking, demonstrating how standard RL causes agents to generate redundant subtasks to exploit retrieval metrics.
  2. The framework elegantly blends a Counterfactual Reward (measuring actual value-add over the raw query) with a Preference Reward to eliminate bloated outputs.
  3. The authors created and released MTDTool, filling a community gap with the first multi-turn mobile interaction benchmark featuring explicit process-level annotations.
  4. By training the agent to master the logic of decomposition rather than memorizing specific tool lists, the model achieves highly robust transferability to entirely unseen tools.
  5. PCTD is backed by extensive ablation studies and consistently outperforms highly capable baselines, including state-of-the-art LLMs, across multiple benchmarks.
Critical weaknesses
  1. The counterfactual reward is an operational heuristic rather than a mathematically rigorous causal model. It lacks a formal structural causal framework and does not control for confounding text variables, making the claim of causality weak.
  2. The balance between the counterfactual and preference rewards is highly sensitive and manually tuned to 0.2. The paper lacks a principled or dynamic mechanism to automatically adapt this weight when transferring to new tasks or domains.
  3. Decomposition quality is evaluated using a panel of closed-source LLMs rather than objective, deterministic metrics or human annotators. This introduces potential length and stylistic biases into the primary qualitative results.
  4. The newly introduced MTDTool benchmark is restricted entirely to mobile phone interaction datasets. The paper fails to prove that the PCTD framework generalizes to broader, complex agent environments like software engineering or web navigation.
Questions for authors
  1. Since your counterfactual reward relies on a shared-environment assumption without a formal structural causal model, it seems a reward heuristic rather than true causal identification? Can you justify otherwise?
  2. Given that the MTDTool benchmark is restricted entirely to mobile interactions, how do you expect the PCTD framework and the specific heuristics used to train the Preference Reward Model to generalize to broader agent domains like software engineering or web navigation or some other domains/data sets?
  3. How do you control for known LLM-as-a-judge limitations (such as length and stylistic bias) when evaluating decomposition quality, and are there plans to validate these qualitative results using human annotators?
  4. Since the performance of your joint reward system is highly sensitive to the manually tuned weight (w_pref = 0.2), have you explored dynamic or adaptive methods to automatically balance these rewards during training based on task complexity?
Reviewer_exbtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes PCTD, a framework for decomposing user requests into atomic subtasks for tool retrieval in multi-turn conversations. It trains the decomposition policy using GRPO with a retrieval reward based on improvements over direct retrieval and a preference reward for decomposition quality. The authors also introduce MTDTool, a synthetic multi-turn benchmark, and report improvements in retrieval performance and generalization to unseen tools on MTDTool and ToolRet.

Critical strengths

The paper is clearly written, and the motivation behind the proposed method is easy to understand.
The main results and ablation studies demonstrate substantial performance gains from the proposed method.

Critical weaknesses

Not much.

Questions for authors

Could you provide statistics on the number of required tools and atomic subtasks per sample, as well as dialogue turns, history length, and how far back relevant information must be retrieved from the conversation? These distributions would help characterize the task complexity of MTDTool beyond the scenario categories.

Comments on the paper

3 comments

W1 & Q1
Thank you for this valuable comment, which is important for strengthening the paper’s motivation. We have added a comparison of duplication rates for ToolQP and an absolute-NDCG reward variant. Specifically, PCTD with Absolute NDCG (w/o Preference) removes the preference reward and replaces the ranking reward max⁡(0,Natom−Nraw)\max(0,N_{\mathrm{atom}}-N_{\mathrm{raw}}) with NatomN_{\mathrm{atom}}, directly rewarding the absolute retrieval performance after decomposition. The results are reported below.

MethodDuplication Rate
ToolQP7.13%
PCTD with Absolute NDCG (w/o Preference)10.93%
PCTD w/o Preference Reward5.24%
PCTD0.70%

The results show that ToolQP has a substantially higher duplication rate than full PCTD. Replacing absolute NDCG with the counterfactual reward reduces the duplication rate from 10.93% to 5.24%. Adding the preference reward further reduces it to 0.70%. These results provide stronger empirical support for the paper’s core motivation: directly optimizing absolute retrieval metrics can encourage repetitive decomposition and other behaviors consistent with reward hacking.


W2 & Q2
Thank you for this insightful comment, which is very helpful to us. To further validate the rationale behind our counterfactual framework, we added a control experiment that replaces the dynamic threshold in the ranking reward with a fixed global threshold of 0.6.

Under PCTD, 15.7% of training inputs have candidates on both sides of the original-query threshold, 47.1% have all candidates above it, and 37.2% have all candidates at or below it. In contrast, under the fixed threshold of 0.6, inputs with candidates straddling the threshold or entirely above it account for 89.1%, substantially higher than PCTD’s 62.8%. This indicates that PCTD’s dynamic threshold more selectively determines which inputs receive positive ranking rewards.

The performance comparison is presented below:

MethodID N@10ID C@10Duplication RateOOD N@10OOD C@10Duplication Rate
Fixed Threshold 0.690.5386.704.01%81.2378.953.13%
PCTD91.3087.850.71%83.1380.731.35%

These results show that rewarding more inputs does not necessarily yield better performance. Compared with the fixed threshold, PCTD improves ID/OOD N@10 by 0.77/1.90 percentage points and C@10 by 1.15/1.78 percentage points, while reducing duplication rates from 4.01%/3.13% to 0.71%/1.35%, respectively. Both settings retain the same hinge-clipping form and differ in whether the threshold is based on the current query’s direct-retrieval performance. Therefore, the gains arise not only from clipping itself, but also from assigning positive ranking rewards only to decompositions that outperform direct retrieval. This directly supports the rationale behind our counterfactual framework.


W3
Thank you for the suggestion. To independently validate the annotation quality of MTDTool, we sampled 1,000 instances according to the scenario distribution. Three annotators independently checked the tool labels, task decompositions, and their consistency with user requests. Inter-annotator agreement reached 90%, and the consolidated review results showed an annotation accuracy of 95% on the sampled instances. This human validation provides independent evidence of the reliability of the automatically generated annotations. We will include the evaluation procedure and results in the revised manuscript.


W3
Thank you for the suggestion. We conducted three independent runs as requested, and the mean and standard deviation results are presented below. Table 2 (Mean ± Standard Deviation)

MethodID N@10ID C@10OOD N@10OOD C@10
PCTD0.9130 ± 5.87e-60.8785 ± 4.53e-50.8314 ± 1.58e-50.8073 ± 1.11e-4
w/o Preference Reward0.9129 ± 2.11e-50.8692 ± 3.11e-50.8194 ± 5.63e-40.7913 ± 5.64e-4
w/o Counterfactual Reward0.9010 ± 4.92e-60.8570 ± 3.39e-60.8037 ± 6.49e-50.7688 ± 2.73e-5

Table 9

wprefw_{\mathrm{pref}}ID N@10ID C@10OOD N@10OOD C@10
00.9129 ± 2.11e-50.8692 ± 3.11e-50.8194 ± 5.63e-40.7913 ± 5.64e-4
0.20.9130 ± 5.87e-60.8785 ± 4.53e-50.8314 ± 1.58e-50.8073 ± 1.11e-4
0.50.9102 ± 1.65e-50.8705 ± 2.97e-50.8240 ± 1.91e-50.8035 ± 7.70e-5
0.80.8838 ± 2.32e-40.8302 ± 4.11e-40.8016 ± 3.89e-40.7626 ± 6.89e-4
10.8637 ± 1.27e-50.8050 ± 2.68e-50.7763 ± 1.72e-50.7294 ± 4.69e-5

The results from three independent runs further validate the robustness of the main conclusions and directly demonstrate that the primary gains are not attributable to random fluctuations in a single training run.

Dear Reviewers, As the deadline for the author-reviewer discussion is approaching, we kindly invite you to review our response. If you have any further questions or concerns, we are more than happy to engage in a deeper discussion. Thank you once again for your time and effort in reviewing our manuscript.

Dear Reviewer, I hope this message finds you well. As the author-reviewer discussion phase will conclude in a few hours, we kindly ask if you might have a moment to review our response. We fully understand that you are busy, but your feedback would be invaluable in helping us further improve our manuscript. Thank you once again for your time and effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.