- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether tool-use agents internally represent task progress. It identifies progress-sensitive directions using externally verified evidence and matched comparisons. It then proposes Probe-Gated Steering (PGS) to improve evidence assimilation and action selection. Experiments show improvements on two travel-planning benchmarks.
Critical strengths
- Meaningful research question and well-motivated analysis. The paper addresses an important question about how agents internally track progress during long-horizon execution. The analysis proceeds logically from identifying progress-sensitive representations to examining their evolution. Externally verified progress and matched comparisons provide a reasonable foundation for these findings.
- A reasonable probe-based method grounded in the findings. PGS naturally extends the analysis by detecting potentially problematic states and selectively applying targeted steering. Distinguishing evidence assimilation from evidence review gives the interventions clear motivations. The component ablations provide supporting evidence for this design.
Critical weaknesses
-
The benchmark scores make the gains difficult to assess. Complete-task success is zero or very low, while Delivery Rate is already near saturation for several baselines. This combination makes the reported improvements appear modest in practical terms. Moreover, the best-of-two protocol, labeled pass@2, obscures single-run performance and may be sensitive to sampling variability. Pass@1 results, repeated evaluations, and confidence intervals are needed to establish whether the gains reliably exceed run-to-run fluctuations.
-
Evaluation is limited to travel planning. The method targets general mechanisms of progress tracking and evidence integration, so its evaluation should extend beyond travel planning. Additional benchmarks, such as WebArena for web interaction or SWE-bench for coding, would test its broader applicability. Any other suitable long-horizon agent benchmark outside travel planning would also help address this limitation.
Questions for authors
See the weankness.
6 comments
Response to Reviewer mEho (2/2)
Q3. Generalizability of the Method
Our existing results already demonstrate the framework's robustness across multiple settings: progress representations are validated across two models and two travel benchmarks, while PGS improves performance across models and both Native and ReAct protocols. We also observe gains on Chinese tasks (CP: 23.3 → 27.2). Ablations support the contributions of EAS and ERS, and show that targeted triggering outperforms random triggering (CP: 27.2 vs. 24.0). Moreover, the two travel benchmarks cover different constraint structures and difficulty levels.
We agree that evaluating additional planning domains is valuable for further assessing the framework's applicability. Within the limited rebuttal period, we conducted additional rollouts on DeepPlanning-Shop and completed both representation analysis and downstream PGS evaluation.
Shopping Planning involves product-attribute matching, multi-product selection, coupon constraints, and budget optimization, providing a distinct planning domain beyond travel-specific entities and constraints. The additional results are as follows.
Progress representation (Qwen3-14B, 263 held-out test pairs per anchor):
| Anchor | (v^E) | (v^D) | (v^P) |
|---|---|---|---|
| Entry | 90.1% | 76.0% | 89.7% |
| Decision | 86.3% | 86.7% | 87.1% |
All directions achieve PairAcc substantially above chance (50%) at both anchors, including the shared progress direction.
Downstream PGS performance:
| Metric | Pass@1 (Base → PGS) | Pass@2 (Base → PGS) |
|---|---|---|
| Match Acc | 21.4% → 23.8% | 36.4% → 44.7% |
| Case Acc | 4.2% → 5.8% | 8.3% → 10.8% |
Match Acc measures micro-averaged partial matching of required products and coupons, whereas Case Acc requires all ground-truth items to be matched. PGS improves both metrics under Pass@1 and Pass@2 evaluation, providing further evidence that the framework extends beyond travel planning.
More generally, the framework is not tied to travel-specific entities or constraints. It can be adapted to other tool-use domains by grounding progress in their verifiable task states and adapting the corresponding progress verifier and learned components. For coding and web/GUI agents, this would involve progress references grounded in code changes or browser/environment states, with appropriate handling of state revisions. We will clarify this scope and the required adaptations in the revision, and further explore these settings in future work.
Dear Reviewer mEho,
We sincerely appreciate your thoughtful assessment and, in particular, your recognition of the significance of our research question and the motivation behind our approach. We have carefully addressed the concerns raised in your review through additional Pass@1 results, further clarification of the evaluation metrics, and a new cross-domain evaluation on DeepPlanning-Shop.
We would like to draw your attention to our detailed rebuttal and respectfully ask that you carefully consider these additional results in reassessing the paper. As the discussion period is nearing its end, we look forward to hearing from you before it closes.
Best, Authors
Response to Reviewer mEho (1/2)
We appreciate your recognition of our work and your careful review and thoughtful suggestions. Below, we address the concerns with additional analyses and a cross-domain evaluation.
Q1. Single-run Performance and the Role of Pass@2
We originally adopted Pass@2 as a complementary evaluation protocol to reduce the variability of evaluation outcomes in long-horizon planning. These tasks involve many steps and are sensitive to local errors; on difficult tasks, Pass@1 scores can be very low and vary substantially across sampled rollouts, making comparisons based on a single rollout less stable. Best-of-two evaluation can partially mitigate single-rollout sampling noise and provide a more stable comparison between methods. Nevertheless, we agree that Pass@1 is important for understanding single-run performance and therefore report additional results below.
Additional Pass@1 results on DeepPlanning-Travel(all values in %):
PGS consistently improves the scorable trajectory ratio and all three plan-quality metrics under Pass@1. In particular, the Composite Score increases by 2.4 points for Qwen3-14B and 6.2 points for Ministral-3-14B. These results show that the improvements are also present under single-rollout evaluation, rather than arising solely from best-of-two selection. We will include more results in the revision and clarify the complementary roles of Pass@1 and Pass@2.
Q2. Interpreting Delivery Rate and End-to-End Planning Quality
We would like to clarify that the practical improvements of PGS are reflected in multiple complementary metrics, rather than Delivery Rate alone.
First, PGS improves Delivery Rate across all evaluated models. The gains are particularly notable for Ministral and Apriel, whose baseline delivery rates are relatively low (82.5% and 71.7%), with improvements of 4.2 and 5.8 percentage points, respectively. Importantly, PGS still yields gains for Qwen models with already high baseline delivery rates (97.5% and 99.2%), improving them by 1.7 and 0.8 percentage points, respectively.
Second, PGS improves overall itinerary quality beyond successful plan delivery. CS measures the commonsense plausibility and execution logic of itineraries (e.g., temporal arrangements, route continuity, and budget consistency), while PS measures satisfaction of explicit personalized hard constraints. Under Pass@1, PGS improves CS/PS by 3.2/1.7 points for Qwen3-14B and 9.1/3.3 points for Ministral-3-14B, yielding Composite Score gains of 2.4 and 6.2 points, respectively. These improvements demonstrate that PGS enhances not only whether a plan is delivered, but also the quality and constraint satisfaction of the resulting itinerary.
Furthermore, end-to-end success is not uniformly zero across our evaluation. On TravelPlanner, PGS improves Final Pass Rate under both protocols. For example, under Native, Qwen3-32B improves from 3% to 5%, and Ministral-3-14B from 0% to 2%; under ReAct, Ministral-3-14B improves from 2% to 4%.
We acknowledge that CaseAcc remains zero on DeepPlanning-Travel in our experiments. This benchmark imposes tightly coupled temporal, spatial, and budget constraints, where a single violation can invalidate the entire plan; its all-or-nothing CaseAcc is therefore substantially stricter than partial plan-quality metrics. This difficulty is also reflected in the benchmark's evaluation of frontier models. Our claim is not that PGS resolves all global planning constraints, but that it improves evidence assimilation and action selection, leading to measurable gains in plan quality and end-to-end success on TravelPlanner.