- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
EvoPlanner separates what a user’s request requires from the evolving plan used to fulfill it. It maintains fixed evidence obligations, a revisable task DAG, and a persistent ledger of verified observations called Witnesses. This allows useful evidence from partially completed tasks to survive task replacement and guide subsequent execution. On StableToolBench’s 765 solvable queries, the system improves macro-average SoPR over the authors’ VeriMap adapters by 10.8 and 17.0 percentage points with two backbones. Ablations and a repair-budget study provide additional evidence for the integrated framework.
Critical strengths
A useful separation of requirements, execution, and evidence. Task success is often too coarse to represent partial progress. EvoPlanner makes this distinction explicit and provides a concrete way to preserve and reuse supported information after replanning. This is a meaningful contribution to the design of tool-using agents.
Substantial gains across both evaluated backbones. EvoPlanner outperforms the corresponding local CoT and VeriMap configurations on all six subsets. The improvements are large overall, and disabling graph revision reduces I3 Inst performance by 9.8 points. These results make the integrated approach promising, even though the source of the gains remains uncertain.
Critical weaknesses
The central contribution is not isolated experimentally. Removing the evidence layer also removes dynamic repair, so this ablation cannot establish the value of persistent support links. There is no direct measurement of evidence reuse or support accuracy. Furthermore, equal tool-call ceilings do not control model computation, and the VeriMap adapter replaces generated Python verification with LLM checks. Consequently, the results do not establish whether the proposed representation improves over a faithful baseline given comparable inference resources.
The evaluation leaves important uncertainty about reproducibility and generalization. The component studies use only 61 development-exposed queries, with one generated answer per query and condition. Repeated judging does not measure agent variability. Section 5 also mentions a different judge for the controlled study without clearly identifying it, while Appendix A.3 leaves historical model routing and auxiliary-component settings unspecified. These gaps make it difficult to determine exactly what produced the reported improvements.
Questions for authors
-
Which models and auxiliary components were used in each reported configuration? Please also clarify the judge used for the component studies and how the 61 I3 Inst queries influenced development.
-
Can you isolate persistent Witness links while retaining the same repair, verification, and synthesis components—for example, by comparing against a system that preserves raw observations but reconstructs their requirement links after revision? How often does retained evidence actually enable successful continuation?
-
What are the token and model-call costs of the local baselines? Does the advantage remain against a faithful VeriMap implementation or a retry baseline with comparable inference cost?