- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces COPA, a framework for reducing repeated model calls in tool-using agents. It has two paths. The first compiles a one-shot plan into an executable workflow, using tool contracts to repair reference and typing mismatches before deterministic execution. The second extracts complete candidate answers already present in a model response, checks their feasibility, and selects an answer without further generation. Experiments on DeepPlanning Shopping, Travel and AppWorld report competitive or better success with substantially fewer model tokens. The frozen-response experiments examine the contributions of plan repair and candidate processing while keeping the generated responses fixed. The paper’s main finding is that some apparent planning failures can be resolved by improving execution and answer return without asking the model to generate more.
Critical strengths
The frozen-response experiments are the strongest part of the paper for me. Applying different repairs and answer readers to the same saved generations makes the contribution of these components much easier to assess. The reported increase of 40 executable plans, and the improvement from 4/96 to 31/96 correct answers in the Qwen setting, provide concrete evidence that useful solutions can already be present in a response but fail to reach execution or the final answer. These experiments support a specific explanation for the gains, which I find more convincing than an end-to-end comparison alone.
The approach addresses a practical source of overhead in tool agents. Contract-aware repair and candidate validation are explicit operations, and neither requires another model call. The shared-tool comparisons also gives useful evidence that large token savings are possible without a corresponding loss in task success or a large increase in tool calls. I particularly like the distinction between a response that contains no solution and one that contains a feasible solution but fails to return it. That distinction has implications for how agent failures should be evaluated.
Critical weaknesses
My main reservation concerns the scope of the efficiency comparisons. The shared-tool Shopping results are useful, but Travel lacks a direct comparison with an interactive baseline using the same scheduler and tools. Some comparisons with LLMCompiler and PIVOT also use known token lower bounds rather than complete token accounting. This does not by itself invalidate the savings, but it makes the different comparisons harder to interpret together. Token usage also leaves out the computational cost of domain operations such as the coupon-aware search in choose. I would like a clearer account of which efficiency claims are supported by matched measurements, and whether the savings carry through to end-to-end runtime.
The method’s applicability depends heavily on well-specified interfaces and the availability of useful domain computation. The compiler repairs representation-level problems, but a plan with valid types and references can still use a parameter incorrectly. The answer processor also requires a complete candidate to already exist in the response. These limitations are acknowledged, so I do not view them as a fundamental flaw. However, they leave the boundary of the contribution somewhat narrow: the evidence is strongest for structured environments where contracts and feasibility checks are already reliable. More detail on failures that remain after compilation and candidate processing would help establish where an interactive agent is still needed.
Questions for authors
Could you provide a compact comparison table showing which tools, backbones and generation settings are shared across methods? Please distinguish measured token totals from lower bounds. For Travel, is there a matched ReAct or CodeAct result using the same scheduler and tools? This would help clarify which conclusions can be drawn from each comparison
How often does the candidate reader encounter contradictory or misleading candidates in the reasoning text? Could you report cases where validation or selection produces an incorrect result, as well as cases where no complete candidate is available? I would like to understand the failure rate of this component alongside the recovery gains.
Could you report end-to-end latency and the non-LLM computation costs, seperately from model tokens? In particular, how much time is spent in domain operations such as choose, and do the main efficiency conclusions remain when this work is included?
4 comments
Follow-Up to Your Review (Second Reminder)
Dear Reviewer Rm9W,
Thank you again for the time and care you put into reviewing our work!
We know this is a busy period, so we wanted to follow up briefly on our point-by-point responses and the revised manuscript, where we highlighted the changes for easy reference.
If you have a chance to look over our responses, we would appreciate hearing whether they address your main concerns. If anything remains unclear, we would be happy to clarify it here.
Thanks again for your support!
Sincerely yours,
The Authors of Submission 5204
Follow-Up to Your Review (Third Reminder)
Dear Reviewer Rm9W,
We are following up on our point-by-point responses and the revised manuscript, which we posted several days ago. We believe these materials further address your concerns and highlight the changes we made in response to your feedback.
As the end of the Author-Reviewer Discussion phase is fast approaching, please let us know whether our responses address your main concerns or if any points still need clarification. We are available to discuss them here.
Thank you again for your time and effort devoted in this review!
Best,
The Authors of Submission 5204
Follow-Up to Your Review (Fourth Reminder)
Dear Reviewer Rm9W,
As the discussion period draws to a close, we would like to briefly follow up on a point you raised in your review.
We appreciate your suggestion to compare COPA with interactive baselines on Travel using the same scheduler and tools. In the matched comparison, COPA and ReAct each solved 83 of 120 tasks, while CodeAct solved 82. COPA used 54.0% fewer tokens than ReAct and 61.1% fewer than CodeAct. The rebuttal reports the full token totals for all three methods under these shared settings.
If any part of our response would benefit from clarification, we would be happy to discuss it here. Thank you again for your time and feedback.
Best,
The Authors of Submission 5204
Follow-Up to Your Review
Dear Reviewer
Rm9W,Thank you again for the time and effort you put into reviewing our work!
We understand that this might be a super busy period for you. We wanted to briefly let you know that we have posted point-by-point responses to your questions and concerns and uploaded a revised manuscript with the changes highlighted.
Whenever you have a chance, we would be grateful for your feedback on whether the responses address your concerns. We would be happy to clarify anything that remains open.
Thank you!
Best,
The Authors of Submission
5204