- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper models agent workflows as budget-augmented stochastic Petri nets and introduces capacity curves to measure the minimum token budget required for a target success rate. It derives conditions under which budget constraints reverse configuration rankings and characterizes optimal budget-aware allocation in the model. An analysis of 8,190 published AppWorld trajectories shows substantial ranking changes under counterfactual budget caps, although adaptive budget-aware agents are not evaluated through new deployments.
Critical strengths
- Task-level token budgeting is a practically valuable research topic. The paper incorporates the remaining budget into the workflow state and uses capacity curves to characterize the budget needed for a target success rate, informing system selection and resource planning.
- The analysis is systematic, covering simplified-model calculations, cost reconstruction from published trajectories, ranking changes under budget constraints, failure types, and cascade strategies. These analyses yield several findings about the relationships among token budgets, accumulated context costs, and task completion.
Critical weaknesses
- The evaluation mainly applies budget cutoffs to existing trajectories from agents that were not informed of their remaining budget. An important missing comparison is to run agents with the remaining budget provided at each decision step, allowing them to adjust their plans and spending. This would help assess whether the offline results reflect performance under actual budget-aware execution.
- The main empirical findings come from a single benchmark, leaving their generalizability unclear. The ranking reversals and the reported 88% token-budget saving from configuration selection are demonstrated on AppWorld, whose code-execution interface, tool-output lengths, and history-trimming rules may substantially affect the costs of different architectures. The supplementary OpenManus analysis mainly examines cost-growth patterns rather than validating similar ranking reversals.
Questions for authors
- How do the capacity curves and rankings change when agents are informed of their remaining budget at each decision step and allowed to adapt their behavior?
- Do similar ranking reversals and token-budget savings occur on another benchmark with different tool interfaces and context-management rules?
2 comments
Dear Reviewer,
Thank you again for your great work on reviewing this paper submission to ICLR 2027.
The authors have now submitted their rebuttal. The open reviewer-author discussion phase will end soon, on November 18 at 11:59 PM AoE.
Please read the authors' responses, determine whether any further clarification is needed, and respond to the rebuttal at your earliest convenience. Please be sure to acknowledge the authors' responses and update your review accordingly.
Best regards, Your AC
Response to E3Ef: both gaps are real — what the paper does and does not show; the 88% re-checked on held-out tasks
Thank you for the constructive review. You identify the two gaps we most want to close, and we agree with both.
W1/Q1: agents told their remaining budget. We have not run this comparison, and it is the paper's main limitation: every empirical result concerns budget-blind configurations, as §6 says (l.326–328, 446–447; Limitations l.484–486). Within that scope the replay is exact, because a budget-blind agent's trajectory does not depend on B, but it does not tell us what an informed agent would do. The comparison we consider right uses one ReAct agent on AppWorld
test_normalin three arms at matched hard caps (around the crossing region of Fig. 2):Arm (b) is the idea of Budget Tracker (Liu et al., 2025), which shows search agents their remaining tool-call budget at every step and improves their budget scaling; we will cite it. Our own model predicts gains mainly near the onset: π⋆ gains nothing at the 0.90 target and 0.042 at B=14 (§5, l.310–315). We therefore claim no deployment benefit for budget-aware allocation; §5 is a normative model, and we label its allocation results as model-based. If we can complete this comparison before the discussion closes, we will post it here.
W2/Q2: a second benchmark. We agree, and
test_challengedoes not answer it: it shares AppWorld's interface and trimming rules, so it shows only that the reversal is not an artifact of one task split (Table M14: the top configuration has zero success below 29k tokens, the third returns 0.165 at 10k). OpenManus (E6) validates the cost reconstruction only. Our own E14 shows why your concern matters: removing AppWorld's history trimming moves the cost tail by orders of magnitude (Table M12). We will state the single-benchmark scope in the abstract and §6.The 88% figure, checked for selection effects (new). We selected on one random half of
test_normaland evaluated on the other (1,000 splits), and we selected ontest_normaland evaluated ontest_challenge:test_normal→test_challengetest_normalThe 88% sits where the pick's ceiling (0.339) barely exceeds the target. We will lead with about 90% at targets 0.10–0.20 and report these intervals.
Theory. Reviewers Q1wf and 6FPy found that Theorem 2 needs an extra hypothesis once costs are stochastic; we fix it (general response). The crossing condition (Thm 3), the inversion interval (Thm 1), and all model and replay numbers stand.
We would welcome your view on whether the three-arm design above is the comparison you had in mind.