- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces ExpGym to evaluate LLM agents under limited feedback budgets in hyperparameter tuning, multi-hop search, and evidence audit. It proposes PoolAct to coordinate multiple agents through shared feedback and exploration state. Experiments on six models show that feedback budgets affect performance and model rankings, while PoolAct reduces redundant requests and improves results.
Critical strengths
- expgym provides a framework for evaluating agents under limited feedback budgets across three different tasks.
- poolact consistently improves evidence-audit performance across all six models under both budget settings.
- The component ablations provide useful support for the benefits of feedback sharing and coordinated action selection.
Critical weaknesses
- The paper does not compare poolact with a single agent given the same total feedback budget, leaving unclear when multi-agent coordination is preferable to increasing a single agent’s budget.
- Agents are not told the budget ceiling or per-call costs before their first action, limiting budget-aware planning under tight constraints.
- The decision lock serializes reasoning and action selection across agents, allowing only tool execution to run in parallel. This may increase latency and limit parallel speedups.
Questions for authors
- How does PoolAct compare with a single agent given the same total feedback budget and a comparable overall step limit?
- What fraction of search instances are solvable by an ideal retrieval strategy within the 900-second Tight Budget under the current withholding rule?
- How do the main results change when the initial prompt explicitly states the budget ceiling and estimated per-call costs?
2 comments
Reply to reviewer z32G (Part 2/2)
W3: The decision lock serializes reasoning and action selection across agents, allowing only tool execution to run in parallel. This may increase latency and limit parallel speedups.
R4: Thank you for this question. The lock covers only graph reading, action selection and request registration, and it is released before tool execution. In the matched NASBench-101 ablation in Section 5.5, both variants retain the shared cache and exploration graph:
| Matched lock ablation | Gap Score | Tokens (K) | Feedback charge (s) |
|---|---|---|---|
| Graph, no decision lock | 87.68 | 464.88 | 120,312.8 |
| PoolAct, with decision lock | 96.04 | 410.95 | 110,056.0 |
Adding the decision lock gains 8.36 Gap points, uses 11.60% fewer tokens and requires 2.85 fewer hours of charged training work across the four members (8.53% less).
To measure parallel speedup with training waits included, we compared PoolAct with a single agent given the same total allowance (see W1/Q1). The four PoolAct members complete 30.07 hours of charged feedback in 8.63 hours, about 3.5 concurrent jobs on average, and finish 3.2× sooner than the single agent; all serialized model calls take 7.5% of that time. We have detailed the matched ablation in Appendix F.2 and the completion-time comparison in Appendix F.5 of the revised manuscript.
Thank you for helping us clarify these comparisons.
Reply to reviewer z32G (Part 1/2)
We thank the reviewer for the thoughtful comments on budget allocation and deployment costs. We provide detailed responses to your specific concerns below.
R1: Thank you for asking for this comparison. We added a direct comparison with Long N1: one GLM-5.3 agent receives four workers' combined feedback allowance and step cap (4B, 120 steps), while each PoolAct member receives B and 30 steps. Both methods use a common deadline fixed before execution:
Both methods are scored by their best completed, visible configuration, and the simulation includes recorded training durations and measured inference and coordination time. Over the full runs, Long N1 and PoolAct use 27.44 and 30.07 hours of charged feedback. PoolAct gains 8.63 points at the deadline and wins all nine pairs, while terminal quality is similar. Multi-agent coordination is therefore preferable when results are needed within a practical completion window. In workflows such as scientific discovery and system optimization, giving one agent several times longer for serial experiments can take days, and that time cannot be assumed available. We have added the detailed comparison to Appendix F.5, with a main-text summary in Section 5.5 of the revised manuscript.
R2: Thank you for this question. We verified complete-answer certificates for 22 of 73 questions (30.1%) within the 900-second Tight Budget. This is a constructive lower bound, not a solvability ceiling. We constructed evidence certificates offline and replayed their article sets through the released tool, verifying total costs strictly below 900 s. We have added the certificate construction and results to Appendix F.7 of the revised manuscript. Since set F1 also credits correct partial answers, the certificates give F1 lower bounds of 74.55 (Whois) and 45.85 (Whatis), well above current agent scores.
R3: We added the 900 s ceiling and the 280–320 s cost per new article to GLM-5.3's initial prompt under Tight Budget:
F1 increases by only 0.95 points across all 39 questions, suggesting that upfront cost information has little effect on performance in this experiment. As detailed in Appendix B.2, agents already learn the ceiling after the first paid call, since each paid observation reports the incurred cost and remaining budget. For HPO, exact per-call costs are not given in advance and depend on the configuration; as detailed in Appendix E.1, three configurations in one NASBench-201 task cost 3,641 s, 13,525 s and 9,682 s. We have added the prompts and results to Appendix F.8 of the revised manuscript.