- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Acquire2Act isolates the acquisition-to-execution boundary: controllers choose ACQUIRE or ADVANCE under controlled information states on 200 tasks, with fixed fresh executors. Controllers respond to new and restored facts, yet often advance prematurely or keep acquiring after construction-defined complete information.
Critical strengths
The measurement contract cleanly separates information control from downstream execution, and the matched restoration experiment is a strong intervention for distinguishing responsiveness from appropriate exit. Trajectory diagnostics (local target completion vs. sufficiency, avoidable-target persistence) go beyond final task success alone.
Critical weaknesses
Thin related works Haven't seen a paper with this little related works......
Lack of human verification Use LLM to judge LLM, and no human verfication.
Questions for authors
Please improve the layout when you resubmit.
2 comments
Thank you for the detailed response. I will keep my score.
Response to Reviewer Rbf3
Thank you for the feedback. We address the literature coverage, human verification, and layout in the revised manuscript.
1. Related work
We have expanded the literature coverage. Section 2 now develops four strands: clarification and underspecification; information acquisition and sufficient evidence; clarification timing and stopping decisions; and agent execution and process evaluation. It explains connections to and differences from nearby studies, including SAGE-Agent, The Missing Complement, and Ask Early, Ask Late, Ask Right.
The Introduction and Related Work clarify our focus: the decision reliability of LLM agent controllers under information uncertainty, jointly examining acquisition-versus-advancement choices, acquisition targets, and policy updates under controlled information states and fixed execution conditions. We also added relevant citations throughout the Methods, analyses, Discussion, and appendices for task sources, user simulation, model-assisted evaluation, behavioral testing, and statistical methods. The bibliography has expanded from 9 to 30 entries; the main text remains within 10 pages.
2. Human verification
Primary task success uses domain task evaluators under the fixed scoring protocol, not an LLM judge. Controlled information construction and execution-based screening support the evaluation materials. Task and information construction also underwent a human audit of 18 tasks and 84 applicable judgments per return, reported in Appendix C.2.
LLM ratings support further behavioral analysis of timing, target alignment, handoff executability, and avoidable redundancy. The existing protocol uses two separate model evaluations blind to model identity and task outcomes, followed by arbitration of disagreements.
We added human validation on a stratified sample of 120 states, covering all three domains, both actions, and all eight configurations. After two rounds of human initial review and arbitration, exact agreement between final human labels and original model labels is:
The last two comparisons concern acquisition states. Appendix C.5, Table 17 (p. 23), reports the procedure and results, with pointers in Section 3.3 and an updated Discussion.
3. Layout
We revised the overview and its placement. Figure 1 now appears at the top of page 2, presenting task construction, controller evaluation, and main results as three full-width rows. The result panel uses vertical bars, with larger text and nodes at manuscript scale; its caption was shortened while protocol details remain in Section 3.
Table 1 now follows the corresponding Section 4.1 result discussion. Figures 3 and 4 appear near the complete- and partial-restoration analyses on pages 8 and 9. We removed excess bottom whitespace from the statistical figure displays and standardized figure/table caption spacing, retaining table captions above and figure captions below. The revised manuscript has 10 main-text pages; the reported task results are unchanged.