acceptodds
Under review as a conference paper at ICLR 2027

Acquire2Act: Evaluating Information-to-Action Boundaries in LLM Agents

Abstract

An agent that gathers information must decide when to stop gathering and start executing. Does its control policy adapt appropriately as evidence becomes available? We study this question with Acquire2Act, a controlled evaluation of the acquisition-to-execution boundary on 200 application, data-science, and software-engineering tasks. Acquire2Act varies source-grounded information while fixing execution infrastructure, exposing acquisition targets, verified facts, and advancement decisions. Eight controller configurations achieve 60.5–70.0% task success, versus 86.5% for a privileged information-need policy. Natural trajectories show strong responses to information progress, but reports of local target completion can coexist with premature advancement, and already avoidable acquisition targets can persist. In a matched restoration experiment on 130 tasks, supplying the complete construction-defined fact set increases advancement by 48.5 and 36.9 percentage points for two controllers. Nevertheless, they continue acquiring information on 40.8% and 34.6% of complete-information inputs. A separate ablation does not find a consistent positive effect of an explicit completion-status field. These results distinguish information responsiveness from appropriate boundary control: controllers use new evidence, yet neither observed progress nor restored facts ensure reliable acquisition exit under the fixed task and execution contract.

Primary area:
datasets and benchmarks
Keywords:
agent evaluation, information sufficiency, tool-using agents, uncertainty, benchmark, decision making

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_Rbf3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Acquire2Act isolates the acquisition-to-execution boundary: controllers choose ACQUIRE or ADVANCE under controlled information states on 200 tasks, with fixed fresh executors. Controllers respond to new and restored facts, yet often advance prematurely or keep acquiring after construction-defined complete information.

Critical strengths

The measurement contract cleanly separates information control from downstream execution, and the matched restoration experiment is a strong intervention for distinguishing responsiveness from appropriate exit. Trajectory diagnostics (local target completion vs. sufficiency, avoidable-target persistence) go beyond final task success alone.

Critical weaknesses

Thin related works Haven't seen a paper with this little related works......

Lack of human verification Use LLM to judge LLM, and no human verfication.

Questions for authors

Please improve the layout when you resubmit.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Rbf3

Thank you for the feedback. We address the literature coverage, human verification, and layout in the revised manuscript.

1. Related work

We have expanded the literature coverage. Section 2 now develops four strands: clarification and underspecification; information acquisition and sufficient evidence; clarification timing and stopping decisions; and agent execution and process evaluation. It explains connections to and differences from nearby studies, including SAGE-Agent, The Missing Complement, and Ask Early, Ask Late, Ask Right.

The Introduction and Related Work clarify our focus: the decision reliability of LLM agent controllers under information uncertainty, jointly examining acquisition-versus-advancement choices, acquisition targets, and policy updates under controlled information states and fixed execution conditions. We also added relevant citations throughout the Methods, analyses, Discussion, and appendices for task sources, user simulation, model-assisted evaluation, behavioral testing, and statistical methods. The bibliography has expanded from 9 to 30 entries; the main text remains within 10 pages.

2. Human verification

Primary task success uses domain task evaluators under the fixed scoring protocol, not an LLM judge. Controlled information construction and execution-based screening support the evaluation materials. Task and information construction also underwent a human audit of 18 tasks and 84 applicable judgments per return, reported in Appendix C.2.

LLM ratings support further behavioral analysis of timing, target alignment, handoff executability, and avoidable redundancy. The existing protocol uses two separate model evaluations blind to model identity and task outcomes, followed by arbitration of disagreements.

We added human validation on a stratified sample of 120 states, covering all three domains, both actions, and all eight configurations. After two rounds of human initial review and arbitration, exact agreement between final human labels and original model labels is:

DimensionExact agreement
Timing111/120 (92.5%)
Target alignment113/120 (94.2%)
Handoff executability55/60 (91.7%)
Avoidable redundancy53/60 (88.3%)

The last two comparisons concern acquisition states. Appendix C.5, Table 17 (p. 23), reports the procedure and results, with pointers in Section 3.3 and an updated Discussion.

3. Layout

We revised the overview and its placement. Figure 1 now appears at the top of page 2, presenting task construction, controller evaluation, and main results as three full-width rows. The result panel uses vertical bars, with larger text and nodes at manuscript scale; its caption was shortened while protocol details remain in Section 3.

Table 1 now follows the corresponding Section 4.1 result discussion. Figures 3 and 4 appear near the complete- and partial-restoration analyses on pages 8 and 9. We removed excess bottom whitespace from the statistical figure displays and standardized figure/table caption spacing, retaining table captions above and figure captions below. The revised manuscript has 10 main-text pages; the reported task results are unchanged.

Thank you for the detailed response. I will keep my score.

Reviewer_MPRuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Acquire2Act, an evaluation framework designed to investigate when LLM agents should stop acquiring information and proceed to task execution. The framework constructs controlled information states across 200 tasks from AppWorld, DA-Code, and SWE-Lancer, and evaluates eight controller configurations under a fixed execution contract. Through natural trajectory analysis, matched information-restoration experiments, and a completion-status ablation, the authors examine how agents respond to information changes and whether their acquisition-to-execution decisions are appropriate. The results show that although controllers respond substantially to newly available information, they frequently advance prematurely or continue acquiring information even when the construction-defined information requirements have been satisfied. The paper highlights an important distinction between information responsiveness and reliable boundary control.

Critical strengths

Meaningful research problem. The paper investigates an important aspect of agent reliability that is not adequately captured by conventional task-success metrics. Distinguishing information acquisition from execution decisions provides a useful perspective for understanding agent behavior. Well-structured experimental framework. The separation between controllers and executors helps isolate acquisition decisions from downstream execution. The controlled information states and matched restoration experiments provide stronger evidence than purely observational trajectory analysis. Informative empirical findings. The paper identifies several interesting failure patterns, including premature advancement following locally successful acquisition, persistent unnecessary information gathering, and continued acquisition under complete-information conditions. These findings provide useful insights into the limitations of existing LLM controllers. Careful reporting and analysis. The authors report confidence intervals, distinguish observational associations from controlled interventions, and acknowledge limitations related to task construction, automated evaluation, and experimental generalizability.

Critical weaknesses

Limited methodological novelty. The primary contribution is an evaluation framework rather than a new algorithm or control mechanism. Although the research problem is relevant, the paper does not sufficiently demonstrate how its evaluation methodology advances beyond existing work on clarification, information sufficiency, and agent stopping decisions. Insufficient validation of evaluation judgments. The behavioral analysis relies heavily on model-assisted reviewers from the same model family. The construction audit covers only 18 tasks, and the reported Cohen's kappa of 0.253 raises concerns about annotation reliability. Without stronger independent human validation, conclusions concerning premature advancement and unnecessary acquisition remain somewhat dependent on potentially imperfect judgments. Potential confounding factors in the controlled experiments. The information-restoration experiment changes both information content and input length. Without length-matched irrelevant-information controls, it is difficult to determine whether the observed advancement changes are driven specifically by improved information sufficiency. Furthermore, the main controlled experiments involve only two controller configurations, which limits the generalizability of the findings. Limited connection to downstream task performance. The controlled restoration experiment measures controller decisions without executing the resulting actions. Therefore, the paper does not establish whether improving boundary decisions would lead to higher task success rates or lower execution costs. Additionally, the privileged need-guided reference has access to information unavailable to ordinary controllers, making its performance gap difficult to interpret as a direct measure of controller deficiencies. Limited reproducibility and robustness evidence. The complete-chain repeatability analysis covers only nine tasks, and some repeated executions produce different outcomes. The benchmark also involves calibration-informed task selection and hosted model configurations, which may affect the stability and generalizability of the reported results.

Questions for authors

Can the authors provide independent human annotations for a representative subset of controller states and report agreement with the model-assisted judgments? How sensitive are the main conclusions to potential annotation errors? Could the authors introduce a length-matched irrelevant-information control in the restoration experiment to distinguish the effect of information sufficiency from the effect of increased context length? Would correcting premature advancement and unnecessary acquisition decisions improve actual task success rates or execution efficiency? Can the authors provide an end-to-end evaluation demonstrating the practical impact of more accurate boundary control? Why were only Sol/max and Grok/xhigh selected for the controlled restoration and completion-status experiments? Are similar behavioral patterns observed across the other controller configurations? How do the authors distinguish genuinely unnecessary information acquisition from reasonable additional verification when the construction-defined fact set may not cover every operational uncertainty encountered by an agent?

3 comments
AuthorsOpenReview ↗

Response to Reviewer MPRu (1/3)

Thank you for the detailed feedback. We have clarified the evaluation contribution, added human validation and random-label sensitivity analysis, and revised the presentation. Below we address the five weaknesses and five questions together where they concern the same evidence.

1. Evaluation contribution — Weakness 1

Acquire2Act contributes an evaluation methodology for LLM controller decision reliability under information uncertainty. Across executable tasks, controlled information states, and fixed execution conditions, it jointly examines whether to acquire or advance, what information to acquire, and how decisions change with new evidence.

We expanded the comparison with closely related work. SAGE-Agent selects clarification questions and execution under tool-parameter uncertainty; The Missing Complement studies state-conditioned sufficient-evidence recovery and downstream effects; Ask Early, Ask Late, Ask Right examines clarification timing and execution benefits. Acquire2Act makes acquisition targets and the acquisition-to-execution boundary separately inspectable through fixed executors and information-transfer rules, natural trajectories, and matched information restoration. These conditions let us distinguish information responsiveness from appropriate control.

The construction also supplies an execution-based foundation for this measurement. Source-grounded facts are grouped into functional information clusters. Retaining them or removing one or two qualified clusters, together with dependent facts, creates S0/S1/S2. Repeated fixed-executor runs qualify gaps whose deletion reduces task success or preserves success while increasing execution cost. Figure 1 and Section 3.1 now expose this construction pipeline; Sections 1–2 clarify the research focus and comparisons, and Appendix A.2 gives the screening criteria.

2. Validation of judgments — Weakness 2 and Question 1

Primary task success is computed by domain task evaluators, not an LLM judge. Model-assisted ratings diagnose controller behavior; task construction has its own human audit.

The original construction audit contains 18 tasks and 84 applicable judgments. Observed agreement is high: 76/84 judgments (90.5%). Cohen's κ = 0.253 is lower because most labels in both returns are “pass.” This concentration raises the chance-agreement baseline that κ subtracts, yielding a low chance-corrected value alongside high observed agreement. Consulting the third return for disagreements yields 79 passes, one failure, and four unresolved judgments. Appendix C.2–C.4 retains these outcomes and the post-audit task-exclusion analysis.

We added a stratified human check of 120 states from 103 tasks, covering all three domains, ACQUIRE/ADVANCE, and all eight configurations. After two rounds of human initial review and arbitration, agreement between final human and original model labels is:

DimensionExact agreement
Timing111/120 (92.5%)
Target alignment113/120 (94.2%)
Handoff executability55/60 (91.7%)
Avoidable redundancy53/60 (88.3%)

The last two dimensions concern acquisition states. Section 3.3 and Appendix C.5, Table 17 (p. 23), report the protocol and results.

We also added uniformly sampled, bidirectional random flips of 1,458 binary advancement-timing labels and 1,444 binary acquisition-redundancy labels. Each integer flip percentage from 0% to 50% uses 5,000 repetitions. We check two descriptions: most advances following resolved S2 acquisitions remain premature; and after a first fact stall, subsequent acquisition is more often avoidable when the preceding acquisition was already avoidable. At 25% random flips, both hold in 96.44% of repetitions; at 26%, retention is 94.70%. Under the 95% retention criterion, 25% is the largest tested integer percentage in the contiguous passing range from zero. Appendix C.6 (pp. 23–24), Tables 18–19, gives the protocol, distributions, and thresholds.

AuthorsOpenReview ↗

Response to Reviewer MPRu (2/3)

3. Input length and configuration selection — Weakness 3, Questions 2 and 4

Long-context degradation is not a sufficient explanation for the decision changes in C1. C1 restores the task's original missing facts while keeping the objective, orientation, instruction, tools, and empty history fixed. Across all 752 primary requests, median restoration increments are only 86 tokens for Sol and 94 for Grok. Each configuration has 129/130 tasks with increments of at most 500 tokens, and even the largest complete inputs contain just 2,250 and 2,602 tokens.

For comparison, RULER (Hsieh et al., 2024) evaluates performance across 4K–128K contexts and reports degradation as context length increases. Lost in the Middle (Liu et al.) examines how input length and the position of relevant information affect multi-document question answering and key-value retrieval. These findings do not establish that restoring a median of 86/94 task-relevant tokens in our short inputs explains the observed decision changes through length alone. Section 5.1 and Appendix B.3, Table 8 (p. 19), report our actual input scale.

Equal-length irrelevant padding is not a pure length control: it introduces distractor content. Shi et al. (ICML 2023), Large Language Models Can Be Easily Distracted by Irrelevant Context, show that even one irrelevant sentence can affect short-problem behavior. Padding therefore introduces an additional semantic intervention. Our comparison restores the task's original missing facts, whose execution relevance was established during construction. The control should be assessed against this information-restoration question, rather than assuming irrelevant text is behaviorally neutral.

The eight configurations form the primary evaluation. C1/C2 are targeted follow-ups contrasting two observed behavior profiles: Sol/max's pronounced avoidable long acquisition chains and Grok/xhigh's more frequent post-acquisition advancement and lighter stalling. They test responses under the same intervention protocol; the eight-configuration natural trajectories provide the broader behavioral observations. This selection rationale is now explicit in Section 5.1.

AuthorsOpenReview ↗

Response to Reviewer MPRu (3/3)

4. Connection to downstream performance — Weakness 4 and Question 3

The study includes complete acquisition-and-execution results on the same 200 tasks. Under the fixed execution infrastructure, the target-guided baseline succeeds on 173/200 (86.5%), versus 60.5%–70.0% for the eight configurations. It uses 128.7K provider tokens per task on average; the equal-weighted eight-configuration mean is 168.0K, 30.5% higher. Each configuration contributes 200 effective runs. Figure 1, Table 1 (p. 5), Appendix A.5 (p. 17), and Table 7 (p. 19) report these results.

The baseline is told what information to acquire; the executor must obtain the answers and complete the task. It receives functional descriptions of missing needs, requests targets in a preset order, and advances after completion reports. Ordinary executors select sources, retrieve facts, and perform the task. These complete-run results connect useful acquisition-target guidance to downstream performance.

Construction supplies a second layer of execution evidence. Standard screening requires at least two H+ successes in three runs; designated boundary cases require four in five. Removing a qualified cluster must either reduce success or preserve protected success while increasing execution cost. For the latter, successful H+ trials must have no higher mean query or token count, plus either at least one fewer query and a 20% query reduction, or a 20% token reduction (Section 3.1; Appendix A.2). Thus, tested gaps have execution-verified relevance.

5. Repeatability, calibration, and reproducibility — Weakness 5

We evaluated eight configurations on the same 200 tasks and conducted matched restoration on all 130 S1/S2 tasks. We also ran two complete chains for each of Sol/max and Grok/xhigh on nine preselected tasks, yielding 36 repeated-run outcomes. Task-level success/failure agreement is 6/9 for Sol and 7/9 for Grok. Sol succeeds on 5/9 and 6/9; Grok on 6/9 in both, with two tasks exchanging success and failure. Appendix C.1 and Table 13 (p. 21) report these outcomes.

Calibration serves the benchmark's design objective: making decision quality visible in task success and execution cost. We first verify executor capability, then examine the within-task relationship between decision quality and execution utility to select suitable tasks. The task set is frozen before formal evaluation. Section 3.1 and Appendix A.2 explain this purpose and the Terra/Sol calibration roles.

The paper provides model identifiers, reasoning settings, output budgets, prompts, and execution/retry rules (Appendix A.4, Table 4; Appendix D). Upon acceptance, we will immediately open-source the experimental code, evaluation configurations, and supporting analysis data and scripts, as stated in the reproducibility statement (p. 11).

6. Reasonable verification versus separate acquisition — Question 5

ADVANCE starts task execution with continued access to environment queries, code inspection, and implementation checks. Repeated H+ execution verifies the basis for entering that stage. The executor obtains implementation details as needed during execution; the controller uses separate acquisition to resolve information gaps that block this transition (Section 3.2).

For example, an email-scheduling H+ input already specifies eight email bindings, reply content, the 96-hour offset, and API parameters. Its instructions authorize connected-account operations and access to account information through Supervisor. Sol nevertheless requests Gmail credentials as a separate ACQUIRE target. The executor can obtain those credentials before calling the email API. This existing C1 example illustrates execution preparation being treated as a separate pre-execution requirement; Appendix B.3 (p. 20) presents it and the relevant protocol.

Comments on the paper

3 comments
Area_Chair_gieiOpenReview ↗

Review

The paper studies whether an agent’s control policy handles the boundary between gathering information and executing. It builds Acquire2Act on 200 tasks from AppWorld, DA-Code, and SWE-Lancer.

Strengths: A controller sees the task, verified facts, and acquisition history and chooses ACQUIRE or ADVANCE, while a fixed executor performs acquisitions and the final commit in fresh environments. The study covers three task substrates and eight controller configurations from several model families.

The matched restoration experiment provides clear evidence of information responsiveness. It changes the available facts while holding the task, instructions, and empty history fixed. The substantial advancement shifts, together with the partial-restoration conditions, show that controllers respond to the supplied information rather than simply following a fixed tendency to acquire or proceed.

weaknesses: The stopping judgments need stronger support. Continued acquisition after full restoration is not necessarily unnecessary. The probe does not test whether it improves success or cost, and some “required” information qualifies only because it makes execution more efficient.

The behavioral labels need independent validation. The same model family evaluates decisions and participates in the experiments. Agreement between model judgments does not establish accuracy, and the construction audit does not validate these labels.

The findings may be specific to acquisition-prone controllers. The controlled probes cover only Sol and Grok, which acquire relatively often. Sol also accounts for most transitions without new facts. Results from lower-acquisition controllers would help establish how broadly the findings apply.

Key controls and baselines are missing. Fact restoration changes both content and prompt length. Length-matched controls and simple stopping policies would clarify the effect and its practical value. Budget failures also need a clearer breakdown.

Questions: The paper should explain what it adds beyond CaRT and Sufficient Context, which already study stopping decisions and evidence sufficiency. How is the proposed work improving on these methods? After full restoration, how do success and total cost compare between immediate execution and further acquisition? Can you validate the behavioral labels with human reviewers and a different model family? Why were Sol and Grok selected, and do the findings hold for controllers that acquire less often?

AuthorsOpenReview ↗

Response to the Area Chair

Thank you for your comments. We respond below in the order of the points raised and link to our replies to Reviewer MPRu for detailed results.

1. Continued acquisition after full information restoration. Our construction process uses repeated runs to verify that the fixed executor can complete tasks from H+. It also tests how removing information affects success or cost. After ADVANCE, the executor can still query the environment, inspect code, and check implementation details. Details available during execution therefore do not need to be acquired separately before the controller hands over the task. Gathering these details in a separate earlier stage is the redundant preparation we discuss. See R1-C, Section 6, for the execution protocol and an example.

2. Whether further acquisition improves success or reduces cost. For success, H+ has already passed the repeated-execution success threshold, providing a tested basis for starting execution. The executor can also obtain implementation details during execution. For cost, a separate ACQUIRE adds controller calls and acquisition-executor work. Any later savings must cover this added cost before there is a net benefit. See R1-C, Section 4, for the success and cost comparison from complete runs.

3. Information selected for execution efficiency. We have two clear screening routes: removing information reduces task success, or retaining it substantially improves execution efficiency without reducing success. Both routes have strict, demanding admission criteria. A benefit alone is not enough for inclusion. See Appendix A.2 and R1-C, Section 4, for the detailed thresholds.

4. Independent validation of behavioral labels. We added human validation of 120 states covering all three domains, both actions, and all eight configurations. Agreement between the final human labels and the original model labels is reported in the table in R1-A, Section 2, and manuscript Table 17.

5. Controlled experiments on Sol and Grok. We selected these configurations to compare different behaviors: Sol tends to acquire too much information, while Grok advances more readily after acquisition and stalls less. Their mean acquisition counts are 2.42 and 0.89 per task. In this comparison, Grok is exactly the controller you ask for: it acquires less and is more inclined to advance. Describing both as having the same preference for acquisition misses this contrast in our design. See R1-B for the selection rationale.

6. Information content and input length. For Sol/Grok, inputs with missing information average 1,515/1,799 tokens, while complete inputs average 1,625/1,915 tokens. Full restoration adds 110/116 tokens on average; restoring one information cluster adds 76/80 tokens. These are small changes in short inputs. Adding irrelevant text to match length also introduces semantic distractions. See R1-B for the length analysis and supporting literature.

7. A finer breakdown of budget failures. Counting budget exhaustion as failure is appropriate because the task was not completed within the evaluation budget. We agree that this category deserves further behavioral analysis, including repeated attempts to obtain unavailable or unnecessary information and continued failure to advance. This is a useful direction for follow-up analysis. Table 1 reports the current budget-failure counts for each configuration.

8. Contribution beyond CaRT and Sufficient Context. CaRT studies how to learn when to stop; Sufficient Context studies whether context is sufficient to answer a question. Our broader goal is to study controller decisions under information uncertainty: whether to acquire or advance, what to acquire, and how to update decisions as new information arrives. Our method groups facts into functional information clusters, removes and restores them under controlled conditions, and screens tasks using execution results. This construction makes controller decision quality visible in task success and execution cost with a fixed executor. The construction pipeline is the core contribution of our evaluation method; see R1-A, Section 1. The requested complete-run results are in R1-C, Section 4.

Area_Chair_gieiOpenReview ↗

Reminder: Please respond to the author rebuttal

Dear Reviewers MPRu and Rbf3, the authors have responded to your reviews with a revised paper, including expanded related work and a new human validation of the behavioral labels, and the author-reviewer discussion ends soon. Please read their response, post a brief reply on whether your concerns are addressed, and update your scores if your assessment has changed. Thank you! AC.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.