- Rating
- 2
- Confidence
- 3
- Informativeness
- 3(+1)
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper studies when solving a task requires both acquiring external information and using enhanced computation.It introduces four matched solver classes corresponding to the availability of these two resources and characterizes necessity through their optimal achievable risks.
The main theoretical contributions include an exact frontier relating retained information,query budget,and acquisition coverage;a distinction between acquiring the relevant records and identifying the correct answer; and computational lower bounds for tasks involving externally acquired information.Experiments on controlled tasks,FinQA,and HotpotQA illustrate how retrieval and answer processing can create different bottlenecks. Additional analyses examine evidence removal,resource costs,and transfer across models.
The results concern necessity relative to specified tasks,interfaces,and computational restrictions.They do not establish that adaptive agent workflows are universally necessary.
Critical strengths
1.The framework makes resource necessity precise and yields useful distinctions.
The paper compares optimal risks over explicitly defined solver classes,rather than interpreting the failure of an individual baseline as proof of necessity.The localization results connect retained information and query budget to achievable coverage.The distinction between coverage and answer identification is particularly informative: observing all potentially relevant records can still leave uncertainty when the solver cannot retain the target identity.These results provide a concrete account of how information acquisition and computation can impose separate limitations.
2.The empirical analysis includes meaningful controls and openly reports limitations.
The paper reports paired comparisons,cluster-aware uncertainty estimates,resource accounting,and evidence- removal interventions.Appendix H.2 also examines whether program-based improvements can be explained merely by invalid direct-answer formats.Importantly,Appendix J reports weak or negative transfer to a second model instead of presenting the primary model's results as universal.These analyses make the empirical evidence more informative and appropriately qualify its scope.
Critical weaknesses
1.The significance of the incremental scientific contribution remains insufficiently established.
The exact frontiers and separation examples are useful,but their relationship to prior work on bounded optimality and computationally constrained information needs sharper articulation.The controlled constructions establish conditional necessity under particular restrictions.The natural-task experiments compare implemented workflows and primarily illustrate the proposed perspective.
The evidence for additional practical diagnostic value is also limited.In Appendix H.4,diagnosis-assisted selection and development search select the same tested configurations.Appendix J further shows that the main empirical patterns depend substantially on the model and its ability to use the interface.These observations do not invalidate the theory,but they leave me uncertain whether the combined contribution provides sufficiently substantial new knowledge for acceptance.
A clearer account of the principal new result,its distinction from the closest prior frameworks,and the intended evidential role of the natural-task experiments could materially improve my assessment.
2.Reproducibility-critical details of the natural-task experiments are not sufficiently specified in the supplied manuscript.
The reported comparisons depend on retrieval construction,controller prompts,query updates and stopping rules,answer prompts,decoding settings,and the program language and executor.The manuscript provides substantial aggregate results,but the supplied document does not give enough detail to reconstruct these procedures reliably.
This matters because program-based processing produces large improvements for the primary model,while the second model frequently produces invalid programs.Without the exact interface and execution rules,it is difficult to assess how much the observed behavior depends on those implementation choices.
Questions for authors
1.What is the principal new scientific result relative to the closest work on bounded optimality and computationally constrained information?Please distinguish the contribution of the organizing framework from that of the exact frontier and separation theorems
2.Where can the complete natural-task protocols be found?In particular,please identify the prompts, retrieval and record-construction rules,adaptive query and stopping procedures,decoding settings, program grammar and executor,and evaluated sample identifiers.
3.What practical diagnostic benefit is supported beyond the illustrative comparisons?Given the identical selected configurations in Appendix H.4 and the model-dependent results in Appendix J,please clarify whether the intended contribution is primarily explanatory organization,improved configuration selection, reduced search effort,or another measurable benefit.
8 comments
Response to W1 (Part 2):
To test practical diagnostic value prospectively, we completed a new Qwen2.5-14B-Instruct study with three 60-question calibration sets and one 600-question held-out set per task. We froze the recommendations before evaluation. All three calibrations selected program readout at B4 for FinQA and increased acquisition from B4 to B8 with direct readout for HotpotQA, relative to B4 direct answering. The measured gains were 22.17 percentage points on FinQA, with a 95% cluster interval of [17.54, 27.27], and 6.17 points on HotpotQA, with [3.57, 8.81]. Both recommendations achieved the largest measured gains among the prespecified single-resource interventions. The six-candidate selection comparison used equal logical model-call allowances and matched direct search in average held-out score, with diagnosis additionally using calibration support annotations. The practical benefit supported here is the ability to formulate resource interventions before evaluation and test their predicted effects on held-out cases.
Response to W2 (Part 1):
We have prepared an indexed reproduction specification and verified the documented procedures against the saved original runs. The five groups covered are main FinQA, main HotpotQA, Mistral FinQA, Mistral HotpotQA, and FinQA supporting-record replacement with zero added distractors. We took this approach because assessing processing gains requires a clear account of how evidence reaches the model, how the model uses that evidence, and how answers are evaluated.
Our experimental design compares readouts on the same acquired history to examine changes in evidence use. We have organized the reproduction details around that comparison, from record construction and acquisition to answer generation and scoring. The documentation brings together the evaluated question IDs and dataset revisions, exact controller and answer prompts, message assembly, decoding settings, and program execution rules. The file index identifies the corresponding source files and recorded runtime configurations. The main cohorts contain 1,000 questions per task, and the replacement cohort contains 961 FinQA questions selected by the recorded fragment-matching rule.
For acquisition, FinQA records retain the supplied text and table headers, while HotpotQA records retain article titles and sentence indices. Records are split into chunks of at most 256 whitespace-delimited words. Fixed retrieval applies BM25 with k1 = 1.2 and b = 0.75 to the original question within the supplied candidate pool, with source order breaking ties. Adaptive acquisition starts with up to two chunks and permits up to three controller calls. Each call receives the question, accumulated observations, and remaining budget, then returns a query or a stop action. Each new query ranks the full candidate pool using the same BM25 rule and tie-breaking order. Previously acquired chunks are then excluded, and up to two highest-ranked unseen chunks are appended to the history within an eight-chunk total budget. Stopping occurs after an explicit stop, invalid or truncated control output, a search adding zero chunks, candidate exhaustion, or completion of the call or chunk budget.
For processing, each readout receives the history produced by the designated acquisition procedure. Draft-enhanced readout appends the generated draft verbatim as an assistant message, followed by the final-format instruction. The FinQA program readout accepts a JSON program comprising at most 20 flat, two-argument operations from ten arithmetic, comparison, and table-aggregation operations. Operands can be numeric literals, constants, or references to earlier results. Table operations access only rows returned as complete single chunks with structured row metadata in the acquired history. The final operation supplies the answer. The executor validates grammar, references, visible-row access, and numerical bounds before applying the official FinQA arithmetic semantics. The scoring calls use an exponent bound of 16. Final format errors, truncation, and execution failures remain in the full evaluation denominator and receive incorrect scores.
These execution details are particularly relevant to the concern about model-dependent program gains. The prompt instructs the model to select visible values, and the executor checks arithmetic and table access. The original Mistral comparisons use the corresponding Qwen task instructions, program grammar, executor, and scoring rules, while Mistral generates the adaptive controller outputs. Reporting valid-output rates alongside answer quality allows the comparison to account explicitly for interface use. This provides the procedural basis for interpreting the results in Appendix J.
Response to W2 (Part 2):
We also checked the effective generation settings against the runtime records. The runs use BF16, temperature 0, seed 1729, and each model's native chat template. The main Qwen context limit is 12,288 tokens, and the Mistral context limit is 32,768. Controller and draft limits are 192 and 512 generated tokens. Final limits are 2,048 for FinQA and 512 for HotpotQA. The reproduction specification records the exact prompt strings and continuation instructions together with software versions and model and tokenizer asset hashes.
To verify these descriptions against the reported experiments, we used the preserved model responses to replay all 65,962 requests across these five original experiment groups. The offline replay reconstructed messages, acquisition histories, query updates, and stopping events. The request total includes intermediate controller and draft calls. We reran the original parsing and execution procedures with the official scoring functions on all 43,844 final outputs. Every per-question score and validity flag matched the original analysis, as did the model-call, returned-block, and input/output-token counts. This agreement connects the documented procedures to the reported natural-task results.
Response to Q1:
The principal scientific result is to establish task structures under which an agent's acquisition and computational capabilities are jointly necessary, and to explain why the same necessary configuration can require different resource investments. The four-class framework defines the resource comparison, while the exact frontier and separation theorems establish the conditions under which that comparison yields joint necessity.
We developed the framework by combining the program-level perspective of bounded optimality with the question of which evidence a resource-constrained solver can acquire and use. The task, interface, and query budget are fixed across the four classes, and each class may redesign the complete acquisition, stopping, and answering procedure. Relative to the closest perspectives discussed in W1, this formulation makes computation relevant both to the observations that enter the history and to the prediction made from those observations.
The structural results provide the guarantees for this comparison. In localization, the exact coverage and terminally relaxed risk frontiers quantify how retained information and query budget constrain acquisition across all permitted encoders, randomized adaptive policies, and stopping rules. Combined with the passive-risk bound and an explicit successful agent, these frontiers establish a parameter region where both capabilities are necessary. The processing construction establishes joint necessity even when the acquired history contains all raw evidence, because the stated depth restriction limits how the evidence can affect the answer. These two results explain why acquiring the target record and processing complete evidence can lead to the same necessary configuration while calling for different interventions. Our response to W1 gives the precise frontiers, separation conditions, and constructions, together with the comparison to Russell and Subramanian (1995) and Xu et al. (2020).
Response to Q2 (Part 1):
The revised manuscript and executable reproduction materials are available in our anonymous repository. We have also update the manuscript on the website. The original natural-task protocols are consolidated in Appendix F.1 through F.7, and the additional prospective diagnosis experiment is documented in the diagnosis directory. We have organized these materials so that readers can follow each comparison from the evaluated questions and acquired records to the generated answers, configuration choices, and reported statistics.
Appendix F.1 specifies the dataset versions and evaluated cohorts. Appendix F.2 describes evidence-record construction, BM25 retrieval, adaptive query updates, and every stopping condition. Appendix F.3 explains how controller, draft, and final-answer messages are assembled. Appendix F.4 and Table F.1 give the inference settings and generation limits. Appendix F.5 specifies the program grammar, executor checks, output parsing, and scoring rules. Appendices F.6 and F.7 describe the paired statistical comparisons and resource accounting.
The repository includes the verbatim instructions in docs/PROMPTS.md, exact message templates in metadata/prompts_exact.json, evaluated question identifiers and shard assignments in metadata/sample_ids.json, and recorded runtime settings and model asset hashes in metadata/runtime_settings.json. The repository README provides commands for preparing the public data, running the original configurations, and replaying and scoring complete outputs. FILE_INDEX.md connects these files to the corresponding appendix sections and manuscript comparisons.
We thank you for the careful review and constructive questions about the scientific contribution, reproducibility, and practical value of our framework. Your comments helped us clarify what the joint comparison establishes, provide the implementation details needed for reproduction, and explain the diagnostic benefits demonstrated by the experiments.
Response to W1 (Part 1):
The principal scientific contribution is to identify task structures that make an agent's acquisition and computational capabilities jointly necessary, and to establish how those structures lead to different resource requirements. We developed the four-class comparison to ask which simpler configurations can meet a target risk when each alternative may redesign the complete acquisition, stopping, and answering procedure.
Bounded optimality evaluates the best program implementable on a given architecture in a task environment (Russell and Subramanian, 1995). We build on this program-level comparison by varying acquisition access and computational capability while fixing the task, interface, and query budget. Predictive V-information measures the prediction benefit of supplied observations within a restricted predictive family (Xu et al., 2020). Our analysis also treats the observations available to the solver as an outcome of resource-constrained acquisition. Computation can determine which records enter the history as well as how the solver uses those records. The four-class framework organizes this comparison, and the structural theorems establish concrete conditions for joint necessity.
The localization frontier makes this distinction quantitative. Consider M independent fair record bits and an initial request, independent of those bits, specifying a uniform target address. All later access to the request passes through a b-bit summary, and acquisition uses at most B queries. Theorems 1 and 2 give the exact optimal coverage and terminally relaxed risk
α=min{1,2bB/M},Rˉ10=21(1−α)≤R10.
The relaxation grants the final rule unrestricted computation and the full acquired history, including the original request. The frontier covers every permitted encoder, randomized adaptive acquisition policy, and stopping rule, and a grouping construction attains the bound. Increasing the retained summary by one bit or doubling the query budget yields the same optimal coverage under this model. Combining the frontier with passive risk 1/2 and an agent that retains the target address and reads the corresponding record establishes joint necessity whenever B≥1, 0≤η<1/2, and 2bB/M<1−2η. The framework specifies the comparison, while these results identify a parameter region in which the full agent configuration is uniquely sufficient.
We then examine why the same necessary configuration can require a different investment. In Section 3.3, two fixed queries acquire all raw bits. Full-data simulation transfers the depth restriction to every allowed interactive solver, giving Rˉ10=0<R10=1/2 under the stated separation conditions. An agent with sufficient depth combines the bits and achieves zero risk. Here additional processing addresses the remaining limitation, whereas the localization frontier identifies a limitation in acquiring the target record. Appendix D.4 further shows why coverage and answer identification require separate treatment. With two addresses and a zero-bit summary, increasing the query budget from one to two achieves complete coverage while actual optimal risk remains 1/4. Retaining one address bit instead permits zero risk with one query. Together, these results connect agent irreplaceability to specific choices about retained context, acquisition, and processing.
We designed the natural-task experiments to examine these intervention predictions in implemented language-model workflows. Fixed-history comparisons test changes in evidence use, support replacement tests sensitivity to available evidence, and resource accounting measures the associated costs. Appendix J makes model and interface competence part of this empirical assessment. Our added controlled parity experiments with Mistral separately measure successful execution and answer correctness, helping distinguish access to a processing capability from successful use of that capability.