acceptodds
Under review as a conference paper at ICLR 2027

When Are Agents Irreplaceable? A Task-Level Theory of Acquisition and Computation

Abstract

The success of an agent does not establish which capabilities the task requires. We introduce task-level resource diagnosis to determine when an agent is irreplaceable and when simpler configurations suffice. For each fixed-target task, we compare class-optimal risks with the interface and query budget held fixed across solver classes, allowing solvers in each class to redesign their solutions within the specified resource constraints. Exact evidence-coverage frontiers and processing risk bounds, together with successful agent constructions, establish conditions under which acquisition and enhanced computation are jointly necessary. Computation shapes both evidence acquisition and the answers derived from observations. Broader coverage need not reduce risk, and complete evidence can still require stronger computation. These mechanisms explain why tasks requiring the same agent capabilities can benefit from different resource investments. Controlled tasks test these predictions. Natural-language experiments show how interventions on acquisition and processing improve answer quality and affect resource use. The framework connects agent irreplaceability to task structure, identifying indispensable capabilities and guiding interventions on the limitations that prevent success.

Primary area:
learning theory
Keywords:
agentic inference, information acquisition, computational constraints, resource trade-offs, risk lower bounds

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3DPVOpenReview ↗
Rating
2
Confidence
3
Informativeness
3(+1)
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper studies when solving a task requires both acquiring external information and using enhanced computation.It introduces four matched solver classes corresponding to the availability of these two resources and characterizes necessity through their optimal achievable risks.

The main theoretical contributions include an exact frontier relating retained information,query budget,and acquisition coverage;a distinction between acquiring the relevant records and identifying the correct answer; and computational lower bounds for tasks involving externally acquired information.Experiments on controlled tasks,FinQA,and HotpotQA illustrate how retrieval and answer processing can create different bottlenecks. Additional analyses examine evidence removal,resource costs,and transfer across models.

The results concern necessity relative to specified tasks,interfaces,and computational restrictions.They do not establish that adaptive agent workflows are universally necessary.

Critical strengths

1.The framework makes resource necessity precise and yields useful distinctions.

The paper compares optimal risks over explicitly defined solver classes,rather than interpreting the failure of an individual baseline as proof of necessity.The localization results connect retained information and query budget to achievable coverage.The distinction between coverage and answer identification is particularly informative: observing all potentially relevant records can still leave uncertainty when the solver cannot retain the target identity.These results provide a concrete account of how information acquisition and computation can impose separate limitations.

2.The empirical analysis includes meaningful controls and openly reports limitations.

The paper reports paired comparisons,cluster-aware uncertainty estimates,resource accounting,and evidence- removal interventions.Appendix H.2 also examines whether program-based improvements can be explained merely by invalid direct-answer formats.Importantly,Appendix J reports weak or negative transfer to a second model instead of presenting the primary model's results as universal.These analyses make the empirical evidence more informative and appropriately qualify its scope.

Critical weaknesses

1.The significance of the incremental scientific contribution remains insufficiently established.

The exact frontiers and separation examples are useful,but their relationship to prior work on bounded optimality and computationally constrained information needs sharper articulation.The controlled constructions establish conditional necessity under particular restrictions.The natural-task experiments compare implemented workflows and primarily illustrate the proposed perspective.

The evidence for additional practical diagnostic value is also limited.In Appendix H.4,diagnosis-assisted selection and development search select the same tested configurations.Appendix J further shows that the main empirical patterns depend substantially on the model and its ability to use the interface.These observations do not invalidate the theory,but they leave me uncertain whether the combined contribution provides sufficiently substantial new knowledge for acceptance.

A clearer account of the principal new result,its distinction from the closest prior frameworks,and the intended evidential role of the natural-task experiments could materially improve my assessment.

2.Reproducibility-critical details of the natural-task experiments are not sufficiently specified in the supplied manuscript.

The reported comparisons depend on retrieval construction,controller prompts,query updates and stopping rules,answer prompts,decoding settings,and the program language and executor.The manuscript provides substantial aggregate results,but the supplied document does not give enough detail to reconstruct these procedures reliably.

This matters because program-based processing produces large improvements for the primary model,while the second model frequently produces invalid programs.Without the exact interface and execution rules,it is difficult to assess how much the observed behavior depends on those implementation choices.

Questions for authors

1.What is the principal new scientific result relative to the closest work on bounded optimality and computationally constrained information?Please distinguish the contribution of the organizing framework from that of the exact frontier and separation theorems

2.Where can the complete natural-task protocols be found?In particular,please identify the prompts, retrieval and record-construction rules,adaptive query and stopping procedures,decoding settings, program grammar and executor,and evaluated sample identifiers.

3.What practical diagnostic benefit is supported beyond the illustrative comparisons?Given the identical selected configurations in Appendix H.4 and the model-dependent results in Appendix J,please clarify whether the intended contribution is primarily explanatory organization,improved configuration selection, reduced search effort,or another measurable benefit.

8 comments

We thank you for the careful review and constructive questions about the scientific contribution, reproducibility, and practical value of our framework. Your comments helped us clarify what the joint comparison establishes, provide the implementation details needed for reproduction, and explain the diagnostic benefits demonstrated by the experiments.

Response to W1 (Part 1):

The principal scientific contribution is to identify task structures that make an agent's acquisition and computational capabilities jointly necessary, and to establish how those structures lead to different resource requirements. We developed the four-class comparison to ask which simpler configurations can meet a target risk when each alternative may redesign the complete acquisition, stopping, and answering procedure.

Bounded optimality evaluates the best program implementable on a given architecture in a task environment (Russell and Subramanian, 1995). We build on this program-level comparison by varying acquisition access and computational capability while fixing the task, interface, and query budget. Predictive V-information measures the prediction benefit of supplied observations within a restricted predictive family (Xu et al., 2020). Our analysis also treats the observations available to the solver as an outcome of resource-constrained acquisition. Computation can determine which records enter the history as well as how the solver uses those records. The four-class framework organizes this comparison, and the structural theorems establish concrete conditions for joint necessity.

The localization frontier makes this distinction quantitative. Consider M independent fair record bits and an initial request, independent of those bits, specifying a uniform target address. All later access to the request passes through a b-bit summary, and acquisition uses at most B queries. Theorems 1 and 2 give the exact optimal coverage and terminally relaxed risk

α=min⁡{1,2bB/M},Rˉ10=12(1−α)≤R10.\alpha=\min\{1,2^bB/M\},\qquad \bar R_{10}=\tfrac12(1-\alpha)\le R_{10}.

The relaxation grants the final rule unrestricted computation and the full acquired history, including the original request. The frontier covers every permitted encoder, randomized adaptive acquisition policy, and stopping rule, and a grouping construction attains the bound. Increasing the retained summary by one bit or doubling the query budget yields the same optimal coverage under this model. Combining the frontier with passive risk 1/2 and an agent that retains the target address and reads the corresponding record establishes joint necessity whenever B≥1B\ge1, 0≤η<1/20\le\eta<1/2, and 2bB/M<1−2η2^bB/M<1-2\eta. The framework specifies the comparison, while these results identify a parameter region in which the full agent configuration is uniquely sufficient.

We then examine why the same necessary configuration can require a different investment. In Section 3.3, two fixed queries acquire all raw bits. Full-data simulation transfers the depth restriction to every allowed interactive solver, giving Rˉ10=0<R10=1/2\bar R_{10}=0<R_{10}=1/2 under the stated separation conditions. An agent with sufficient depth combines the bits and achieves zero risk. Here additional processing addresses the remaining limitation, whereas the localization frontier identifies a limitation in acquiring the target record. Appendix D.4 further shows why coverage and answer identification require separate treatment. With two addresses and a zero-bit summary, increasing the query budget from one to two achieves complete coverage while actual optimal risk remains 1/4. Retaining one address bit instead permits zero risk with one query. Together, these results connect agent irreplaceability to specific choices about retained context, acquisition, and processing.

We designed the natural-task experiments to examine these intervention predictions in implemented language-model workflows. Fixed-history comparisons test changes in evidence use, support replacement tests sensitivity to available evidence, and resource accounting measures the associated costs. Appendix J makes model and interface competence part of this empirical assessment. Our added controlled parity experiments with Mistral separately measure successful execution and answer correctness, helping distinguish access to a processing capability from successful use of that capability.

Response to W1 (Part 2):

To test practical diagnostic value prospectively, we completed a new Qwen2.5-14B-Instruct study with three 60-question calibration sets and one 600-question held-out set per task. We froze the recommendations before evaluation. All three calibrations selected program readout at B4 for FinQA and increased acquisition from B4 to B8 with direct readout for HotpotQA, relative to B4 direct answering. The measured gains were 22.17 percentage points on FinQA, with a 95% cluster interval of [17.54, 27.27], and 6.17 points on HotpotQA, with [3.57, 8.81]. Both recommendations achieved the largest measured gains among the prespecified single-resource interventions. The six-candidate selection comparison used equal logical model-call allowances and matched direct search in average held-out score, with diagnosis additionally using calibration support annotations. The practical benefit supported here is the ability to formulate resource interventions before evaluation and test their predicted effects on held-out cases.

Response to W2 (Part 1):

We have prepared an indexed reproduction specification and verified the documented procedures against the saved original runs. The five groups covered are main FinQA, main HotpotQA, Mistral FinQA, Mistral HotpotQA, and FinQA supporting-record replacement with zero added distractors. We took this approach because assessing processing gains requires a clear account of how evidence reaches the model, how the model uses that evidence, and how answers are evaluated.

Our experimental design compares readouts on the same acquired history to examine changes in evidence use. We have organized the reproduction details around that comparison, from record construction and acquisition to answer generation and scoring. The documentation brings together the evaluated question IDs and dataset revisions, exact controller and answer prompts, message assembly, decoding settings, and program execution rules. The file index identifies the corresponding source files and recorded runtime configurations. The main cohorts contain 1,000 questions per task, and the replacement cohort contains 961 FinQA questions selected by the recorded fragment-matching rule.

For acquisition, FinQA records retain the supplied text and table headers, while HotpotQA records retain article titles and sentence indices. Records are split into chunks of at most 256 whitespace-delimited words. Fixed retrieval applies BM25 with k1 = 1.2 and b = 0.75 to the original question within the supplied candidate pool, with source order breaking ties. Adaptive acquisition starts with up to two chunks and permits up to three controller calls. Each call receives the question, accumulated observations, and remaining budget, then returns a query or a stop action. Each new query ranks the full candidate pool using the same BM25 rule and tie-breaking order. Previously acquired chunks are then excluded, and up to two highest-ranked unseen chunks are appended to the history within an eight-chunk total budget. Stopping occurs after an explicit stop, invalid or truncated control output, a search adding zero chunks, candidate exhaustion, or completion of the call or chunk budget.

For processing, each readout receives the history produced by the designated acquisition procedure. Draft-enhanced readout appends the generated draft verbatim as an assistant message, followed by the final-format instruction. The FinQA program readout accepts a JSON program comprising at most 20 flat, two-argument operations from ten arithmetic, comparison, and table-aggregation operations. Operands can be numeric literals, constants, or references to earlier results. Table operations access only rows returned as complete single chunks with structured row metadata in the acquired history. The final operation supplies the answer. The executor validates grammar, references, visible-row access, and numerical bounds before applying the official FinQA arithmetic semantics. The scoring calls use an exponent bound of 16. Final format errors, truncation, and execution failures remain in the full evaluation denominator and receive incorrect scores.

These execution details are particularly relevant to the concern about model-dependent program gains. The prompt instructs the model to select visible values, and the executor checks arithmetic and table access. The original Mistral comparisons use the corresponding Qwen task instructions, program grammar, executor, and scoring rules, while Mistral generates the adaptive controller outputs. Reporting valid-output rates alongside answer quality allows the comparison to account explicitly for interface use. This provides the procedural basis for interpreting the results in Appendix J.

Response to W2 (Part 2):

We also checked the effective generation settings against the runtime records. The runs use BF16, temperature 0, seed 1729, and each model's native chat template. The main Qwen context limit is 12,288 tokens, and the Mistral context limit is 32,768. Controller and draft limits are 192 and 512 generated tokens. Final limits are 2,048 for FinQA and 512 for HotpotQA. The reproduction specification records the exact prompt strings and continuation instructions together with software versions and model and tokenizer asset hashes.

To verify these descriptions against the reported experiments, we used the preserved model responses to replay all 65,962 requests across these five original experiment groups. The offline replay reconstructed messages, acquisition histories, query updates, and stopping events. The request total includes intermediate controller and draft calls. We reran the original parsing and execution procedures with the official scoring functions on all 43,844 final outputs. Every per-question score and validity flag matched the original analysis, as did the model-call, returned-block, and input/output-token counts. This agreement connects the documented procedures to the reported natural-task results.

Response to Q1:

The principal scientific result is to establish task structures under which an agent's acquisition and computational capabilities are jointly necessary, and to explain why the same necessary configuration can require different resource investments. The four-class framework defines the resource comparison, while the exact frontier and separation theorems establish the conditions under which that comparison yields joint necessity.

We developed the framework by combining the program-level perspective of bounded optimality with the question of which evidence a resource-constrained solver can acquire and use. The task, interface, and query budget are fixed across the four classes, and each class may redesign the complete acquisition, stopping, and answering procedure. Relative to the closest perspectives discussed in W1, this formulation makes computation relevant both to the observations that enter the history and to the prediction made from those observations.

The structural results provide the guarantees for this comparison. In localization, the exact coverage and terminally relaxed risk frontiers quantify how retained information and query budget constrain acquisition across all permitted encoders, randomized adaptive policies, and stopping rules. Combined with the passive-risk bound and an explicit successful agent, these frontiers establish a parameter region where both capabilities are necessary. The processing construction establishes joint necessity even when the acquired history contains all raw evidence, because the stated depth restriction limits how the evidence can affect the answer. These two results explain why acquiring the target record and processing complete evidence can lead to the same necessary configuration while calling for different interventions. Our response to W1 gives the precise frontiers, separation conditions, and constructions, together with the comparison to Russell and Subramanian (1995) and Xu et al. (2020).

Response to Q2 (Part 1):

The revised manuscript and executable reproduction materials are available in our anonymous repository. We have also update the manuscript on the website. The original natural-task protocols are consolidated in Appendix F.1 through F.7, and the additional prospective diagnosis experiment is documented in the diagnosis directory. We have organized these materials so that readers can follow each comparison from the evaluated questions and acquired records to the generated answers, configuration choices, and reported statistics.

Appendix F.1 specifies the dataset versions and evaluated cohorts. Appendix F.2 describes evidence-record construction, BM25 retrieval, adaptive query updates, and every stopping condition. Appendix F.3 explains how controller, draft, and final-answer messages are assembled. Appendix F.4 and Table F.1 give the inference settings and generation limits. Appendix F.5 specifies the program grammar, executor checks, output parsing, and scoring rules. Appendices F.6 and F.7 describe the paired statistical comparisons and resource accounting.

The repository includes the verbatim instructions in docs/PROMPTS.md, exact message templates in metadata/prompts_exact.json, evaluated question identifiers and shard assignments in metadata/sample_ids.json, and recorded runtime settings and model asset hashes in metadata/runtime_settings.json. The repository README provides commands for preparing the public data, running the original configurations, and replaying and scoring complete outputs. FILE_INDEX.md connects these files to the corresponding appendix sections and manuscript comparisons.

Reviewer_5TtGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies when a task requires both access to external evidence and additional computation. It compares the best achievable performance across four solver classes, holding the task, interface, and query budget fixed. Every class may redesign its acquisition, stopping, and answering procedures within its resource limits. The analysis derives exact evidence-coverage results under a restriction on retained information and processing lower bounds under a restriction on circuit depth. Together with explicit constructions, these results establish joint resource requirements for two task families. Experiments on controlled tasks, FinQA, and HotpotQA examine how acquisition and processing affect answer quality and resource use.

Critical strengths

A competing class can redesign its complete procedure, and the lower bounds account for randomized acquisition, adaptive queries, and early stopping. The exact coverage results, including the extension to sets of jointly required records, are the strongest technical part of the paper.

The analysis distinguishes the information acquired by a solver from its ability to retain and use that information. Allowing an unrestricted final rule to access the full history, including the original request, helps isolate the source of the remaining error. Appendix D.4 illustrates why collecting every record can still leave uncertainty about the answer. The experiments comparing answering procedures on the same acquired evidence help illustrate this distinction.

Critical weaknesses

The theoretical advance over earlier work on bounded optimality and information under computational constraints appears modest. Proposition 1 follows directly from class inclusion. The main localization frontier uses a counting argument under an independence assumption, and the processing separation uses the standard bound on how many inputs can affect a circuit of limited depth and bounded fan-in. The treatment of adaptive acquisition and overlapping evidence sets adds useful detail. The strongest guarantees concern the specified lookup and circuit models, and the main contribution is their organization into a common framework. The authors should clarify what the four-class comparison adds beyond the separate coverage and circuit-depth results.

The practical value of the diagnosis is not yet well demonstrated. Appendix H.4 reports that diagnosis and direct development search select the same configurations, with limited detail about the diagnostic decision rule or the evaluation effort involved. The natural-language experiments establish the effects of particular retrieval and answering procedures, but their relationship to the formal resource classes needs further explanation. Appendix J also shows that processing gains depend on the model's ability to use the program interface. A practical diagnosis needs to distinguish such interface failures from a task's computational requirements. The paper should specify the selection procedure, including the acquisition and stopping rules, program-interface details, and evaluation effort. Evidence that the diagnosis predicts useful interventions on held-out cases would strengthen the empirical case.

Questions for authors

The authors should clarify what the joint comparison of the four solver classes establishes beyond the separate evidence-coverage and circuit-depth bounds. A concrete result or example showing the additional conclusion obtained from their combination would help establish the theoretical contribution. Appendix H.4 needs a clearer account of how the diagnosis selects a configuration, including the information and evaluation effort required. A comparison with direct development search on held-out tasks would help establish its practical benefit. The authors should also explain how failures to use the program interface in Appendix J are distinguished from limitations in a task's computational requirements.

5 comments

We thank you for the careful review and for highlighting the importance of comparing complete solver classes and distinguishing evidence acquisition from evidence use. Your questions prompted us to clarify the conclusion established by the joint comparison and evaluate diagnostic recommendations on held-out questions. Below, we explain the concrete necessity example, the diagnostic procedure and evaluation effort, and the interface controls that help interpret the experimental results.

Response to W1:

The joint comparison establishes when acquisition and enhanced computation are both necessary for the same task, allowing every alternative solver to redesign the complete procedure for acquisition, stopping, and answering. Building on resource-constrained optimality, we compare computation on initial information with acquisition under a restricted controller, while fixing the task, interface, and query budget across classes. Proposition 1 expresses this comparison through class inclusion. The structural bounds and successful constructions in Section 3 establish the task conditions under which the full agent configuration is uniquely sufficient.

The numerical example in Appendix C.5 makes this conclusion concrete. Consider four external records and a target address drawn with probabilities (0.4,0.3,0.2,0.1)(0.4,0.3,0.2,0.1). Each record contains an independent Bernoulli$(1/4)$ bit, and the record contents are independent of the target address. The initial request specifies the target address, and the answer is the bit stored at that address. Fix the query budget at B=1B=1 and the target risk at η=0.1\eta=0.1. The restricted controller passes the request through a constant summary, whereas the enhanced controller can retain the two-bit target address. Encoding and subsequent computation are otherwise unrestricted in both classes.

For both passive classes, the best prediction has risk R00=R01=0.25R_{00}=R_{01}=0.25. For the restricted acquisition class, optimal target coverage is 0.40.4. The localization result therefore gives R10≥Rˉ10=0.25(1−0.4)=0.15R_{10}\geq\bar R_{10}=0.25(1-0.4)=0.15, where Rˉ10\bar R_{10} allows unrestricted terminal computation and access to the original request. The full agent retains the target address, queries the corresponding record, and returns the observed bit, achieving R11=0R_{11}=0. These results jointly give

\qquad \mathcal M_{0.1}=\{(1,1)\}.$$ The coverage bound quantifies how often the restricted controller can acquire the target record. Combining this bound with the passive risks and the successful agent construction establishes that both capabilities are necessary at the specified risk target. The lower bounds cover every solver in the declared restricted classes. Moreover, answering after a target read requires only copying a bit. The computational resource is necessary here because retaining target information enables the correct acquisition decision. This example shows how the comparison captures computation acting throughout the agent's control process. The processing construction in Section 3.3 gives a complementary conclusion. Under the stated depth conditions, the full agent configuration is again uniquely sufficient for any target risk below $1/2$, while complete evidence is already accessible and $\bar R_{10}=0<R_{10}=1/2$. Additional depth enables the required combination of that evidence. The matched comparison identifies the necessary agent capabilities, while the relaxed risks locate the limiting mechanism in these two constructions. The two constructions explain why tasks requiring the same agent configuration can call for different resource investments. **Response to W2 (Part 1)**: Our new experiment shows that diagnosis can guide useful resource interventions on held-out questions. Our aim in Appendix H.4 was to help decide where additional resources should be invested. Your suggestion prompted a prospective test, with the diagnostic rule and recommendations frozen before evaluation. We use Qwen2.5-14B-Instruct on FinQA and HotpotQA. Each task has three disjoint 60-question calibration sets and one common 600-question evaluation set. Splits are separated by financial report or candidate-article component and exclude sources from our documented earlier experiments. Following W1, we evaluate concrete procedures guided by the formal comparison. Each task has six candidates combining BM25 at $B=2,4,8$ with two answering procedures. Retrieval uses the original question once and stops at $B$ blocks or pool exhaustion. FinQA compares direct answers with programs of up to 20 operations in a fixed domain-specific language, executed on visible records. HotpotQA compares direct answers with evidence organization followed by answering from the records and notes.

Response to W2 (Part 2):

For each answering procedure rr, we pool calibration outputs at B=2,8B=2,8 to estimate correctness conditional on complete or incomplete annotated support, using add-one smoothing. These estimates, p^r,1\widehat p_{r,1} and p^r,0\widehat p_{r,0}, combine with calibration coverage C^B\widehat C_B to predict accuracy as

A^r,B=C^Bp^r,1+(1−C^B)p^r,0.\widehat A_{r,B}=\widehat C_B\widehat p_{r,1}+(1-\widehat C_B)\widehat p_{r,0}.

Candidates within two percentage points of the highest prediction are ranked by fewer model calls, smaller BB, then direct answering. We also freeze a choice among B4 direct, B8 direct, and B4 enhanced to test which single resource to increase.

At the main calibration budget, diagnosis and interpolation use the same endpoint outputs on 60 questions, while direct development search evaluates all six candidates on 40 questions. Each method accesses outputs costing 240 calls on FinQA or 360 on HotpotQA per calibration set, including HotpotQA's intermediate organization calls. Diagnosis additionally uses support annotations for the 60 questions, taken directly from FinQA's annotated record indices and HotpotQA's supporting facts. The existing dataset annotations supply all required support labels. The complete comparative study generates 11,700 calls. Selector access budgets, annotation requirements, and token use are recorded separately.

Across all three calibration sets, diagnosis recommends B4 program answering for FinQA and B8 direct answering for HotpotQA. Both frozen recommendations achieve the largest observed held-out gains among the predefined single-resource interventions. Gains over B4 direct answering have 95% paired bootstrap intervals over reports or article components.

TaskRecommendationGain in percentage points95% interval
FinQAB4 program+22.17[17.54, 27.27]
HotpotQAB8 direct+6.17[3.57, 8.81]

For the acquisition, processing, and joint interventions, the mean absolute errors between predicted and observed gains, averaged over the three interventions and three calibration sets, are 5.47, 6.79, and 6.66 percentage points for diagnosis, search, and interpolation on FinQA, and 1.69, 3.53, and 2.77 on HotpotQA. In six-candidate selection, all methods attain the same mean held-out score, 46.67% accuracy on FinQA and 52.50% exact match on HotpotQA. The 98.75% intervals for diagnosis minus search are [0, 0] and [-1.21, 1.16] percentage points, respectively. We ground the practical contribution in frozen intervention predictions and their realized benefits.

Your interface concern also motivates separating response and execution status from correctness. For B4 FinQA, validity requires a correctly formatted direct answer or a program that parses and executes successfully. The full 600-question evaluation gives

ProcedureInvalidValid but incorrectCorrect
Direct16534095
Program88284228

Valid but incorrect outputs pass these checks but produce an incorrect final answer. Invalid outputs include malformed responses, truncation, and execution failures such as references to table rows unavailable to the executor. All cases remain in the denominator. The paired gain evaluates the complete answering procedure, including representation and execution.

To isolate interface effects further, we use paired Mistral-7B-Instruct-v0.3 parity controls with 100 instances per configuration and a predefined tool aggregating visible bits. At 64 and 256 bits, constrained operation selection raises valid execution under complete evidence from 3% and 16% to 100%, with accuracy reaching 100%. Under half evidence, execution also succeeds throughout, while accuracy is 41% and 49%. These controls distinguish successful access to executable aggregation from evidence completeness and guide our interpretation of the workflow failures in Appendix J.

Response to Q1:

  1. The joint comparison establishes which configurations can replace the full agent at a specified risk target. Our response to W1 gives a concrete example in which the passive risks, the acquisition lower bound, and the successful agent construction jointly establish that (1,1)(1,1) is uniquely sufficient at η=0.1\eta=0.1. The bounds cover every solver in the two intermediate classes, including redesigned acquisition, stopping, and answering procedures. W1 also contrasts localization and processing to explain why the same necessary configuration can call for different resource investments.

  2. The diagnosis guides useful resource interventions on held-out questions. Our response to W2 provides the prediction formula, selection rule, acquisition and stopping procedures, and evaluation effort. The predictor combines support coverage with conditional correctness estimated from endpoint outputs. Support labels come directly from the existing FinQA and HotpotQA annotations, and their use is included in the calibration requirements. At the main calibration budget, diagnosis and direct search have equal model-call access budgets. The frozen single-resource recommendations produce the held-out improvements reported in W2. Diagnosis has lower observed mean absolute error in predicting intervention gains, while the mean scores from six-candidate selection match direct search. Our practical claim therefore rests on recommendations made before evaluation and their measured benefits. W2 reports the comparisons and separates selector access budgets from the complete study's expenditure.

  3. Interface validity and answer correctness are evaluated separately. The FinQA decomposition in W2 separately counts invalid outputs, valid but incorrect answers, and correct answers, with every case retained in the evaluation denominator. This makes errors after successful execution visible alongside format and execution failures. The paired Mistral parity controls in W2 examine operation syntax with evidence and the predefined aggregation tool held fixed, and evidence completeness with the constrained interface in place. Execution succeeds under both complete and partial evidence, while final-answer accuracy depends on evidence completeness. The tool aggregates only visible bits. We use these controls to interpret the workflow failures in Appendix J. Formal computational requirements follow from the declared solver classes and lower bounds in W1, while these experiments assess how particular implementations use the available evidence and computation.

Reviewer_KrpLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that an agent's success doesn't accurately show whether an agent is needed to complete that task. The paper explores this idea given two features of the agent's skillset: The use of tool or information retrieval skills and comprehensive reasoning. The authors argue that an agent is irreplacable when information acquisition and extended computation are both strictly necessary. Through controlled and natural language experiments, the authors show that enhanced computation can beat information retrieval on certain tasks (FinQA achieves higher success when computation is enhanced without any additional information), that searching can improve performance without enhanced computation (HotpotQA step-wise search is more efficient than global information ingestion without sacrificing performance), and that missing information cannot always be replaced with extended reasoning.

Critical strengths

The conceptual framing is strong, and formalizes agent necessity in a clear and testable way. The technical formulation of the propositions required to complete this diagnosis is clear and understandable.

The authors identify clear failure modes (inability to find information and inability to synthesize information) that identify when agents are needed, and experiments displaying these ideas on a natural language model and identify useful behavioral aspects of LLMs under these tasks (enhanced computation can overcome the need to search exhaustively for information, and targeted search and retrieval can reduce the need for enhanced computation).

Critical weaknesses

The authors propose the central goal of this work as determining when an agent is irreplacable, but the downstream tasks explored with real language models (FinQA, HotpotQA) are not used to test agent irreplacability.

The LLMs are not used on the toy tasks in the first half of the paper (localization/parity); consequentially, the theory of irreplaceability is not demonstrated with natural language experiments, weakening the premise of the paper. Further, the results in appendix I show that the trends exhibited by Qwen do not translate to another model family (Mistral), weakening the findings.

Questions for authors
  1. Do the controlled localization/parity experiments replicate the same results when applied to Qwen?
  2. Does task-based irreplaceability hold across model sizes and families?
5 comments

We thank the reviewer for the careful reading and constructive feedback. Your questions helped us clarify the connection between our theoretical analysis and language-model behavior and motivated additional controlled experiments across model sizes and families. We address each concern below and explain how the new results complement the evidence in the submission.

Response to W1:

The FinQA and HotpotQA experiments test the practical implications of our resource analysis for agent design. Our research question encompasses both the conditions requiring the full agent configuration and the conditions under which simpler alternatives suffice.

Section 3.4 shows that tasks requiring the same combination of acquisition and enhanced computation can be limited by evidence localization or by the use of acquired evidence. These mechanisms suggest different resource investments. We designed the natural-language comparisons around this distinction, varying the answering procedure on fixed acquisition histories and intervening on the supporting evidence. The aim is to examine when better use of available evidence helps and when improving acquisition is more effective.

On FinQA, program readout increases accuracy from 17.1% to 50.3% on the same eight-block acquisition histories, a paired gain of 33.2 percentage points with a 98.75% cluster interval of [28.30, 37.93] (Section 4.3). This improvement also changes the trade-off between evidence quantity and answer quality. Program readout with four retrieved blocks achieves 39.7%, compared with 17.5% for direct answering with all candidate evidence, averaging 31.60 blocks (Section 4.4).

On HotpotQA, adaptive acquisition improves direct-answering exact match from 48.0% to 52.2% with at most eight evidence blocks. The coverage analysis shows that the net gain is concentrated in questions whose annotated-support coverage changes (Section 4.3).

We further examine the importance of supporting evidence through the replacement experiment in Appendix I. On 961 eligible FinQA questions with zero added distractors, replacing one annotated supporting record with an unrelated fragment of similar length reduces program accuracy from 61.19% to 9.26%. Adding an intermediate draft yields 65.04% with complete support and 11.86% with missing support. Restoring support produces a much larger accuracy gain than adding a draft in this comparison.

The theoretical results provide the class-level necessity guarantees, while the natural-language experiments assess their practical implications for the evaluated agent workflows. To examine the controlled task mechanisms with language models directly, we have also completed localization and parity experiments with Qwen and Mistral, reported in our response to Weakness 2.

Response to W2 (Part 1):

We have now evaluated localization and parity with Qwen2.5-14B-Instruct, Qwen2.5-7B-Instruct, and Mistral-7B-Instruct-v0.3. Across the three models, these experiments reproduce the qualitative benefits of restoring target identity in localization and providing executable aggregation for complete parity evidence. We use 600 instances per core configuration, paired across interventions, with the same task instances and allowed resources across models.

For localization, M=64 addresses contain independent fair bits, and the target address is uniformly distributed. The 2-bit summary identifies one of four groups of 16 candidate addresses. Each model chooses up to 16 addresses to query. The restricted readout receives the summary and queried records. We then restore the exact target address at readout while keeping that model's query history fixed. A separate condition retains the full 6-bit address and allows one query. This design distinguishes obtaining the target record from retaining the information needed to identify that record.

For parity, two records each contain N/2 independent fair bits. We supply both records in the complete-evidence condition and one record in the half-evidence condition. Direct answering, CoT, and tool-assisted processing receive the same raw evidence within each condition. CoT has a 4096-token reasoning budget. The tool condition lets the model select a supplied operation, xor_visible, or a binary guess. The operation aggregates only visible records, following the computation-axis definition in Appendix B. After execution, the model predicts the parity of all N bits. Operation selection uses a constrained interface, and all methods use a binary final-answer interface.

Response to W2 (Part 2):

The table reports accuracy in percent, with parity results at N=1024.

ConfigurationQwen 14BQwen 7BMistral 7B
Localization, 2-bit summary, up to 16 queries52.8353.0048.17
Localization, same history with target address restored99.5093.8371.00
Localization, full address, up to 1 query100.0097.0095.50
Parity, complete evidence, direct51.0051.1751.00
Parity, complete evidence, CoT47.3347.3347.00
Parity, complete evidence, aggregation tool100.00100.00100.00
Parity, half evidence, aggregation tool47.6754.6753.50

In the 2-bit condition, target-record coverage is 600/600 for both Qwen models and 348/600 for Mistral. For both Qwen models, the target record is already present in every acquired history, so the restored-address comparison directly measures the benefit of reinstating the information needed to identify that record. For Mistral, the history covers 58% of targets, so the 71.00% restored-address accuracy reflects both evidence acquisition and subsequent use.

In the full-address condition, target-record coverage is 600/600, 565/600, and 522/600 for Qwen 14B, Qwen 7B, and Mistral, respectively. Mistral answers all 522 covered targets correctly and makes 51 correct predictions among the 78 uncovered cases, giving an overall accuracy of 573/600 (95.50%). We report the uncovered-case count as a finite-sample outcome. Under the stated independent fair-bit model, the success probability for an unread target is 1/2. An audit of the recorded inputs confirms that the readouts receive exactly the declared query histories.

The parity comparison shows a different resource dependence. With complete evidence, access to executable aggregation raises accuracy to 100% at N=1024 for all three models. With half the evidence, the same tool-assisted workflow yields final accuracies of 47.67% to 54.67%. Across the tested lengths of 64, 256, and 1024 bits, complete-evidence tool-assisted accuracy ranges from 93% to 100%. These comparisons connect evidence completeness and processing interventions to the mechanisms studied in Section 3.

Appendix J shows that gains from the original FinQA and HotpotQA workflows depend on the model. For FinQA specifically, the recorded invalid programs identify interface adherence as an obstacle to using the available computation. We therefore examine the controlled task mechanisms through an explicit, executable aggregation interface. All Mistral core tool calls issued through the constrained operation-selection interface execute successfully. This setting allows us to assess how access to executable aggregation affects final-answer accuracy. At N=1024, complete-evidence tool-assisted processing improves over CoT by 53.00 percentage points, with a 98.75% paired interval of [47.83, 58.00]. The new results show that Mistral exhibits the same qualitative resource-intervention patterns as the tested Qwen models, while accuracy levels depend on the model and workflow.

Response to Q1:

Yes. The controlled experiments with Qwen2.5-14B-Instruct reproduce the main qualitative resource-intervention patterns predicted by our analysis.

In localization, restoring target identity improves answering on the same query histories, which contain the target record in every instance. Retaining the full address also enables accurate answering with a single-query budget. For parity, access to the supplied aggregation operation improves performance with complete evidence, while the same tool-assisted workflow achieves approximately 50% accuracy with half the evidence. These comparisons demonstrate the distinct roles of retaining target information and processing acquired evidence in the tested Qwen workflows.

Our response to Weakness 2 provides the experimental settings and numerical results. Our response to Weakness 1 explains how these mechanism tests complement the class-level theoretical guarantees.

Response to Q2:

The same qualitative resource-intervention patterns appear across the tested Qwen sizes and between Qwen and Mistral at the 7B scale, with model-dependent accuracy levels.

We compare Qwen2.5-7B-Instruct with Qwen2.5-14B-Instruct within one family, and Qwen2.5-7B-Instruct with Mistral-7B-Instruct-v0.3 across families. As reported in our response to Weakness 2, all three models benefit from preserving target information in localization and from executable aggregation with complete parity evidence. That response provides the numerical results and explains the observed model differences.

Following the distinction in our response to Weakness 1, these comparisons provide empirical support for the resource mechanisms underlying agent irreplaceability. The theoretical necessity guarantees apply to the specified tasks, interfaces, budgets, and solver classes.

Dear Reviewer KrpL,

Thank you again for your careful review. We wanted to briefly follow up on our responses and additional experiments. Following your questions, we evaluated controlled localization and parity tasks with Qwen at two model sizes and with Mistral. Our responses to W2 and Q1–Q2 report the results and clarify the resource-intervention patterns observed across these models.

With the discussion deadline approaching, we would appreciate your thoughts on whether these additions address your concerns. Please let us know if there are points you would like us to clarify further while the discussion is open.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Author Summary on Agent Irreplaceability and Resource Allocation

Dear Area Chair,

We thank you and the reviewers for assessing our work. When an agent succeeds, its performance reflects both the demands of the task and the design of its workflow. Our paper asks which of the agent’s capabilities remain indispensable after competing solvers have had the freedom to redesign their entire solutions.

Our Contribution

Task-level agent irreplaceability. We compare the best attainable risks of four solver classes that differ in access to external evidence and enhanced computation. We hold the task, interface, and query budget fixed across classes. Each solver can choose what to query, when to stop, and how to answer. This comparison identifies the minimal sufficient configurations for a target risk and establishes joint necessity when only the full agent configuration can attain that target.

Structural limits on agent simplification. We prove joint necessity for two distinct reasons. In localization, an exact coverage frontier shows how retained address information and the query budget limit a solver’s ability to find the target record. In parity, even a solver with access to every input bit can have too little computational depth to combine them. In each family, the lower bounds cover every permitted alternative, and an explicit agent construction attains the target.

Task-dependent resource investment. The same capability requirement can call for different design choices. In our two-address example, increasing the query budget until both records can be read leaves the optimal risk unchanged. Retaining one address bit instead permits an exact answer with one query. In parity, the useful investment is additional depth to process evidence already in hand. The natural-task experiments connect this distinction to resource use. Better processing on FinQA achieves higher accuracy with fewer records, while adaptive acquisition on HotpotQA saves records and tokens at the cost of more model calls.

Reviewer Recognition

Reviewer 5TtG highlighted the exact coverage results and the distinction between acquiring evidence and being able to use it. The reviewer confirmed that the prospective study addressed the request for a diagnostic procedure and held-out evidence for its recommendations.

Reviewer KrpL confirmed that the Qwen and Mistral experiments on localization and parity answered the question about transfer from controlled tasks to language models and strengthened the paper’s technical completeness.

Reviewer 3DPV valued the comparison over whole solver classes and the result that even complete coverage can leave the answer uncertain when the target’s identity is lost. After the discussion, the reviewer raised their assessments of soundness and informativeness.

Discussion Outcomes

The discussion led us to test whether a simpler FinQA workflow could preserve the accuracy achieved with both retrieval and executable arithmetic. On the same 600 held-out questions, Mistral answered 162 correctly with both resources, compared with 26 for the stronger single-resource alternative. Qwen showed the same pattern. In both model families, executable arithmetic yielded a much larger gain once retrieval supplied the figures needed for the calculation. That dependence explains why giving up either resource cost accuracy among the workflows we tested.

Practical Impact

A designer deciding where to invest needs to know where useful evidence is being lost or underused. We used calibration questions to estimate how often retrieval found the supporting records and how well Qwen answered when those records were available. Starting from the same workflow on both tasks, we recommended executing arithmetic on the records already retrieved for FinQA and expanding retrieval while keeping the answering procedure fixed for HotpotQA. We committed to those choices before evaluation. On each task, the recommended change produced the largest held-out gain among the prespecified single-resource interventions.

We respectfully ask you to consider the paper for acceptance. We believe its contribution lies in connecting two decisions in agent design that are often made separately. The class comparison establishes which capabilities a task requires. Identifying why those capabilities are needed gives a principled basis for deciding where to invest next.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.