- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether adaptive tool use provides an information advantage over choosing queries in advance. The authors define this advantage through Bayes-optimal task utility and derive a finite-budget upper bound using pairwise response overlap and fractional matching. Moreover, controlled network-diagnosis experiments show that changing the tool interface can create or remove this advantage. The model experiments further distinguish between using adaptively selected evidence and discovering effective adaptive query policies.
Critical strengths
-
The distinction between information value and bounded-model performance is valuable. Equation 2 explains why an observed adaptive gain does not establish better information acquisition, giving a concrete basis for evaluating claims about tool-using agents.
-
The experiments isolate the effect of the tool interface while holding the underlying task fixed. The matched evidence format, presentation control, and model-selected querying comparison strengthen the finding that using better evidence and selecting it are separate capabilities.
Critical weaknesses
-
The practical contribution of the main certificate remains limited in the provided experiments. In Section 4, all 15 certified zero-gap cases already have a pre-committed solution achieving perfect-information utility. The noisy sensitivity study tightens the bound but leaves every verdict inconclusive, while the headline results rely on exact search. These limitations are acknowledged, but stronger evidence is needed that the proposed certificate enables useful audits beyond simpler bounds or tractable exact computation.
-
Applicability to existing agent benchmarks remains uncertain. The main empirical evidence concerns small-budget diagnosis tasks, and the model study selects topologies for the desired interface contrast and supplies externally optimised queries. This supports the stated mechanism, but does not yet establish how useful the audit is when the latent hypotheses and response channels are difficult to specify.
Questions for authors
- Can you identify a non-saturation setting where the affinity certificate gives a meaningfully small upper bound on adaptive information value and exact search is impractical? A comparison of bound tightness and runtime can be helpful for establishing its practical contribution.
- Section 7 describes the oracle gap as a maximum effect for power planning. However, Eq. 2 allows the observed model contrast to exceed this gap when the optimisation shortfalls differ. What additional assumptions justify this recommendation?