- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies task-free discovery of parameter-level connections between tools. It proposes ICHNAEA, an explorer that maintains Beta beliefs over candidate connections, selects calls using predicted acceptance and information gain, and stores successful calls as recipes. When the environment's data change, these recipes are replayed to reconstruct valid inputs for further exploration. Experiments show improved graph reuse and downstream target-function reachability in procedural tool worlds. Results on AppWorld reveal limitations from coincidental argument acceptance and provide limited evidence of additional benefit over documentation alone.
Critical strengths
-
Learning reusable tool connections before receiving a task is a useful problem. The distinction between persistent connection knowledge and transient input values is well motivated, and recipe replay provides a concrete mechanism for rebuilding inputs after data changes.
-
The paper includes systematic component ablations and held-back-seed evaluations in its procedural environments. It also candidly reports negative results on AppWorld, including cases where replay reinforces coincidental connections.
Critical weaknesses
-
The method does not adequately distinguish executable connections from semantically meaningful ones. Its claim rule relies on successful execution and an acceptance-probability threshold, which do not establish that an output serves the intended semantic role of the receiving parameter. The paper's example of a review rating being accepted as a song ID illustrates this directly. Replay can then reinforce the same coincidence and increase confidence in a spurious connection. Although specificity probes are collected, they do not affect the claim rule. This is a central unresolved limitation for a method intended to provide reusable tool-composition knowledge. The AppWorld results demonstrate its practical impact: with the fixed planner, adding reconstruction reduces expert-endorsed goal reach from 63% to 31% on Spotify while increasing unendorsed reach from 16% to 44%. Unendorsed routes are not necessarily all incorrect, but these results do not establish improved connection quality.
-
The evaluation does not sufficiently establish benefits for realistic task completion. The strongest positive results come from an author-designed generator with a fixed chain structure, negligible identifier collisions, and explicit feedback identifying rejected arguments. These assumptions substantially simplify discovery and largely avoid the semantic ambiguity exposed by AppWorld. The downstream study uses only Gemini 3.5 Flash and evaluates successful invocation of target functions rather than completion of full user tasks. On AppWorld, documentation alone achieves 95/99 successful attempts, compared with 96/99 using the graph learned with reconstruction and transfer; the original AppWorld tasks are not evaluated. Thus, the synthetic improvements do not yet establish useful gains in realistic agent performance. Moreover, reduced downstream calls do not by themselves demonstrate overall efficiency once the upfront exploration budget is included, and there is no comparison against an LLM explorer under a comparable budget.
Questions for authors
-
Additional evidence on connection validity: Can the authors evaluate a claim rule that incorporates the collected specificity probes or another semantic-consistency check, and show whether it reduces coincidental connections while preserving useful coverage on AppWorld? This would directly address the failure mode introduced by replay.
-
Additional evidence on practical utility: Can the authors demonstrate improved end-to-end task success on a more realistic evaluation, using task-level correctness rather than target-function reachability? A comparison with documentation-only and LLM-based exploration baselines, accounting for both exploration and downstream costs over a stated number of tasks, would materially affect my assessment.