acceptodds
Under review as a conference paper at ICLR 2027

Compile the Catalog, Not the Task: ToolGraph as a Tool-Use Agent Harness that Updates, Scales, and Transfers

Abstract

Tool-use agents must chain interdependent calls, and which outputs satisfy which later inputs is a property of the deployed tool catalog, not of any one task or model. We argue that this structure should be compiled once into a persistent artifact that any frozen agent can read. ToolGraph is a directed, weighted tool-dependency graph built from tool interfaces alone: parameter matching and counterfactual probes, in which an evaluator LLM compares a candidate chain with the same query answered without its proposed predecessor, propose edges, and a Bayesian combination of match and information-gain evidence sets each edge's confidence. Construction reads no task data, reward, or expert label. A frozen planner reads the graph through a top- injection ranked by personalized PageRank or by a goal-conditioned action value composed from graph paths. We study ToolGraph as an agent harness along three axes. Update (continual learning): the agent stays frozen while the graph learns. Create/read/update/delete (CRUD) operations driven by execution outcomes raise completion on held-out -Bench tasks by – points for two planners, with the gain saturating by – of the update stream. Scale (test-time scaling): spending test-time compute on an LLM scorer that reads graph evidence outperforms Language Agent Tree Search in all six ToolComp and -Bench settings while using – fewer LLM calls per task. Transfer: with no training, subgraphs retrieved from a -node ToolGraph compiled on Toolathlon's catalog lift DeepSeek-V3.2 on -Bench from to , while the reverse direction trails the target's own graph. Across three benchmarks and three backbones, the reader matters more than the graph: the same edges help or hurt depending on how the planner reads them.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
test time scaling, q value, graph, agent, tool use, tool graph, continual learning, agent harness

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_axvBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents ToolGraph, a tool-use agent harness that compiles dependencies between tools into a persistent, weighted graph using parameter matching and counterfactual probing. The graph provides action recommendations to frozen LLM planners and is refined using execution outcomes. The authors investigate three capabilities of the framework, including continual graph updates, test-time scaling through graph-guided action scoring, and transfer across disjoint tool catalogs. Experiments on ToolComp, τ²-Bench, and Toolathlon show improvements in task completion, with the LLM-based graph reader outperforming other graph-reading strategies.

Critical strengths
  1. The paper introduces a unified framework that supports continual learning, test-time scaling, and cross-catalog transfer through a shared tool-dependency graph without modifying the underlying LLM planner.

  2. The proposed counterfactual probing mechanism evaluates tool dependencies by comparing the information gained from chaining tools against using individual tools, providing evidence beyond simple parameter matching.

  3. The paper provides extensive experiments across three benchmarks and three LLM backbones, including comparisons of four graph-reading strategies, inference cost analysis, and evaluations of continual graph updates and cross-catalog transfer.

Critical weaknesses

I appreciate the authors' efforts. However, I have several concerns that I believe should be addressed to strengthen the paper.

  1. My main concern is the lack of a systematic study showing that incorrect tool selection or tool arguments are the primary causes of task failures. I believe that selecting an incorrect read tool or providing incorrect arguments to a read tool may not necessarily lead to task failure. In contrast, incorrect write operations and policy violations can have more significant consequences, particularly in environments such as τ-Bench, as discussed in [1]. Although the authors demonstrate that constructing ToolGraph improves task completion, these results do not establish that tool dependencies are the primary source of failures. I encourage the authors to conduct a comprehensive failure analysis to identify the dominant error types and clarify which failures ToolGraph actually mitigates.

  2. The authors propose a method for constructing tool-dependency graphs. However, several previous studies [2–5] have also explored dependency-based graph construction, primarily for generating tasks and interactive environments. Although these works use dependency graphs for different downstream purposes, comparing their graph construction approaches with ToolGraph would help clarify the novelty of the proposed method and whether its construction strategy provides additional benefits.

  3. Using a portion of the benchmark tasks for training raises concerns about potential data contamination and the interpretation of generalization results. Although τ²-Bench provides an official train/test split, the original paper does not establish this split as its evaluation protocol. The training tasks were originally introduced for exploring agent training rather than defining a standard evaluation split for τ²-Bench. I encourage the authors to use external training data [6-7] for their graph updates and evaluate on the complete τ²-Bench evaluation set. This would provide stronger evidence that the improvements generalize beyond the benchmark-specific training tasks.

  4. Numerous training-based and training-free methods have been proposed to improve LLM agents' tool-use performance, but the paper lacks sufficient comparisons with these approaches. In particular, I suggest including EvoTool [8] and CAST [9] as additional baselines on τ²-Bench, where applicable. These comparisons would help clarify how ToolGraph performs relative to alternative tool-use improvement techniques and whether its graph-based guidance offers complementary benefits.


References:

[1] IRMA: How Can Input Reformulation Improve Tool Usage Accuracy in a Complex Dynamic Environment? A Study on τ-Bench, https://arxiv.org/abs/2508.20931

[2] APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay, https://arxiv.org/abs/2504.03601

[3] EnvScaler: Scaling Tool-Interactive Environments for LLM Agent via Programmatic Synthesis, https://arxiv.org/abs/2601.05808

[4] VULCAN: Where Agents Learn by Living in Simulated Tool Environments, https://openreview.net/forum?id=Ht4HB4pIQq

[5] ScaleEnv: Scaling Environment Synthesis from Scratch for Generalist Interactive Tool-Use Agent Training, https://arxiv.org/abs/2602.06820

[6] Simia: https://huggingface.co/datasets/Simia-Agent/Simia-Tau-SFT-90k-Hermes

[7] AReaL: https://huggingface.co/datasets/inclusionAI/AReaL-tau2-data

[8] EvoTool: Self-Evolving Tool-Use Policy Optimization in LLM Agents via Blame-Aware Mutation and Diversity-Aware Selection, https://arxiv.org/abs/2603.04900

[9] CAST: Critique-Aware Supervision for Training Reliable Long-Horizon Tool-Calling Agents, https://arxiv.org/abs/2608.30147

Questions for authors

Please read the weaknesses.

2 comments

W1: failure analysis. We analysed the 94 failures of our method (TG Q_llm) on τ², pooled over the three planners.

Failure typeCountShare
Wrong or missing write, all reads fine5558.5%
— forbidden by policy1617.0%
— wrong arguments1718.1%
— required write never made1617.0%
— wrong tool or order66.4%
A bad read leading to a wrong write1313.8%
Wrong read tool alone33.2%
User-simulator or gold-label ambiguity1516.0%
Early stop or transfer88.5%

What the table shows:

  • A wrong read tool alone caused 3 of 94 failures. Read errors matter mainly when wrong arguments lead to a wrong write.
  • Policy-forbidden writes and wrong write arguments are each about 17% of failures.
  • The baselines show the same pattern: wrong or missing writes are the largest class in every arm.
  • Counting every case where the agent picked the wrong tool or failed to pick a needed one, the strict count is 9 of 94 (9.6%): wrong tool or order (6) and wrong read tool (3).
  • It rises to 21 of 94 (22.2%) if we add the "never made" writes where the agent called user-only tools itself or never chose the diagnostic tool. All 16 "never made" writes are telecom roaming omissions; in 4 of them the agent had the evidence (a roaming flag or a device status) and did not act on it, which leaves 12.

What ToolGraph mitigates.

  • Relative to the planner alone, the full system reduces no-action failures (28 → 12, pooled) and extra-write failures (22 → 16).
  • In telecom it reduces missed roaming diagnoses, and topology matters there: with a random topology, wrong-phone-line writes rise to 24 of 80 Ministral and DeepSeek runs.

The revised paper adds this analysis as an appendix and uses it to explain the graph controls.

W2: APIGen-MT, EnvScaler, VULCAN, ScaleEnv. All four are now cited. They build tool-dependency graphs to synthesize tasks or environments for training, so the graph feeds a data generator. Ours is the artifact a frozen planner reads and updates at inference.

W3: splits and external data.

  • τ²-Bench has a public train/test partition released for RL, and we use it so the update stream never touches an evaluated task. The leaderboard protocol scores all 278 tasks, and our wording "official split" is corrected.
  • ToolComp and Toolathlon publish no split, so we use our own random 60/40 splits, and all arms are scored on the same fixed evaluation tasks.
  • No evaluation task is read during construction or updating.

W4: EvoTool and CAST. Both are now cited. They train or evolve the planner, while ToolGraph changes only what a frozen planner reads, so they are complementary rather than competing. Combining them is a natural follow-up.

My main concerns have been addressed. I am willing to increase my score.

Reviewer_tyBLOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This study proposes ToolGraph, a persistent, directed, and weighted tool-dependency graph compiled from parameter matching of tool interfaces and LLM-simulated counterfactual probes. The system is read by a frozen planner through various readers, including personalized PageRank and a graph-aware LLM action scorer. The paper comprehensively evaluates this harness across three dimensions: continual learning updates, test-time compute scaling, and cross-catalog transfer.

Critical strengths

The paper externalizes the dynamically changing tool catalog into an incrementally maintainable, persistent artifact while keeping the planner frozen, offering a valuable new perspective for agent continual learning. Furthermore, its experimental design is exceptionally broad, validating core claims across multiple benchmarks and backbone networks, while clearly demonstrating the delayed emergence and saturation behaviors of performance gains.

Critical weaknesses

First, the chosen baselines are remarkably weak, relying repeatedly on 2023 methods such as ReAct, and Reflexion; comparisons with state-of-the-art (SOTA) methods in tool use and graph-structured planning from 2024 to 2026 are needed. Second, the core empirical comparison fails to decouple the graph structure from stronger LLM decision support, as both TG-PPR and QllmQ_{llm} benefit from extra verifier feedback and rich context, effectively attributing the credit of a powerful LLM critic to the graph. Moreover, it lacks foundational control experiments such as a graph-free QllmQ_{llm} or a graph with randomized edge weights. "Counterfactual" probing is essentially LLM text simulation rather than real-world environment interaction; the Bayesian posterior in Equation (2) lacks a rigorous probabilistic observation model, and the action-value model in Equation (3) remains ambiguous regarding state-related definitions and incoming edge aggregation. Finally, does CRUD continual learning lack a "result-agnostic update" control, and does the distillation of QembQ_{emb} flaw the "task-free" claim? Direct evaluation of the graph quality itself is needed.

Questions for authors

Please refer to Weaknesses.

1 comment

W2 and W3: decoupling the graph from the LLM critic; graph-free and randomized-weight controls. We ran the same Q_llm scorer and critic with the graph replaced by (i) no graph, (ii) a random topology with the same nodes and number of edges, and (iii) the real topology with permuted edge weights (same weight distribution).

Pass rates (%). TC = ToolComp, TA = Toolathlon; M / G / D = Ministral / Gemma / DeepSeek:

Armτ²-Mτ²-Gτ²-DTC-MTC-GTC-DTA-MTA-GTA-D
Real graph53.073.080.061.473.171.16.118.221.2
No graph48.065.085.056.372.173.10.06.19.1
Random topology56.974.677.750.068.077.00.03.09.1
Permuted edge weights53.068.080.061.977.275.13.06.112.1
Planner + verifier, no graph, no critic47.062.075.0

Why random graphs barely change τ² and ToolComp.

  • τ². Of the 94 failures we read by hand, 58.5% are wrong or missing writes with every read correct. Only 9.6% involve a wrong tool or order. When the dominant errors are wrong arguments and policy violations and the tool list is small, a graph over tool dependencies has little to correct.
  • Telecom. Topology shows up here, where tool order matters. With a random topology, agents write to the customer's wrong phone line in 24 of 80 Ministral and DeepSeek runs, against 8 and 12 of 80 with the real graph.
  • ToolComp. It has 11 tools, and 79 of its 197 tasks use only two. The graph has 35 edges, so there is little for the graph to select among.
  • Toolathlon. It has about 620 tools, and the reader re-ranks the personalized-PageRank top 10. Here tool selection matters, and every control falls well below the real graph (at most 3.0 / 6.1 / 12.1 against 6.1 / 18.2 / 21.2).

Counterfactual probing. We agree that it is LLM text simulation, not environment interaction. The paper now describes the probes as LLM-simulated and explains how each probe type is scored.

W1: baselines. LATS (2024) is our search baseline. IRMA, EvoTool and CAST are now cited and discussed. EvoTool and CAST evolve or train the planner (EvoTool's code is not released), so they are complementary to a harness that keeps the planner frozen. We list the missing 2025–26 external comparison as a limitation.

W4: Eq. 2 and Eq. 3.

Eq. 2. Yes, this is not a derived probabilistic model.

  • The likelihood is a sigmoid of the probe's gain score, σ((IG − 0.5)/0.15), with 1 − σ for "no edge".
  • The edge weight is the mean of the per-component posteriors.
  • We now call it a score combination in Bayes form and drop "Bayesian".
  • We now also state three facts the paper omitted:
    1. Same-class edges are filtered on a match score ≥ 0.3, not on gain > 0.
    2. Schema-matching edges use a fixed likelihood of 0.5, so their posterior equals their schema confidence.
    3. The posterior is set to 1.0 when the prior is 1.0, which affects 8 τ² edges and 79 Toolathlon edges.

Eq. 3. p_a is the max and c_a the mean over all incoming edges of the tool in the catalog graph. They do not depend on which predecessors have already run in the current state. We added the missing definitions:

  • E(s) is the set of entity names from the conversation and tool outputs.
  • The goal tool is inferred by embedding similarity plus an LLM call over write-like and terminal tools.
  • Tools with no incoming edge get 0.5.
  • The value is mapped affinely to [0, 1].
  • γ and κ were fixed, not swept.

W5: result-agnostic control, "task-free", and graph quality.

Result-agnostic control. Yes, it is missing; we have not run a control that applies the same updates regardless of outcome.

  • The statement is now in the main text, next to the CRUD claim. The gain cannot yet be split into learning which transitions succeed and learning which transitions the training tasks use.
  • How much weight moves: a failure adds 0.3 and a pass adds 1.0, so an edge that passes only one time in four still gains weight.
  • On τ², the update stream produced 36 edge creations and no deletions. Updates are therefore mostly additions and upward weight shifts.

"Task-free." We narrowed the claim.

  • The harness is outcome-supervised. The update stream uses the benchmark's pass/fail labels (an LLM goal-completion evaluator could supply them instead).
  • Training-split signals also enter the edge confidences.
  • The training split is used to choose the Table 6 hyperparameters, and Q_emb is distilled from Q_dp on training-split states.
  • It needs no planner fine-tuning and no hand-written task labels, and no evaluation task is read.
  • The paper says this wherever "task-free" appeared.

Graph quality. We report edge provenance by evidence type (table in our reply to x18B, Q4). No expert-labelled dependency graph exists for these catalogs, so a direct audit is listed as future work.

Reviewer_x18BOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper builds a weighted tool-dependency graph once per tool catalog. Edges come from LLM-based parameter matching and from LLM-simulated probes that compare a two-tool chain with the second tool alone. A frozen planner reads the graph through one of four rankers: PageRank, two structural value estimators, or an LLM scorer. The paper evaluates three uses of the graph on ToolComp, tau2-Bench and Toolathlon: updating it from execution outcomes, using it as a cheaper alternative to tree search, and transferring it across catalogs. The main claims are that the LLM scorer is the best reader everywhere, that outcome-based updates add 6 to 7 points, that it beats LATS with far fewer calls, and that a Toolathlon graph helps DeepSeek-V3.2 on tau2-Bench.

Critical strengths

The reader comparison is well controlled. All graph-based arms share one graph and one prompt interface, so the paper can isolate how the graph is read. The finding that the structural value estimators do worse than plain PageRank is useful, and the limitations section is candid.

The graph is a model-agnostic artifact, and per-step scoring is much cheaper than tree search in LLM calls, which would matter in practice if the gains can be attributed to the graph.

Critical weaknesses
  1. Limited novelty. Building a tool graph from schemas and reading it with HippoRAG-style PageRank follows Liu et al. (2025a), as the authors acknowledge. A training-free, catalog-level dependency graph for tool retrieval also appears in Graph RAG-Tool Fusion, a plug-and-play approach requiring no model fine-tuning, which is not cited. An uncited June 2026 preprint also calls its method ToolGraph and also evaluates on tau2-Bench. It builds topology from tool schemas, takes edge weights from successful trajectories, and uses best-of-N scoring to bias tool choices, which is close to this paper's update-plus-rescoring setup. The LLM scorer itself is standard step-level best-of-n with an LLM verifier. What is left is the simulated probe weighting, and that is never ablated.

  2. The gains may come from the LLM critic rather than the graph. The key control is missing: the same LLM scorer, with the same policy, history and number of candidates, but no graph or a random graph. Reflexion is not that control. The existing results point the other way. On ToolComp, shrinking the subgraph to a few nodes or pruning more than half the edges does not hurt (Table 5). With a stronger verifier, the structural reader jumps by more than 10 points even though it makes no model call, which the authors attribute to the verbalizer. And a graph from an unrelated catalog beats tau2-Bench's own graph (88 vs 80), which contradicts the title's claim that the catalog itself should be compiled.

  3. Several numbers do not line up, and the effects are close to noise.

    • The static point in Figure 3 (68 for Gemma, 78 for DeepSeek) differs from Table 1 for the same setting (73 and 80). Measured against Table 1, the update gain drops to 2 to 4 points.
    • Table 2 token costs for Gemma and DeepSeek on tau2-Bench do not match the n=8 setting listed in Table 6, given what Table 5 reports for n=8.
    • The Toolathlon percentages cannot all come from 43 tasks under any one rounding rule.
    • The two construction-cost counts in Table 8 differ by about 7x for tau2-Bench.
    • The LATS baseline uses imagined rollouts and scores about the same as Reflexion, so beating it says little.
    • Results come from a single seed, and the authors' own replicate spread is 2 to 3 points.
  4. The method overclaims what it does. The same-class probe prompt asks directly for a gain score without simulating a baseline, so most edges in a large catalog are probably not counterfactual at all. The probe prompt hard-codes judgments about specific ToolComp tools, and 12 Toolathlon edges were added by hand, which conflicts with "no expert label." On the update side, failures add only 0.3 while passes add 1, so weights mostly track usage frequency. Under the stated constants, deleting an edge would need dozens of failures with no passes, so DELETE likely never fires in a 178-task stream.

Questions for authors
  1. How does the LLM scorer perform with no graph, a random graph, and a schema-only graph, keeping everything else fixed?
  2. Why does Figure 3's static point differ from Table 1? Can you report results over at least three seeds?
  3. Were the Table 2 costs measured with the Table 6 settings, and do the call counts include planner calls for both methods?
  4. What share of edges came from the same-class prompt? How many CREATE and DELETE operations actually fired?
3 comments

Q1. How does the LLM scorer perform with no graph, a random graph, and a schema-only graph, keeping everything else fixed?

We ran the same Q_llm scorer and critic with the graph replaced by:

  • (i) no graph (node catalog kept, all edges removed);
  • (ii) a random topology with the same nodes and number of edges;
  • (iii) the real topology with permuted edge weights (same weight distribution).

Pass rates (%). TC = ToolComp, TA = Toolathlon; M / G / D = Ministral / Gemma / DeepSeek:

Armτ²-Mτ²-Gτ²-DTC-MTC-GTC-DTA-MTA-GTA-D
Real graph53.073.080.061.473.171.16.118.221.2
No graph48.065.085.056.372.173.10.06.19.1
Random topology56.974.677.750.068.077.00.03.09.1
Permuted edge weights53.068.080.061.977.275.13.06.112.1
Planner + verifier, no graph, no critic47.062.075.0

Notes:

  • ToolComp is over all 197 evaluation tasks; the submitted Q_llm cells were on a 177-task subset.
  • Toolathlon is over 33 tasks.
  • On τ², the no-graph arm ran at n = 4 / 8 / 8 candidates.
  • On Toolathlon there is no critic, so the no-graph arm is flat top-k re-ranked by the verifier.

Why random graphs barely change τ² and ToolComp.

  • τ². Tool selection and chaining explain a minority of failures. Of the 94 TG Q_llm failures we read by hand, 55 (58.5%) are wrong or missing writes with every read correct: 16 writes the policy forbids, 17 with wrong arguments, and 16 required writes that were never made. Only 9 (9.6%) involve a wrong tool or order. When the dominant errors are wrong arguments and policy violations and the tool list is small, a graph over tool dependencies has little to correct.
  • Telecom is the one place where topology shows up, because tool order matters there. With a random topology, agents write to the customer's wrong phone line in 24 of 80 Ministral and DeepSeek runs, against 8 and 12 of 80 with the real graph. On telecom, the real graph scores 40.0 / 65.0 / 72.5 against 30.0 / 42.5 / 62.5 for the planner with the same verifier and no graph.
  • ToolComp has a very small tool pool: 11 tools in total, 79 of the 197 tasks use only google_search and python_interpreter, and the graph has 35 edges, of which a random topology keeps 7. With so little for the graph to select among, the controls move results in both directions (−11 to +6 points) without a consistent sign.

Toolathlon. The catalog is more than an order of magnitude larger: about 620 tools (604 MCP and 16 local), 576 atomic and 42 meta nodes, and about 10,000 edges. Our reader re-ranks the personalized-PageRank top 10 tools, at most two per server. Tool selection matters more at this scale, and that is where the tool graph pays off: every control falls to at most 3.0 / 6.1 / 12.1, against 6.1 / 18.2 / 21.2.

Schema-only graph. Parameter matching alone supports 0%, 4.5% and 3.9% of the kept edges, so a schema-only graph is nearly empty, and the no-graph arm bounds it.

W1 (novelty). We have cited and discussed Graph RAG-Tool Fusion and the June 2026 preprint (Tang, arXiv:2606.23112).

  • Graph RAG-Tool Fusion retrieves tools over a hand-built, unweighted knowledge graph. ToolGraph is built automatically, weights every edge, and is read at every decision.
  • Tang (2026) estimates weights from successful rollouts and trains the agent with DPO. ToolGraph adds interface-derived probe evidence, and the planner is never trained.

The LLM scorer is standard step-level best-of-n by design. The table above isolates what the graph adds to it.

W2 (88 vs 80). With no edges, DeepSeek reaches 85 on τ², so part of the cross-catalog gain is the scorer's. The cross-catalog subgraphs are also larger, and we have no size-matched control. The revised Section 7.5 states both.

Q2. Why does Figure 3's static point differ from Table 1? Can you report three seeds?

  • Figure 3's static point and every update checkpoint use the anchor configuration (1 hop, θ = 0, N0, 4 candidates), so +7 / +6 is the like-for-like update gain.
  • Table 1's τ² cells come from a separate run.
  • A second anchor run (Table 5, n = 4 block, scored with Table 1's task weights) gives 71 / 79, a gain of +4 / +5.
  • We ran one trial per task with one seed, so we cannot yet report three seeds.

Q3. Were the Table 2 costs measured with the Table 6 settings, and do the call counts include planner calls for both methods?

  • The τ² and ToolComp Q_llm cells of Tables 1 and 2 use the anchor configuration (n = 4) for every planner. That is why the G and D token costs match Table 5's n = 4 row; the Table 6 caption is corrected.
  • The LATS τ² costs count its search calls.
  • We did not log calls by role, so we cannot break down or confirm planner calls for both methods.

Q4. What share of edges came from the same-class prompt? How many CREATE and DELETE operations fired?

CatalogKept edgesSame-class involvedPositional probe onlySchema only
ToolComp3520 (57.1%)15 (42.9%)0
τ²-Bench1,3031,064 (81.7%)180 (13.8%)59 (4.5%)
Toolathlon10,2265,360 (52.4%)4,460 (43.6%)394 (3.9%)

DELETE fired 0 times. CREATE fired 36 times on τ², 6 on ToolComp, and 11 on Toolathlon.

W3 (numbers). We audited every number you flagged against our run records. Each one, and how the revision reconciles it:

Flagged numberCauseRevision
Figure 3 static (68/78) vs Table 1 (73/80)Separate runs; Figure 3 uses the anchor configuration throughoutCaptions say so. Update gain: +7/+6 like-for-like, +4/+5 against a second anchor run
Table 2 costs vs Table 6 n = 8The τ² and ToolComp Q_llm cells use the anchor (n = 4)Table 6 caption corrected
Toolathlon % not k/43Ten of the 43 test tasks need Docker-hosted servers; rates are k/33Denominator stated as 33
ToolComp denominatorsQ_llm and Q_emb were run on a 177-task audited subset, other arms on 197Q_llm rerun on 197 (61.4/73.1/71.1); Q_emb marked as subset
Toolathlon flat and TG-PPR cellsTranscription errors from our results sheetCorrected to the run records
Opus-verifier Q_llm/D 88 (and two other cells)Transcription errors from our results sheetCorrected to the run records (Q_llm/D = 86)
Toolathlon kept edges 10,226 vs evidence total 10,21412 edges lack a provenance tag in the exportFootnote corrected; nothing was added by hand
Table 8, two construction countsTwo differently instrumented construction runsBoth kept and labelled

With the 197-task ToolComp numbers, Q_llm beats LATS on all three τ² settings at 15–28× fewer calls and is within 2–5 points of it on ToolComp (+2.0 / −2.0 / −4.5) at 6–29× fewer calls. We agree that our LATS, with imagined rollouts and an unswept budget, is one operating point, and Table 2 is now described that way.

W4.

  • Same-class pairs. These are scored by a single call that returns a match score and a gain score; we keep the component when its match score is ≥ 0.3. The paper now states this scoring rule, and the evidence shares are in the Q4 table above.
  • Probe prompt. Its examples name ToolComp tools; they were used unchanged for all catalogs, and the paper now discloses this.
  • The 12 Toolathlon edges. The footnote calling them hand-added was wrong: all 10,226 edges come from the automatic pipeline. Twelve of them (0.1%) lack a provenance tag in the exported graph, which is why the evidence columns sum to 10,214. The corrected footnote says exactly that.
  • Updates. A failure adds 0.3 and a pass 1.0, so an edge with long-run pass rate p drifts to p / (p + 0.3(1 − p)). DELETE needs p < 1.6%, which is why it never fired.

I appreciate the authors' diligent and detailed responses, as well as the supplementary experiments. Several issues have been addressed, including the verification of figures, the denominator used for Toolathlon, the origin of the 12 edges, the addition of references, and the analysis of failure cases. However, after reviewing all the comments and responses, my core concerns remain; therefore, I am maintaining my original score.

The contribution of the graph structure itself remains unclear. New control experiments show that, on τ²-Bench and ToolComp, using no graph, a random topology, or shuffled weights yields results comparable to—or in some settings, even better than—those obtained with the actual graph. For instance, on τ²-Bench, DeepSeek achieves a score of 85 without a graph, compared to 80 with the actual graph. Only Toolathlon shows a significant performance gap; however, this benchmark consists of only 33 tasks—meaning the scores of 6.1/18.2/21.2 correspond to just 2, 6, and 7 tasks, respectively—and the results are based on a single random seed. The gains from continuous updates fall within the range of statistical noise. Compared to the second "anchor" run, the observed gains are only +4/+5, which is consistent with the 2–3 point fluctuation the authors themselves reported for repeated runs. The experiments still lack multi-seed results and a control condition for the updates themselves (i.e., a baseline to determine if the updates are actually meaningful). Furthermore, the "DELETE" operation was never triggered, and the updates primarily reflect the frequency of tool usage. The novelty is limited. The scoring mechanism is essentially a standard step-level "best-of-n" approach combined with a verifier. Since the results on τ²-Bench and ToolComp remain largely unchanged when the weights are shuffled, it suggests that the only unique component—the weighting of probes—is ineffective. Efficiency comparisons cannot be verified. The authors did not record the number of calls broken down by role; consequently, it is impossible to confirm whether the cost metrics for the two methods were calculated using a consistent basis.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.