acceptodds
Under review as a conference paper at ICLR 2027

Mosaic: Confidence-Aware Contrastive Learning over Multiplex Tool Graphs for Robust Task Planning

Abstract

Task planning requires LLM-based agents to translate a complex user request into an ordered sequence of tool invocations. Graph-based task planners address this problem by learning over a tool graph that encodes which tools can be invoked in sequence, and consistently outperform planners that rely on LLMs alone. However, the tool graph is rarely available and is built heuristically, either from historical tool invocations or from input-output type compatibility; therefore, its edges vary widely in reliability, yet existing planners treat them as equally trustworthy binary connections and typically rely on a single construction. We propose Mosaic (**M**ultiplex c**O**nfidence-aware **S**tructure-**A**ugmented **I**dentity-aware **C**ontrastive learning), a model-agnostic plug-in method that makes the tool representations of graph-based planners robust to unreliable tool graphs. Mosaic (i) combines the historical and dependency graphs into a confidence-weighted multiplex tool graph; (ii) trains relation-specific graph encoders with a contrastive objective whose augmentations drop low-confidence edges more aggressively and which aligns each tool across the two relations; and (iii) fuses the resulting embeddings with learnable, graph-agnostic tool identity embeddings. Across three TaskBench domains, three LLMs, and two state-of-the-art planners, Mosaic consistently improves planning performance (with relative accuracy gains of up to 22.25%), remains more robust under structural noise injected into the tool graphs, and adds only marginal computational overhead.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Task Planning, Large Language Models, Graph Neural Networks

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UuCeOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MOSAIC, a plug-in framework that improves tool representations for graph-based task planning. It combines historical and input–output dependency graphs, assigns heuristic confidence weights to their edges, and applies top-k sparsification. Relation-specific graph encoders are trained with confidence-aware edge-dropping augmentations and intra- and cross-graph contrastive objectives. The resulting representations are fused with learnable tool identity embeddings and integrated into existing planners. Experiments on three TaskBench domains, two planning architectures, and multiple LLM backbones show consistent improvements in tool-selection and dependency-prediction metrics. Component ablations, synthetic graph-noise experiments, and efficiency measurements provide further validation.

Critical strengths

A well-motivated and reusable design. MOSAIC addresses the unreliability of heuristic tool graphs by combining dependency and historical information with confidence-aware representation learning. Its integration into two different planners without redesigning their downstream planning procedures demonstrates practical applicability beyond a single architecture.

Consistent improvements across experimental settings. MOSAIC improves all three reported metrics across the main combinations of domains, LLMs, and planners, with substantial gains in several settings. For example, GTool with Mistral-7B improves tool-set accuracy from 59.40% to 69.20% on Multimedia. These results, supported by component ablations, provide meaningful evidence for the effectiveness of the overall approach.

Critical weaknesses

The role of the core confidence mechanism is not sufficiently established. Semantic similarity, type rarity, and transition frequency are plausible weighting heuristics, but the paper does not directly establish that they identify unreliable connections. The confidence-weighting ablation simultaneously removes edge weighting, sparsification, and adaptive dropout, making their contributions difficult to distinguish. Controlled comparisons that vary weighting and dropout separately, while retaining the same two graphs and comparable sparsity, would help determine whether the gains reflect reliability estimation rather than general regularization.

The robustness evidence is limited by the perturbation protocol and single-seed evaluation. Noise experiments perturb one graph at a time while leaving the other intact, so improvements may arise from access to complementary clean information rather than the proposed confidence mechanism. All experiments also use a single random seed, leaving the stability of smaller gains and fluctuating noise results unclear. Comparing against an otherwise identical dual-graph model with uniform weighting, testing simultaneous corruption of both graphs, and reporting multiple training and perturbation seeds would materially strengthen the robustness claim.

Questions for authors

Confidence mechanism. Could you isolate the effects of edge weighting and adaptive dropout while keeping the two graphs and sparsity comparable? This would help establish whether the proposed confidence scores offer benefits beyond graph fusion and general regularization. An analysis of rare but valid transitions would also clarify whether frequency-based weighting suppresses useful connections.

Robustness and stability. A comparison with an otherwise identical dual-graph model using uniform weights would help interpret the noise results. Please also consider simultaneous corruption of both graphs and multiple training and perturbation seeds to establish whether the observed robustness gains are stable.

Implementation clarification. How is the type-specificity weight computed for Daily Life’s fully connected graph? Please explain how Equation (1) produces the nonzero weights in Table 5. Also, please clarify the effective evaluation denominators: some reported Accuracy values do not appear consistent with averaging binary outcomes over 500 examples in a single run.

6 comments

(Part 1/4)

We thank the reviewer for the thorough review and the precise questions. We ran the requested experiments and clarified the implementation points. The new material is in the revised PDF, highlighted in blue, with the new experiments in Appendix E.4 to E.9.

1. Confidence mechanism (W1, Q1)

In the original w/o Confidence Weighting ablation all edges have weight 1. Top-k sparsification ranks edges by weight, and confidence-aware dropout derives drop probabilities from the weights, so with unit weights the first has nothing to rank and the second reduces to uniform dropout. Removing the weights therefore necessarily removes the other two mechanisms.

To isolate the dropout, we keep the same two graphs, the same weights and the same top-k sparsification, and replace only the confidence-aware dropout with uniform edge dropout (w/o Confidence-aware Dropout). Since the reviewer also asks whether the gains go beyond general regularization, we add a second control, also suggested by Reviewer Adig, that keeps the dropout adaptive but removes the confidence from it: w/ GCA-based Dropout uses the adaptive edge dropout of GCA (Zhu et al., 2021), where drop probabilities depend on graph topology (log degree centrality of the target node) instead of edge confidence, with the same average and maximum drop rates as ours. Together with w/o Graph Sparsification in Table 2, which keeps the weights and removes top-k, each mechanism is now evaluated with the others fixed. Results on Multimedia (MM) and Daily Life (DL) with Mistral-7B over 3 seeds (Appendix E.6, Table 11):

GNN4Plan

DatasetSettingn-F1l-F1Acc
MMBaseline76.92 ± 1.5056.92 ± 1.0440.47 ± 1.38
w/o Confidence-aware Dropout76.90 ± 2.1556.64 ± 2.5540.74 ± 2.34
w/ GCA-based Dropout76.87 ± 2.1856.41 ± 2.7339.39 ± 3.85
Baseline+MOSAIC78.22 ± 0.7458.41 ± 0.7242.56 ± 0.65
DLBaseline90.19 ± 0.3769.88 ± 0.9460.22 ± 0.83
w/o Confidence-aware Dropout90.06 ± 0.1569.73 ± 0.2659.60 ± 0.31
w/ GCA-based Dropout90.30 ± 0.2370.21 ± 0.4660.36 ± 0.83
Baseline+MOSAIC90.40 ± 0.3070.41 ± 0.4860.63 ± 0.72

GTool

DatasetSettingn-F1l-F1Acc
MMBaseline88.66 ± 0.4572.36 ± 1.4360.07 ± 1.89
w/o Confidence-aware Dropout90.51 ± 0.2276.15 ± 0.8466.80 ± 0.20
w/ GCA-based Dropout90.67 ± 0.7176.94 ± 0.5667.60 ± 0.92
Baseline+MOSAIC90.72 ± 0.6877.79 ± 0.9569.40 ± 1.51
DLBaseline96.69 ± 0.7085.86 ± 1.6481.80 ± 2.60
w/o Confidence-aware Dropout98.09 ± 0.1588.60 ± 0.7687.93 ± 0.31
w/ GCA-based Dropout97.97 ± 0.0988.02 ± 0.6187.13 ± 0.50
Baseline+MOSAIC98.03 ± 0.1088.61 ± 0.3187.53 ± 0.12

MOSAIC is the best setting on every metric for both planners on Multimedia and for GNN4Plan on Daily Life. The only exception is GTool on Daily Life, where w/o Confidence-aware Dropout is slightly higher in n-F1 (98.09 against 98.03) and accuracy (87.93 against 87.53), on a dataset where performance is close to saturation, while MOSAIC has the best l-F1. w/ GCA-based Dropout, which keeps the dropout adaptive but driven by topology instead of confidence, stays below MOSAIC on every metric in all four settings. With GNN4Plan on Multimedia, both variants fall back to the level of the baseline, so most of the gain there comes from the confidence-aware dropout. Since GCA-based dropout uses the same graphs, the same sparsity and the same amount of regularization, the gap between it and MOSAIC can be attributed to the confidence scores rather than to graph fusion or generic regularization.

On rare but valid transitions: giving infrequent transitions a lower weight is the intended behavior of the historical graph, which leverages usage frequency as a proxy for reliability. A transition that users have performed only a few times is less likely to be a sound step, although it may still be valid, and frequency is the only evidence of this kind that execution logs provide. For this reason, MOSAIC treats frequency as a soft signal: the historical weight νijνˉ\frac{\nu_{ij}}{\bar{\nu}} lowers the weight of rare transitions but keeps them in the graph, top-k removes a rare edge only if its source tool has at least kk outgoing transitions with a higher weight, and the dropout probability is capped at pmaxp_{\text{max}}, so no edge is dropped deterministically.

(Part 2/4)

2. Robustness and stability (W2, Q2)

We made three changes to the robustness study, following the reviewer's suggestions.

First, both graphs are now corrupted at the same time. For each ratio from 10% to 90%, false edges are added to both graphs between tools that are not linked in the clean graphs. Their types (dependency graph) and co-occurrence counts (historical graph) are drawn from those of the true edges, and all weights are recomputed with Equations 1 and 2, so false and true edges have the same weight distribution (median dependency weight 0.20 against 0.19 at 10% noise on HuggingFace). MOSAIC therefore cannot rely on a clean second graph, nor single out the false edges by their weights.

Second, we compare with a dual-graph version of each baseline that is identical to the original planner except that it encodes both graphs, unweighted and not sparsified, with two separate GNNs and fuses the two embeddings as MOSAIC does. It has no confidence weights, no contrastive objective and no identity embeddings, so the gap to MOSAIC measures what our method adds beyond using both graphs.

Third, all results are averaged over 3 seeds, and each seed is used both to generate the perturbed graphs and to train the model.

Accuracy on HuggingFace with CodeLlama-13B (Appendix E.7, Figure 6):

Noise (%)GNN4Plan dual-graphGNN4Plan+MOSAICGTool dual-graphGTool+MOSAIC
030.53 ± 1.3032.40 ± 0.3145.07 ± 1.4749.80 ± 0.92
1030.79 ± 1.8531.40 ± 0.8147.40 ± 0.5348.13 ± 1.33
2531.00 ± 0.3131.20 ± 0.5046.93 ± 3.6147.60 ± 0.80
5030.59 ± 1.1631.33 ± 0.2345.93 ± 1.9748.53 ± 1.60
7530.59 ± 1.1630.26 ± 1.7547.67 ± 3.2048.80 ± 2.96
9029.26 ± 1.1230.19 ± 1.5048.87 ± 1.9448.33 ± 1.10

MOSAIC stays above the dual-graph version on the clean graphs and at most noise ratios with both planners: GTool+MOSAIC from 10% to 75%, and GNN4Plan+MOSAIC at 10%, 25%, 50% and 90%, where the margin is the largest. In the remaining cases the two are within one standard deviation. From the clean graphs to 90% noise, the accuracy of MOSAIC decreases by 2.21 points with GNN4Plan and by 1.47 with GTool, although both graphs are corrupted at once and the false edges cannot be told apart from the true ones by their weights. On the clean graphs, the gap to the dual-graph version (32.40 against 30.53 with GNN4Plan, 49.80 against 45.07 with GTool) also shows that the gains of MOSAIC go beyond the use of both graphs.

(Part 3/4)

We also repeated the main comparison over 3 seeds, for both planners on HuggingFace (HF) and Multimedia (MM) with all three LLMs (Appendix E.4, Table 9). Mean ± std over seeds; Δ is the relative change of each seed, averaged over seeds:

LLMModelHF n-F1HF l-F1HF AccMM n-F1MM l-F1MM Acc
Qwen3-14BGNN4Plan79.94 ± 0.2461.45 ± 0.6846.00 ± 1.0684.22 ± 1.6470.01 ± 1.5755.60 ± 1.11
GNN4Plan+MOSAIC80.58 ± 0.5262.44 ± 0.6246.33 ± 1.1085.38 ± 0.2671.38 ± 0.1356.93 ± 0.12
Δ (%)+0.81 ± 0.95+1.63 ± 2.06+0.79 ± 4.63+1.41 ± 1.95+1.98 ± 2.28+2.42 ± 1.88
GTool81.01 ± 0.8559.31 ± 0.4945.00 ± 1.2286.47 ± 0.8570.06 ± 1.1458.07 ± 2.90
GTool+MOSAIC81.66 ± 1.3560.41 ± 2.0745.07 ± 2.0487.11 ± 0.5170.69 ± 0.6459.40 ± 0.72
Δ (%)+0.81 ± 1.92+1.84 ± 2.69+0.12 ± 1.82+0.75 ± 1.25+0.92 ± 1.85+2.46 ± 4.96
CodeLlama-13BGNN4Plan72.96 ± 0.7450.61 ± 0.8431.46 ± 0.4077.96 ± 0.5558.67 ± 0.8341.72 ± 1.28
GNN4Plan+MOSAIC74.11 ± 0.6651.12 ± 0.8232.40 ± 0.3179.21 ± 0.2360.47 ± 0.3044.38 ± 0.35
Δ (%)+1.58 ± 1.86+1.04 ± 2.92+2.98 ± 0.40+1.60 ± 0.93+3.09 ± 1.69+6.45 ± 4.03
GTool81.35 ± 0.5560.10 ± 1.2444.33 ± 1.0187.85 ± 1.3772.12 ± 2.1658.53 ± 3.83
GTool+MOSAIC82.64 ± 0.4163.58 ± 0.1149.80 ± 0.9289.54 ± 0.4574.92 ± 0.6063.33 ± 0.46
Δ (%)+1.59 ± 1.06+5.82 ± 2.34+12.36 ± 2.88+1.94 ± 1.27+3.93 ± 2.53+8.50 ± 7.02
Mistral-7BGNN4Plan68.20 ± 0.5745.44 ± 0.4829.45 ± 0.8276.92 ± 1.5056.92 ± 1.0440.47 ± 1.38
GNN4Plan+MOSAIC69.85 ± 0.2547.29 ± 0.1330.74 ± 0.2478.22 ± 0.7458.41 ± 0.7242.56 ± 0.65
Δ (%)+2.42 ± 0.49+4.07 ± 0.82+4.45 ± 3.04+1.74 ± 2.90+2.66 ± 3.08+5.27 ± 5.01
GTool80.22 ± 0.5058.14 ± 1.5939.60 ± 0.8788.66 ± 0.4572.36 ± 1.4360.07 ± 1.89
GTool+MOSAIC82.68 ± 0.8062.96 ± 1.2847.40 ± 0.8790.72 ± 0.6877.79 ± 0.9569.40 ± 1.51
Δ (%)+3.08 ± 1.02+8.33 ± 3.11+19.72 ± 2.46+2.32 ± 0.59+7.53 ± 1.03+15.56 ± 1.25

MOSAIC improves the mean of every metric in every configuration. With Qwen3-14B the gains are moderate but positive and within the seed variance, as they already were in Table 1. With CodeLlama-13B and Mistral-7B the accuracy gain is larger than the standard deviation in every configuration.

Reviewer_ecGUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MOSAIC, a plug-in for graph-based tool planners that aims to make tool representations less sensitive to unreliable graph structure. It builds a confidence-weighted multiplex graph from dependency/schema information and historical tool transitions. Relation-specific GNN encoders are trained with confidence-aware contrastive learning, using within-relation augmentations and cross-relation alignment. The resulting representations are fused with learnable tool identity embeddings. Experiments integrate MOSAIC into GNN4Plan and GTool across three TaskBench domains and three LLMs. The reported results show improvements in planning metrics, with a largest relative accuracy gain of 22.25%, along with structural-noise experiments and component ablations. The contribution is a practical representation-learning module for improving existing planners when their tool graphs contain uncertain edges.

Critical strengths

The treatment of graph uncertainty is well motivated. Dependency graphs and historical transitions provide different kinds of evidence, and neither source is uniformly reliable. MOSAIC keeps the relations separate during encoding and uses confidence scores to guide which edges are dropped during contrastive training. I find this design sensible because it gives the model a way to use both sources without treating every observed edge as equally trustworthy. Its integration into existing planners also makes the contribution relatively easy to adopt.

The component ablations are an important strength. They examine confidence weighting, the intra- and cross-relation objectives, and the identity-embedding fusion, so the evaluation goes beyond comparing the complete system with its backbones. The reported gains across two planners, three LLMs and three domains also make the results more persuasive than a single-configuration demonstration. I particularly appreciate the structural-noise experiments, since they directly address the problem the method is designed to solve, even though the corruption model could be more realistic.

Critical weaknesses

My main concern is how much the results depend on clean historical information and a fixed tool catalog. The historical graph uses ground-truth transitions from the training set, while identity embeddings are learned for individual tools. This setting may provide better historical evidence than would be available from incomplete or imperfect agent logs. The authors acknowledge the transductive nature of the identity embeddings, but generalization to unseen tools is not quantified. I would like the paper to distinguish more clearly between improving planning within a known catalog and supporting a changing catalog. This is a question about the scope of the evidence; the use of training transitions alone does not establish that test information has leaked.

The robustness claims also need stronger validation. Random edge addition is a useful controlled test, but it does not capture systematic errors such as over-connecting tools with common argument types or adding misleading co-occurrence edges. The evaluation reports plan-structure metrics rather than end-to-end execution success, so it is unclear how much the improvements translate into completed tasks. Variability across seeds is also missing. A breakdown across several seeds would give clearer picture of whether the smaller gains are stable. Together, these gaps limit how confidently I can interpret the results as evidence of robust planning in realistic tool environments.

Questions for authors

Could you clarify exactly which data are used to construct the historical graph and whether the tool catalog is shared across training and evaluation? How much of the improvement remains when the historical graph is built from a small subset of training plans or from imperfect agent trajectories? This would help assess the dependence on high-quality historical information.

Do the robustness gains remain under structured corruptions, such as over-connecting common schema types or introducing spurious co-occurrence edges? Also, is there execution-level evidence that the improvements in n-F1, l-F1 and Acc lead to better task completion, or should the conclusions be restricted to plan-structure quality?

Could you report the variation across random seeds for the main comparisons, with the results for each backbone shown seperately? A sensitivity check for the sparsification and contrastive-dropout settings would also help establish whether the gains depend on a narrow hyperparameter configuration.

5 comments

(Part 1/4)

We thank the reviewer for the detailed review and for the questions on the scope of our evidence. We ran new experiments for each of them. The new material is in the revised PDF, highlighted in blue, with the new experiments in Appendix E.4 to E.9.

1. Data used for the historical graph and tool catalog (Q1)

Following GNN4Plan and GTool, the queries are split into training, validation and test sets. The historical graph is built only from the ground-truth plans of the training queries: an edge (ti,tj)(t_i, t_j) exists if tjt_j follows tit_i in at least one training plan, and its weight is the number of such transitions divided by the average count. Validation and test plans are never used, so there is no leakage. The dependency graph does not depend on the split, since it is given by the input and output types of the tools. We clarified this aspect further in Section D.1 from the Appendix.

The tool catalog is shared by all splits, as in the original benchmarks. This setting is transductive with respect to the tools: every tool is known at training time, and the task is to predict the plan of unseen test queries over that catalog. We agree that the paper should separate this setting from a changing catalog more clearly, and we address the latter in point 3.

2. Dependence on high-quality historical information (Q1)

We removed a growing fraction of the edges of the historical graph, from 10% to 90%, and retrained MOSAIC on each incomplete graph. Removing these edges corresponds to constructing the historical graph from incomplete or imperfect historical trajectories; conversely, the baselines keep the complete dependency graph. Results on HuggingFace with CodeLlama-13B over 3 seeds (Appendix E.7, Table 12):

MethodMissing edges (%)n-F1l-F1Acc
GNN4Plannone72.96 ± 0.7450.61 ± 0.8431.46 ± 0.40
GNN4Plan+MOSAIC074.11 ± 0.6651.12 ± 0.8232.40 ± 0.31
1072.48 ± 1.1949.81 ± 1.5131.40 ± 0.76
2573.18 ± 0.6550.43 ± 0.6432.20 ± 0.61
5072.10 ± 1.8749.32 ± 1.9830.66 ± 1.78
7573.50 ± 0.3551.24 ± 0.3732.46 ± 0.40
9072.43 ± 1.3149.47 ± 1.7831.60 ± 0.99
GToolnone81.35 ± 0.5560.10 ± 1.2444.33 ± 1.01
GTool+MOSAIC082.64 ± 0.4163.58 ± 0.1149.80 ± 0.92
1083.00 ± 0.7463.53 ± 1.0448.47 ± 2.70
2582.35 ± 0.9062.82 ± 1.4047.53 ± 2.44
5082.23 ± 0.9962.20 ± 2.7746.80 ± 1.97
7583.13 ± 0.6363.72 ± 1.2248.60 ± 1.80
9081.96 ± 2.0561.68 ± 3.5846.93 ± 3.29

GTool+MOSAIC stays above GTool on every metric at every level, even with 90% of the historical edges missing (46.93 against 44.33 accuracy). GNN4Plan+MOSAIC remains on par with GNN4Plan: it is never below it by more than one standard deviation, and its accuracy is higher at three of the five levels, although GNN4Plan does not use the historical graph and is therefore unaffected by the removal. As expected, removing historical information has some cost (the accuracy of GTool+MOSAIC goes from 49.80 on the complete graph to between 46.80 and 48.60), but the drop is moderate even when most of the graph is missing, so the gains of MOSAIC do not require a complete historical graph.

(Part 2/4)

3. Generalization to unseen tools

To test a changing catalog, we simulate tools added after training. For each seed, we randomly hold out 20% of the tools (8 of the 40 tools of Multimedia): every training and validation query whose plan uses one of them is discarded, and the held-out tools are removed from both graphs with all their edges, so the planner never sees them during training (about 1,800 of the 3,000 training queries remain). At test time, the held-out tools are added back together with their edges in both graphs; their historical edges come from the training trajectories that were discarded, which simulates tools that have accumulated some usage history after deployment. The model is evaluated on the original test set, split into Seen queries, which use only tools available during training, and Unseen queries, which use at least one held-out tool (about 275 of the 500). MOSAIC needs almost no change for this setting: its GNN encoders build a tool representation from the description embedding and the edges, so a new tool gets one as soon as it is connected to the graphs. The only component tied to the training catalog is the identity embedding, which we set to zero for new tools. The baseline receives the same restored catalog and graphs. We ran this experiment with GTool (the most recent planner) on Multimedia with DeepSeek-R1, over 3 seeds (Appendix E.9, Table 14):

SubsetModeln-F1l-F1Acc
SeenGTool74.96 ± 3.2751.65 ± 3.8134.95 ± 2.58
GTool+MOSAIC80.01 ± 0.2758.17 ± 3.2045.61 ± 4.92
Δ (%)+6.87 ± 4.49+12.74 ± 3.03+30.39 ± 8.32
UnseenGTool65.66 ± 3.8038.72 ± 5.7219.41 ± 5.98
GTool+MOSAIC73.69 ± 1.7049.37 ± 0.4931.28 ± 2.13
Δ (%)+12.52 ± 8.08+29.42 ± 19.71+71.69 ± 53.70

MOSAIC improves GTool on both subsets, on every metric and for every seed. Both models perform worse on Unseen queries, as expected, but MOSAIC improves GTool by a similar or larger margin there (+11.87 accuracy points, against +10.66 on Seen), even though the identity embedding of the new tools is zero, which indicates that the confidence-weighted graph encoders transfer to tools that were not in the training catalog. Because the baseline is lower on Unseen queries, the same margin becomes a larger relative gain (+71.69% accuracy), and it also explains the large standard deviations of Δ: GTool's Unseen accuracy ranges from 13.04 to 24.91 across seeds, while MOSAIC stays between 29.71 and 33.70, and a relative change amplifies this when the baseline is low.

(Part 3/4)

4. Structured corruptions (Q2)

Regarding over-connection through common schema types, we clarify that our dependency graphs already contain all possible edges satisfying input-output type compatibility, including potentially spurious connections induced by common types such as “text”. Therefore, the over-connection scenario described by the reviewer is already present in the original dependency graphs and directly motivates our confidence-weighting and sparsification mechanisms. Adding further type-compatible edges would not introduce new connections, as these are already included in the graph.

That being said, inspired by the interesting intuition provided by this reviewer, and the suggestion by Reviewer UuCe, we decided to add a further perturbation protocol that corrupts both graphs at the same time, with false edges whose types and weights follow the same distributions as those of the true edges, so that they cannot be told apart by these properties. For each ratio from 10% to 90%, we add false edges between tools that are not linked in the clean graphs. In the dependency graph, the type of each false edge is drawn from the type distribution of the true edges, and all weights are recomputed with Equation 1. Since common types dominate that distribution, most false edges carry common types such as “text”, which is close to the “over-connection through common argument types” that the reviewer describes. In the historical graph, each false edge gets a co-occurrence count drawn from the counts of the true edges, so it behaves like a “spurious but plausible co-occurrence”. False and true edges then have the same type and weight distributions (for example, median dependency weight 0.20 against 0.19 at 10% noise on HuggingFace). Since the original baselines use only the dependency graph, we compare with a dual-graph version of each: the same planner, which encodes both graphs, unweighted, with two separate GNNs and fuses the two embeddings as MOSAIC does, without confidence weights, contrastive learning or identity embeddings. Accuracy on HuggingFace with CodeLlama-13B over 3 seeds (Appendix E.7, Figure 6):

Noise (%)GNN4Plan dual-graphGNN4Plan+MOSAICGTool dual-graphGTool+MOSAIC
030.53 ± 1.3032.40 ± 0.3145.07 ± 1.4749.80 ± 0.92
1030.79 ± 1.8531.40 ± 0.8147.40 ± 0.5348.13 ± 1.33
2531.00 ± 0.3131.20 ± 0.5046.93 ± 3.6147.60 ± 0.80
5030.59 ± 1.1631.33 ± 0.2345.93 ± 1.9748.53 ± 1.60
7530.59 ± 1.1630.26 ± 1.7547.67 ± 3.2048.80 ± 2.96
9029.26 ± 1.1230.19 ± 1.5048.87 ± 1.9448.33 ± 1.10

MOSAIC stays above the dual-graph version on the clean graphs and at most noise ratios with both planners: GTool+MOSAIC from 10% to 75%, and GNN4Plan+MOSAIC at 10%, 25%, 50% and 90%, where the margin is the largest. In the remaining cases the two are within one standard deviation. From the clean graphs to 90% noise, where false edges almost double the edges of each graph, the accuracy of MOSAIC decreases by 2.21 points with GNN4Plan and by 1.47 with GTool. MOSAIC therefore remains effective under over-connection through common types and spurious but plausible co-occurrences, even when these affect both graphs at once.

5. Execution-level evidence (Q2)

Our work, like GNN4Plan and GTool, addresses task planning: given a query, predict which tools to call and in which order. The tools in TaskBench (Shen et al., 2024) are not executable services, so task completion cannot be measured on these benchmarks, and our conclusions are about the quality of the predicted plans. Nevertheless, to provide a closer performance assessment to what task execution would imply, we computed another metric “Exact Match” (EM) which requires the full ordered plan to be correct. MOSAIC improves it as well. Results on Multimedia with Qwen3-14B over 3 seeds (Appendix E.5, Table 10):

ModelEMΔ (%)
GNN4Plan53.47 ± 1.10
GNN4Plan+MOSAIC54.73 ± 0.12+2.40 ± 1.93
GTool55.80 ± 2.82
GTool+MOSAIC57.07 ± 0.23+2.44 ± 5.12

We state this scope explicitly in the paper, as evaluating MOSAIC on benchmarks with executable tools is a natural next step (i.e., see the updated Conclusion and Limitations sections).

Reviewer_AdigOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MOSAIC, a plug-in for graph-based tool planners. It combines confidence-weighted dependency and historical tool graphs, contrastive learning across graph views, and learnable tool identity embeddings. On three TaskBench domains, three LLMs, and two planning backbones, the full method improves the reported tool-selection and link-prediction metrics over GNN4Plan and GTool.

Critical strengths

The paper addresses a concrete limitation of existing graph-based planners: heuristic tool-graph edges have unequal reliability, yet are commonly treated as binary connections. Improvements across two different planning backbones make the complete approach practically interesting. For example, on HuggingFace with Mistral-7B, GTool’s tool-set accuracy rises from 39.10% to 47.80%, while link F1 rises from 56.31% to 62.44%.

Critical weaknesses

My main concern is the incremental technical contribution and whether its task-specific choices account for the gains. Adaptive graph augmentation, multi-view contrastive learning, and identity embeddings all have close precedents. The ablations provide some support for the complete design, but the “without confidence weighting” setting changes edge weighting, sparsification, and dropout together. A comparison with simpler weighted two-graph models and an existing adaptive augmentation method under comparable capacity would better establish what MOSAIC adds beyond combining known techniques. The evaluation also leaves the planning benefit somewhat uncertain. The headline accuracy metric checks the selected tool set, not the correctness of the full ordered plan. The three tool catalogs contain only 23–40 tools, and results use a single random seed.

Questions for authors
  • Under the same two-graph architecture and parameter budget, how does MOSAIC’s confidence-based augmentation compare with uniform dropout and an existing adaptive augmentation method such as GCA? Can edge weighting, Top-k sparsification, and adaptive dropout be ablated separately?
  • Can you report exact ordered-plan accuracy and variation across random seeds for the main comparisons? These results would clarify whether the gains in tool-set accuracy translate into more reliable plans.
4 comments

(Part 1/3)

We thank the reviewer for having carefully assessed the paper and the concrete suggestions. We ran all the requested experiments. The new material is in the revised PDF, highlighted in blue, with the new experiments in Appendix E.4 to E.9. We answer each point below.

1. Technical contribution and what MOSAIC adds beyond known techniques

We agree that adaptive augmentation, multi-view contrastive learning and identity embeddings each have precedents in graph learning, and we cite them in the paper. What MOSAIC contributes is how they are adapted to graph-based task planning, where the tool graph is built heuristically and its edges have very different reliability. The confidence weights are derived from how each tool graph is built (type specificity and description similarity for the dependency graph, transition frequency for the historical graph), and the same weights drive sparsification and augmentation. The cross-relation term aligns two graphs that existing planners, such as GNN4Plan and GTool, use one at a time. The question is whether these task-specific choices explain the gains, and the new ablations below address it directly.

2. Separating edge weighting, sparsification and dropout (Q1)

In the original w/o Confidence Weighting variant all edges have weight 1. Top-k sparsification ranks edges by weight and confidence-aware dropout assigns drop probabilities from the weights, so with unit weights neither can be applied: sparsification has nothing to rank, and dropout becomes uniform. This is why that variant changes the three components together.

The other two components can be isolated while keeping the same two graphs, the same architecture and the same number of parameters:

  1. w/o Graph Sparsification (Table 2 of the paper) keeps the weights and the confidence-aware dropout and removes top-k.
  2. w/o Confidence-aware Dropout (new) keeps the weights and top-k, and replaces our dropout with uniform edge dropout.
  3. w/ GCA-based Dropout (new) keeps the weights and top-k, and replaces our dropout with the adaptive edge dropout of GCA (Zhu et al., 2021). Edge importance is the log degree centrality of the target node, and the average and maximum drop rates are the same as ours, so only the criterion for choosing which edges to drop changes.

Results on Multimedia (MM) and Daily Life (DL) with Mistral-7B over 3 seeds (Appendix E.6, Table 11):

GNN4Plan

DatasetSettingn-F1l-F1Acc
MMBaseline76.92 ± 1.5056.92 ± 1.0440.47 ± 1.38
w/o Confidence-aware Dropout76.90 ± 2.1556.64 ± 2.5540.74 ± 2.34
w/ GCA-based Dropout76.87 ± 2.1856.41 ± 2.7339.39 ± 3.85
Baseline+MOSAIC78.22 ± 0.7458.41 ± 0.7242.56 ± 0.65
DLBaseline90.19 ± 0.3769.88 ± 0.9460.22 ± 0.83
w/o Confidence-aware Dropout90.06 ± 0.1569.73 ± 0.2659.60 ± 0.31
w/ GCA-based Dropout90.30 ± 0.2370.21 ± 0.4660.36 ± 0.83
Baseline+MOSAIC90.40 ± 0.3070.41 ± 0.4860.63 ± 0.72

GTool

DatasetSettingn-F1l-F1Acc
MMBaseline88.66 ± 0.4572.36 ± 1.4360.07 ± 1.89
w/o Confidence-aware Dropout90.51 ± 0.2276.15 ± 0.8466.80 ± 0.20
w/ GCA-based Dropout90.67 ± 0.7176.94 ± 0.5667.60 ± 0.92
Baseline+MOSAIC90.72 ± 0.6877.79 ± 0.9569.40 ± 1.51
DLBaseline96.69 ± 0.7085.86 ± 1.6481.80 ± 2.60
w/o Confidence-aware Dropout98.09 ± 0.1588.60 ± 0.7687.93 ± 0.31
w/ GCA-based Dropout97.97 ± 0.0988.02 ± 0.6187.13 ± 0.50
Baseline+MOSAIC98.03 ± 0.1088.61 ± 0.3187.53 ± 0.12

MOSAIC is the best setting on every metric for both planners on Multimedia and for GNN4Plan on Daily Life. The only exception is GTool on Daily Life, where w/o Confidence-aware Dropout is slightly higher in n-F1 (98.09 against 98.03) and accuracy (87.93 against 87.53), on a dataset where performance is close to saturation, while MOSAIC has the best l-F1. w/ GCA-based Dropout, which keeps the dropout adaptive but driven by topology instead of confidence, stays below MOSAIC on every metric in all four settings. With GNN4Plan on Multimedia, both variants fall back to the level of the baseline, so most of the gain there comes from the confidence-aware dropout.

(Part 2/3)

On simpler weighted two-graph models: the w/o Lintra∗+Lcross\mathcal{L}^*_{\text{intra}} + \mathcal{L}_{\text{cross}} variant in Table 2 is a weighted two-graph model with the same encoders, fusion, identity embeddings and parameter budget as MOSAIC, trained without any contrastive objective. It stays below the full model on both backbones.

Additionally, for the robustness studies requested by Reviewers ecGU and UuCe, we implemented a dual-graph version of each planner, which is another variant close to the simpler two-graph models suggested by the reviewer. The same planner encodes the dependency and historical graphs, unweighted, with two separate GNNs and fuses the two embeddings as MOSAIC does, but without confidence weights, contrastive learning or identity embeddings. We compared it with MOSAIC on the clean graphs and under a perturbation protocol that corrupts both graphs at the same time, adding from 10% to 90% false edges whose types and weights follow the same distributions as those of the true edges, so that they cannot be told apart by these properties. Accuracy on HuggingFace with CodeLlama-13B over 3 seeds (Appendix E.7, Figure 6):

Noise (%)GNN4Plan dual-graphGNN4Plan+MOSAICGTool dual-graphGTool+MOSAIC
030.53 ± 1.3032.40 ± 0.3145.07 ± 1.4749.80 ± 0.92
1030.79 ± 1.8531.40 ± 0.8147.40 ± 0.5348.13 ± 1.33
2531.00 ± 0.3131.20 ± 0.5046.93 ± 3.6147.60 ± 0.80
5030.59 ± 1.1631.33 ± 0.2345.93 ± 1.9748.53 ± 1.60
7530.59 ± 1.1630.26 ± 1.7547.67 ± 3.2048.80 ± 2.96
9029.26 ± 1.1230.19 ± 1.5048.87 ± 1.9448.33 ± 1.10

On the clean graphs (0%), MOSAIC improves over the dual-graph version with both planners (32.40 against 30.53 with GNN4Plan, 49.80 against 45.07 with GTool), so using both graphs does not by itself explain the gains: the confidence weights, the contrastive objectives and the identity embeddings add to it. Under perturbation, MOSAIC stays above the dual-graph version at most noise ratios, and the two are within one standard deviation in the remaining cases.

3. Exact ordered-plan accuracy and variation across seeds (Q2)

Accuracy checks the predicted tool set, so we added Exact Match (EM): the fraction of test queries whose predicted plan is identical to the ground truth, with the same tools, connected by the same links, and no missing or extra tool or link. Results on Multimedia with Qwen3-14B over 3 seeds (Appendix E.5, Table 10):

ModelEMΔ (%)
GNN4Plan53.47 ± 1.10
GNN4Plan+MOSAIC54.73 ± 0.12+2.40 ± 1.93
GTool55.80 ± 2.82
GTool+MOSAIC57.07 ± 0.23+2.44 ± 5.12

The gains in tool-set accuracy carry over to exact ordered plans.

(Part 3/3)

We also repeated the main comparison over 3 random seeds for both planners, on HuggingFace (HF) and Multimedia (MM) with all three LLMs (Appendix E.4, Table 9). Mean ± std over seeds; Δ is the relative change of each seed, averaged over seeds:

LLMModelHF n-F1HF l-F1HF AccMM n-F1MM l-F1MM Acc
Qwen3-14BGNN4Plan79.94 ± 0.2461.45 ± 0.6846.00 ± 1.0684.22 ± 1.6470.01 ± 1.5755.60 ± 1.11
GNN4Plan+MOSAIC80.58 ± 0.5262.44 ± 0.6246.33 ± 1.1085.38 ± 0.2671.38 ± 0.1356.93 ± 0.12
Δ (%)+0.81 ± 0.95+1.63 ± 2.06+0.79 ± 4.63+1.41 ± 1.95+1.98 ± 2.28+2.42 ± 1.88
GTool81.01 ± 0.8559.31 ± 0.4945.00 ± 1.2286.47 ± 0.8570.06 ± 1.1458.07 ± 2.90
GTool+MOSAIC81.66 ± 1.3560.41 ± 2.0745.07 ± 2.0487.11 ± 0.5170.69 ± 0.6459.40 ± 0.72
Δ (%)+0.81 ± 1.92+1.84 ± 2.69+0.12 ± 1.82+0.75 ± 1.25+0.92 ± 1.85+2.46 ± 4.96
CodeLlama-13BGNN4Plan72.96 ± 0.7450.61 ± 0.8431.46 ± 0.4077.96 ± 0.5558.67 ± 0.8341.72 ± 1.28
GNN4Plan+MOSAIC74.11 ± 0.6651.12 ± 0.8232.40 ± 0.3179.21 ± 0.2360.47 ± 0.3044.38 ± 0.35
Δ (%)+1.58 ± 1.86+1.04 ± 2.92+2.98 ± 0.40+1.60 ± 0.93+3.09 ± 1.69+6.45 ± 4.03
GTool81.35 ± 0.5560.10 ± 1.2444.33 ± 1.0187.85 ± 1.3772.12 ± 2.1658.53 ± 3.83
GTool+MOSAIC82.64 ± 0.4163.58 ± 0.1149.80 ± 0.9289.54 ± 0.4574.92 ± 0.6063.33 ± 0.46
Δ (%)+1.59 ± 1.06+5.82 ± 2.34+12.36 ± 2.88+1.94 ± 1.27+3.93 ± 2.53+8.50 ± 7.02
Mistral-7BGNN4Plan68.20 ± 0.5745.44 ± 0.4829.45 ± 0.8276.92 ± 1.5056.92 ± 1.0440.47 ± 1.38
GNN4Plan+MOSAIC69.85 ± 0.2547.29 ± 0.1330.74 ± 0.2478.22 ± 0.7458.41 ± 0.7242.56 ± 0.65
Δ (%)+2.42 ± 0.49+4.07 ± 0.82+4.45 ± 3.04+1.74 ± 2.90+2.66 ± 3.08+5.27 ± 5.01
GTool80.22 ± 0.5058.14 ± 1.5939.60 ± 0.8788.66 ± 0.4572.36 ± 1.4360.07 ± 1.89
GTool+MOSAIC82.68 ± 0.8062.96 ± 1.2847.40 ± 0.8790.72 ± 0.6877.79 ± 0.9569.40 ± 1.51
Δ (%)+3.08 ± 1.02+8.33 ± 3.11+19.72 ± 2.46+2.32 ± 0.59+7.53 ± 1.03+15.56 ± 1.25

MOSAIC improves the mean of every metric in every configuration. With Qwen3-14B the gains are moderate but positive and within the seed variance, as they already were in Table 1. With CodeLlama-13B and Mistral-7B the accuracy gain is larger than the standard deviation in every configuration.

4. Size of the tool catalogs

The TaskBench (Shen et al., 2024) domains, from which HuggingFace, Multimedia, and Daily Life datasets are taken, have 23 to 40 tools. We added UltraTool (Huang et al., 2024), preprocessed as in GNN4Plan. Since tool input/output types are unavailable in this case, we connect tools sharing at least one domain in both directions, weighting edges by normalized domain overlap and rescaled cosine similarity of tool descriptions. Historical edges capture training-set tool transitions. This yields 260 tools, 22,014 dependency edges (density 0.327) and 534 historical edges (density 0.008), with 2,724/303/500 train/validation/test queries. Results (Appendix E.8, Table 13):

LLMModeln-F1l-F1Acc
Mistral-7BGNN4Plan65.7338.1833.60
GNN4Plan+MOSAIC67.8340.5137.04
Δ (%)+3.19+6.11+10.24
GTool80.2758.8756.60
GTool+MOSAIC82.9563.1560.20
Δ (%)+3.34+7.27+6.36
DeepSeek-R1GNN4Plan59.3933.1732.33
GNN4Plan+MOSAIC64.1435.6337.75
Δ (%)+8.01+7.40+16.77
GTool79.9658.4854.80
GTool+MOSAIC80.2460.1757.00
Δ (%)+0.34+2.89+4.01

MOSAIC improves both planners on all metrics with a catalog more than six times larger than the largest TaskBench one, and with a very sparse historical graph.

References

Zhu et al., Graph Contrastive Learning with Adaptive Augmentation. WWW 2021.

Shen et al., TaskBench: Benchmarking Large Language Models for Task Automation. NeurIPS 2024.

Huang et al., Planning, Creation, Usage: Benchmarking LLMs for Comprehensive Tool Utilization in Real-World Complex Scenarios. Findings of ACL 2024.

Dear Reviewer Adig,

We would like to thank you once again for your detailed review and for the time you dedicated to our work. Your comments led to several additions to the paper, which we believe have substantially improved its quality and clarity.

As the end of the discussion period is approaching, we would be grateful to know whether our responses have addressed your concerns, or whether there is anything else you would like us to clarify. We would be happy to provide further details or results.

Best regards,
The Authors

Comments on the paper

1 comment

We sincerely thank the Reviewers and the Area Chair for their time and the valuable feedback, which helped us improve the quality of the paper. We have addressed all the points raised and revised the manuscript accordingly; all changes are highlighted in blue, with the new experiments presented in Appendix E.4 to E.9.

Our answers and modifications may be summarized as follows:

Reviewer Adig

  • Separate ablations of confidence-aware dropout, with uniform dropout and GCA-based adaptive dropout under the same graphs, sparsification and parameter budget (Appendix E.6).
  • Comparison with a dual-graph version of each planner, on clean and perturbed graphs (Appendix E.7).
  • Exact Match on full ordered plans (Appendix E.5) and results over 3 seeds for all main comparisons (Appendix E.4).
  • Evaluation on UltraTool, a catalog of 260 tools (Appendix E.8).

Reviewer ecGU

  • Clarification of the data used to build the historical graph, with no leakage from validation or test plans (Appendix D.1).
  • Robustness to incomplete historical graphs, with up to 90% of the edges removed (Appendix E.7).
  • Inductive experiments with tools unseen during training (Appendix E.9).
  • Structured corruption of both graphs with realistic false edges (Appendix E.7), and explicit statement of the planning scope in the Conclusion and Limitations.
  • Results over 3 seeds (Appendix E.4).

Reviewer UuCe

  • Separate ablations isolating confidence-aware dropout from weighting and sparsification (Appendix E.6), and discussion of rare but valid transitions.
  • Simultaneous corruption of both graphs, comparison with a dual-graph baseline, and multiple training and perturbation seeds (Appendix E.7), together with multi-seed main results (Appendix E.4).
  • Clarifications on type specificity in Daily Life, the weights in Table 5, and the accuracy denominators, with corrections to a few transcription errors in the paper.

As the end of the discussion period is approaching, we would be grateful if you could let us know whether there are any remaining concerns or aspects you would like to discuss further. We would be happy to address them.

Kind regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.