- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces MOSAIC, a plug-in framework that improves tool representations for graph-based task planning. It combines historical and input–output dependency graphs, assigns heuristic confidence weights to their edges, and applies top-k sparsification. Relation-specific graph encoders are trained with confidence-aware edge-dropping augmentations and intra- and cross-graph contrastive objectives. The resulting representations are fused with learnable tool identity embeddings and integrated into existing planners. Experiments on three TaskBench domains, two planning architectures, and multiple LLM backbones show consistent improvements in tool-selection and dependency-prediction metrics. Component ablations, synthetic graph-noise experiments, and efficiency measurements provide further validation.
Critical strengths
A well-motivated and reusable design. MOSAIC addresses the unreliability of heuristic tool graphs by combining dependency and historical information with confidence-aware representation learning. Its integration into two different planners without redesigning their downstream planning procedures demonstrates practical applicability beyond a single architecture.
Consistent improvements across experimental settings. MOSAIC improves all three reported metrics across the main combinations of domains, LLMs, and planners, with substantial gains in several settings. For example, GTool with Mistral-7B improves tool-set accuracy from 59.40% to 69.20% on Multimedia. These results, supported by component ablations, provide meaningful evidence for the effectiveness of the overall approach.
Critical weaknesses
The role of the core confidence mechanism is not sufficiently established. Semantic similarity, type rarity, and transition frequency are plausible weighting heuristics, but the paper does not directly establish that they identify unreliable connections. The confidence-weighting ablation simultaneously removes edge weighting, sparsification, and adaptive dropout, making their contributions difficult to distinguish. Controlled comparisons that vary weighting and dropout separately, while retaining the same two graphs and comparable sparsity, would help determine whether the gains reflect reliability estimation rather than general regularization.
The robustness evidence is limited by the perturbation protocol and single-seed evaluation. Noise experiments perturb one graph at a time while leaving the other intact, so improvements may arise from access to complementary clean information rather than the proposed confidence mechanism. All experiments also use a single random seed, leaving the stability of smaller gains and fluctuating noise results unclear. Comparing against an otherwise identical dual-graph model with uniform weighting, testing simultaneous corruption of both graphs, and reporting multiple training and perturbation seeds would materially strengthen the robustness claim.
Questions for authors
Confidence mechanism. Could you isolate the effects of edge weighting and adaptive dropout while keeping the two graphs and sparsity comparable? This would help establish whether the proposed confidence scores offer benefits beyond graph fusion and general regularization. An analysis of rare but valid transitions would also clarify whether frequency-based weighting suppresses useful connections.
Robustness and stability. A comparison with an otherwise identical dual-graph model using uniform weights would help interpret the noise results. Please also consider simultaneous corruption of both graphs and multiple training and perturbation seeds to establish whether the observed robustness gains are stable.
Implementation clarification. How is the type-specificity weight computed for Daily Life’s fully connected graph? Please explain how Equation (1) produces the nonzero weights in Table 5. Also, please clarify the effective evaluation denominators: some reported Accuracy values do not appear consistent with averaging binary outcomes over 500 examples in a single run.
6 comments
(Part 2/4)
2. Robustness and stability (W2, Q2)
We made three changes to the robustness study, following the reviewer's suggestions.
First, both graphs are now corrupted at the same time. For each ratio from 10% to 90%, false edges are added to both graphs between tools that are not linked in the clean graphs. Their types (dependency graph) and co-occurrence counts (historical graph) are drawn from those of the true edges, and all weights are recomputed with Equations 1 and 2, so false and true edges have the same weight distribution (median dependency weight 0.20 against 0.19 at 10% noise on HuggingFace). MOSAIC therefore cannot rely on a clean second graph, nor single out the false edges by their weights.
Second, we compare with a dual-graph version of each baseline that is identical to the original planner except that it encodes both graphs, unweighted and not sparsified, with two separate GNNs and fuses the two embeddings as MOSAIC does. It has no confidence weights, no contrastive objective and no identity embeddings, so the gap to MOSAIC measures what our method adds beyond using both graphs.
Third, all results are averaged over 3 seeds, and each seed is used both to generate the perturbed graphs and to train the model.
Accuracy on HuggingFace with CodeLlama-13B (Appendix E.7, Figure 6):
| Noise (%) | GNN4Plan dual-graph | GNN4Plan+MOSAIC | GTool dual-graph | GTool+MOSAIC |
|---|---|---|---|---|
| 0 | 30.53 ± 1.30 | 32.40 ± 0.31 | 45.07 ± 1.47 | 49.80 ± 0.92 |
| 10 | 30.79 ± 1.85 | 31.40 ± 0.81 | 47.40 ± 0.53 | 48.13 ± 1.33 |
| 25 | 31.00 ± 0.31 | 31.20 ± 0.50 | 46.93 ± 3.61 | 47.60 ± 0.80 |
| 50 | 30.59 ± 1.16 | 31.33 ± 0.23 | 45.93 ± 1.97 | 48.53 ± 1.60 |
| 75 | 30.59 ± 1.16 | 30.26 ± 1.75 | 47.67 ± 3.20 | 48.80 ± 2.96 |
| 90 | 29.26 ± 1.12 | 30.19 ± 1.50 | 48.87 ± 1.94 | 48.33 ± 1.10 |
MOSAIC stays above the dual-graph version on the clean graphs and at most noise ratios with both planners: GTool+MOSAIC from 10% to 75%, and GNN4Plan+MOSAIC at 10%, 25%, 50% and 90%, where the margin is the largest. In the remaining cases the two are within one standard deviation. From the clean graphs to 90% noise, the accuracy of MOSAIC decreases by 2.21 points with GNN4Plan and by 1.47 with GTool, although both graphs are corrupted at once and the false edges cannot be told apart from the true ones by their weights. On the clean graphs, the gap to the dual-graph version (32.40 against 30.53 with GNN4Plan, 49.80 against 45.07 with GTool) also shows that the gains of MOSAIC go beyond the use of both graphs.
(Part 3/4)
We also repeated the main comparison over 3 seeds, for both planners on HuggingFace (HF) and Multimedia (MM) with all three LLMs (Appendix E.4, Table 9). Mean ± std over seeds; Δ is the relative change of each seed, averaged over seeds:
| LLM | Model | HF n-F1 | HF l-F1 | HF Acc | MM n-F1 | MM l-F1 | MM Acc |
|---|---|---|---|---|---|---|---|
| Qwen3-14B | GNN4Plan | 79.94 ± 0.24 | 61.45 ± 0.68 | 46.00 ± 1.06 | 84.22 ± 1.64 | 70.01 ± 1.57 | 55.60 ± 1.11 |
| GNN4Plan+MOSAIC | 80.58 ± 0.52 | 62.44 ± 0.62 | 46.33 ± 1.10 | 85.38 ± 0.26 | 71.38 ± 0.13 | 56.93 ± 0.12 | |
| Δ (%) | +0.81 ± 0.95 | +1.63 ± 2.06 | +0.79 ± 4.63 | +1.41 ± 1.95 | +1.98 ± 2.28 | +2.42 ± 1.88 | |
| GTool | 81.01 ± 0.85 | 59.31 ± 0.49 | 45.00 ± 1.22 | 86.47 ± 0.85 | 70.06 ± 1.14 | 58.07 ± 2.90 | |
| GTool+MOSAIC | 81.66 ± 1.35 | 60.41 ± 2.07 | 45.07 ± 2.04 | 87.11 ± 0.51 | 70.69 ± 0.64 | 59.40 ± 0.72 | |
| Δ (%) | +0.81 ± 1.92 | +1.84 ± 2.69 | +0.12 ± 1.82 | +0.75 ± 1.25 | +0.92 ± 1.85 | +2.46 ± 4.96 | |
| CodeLlama-13B | GNN4Plan | 72.96 ± 0.74 | 50.61 ± 0.84 | 31.46 ± 0.40 | 77.96 ± 0.55 | 58.67 ± 0.83 | 41.72 ± 1.28 |
| GNN4Plan+MOSAIC | 74.11 ± 0.66 | 51.12 ± 0.82 | 32.40 ± 0.31 | 79.21 ± 0.23 | 60.47 ± 0.30 | 44.38 ± 0.35 | |
| Δ (%) | +1.58 ± 1.86 | +1.04 ± 2.92 | +2.98 ± 0.40 | +1.60 ± 0.93 | +3.09 ± 1.69 | +6.45 ± 4.03 | |
| GTool | 81.35 ± 0.55 | 60.10 ± 1.24 | 44.33 ± 1.01 | 87.85 ± 1.37 | 72.12 ± 2.16 | 58.53 ± 3.83 | |
| GTool+MOSAIC | 82.64 ± 0.41 | 63.58 ± 0.11 | 49.80 ± 0.92 | 89.54 ± 0.45 | 74.92 ± 0.60 | 63.33 ± 0.46 | |
| Δ (%) | +1.59 ± 1.06 | +5.82 ± 2.34 | +12.36 ± 2.88 | +1.94 ± 1.27 | +3.93 ± 2.53 | +8.50 ± 7.02 | |
| Mistral-7B | GNN4Plan | 68.20 ± 0.57 | 45.44 ± 0.48 | 29.45 ± 0.82 | 76.92 ± 1.50 | 56.92 ± 1.04 | 40.47 ± 1.38 |
| GNN4Plan+MOSAIC | 69.85 ± 0.25 | 47.29 ± 0.13 | 30.74 ± 0.24 | 78.22 ± 0.74 | 58.41 ± 0.72 | 42.56 ± 0.65 | |
| Δ (%) | +2.42 ± 0.49 | +4.07 ± 0.82 | +4.45 ± 3.04 | +1.74 ± 2.90 | +2.66 ± 3.08 | +5.27 ± 5.01 | |
| GTool | 80.22 ± 0.50 | 58.14 ± 1.59 | 39.60 ± 0.87 | 88.66 ± 0.45 | 72.36 ± 1.43 | 60.07 ± 1.89 | |
| GTool+MOSAIC | 82.68 ± 0.80 | 62.96 ± 1.28 | 47.40 ± 0.87 | 90.72 ± 0.68 | 77.79 ± 0.95 | 69.40 ± 1.51 | |
| Δ (%) | +3.08 ± 1.02 | +8.33 ± 3.11 | +19.72 ± 2.46 | +2.32 ± 0.59 | +7.53 ± 1.03 | +15.56 ± 1.25 |
MOSAIC improves the mean of every metric in every configuration. With Qwen3-14B the gains are moderate but positive and within the seed variance, as they already were in Table 1. With CodeLlama-13B and Mistral-7B the accuracy gain is larger than the standard deviation in every configuration.
(Part 1/4)
We thank the reviewer for the thorough review and the precise questions. We ran the requested experiments and clarified the implementation points. The new material is in the revised PDF, highlighted in blue, with the new experiments in Appendix E.4 to E.9.
1. Confidence mechanism (W1, Q1)
In the original w/o Confidence Weighting ablation all edges have weight 1. Top-k sparsification ranks edges by weight, and confidence-aware dropout derives drop probabilities from the weights, so with unit weights the first has nothing to rank and the second reduces to uniform dropout. Removing the weights therefore necessarily removes the other two mechanisms.
To isolate the dropout, we keep the same two graphs, the same weights and the same top-k sparsification, and replace only the confidence-aware dropout with uniform edge dropout (w/o Confidence-aware Dropout). Since the reviewer also asks whether the gains go beyond general regularization, we add a second control, also suggested by Reviewer Adig, that keeps the dropout adaptive but removes the confidence from it: w/ GCA-based Dropout uses the adaptive edge dropout of GCA (Zhu et al., 2021), where drop probabilities depend on graph topology (log degree centrality of the target node) instead of edge confidence, with the same average and maximum drop rates as ours. Together with w/o Graph Sparsification in Table 2, which keeps the weights and removes top-k, each mechanism is now evaluated with the others fixed. Results on Multimedia (MM) and Daily Life (DL) with Mistral-7B over 3 seeds (Appendix E.6, Table 11):
GNN4Plan
GTool
MOSAIC is the best setting on every metric for both planners on Multimedia and for GNN4Plan on Daily Life. The only exception is GTool on Daily Life, where w/o Confidence-aware Dropout is slightly higher in n-F1 (98.09 against 98.03) and accuracy (87.93 against 87.53), on a dataset where performance is close to saturation, while MOSAIC has the best l-F1. w/ GCA-based Dropout, which keeps the dropout adaptive but driven by topology instead of confidence, stays below MOSAIC on every metric in all four settings. With GNN4Plan on Multimedia, both variants fall back to the level of the baseline, so most of the gain there comes from the confidence-aware dropout. Since GCA-based dropout uses the same graphs, the same sparsity and the same amount of regularization, the gap between it and MOSAIC can be attributed to the confidence scores rather than to graph fusion or generic regularization.
On rare but valid transitions: giving infrequent transitions a lower weight is the intended behavior of the historical graph, which leverages usage frequency as a proxy for reliability. A transition that users have performed only a few times is less likely to be a sound step, although it may still be valid, and frequency is the only evidence of this kind that execution logs provide. For this reason, MOSAIC treats frequency as a soft signal: the historical weight νˉνij lowers the weight of rare transitions but keeps them in the graph, top-k removes a rare edge only if its source tool has at least k outgoing transitions with a higher weight, and the dropout probability is capped at pmax, so no edge is dropped deterministically.