- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes SkillGATE, a skill retrieval framework that combines a skill graph, hierarchical indexing, and gate-aware Monte Carlo Tree Search (MCTS). The method introduces four search actions: node-aware descent, skill-aware descent, horizontal search, and pruning. It uses historical rewards and entropy to guide action selection and employs a dual-path backpropagation mechanism to update search statistics. Experiments on six benchmarks demonstrate performance improvements over various retrieval and reranking methods.
Critical strengths
-
The combination of skill graphs and hierarchical indexing enables cross-region exploration and helps recover from incorrect hierarchical routing decisions.
-
The paper provides extensive experiments, including component ablations, comparisons of different search strategies, hyperparameter sensitivity analyses, and scalability evaluations.
Critical weaknesses
W1. The proposed method introduces four search actions, a heuristic gating function, a specialized reward function, and a dual-path backpropagation mechanism. Although the authors conduct component ablations and compare their method with Greedy, UCT, and PUCT, these experiments do not sufficiently isolate the benefits of the complex MCTS framework itself. In particular, Table 2 shows that removing skill-aware descent leads to performance even worse than the original Ada-002 retriever in Table 1. Meanwhile, Table 7 lacks a simple baseline that combines hierarchical indexing and skill-aware graph propagation without MCTS-based search feedback.
W2. Several structured retrieval baselines perform poorly, even compared with simple retrieval methods. However, the original objectives of these methods may differ from the single-skill ranking task considered in this paper. The paper does not sufficiently clarify how these baselines were adapted to the current evaluation setting or whether their original configurations were faithfully preserved.
Questions for authors
-
Why is such a complex MCTS framework necessary? Could the authors include a simpler baseline that combines hierarchical indexing and skill-aware graph propagation without MCTS-based search feedback? Furthermore, what motivates the specific formulations of the gating functions for the four actions, and have alternative mappings been explored?
-
Do the search strategies compared in Table 7 use equivalent computational budgets? Could the authors provide recall–latency curves for different numbers of search iterations and compare the methods under matched computational budgets?
Thank you for your response, I still have some concerns unresolved:
The newly added Non-MCTS baseline provides evidence for the contribution of adaptive search, but its implementation details remain unclear, particularly how the search actions are performed without MCTS and whether the computational budgets are comparable.
You should explain why removing skill-aware descent causes performance to fall below the original Ada-002 retriever, do you investigate its interactions with other components?
You report overall retrieval latency and iteration-based Hit rates, but do not provide Recall–Latency curves or matched-budget comparisons across search strategies, leaving the computational fairness of the comparisons unclear.