I do think the work has some weaknesses, specifically when it comes to contrasting the method against PUCT based selections: What is the theoretical difference between placing a prior on the UCT selection and placing a prior on the subtree selection, especially since you only use T=2 rounds of message passing?
In fact, I would expect that you can even rewrite the PUCT and node scoring objectives into each other by using the probability chain rule along the root-to-leaf paths.
We can observe this exact similarity in the benchmarks as well: Switching from UCT to PUCT largely removes the advantage of the selection mechanisms over plain MCTS.
At the very least, I would expect more independent seeds to be necessary: The “Deep Reinforcement Learning at the Edge of the Statistical Precipice” paper cited in this work that robust statistics (i.e. IQM+bootstrap CI) start producing reasonable results at 10 seeds. Considering that the methods here are all quite close we would need at least those 10 seeds, perhaps more to make any inference.
I think this also affects the conclusions drawn in section 5.4
Its improvement over MCTS is most pronounced in the mean, with smaller gains, within statistical error, in the median and IQM and a comparable optimality gap.
In words: the method is only better when a very noise sensitive statistic is used and becomes indistinguishable when using the robust estimators of performance (median, IQM and optimality gap).
This might be because robust statistics sacrifice some statistical power for repeatability, but then I would expect an increase in statistical power to reveal the advantage under robust statistics as well.
There is also a confounder in that LASER uses Danihelka et al.’s scoring method, which is known to outperform vanilla MCTS (especially for low number of planning rollouts as they are used here). At the very least you should test against MCTS with and without Danihelka’s scoring protocol to prevent spurious improvements caused by the more efficient scoring rule.
The results in table 2 are also a little bit strange: Even with additional tree traversal costs it makes, in my opinion, no sense that running an additional neural network is faster than the really cheap PUCT scoring rule. How expensive can a tree traversal possibly be? This has 50 trials, and the number of comparisons scales between n^2 and nlogn depending on how diverse the exploration is. LASER still needs K full views of the tree, which then produces 2N operations (due to message passing) without the logN reduction produced by the tree structure. Then it performs local MCTS there and propagates the information up (the “down” search is cheaper, the “up” broadcasting is more expensive). I cannot imagine that – at worst – 50 indexing operations into a tree will be more expensive than running a neural network.
Do you have some standard deviation data? Could it be that there are some spikes which materially affect the results? (or, alternatively, which MCTS implementation is used and how optimized is it)
Thank you for taking the time to review our paper and for your constructive feedback! We are glad you found the idea insightful and the experiments thorough.
Please see our common response above.
In the paper, we showed that LASER with a selector trained with K=5 generalizes to larger K at test time (Figure 3). This experiment only tested generalization to larger search trees within the same domain. To assess cross-domain transfer, we evaluated each source-trained selector on each target domain in Table 11.
Table 11. Transfer across domains under UCT and PUCT selection.
The selector transfers across problem sizes within the sliding-puzzle family, though transfer is weaker from the 15-puzzle to the 35-puzzle. Across families (Sokoban ↔ sliding puzzles), performance is generally close to or only slightly above the random-selector baseline. This pattern suggests that LASER learns allocation strategies that generalize within related tasks while adapting to the search characteristics of each domain. The limited transfer across families also supports the value of learning, as a single fixed scoring heuristic may struggle to capture the allocation strategies needed across these domains.
In Sokoban and puzzles, training the selector is a one-time cost of ~0.5-1.2 GPU-hours under UCT and ~1.4-4 GPU-hours under PUCT until its performance saturates. Planning costs at most ~2 seconds per episode and scales linearly with the search budget, so the one-time training cost equals the planning cost of only a few thousand episodes. In Atari domain, the selector is learning in parallel with other components and the extra training time is negligible.
MCTSnets and D-TSN address related questions, but a direct comparison would confound the effect of search-budget allocation with changes to the search algorithm or value network architecture.
MCTSnets is a learned version of MCTS trained on substantially more expert data than LASER on Sokoban (250K vs. 20K trajectories). LASER keeps local MCTS fixed and learns to allocate search to promising areas of the tree. MCTSnets could replace LASER’s local MCTS by providing a differentiable search procedure that might give the selector a denser training signal. The two approaches are therefore orthogonal. We leave this integration and its evaluation to future work.
D-TSN embeds best-first search in the value network to learn a more accurate value function from expert data. D-TSN therefore uses search to learn a value function and only indirectly to select the optimal action at the root. Given this incompatibility in purpose and mechanism, a direct comparison with LASER would not be indicative of LASER’s performance benefits.