- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper is based around black-box, combinatorial optimization where evaluations of the combinatorial function being optimized are a realizations of a stochastic sequence. However, simulating these stochastic sequences to completion in order to obtain outcome observations can be costly. This paper proposes a method to balance the extent to which the sequences are generated to trade-off between costs and performance. The paper validates its approach in simulated automated electric taxi charging domain to provide evidence of its capabilities.
Critical strengths
- I found the problem statement well motivated. I also consider the problem of interest to the search/planning community.
- The approach seems intuitive and sound.
Critical weaknesses
The main issue I view in the paper is the limited evaluation. The approach introduced is only evaluated in a single domain. All results demonstrate the performance of their algorithm in only this domain. Furthermore, the paper lacks details on the baselines they compare against. I cannot validate the fairness of the comparisons --- LAMF-MCTS receives a simulation budget of 10,000 samples. However, there is no mention of the compute costs of the baseline algorithms. Do they get more or less resources compared to LAMF-MCTS?
Additionally, the description of "side-information" seems context specific. Is the side-information useful in other domains? How do you compute for PUCT without side information?
The paper does not describe how the hyperparameters were set? There is a table of these values but no description on how the values were chosen.
I also am curious about the choice to report the running-best values. That seems like it with high variance outcomes, we'd have a skewed result.
Questions for authors
- The paper states that controls a trade-off between performance and cost. But the way is computed in Equation 1, does it do so? Depending on the scales of and , may or may not have any noticeable effect on evaluations.