- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates partially observable restless multi-armed bandits (RMABs) in continuous latent spaces where the dynamics are unknown or vary over time. The proposed method, BIRD, consists of modeling the actor’s priorities using a shared belief encoder and a diffusion actor, combined with selection via a Top-K projection. It has been theoretically demonstrated that this approach yields updates that satisfy budget constraints. Furthermore, experiments confirm its effectiveness.
Critical strengths
In the field of partially observable restless multi-armed bandits (RMABs), a significant method has been proposed as a solution to the issue of not satisfying the assumptions of conventional RMAB methods.
The proposal is presented in a structure that is easy to understand from a theoretical perspective and includes theoretical guarantees.
From these perspectives, this is a meaningful proposal that holds promise for the future development of this field.
Critical weaknesses
This paper argues that a new method is necessary because the classical Whittle index method cannot be applied to partially observed RMABs, given that it assumes discrete states. On the other hand, several studies on partially observed RMABs have been published, such as [1]–[4]. Given this context, concerns arise as to whether the problem formulation in this paper is appropriate.
(There is a possibility that the authors are claiming achievements while concealing issues that have already been resolved.) This point constitutes a major problem regarding the structure of this paper.
Regarding methodological questions: How is the value of K in TopK determined? It is difficult to understand how performance varies with K based solely on the two results for K = 5 and K = 20. If setting K leads to catastrophic results, it is difficult to highly evaluate the validity of the results at this stage.
[1] J. Kuhn et al., Exploration vs Exploitation with Partially Observable Gaussian Autoregressive Arms, International Conference on Performance Evaluation Methodologies and Tools, 2014
[2] Y. Zhao et al., Towards a Pretrained Model for Restless Bandits via Multi-arm Generalization, IJCAI 2024
[3] A. Mate et al., Collapsing Bandits and Their Application to Public Health Intervention, NeuRIPS 2020
[4] K. Liu, Relaxed Indexability and Index Policy for Partially Observable Restless Bandits, Management Science Vol.17 No.12
Questions for authors
Please clarify how this study relates to previous research regarding partially observed RMABs.
Please explain the methodology for setting K and the changes associated with different values of K.
We thank the reviewer for their careful review and feedback. We briefly summarize the response to your questions below.
Q1 connection to other partially observed RMABs: the major difference is the use of transformer to encode the historical trajectory and the use of diffusion head to predict the index. Our model benefits from the expressiveness of transformer and the multimodality and exploration benefit of diffusion model. We will add more experiments to compare with other partially observed RMAB methods to validate the improvement led by transformer and diffusion model separately.
Q2 choice of parameter K: Our experiments evaluate different problem sizes and budgets, including (N, K) = (20, 5), (40, 10), and (100, 25), where K is approximately 25% of N. These experiments only demonstrate the scalability of BIRD under different resource budgets, not aiming to test the change in terms of different K.