- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The article presents TestBED, a modular and extensible software framework for developing and assessing Bayesian experimental design (BED) methods. It brings together the majority of current methods, and a suite a test tasks. Most importantly, it defines an interface between different components of BED methods that allows for modular extensibility. This will allow researchers in BED to have a set of test tasks, and standardised implementations of core components, with which to implement and experiment with new developments. In turn, this will enable much greater comparability between BED methods, in contrast with the current state of affairs (everybody develops everything from scratch, including the test tasks, so results are rarely like for like comparable).
The article demonstrates an immediate benefit of the TestBED in discovering clearly that using time as a feature in the design state improves performance in methods that were originally released without time dependence. Similar ablations/borrowing of ideas are likely to follow.
Critical strengths
The article is, in my opinion, a significant enabler of future research in BED
The article is very clear, and the software is already available to download
I would like to use this software suite in my own research!
Critical weaknesses
There isn't a large core "research" contribution here. However, I think that the contribution to future development of BED is likely to be significant
Questions for authors
None
Thank you for your positive assessment of our work, and are encouraged to hear that you are likely to find TestBED useful in your own research. We are looking forward to seeing what TestBED will enable the BED community to achieve.
We appreciate the reviewer’s comment that TestBED is likely to contribute significantly to the development of future BED methodology. While the infrastructure of TestBED is the primary contribution of our work, we also highlight several scientific contributions that go beyond the framework itself:
As noted by the reviewer, we see these findings as demonstrative of the kinds of future research TestBED will additionally enable.