- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper is concerned with constructing a strong prior distribution for multi-objective Bayesian optimization over m criteria. The construction is based on E experts, considered frozen (and usually given by prompts to an LLM). Each expert e provides a score mu_{e j}(x) for candidate x and criterion j, as well as some confidence statement c_e(x). Neither is necessarily well-calibrated. The work aims to turn this input into a useful prior, which is used in two ways. First, it provides a prior mean function which is added to the residual (prior 0 mean) GP. Second, it provides some "counterfactual" confidence estimate, based on which the overall confidence/reliability of experts is judged and taken into account.
There is quite some prior literature on prior construction for BO, also MOBO, and what is proposed here is nothing very novel given. The presentation of this work is really quite poor, which makes it hard to say much more. From a lot of undefined lingo over undefined notation to totally unmotivated expressions (containing sigmas in sigmas, softmax, clipping, and a large number of free parameters, which the authors likely tuned until they got good results). The presentation does essentially nothing to structure this development in any sensible way or guide the reader.
Critical strengths
The overall problem addressed here is potentially important and timely.
I cannot detect any other strength of this work.
Critical weaknesses
The paper has a poor presentation. It uses a lot of undefined lingo like "markets, capital, reputation, market trust, real-market capital, and even "shadow capital". It is not even completely clear what the inputs are. I guess these are mu_{e j} and c_e. But where are K_{e j, t} coming from? They are then combined in some complex, unmotivated expressions, involving lots of things from the ML toolbox, like sigmas nested in sigmas, clipping, or softmax. The chain is xi_j -> alpha_{e j, t} -> v_{e j} -> bar{mu}_j, chi_j -> G_j. This ends up with 3 "arms" based on using G_j with different values for d_e(x). Just from this vocabulary (!!), I am guessing there is some bandit vote involved here. Crucially, it is certainly not clear from the writing why this complex construction would amend anything about the inputs possibly being uncalibrated, biased, or whatever else the authors are saying in their overall motivation.
At a later point (maybe also in the Appendix), they talk about "shadow", and in the abstract I see the term "replay". None of this is explained or connected. Reading this is quite a frustrating exercise.
And by the way, the expression come with a large number of free parameters:
- a_xi, tau_xi
- T
- alpha_min, alpha_max
- eta, lambda, eta_p, ...
- s_min, ...
In Table 5, the authors list the values they chose for all these many free parameters. I have no doubt their results improve over all the other options they compared against, given this large tuning space. So, apart from being obscure and unmotivated, the method is also not practically useful, because way too many values have to be adjusted.
Questions for authors
I have no idea what all this lingo about "markets" and "reputation" and "trust" is all above, what it has to do with MOBO, or how you got to the expressions you use here.
Just one observation of me playing around with your expressions:
Denote C_j := E^{-1} sum_e v_{e j}. Then:
- mu_j = E^{-1} sum_e v_{e j} mu_{e j} C_j^{-1}
- xi_j = clip( C_j, 0, 1). The clipping to 0 is not needed, since C_j >= 0
- G_j = xi_j mu_j = E^{-1} sum_e v_{e j} mu_{e j} * F_j
where F_j = clip( C_j, 0, 1) / C_j = 1 I(C_j <= 1) + C_j^{-1} I(C_j > 1). So, F_j = 1 (flat!) unless C_j > 1. What is so special about 1? alpha_{e j, t} lies in [0, E] before clipping and [alpha_min, alpha_max] = [0.02, 2.5] after clipping (why?). Why does the clipping not depend on E? And d_e(x) = 1 for one, = c_e(x) for the other arm, so comes from the input and is maybe in [0, 1].
If you like to answer any of my concerns / questions, please do it concisely. Any long-winded AI generated slop not really addressing the concerns will not be read.