- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In a concise summary, the paper studies the setting where additional information sources are efficiently and consistently matched with experts which predict labels for given inputs. It contributes methods to consistently compute the errors with surrogates and efficiently via budgeted source observations. It validates the results through experiments and analyses for unbiasedness and optimality bounds.
Critical strengths
A substantive strength of the paper can possibly be the comprehensiveness of the experimental body. However, it is hard to evaluate the rigorousness and reproducibility.
Regarding technical correctness, the analyses are individually easy to follow in general.
Critical weaknesses
The submission is not really clear. Many times the terminology changes abruptly. The presentation of many central elements are not eased into in the first few pages. Also, for technical correctness, there are some potential issues, please see questions.
Regarding presenting novel findings, the importance of many theoretical findings are questionable. Possibly besides showing the consistency of surrogates, the other findings and contributions seem incremental, e.g., pairing of expert and source, budgeting etc.
Questions for authors
Abstract seems to be written in a general language so its hard to pinpoint the exact contributions.
Line 40: "cheapest expected cost under its best source, including no advice", do you mean all the sources are possibly detrimental or did you actually want to point out a trade-off?
Line 58: the use of the term "reference" in Page 1 is confusing. Is the reference outcome whether a fixed reference expert succeeded or not? Also, to clarify, there are experts (with prediction capabilities), paid sources (information before predictions), and paid actions (decisions) which result in loss/error?
Line 163: arbitrary notation changes makes it harder to understand. For example, please define policy loss in (5).
Why is Theorem 3 important? It deals with specific surrogate losses, why are they important?
Line 251: is there two scenarios, either no advice or (j,k) paid pair?
Line 256: why is the surrogate loss for no advice always zero, isn't it also a pair (j,0)? Below here you seem to deduce as such from the equation; however, that would be a circular argument since you disregarded the contribution of no advice already by not introducing a term with (1-I_t). Please clarify.
(12): the normalization with B is not tight since it is a bound, and the probabilities seem heavily dependent on this normalization factor B, why?
Thank you for your answers and responses. My concerns have been generally addressed. I will raise my rating to 3.