- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies a setting where the real annotation f(X) would be expensive while people have a proxy annotation g(X), and people wish to get a data-efficient estimation of the real value. So that the authors proposed to use an unbiased estimation as to reduce the variance. They further proposed a constrained projection version of this unbiased estimation to control the range. They also derived some criterions on when should people use this estimation. They conducted some experiments on toy cases and real experiments involving LLMs such as Gemini and LLama.
Critical strengths
- They proposed a constrained optimization framework, which might be of some novelty.
Critical weaknesses
- The authors claimed that "the preparation of figures were independently completed by the authors", however, the Figure 2 is AI-generated w.h.p.
- The theoretical part is trivial, as it's a basic idea in statistical inference to control variance, for example, see [1]. The only contribution might be to proposed the constrained projection method, however, the experimental advantage of projection is very marginal, as shown in the LLM experiments.
- The experiments are mostly based on manually designed toy cases. In the only experimental part involving modern LLMs, the authors admitted that "A favorable variance comparison therefore informs the control signal but does not establish cost-saving qualification." and "Every real-model setting remains unqualified, with no measured accepted four-resource ledger." It can be seen that the improvement in practice is very marginal, or sometimes even worse: " Full-context Skywork gives gains of .67% and .54% on Llama–Gemini and Llama–Claude, though both remain worse than raw.".
- Their proposed criterion doesn't hold in practice as admitted in the paper, "All combinations return HOLD; no fresh evaluation runs."
- The paper writing seems AI-native.
[1] Prediction-Powered Inference. https://arxiv.org/abs/2301.09633.
Questions for authors
I don't have questions. I think this paper cannot get accepted.