- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors run a series of controlled experiments on LLM judges for research ideation systems, and find effects such as order sensitivity and measure agreement with human ratings from existing papers.
Critical strengths
There is growing interest in LLM-based ideation, and clearly the rapid advances in AI-for-science (and math) results suggest that more is coming in this space. Understanding LLM as a judge is important if we want to scale up evaluations or if we want to use the judges as verifier targets for test-time-search.
Critical weaknesses
It is not clear to me that this paper adds much beyond the existing literature on llm-as-a-judge's biases, as well as various more controlled human evaluation studies on the quality of llm-as-a-judge methods when compared to humans.
While the paper has a human eval section, this is actually reusing human eval data from past studies, and so the main contribution of the paper is in studying various forms of biases (like order sensitivity) and measuring changes as a function of architecture interventions (like retrieval).
However, the former is well studied in past work on llm judgements, and the latter is ungrounded, and it's unclear to me what the 'ground truth' should be in this case.
For all these reasons, it's not clear that there's much of a clear contribution beyond collecting a bunch of experimental observations. The paper is also written in this way, with sections essentially serving as collections of observations. If the observations are robust and timeless, maybe this is fine, but the paper does not do much to convince me of the novelty and robustness of these findings.
Questions for authors
For the llm as a judge biases parts, are you finding any new phenomena?
What are some more general, concrete prescriptions that you have for llm-as-a-judge for ideation as a result of your experiments? do you have any stronger falsifiable claims that you're making about the practice of llms for ideation?