- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper discusses LLM judge reliability, providing a framework and running a couple experiments on Qwen judges.
Critical strengths
none
Critical weaknesses
- This paper is very hard to follow - I would recommend a thorough rewrite starting from the abstract and intro to really focus your ideas. It is highly disorganized throughout, very challenging for a reader to understand what the contribution is beyond something to do with judge reliability. I don't want to overclaim exactly in what way AI was used to write this paper but I will say that it reads as AI slop: sentences such as "We present this contrast as a substantive, worked demonstration of pseudo-replication in LLM evaluation, not as a record of what changed between drafts of this manuscript” suggest this is an intermediate artifact outputted in a research project by an agent rather than a worthwhile submission
- experiments seem quite narrow: even beyond the disorganized communication, it seems that only a single small dataset was used (authors say themselves that these results cannot be generalized, " All findings hold only for these two Qwen2.5 judges and 30 MTBench examples"
Questions for authors
n/a