- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
From the paper writing, I was not able to understand what the paper is about, except that it addresses multiple pairwise preference selections by LLM judges. Specifically,
Line 038 states what I assume is the central research question of this paper:
Which joint response patterns can rule out every shared-ranking explanation?
However, there is no explanation what the "joint response pattern" is and what "ruling out shared-ranking explanations refers to". Because of the lack of clarity, the motivation and purpose of the following theoretical analysis also remains unclear.
I also tried understanding the paper from the perspective of the empirical validation in Sec. 6. However, the experimental setup is not explained there either. It seems that somehow texts in different styles where judged by closed/API LLMs, but what exactly is judged and with what purpose is not stated.
Finally, the implications for preference learning in Sec. 8 are opaque, too:
Preserve the labelled joint answers, use independent pilot and confirmation units to select promising comparisons, and report the dependence budgets under which a shared-order explanation is rejected.
None of this was directly tested, and how it would be implemented in practice is not stated either.
In summary, if there is actually a meaningful contribution, it needs to be stated in a much clearer way.
Critical strengths
I do not see any strengths in favor of acceptance, except the general topic of preference learning is relevant, and insights that resolve judge nondeterminism or contradictory binary decisions is useful.
Critical weaknesses
As mentioned above, the paper is unclear to an extent that makes judging anything else very difficult. At least, the paper should clearly state its motivation, which problem it addresses, and how it does that. In terms of writing, it should avoid jargon and introduce all concepts and settings properly.
Questions for authors
My main question is to clarify which problem this paper addresses, concretely.