- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper stuties the reliability of LLM-as-judge methods. The author focus on two orthogonal axes: cross-session rank stability and cross-model scale calibration.
The paper did experiment on a 188k Chinese criminal court judgments dataset scored across six ordinal dimensions using Qwen2.5-14B-Instruct and 10 other open-weight models, and discovered that judge can switch between two nearly orthogonal scoring modes across sessions. The paper then proposed a 20 document reference anchoring and multi-session voting method to detect and mitigate the issue.
Critical strengths
-
the paper demonstrates why within-session retesting and marginal score calibration are insufficient, the cross model analysis across 11 open-weight model judges shows that even when model is bit2bit deterministic, their document ranking still has a low correlation, this provides solid evidence for the claim.
-
The 20-document reference anchor protocal is simple, mechanism-agnostic and computationally effecient.
Critical weaknesses
-
Mode B appeared in 4/16 campaigns, but didn't appear in 8 controlled conditions and 63 new sessions (M.7, M.12), and its root cause is unknown. The paper's main claim of mode B is weakened given it's only relying on those 4 campaigns.
-
"5/6 DID sign reversal" is overstated given sampling uncertainty and permutation null expectations. B-mode DID estimates are all close to zero, which is what random re-permutation predicts.
-
In axis 2, the calibration is measured by comparing judges against Qwen2.5-14B's scores, not against human labels (the human labeled data is only 8 documents and a single annotator). This only shows that other LLMs disagree with Qwen2.5-14B so this does not establish a new reliability axis.
Questions for authors
-
Since mode B wasn't reproduced in later sessions, coud you provide more comment on data side explainations considered to rule out the data-misalignment error?
-
For the DID hazard, could you report the B-mode estimates against a permutation null (A scores randomly re-permuted)?
-
The abstract says the anchor classifies "all 16 campaigns," but Table M.4 lists 7.