- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work covers the use case of LLMs being used as adjudicators in solving online disputes in e-commerce websites or wikipedia. Since only relying on how much an LLM adjudicator agreed with human decision in the past (agreement rate) can be erroneous due to any form of bias i.e, adjudicator only favoring party "A" for specific type dispute, the authors argue that it is important to also consider the legibility which is essentially how well we (proxied using LLMs) can forecast the decision of the LLM adjudicator. As the illegibility grows, the author present the safety margin that parties can know to protect themselves and it came out to be around 33-51% which showed that predictions from LLMs on adjudicator barely protected a party
The forecast error is splitted into illegibility, the LLM observers' misunderstanding of the adjudicator, and adjudicator's randomness.
They conducted experiments on Chinese e-commerce disputes and Wikipedia deletion debates using 8 frontier LLM models as adjudicators and found that:
- The choice of LLM adjudicator affected the outcome in 1/3rd of the cases
- Adjudicators with similar agreement rate differed up to 2.35x in illegibility
- Legibility strongly depended on the observer LLMs
- Showing observer LLMs the instructions given to adjudicator LLMs or even its past decisions didn't really help in forecasting adjudicator's decision
Critical strengths
- Timely problem: LLMs are increasingly taking up such adjudicator based roles and it is important to be able to predict their behavior.
- Appropriate controls and good experimental design: Both propositions regarding the forecast error split and safety margin are correct. Sufficient samples were considered for valid reporting along with 5 trials for each. The authors also show how disagreement between adjudicators exceeds what randomness would produce. The authors also openly report failed controls and other limitations.
Critical weaknesses
- Legibility depending on observer: This was the most surprising result but also most concerning. If the two LLM observers ranked the adjudicators with such a less correlation, how do we know for sure whether the "legibility" being measured is the property of adjudicator and not both adjudicator and observer?
- Prompt formatting experiments: Since its known that phrasing prompts differently - emotional tone/rigid persona/urgent tone results in different outcomes, accounting for this is missing from the work.
Questions for authors
Based on the weaknesses, I have two questions
-
Can authors add an observer independent measure? For example, on similar cases do we notice similar decisions and vice versa. Additionally, maybe some sort of self prediction where we can compare adjudicator's own forecast of its own ruling and compare with forecasting of an assumed generic AI judge.
-
Can authors add a small experiment on prompt sensitivity to show how the scores/findings generalise?
2 comments
Dear Reviewer MLQA,
Thank you again for your positive feedback. If you have other questions about the paper, let us know and we'll try to answer before the end of the discussion period (in 12 hours). Thanks again!
The Authors
Dear Reviewer MLQA,
Thank you for your review, and we appreciate your positive comments, as well as your suggestions for improvement. We ran many more experiments, for a total of 2.2k USD, which we hope will strengthen the paper. To address each of your points:
We agree that legibility depends on both the judge and the observer. We've made that explicit and tested the dependence across four observer families. We added F3 (DeepSeek V4.1 Flash) and F4 (GLM 5.3 Flash), whose families aren't on the judge roster. Each dataset now has 200 contested cases, split equally between the original sample (which we now refer to as Analysis) and the new sample (Holdout), plus 100 cases outside the contested sample (Comparison).
On AfD Holdout, all six dispute-only observer-rank correlations are 0.762-0.929. On VerdictBench Holdout, F1-F2 is −0.048 [−0.619, 0.572].
For the consistency test, we matched each Analysis case with its five closest other Analysis cases in the frozen text and text+image embeddings. We chose neighbors without using rulings. We compared squared differences in ruling probabilities between neighbors with those across all distinct case pairs, correcting both for the noise from estimating probabilities with five rulings. We used a delete-one-case jackknife, removing every pair involving the omitted case (see Appendix C in the revised paper).
With text embeddings, we found mean adjusted distances between neighbors versus all pairs of 0.320 versus 0.360 on VerdictBench and 0.411 versus 0.426 on AfD. Table 21 lists every judge and both representations. For example, VerdictBench/J3's reduction is 0.107 [0.026, 0.187], pointwise.
For self-prediction, we asked J3/J7 to forecast either a fresh independent instance of themselves or an unspecified automated judge, in independent sessions, on 50 cases per dataset. We scored both forecasts against the same five newly collected rulings under the original instructions. Self-minus-generic excess error is 0.054 [−0.030, 0.139] and −0.010 [−0.038, 0.016] on VerdictBench, and 0.007 [−0.009, 0.034] and 0.002 [−0.006, 0.012] on AfD, for J3/J7 respectively. See Table 18 in the revised paper.
For prompt sensitivity, we collected five rulings per case from J3/J7 under different variants: the original instructions, a neutral paraphrase, urgency, vague rules and substantive rules. We gave F1/F2 the corresponding judge instructions and collected new controls on the same cases. AfD/J7's neutral paraphrase has a noise-adjusted squared ruling-probability distance of 0.056 [0.010, 0.117] from the control and reduces reference agreement by 0.105 [0.025, 0.195] on the 40 cases with binary references. The substantive rules also changed ruling probabilities, with mixed effects on forecasting. We report all variants and pointwise intervals in Tables 15-16.
We also tested rephrased and checklist versions of the observer task. We collected fresh controls for F2 in both dispute-only and rules-only conditions, and for both F1 and F2 in the matched 50-case rules-only comparison. In that comparison, F2's VerdictBench checklist reduces excess error by 0.031 [0.004, 0.070] (see Table 17).
On the history result described in your summary:
We extended the history analysis to 500 rulings as well. The text-only linear model improves on the VerdictBench base-rate baseline by 0.026 [0.014, 0.039], including a small judge-specific component (see Table 20).
Thank you again for your review, and we hope that you’ll support our acceptance! If you have any questions between now and the end of the discussion period, we’ll do our best to address them.
Best,
The Authors