acceptodds
Under review as a conference paper at ICLR 2027

Equally Accurate, Unequally Legible: Measuring the Legibility of LLM Adjudicators

Abstract

Large language models are increasingly used to resolve real disputes, from e-commerce claims to prediction-market arbitration. An adjudicator is chosen when a contract is signed, long before the disputes it will rule on exist. At that moment, the only evidence about a judge is its record, usually summarized by reference agreement: how often its rulings match those of the institution it would replace (e.g., a human jury). This tells a party how closely the judge tracks that institution, but not whether the party can anticipate its rulings. Following the legal-realist view of law as the prediction of what judges will do, we introduce adjudicative legibility: how well an external forecaster can predict an adjudicator's ruling on an unseen dispute, given the dispute and, optionally, its published rules and past rulings. We study eight LLM adjudicators on contested e-commerce and article-deletion disputes. On more than a third of them, the outcome depends on which adjudicator is chosen, and adjudicators of similar accuracy differ by up to in illegibility. We prove that a party can protect itself by demanding a safety margin that grows with a judge's illegibility; on these disputes, this margin is comparable to that with no forecast at all. A party cannot meaningfully consent to a judge it cannot anticipate; we therefore argue that as LLMs take on binding adjudicative roles, legibility should be measured as routinely as accuracy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
llm, adjudication, law, legibility, predictability

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_MLQAOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work covers the use case of LLMs being used as adjudicators in solving online disputes in e-commerce websites or wikipedia. Since only relying on how much an LLM adjudicator agreed with human decision in the past (agreement rate) can be erroneous due to any form of bias i.e, adjudicator only favoring party "A" for specific type dispute, the authors argue that it is important to also consider the legibility which is essentially how well we (proxied using LLMs) can forecast the decision of the LLM adjudicator. As the illegibility grows, the author present the safety margin that parties can know to protect themselves and it came out to be around 33-51% which showed that predictions from LLMs on adjudicator barely protected a party

The forecast error is splitted into illegibility, the LLM observers' misunderstanding of the adjudicator, and adjudicator's randomness.

They conducted experiments on Chinese e-commerce disputes and Wikipedia deletion debates using 8 frontier LLM models as adjudicators and found that:

  1. The choice of LLM adjudicator affected the outcome in 1/3rd of the cases
  2. Adjudicators with similar agreement rate differed up to 2.35x in illegibility
  3. Legibility strongly depended on the observer LLMs
  4. Showing observer LLMs the instructions given to adjudicator LLMs or even its past decisions didn't really help in forecasting adjudicator's decision
Critical strengths
  1. Timely problem: LLMs are increasingly taking up such adjudicator based roles and it is important to be able to predict their behavior.
  2. Appropriate controls and good experimental design: Both propositions regarding the forecast error split and safety margin are correct. Sufficient samples were considered for valid reporting along with 5 trials for each. The authors also show how disagreement between adjudicators exceeds what randomness would produce. The authors also openly report failed controls and other limitations.
Critical weaknesses
  1. Legibility depending on observer: This was the most surprising result but also most concerning. If the two LLM observers ranked the adjudicators with such a less correlation, how do we know for sure whether the "legibility" being measured is the property of adjudicator and not both adjudicator and observer?
  2. Prompt formatting experiments: Since its known that phrasing prompts differently - emotional tone/rigid persona/urgent tone results in different outcomes, accounting for this is missing from the work.
Questions for authors

Based on the weaknesses, I have two questions

  1. Can authors add an observer independent measure? For example, on similar cases do we notice similar decisions and vice versa. Additionally, maybe some sort of self prediction where we can compare adjudicator's own forecast of its own ruling and compare with forecasting of an assumed generic AI judge.

  2. Can authors add a small experiment on prompt sensitivity to show how the scores/findings generalise?

2 comments

Dear Reviewer MLQA,

Thank you for your review, and we appreciate your positive comments, as well as your suggestions for improvement. We ran many more experiments, for a total of 2.2k USD, which we hope will strengthen the paper. To address each of your points:

how do we know for sure whether the "legibility" being measured is the property of adjudicator and not both adjudicator and observer?

We agree that legibility depends on both the judge and the observer. We've made that explicit and tested the dependence across four observer families. We added F3 (DeepSeek V4.1 Flash) and F4 (GLM 5.3 Flash), whose families aren't on the judge roster. Each dataset now has 200 contested cases, split equally between the original sample (which we now refer to as Analysis) and the new sample (Holdout), plus 100 cases outside the contested sample (Comparison).

On AfD Holdout, all six dispute-only observer-rank correlations are 0.762-0.929. On VerdictBench Holdout, F1-F2 is −0.048 [−0.619, 0.572].

Can authors add an observer independent measure? For example, on similar cases do we notice similar decisions and vice versa.

For the consistency test, we matched each Analysis case with its five closest other Analysis cases in the frozen text and text+image embeddings. We chose neighbors without using rulings. We compared squared differences in ruling probabilities between neighbors with those across all distinct case pairs, correcting both for the noise from estimating probabilities with five rulings. We used a delete-one-case jackknife, removing every pair involving the omitted case (see Appendix C in the revised paper).

With text embeddings, we found mean adjusted distances between neighbors versus all pairs of 0.320 versus 0.360 on VerdictBench and 0.411 versus 0.426 on AfD. Table 21 lists every judge and both representations. For example, VerdictBench/J3's reduction is 0.107 [0.026, 0.187], pointwise.

Additionally, maybe some sort of self prediction where we can compare adjudicator's own forecast of its own ruling and compare with forecasting of an assumed generic AI judge.

For self-prediction, we asked J3/J7 to forecast either a fresh independent instance of themselves or an unspecified automated judge, in independent sessions, on 50 cases per dataset. We scored both forecasts against the same five newly collected rulings under the original instructions. Self-minus-generic excess error is 0.054 [−0.030, 0.139] and −0.010 [−0.038, 0.016] on VerdictBench, and 0.007 [−0.009, 0.034] and 0.002 [−0.006, 0.012] on AfD, for J3/J7 respectively. See Table 18 in the revised paper.

Can authors add a small experiment on prompt sensitivity to show how the scores/findings generalise?

For prompt sensitivity, we collected five rulings per case from J3/J7 under different variants: the original instructions, a neutral paraphrase, urgency, vague rules and substantive rules. We gave F1/F2 the corresponding judge instructions and collected new controls on the same cases. AfD/J7's neutral paraphrase has a noise-adjusted squared ruling-probability distance of 0.056 [0.010, 0.117] from the control and reduces reference agreement by 0.105 [0.025, 0.195] on the 40 cases with binary references. The substantive rules also changed ruling probabilities, with mixed effects on forecasting. We report all variants and pointwise intervals in Tables 15-16.

We also tested rephrased and checklist versions of the observer task. We collected fresh controls for F2 in both dispute-only and rules-only conditions, and for both F1 and F2 in the matched 50-case rules-only comparison. In that comparison, F2's VerdictBench checklist reduces excess error by 0.031 [0.004, 0.070] (see Table 17).

On the history result described in your summary:

past decisions didn't really help in forecasting adjudicator's decision

We extended the history analysis to 500 rulings as well. The text-only linear model improves on the VerdictBench base-rate baseline by 0.026 [0.014, 0.039], including a small judge-specific component (see Table 20).

Thank you again for your review, and we hope that you’ll support our acceptance! If you have any questions between now and the end of the discussion period, we’ll do our best to address them.

Best,

The Authors

Dear Reviewer MLQA,

Thank you again for your positive feedback. If you have other questions about the paper, let us know and we'll try to answer before the end of the discussion period (in 12 hours). Thanks again!

The Authors

Reviewer_uREhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper sumarized that current LLM arbitrators are evaluated based solely on reference agreements, however, from a legal realist perspective, parties need predictability after selecting an arbitrator to decide whether to refuse, renegotiate, or commit. The authors studied the performance of eight arbitrators based on large language models in e-commerce disputes and article deletion disputes, and proposed that predictability should be incorporated into the routine evaluation index system just like accuracy.

Critical strengths
  1. The concepts are clearly formalized and relevant to decision-making. The paper transforms legal prediction theory into computable indicators and separates observer-eliminable errors from the model's inherent randomness through Brier decomposition. And this paper is clearly written and the figures are well presented.

  2. The empirical design is relatively robust. The strong policy implications of procedural disclosure not purchasing interpretability and records only revealing the base rate are directly supported by the F4 baseline rate comparison and individual component intervals.

Critical weaknesses
  1. The data scale and statistics are relatively small. Each scenario only has about 100 contested cases and 2 LLM observers, and similar settings are also relatively small. It's true that legal datasets, especially recent data, are difficult to collect, but to my knowledge, there are still some historical legal datasets collected by other countries or institutions that are usable.

  2. F1 and J1, F2 and J3 share the same model family, while the paper also reports that the legibility ranking is observer-dependent. This could potentially introduce bias.

  3. The contributions of this paper seem to focus primarily on providing an auxiliary metric, lacking a fundamental methodology or any truly essential meaning for the adjudicator.

Questions for authors

Questions are the same as the weekness, thanks.

2 comments

Dear Reviewer uREh,

Thank you for your review! We’ve added around 2.2k USD’s worth of experiments, and we believe we have addressed your concerns. Going over each one:

The data scale and statistics are relatively small.

[…] historical legal datasets collected by other countries or institutions that are usable.

On sample size, we now have 200 contested cases per dataset, split equally between the original sample (which we now call Analysis to distinguish it from the others) and the new sample (Holdout), plus 100 cases outside the contested sample (Comparison). We added two observer families, prompt and rule interventions, self-prediction tests, and a history-length analysis through 500 rulings. We expanded the samples within the same two settings.

Under dispute-only forecasting, our simultaneous intervals did not distinguish any judge pair on VerdictBench Holdout. On AfD Holdout, we found differences between judges with all four observers. We also found that the one-sided AfD Comparison cases were much easier to forecast: the ensemble's rules-only excess Brier error is 0.011 [0.003, 0.022] (Appendix D).

F1 and J1, F2 and J3 share the same model family, while the paper also reports that the legibility ranking is observer-dependent. This could potentially introduce bias.

To address the shared model families, we added F3 (DeepSeek V4.1 Flash) and F4 (GLM 5.3 Flash); neither family appears in the judge roster. We report all four observers' matrices, repeat variability and comparisons. We also checked ensembles excluding F1 or F2, the F3-F4 ensemble, and results excluding J1/J3 as targets.

On AfD Holdout, all six dispute-only observer-rank correlations are 0.762-0.929. On VerdictBench Holdout, F1-F2 is −0.048 with a wide interval containing zero.

The contributions of this paper seem to focus primarily on providing an auxiliary metric, lacking a fundamental methodology or any truly essential meaning for the adjudicator.

On the topic of methodological contribution, we distinguish observer error from judge randomness in Proposition 1, then use a finite-rerun correction to estimate that error without bias. In Proposition 2, we derive a sufficient safety margin for participation from the forecast error. On the topic of metrics, reference agreement tells us how often a judge matches human decisions, while our forecast-error measure tells us how well a party can anticipate the judge's ruling. The latter, as we show in the paper, is critical when picking machine adjudicators in a way that regular accuracy doesn’t capture alone.

We’ve also added further tests: We tested observer selection on 50 cases per dataset reserved for observer selection (History). We compared four observers and their equal-weight average (ensemble), then fixed the selected ensemble before Holdout evaluation. On VerdictBench, its dispute-only error is 0.099 [0.086, 0.114], compared with F1's 0.134 [0.110, 0.157]. The improvement is 0.034 [0.015, 0.054]. We report all candidates in Table 14, including the unresolved comparisons on AfD. We evaluated a participation decision. We set the payoff to +1 for A, −1 for B and 0 for abstention, with a judge sampled uniformly from the tested roster. Accepting only when the rules-only forecast satisfies q≥0.7 covers 35% of VerdictBench Holdout cases and gives estimated payoff 0.263 [0.187, 0.337] per incoming case, compared with 0.127 for always accepting (Appendices B.4 and D).

We also revisited the short-history finding you mention:

records only revealing the base rate

We extended the history analysis to 500 past rulings. The text-only linear model improves on the VerdictBench base-rate baseline by 0.026 [0.014, 0.039], including a small judge-specific component (Table 20 in the revised paper).

Thank you again for your review, and if we’ve addressed your concerns, we hope you’ll consider raising our score.

We know that the remaining discussion period is short (we spent all of it collecting additional results), but if you have any further questions, we’ll try to answer them before the end!

Best,

The Authors

Dear Reviewer uREh,

Thank you again for your feedback. In case you have other questions about the paper, let us know and we'll try to answer before the end of the discussion period (in 12 hours).

The Authors

Reviewer_D3mmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes adjudicative legibility as a criterion for evaluating LLM adjudicators. Legibility is defined as the Brier score of an external observer's forecast of an adjudicator's ruling on an unseen dispute, where the observer sees the dispute and, optionally, the adjudicator's published rules and a record of its past rulings. Using eight LLM adjudicators on contested disputes from two institutions, the authors argue that adjudicators with similar reference agreement differ substantially in legibility, and that neither short procedural rules nor small records of past rulings close this gap.

Critical strengths

The paper addresses a question of clear practical relevance. The authors point out that, unlike most uses of LLMs, an adjudicator is chosen before the disputes it will rule on occur, and they use this to shift the evaluation target from whether a judge agrees with a reference to whether a party can anticipate its rulings. The resulting concept could be useful to the community beyond this paper.

The framework is formally grounded, and the statistical analysis is careful. Each adjudicator is rerun five times per test case to estimate its ruling probability, and a finite-sample correction is applied so that the adjudicator's own noise is separated from illegibility. Comparisons between adjudicators use paired bootstrap and simultaneous intervals that hold jointly for all 28 pairs.

Some of the negative results are directly useful to practitioners. For instance, the LLM observers read a non-existent policy into noise: they forecast the seeded-coin control C1 as though it followed a policy.

Critical weaknesses

The main conclusions depend heavily on the choice of observer, and the observers themselves are not clearly reliable. In the dispute-only condition, the point estimates of the Spearman correlation between F1's and F2's rankings of the eight adjudicators by illegibility are only 0.05 on e-commerce disputes and 0.00 on deletion debates (Table 5), and F1's forecasts vary noticeably between repeated runs (Appendix B.5). The authors treat the observer-relativity of legibility as a feature of the definition, but it also means that the ranking reflects the observer to a considerable extent, not only the adjudicator. If the ranking changes with the observer, it is unclear which observer a party should rely on in practice.

With only 100 test cases per dataset, the evidence is too limited to support broad institutional conclusions. VerdictBench cases are sampled with a juror margin of at most 7 of 17, which admits near-ties such as a 9 to 8 vote. On such cases, the jury majority is itself close to a coin flip, so the reference is of limited reliability. The paper also does not include a comparison set of non-contested disputes, which leaves open whether the findings generalize to typical cases.

The C2 control has limited power. The authors themselves describe the original constitutions as short and procedural, roughly a hundred words asking for impartiality, attention to the evidence, and a binary ruling, so they differ only modestly from C2's vague constitution. That C2 and J7 show no measurable difference in legibility is therefore not enough to conclude that legibility is independent of how specific the rules are.

Questions for authors
  1. Could the authors add more observers, for example from other model families or human forecasters, and report which conclusions hold across observers?
  2. Could they also report reference agreement and legibility on a random sample of non-contested disputes, to clarify the scope of the findings?
2 comments

Dear Reviewer D3mm,

Thank you for your feedback! We’ve added many experiments, at the cost of around 2.2k USD, which we hope should address your concerns. Going one by one:

With only 100 test cases per dataset, the evidence is too limited to support broad institutional conclusions.

We've expanded each dataset to the 100 original contested cases (which we named Analysis to distinguish them from the others), 100 additional contested cases (Holdout), and 100 cases outside the contested sample (Comparison).

We also checked whether the judge-pair legibility differences carried over to Holdout. Under dispute-only forecasting, the simultaneous intervals did not distinguish any pair on VerdictBench. On AfD, we found differences between judges with all four observers.

The main conclusions depend heavily on the choice of observer, and the observers themselves are not clearly reliable.

On observer reliability, we added F3 (DeepSeek V4.1 Flash) and F4 (GLM 5.3 Flash), whose families aren't on the judge roster, and ran three forecasts per Analysis case and condition. Their mean absolute differences between repeats range from 0.060 to 0.109. We've included full matrices, comparisons, rank uncertainty, calibration and family-exclusion checks in the revised paper. On AfD Holdout, all six dispute-only observer-rank correlations are 0.762-0.929.

it is unclear which observer a party should rely on in practice.

To pick an observer, we tested the four individuals and their equal-weight average (ensemble) on 50 cases per dataset reserved for observer selection (History). We chose the ensemble before evaluating it on Holdout. On VerdictBench, its dispute-only excess Brier error is 0.099 [0.086, 0.114], compared with F1's 0.134 [0.110, 0.157]. The improvement is 0.034 [0.015, 0.054]. We found lower error than for every individual observer under both information conditions, with pointwise intervals excluding zero. On AfD, we couldn't distinguish the ensemble from F2, or from F3 in the dispute-only condition (see Appendix B.4, Table 14).

Could they also report reference agreement and legibility on a random sample of non-contested disputes, to clarify the scope of the findings?

For the Comparison cases, we sampled VerdictBench disputes outside the original juror-margin cutoff. On AfD, we used one-sided discussions: 93 Delete-only and seven Keep-only.

We found agreement rates of 0.626-0.850 on VerdictBench Comparison and 0.955-1.000 on the 94 AfD Comparison cases with binary closures. Ensemble rules-only error is 0.067 [0.055, 0.080] and 0.011 [0.003, 0.022], respectively. The one-sided AfD cases were much easier to forecast than the contested cases.

On such cases, the jury majority is itself close to a coin flip, so the reference is of limited reliability.

We agree that a narrow jury majority should not be treated as ground truth: instead, we use it to measure reference agreement (which, as we state in the paper, is orthogonal to accuracy/correctness). For legibility, we score forecasts against repeated judge rulings, without using the jury majority as the target. We also report results by juror margin and distance from the full buyer vote share. On VerdictBench Holdout, the mean agreement is 0.533 [0.451, 0.625] for margins 1-3 and 0.645 [0.549, 0.732] for margins 5-7 (Appendix D).

The C2 control has limited power.

We agree that the original C2 comparison was too weak to rule out an effect of rule specificity. We added rules specifying requirements in terms of evidence and tie-breaks for J3/J7 on 50 cases per dataset, with five rulings per condition and new controls under the original instructions. We showed observers the exact rules. The new rules changed both judges' ruling probabilities in both datasets (all four intervals exclude zero). The forecasting effects are mixed: F1 improves on VerdictBench/J7 by 0.092 [0.021, 0.175], but F2's point estimate worsens (Tables 15-16). We've revised the claim about specificity to match.

We hope that our new results address your concerns, and if they do, you will see fit to raise your score. We once again thank you for your review and look forward to the rest of this discussion period!

Best,

The Authors

Dear Reviewer D3mm,

Thank you again for your feedback. If you have further questions about the paper, please let us know and we'll try to answer before the end of the discussion period (in approximately 12 hours).

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thank you for your reviews + all our new experiments

Dear reviewers,

Thanks for your feedback, which has helped us significantly strengthen the paper. We believe that our new results, which were collected at the cost of 2.2k USD, address all reviewers’ concerns. We’ve added individual responses to each reviewer, but as a summary:

  1. To address D3mm's and uREh's concerns about sample size, we've doubled the contested sample and added 100 cases outside the contested sample (which we now refer to as Comparison) per dataset. We now have 200 contested cases in each dataset, split equally between the original sample (Analysis) and the new sample (Holdout), for 600 cases in total.
  2. To address observer reliability and shared model families (D3mm, uREh and MLQA), we added F3 (DeepSeek V4.1 Flash) and F4 (GLM 5.3 Flash), whose families aren't on the judge roster, and ran three forecasts per Analysis case and condition. See Appendix D for the full results.
  3. For D3mm's question about which observer a party should rely on, we compared the four individuals and their equal-weight average (ensemble) on 50 cases per dataset reserved for observer selection (“History”). We selected the ensemble and fixed that choice before Holdout evaluation. On VerdictBench, its dispute-only excess Brier error is 0.099 [0.086, 0.114], compared with F1's 0.134 [0.110, 0.157]. The improvement is 0.034 [0.015, 0.054]. We found lower error than for every individual observer under both information conditions, with pointwise intervals excluding zero. On AfD, we couldn't distinguish the ensemble from F2, or from F3 in the dispute-only condition. We report every candidate in Table 14. The abstention analysis in Appendices B.4 and D also addresses uREh's concern about practical value.
  4. For D3mm's concern about the weak C2 control and MLQA's prompt-sensitivity question, we reran J3/J7 on 50 cases per dataset, collecting five rulings per case under the original, paraphrased, urgent and vague instructions, and substantive rules. We showed observers the corresponding judge instructions. The substantive policies changed ruling probabilities in all four judge/dataset comparisons, but their forecasting effects were mixed (Tables 15-16). For MLQA, we also varied the observer's task wording (Table 17), compared self-prediction with forecasting a generic judge (Table 18), and measured consistency between similar cases without an LLM forecaster (Table 21). We found no clear self-prediction advantage in the four comparisons.
  5. On generalization (D3mm and uREh), the simultaneous intervals did not distinguish any judge pair on VerdictBench Holdout under dispute-only forecasting. On AfD Holdout, we found differences between judges with all four observers, and their rankings agreed more strongly. The one-sided AfD Comparison cases were much easier to forecast: ensemble rules-only error is 0.011 [0.003, 0.022].
  6. We also revisited the short-history finding noted by uREh and MLQA, extending the analysis to 500 rulings. The text-only linear model improves on the VerdictBench base-rate baseline by 0.026 [0.014, 0.039], including a small judge-specific component (Table 20).

Overall, we’re very thankful for the reviewers’ feedback, and with apologies for the short time left before the end of the discussion period (we were busy collecting all of these results), we’re looking forward to further productive discussion!

Best,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.