- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Adjudex-Labor, a benchmark for evaluating whether LLM deviations from human labor-dispute adjudications systematically favor employers or employees. Using 804 real cases with both arbitration and court outcomes, the paper proposes complementary metrics for overall disagreement and party-relative directional imbalance, and evaluates multiple LLM configurations in both single-stage and multi-stage settings. The main results show systematic employer-favoring deviations and further employer-ward shifts when downstream judgments rely on LLM-generated arbitration outcomes.
Critical strengths
- The paper identifies an important evaluation dimension that standard agreement or accuracy metrics miss. The distinction between DDR and PPR is useful. Two models can disagree with human adjudicators at similar rates while imposing very different distributive consequences across employers and employees. The paper also shows that higher institutional agreement can coexist with greater positional imbalance, which makes a strong case that aggregate legal accuracy alone is insufficient for evaluating LLM adjudication systems.
- The multi-stage adjudication design goes beyond isolated legal prediction. The paper studies how upstream arbitration outcomes affect downstream judicial decisions and evaluates all arbitrator-judge model combinations. This is important because deployed AI decision systems may operate in pipelines where one model's output becomes another model's input. The finding that positional tendencies can persist, amplify, or reverse across stages would not be visible in a single-stage evaluation.
- The paper is appropriately cautious about institutional references. It generally avoids treating human arbitration or court outcomes as normative ground truth and instead frames the benchmark in terms of directional deviation relative to institutional decisions. This is important for interpreting the reported employer-favoring and employee-favoring tendencies.
Critical weaknesses
1. The benchmark currently lacks sufficient auditability and reproducibility.
The paper's main contribution depends on a newly constructed benchmark derived from real legal documents, but the current submission does not provide the benchmark data, construction code, or even a small set of complete, inspectable examples. This makes it difficult to independently verify case preprocessing, label extraction, leakage control, prompt construction, and output parsing. Even if the underlying legal documents cannot be fully released because of licensing or privacy constraints, the paper should provide anonymized or synthetic representative cases together with benchmark construction and evaluation code.
2. The external validity of the reported employer-favoring tendency is difficult to assess because the benchmark is highly selected.
The 804 cases are not a random sample of labor disputes. They must proceed through both arbitration and subsequent litigation, retain employment-status disputes as a substantive issue, and contain sufficiently clear claims and binary institutional outcomes. These requirements are understandable, but they may induce substantial selection effects. The results therefore strongly support conclusions about this selected litigation population, but extrapolation to labor disputes more broadly or to real deployment populations is less well justified.
3. Generation-level robustness is insufficiently documented.
The paper does not report several important decoding parameters, such as temperature, top-p, random seed, or the number of generations per case. The reported case-level bootstrap quantifies uncertainty across benchmark cases, but it does not capture stochastic variation in repeated model generations for the same input. Since several central conclusions concern directional shifts at the configuration level, at least a representative repeated-generation or multi-seed analysis would be important for establishing their stability.
4. The main sequential experiment establishes outcome propagation, but not richer reasoning propagation.
In the large-scale analysis, the downstream judge receives only the binary upstream arbitration outcome. This is a clean design for isolating the effect of replacing a human outcome with an LLM outcome, but it does not test whether model-generated legal reasoning, evidence interpretation, or argumentation itself propagates and amplifies positional tendencies. The smaller monetary experiment partially addresses this, but remains exploratory. The main claim should therefore remain scoped to outcome propagation rather than stronger reasoning self-reinforcement.
5. The technical novelty is limited, and the paper could be more concise.
The main contribution lies in the benchmark framing, the party-relative evaluation perspective, and the empirical study of a multi-stage legal process, rather than in a substantially new technical method. DDR is essentially an overall disagreement measure, while PPR captures directional asymmetry in those disagreements. This is still a meaningful contribution, but the paper's novelty depends heavily on the benchmark and empirical framing. The manuscript also repeats several central observations across multiple sections, and a more concise presentation would improve information density.
Questions for authors
- Can the authors provide inspectable benchmark artifacts during review, at least in the form of several anonymized representative cases containing processed case materials, Human Arbitrator and Human Judge labels, the exact LLM input prompts, and representative model outputs? Can the benchmark construction and evaluation code also be made available?
- Can the authors report the complete generation setup, including temperature, top-p, random seed where applicable, maximum generation length, and the number of generations per model-case-condition? If stochastic decoding is used, can repeated-generation or multi-seed results be provided for a representative subset?
- How representative are the final 804 cases of the broader source population? Can the authors report the number of cases remaining after each filtering stage and, where possible, compare retained and excluded cases on observable characteristics?
- The paper is appropriately careful in treating human decisions as institutional references rather than normative ground truth. Should the core claim therefore be interpreted strictly as a systematic directional deviation relative to observed institutional decisions, or do the authors intend to make a stronger fairness or legal-bias claim?
- The appendix states that the 50-case monetary sample was selected after preliminary analysis. What exactly were the selection criteria, and were they fixed before inspecting model monetary outputs?
8 comments
(Q2) Generation Settings and Robustness
1. Generation settings (Appendix D.1). Table 5 reports the decoding settings for all 14 configurations in the 804-case experiment. We deliberately configured high maximum output-token limits to reduce truncation risks. Table 5 therefore reports the maximum observed completion tokens, distinguishing actual usage from configured limits.
2. Generation counts and experimental design (Appendix D.1). We retained one response per case and configuration under 3 input conditions, yielding 33,768 direct outputs across 804 cases and 14 configurations.
3. Repeated-generation experiments (Appendix D.4). We conducted two additional runs on a stratified subset of 60 employment-status cases across five representative configurations (Appendix D.4.1), and on all 50 monetary-award cases across 14 configurations (Appendix D.4.2). The appendices report the experimental procedures, output consistency, and stability of the results.
4. Statistical robustness (Appendix D.2). We report configuration-level 95% confidence intervals for DDR and PPR. In same-configuration chains, replacing human arbitration with LLM arbitration increased mean PPR by +8.53 pp (95% CI: [+5.09, +11.96]) toward employers and mean DDR by +9.53 pp (95% CI: [+7.81, +11.27]). These results support the directional shift under case-level resampling.
(Q3+W2) Sample Selection and Representativeness
We constructed the 804-case dataset to enable matched comparisons between arbitration and court decisions in the same labor dispute. Accordingly, retained cases must involve a substantive employment-relationship dispute with identifiable outcomes at both stages.
1. Case Retrieval and Screening. We retrieved approximately 15,000 first-instance judgments from China Judgments Online and PKULaw concerning the confirmation of employment relationships. After removing 2,320 duplicates, 12,680 documents remained. We subsequently excluded 3,540 documents with incomplete text and 3,130 lacking identifiable claims, defenses, or judicial examination of evidence, leaving 6,010 documents. We further excluded 5,081 cases in which employment status was not a substantive issue but appeared only as background or in procedural matters, leaving 929 cases.After removing 59 highly similar template-based cases and 66 cases without clearly identifiable arbitration outcomes, the final dataset comprised 804 cases.
2. Sample Composition and Scope. We characterize these 804 cases as a well-defined and analytically relevant subset rather than a statistically representative sample of broader labor disputes. All cases underwent both arbitration and subsequent litigation, making them suitable for examining positional tendencies across linked adjudicative stages. The sample includes 416 court-recognized employment relationships (51.7%) and 388 non-recognized relationships (48.3%), providing substantial observations in both reference-outcome groups for estimating PPR.
3. External Validity. Our selection criteria are designed to support matched comparisons across arbitration and litigation stages, requiring identifiable decisions and sufficient case materials at both stages. While this creates a selected litigation population rather than a representative sample of all labor disputes, it provides a well-defined setting for examining positional tendencies in sequential adjudication. Our findings should therefore be interpreted within this scope, where the benchmark offers systematic evidence of directional deviations from institutional decisions and their propagation across adjudicative stages.
(Q1+W1) Benchmark Auditability and Reproducibility
To address the reviewer's concerns, we have provided an anonymous review repository containing de-identified real-case examples, complete experimental inputs for these examples, model responses, case-level results, and evaluation code. All materials are accessible without sign-in.
Case materials and experimental auditability. We provide five de-identified real cases containing case facts, parties' arguments, key evidence, arbitration outcomes, court reasoning, and both human decision labels. These examples cover all four human arbitrator–judge outcome combinations, as well as both agreements and disagreements between model and human decisions.
Prompts, information separation, and output verification. The repository includes three original prompt templates, complete de-identified inputs for all five cases under the three conditions, and 15 corresponding raw responses from GPT-5.6-Sol NT. Reviewers can inspect how information is used across conditions, how upstream arbitration outcomes are introduced, and how model responses correspond to the scored labels.
Case-level data and reproducibility. We provide anonymized case-level results for 804 cases across 14 model configurations, together with runnable evaluation code. Without rerunning the models, reviewers can reproduce DDR, PPR, and the results for all 196 arbitrator–judge configuration pairings, including the central finding of an average PPR shift of +8.53 percentage points in same-configuration chains when replacing upstream human arbitration with LLM arbitration. The repository also includes execution instructions, dependencies, and expected outputs.
Appendix A of the revised manuscript further documents case screening, evidence processing, and label extraction.