- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
JurisBench-Omni contains 1,650 multiple-choice questions derived from 150 Chinese civil hearings paired with judgment documents. Its eleven tasks cover role attribution, intention understanding, evidence recognition, and hearing-to-judgment grounding. Validation includes two fully evaluated omni-modal models, a partial evaluation of a third model, modality ablations, and case-level analyses.
Critical strengths
The use of authentic hearings paired with corresponding judgments addresses a relevant gap in text-centered legal evaluation and could support further research on courtroom understanding.
Critical weaknesses
-
The benchmark contribution is insufficiently established, and the related-work positioning is incomplete. The paper does not adequately distinguish its judicial-practice framing from recent professional legal benchmarks such as JurisBench, or systematically compare its capabilities and data coverage with existing multimodal legal resources. JurisBench already evaluates tasks organized around judicial workflows. aclanthology.org With only 150 civil cases and several tasks centered on speaking activity, gaze, movements, and temporal alignment, the submission provides limited evidence of a substantial new legal capability formulation. In particular, these observable behaviors do not directly establish “intention understanding.” The overview figure relies on generic illustrations and crowded text boxes, without clearly explaining the novel challenges or the logic connecting the tasks to judicial practice.
-
The comparisons and analyses are too limited to validate the benchmark’s significance and reliability. Only two models receive full evaluation, and the ASR-transcript baseline central to the motivation is missing. Much of the analysis describes aggregate accuracy and plots, without investigating the causes of failure. Figure 6 raises unresolved concerns: G3 achieves 56% with questions and options alone, exceeding its 45.3% full-input accuracy, while G2 scores 60% with either audio-only or full input. Moreover, G3 asks about content in an unseen judgment document, making its answer potentially underdetermined. Missing human baselines and agreement statistics, together with the mismatch between the public release and manuscript task definitions, further weaken confidence in the conclusions.
Questions for authors
- What substantive capability or diagnostic insight does this benchmark establish beyond existing judicial-workflow benchmarks and multimodal legal datasets?
- How can G3 be answered reliably without the judgment document? What is human performance under exactly the same input conditions?
- Do the claimed multimodal benefits remain after comparison with ASR-transcript and majority-label baselines?
- Which released dataset and prediction version corresponds to the task definitions and results in this submission?