acceptodds
Withdrawn from ICLR 2027

JurisBench-Omni: A Judicial-Practice-Oriented Benchmark for Legal Omni-Modal Evaluation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models (LLMs) have achieved significant progress in *multi-modal* tasks; however, they still face challenges in the *omni-modal* understanding of real-world signals. Court hearings constitute one of the most consequential stages of judicial practice and are inherently omni-modal: judges, litigants, and witnesses jointly convey legal meaning through oral arguments, facial expressions, bodily actions, procedural contexts, spatial environments, causal relations, and other symbolic cues (e.g., silence). Much of this meaning is compressed into written judgments. Current legal AI evaluation remains predominantly text-centric or treats audio and video as separated and auxiliary signals appended to general-purpose multi-modal templates and, as a result, fails to assess whether models can comprehend judicial hearings in a human-like manner by integrating speech, actions, and intentions into a coherent, continuous task representation. We introduce JurisBench-Omni, a benchmark for legal omni-modal evaluation constructed directly from real hearing videos and their corresponding judgment documents. The benchmark decomposes judicial competence into four omni-modal dimensions: *role attribution*, *intention understanding*, *evidence recognition*, and *hearing-to-judgment grounding*.

Primary area:
datasets and benchmarks
Keywords:
omni-modal benchmark, legal AI, court hearing understanding, audio-visual reasoning, multimodal large language models, judicial evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_rERbOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

JurisBench-Omni contains 1,650 multiple-choice questions derived from 150 Chinese civil hearings paired with judgment documents. Its eleven tasks cover role attribution, intention understanding, evidence recognition, and hearing-to-judgment grounding. Validation includes two fully evaluated omni-modal models, a partial evaluation of a third model, modality ablations, and case-level analyses.

Critical strengths

The use of authentic hearings paired with corresponding judgments addresses a relevant gap in text-centered legal evaluation and could support further research on courtroom understanding.

Critical weaknesses
  1. The benchmark contribution is insufficiently established, and the related-work positioning is incomplete. The paper does not adequately distinguish its judicial-practice framing from recent professional legal benchmarks such as JurisBench, or systematically compare its capabilities and data coverage with existing multimodal legal resources. JurisBench already evaluates tasks organized around judicial workflows. aclanthology.org With only 150 civil cases and several tasks centered on speaking activity, gaze, movements, and temporal alignment, the submission provides limited evidence of a substantial new legal capability formulation. In particular, these observable behaviors do not directly establish “intention understanding.” The overview figure relies on generic illustrations and crowded text boxes, without clearly explaining the novel challenges or the logic connecting the tasks to judicial practice.

  2. The comparisons and analyses are too limited to validate the benchmark’s significance and reliability. Only two models receive full evaluation, and the ASR-transcript baseline central to the motivation is missing. Much of the analysis describes aggregate accuracy and plots, without investigating the causes of failure. Figure 6 raises unresolved concerns: G3 achieves 56% with questions and options alone, exceeding its 45.3% full-input accuracy, while G2 scores 60% with either audio-only or full input. Moreover, G3 asks about content in an unseen judgment document, making its answer potentially underdetermined. Missing human baselines and agreement statistics, together with the mismatch between the public release and manuscript task definitions, further weaken confidence in the conclusions.

Questions for authors
  • What substantive capability or diagnostic insight does this benchmark establish beyond existing judicial-workflow benchmarks and multimodal legal datasets?
  • How can G3 be answered reliably without the judgment document? What is human performance under exactly the same input conditions?
  • Do the claimed multimodal benefits remain after comparison with ASR-transcript and majority-label baselines?
  • Which released dataset and prediction version corresponds to the task definitions and results in this submission?
Reviewer_PiKbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

JurisBench-Omni evaluates audio-visual understanding of Chinese civil court hearings. It contains 150 real cases paired with judgment documents and 1,650 multiple-choice questions across 11 tasks. The tasks cover participant roles, observable behavior, evidence discussions, and connections between hearings and judgments. The authors evaluate two main models and conduct input-modality ablations.

Critical strengths
  1. The benchmark uses real hearing recordings and corresponding judgment documents, with annotation and review by people with legal training. Overall, it provides a useful test set to evaluate the model's capability in judicial scenario.
  2. The task breakdown and modality ablations provide useful information about which aspects of hearing understanding remain difficult.
Critical weaknesses
  1. A transcript-based baseline is missing. The paper argues that transcription loses important information, but does not directly compare audio-video models with a text model receiving the ASR transcript (cascaded system). This comparison would help establish the added value of audio and visual cues.
  2. The paper does not report a human baseline. Asking independent human participants to answer the questions using the same inputs as the models would provide a performance reference and help check whether the questions are answerable.
  3. G3 may not be answerable from the provided input. It asks whether hearing content appears in the final judgment, but the judgment is not given to the model (line 289). A hearing clip alone may not determine what the final document actually includes.
Questions for authors
  1. How can G3 be answered without access to the final judgment? Could you clarify what information in the provided clip determines the answer?
  2. How does an ASR-transcript-based text model perform on the same questions? How do human export perform in JurisBench-Omni?
Reviewer_wu6LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces JurisBench-Omni, a benchmark for understanding Chinese civil court hearings using audio and video. It includes 150 cases with matched judgments and 1,650 multiple-choice questions on participant roles, intentions, evidence, and links to the judgment. Results from two models show that audio and video help when used together, but many tasks remain challenging.

Critical strengths

The benchmark studies a useful problem with real hearing videos and matched judgment documents. The tasks go beyond legal knowledge and judgment prediction to examine how well models understand the hearing itself.

The modality ablations are helpful. Full audio-video input performs better overall than audio or video alone (Figure 6), while the task-level results show where each modality helps.

Critical weaknesses

G3 asks whether hearing content appears in the final judgment, but the judgment is not provided to the model. It is unclear how this can be answered reliably. The higher score with question-only input also needs explanation.

The paper does not report annotation agreement or human performance using the same inputs as the models. These checks would help determine whether low scores reflect model limitations or insufficient context. Parsing and interface failures should also be reported separately.

A transcript-only baseline is missing, despite the motivation that transcription loses important information. Also, the “intention understanding” tasks mainly assess behavior and temporal alignment, so this description seems broader than what is actually tested.

Questions for authors

Questions For Authors: See the “Critical Weaknesses” section.

Other minor comments:

  1. Please report confidence intervals for the main results.
  2. Table 3 should account for the different chance levels across tasks rather than use a common 50% color threshold.
Reviewer_KckqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents JurisBench-Omni, a benchmark of real Chinese court hearings (150 cases, 1,650 questions, 11 tasks) for evaluating omni-modal (joint audio-video) understanding of judicial hearings. Its claims are supported by human verification of the questions and by evaluating two models, including modality ablations.

Critical strengths
  • The motivation for why this matters is clear, and it resolved many of my initial questions.
  • Genuine recordings, a good number of case categories, and a reasonable filtering and QA pipeline.
Critical weaknesses
  • Only two models are mainly evaluated, which cannot justify conclusions. Limited max video input length shouldn’t be a make-or-break reason to exclude models, since there are ways around it (chunking, frame sampling). Generally, benchmarks need good representation of families and sizes. I am aware of many other candidates, so I would expect that 3-4 more could help the paper significantly, because two models cannot show whether the findings generalize. Additionally, it is a single run per model, with no bootstrap CIs or significance tests. I would expect the next version to have bootstrap CIs and repeated runs, and preferably report decoding settings.

  • The best way to justify a benchmark where models score badly is to show that the skill being tested is very important to people. The topic here, judicial drafting, is important, and the paper makes a fair point that relying only on speech-to-text may present issues. But the results shown are mixed because, for instance, in the Qwen Flash ablation (and note that this is the only model ablated), audio-only inputs are well above chance on G1, G2 and G3, and even text-only inputs are above chance on G2 and G3, and more importantly, the tasks that most need video (R1, R2) are near chance. Video clearly helps on I2 and I3, but the paper never shows that these skills matter for any real workflow. Also, audio models can already help tell speakers apart, detect emotion, and pick up other cues from speech patterns. So the question is why video specifically matters and why anyone should work on omni-modality instead of just improving audio models, which presumably would be less effort. What could help are examples where visual cues change the outcome and audio alone fails, plus a few model outputs and failure cases. Without further evidence, the results themselves are not supporting the paper’s main premise that video is needed.

Questions for authors
  • All hearings and prompts are in Mandarin, which may favor Chinese-trained models like Qwen. Could this explain the gap between Gemini and Qwen?

Suggestions (not questions):

  • Figure 1 is a bit blurry. Please use a vector PDF and check alignment and consistency of objects. It also looks AI-generated to me. If so, the AI Use Statement should mention it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.