- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces SG-LegalClarify, a benchmark for legal reasoning and clarification in Singapore law. It contains 1,100 lawyer-authored source questions, including 760 with paired underspecified and fully specified versions, yielding 1,860 evaluation instances. Responses are evaluated using lawyer-authored IRAC rubrics. Experiments on 19 models show that web access increases the average mandatory rubric score from 53.8% to 63.0%, while models still struggle to determine when clarification is needed. The paper also uses human annotations to validate automated grading and parts of the interaction pipeline.
Critical strengths
-
Fully specified and underspecified versions share the same underlying fact assignment and rubric, enabling comparisons between receiving facts upfront and eliciting them through interaction. Including fully specified questions also allows the benchmark to measure unnecessary clarification.
-
Lawyers authored and reviewed questions across ten legal domains. Fine-grained rubrics and human validation provide a useful foundation for evaluating models on Singapore law.
Critical weaknesses
-
Figure 1 labels a question as fully specified without stating the applicable road speed limit, while the rubric assumes 90 km/h. Speed limits can differ across expressway sections and affect the penalty. Unless an explicit default assumption is provided, asking about the speed limit is reasonable but could be counted as unnecessary clarification. This raises concerns about whether the benchmark consistently distinguishes necessary from unnecessary questions.
-
Both annotators identified over-disclosure in four of the 24 simulator conversations evaluated. The aligned validation of the turn classifier covers only nine positive FS decisions. These errors could affect fact coverage and final scores, but their impact on the reported results has not been quantified.
Questions for authors
-
Why is the Figure 1 question labeled FS without specifying the road speed limit? Are models given an explicit default assumption? How are other FS questions checked for omitted facts that could change the answer?
-
How reliable are fact-request recognition and fact disclosure under turn-level human evaluation covering both positive and negative cases? On a representative subset, do US scores and US–FS gaps remain stable after correcting interaction errors?
2 comments
Official Comment by Reviewer oNHH
Thank you for the additional experiments, which address my some concerns. I will maintain my original positive score.
Response to Reviewer oNHH
Thank you for your thoughtful review. We appreciate that you believe our paired FS-US design and lawyer-authored rubrics are a useful foundation. Both of your concerns led to changes: a corrected Figure 1 and a new turn-level evaluation of both LLM helpers (E6).
W1/Q1. Figure 1 and omitted facts in FS questions. Figure 1 shows an abbreviated version of the item, shortened to fit the space, and the abbreviation dropped the speed limit. We agree that this wasn't clear. The original FS question, which models receive, is much longer and states the limit: "...around 115 KM/h where the speed limit is 90KM/H." Models get no default assumptions. FS questions are written to include every fact needed to determine the answer (§2.1), as this one does, so a question about the limit would rightly count as unnecessary. The revised Figure 1 includes the limit, and its caption notes that the example is abbreviated.
For other FS questions, every item was reviewed by a second lawyer who checked its specification label, among other fields, and could relabel it (App. A). Residual omissions also cannot explain our clarification findings: all 19 models see the same FS items, yet their FS false-positive rates range from 3.5% (Mistral 3 Large) to 64.1% (GPT-5.6 Luna), so the spread captures model behavior.
W2/Q2. Interaction errors. We added the turn-level evaluation you asked for, covering both positive and negative cases (E6). An annotator labeled a new sample from all 19 models: 152 turns, stratified by the classifier's call (request or not) and by FS versus US, 38 per cell, plus 403 fact rows from 120 simulator replies in 76 US conversations.
Agreement between annotators is κ = 0.85 on 40 doubly labeled turns, and κ = 0.95 and 0.80 for disclosed and requested facts on 25 doubly labeled replies.
Both helpers err rarely, and the more frequent simulator error, over-disclosure, can only raise US scores. Correcting it would only widen rather than close the FS-US gaps we report in the findings.
We hope these changes resolve your concerns and would be grateful if you could consider further raising your score.