acceptodds
Under review as a conference paper at ICLR 2027

SG-LegalClarify: a Large-scale Benchmark for Legal Reasoning and Ambiguity Recognition in Singapore Law

Abstract

Singapore offers a distinctive testbed for legal reasoning by large language models (LLMs). Its legal system combines inherited English common law with jurisdiction-specific statutes, requiring models to distinguish local doctrine from generic common-law intuition. This setting also highlights the challenge of recognizing underspecification in lay legal queries, because the facts material to a legal outcome depend on the governing doctrine. Yet legal benchmarks rarely cover Singapore law or test this form of ambiguity recognition. To address these gaps, we introduce SG-LegalClarify, a large-scale benchmark for open-ended legal reasoning and ambiguity recognition in Singapore law. SG-LegalClarify contains 1,100 lawyer-authored source questions spanning 10 legal domains, including 340 fully specified and 760 underspecified questions. Each underspecified question has a fixed hidden-fact assignment and a fully specified counterpart, yielding 1,860 evaluation instances. Responses are graded with lawyer-authored Issue, Rule, Application, and Conclusion (IRAC) rubrics. We evaluate 19 LLMs, closed-book and with web search, using a multi-turn pipeline in which models decide whether to answer or clarify, and a user simulator reveals hidden facts only when asked. We find the benchmark far from saturated: the best models satisfy only 69.8% of mandatory rubric items closed-book (average 53.8%) and 77.8% with web search (average 63.0%). Models also struggle to recognize when clarification is necessary. We will release the public split of our benchmark, the model outputs on it, and the evaluation pipeline to support future research.

Primary area:
datasets and benchmarks
Keywords:
legal benchmark, legal reasoning, ambiguity recognition, underspecification, clarification, Singapore law, multi-turn evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_oNHHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SG-LegalClarify, a benchmark for legal reasoning and clarification in Singapore law. It contains 1,100 lawyer-authored source questions, including 760 with paired underspecified and fully specified versions, yielding 1,860 evaluation instances. Responses are evaluated using lawyer-authored IRAC rubrics. Experiments on 19 models show that web access increases the average mandatory rubric score from 53.8% to 63.0%, while models still struggle to determine when clarification is needed. The paper also uses human annotations to validate automated grading and parts of the interaction pipeline.

Critical strengths
  1. Fully specified and underspecified versions share the same underlying fact assignment and rubric, enabling comparisons between receiving facts upfront and eliciting them through interaction. Including fully specified questions also allows the benchmark to measure unnecessary clarification.

  2. Lawyers authored and reviewed questions across ten legal domains. Fine-grained rubrics and human validation provide a useful foundation for evaluating models on Singapore law.

Critical weaknesses
  1. Figure 1 labels a question as fully specified without stating the applicable road speed limit, while the rubric assumes 90 km/h. Speed limits can differ across expressway sections and affect the penalty. Unless an explicit default assumption is provided, asking about the speed limit is reasonable but could be counted as unnecessary clarification. This raises concerns about whether the benchmark consistently distinguishes necessary from unnecessary questions.

  2. Both annotators identified over-disclosure in four of the 24 simulator conversations evaluated. The aligned validation of the turn classifier covers only nine positive FS decisions. These errors could affect fact coverage and final scores, but their impact on the reported results has not been quantified.

Questions for authors
  1. Why is the Figure 1 question labeled FS without specifying the road speed limit? Are models given an explicit default assumption? How are other FS questions checked for omitted facts that could change the answer?

  2. How reliable are fact-request recognition and fact disclosure under turn-level human evaluation covering both positive and negative cases? On a representative subset, do US scores and US–FS gaps remain stable after correcting interaction errors?

2 comments
AuthorsOpenReview ↗

Response to Reviewer oNHH

Thank you for your thoughtful review. We appreciate that you believe our paired FS-US design and lawyer-authored rubrics are a useful foundation. Both of your concerns led to changes: a corrected Figure 1 and a new turn-level evaluation of both LLM helpers (E6).

W1/Q1. Figure 1 and omitted facts in FS questions. Figure 1 shows an abbreviated version of the item, shortened to fit the space, and the abbreviation dropped the speed limit. We agree that this wasn't clear. The original FS question, which models receive, is much longer and states the limit: "...around 115 KM/h where the speed limit is 90KM/H." Models get no default assumptions. FS questions are written to include every fact needed to determine the answer (§2.1), as this one does, so a question about the limit would rightly count as unnecessary. The revised Figure 1 includes the limit, and its caption notes that the example is abbreviated.

For other FS questions, every item was reviewed by a second lawyer who checked its specification label, among other fields, and could relabel it (App. A). Residual omissions also cannot explain our clarification findings: all 19 models see the same FS items, yet their FS false-positive rates range from 3.5% (Mistral 3 Large) to 64.1% (GPT-5.6 Luna), so the spread captures model behavior.

W2/Q2. Interaction errors. We added the turn-level evaluation you asked for, covering both positive and negative cases (E6). An annotator labeled a new sample from all 19 models: 152 turns, stratified by the classifier's call (request or not) and by FS versus US, 38 per cell, plus 403 fact rows from 120 simulator replies in 76 US conversations.

Turn levelResult
Classifier called a fact request74 of 76 confirmed (97%)
Classifier called no request70 of 76 confirmed (92%)
Classifier, weighted to the real mix of turns96% accuracy, 98% recall; κ = 0.89 with the annotator
Simulator, requested facts146 of 150 given (97%)
Simulator, unrequested facts243 of 253 withheld (96%); 8 of 76 conversations contain any

Agreement between annotators is κ = 0.85 on 40 doubly labeled turns, and κ = 0.95 and 0.80 for disclosed and requested facts on 25 doubly labeled replies.

Both helpers err rarely, and the more frequent simulator error, over-disclosure, can only raise US scores. Correcting it would only widen rather than close the FS-US gaps we report in the findings.

We hope these changes resolve your concerns and would be grateful if you could consider further raising your score.

Reviewer_oNHHOpenReview ↗

Official Comment by Reviewer oNHH

Thank you for the additional experiments, which address my some concerns. I will maintain my original positive score.

Reviewer_dPtWOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces SG-LegalClarify, a benchmark for evaluating legal reasoning and selective clarification under Singapore law. It contains 1,100 lawyer-authored source questions across 10 legal domains, yielding 1,860 evaluation instances with paired underspecified and fully specified questions. The authors evaluate 19 LLMs using a two-stage sufficiency gate, an LLM-based user simulator, and IRAC-based rubrics graded by a selected panel of three LLM judges. Results show that clarification decisions remain unreliable and that web search improves legal reasoning scores. While the benchmark addresses a relevant evaluation problem, the validity of its evaluation pipeline and the comparability of its experimental conditions remain concerns.

Critical strengths
  1. The dataset covers 10 domains of Singapore law and was authored and reviewed by qualified lawyers. The paired underspecified and fully specified questions provide a useful framework for studying clarification behavior alongside legal answer quality.

  2. The IRAC-based rubrics distinguish issue identification, rule recall, application, and conclusions. This offers more diagnostic insight than evaluating answers solely through an overall accuracy score.

Critical weaknesses
  1. A separate sufficiency gate determines whether a model can answer directly or must enter a clarification dialogue. This artificial routing changes the model’s subsequent behavior and does not faithfully measure an unconstrained, end-to-end clarification policy. Furthermore, the LLM-based simulator achieves only 89.1% fact-disclosure accuracy in a small validation subset, with all reported errors involving over-disclosure. The resulting scores may therefore reflect pipeline artifacts rather than the models’ intrinsic clarification capabilities.

  2. Web-enabled models use different search systems and substantially different tool configurations, making the reported gains difficult to attribute to model capability. Meanwhile, the production judge panel is selected using agreement on the same human-annotated reference set used for evaluation design; human annotators themselves exhibit only moderate agreement (Fleiss’ κ = 0.607), with no adjudication. These limitations weaken confidence in the reported rankings and performance differences.

Questions for authors
  1. How do the authors establish that the separate sufficiency gate does not materially distort clarification behavior? Have they compared the proposed pipeline against an end-to-end interaction setting in which models freely decide whether to answer or ask questions?

  2. Why is the production judge panel selected on the same human-annotated reference set used to assess judge agreement? Can the authors report results from an independently annotated test set evaluated by the fixed panel and adjudicated legal experts?

  3. How can the authors justify comparing models using provider-native search against models using a custom search pipeline with different tool-call limits and execution behavior? Would the main conclusions persist under a standardized search environment?

2 comments
AuthorsOpenReview ↗

Response to Reviewer dPtW (1/2)

Thank you for your thoughtful review. We appreciate that you believe our benchmark addresses a relevant problem with lawyer-authored data, a useful paired design and diagnostic IRAC rubrics. We explain the rationale for the gate and test it with new experiments (E1, E2), rerun search with a common tool (E4), and answer the simulator and judge-selection concerns with new analyses of human labels (E5, E6). The conclusions hold in every case; where numbers moved, we report so.

W1/Q1. Does the sufficiency gate distort clarification? We designed the gate over many rounds of pilots. We noticed that, without it, whether a model asked conflated recognition with style: some models asked often, while others seldom asked and instead enumerated scenarios with an answer for each. Both are valid ways to handle missing facts, so asking alone cannot show whether a model noticed the gap. The gate instead asks every model the same explicit question, whether the facts given are legally sufficient for a decisive conclusion (App. D), and routes it to fact gathering only if it says no. Models are thus told to ask, but only when they recognize the need, which isolates the key capability we target: ambiguity recognition, or recognizing that legally material facts are missing. Keeping the gate closed-book also makes the decision comparable across search conditions.

That said, we agree that an unconstrained policy is also worth measuring. We conducted E1 to rerun all 19 models with one unified prompt and no gate, letting each freely ask or answer, with the rest of the pipeline unchanged. It uses a stratified subset of 186 instances from 110 source questions, none from the judge-selection set (seed 42; 95% CIs from 2,000 bootstrap resamples of source questions).

Closed-book, 19 models, 186 instancesGateFree choiceDifference [95% CI]
US ask rate (TPR)66.4%72.0%
FS ask rate (FPR)31.5%34.3%
Balanced accuracy0.6740.689
Mandatory score, overall59.859.3−0.5 [−1.4, +0.4]
Mandatory score, US56.957.4+0.6 [−0.7, +1.9]
Paired FS minus US gap (points)6.44.3
r(TPR, FPR) across models0.86 [0.61, 0.92]0.89 [0.60, 0.94]

Without the gate, models ask somewhat more on both US and FS items, balanced accuracy barely moves, and mandatory scores stay within noise. The FS-US gap persists, though smaller (6.4 to 4.3 points), and the coupling between detecting missing facts and over-asking is unchanged (r = 0.86 versus 0.89). Because the free-choice runs are new, we also checked for API drift (E2): rerunning the gate call two months later agrees with the original run as closely as two of its seeds agree with each other (κ = 0.82 versus 0.81), with no model drifting. Across seeds, model rankings by detection rate correlate at ρ = 0.93, and the paper's model-level correlations reproduce exactly from the stored runs. The revision adds the pilot rationale to §3.2 and reports both protocols in App. H.

W1. Simulator over-disclosure. All 14 errors in the submission's audit were over-disclosures, and the simulator never withheld a requested fact there (98/98). A new reply-level annotation (E6) of 76 US conversations from all 19 models shows that errors are rare in both directions: 96% of unrequested facts are withheld (243 of 253), 97% of requested facts are given (146 of 150), and 8 of 76 conversations (11%) contain any unrequested fact. Over-disclosure, the more frequent error, hands the model facts it did not ask for, so it can only raise US scores and shrink FS-US gaps. Our missing-fact findings are therefore, if anything, conservative.

AuthorsOpenReview ↗

Response to Reviewer dPtW (2/2)

W2/Q3. Heterogeneous search tools. We agree that mixing provider-native search with our own tool confounds the web-search comparison. E4 reruns the eight native-search models through the same Serper-Jina tool and limits as the other 11 models, on all 930 public-split instances (seed 42, paired with the paper's native seed-42 run), graded by the same judge panel and prompt.

ModelClosed bookNative searchCommon toolCommon minus native [95% CI]
Fable 570.574.376.2+2.0 [+0.9, +3.1]
Opus 4.867.973.875.4+1.6 [+0.5, +2.8]
Sonnet 562.270.872.3+1.4 [+0.2, +2.7]
GPT-5.6 Sol65.471.170.3−0.8 [−1.9, +0.2]
GPT-5.6 Terra60.368.366.6−1.6 [−2.9, −0.4]
GPT-5.6 Luna59.567.662.7−4.9 [−6.1, −3.7]
Gemini 3.1 Pro58.160.861.4+0.5 [−0.8, +1.9]
Gemini 3.5 Flash63.272.274.4+2.3 [+1.3, +3.3]

Differences and 95% CIs (paired bootstrap over instances) are computed before rounding. Search still helps all eight models over closed book (+3.2 to +11.2 points), and seven stay within 2.3 points of native search; native scores vary by up to 1.6 points across seeds, so we treat such differences as small. The OpenAI drop comes from tool use: GPT-5.6 Luna makes no search call in 39% of its common-tool trials. When the OpenAI models do search, they match native search within 1 point; their deficit lies almost entirely in trials without a search (10.6 to 11.8 points lower).

With these scores swapped in, the 19-model web ranking has Spearman 0.98 against the native-search ranking on the same instances: Kimi K3 stays first, the bottom seven are unchanged, and no model moves more than two places. Ranks 2 to 6 reshuffle within a 2-point band (72 to 76), inside seed noise, and the paper makes no ordering claims within that band. Gate accuracy moves by at most 3 points per model, as expected since the gate is closed-book in both conditions. The revision adds these results to App. G, with a one-sentence summary in §3.2, keeping provider-native search as the main web condition since it reflects how these models are deployed. For the camera-ready, we will extend the table to all three seeds.

W2/Q2. Judge selection and human agreement. We agree that selecting and evaluating a panel on the same reference inflates in-sample agreement, so we measure by how much with source-grouped five-fold cross-validation, which reruns the full selection over all 969 panels inside each training fold and scores only held-out sources. To address the lack of adjudication, E5 scores the panel on the criteria where both lawyers agree (about 80% of the 6,522), a high-confidence reference. There the panel matches the lawyers 93.0% of the time on held-out questions (κ = 0.86 [0.83, 0.89]) versus 93.6% in-sample (κ = 0.87), so selection inflates agreement by about 0.01 in κ. The production panel is selected in two of the five folds, and the result holds across 20 other fold assignments (κ = 0.85 to 0.86) and across IRAC dimensions (92.4% to 94.4%). On the moderate lawyer agreement (κ = 0.607), the panel agrees with each lawyer (κ = 0.701 and 0.694) more than the lawyers agree with each other, and lawyer disagreements recur on borderline calls, such as whether a fact is merely mentioned or actually applied (App. E).

We hope these results show that the concerns were addressable and that the conclusions survive each alternative design. If so, we would be grateful if you would consider raising your scores.

Reviewer_f7VaOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SG-LEGALCLARIFY, a legal consultation benchmark that pairs fully specified and underspecified questions to evaluate how effectively LLMs handle missing information and determine when clarification is needed. The benchmark provides a useful setting for investigating model performance under different levels of information completeness. The authors conduct comprehensive experiments involving 19 models, multiple runs, and both closed-book and web-enabled settings. However, the novelty of the proposed benchmark appears limited, given the similarities with existing legal consultation benchmarks. A more systematic comparison with existing benchmarks would help establish the unique contributions of SG-LEGALCLARIFY. Furthermore, the evaluation relies heavily on LLM-based components, including a user simulator and a three-model judging panel. Additional validation against human or legal expert evaluations is needed to establish the reliability of the reported results.

Critical strengths
  1. The design of pairing fully specified and underspecified questions is a useful setting for showing the model performance differences when the model given specified question or not, which can further verified the model’s understandable ability.
  2. The authors have done comprehensive experiments of evaluating 19 models, multiple runs, closed-book and web-enabled settings.
Critical weaknesses
  1. Limited novelty of the proposed SG-LEGALCLARIFY dataset. The authors acknowledge some other legal consultation benchmarks, such as LeCoDe, DLawBench, and FSDBench. Specifically, DLawBench already lets a model choose between further client questions and submitting a legal memorandum, which is very similar to the specified and unspecified question settings of SG-LEGALCLARIFY. It is recommended to have a systematic benchmarks comparison experiment.
  2. The evaluation relies a lot on LLM-as-a-judge. Such as user simulator and three-model judging panel. A detailed LLM-as-a-judge experiment about comparing human/legal expert evaluation is needed.
Questions for authors

Refers to the weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer f7Va

Thank you for your thoughtful review. We appreciate that you find our paired design useful and our experiments comprehensive. We address your novelty concern with a systematic comparison table and a new ranking analysis (E3), and your reliability concern with the expert validation already in the paper plus two new analyses (E5, E6).

W1. Novelty relative to DLawBench, LeCoDe and FSDBench. We agree DLawBench is the closest work: its model also chooses between asking the client and submitting a memo. The difference is what each benchmark can observe. Every DLawBench case begins from a neutral opening with facts withheld, and DLawBench does not score unnecessary questions (its authors list turn-level question scoring as future work). LeCoDe measures question efficiency only by turn count, and FSDBench scores fact recovery without grading the legal answer. SG-LegalClarify adds two novel design elements: labeled answerable inputs, so over-clarification is measurable, and US/FS pairs sharing one fact assignment and rubric, which isolate the cost of eliciting facts versus receiving them. We added the following table to the Related Work section:

BenchmarkLawLabeled answerable inputsSame facts given vs. elicited, one rubricLegal answer graded
LeCoDeChinaNoNoYes (overlap metrics, LLM ratings)
FSDBenchPoland (tax)NoNoNo (fact recovery only)
DLawBenchChina, U.S.NoNoYes (issue-resolution rubrics)
SG-LegalClarifySingaporeYes (1,100 FS)Yes (760 pairs)Yes (lawyer-written IRAC rubrics)

These elements change what a benchmark can see. Without answerable inputs, a benchmark cannot see that detecting missing facts moves in lockstep with over-asking (r = 0.90 across models, Figure 4). Ranking models by US detection alone, as a benchmark without FS items must, agrees only weakly with ranking by balanced gate accuracy (E3: Spearman ρ = 0.35 across 19 models). For example, GPT-5.6 Luna ranks 1st on detection but 15th once its 64.1% false-alarm rate on FS items counts, while Gemini 3.5 Flash rises from 13th to 3rd. Without pairs, a benchmark also cannot measure the missing-fact penalty, which all 19 models pay (Figure 5).

W2. Validation against legal experts. We agree this matters. Our submission already included meta evaluation (§3.4, App. E): nine lawyers labeled 6,522 rubric criteria on 558 conversations from three response models, two lawyers per criterion, and we compared all 19 candidate judges and 969 three-judge panels against them, with panel selection cross-validated by source question. Nevertheless, we conducted further validation for the LLM helpers. The table summarizes the human evidence for every LLM component that affects reported scores:

ComponentEvidenceResult
Judge panelExisting: 6,522 criteria, each labeled by two lawyersAgrees with each lawyer (κ = 0.701 and 0.694) more than the lawyers agree with each other (κ = 0.607)
Judge panelNew (E5): criteria where both lawyers agree; fresh source-grouped 5-fold CV93.0% agreement on held-out questions (κ = 0.86); 92.4% to 94.4% across IRAC dimensions
User simulatorExisting: 55 double-annotated tasksNever withheld a requested fact (98/98); all 14 errors were over-disclosures
User simulatorNew (E6): 403 fact rows from 76 US conversations97% of requested facts given; 96% of unrequested facts withheld
Turn classifierNew (E6): 152 turns, with and without fact requests96% accuracy on the real mix of turns; κ = 0.89 with the annotator

We hope these additions make the contribution and its reliability clear. If they address your concerns, we would be grateful if you would consider raising your score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank all three reviewers for their careful and constructive reviews. Their concerns converge on four questions: how SG-LegalClarify differs from recent consultation benchmarks, whether the sufficiency gate distorts behavior, how reliable the LLM judges and helpers are, and whether web-search comparisons are fair. We ran six new experiments and analyses. None changes the paper's main conclusions.

IDNew experiment or analysisRaised byHeadline result
E1Free choice vs. gate: one unified prompt, no gate; using a stratified 186-instance subset (seed 42)dPtWBalanced accuracy 0.674 with the gate vs. 0.689 without; mandatory score changes by −0.5 points [−1.4, +0.4]
E2Gate reproducibility: gate rerun two months later, and compared across seedsdPtWThe rerun agrees with the original run as closely as two seeds agree (κ 0.82 vs. 0.81); detection-rate rankings across seeds correlate at ρ = 0.93
E3Model ranking by US detection alone vs. by balanced gate accuracy (19 models, 3 seeds)f7VaSpearman ρ = 0.35; GPT-5.6 Luna ranks 1st on detection but 15th on balanced accuracy
E4The eight native-search models rerun with the search tool and limits of the other 11 (930 public-split instances, seed 42)dPtWSearch still helps all eight (+3.2 to +11.2 points); seven stay within 2.3 points of native search; 19-model ranking Spearman 0.98
E5Judge panel on the criteria where both lawyers agree, with a fresh source-grouped 5-fold cross-validationf7Va, dPtW93.0% agreement on held-out questions (κ = 0.86); 92.4% to 94.4% across IRAC dimensions
E6Turn-level annotation of the turn classifier and simulator (152 turns; 403 fact rows from 76 US conversations; all 19 models)oNHH, dPtWClassifier 96% accurate on the real mix of turns (κ = 0.89); simulator gives 97% of requested facts and withholds 96% of unrequested ones

Revised PDF (changes in blue): Figure 1 now includes the speed limit stated in the original question, and its caption notes that the example is abbreviated (§1); the pilot rationale for the gate and pointers to the robustness checks (§3.2); agreement on lawyer-consensus criteria (§3.4, App. E); common-tool search results (App. B, App. G); free choice vs. gate, gate reproducibility, and the detection-versus-selectivity ranking (new App. H); a benchmark comparison table (§5); updated limitations (§6).

We appreciate the reviewers' and Area Chair's consideration of the revised work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.