- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper proposes a branch-aware benchmark for multi-turn legal consultation. The authors argue that the advice for a given legal consultation case should depend on the user's varying goals, even when the meta case facts stay the same. Therefore, they design a benchmark in which each consultation case is decoupled into multiple branches with varying goals to more comprehensively evaluate LLMs' capabilities in legal consultation.
Critical strengths
S1. The idea of branch-aware evaluation on legal consultation is well-motivated.
S2. The number of meta cases and branches is substantial.
S3. Two practicing lawyers have manually reviewed and corrected all the cases.
Critical weaknesses
W1. All the meta cases and branches are LLM-generated. Although two lawyers have reviewed and corrected all the data, it is unclear what review/correction standards have been applied to ensure the practicality of the LLM-generated advices.
W2. Although the dataset covers 756 categories of civil cases, only two practicing lawyers were involved in reviewing and revising the legal advices. These categories span a wide range of specialized legal domains, and only very few lawyers can possess deep expertise across as many as 756 subfields. Therefore, I am skeptical that relying on just two lawyers is sufficient to ensure the quality of all LLM-generated legal advices.
W3. The benchmarking protocol relies on LLM-as-a-judge to evaluate the advice quaility. Although the authors have conducted human evaluation to see the agreement between expert rankings and the LLM judge, they do not report inter-expert (human–human) agreement, so it is unclear whether the judge’s agreement approaches the level of agreement among human experts.
W4. The range of the evaluated models is too limited, and the models are somewhat outdated. The authors only consider the GPT-5.4, Qwen-3.6, Qwen-3.5 families, and thus it is unclear how frontier LLMs and legal LLMs perform on this benchmark.
W5. For LLM-as-a-judge evaluation, only DeepSeek-V4-Pro is tested.
Questions for authors
Q1. How many years of legal practice experience do the two lawyers have, and what are their areas of specialization?
Q2. Why only DeepSeek-V4-Pro was selected as the judge?
8 comments
Response to Reviewer rEAZ (W4-W5)
| Method | Avg. Recall (%) ↑ | Avg. Precision (%) ↑ | Avg. F1 (%) ↑ | Avg. Advice ↑ | Avg. r_q (%) ↑ | Avg. r_a ↑ | Cost ↓ | Cost Saving ↑ |
|---|---|---|---|---|---|---|---|---|
| GPT-6.1-Sol | 68.37 | 26.01 | 36.20 | 2.16 | 23.42 | 1.25 | $85.13 | — |
| + CFCol (4B) | 80.72 | 24.72 | 37.39 | 2.29 | 27.55 | 1.43 | $60.88 | 28.49% |
| + CFCol (9B) | 80.72 | 24.72 | 37.39 | 2.32 | 27.55 | 1.48 | $61.36 | 27.92% |
| MiMo-v2.6-Flash | 62.55 | 21.77 | 32.30 | 1.90 | 18.68 | 1.12 | $8.82 | — |
| + CFCol (4B) | 75.15 | 20.91 | 32.72 | 2.06 | 21.54 | 1.29 | $5.01 | 43.20% |
| + CFCol (9B) | 75.15 | 20.91 | 32.72 | 2.09 | 21.54 | 1.34 | $5.04 | 42.86% |
| GLM-5.3 | 65.05 | 22.68 | 33.63 | 2.09 | 20.43 | 1.45 | $60.43 | — |
| + CFCol (4B) | 77.89 | 21.43 | 33.61 | 2.20 | 23.14 | 1.56 | $30.00 | 50.36% |
| + CFCol (9B) | 77.89 | 21.43 | 33.61 | 2.22 | 23.14 | 1.63 | $30.23 | 49.98% |
W5 & Q2: Choice of a single judge.
Thank you for this question. We use a single judge for three reasons. First, a single strong LLM judge is common practice in legal consultation benchmarks. For example, LeCoDe [1] uses GPT-4o as its only judge. Second, we collected the data and ran the experiments from January to August 2026. During that period, DeepSeek-V4-Pro was one of the most cost-efficient flagship models, which made large-scale evaluation affordable. Third, DeepSeek-V4-Pro differs from the backbone models of the consultation agents. This avoids self-preference bias, where a judge favors outputs from its own model family.
To address this concern further, we follow DLawBench [2] and add a multi-judge evaluation in this rebuttal. It uses three advanced models: Kimi K3, Gemini 3.8 Flash, and the original DeepSeek-V4-Pro. With the same evaluation prompt, the three judges score each legal advice response independently, and the median of the three scores is the final score. A concise version is shown below. Single: DeepSeek-V4-Pro only (from Table 3). Multi: median of three judges (from Table 13). All values are averaged over ID and OOD. The complete results are in Table 13 of the revised version.
The results show that the multi-judge scores stay close to the single-judge scores. They are slightly higher for most models, while the GPT-5.4 advice score is slightly lower. The conclusions do not change. CFCol consistently improves over every cloud-only baseline, and the baselines keep the same ranking. Together with the strong agreement with expert rankings, we argue that a single DeepSeek-V4-Pro judge is reliable.
| Method | Advice Mean ↑ (Single / Multi) | r_a ↑ (Single / Multi) |
|---|---|---|
| GPT-5.4 | 2.10 / 2.09 | 1.35 / 1.38 |
| + CFCol (4B) | 2.28 / 2.26 | 1.49 / 1.54 |
| + CFCol (9B) | 2.32 / 2.30 | 1.55 / 1.47 |
| GPT-5.4-mini | 1.78 / 1.89 | 1.10 / 1.28 |
| + CFCol (4B) | 2.00 / 2.06 | 1.21 / 1.33 |
| + CFCol (9B) | 1.98 / 2.06 | 1.20 / 1.33 |
| Qwen3.6-Plus | 1.56 / 1.72 | 0.88 / 1.04 |
| + CFCol (4B) | 1.81 / 1.95 | 1.00 / 1.19 |
| + CFCol (9B) | 1.83 / 1.91 | 1.02 / 1.12 |
Response to Reviewer rEAZ (Q1 and Reference)
Q1: The two lawyers' practice areas.
Both lawyers have over 15 years of legal practice experience. The first specializes in divorce-related property disputes, private lending disputes, commercial disputes, labor arbitration, workers' compensation claims, criminal defense, applications for release on bail pending trial, and traffic accident cases.
The second specializes in commercial disputes, marriage and family law, labor arbitration, real estate disputes, foreign-related disputes, and criminal defense.
Thank you again for your constructive feedback. We will incorporate your suggestions into the final version. We hope these responses address your concerns and welcome further discussion.
[1] Yuan, Weikang, et al. LeCoDe: A Benchmark Dataset for Interactive Legal Consultation Dialogue Evaluation. ACL 2026.
[2] Zhang, Li, et al. DLawBench: Evaluating LLMs Through Multi-Turn Legal Consultation. EMNLP 2026.
[3] McHugh, M. L. Interrater reliability: the kappa statistic. Biochemia Medica, 2012.
Response to Reviewer rEAZ (W1-W3)
Dear Reviewer rEAZ,
Thank you for your constructive feedback. We address your concerns in the point-by-point responses below.
W1: Review standards for LLM-generated advice.
We apologize for the confusion. First, the meta-cases are not freely generated by LLMs. Each one comes from a real, publicly accessible Chinese civil dispute that we retrieved by web search and that can be traced to its source. The LLM only de-identifies the case, structures it, and expands it into factual branches (Sec. 3.1, App. A.2), because branch-level multi-turn legal consultation data are very scarce.
Second, every case passes two layers of human verification under explicit standards.
Layer 1: Five law graduates take part in the review. Each case is independently reviewed by two of them, taking about 15 minutes per case (App. A.2). All of them have passed the National Legal Professional Qualification Examination and have internship experience at law firms.
Layer 2: After revision, a practicing lawyer gives each case a final review and corrects it where needed. Only cases that pass this review are included.
Advice quality review standard. We apologize for the confusion. As stated in lines 143–144 of the original submission, each piece of reference advice consists of four parts: recommended solution, applicable conditions, procedural steps, and precautions. To review advice quality, annotators and lawyers apply the top-score criteria of our LLM-as-a-judge evaluation rubric (lines 950–966 of the revised version) as the review standard. A piece of reference advice passes review only if it meets all three requirements:
We will state this standard explicitly in the final version.
W2: Sufficiency of two lawyers.
The two lawyers are the final layer of quality control, not the only reviewers. As described in the "Verify" and "Revise" stages of Sec. 3.1 and in App. A.2 (Details of the Generate–Verify–Revise Construction Pipeline), two qualified law graduates independently review each case before it reaches the lawyers, spending about 15 minutes per case.
Table 8 and the ablation of double versus single annotation in App. A.5 show the effect of this first layer. On the same 65 cases, lawyers found only 1 remaining substantive issue after double-annotator review, compared with 19 after single-annotator review. The first layer therefore already brings case quality to a high level, and the lawyers' review serves mainly as a final safeguard. This greatly reduces the lawyers' workload.
In addition, both lawyers are civil-law practitioners with over 15 years of experience. In China, civil practice routinely covers contract, marriage and inheritance, labor, tort, real estate and construction, and debt disputes. Given our collaborators' expertise, we deliberately limit the benchmark to Chinese civil law and leave other areas, such as criminal law, to future work.
W3: Inter-expert agreement.
In our response to W2 of Reviewer 1jif, we report a human–model consistency study of advice quality scores. We asked legal experts and the LLM judge to independently score the same 400 factual branches, following the protocol described in Sec. 3.2. The two achieved 89.5% exact agreement and a linearly weighted Cohen's κ of 0.832 (95% CI [0.814, 0.866]; bias-corrected bootstrap, 10,000 resamples). These results show almost perfect agreement between the LLM judge and human experts [3].
To assess inter-expert agreement, and following your suggestion, we asked two additional legal experts to independently score the same 400 legal advice responses using the same 0–3 rubric (200 responses per person). The experts assigned identical scores to 367 of the 400 responses (91.75% exact agreement), with a linearly weighted Cohen's κ of 0.925 (95% CI [0.897, 0.949]). This estimate also falls within the almost perfect range [3]. These results further support the reliability of our LLM-based evaluation of advice quality.
W4: More models.
Thank you for this question. When we ran our experiments, GPT-5.4 and the Qwen3.6 family were among the most advanced models available. To address this concern, we ran additional experiments on three newer models: GPT-6.1-Sol, MiMo-v2.6-Flash, and GLM-5.3. For brevity, the table below reports the ID and OOD averages for each metric. CFCol improves performance over each base model while reducing cost. This further supports both the effectiveness and the efficiency of our method.