acceptodds
Under review as a conference paper at ICLR 2027

LawBranch: Branch-Aware Evaluation of Multi-Turn Legal Consultation

Abstract

Large language models (LLMs) hold promise for making legal consultation more accessible. In realistic legal consultation, however, user queries are often initially underspecified and incomplete, so legal experts must ask clarification questions over multiple turns to elicit missing legal facts. Since different answers can lead to substantially different legal reasoning and advice, existing benchmarks, which typically evaluate a single fixed scenario per case and report average-case metrics, obscure failures on consequential factual branches. Meanwhile, cloud-based LLMs achieve strong performance but incur high costs and privacy risks, whereas smaller local models are more affordable yet typically lack sufficient legal-reasoning capabilities. To address these gaps, we introduce LawBranch, a branch-aware Chinese civil-law benchmark comprising 1,500 human-verified cases and 5,399 factual branches. Further analysis reveals that both clarification and advice quality degrade consistently in multi-turn dialogue, while worst-branch reliability is at least 40% lower than the corresponding average-branch metric. We therefore propose Clarification-First Edge-Cloud Collaboration (CFCol), where a cloud agent handles clarification planning and legal advice generation, while an on-device agent and a memory manager are responsible for de-identification and dialogue coordination. Across diverse cloud and edge LLM instantiations, CFCol improves both clarification and advice performance, reduces cloud cost by at least 25%, and offers privacy advantages over cloud-only deployment.

Primary area:
datasets and benchmarks
Keywords:
Multi-turn Legal consultation, benchmark, worst-branch reliability, edge-cloud collaboration, privacy-preserving NLP, Chinese civil law

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_rEAZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes a branch-aware benchmark for multi-turn legal consultation. The authors argue that the advice for a given legal consultation case should depend on the user's varying goals, even when the meta case facts stay the same. Therefore, they design a benchmark in which each consultation case is decoupled into multiple branches with varying goals to more comprehensively evaluate LLMs' capabilities in legal consultation.

Critical strengths

S1. The idea of branch-aware evaluation on legal consultation is well-motivated.

S2. The number of meta cases and branches is substantial.

S3. Two practicing lawyers have manually reviewed and corrected all the cases.

Critical weaknesses

W1. All the meta cases and branches are LLM-generated. Although two lawyers have reviewed and corrected all the data, it is unclear what review/correction standards have been applied to ensure the practicality of the LLM-generated advices.

W2. Although the dataset covers 756 categories of civil cases, only two practicing lawyers were involved in reviewing and revising the legal advices. These categories span a wide range of specialized legal domains, and only very few lawyers can possess deep expertise across as many as 756 subfields. Therefore, I am skeptical that relying on just two lawyers is sufficient to ensure the quality of all LLM-generated legal advices.

W3. The benchmarking protocol relies on LLM-as-a-judge to evaluate the advice quaility. Although the authors have conducted human evaluation to see the agreement between expert rankings and the LLM judge, they do not report inter-expert (human–human) agreement, so it is unclear whether the judge’s agreement approaches the level of agreement among human experts.

W4. The range of the evaluated models is too limited, and the models are somewhat outdated. The authors only consider the GPT-5.4, Qwen-3.6, Qwen-3.5 families, and thus it is unclear how frontier LLMs and legal LLMs perform on this benchmark.

W5. For LLM-as-a-judge evaluation, only DeepSeek-V4-Pro is tested.

Questions for authors

Q1. How many years of legal practice experience do the two lawyers have, and what are their areas of specialization?

Q2. Why only DeepSeek-V4-Pro was selected as the judge?

8 comments
AuthorsOpenReview ↗

Response to Reviewer rEAZ (W1-W3)

Dear Reviewer rEAZ,

Thank you for your constructive feedback. We address your concerns in the point-by-point responses below.

W1: Review standards for LLM-generated advice.

We apologize for the confusion. First, the meta-cases are not freely generated by LLMs. Each one comes from a real, publicly accessible Chinese civil dispute that we retrieved by web search and that can be traced to its source. The LLM only de-identifies the case, structures it, and expands it into factual branches (Sec. 3.1, App. A.2), because branch-level multi-turn legal consultation data are very scarce.

Second, every case passes two layers of human verification under explicit standards.

Layer 1: Five law graduates take part in the review. Each case is independently reviewed by two of them, taking about 15 minutes per case (App. A.2). All of them have passed the National Legal Professional Qualification Examination and have internship experience at law firms.

Layer 2: After revision, a practicing lawyer gives each case a final review and corrects it where needed. Only cases that pass this review are included.

Advice quality review standard. We apologize for the confusion. As stated in lines 143–144 of the original submission, each piece of reference advice consists of four parts: recommended solution, applicable conditions, procedural steps, and precautions. To review advice quality, annotators and lawyers apply the top-score criteria of our LLM-as-a-judge evaluation rubric (lines 950–966 of the revised version) as the review standard. A piece of reference advice passes review only if it meets all three requirements:

  1. Solution and steps: they are complete, reasonable, and executable.
  2. Precautions: they are accurate, complete, and reasonable, and they cover the main risks.
  3. Legal correctness: the legal relationships, legal reasoning, and procedures are correct.

We will state this standard explicitly in the final version.

W2: Sufficiency of two lawyers.

The two lawyers are the final layer of quality control, not the only reviewers. As described in the "Verify" and "Revise" stages of Sec. 3.1 and in App. A.2 (Details of the Generate–Verify–Revise Construction Pipeline), two qualified law graduates independently review each case before it reaches the lawyers, spending about 15 minutes per case.

Table 8 and the ablation of double versus single annotation in App. A.5 show the effect of this first layer. On the same 65 cases, lawyers found only 1 remaining substantive issue after double-annotator review, compared with 19 after single-annotator review. The first layer therefore already brings case quality to a high level, and the lawyers' review serves mainly as a final safeguard. This greatly reduces the lawyers' workload.

In addition, both lawyers are civil-law practitioners with over 15 years of experience. In China, civil practice routinely covers contract, marriage and inheritance, labor, tort, real estate and construction, and debt disputes. Given our collaborators' expertise, we deliberately limit the benchmark to Chinese civil law and leave other areas, such as criminal law, to future work.

W3: Inter-expert agreement.

In our response to W2 of Reviewer 1jif, we report a human–model consistency study of advice quality scores. We asked legal experts and the LLM judge to independently score the same 400 factual branches, following the protocol described in Sec. 3.2. The two achieved 89.5% exact agreement and a linearly weighted Cohen's κ of 0.832 (95% CI [0.814, 0.866]; bias-corrected bootstrap, 10,000 resamples). These results show almost perfect agreement between the LLM judge and human experts [3].

To assess inter-expert agreement, and following your suggestion, we asked two additional legal experts to independently score the same 400 legal advice responses using the same 0–3 rubric (200 responses per person). The experts assigned identical scores to 367 of the 400 responses (91.75% exact agreement), with a linearly weighted Cohen's κ of 0.925 (95% CI [0.897, 0.949]). This estimate also falls within the almost perfect range [3]. These results further support the reliability of our LLM-based evaluation of advice quality.

W4: More models.

Thank you for this question. When we ran our experiments, GPT-5.4 and the Qwen3.6 family were among the most advanced models available. To address this concern, we ran additional experiments on three newer models: GPT-6.1-Sol, MiMo-v2.6-Flash, and GLM-5.3. For brevity, the table below reports the ID and OOD averages for each metric. CFCol improves performance over each base model while reducing cost. This further supports both the effectiveness and the efficiency of our method.

AuthorsOpenReview ↗

Response to Reviewer rEAZ (W4-W5)

MethodAvg. Recall (%) ↑Avg. Precision (%) ↑Avg. F1 (%) ↑Avg. Advice ↑Avg. r_q (%) ↑Avg. r_a ↑Cost ↓Cost Saving ↑
GPT-6.1-Sol68.3726.0136.202.1623.421.25$85.13—
+ CFCol (4B)80.7224.7237.392.2927.551.43$60.8828.49%
+ CFCol (9B)80.7224.7237.392.3227.551.48$61.3627.92%
MiMo-v2.6-Flash62.5521.7732.301.9018.681.12$8.82—
+ CFCol (4B)75.1520.9132.722.0621.541.29$5.0143.20%
+ CFCol (9B)75.1520.9132.722.0921.541.34$5.0442.86%
GLM-5.365.0522.6833.632.0920.431.45$60.43—
+ CFCol (4B)77.8921.4333.612.2023.141.56$30.0050.36%
+ CFCol (9B)77.8921.4333.612.2223.141.63$30.2349.98%

W5 & Q2: Choice of a single judge.

Thank you for this question. We use a single judge for three reasons. First, a single strong LLM judge is common practice in legal consultation benchmarks. For example, LeCoDe [1] uses GPT-4o as its only judge. Second, we collected the data and ran the experiments from January to August 2026. During that period, DeepSeek-V4-Pro was one of the most cost-efficient flagship models, which made large-scale evaluation affordable. Third, DeepSeek-V4-Pro differs from the backbone models of the consultation agents. This avoids self-preference bias, where a judge favors outputs from its own model family.

To address this concern further, we follow DLawBench [2] and add a multi-judge evaluation in this rebuttal. It uses three advanced models: Kimi K3, Gemini 3.8 Flash, and the original DeepSeek-V4-Pro. With the same evaluation prompt, the three judges score each legal advice response independently, and the median of the three scores is the final score. A concise version is shown below. Single: DeepSeek-V4-Pro only (from Table 3). Multi: median of three judges (from Table 13). All values are averaged over ID and OOD. The complete results are in Table 13 of the revised version.

The results show that the multi-judge scores stay close to the single-judge scores. They are slightly higher for most models, while the GPT-5.4 advice score is slightly lower. The conclusions do not change. CFCol consistently improves over every cloud-only baseline, and the baselines keep the same ranking. Together with the strong agreement with expert rankings, we argue that a single DeepSeek-V4-Pro judge is reliable.

MethodAdvice Mean ↑ (Single / Multi)r_a ↑ (Single / Multi)
GPT-5.42.10 / 2.091.35 / 1.38
+ CFCol (4B)2.28 / 2.261.49 / 1.54
+ CFCol (9B)2.32 / 2.301.55 / 1.47
GPT-5.4-mini1.78 / 1.891.10 / 1.28
+ CFCol (4B)2.00 / 2.061.21 / 1.33
+ CFCol (9B)1.98 / 2.061.20 / 1.33
Qwen3.6-Plus1.56 / 1.720.88 / 1.04
+ CFCol (4B)1.81 / 1.951.00 / 1.19
+ CFCol (9B)1.83 / 1.911.02 / 1.12
AuthorsOpenReview ↗

Response to Reviewer rEAZ (Q1 and Reference)

Q1: The two lawyers' practice areas.

Both lawyers have over 15 years of legal practice experience. The first specializes in divorce-related property disputes, private lending disputes, commercial disputes, labor arbitration, workers' compensation claims, criminal defense, applications for release on bail pending trial, and traffic accident cases.

The second specializes in commercial disputes, marriage and family law, labor arbitration, real estate disputes, foreign-related disputes, and criminal defense.

Thank you again for your constructive feedback. We will incorporate your suggestions into the final version. We hope these responses address your concerns and welcome further discussion.

[1] Yuan, Weikang, et al. LeCoDe: A Benchmark Dataset for Interactive Legal Consultation Dialogue Evaluation. ACL 2026.

[2] Zhang, Li, et al. DLawBench: Evaluating LLMs Through Multi-Turn Legal Consultation. EMNLP 2026.

[3] McHugh, M. L. Interrater reliability: the kappa statistic. Biochemia Medica, 2012.

Reviewer_BidoOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes LawBranch, a branch-aware multi-turn legal consultation benchmark designed to evaluate the reliability of LLMs across different factual branches of the same meta-case. The paper also introduces CFCol, a collaborative framework that combines cloud-based LLM reasoning with on-device models to improve efficiency while preserving user privacy. The proposed benchmark addresses an important limitation of existing legal consultation benchmarks, and the framework presents a promising approach to balancing reasoning capability, efficiency, and privacy. However, several aspects require further investigation, including the reliability of LLM-as-a-judge evaluation, the contribution of individual components in CFCol, and the justification for adopting SFT instead of alternative training methods such as reinforcement learning.

Critical strengths
  1. The proposed LawBranch benchmark focuses on branch-aware multi-turn legal consultation, providing a reasonable evaluation setting to assess whether a model can maintain reliable performance across different factual branches of the same meta-case.
  2. The proposed CFCol method designs a good workflow of utilised cloud based LLM and on-device model for having the efficiency and privacy simultaneously.
Critical weaknesses
  1. The evaluation of the final legal advice is entirely relying on LLM-as-a-judge. Although the score rubrics are designed by legal expert, the results remain a concern by the uncertainty of LLM generation. Providing a small proportion of human/legal expert evaluation for the dataset is recommended to prove the reliability of LLM-as-a-judge evaluation.
  2. The contribution of different parts of the proposed CFCol should be futher investigated. CFCol combines serval different techniques, such as cloud-based reasoning, on-device coordination, structural memory and so on. An ablation study of different parts of CFCol is needed to prove the necessary of utilising all the components of the proposed approach.
  3. Comparison of model training methods of CFCol are needed. The on-device agent is based on LLM models like qwen model, where RL post-training method like GRPO is proved to have good performance on. The reason of why using SFT for model training instead of well-known RL training methods is also needed to be clarified.
Questions for authors

Refers to the weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Reviewer Bido (W1-W2)

Dear Reviewer Bido,

Thank you for your constructive feedback. We address your concerns in the point-by-point responses below.

Weakness 1. Thank you for this question. Following existing work on multi-turn legal consultation [1, 2], we use LLM-based matching and judging for the clarification and advice metrics. Notably, LLM-as-a-judge has been shown to align more closely with human experts than traditional automatic metrics used in previous legal benchmarks, such as BERTScore and ROUGE-L [4]. LLM-as-a-judge is also used in nearly all recent legal consultation work (single- and multi-turn) and in other legal generation tasks on widely used legal benchmarks [1, 2, 5, 6].

In Appendix A.5 (Data Quality Validation, lines 1019–1025) of the original submission, we analyzed the consistency between the LLM judge and human experts for model ranking. We sampled 200 branches from the idealized single-run experiments, each with four advice responses from different models and configurations. As shown in Fig. 10 of the original submission, pairwise accuracy (for ordering) and NDCG (for top-ranking quality) are approximately 0.8 and 0.9, respectively. These results show that our LLM-as-a-judge protocol aligns well with expert judgments, especially for high-quality responses, and thus supports its use as a practical surrogate for expert evaluation.

In this rebuttal, we provide two additional expert–model consistency experiments.

Advice quality score: human–model consistency. We asked legal experts and the LLM judge to independently score the same 400 factual branches, following the protocol described in Sec. 3.2. The two achieved 89.5% exact agreement and a linearly weighted Cohen's κ of 0.832 (95% CI [0.814, 0.866]; bias-corrected bootstrap, 10,000 resamples). The results show almost perfect agreement between the LLM judge and human experts [3].

Question matching: human–model consistency. We sampled 400 question lists from the Table 3 experiments in the original submission. These lists cover 200 factual branches drawn from 200 cases, and each branch has one list generated by GPT-5.4 and one by Qwen3.6-Plus acting as the consultation agent. For each ground-truth question–answer pair in Fk={(qk,j,ak,j)}j=1KF\mathcal{F}_k = \{(q_{k,j}, a_{k,j})\}_{j=1}^{K_F}, legal experts independently judged whether the agent had asked the corresponding question (Yes/No), and we compared their judgments with the LLM matching results. The two achieved 91.7% exact agreement and a Cohen's κ of 0.813 (95% CI [0.766, 0.858]). The point estimate falls within the "almost perfect" range [3], indicating that LLM-based matching closely tracks expert judgment.

These findings indicate that the LLM judge can serve as a reliable proxy for expert assessment of advice quality and question matching, allowing scalable evaluation without extensive manual annotation.

Weakness 2. Thank you for this question. The original submission already includes this ablation study. CFCol has three components: a cloud agent, an on-device agent, and a memory manager (MM).

If we keep only cloud-based reasoning, we get the cloud-only baselines (e.g., GPT-5.4, GPT-5.4-mini, and Qwen3.6-Plus in Table 3 of the original submission).

If we keep only on-device coordination, we get the edge-only baselines (e.g., Qwen3.5-9B and, in the revised version, the same model fine-tuned on our data, Qwen3.5-9B SFT). As shown in Tables 3 and 4 of the original submission, CFCol consistently outperforms both the cloud-only and edge-only settings, and it costs much less than the cloud-only baselines. This shows that cloud–edge collaboration improves consultation performance while also reducing cost.

We further report results with the structured memory removed (w/o MM) in Table 12 of Appendix D.3.3 of the original submission. Removing MM causes a significant drop across all metrics, and the drop grows as the cloud model becomes stronger. This highlights the importance of the memory manager for explicit state tracking in cloud–edge collaboration.

AuthorsOpenReview ↗

Response to Reviewer Bido (W3 and Reference)

Weakness 3. Thank you for this suggestion. We adopt SFT because CFCol leaves complex legal reasoning to the cloud model. The on-device agent only handles de-identification and dialogue coordination. These are structured tasks with high-quality reference outputs, so SFT learns them well at low cost, while GRPO is better suited to complex reasoning tasks. Moreover, pure GRPO was unstable on small models they often failed to produce the required format and were rejected at the first de-identification check when without SFT.

We further compare SFT with SFT+GRPO on Qwen3.5-4B and Qwen3.5-9B, using the clean setting from Table 3. All variants share the same cloud-generated question list, so their clarification metrics are identical. We therefore report only advice quality and cost below. Full results are in Table 14 of the revised version. The results show that GRPO brings only modest and inconsistent gains. Advice Mean changes by less than 0.1 in five of the six settings, and SFT matches or outperforms SFT+GRPO for GPT-5.4. GRPO mainly improves rar_a, but it requires an extra RL training stage and slightly increases cost in most settings. SFT therefore offers a better trade-off, and GRPO can be added as an optional extension.

BackboneOn-devicePost-trainingAdvice Mean ID ↑OOD ↑Avg. ↑rar_a ID ↑OOD ↑Avg. ↑Avg. Cost ↓
GPT-5.44BSFT2.272.302.281.481.521.49$88.74
GPT-5.44BSFT+GRPO2.052.142.081.391.471.41$90.52
GPT-5.49BSFT2.372.182.321.681.241.55$89.41
GPT-5.49BSFT+GRPO2.272.232.261.601.551.59$91.43
GPT-5.4-mini4BSFT2.021.942.001.211.201.21$35.83
GPT-5.4-mini4BSFT+GRPO2.022.012.021.351.311.34$37.38
GPT-5.4-mini9BSFT2.001.931.981.201.201.20$35.85
GPT-5.4-mini9BSFT+GRPO2.041.982.021.371.281.34$36.47
Qwen3.6-Plus4BSFT1.801.841.810.981.041.00$7.73
Qwen3.6-Plus4BSFT+GRPO1.891.911.901.181.231.19$8.25
Qwen3.6-Plus9BSFT1.841.801.831.031.021.02$7.77
Qwen3.6-Plus9BSFT+GRPO1.811.791.801.091.061.08$7.63

Thank you again for your constructive feedback. We will incorporate your suggestions into the final version. We hope these responses address your concerns and welcome further discussion.

[1] Yuan, Weikang, et al. LeCoDe: A Benchmark Dataset for Interactive Legal Consultation Dialogue Evaluation. ACL 2026.

[2] Zhang, Li, et al. DLawBench: Evaluating LLMs Through Multi-Turn Legal Consultation. EMNLP 2026.

[3] McHugh, M. L. Interrater reliability: the kappa statistic. Biochemia Medica, 2012.

[4] Enguehard, Joseph, et al. LeMAJ (Legal LLM-as-a-Judge): Bridging Legal Reasoning and LLM Evaluation. Proceedings of the Natural Legal Language Processing Workshop, 2025.

[5] Fan, Yu, et al. LEXam: Benchmarking Legal Reasoning on 340 Law Exams. ICLR 2026.

[6] Li, Haitao, et al. LexEval: A Comprehensive Chinese Legal Benchmark for Evaluating Large Language Models. NeurIPS 2024.

Thank you for the detailed responses and additional experiments.

The additional human–model consistency experiments provide convincing evidence supporting the reliability of the llm-as-a-judge evaluation. It is suggested that the human-model consistency results should be included in the main paper rather than the appendix, given that this concern was raised by all four reviewers and is important for establishing the benchmark's reliability.

The clarification of the existing ablation studies also helps address my concerns regarding the contributions of different CFCol components. It is also suggested that the ablation results of different CFCol components should be put into a single table, making it easier for readers to compare different components individual contributions.

I appreciate the additional comparison experiments between SFT and SFT+GRPO. While the results generally support SFT as a more cost-effective approach, SFT+GRPO also outperforms SFT in some models with only a small increase in cost, suggesting that GRPO can still be beneficial in certain settings. Moreover, the claim that pure GRPO is unstable on small models is not directly validated by the newly reported experiments, which only compare SFT with SFT+GRPO.

Reviewer_gZz7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces LawBranch, a branch-aware benchmark for evaluating multi-turn Chinese legal consultation, comprising 1,500 human-verified cases and 5,399 factual branches. The authors further propose CFCol, an edge–cloud collaboration framework that combines cloud-based legal reasoning with on-device dialogue coordination and privacy protection.

Critical strengths
  1. The paper addresses a practical and important problem in evaluating LLMs for legal consultation, particularly the limitations of average-case evaluation in capturing failures across different factual scenarios.
  2. LawBranch is a useful benchmark with a relatively large number of human-verified cases and factual branches. The involvement of legal annotators and practicing lawyers strengthens the quality of the dataset.
  3. The experimental evaluation is fairly comprehensive, covering different models, user behaviors, multi-turn interactions, and worst-branch reliability. The additional analysis provides useful insights into model failures in realistic legal consultations.
  4. CFCol is evaluated across different model configurations, with experiments on performance, inference costs, privacy protection, and memory management
Critical weaknesses
  1. The distinction between LawBranch and existing benchmarks, particularly LeCoDe and DLawBench, is not sufficiently clear. The main novelty appears to lie in factual branching and worst-branch evaluation, rather than a fundamentally new evaluation paradigm.
  2. The benchmark relies heavily on LLM-generated meta-cases and factual branches derived from publicly available legal cases. It remains unclear whether these generated branches adequately represent realistic variations in actual legal disputes. 3.The multi-turn evaluation relies on an LLM-based simulator with predefined behavioral patterns. Such simulations may not fully capture the spontaneous, inconsistent, and unpredictable responses of real legal clients.
  3. Using the minimum score across branches may introduce bias toward cases with more factual branches, as they have more opportunities to produce low scores. The paper does not sufficiently investigate the effect of branch count on this metric.
  4. Legal advice quality is primarily evaluated using an LLM judge against reference solutions. This may overlook alternative but legally valid advice. The relatively limited expert evaluation does not fully establish the reliability of this evaluation protocol. 6.The clarification plan is generated before the multi-turn interaction, with subsequent dialogue coordination handled by the on-device agent. It is unclear how the framework adapts when newly revealed facts contradict the initial clarification plan.
  5. The cost analysis excludes on-device computation, while CFCol often requires more interaction turns. Moreover, the privacy evaluation mainly focuses on predefined synthetic identifiers, leaving broader privacy risks insufficiently explored.
Questions for authors

see above in weakness

5 comments
AuthorsOpenReview ↗

Response to Reviewer gZz7 (W1-W3)

Dear Reviewer gZz7,

Thank you for your constructive feedback. We address your concerns in the point-by-point responses below.

W1: Novelty of our benchmark. Thank you for this question.

First, we see branch-aware evaluation as a new evaluation axis, not merely an extra metric added to existing benchmarks. Existing multi-turn legal benchmarks evaluate each case as one fixed scenario. LeCoDe [1] transcribes single real consultations, and DLawBench [2] changes the client persona while keeping each case's facts the same. Neither can measure whether a model stays reliable when the legally decisive facts change under the same background. Such variation is central to how hypotheticals are used in legal education and practice [3].

LawBranch collects multiple factual branches per case to model this counterfactual structure. This makes the worst-branch reliability metrics (rqr_q, rar_a) possible; they cannot be defined on single-scenario benchmarks. These metrics show what averages hide. Worst-branch scores are about 50% (F1) and 40% (Advice) below the branch-level mean, and most variance lies at the branch level rather than the case level (Fig. 5).

Second, LawBranch provides a larger evaluation set. It contains 1,500 cases and 5,399 branches, all verified by legal experts: double annotation by legally qualified annotators, followed by a final review by practicing lawyers. Each branch is a distinct, expert-verified evaluation scenario. For comparison, LeCoDe has 740 test dialogues (out of 3,696 in total), and DLawBench has 461 cases. In both, every case corresponds to a single fixed set of facts.

We will highlight these differences in the final version.

W2: Realism of the synthesized branches. Thank you for this question.

We first clarify a misunderstanding. The meta-cases are not purely AI-generated. In the GENERATE stage of our benchmark construction (Sec. 3.1), we use Qwen3.6-Max with web search to retrieve cases from publicly available online sources. Every meta-case is grounded in a real user case with a verifiable source URL. We will release this metadata upon acceptance. Only the branches are constructed on top of these real backgrounds.

Second, we generate branches because branch-level data almost never occurs naturally. Real multi-turn consultations are already hard to collect because of privacy and confidentiality, and each record covers only one fixed set of facts. Observing several clients who share the same background but differ in decisive facts is practically impossible. Controlled construction is therefore standard practice. Prior legal benchmarks also use simulated data (e.g., DLawBench [2] and J1-Eval [8]), and varying decisive facts under a fixed background mirrors the hypothetical method used in legal education [3].

Finally, to ensure realism, every meta-case was reviewed by three people: two law graduate students and one practicing lawyer (Sec. 3.1, App. A.2). They removed unrealistic branches and added missing ones. Concretely, reviewers deleted branches in 202/1,500 meta-cases (13.47%) and added branches in 429/1,500 (28.60%). The final branch sets therefore reflect expert judgment rather than raw LLM output, and are as realistic as is currently feasible. To our knowledge, LawBranch is the first multi-turn legal consultation benchmark with expert-verified counterfactual branches under shared real case backgrounds.

W3: Limitation of the LLM-based simulator. Thank you for this point. We agree that an LLM-based simulator cannot fully capture the behavior of real clients, and we will discuss this in the Limitations section. However, real consultation dialogues contain highly sensitive personal information and are hard to collect at scale, so we build our simulator following existing multi-turn legal consultation benchmarks [1, 2]. We adopt the four Interpersonal Circumplex (IPC)-based narrative styles from DLawBench and add a behavioral-robustness evaluation with ten user behaviors (see Table 6). These behaviors cover the concerns raised in the review:

  • Spontaneous: volunteering unasked facts (Prep.), asking why information is needed (Clar.), pressing for advice early (Comp.), and disclosing private information without being asked (Priv.).
  • Inconsistent: correcting earlier statements (Corr.), giving uncertain or incomplete answers (PAmb.), and drawing personal conclusions with no legal basis (Pers.).
  • Unpredictable: refusing to answer (Refu.), and mixing two or three behaviors within one dialogue (Mix.).

Together, these cover a much broader range of client behaviors than prior simulators.

More importantly, the simulator is part of our evaluation setup, not a core contribution. LawBranch and CFCol do not rely on any specific user-behavior pattern, and the simulator is a modular component that can be replaced by a more advanced simulator or by real users. Our findings also hold under this limitation.

AuthorsOpenReview ↗

Response to Reviewer gZz7 (W4-W6)

W4: Effect of branch count on worst-branch metrics. We thank the reviewer for raising this point. To test it, we split cases by branch count. Every meta-case in our benchmark has either three ( 425 cases) or four branches ( 635 cases) , so we compare these two groups. Representative results are shown below.

Modelrqr_q (3 br.)rqr_q (4 br.)rar_a (3 br.)rar_a (4 br.)
Qwen3.5-4B9.939.570.600.57
Qwen3.5-9B11.8412.370.700.73
Legal-One-8B16.3217.440.480.51
LeCoDe-E2E (9B)7.397.760.410.45
D3LM-Full (9B)2.982.570.100.08
ReAct9.078.791.031.06
GPT-5.423.4522.751.301.39
GPT-5.4 + CFCol 4B28.2028.821.541.45
GPT-5.4 + CFCol 9B28.2028.821.541.55
GPT-5.4-mini23.0723.561.121.09
GPT-5.4-mini + CFCol 4B25.7725.201.191.22
GPT-5.4-mini + CFCol 9B25.7725.201.221.19
Qwen3.6-Plus9.689.370.860.89
Qwen3.6-Plus + CFCol 4B18.7319.181.020.98
Qwen3.6-Plus + CFCol 9B18.7319.181.001.04

We observe no systematic downward bias for 4-branch cases. They score higher than 3-branch cases about as often as they score lower, and the gaps are small: at most 1.12 for rqr_q and 0.09 for rar_a. The worst-branch score therefore depends mainly on how difficult the meta-case is, not on how many branches it has, because each branch is a legally distinct scenario with its own decisive facts.

W5: Reliability of the LLM-judge protocol for advice quality.

Thank you for this question. We address it in three parts.

First, LLM-as-a-judge is the standard protocol for this task. It is used in existing multi-turn legal consultation work [1, 2] and in nearly all recent legal consultation and legal generation benchmarks [1, 2, 5, 6]. It also aligns better with human experts than traditional metrics such as BERTScore and ROUGE-L [4].

Second, our protocol scores legal correctness, not textual match. The expert-annotated reference serves only as an anchor for key steps and risks. The judge checks whether the candidate is "materially consistent" with it on four legal elements (relationship, applicable law, reasoning, and procedure). The rubric explicitly accepts "reasonable equivalent wording or minor simplification", and "reasonable concurrence of legal relationships with similar remedy routes is not by itself a score of 0". A score of 0 is reserved for fatal errors that "can mislead the user or harm substantive rights" (the full prompt is in lines 756–783 of the Appendix of the original submission).

Third, a new human–model consistency experiment. Beyond App. A.5, legal experts scored the same 400 branches as the LLM judge. They worked independently, used the same 0–3 protocol, and could not see the judge's scores. The two reached 89.5% exact agreement and a linearly weighted Cohen's κ of 0.832 (95% CI [0.814, 0.866]; bias-corrected bootstrap, 10,000 resamples). This indicates almost perfect agreement [7] and shows that the judge applies the protocol as reliably as the experts do.

W6: Adaptation when newly revealed facts contradict the clarification plan. We thank the reviewer for the question. Not re-planning is a deliberate design choice. Sec. 4 shows that multi-turn interaction degrades the clarification and advice quality of cloud models (Fig. 3). In CFCol, the cloud agent therefore generates the clarification questions once, and the on-device agent then uses them adaptively with the user, while the memory manager tracks confirmed facts, unconfirmed facts, and corrections (App. C.1). This design outperforms both the cloud-only baselines and existing baselines in the clean setting (Table 3) and under diverse user behaviors (Table 4).

Additionally, a consultation typically concerns a single dispute described upfront. If users wish to discuss a completely different matter, they can start a new session, which triggers a new clarification plan. We already acknowledge this limitation in App. E (second point) and leave adaptive multi-stage cloud reasoning to future work.

AuthorsOpenReview ↗

Response to Reviewer gZz7 (W7 and References)

W7: Privacy and end-to-end cost. For brevity, please see our response to W4 of Reviewer 1jif for the details.

Thank you again for your constructive feedback. We will incorporate your suggestions into the final version. We hope these responses address your concerns and welcome further discussion.

[1] Yuan, Weikang, et al. LeCoDe: A Benchmark Dataset for Interactive Legal Consultation Dialogue Evaluation. ACL 2026.

[2] Zhang, Li, et al. DLawBench: Evaluating LLMs Through Multi-Turn Legal Consultation. EMNLP 2026.

[3] Ashley, K. D. Hypothesis Formation and Testing in Legal Argument. 2006.

[4] Enguehard, Joseph, et al. LeMAJ (Legal LLM-as-a-Judge): Bridging Legal Reasoning and LLM Evaluation. Proceedings of the Natural Legal Language Processing Workshop, 2025.

[5] Fan, Yu, et al. LEXam: Benchmarking Legal Reasoning on 340 Law Exams. ICLR 2026.

[6] Li, Haitao, et al. LexEval: A Comprehensive Chinese Legal Benchmark for Evaluating Large Language Models. NeurIPS 2024.

[7] McHugh, M. L. Interrater reliability: the kappa statistic. Biochemia Medica, 2012.

[8] Jia, Zheng, et al. Ready Jurist One: Benchmarking Language Agents for Legal Intelligence in Dynamic Environments. ACL 2026.

Reviewer_1jifOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LawBranch, a branch-aware Chinese civil-law consultation benchmark with 1,500 cases and 5,399 factual branches. It evaluates clarification and legal-advice quality across branches and proposes CFCol, an edge–cloud framework with on-device de-identification and memory management.

Critical strengths
  1. Branch-level and worst-branch evaluation is more realistic than a single average score.
  2. The benchmark includes legal review, out-of-domain testing, behavior analysis, and ablations.
  3. CFCol addresses legal reasoning, privacy protection, and inference cost in one framework.
Critical weaknesses
  1. CFCol is trained on GPT-5.4-generated trajectories, but there is no strict SFT-only control. The reported gains may therefore come from the training data rather than the proposed framework.
  2. Clarification and advice metrics rely on LLM matching and judging, and the judge can see the reference answer. This may introduce evaluation bias.
  3. The benchmark and user behaviors are largely LLM-generated, so the simulator may not reflect real legal consultations.
  4. The privacy and cost claims are idealized: privacy tests use synthetic identifiers, while local inference, training, energy, and maintenance costs are not fully counted.
Questions for authors

Questions:

  1. How much improvement remains compared with an on-device model trained with the same SFT data but without CFCol?
  2. What is the agreement between the LLM evaluator and qualified legal experts?
  3. Can the benchmark be evaluated with real consultation dialogues or broader legal domains?
  4. Do the privacy and cost results hold with real personal information and full end-to-end accounting?

Suggestions:

  1. Add matched SFT-only, prompt-matched, and call-budget-matched baselines.
  2. Add blinded legal-expert evaluation and report agreement and uncertainty.
  3. Expand validation beyond the current Chinese civil-law domains and synthetic user behaviors.
  4. Report complete end-to-end privacy, latency, energy, training, and maintenance costs.
5 comments
AuthorsOpenReview ↗

Response to Reviewer 1jif (W1-W2)

Dear Reviewer 1jif,

Thank you for your constructive feedback. We address your concerns in the point-by-point responses below.

W1 and Q1: SFT ablation. Thank you for this question.

We have compared our proposed CFCol with LeCoDE-E2E [1], fine-tuned on Qwen3.5-9B, in Table 3 of the original submission. E2E is a stronger SFT strategy than the original SFT method in LeCoDE [1].

In this rebuttal, following your suggestion, we additionally report results for Qwen3.5-4B and Qwen3.5-9B trained on the same SFT data. For brevity, the table below reports the ID and OOD averages for each metric. Full results will appear in Table 3 of the revised version. Recall, Precision, F1, and rqr_q are identical for the 4B and 9B variants because both use the same cloud-generated question list. The substantial capability gap between the fine-tuned smaller local models and CFCol indicates that the gains mainly stem from the proposed CFCol framework rather than from the training data.

MethodRecall Avg. (%)Precision Avg. (%)F1 Avg. (%)Advice Avg.r_q Avg. (%)r_a Avg.
Qwen3.5-4B SFT47.1214.3521.711.0810.470.64
Qwen3.5-9B SFT50.8818.3326.141.2412.900.77
GPT-5.4 + CFCol (4B)84.4525.7738.902.2828.571.49
GPT-5.4 + CFCol (9B)84.4525.7738.902.3228.571.55

W2 and Q2: LLM matching and judging. Thank you for this question. Following existing multi-turn legal consultation work [1, 2], we use LLM-based matching and judging for the clarification and advice metrics. Additionally, LLM-as-a-judge is used in nearly all recent legal consultation work (single- and multi-turn) and in other legal generation tasks on widely used legal benchmarks [1, 2, 4–6].

Regarding consistency between the LLM evaluator and qualified legal experts, in Appendix A.5 (Data Quality Validation, lines 1019–1025) of the original submission, we analyzed the consistency between the LLM judge and human experts for model ranking. We sampled 200 branches from the idealized single-run experiments, each with four advice responses from different models and configurations. As shown in Fig. 10 of the original submission, pairwise accuracy for ordering and NDCG for top-ranking quality are approximately 0.8 and 0.9, respectively. These results show that our LLM-as-a-judge protocol aligns well with expert judgments, especially for high-quality responses, and thus supports its use as a practical surrogate for expert evaluation.

In this rebuttal, we provide two additional expert–model consistency experiments.

Advice quality score: human–model consistency. We asked legal experts and the LLM judge to independently score the same 400 factual branches, following the protocol described in Sec. 3.2. The two achieved 89.5% exact agreement and a linearly weighted Cohen's κ of 0.832 (95% CI [0.814, 0.866]; bias-corrected bootstrap, 10,000 resamples). The results show almost perfect agreement between the LLM judge and human experts [3].

Question matching: human–model consistency. We sampled 400 question lists from the Table 3 experiments in the original submission. These lists cover 200 factual branches drawn from 200 cases, and each branch has one list generated by GPT-5.4 and one by Qwen3.6-Plus acting as the consultation agent. For each ground-truth question–answer pair in Fk={(qk,j,ak,j)}j=1KF\mathcal{F}_k = \{(q_{k,j}, a_{k,j})\}_{j=1}^{K_F}, legal experts independently judged whether the agent had asked the corresponding question (Yes/No), and we compared their judgments with the LLM matching results. The two achieved 91.7% exact agreement and a Cohen's κ of 0.813 (95% CI [0.766, 0.858]). The point estimate falls within the "almost perfect" range [3], indicating that LLM-based matching closely tracks expert judgment.

These findings indicate that the LLM judge can serve as a reliable proxy for expert assessment of advice quality and question matching, allowing scalable evaluation without extensive manual annotation.

AuthorsOpenReview ↗

Response to Reviewer 1jif (W3-W4)

W3 and Q3: Benchmark and user behavior.

1. More validation on other domains. Thank you for this helpful suggestion. As discussed in lines 469–475 of the original submission, CFCol generalizes to unseen legal consultation benchmarks without task-specific training. We evaluated it on J1-Eval and DLawBench, which cover a range of legal domains and additional legal tasks (Appendix D.3.1). CFCol outperforms the cloud-only baselines on the primary metrics of both benchmarks while reducing cloud cost. For brevity, we refer you to Appendix D.3.1 of the original submission for the full results.

2. Benchmark concern. We apologize for the confusion. First, the meta-cases are not purely AI-generated. In the GENERATE stage of our benchmark construction (Sec. 3.1), we use Qwen3.6-Max with web search to retrieve cases from publicly available online sources. Every meta-case is grounded in a real user case with a verifiable source URL. We will release this metadata upon acceptance. In addition, all meta-cases were verified by two law graduate students and one practicing lawyer to ensure that they reflect real legal consultations (see Appendix A.2 for details).

Second, we adopt a generate–verify–revise pipeline because real-world multi-turn legal consultation data are scarce [1, 2]. Constructing branch-aware data is even more challenging: it requires variants of the same case that differ only in key legal facts or user objectives, enabling counterfactual evaluation. Even for practicing lawyers with over 15 years of experience, generating such data from scratch is impractical. Their expertise is better used for verification and revision.

Third, this construction follows established practice. In legal education and legal practice, hypothetical variations of case facts are routinely used to examine how legal outcomes change across factual scenarios [7]. In addition, constructing evaluation data by synthesizing or adapting content from legal documents is a common methodology in legal NLP [2].

As the first branch-aware benchmark for multi-turn legal consultation, we argue LawBranch makes a distinct contribution to the literature.

3. User behavior simulator concern. Our user simulator builds on existing multi-turn legal consultation benchmarks [1, 2]. For example, DLawBench defines four user styles based on the agency and communion dimensions of the Interpersonal Circumplex (IPC). We adopt these four styles and add six further behavior types. Together, they cover a wider range of realistic user behaviors than prior simulators, as shown in Table 6. Most importantly, user behavior modeling is part of our evaluation setup, but it is not the core contribution of this work. Our main contributions are: (1) LawBranch, a branch-aware benchmark for multi-turn legal consultation; (2) CFCol, a privacy-preserving edge–cloud collaborative framework; and (3) the empirical findings that multi-turn dialogue degrades model performance and that average-case metrics overestimate real-world reliability. These contributions do not depend on any particular user-behavior taxonomy, so this concern does not affect our main conclusions.

W4 and Q4 (Privacy): Privacy with real personal information.

We agree that synthetic identifiers make our privacy test less realistic. However, using real personal information would raise ethical concerns and risk exposing individuals' private data. As a compromise, we extract realistic user profiles from a public privacy benchmark [8] and inject multiple privacy-sensitive attributes into each of 300 branches sampled from LawBranch. Following the privacy test protocol in Appendix D.3.2, with GPT-5.4 as the target cloud model, CFCol protects privacy-sensitive information in 97.67% of branches with the 4B edge model and 98.33% with the 9B edge model. This shows that CFCol remains reliable when the injected personal information is more realistic.

Edge model (cloud model: GPT-5.4)Safety rate
4B293/300 (97.67%)
9B295/300 (98.33%)
AuthorsOpenReview ↗

Response to Reviewer 1jif (W4 and Reference)

W4 and Q4 (Cost): End-to-end cost accounting. We thank the reviewer for raising this point. The edge models in CFCol (4B/9B) can run on a consumer PC, so, following prior work, we treat their marginal cost as negligible [9]. Since local inference and maintenance costs are difficult to measure, we use OpenRouter API prices as a consistent proxy for all costs (Gemma 3 4B: $0.04/$0.05 per 1M input/output tokens; Qwen3.5-9B: $0.069/$0.102 per 1M input/output tokens). Because OpenRouter does not offer Qwen3.5-4B, we use Gemma 3 4B in its place. Moreover, the training of CFCol is a one-time SFT on about 400 examples (about 3–4 hours on two NVIDIA L40 GPUs), so its per-query cost is negligible. As shown in Table 12, CFCol still saves 16.60%–32.69% in total cost over cloud-only baselines after the edge-side cost is included.

ConfigurationCloud-onlyCFCol (cloud only)Edge costCFCol (total)Cost saving
GPT-5.4 + 4B$120.53$88.74$1.45$90.1925.17%
GPT-5.4 + 9B$120.53$89.41$2.44$91.8523.80%
GPT-5.4-mini + 4B$51.76$35.83$4.26$40.0922.55%
GPT-5.4-mini + 9B$51.76$35.85$7.17$43.0216.89%
Qwen3.6-Plus + 4B$17.59$7.73$4.11$11.8432.69%
Qwen3.6-Plus + 9B$17.59$7.77$6.90$14.6716.60%

Thank you again for your constructive feedback. We will incorporate your suggestions into the final version. We hope these responses address your concerns and welcome further discussion.

[1] LeCoDE: A Benchmark Dataset for Interactive Legal Consultation Dialogue Evaluation. ACL 2026.

[2] DLawBench: Evaluating LLMs Through Multi-Turn Legal Consultation. EMNLP 2026.

[3] McHugh, M. L. Interrater reliability: the kappa statistic. Biochemia Medica, 2012.

[4] LexRAG: Benchmarking Retrieval-Augmented Generation in Multi-Turn Legal Consultation Conversation. SIGIR.

[5] LEXAM: Benchmarking Legal Reasoning on 340 Law Exams. ICLR 2026.

[6] LexEval: A Comprehensive Chinese Legal Benchmark for Evaluating Large Language Models. NeurIPS 2024.

[7] Ashley, K. D. Hypothesis Formation and Testing in Legal Argument. 2006.

[8] The Text Anonymization Benchmark (TAB): A Dedicated Corpus and Evaluation Framework for Text Anonymization. https://github.com/NorskRegnesentral/text-anonymization-benchmark

[9] Cost-efficient Collaboration between On-device and Cloud Language Models. ICML 2025.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary for all reviewers and the AC

Dear all chairs and reviewers,

We thank the AC and all reviewers for their time and constructive feedback. The reviews share several concerns, so we summarize them here, and the individual replies give the full details.

1. LLM judge for question matching and advice quality (Reviewers 1jif, Bido, gZz7, and rEAZ). Following existing multi-turn legal consultation work [1,2], we use an LLM judge for both metrics. Using an LLM judge is also common practice in other legal generation tasks. The original submission already reports human–model consistency. In this rebuttal, following the reviewers, we add further human–model results and a human–human agreement study.

2. Whether the user simulator reflects real clients (Reviewers 1jif and gZz7). We follow existing multi-turn legal consultation work [1,2] to use a simulator for multi-turn evaluation. Using an LLM as the user simulator is also common practice in multi-turn dialogue tasks. Our contributions are CFCol, the LawBranch benchmark, and the findings that multi-turn dialogue degrades model performance and that worst-case results are much worse than the average. Replacing the simulator with a more advanced one would not affect these conclusions.

3. Idealized privacy and cost (Reviewers 1jif and gZz7). Using real personal information in the privacy test would violate community norms. In this rebuttal, we instead evaluate CFCol with profiles from a public privacy benchmark, and it still provides strong privacy protection. For cost, we follow prior work and do not charge on-device inference, because the edge models can run on a personal PC. In this rebuttal, we also report the total cost with the on-device component included. CFCol still saves a substantial amount of cost.

4. Additional experiments. Reviewer Bido asks whether SFT or GRPO is the better training method. Our comparison shows that SFT offers a better trade-off, while GRPO brings only modest and inconsistent gains.

Reviewer Bido also asks for an experiment on the utility of each CFCol component. The original submission already includes this ablation.

Reviewer 1jif asks about evaluation on other data. The original submission already includes zero-shot validation on other datasets, J1-Eval and DLawBench, without task-specific training.

Reviewer rEAZ asks us to evaluate more models. We provide these results, and CFCol still improves performance while reducing cost.

5. Effect of branch count on the worst-branch metrics (Reviewer gZz7). Our results show no clear bias.

6. Whether the branches reflect real consultations, and whether two lawyers are enough (Reviewers gZz7, 1jif, and rEAZ). We use two rounds of review, not the lawyers alone. Law students review each case first, and a practicing lawyer reviews it second. In both rounds, reviewers check whether each branch reflects a real consultation and add, delete, or revise branches as needed. We thus argue these branches reflect real consultations. Our analysis in Table 8 shows that the first round already brings the data quality to a high level. We argue that two lawyers are enough.

7. When newly revealed facts contradict the initial clarification plan (Reviewer gZz7). Our experiments show that longer multi-turn interaction degrades performance, so not re-planning is a deliberate choice. CFcol already outperforms existing work to achieve SOTA performance at lower cost. We thus leave adaptive re-planning to future work.

We hope this summary and the individual replies address the shared concerns, and we are happy to clarify further.

[1] LeCoDe: A Benchmark Dataset for Interactive Legal Consultation Dialogue Evaluation. ACL 2026.

[2] DLawBench: Evaluating LLMs Through Multi-Turn Legal Consultation. EMNLP 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.