acceptodds
Under review as a conference paper at ICLR 2027

LawBranch: Branch-Aware Evaluation of Multi-Turn Legal Consultation

Abstract

Large language models (LLMs) hold promise for making legal consultation more accessible. In realistic legal consultation, however, user queries are often initially underspecified and incomplete, so legal experts must ask clarification questions over multiple turns to elicit missing legal facts. Since different answers can lead to substantially different legal reasoning and advice, existing benchmarks, which typically evaluate a single fixed scenario per case and report average-case metrics, obscure failures on consequential factual branches. Meanwhile, cloud-based LLMs achieve strong performance but incur high costs and privacy risks, whereas smaller local models are more affordable yet typically lack sufficient legal-reasoning capabilities. To address these gaps, we introduce LawBranch, a branch-aware Chinese civil-law benchmark comprising 1,500 human-verified cases and 5,399 factual branches. Further analysis reveals that both clarification and advice quality degrade consistently in multi-turn dialogue, while worst-branch reliability is at least 40% lower than the corresponding average-branch metric. We therefore propose Clarification-First Edge-Cloud Collaboration (CFCol), where a cloud agent handles clarification planning and legal advice generation, while an on-device agent and a memory manager are responsible for de-identification and dialogue coordination. Across diverse cloud and edge LLM instantiations, CFCol improves both clarification and advice performance, reduces cloud cost by at least 25%, and offers privacy advantages over cloud-only deployment.

Then back it, or bet against it.

Related papers

Open the market on this paper to see 7 more related papers.