- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper investigates the problem of Legal Judgment Prediction (LJP). It proposes LegalEvolver, a legal judgment prediction framework that combines legal agent, high-temperature exploration, query rewriting, and evolving rubrics with policy to generate legally informative queries for LJP-oriented retrieval. Experiments on CAIL2018 and CMDL show that LegalEvolver outperforms traditional LJP classifiers, direct LLM reasoning, legal/search agents, and post-training baselines.
Critical strengths
The paper is generally well written with a clear pipeline.
The experiments are extensive, with multiple types of compared methods and ablations justifying the effectiveness.
Critical weaknesses
My major concern about this submission is novelty. The core components of LegalEvolver including legal agent, high-temperature exploration, query rewriting, and evolving rubrics with policy are all existing methods or techniques. The only novelty of this paper seems to be the preference design under the legal judgement scenario, which is insufficient for a general machine learning methodology.
Repeated experiments are lacking. In Table 1, LegalEvolver (61.5) shows higher overall performance than SmartSearch (60.4) by 1.1. Standard deviation is necessary to see if improvement is significant.
Questions for authors
How are evaluation cases and their near-duplicates excluded from the precedent retrieval corpus? Since retrieved precedents preserve law articles, charges, and prison terms, overlap between the evaluation cases and the retrieval database could directly leak target labels.
3 comments
Dear Reviewer b2Fe,
Thank you once again for your valuable comments! As the discussion phase draws to a close, we would like to ask whether we have adequately addressed all your concerns. We look forward to hearing from you!
Thank you for the detailed rebuttal and additional experiments. The empirical evidence strengthened.
However, my main concerns remain:
W1: The proposed framework mainly combines existing techniques, and the methodological novelty beyond prior query-optimization and rubric-evolution methods remains limited.
W2: Additional training rounds and cross-backbone experiments do not replace repeated runs with different random seeds and standard deviations.
Q1: The BM25-based filtering is helpful, but the definition of the 60% similarity threshold and its effectiveness in removing near-duplicates remain unclear.
I maintain my original rating of 1.
W1. Novelty beyond existing agent, query-rewriting, and rubric-evolution techniques
Thank you for this comment.
Legal agents, high-temperature exploration, query rewriting, and reinforcement learning have precedents. We focus on the additional challenges that general-purpose agentic retrieval faces in realistic legal retrieval. We also focus on designing training signals that address these challenges.
In legal judgment prediction, the agent must identify facts that could change the judgment. It must then translate them into retrieval queries. Distinguishing similar charges requires evidence about offense elements. Predicting sentences also requires guidance on mitigating circumstances such as voluntary surrender and restitution. A single case therefore creates multiple evidence needs. The agent must choose among statutes, judicial interpretations, sentencing guidance, and precedents. Retrieved evidence may also change the current hypothesis. Subsequent queries must adapt accordingly.
Our query rewriting preserves facts relevant to the current legal issue. It also specifies the questions that remain unresolved. We compare successful and failed trajectories for the same case. We then turn retrieval differences affecting legal qualification and sentencing into rubrics. These rubrics are specific to retrieval tools and reasoning objectives. They provide feedback for intermediate queries. As the policy improves, we identify remaining retrieval errors in new trajectories. We then update the rubrics accordingly.
In the original CAIL2018 and CMDL evaluation, LegalEvolver achieves an Overall score of 61.5. In comparison, SmartSearch achieves 60.4, which is a difference of 1.1 percentage points. After an additional training round, LegalEvolver reaches 64.6. This score is approximately 4.4 percentage points above SmartSearch after the same additional round. This result further supports the effectiveness of continually updating rubrics to address specific legal retrieval errors and jointly optimizing them with the model policy.
W2. Repeated experiments, standard deviations, and the reliability of the gain
Thank you for this comment.
Due to time constraints, we previously completed only three training rounds. We have now conducted one additional round. We provide the full results and analysis in our response to Reviewer EmAR, Question 1. The key finding is that LegalEvolver's Overall score increases from 61.5 to 64.6, which is a gain of 3.1 percentage points. After the same additional round, it exceeds SmartSearch by 4.4 percentage points. Meanwhile, both Standard GRPO and SmartSearch show slight decreases in Overall after an additional training round. This indicates that longer training alone does not necessarily yield gains. It further supports the effectiveness of rubric–policy co-evolution for continued improvement.
We also conducted cross-backbone experiments to further assess the effectiveness of our method. We provide full results in our response to Reviewer URUY, Question 2. The key finding is that LegalEvolver improves over Agentic RAG and SmartSearch by 14.0 and 4.9 percentage points, respectively, on Qwen3-4B. It improves over them by 8.7 and 4.3 percentage points, respectively, on Qwen3.5-4B. These results show that LegalEvolver outperforms both retrieval-augmented and query-optimization baselines across different base models. They further support the effectiveness and cross-model applicability of rubric–policy co-evolution.
Q1. Excluding evaluation cases and near-duplicates from the precedent corpus
Thank you for this comment.
When constructing all precedent retrieval corpora, we screened them against the CAIL2018 and CMDL test sets. We did this by computing BM25 similarity between the factual descriptions (Fact) of test and corpus cases. Under our similarity computation scheme, we removed every corpus case whose similarity to any test case reached 60% or above. We compare factual descriptions because different cases can share the same charge, law article, or sentence. Facts are more useful for identifying overlapping cases and near-duplicate documents. We designed this filter to exclude test cases and highly similar documents so the agent cannot directly retrieve target judgment information. We will specify the similarity computation, threshold, and exclusion procedure in the revised manuscript.