- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies important problems concerning the reliability and robustness of LLMs in high-stakes decision-making: sensitivity to input prompts and susceptibility to adversarial opinion, grounded in real-world legal cases. Specifically, it constructs a benchmark using 1000 instances drawn from 947 published ECtHR cases across 15 years and embeds generalizable perturbation strategies. It further challenges init model judgments over 3 turns of adversarial opinion challenges. There are some interesting findings about state-of-the-art models, such as the impact of AI safety researcher identity cue on judgments reversals.
Critical strengths
- The problem is timely and I appreciate the motivation behind this work.
- Writing is natural and easy to follow.
- Some findings are interesting, such as the impact of AI safety researcher identity cue on judgments reversals.
Critical weaknesses
- The paper has limited methodology novelty.
- The technical complexity is limited.
- While the paper describes the benchmark as continuously refreshed in the abstract and introduction, I only see experiment results of v1.0. Could the authors demonstrate the automatic benchmark update process more clearly, and explain how they ensure consistency across releases? Do the findings still hold on newly published judgments?
Questions for authors
While the paper describes the benchmark as continuously refreshed in the abstract and introduction, I only see experiment results of v1.0. Could the authors demonstrate the automatic benchmark update process more clearly, and explain how they ensure consistency across releases? Do the findings still hold on newly published judgments?
4 comments
Official reply from authors - part 2
While the paper describes the benchmark as continuously refreshed in the abstract and introduction, I only see experiment results of v1.0. Could the authors demonstrate the automatic benchmark update process more clearly, and explain how they ensure consistency across releases? Do the findings still hold on newly published judgments?
Thanks for the comment. While the submission indeed reports only v1.0, which we also note in Appendix A.2, we agree that a live benchmark should show at least one refresh. We therefore ran one to answer the three parts of this comment directly. We collected 100 new ECtHR judgments decided after the v1.0 cutoff (21 May 2026) and froze this set before any model was evaluated on it. We then ran them through the same pipeline as a matched set of 100 cases from v1.0 (the anchor), on 5.6-Sol and Opus-4.6 and in the same time window. The results are shown in the table below.
| Model | Set | Accuracy | Balanced accuracy | No-violation recall | Summary change (%) | Reversal, no cue (static, %) | Reversal, researcher cue (static, %) |
|---|---|---|---|---|---|---|---|
| 5.6-Sol | Anchor (v1.0) | 85.0 | 76.0 | 16/30 | 10 | 91 | 48 |
| 5.6-Sol | New (post-v1.0) | 77.0 | 62.6 | 8/30 | 12 | 100 | 93 |
| Opus-4.6 | Anchor (v1.0) | 76.0 | 63.8 | 10/30 | 18 | 92 | 1 |
| Opus-4.6 | New (post-v1.0) | 74.0 | 57.6 | 5/30 | 16 | 97 | 0 |
-
Regarding the update process, the new judgments went through the same steps as v1.0, as described in Section 3.2 and Appendix A. We keep full judgments with a clear decision date and an explicit outcome, remove the verdict, standardize the text and the provision, and generate the summaries and challenges automatically. The only manual step is a review of the passages flagged by a keyword check, which were 21 passages in v1.0.
-
Regarding consistency across releases, the anchor rows reproduce the numbers in our paper. The balanced accuracy is 76.0 vs. 75.5 for 5.6-Sol and 63.8 vs. 62.9 for Opus-4.6, and the reversal rates match as well (91 vs. 92.6 and 92 vs. 92.8 without a cue, 48 vs. 51.2 and 1 vs. 1.1 with the researcher cue). The pipeline and the models therefore behave as in the original experiments, and the differences between the anchor rows and the new rows come from the cases. We will evaluate such an anchor together with every new release. Reviewer j8sk asked how stable our rates are across repeated runs, and this anchor is part of our answer there, since it repeats 100 cases and gives the same rates.
-
Regarding the findings, the main ones hold on the new judgments. Challenges without a cue still reverse almost all judgments (100% for 5.6-Sol and 97% for Opus-4.6). Summaries change a similar share of judgments (12% and 16%). The researcher cue still removes almost all reversals for Opus-4.6 (0%), and its accuracy is similar (74.0 vs. 76.0).
-
The refresh also showed two differences that v1.0 alone could not show. First, both models recognize fewer no-violation outcomes on the new cases (8/30 vs. 16/30 for 5.6-Sol and 5/30 vs. 10/30 for Opus-4.6), while their accuracy on violation cases does not drop. This lowers the balanced accuracy by 13.3 points for 5.6-Sol (bootstrap 95% interval 1.7 to 25.0) and by 6.2 points for Opus-4.6, where the interval includes zero (-5.0 to 17.4). Since both models are affected, we believe the new no-violation cases are harder, and we will study this in the next release. Second, the researcher cue helps 5.6-Sol much less than before (93% reversals vs. 48%). With 100 cases we do not want to over-interpret these differences, but we see them as a good argument for keeping the benchmark live. Reviewer FTor asked why the identity cues affect models differently, and this result adds to our answer there, since it suggests that the cue effect of 5.6-Sol is less stable than that of Opus-4.6.
We will add this experiment and a description of the release procedure to the revised version.
While the paper describes the benchmark as continuously refreshed in the abstract and introduction, I only see experiment results of v1.0. Could the authors demonstrate the automatic benchmark update process more clearly, and explain how they ensure consistency across releases? Do the findings still hold on newly published judgments?
Thanks for the question. Since it is the same as the third weakness above, we kindly refer to our response there. In short, the benchmark was refreshed with 100 new judgments, the anchor from v1.0 reproduces our paper, and the main findings hold on the new cases, with two differences that we describe above.
Thanks for the valuable comments
We thank the reviewer EmoP, again for the valuable comments, which led to an experiment that we believe makes the paper stronger. We hope our responses address the concerns, and we would really appreciate it if the reviewer could consider raising the score. We are happy to answer any further questions.
Dear Reviewer EmoP,
Thank you once again for your time and valuable feedback on our submission.
As the rebuttal period is coming to a close, we wanted to kindly follow up to see if you have had a chance to review our responses. We dedicated significant effort to thoroughly addressing your questions and running additional analyses to strengthen the paper based on your suggestions.
If you have any remaining concerns or additional comments, we would be very happy to discuss them further before the deadline. Otherwise, if our responses have satisfactorily resolved your questions, we would greatly appreciate it if you would consider updating your evaluation accordingly.
Thank you again for your time and consideration.
Best regards, The Authors
Official reply from authors - part 1
We thank the reviewer EmoP for the careful reading of our paper and for the positive comments on the motivation. The question on the live refresh was especially helpful for us, and we are delighted to address your concerns and points accordingly.
Thanks for the comment. While we agree that the paper does not propose a new algorithm, since it is a benchmark paper, we would like to point out three parts of the evaluation design that are new and that led to the findings you found interesting.
Every instance carries its own reliability tests. For each case we automatically generate a summary, three paraphrases, country swaps and the challenges, so no manual annotation is needed. As described in Section 3.2, this is what allows the same tests to be applied to newly published judgments. To demonstrate this, we refreshed the benchmark with 100 new judgments, as reported in our response to your third comment.
We measure what a reversal does to correctness. As listed in Table 1, we report the retention of correct judgments and the correction of errors as two separate rates. This changes the reading of our main finding. As shown in Table 9, with the AI safety researcher cue Opus-4.6 keeps 99.0% of its correct judgments, but it fixes only 1.6% of its errors, compared to 96.3% without a cue. A single reversal rate would present the cue as a pure improvement.
The challenge protocol is controlled along several dimensions. As described in Appendix G.1, each initial judgment is challenged under eleven conditions (two pressure levels, four role cues, three consensus cues and two affiliation cues) in a static and an adaptive mode. As shown in Table 10b, we add a placebo cue ("I am reading this case.") and a challenge that agrees with the model. This design is what isolates the effect of the researcher cue. The placebo reduces the reversals of Opus-4.6 from 95.0% to 65.5%, while the researcher cue reduces them to 0.7%.
A related point on the novelty with respect to prior work was raised by Reviewer j8sk. Our answer there is that running the perturbations and the challenges on the same instances shows that a model can be robust to rewording and still give up its judgment under plain disagreement. Reviewer FTor also asked what is behind the reversals, and our answer there is that these control experiments point to deference to the user as the main driver.
Thanks for the comment. While we agree that the challenges themselves are simple, we kept them simple on purpose, and we believe this makes the findings more serious and not less.
A challenge that only says "I disagree" and asks the model to reconsider, without any evidence, is something every user can write. No access to the model and no optimization is needed. As reported in Appendix G.2, this plain disagreement already makes 57.5% (Opus-4.6) and 62.0% (5.6-Sol) of their most robust correct answers wrong, and the directive raises this to 90.1%. If a complex attack were needed to obtain this, the risk in practice would be much smaller.
The effort of the paper is in the scale and the statistical care of the evaluation. We evaluate eight models on 1,000 instances with ten scores per input, and we started 118,206 conversations for the adversarial part. As described in Appendix B, all intervals come from case-cluster bootstrap resampling, the planned contrasts use Bonferroni-adjusted intervals, and the accuracy losses hold under five different scoring rules.
A simple protocol is also easier to reproduce and to rerun on new data. This is what made it possible to refresh the benchmark with new judgments within the rebuttal period.