- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
RI-Bench tests whether senior agents weigh subordinate arguments, avoid adopting a planted factual error, and preserve limitations when citing evidence. It uses public peer-review, court, and parliamentary records with controlled edits, evaluating eight models in 12,392 episodes plus a harness study. Generic review majorities often move decisions, and cited limitations are frequently lost; planted court errors are rarely adopted. Explicit obligation prompts help unevenly. The main contribution is an evaluation of specific role-related behaviors beyond final-answer correctness.
Critical strengths
-
The benchmark targets a useful distinction: a correct-looking final decision does not establish that the agent handled subordinate evidence properly. Fixed underlying records and paired review packages make the observed sensitivity informative. The three duties give concrete failure modes that ordinary task-success scores would miss.
-
The analysis is unusually careful about its limits. It reports case-cluster uncertainty, prompt effects on control behavior, and Parliament results for claims cited under both prompts. Human calibration and the second-harness study expose weaknesses rather than hiding them.
Critical weaknesses
-
Reliability of the semantic endpoints remains uncertain (§4.4, Appendices C–D). Human Parliament loss rates range from 0.14 to 0.85; in Court, the human majority confirms only 5 of 15 machine-labeled adoptions in the stratified sample. Agreement with a human majority does not resolve this underlying ambiguity. A clearer adjudicated rubric and sensitivity analysis are needed to establish which model comparisons and prompt effects survive plausible readings of the labels.
-
The endpoints only partly establish fulfillment of the stated duties. The constructed OpenReview treatment changes scores, stance, and specificity together; human review validation measures specificity without the paper, not substantive correctness. Court non-adoption includes silence, and Parliament scores only cited claims. Section 6 acknowledges these points, but the labels Weigh, Verify, and Relay in Table 3 still invite a stronger interpretation. Keep RI explicitly as avoidance of these three measured failures, and strengthen the evidence that specific minority reviews deserve greater weight.
Questions for authors
-
Using the existing human sample, can you report model/prompt comparisons under an adjudicated rubric and under each annotator's labels, with the sampling strata accounted for where rates are estimated? Which central conclusions remain stable?
-
For a small sample of the constructed OpenReview cases, can paper-aware human assessment confirm that the specific minority review is substantively better supported than the generic majority?
We thank the reviewer. We have uploaded a revised PDF; text changed in response to the reviews is in blue.
1. Which conclusions survive each reading of the labels (Q1). Using the existing stratified sample, we estimated, within each sampling stratum, the probability of each reader's label given the semantic label, and reweighted every model and condition cell of the full run:
Stable under the majority and two of three readers: Court adoption stays below the 0.20 threshold, and both Obligation effects hold. Not stable: under annotator 2, who labels 76 of 100 rulings as corrected against the written guideline, both Obligation effects vanish. Parliament's absolute loss ranges from 0.14 to 0.83 across readers, so we use it only to compare models and prompts under one reading.
For the adjudicated rubric, we wrote a second guideline that replaces each label with factual questions and derives the label mechanically, and piloted it with two new annotators on a stratified sample. The two annotators agree on 6 of 10 Court rulings (κ = 0.42) and 9 of 12 Parliament witness slots (κ = 0.50). A third reader, given both answers and the source text but not the semantic labels, resolved the disagreements; against the adjudicated labels the semantic layer agrees on 8 of 10 rulings (κ = 0.73) and 12 of 12 witness slots. The two remaining Court cases are a ruling where the scorer's own quote failed verification and the pattern fallback applied, and an item whose question is ambiguous (convictions vs. death sentences); we will fix both before the full relabeling of the 200 items for the final version.
2. What the endpoints establish. We agree and have narrowed the claims. §2.1 now states that all three endpoints are failure rates and that RI summarizes the avoidance of these three measured failures, not verified fulfillment of a role. On Court, non-adoption is mostly an explicit statement of the correct fact: of 3,832 rulings, 2,988 state the record's fact, 634 name and correct counsel's error, 111 adopt it and 99 are silent. For OpenReview, the paper-only, score-free, count-balanced and ratio controls now separate score, stance and specificity.
3. Is the specific minority review better supported (Q2)? We ran this check on 4 constructed-vote cases (2 accepted, 2 rejected): two PhD-level ML annotators read the de-identified paper and checked the four main points of the specific dissenter and of one generic majority review, shown in random order without group labels.
Each of 16 points per side was rated accurate / partly accurate / inaccurate / no specific claim to check:
The annotators differ mainly on general statements, which annotator 1 mostly rated accurate and annotator 2 as having no specific claim.