acceptodds
Under review as a conference paper at ICLR 2027

Do Senior Agents Do Their Job? Measuring Role Integrity in Hierarchical Multi-Agent Systems

Abstract

Multi-agent LLM systems increasingly divide long tasks among subagents whose reports a senior agent turns into the final answer. Such a division of labor pays off only if each agent, and above all the senior agent, carries out the duties of its role. Current evaluations, however, mostly score the final answer, and a fluent, well-cited answer cannot show whether the senior agent weighed its subagents' reports or merely aggregated them. To assess whether agents at each level of the hierarchy act in line with their assigned responsibilities, we define role integrity as the preservation of an agent's functional responsibilities throughout hierarchical deliberation. To measure it, we introduce , a behavioral benchmark grounded in the structures and public records of three institutional hierarchies: peer review, appellate adjudication, and parliamentary inquiry. We choose these hierarchies because their rules explicitly assign the senior role a duty toward subordinate material, and together they cover three basic duties: to weigh, to verify, and to relay. Rather than scoring output correctness, we measure how senior agents' outputs respond to controlled changes in subordinate material, using rule-based checks and a rubric-guided LLM judge calibrated against human annotation. Across eight LLMs and over 12,000 agentic episodes, senior agents rarely adopt a planted procedural error (2.9% of rulings) but often follow a majority of generic reviews (up to 58% of decisions) or drop witnesses' stated limitations (up to 65% of cited claims). Obligation prompts reduce these failures without removing them, and they also shift behavior in the control conditions. These findings motivate direct, obligation-specific evaluation of role integrity rather than inferring it from general capability or explicit role instructions.

Primary area:
datasets and benchmarks
Keywords:
multi-agent systems, LLM agents, benchmark, role integrity, agentic evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_EG54OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

RI-Bench tests whether senior agents weigh subordinate arguments, avoid adopting a planted factual error, and preserve limitations when citing evidence. It uses public peer-review, court, and parliamentary records with controlled edits, evaluating eight models in 12,392 episodes plus a harness study. Generic review majorities often move decisions, and cited limitations are frequently lost; planted court errors are rarely adopted. Explicit obligation prompts help unevenly. The main contribution is an evaluation of specific role-related behaviors beyond final-answer correctness.

Critical strengths
  1. The benchmark targets a useful distinction: a correct-looking final decision does not establish that the agent handled subordinate evidence properly. Fixed underlying records and paired review packages make the observed sensitivity informative. The three duties give concrete failure modes that ordinary task-success scores would miss.

  2. The analysis is unusually careful about its limits. It reports case-cluster uncertainty, prompt effects on control behavior, and Parliament results for claims cited under both prompts. Human calibration and the second-harness study expose weaknesses rather than hiding them.

Critical weaknesses
  1. Reliability of the semantic endpoints remains uncertain (§4.4, Appendices C–D). Human Parliament loss rates range from 0.14 to 0.85; in Court, the human majority confirms only 5 of 15 machine-labeled adoptions in the stratified sample. Agreement with a human majority does not resolve this underlying ambiguity. A clearer adjudicated rubric and sensitivity analysis are needed to establish which model comparisons and prompt effects survive plausible readings of the labels.

  2. The endpoints only partly establish fulfillment of the stated duties. The constructed OpenReview treatment changes scores, stance, and specificity together; human review validation measures specificity without the paper, not substantive correctness. Court non-adoption includes silence, and Parliament scores only cited claims. Section 6 acknowledges these points, but the labels Weigh, Verify, and Relay in Table 3 still invite a stronger interpretation. Keep RI explicitly as avoidance of these three measured failures, and strengthen the evidence that specific minority reviews deserve greater weight.

Questions for authors
  1. Using the existing human sample, can you report model/prompt comparisons under an adjudicated rubric and under each annotator's labels, with the sampling strata accounted for where rates are estimated? Which central conclusions remain stable?

  2. For a small sample of the constructed OpenReview cases, can paper-aware human assessment confirm that the specific minority review is substantively better supported than the generic majority?

1 comment

We thank the reviewer. We have uploaded a revised PDF; text changed in response to the reviews is in blue.

1. Which conclusions survive each reading of the labels (Q1). Using the existing stratified sample, we estimated, within each sampling stratum, the probability of each reader's label given the semantic label, and reweighted every model and condition cell of the full run:

QuantityMachineAnnot. 1Annot. 2Annot. 3Majority
Court adoption0.030.04 [0.02, 0.10]0.13 [0.04, 0.23]0.00 [0.00, 0.01]0.01 [0.00, 0.02]
Court correction gain (O−V)0.160.14 [0.12, 0.16]0.01 [−0.01, 0.04]0.13 [0.11, 0.15]0.15 [0.14, 0.16]
Parliament loss0.390.46 [0.37, 0.55]0.14 [0.07, 0.21]0.83 [0.77, 0.90]0.43 [0.34, 0.52]
Parliament loss reduction (V−O)0.140.08 [0.06, 0.11]0.00 [−0.02, 0.02]0.04 [0.02, 0.06]0.08 [0.05, 0.10]

Stable under the majority and two of three readers: Court adoption stays below the 0.20 threshold, and both Obligation effects hold. Not stable: under annotator 2, who labels 76 of 100 rulings as corrected against the written guideline, both Obligation effects vanish. Parliament's absolute loss ranges from 0.14 to 0.83 across readers, so we use it only to compare models and prompts under one reading.

For the adjudicated rubric, we wrote a second guideline that replaces each label with factual questions and derives the label mechanically, and piloted it with two new annotators on a stratified sample. The two annotators agree on 6 of 10 Court rulings (κ = 0.42) and 9 of 12 Parliament witness slots (κ = 0.50). A third reader, given both answers and the source text but not the semantic labels, resolved the disagreements; against the adjudicated labels the semantic layer agrees on 8 of 10 rulings (κ = 0.73) and 12 of 12 witness slots. The two remaining Court cases are a ruling where the scorer's own quote failed verification and the pattern fallback applied, and an item whose question is ambiguous (convictions vs. death sentences); we will fix both before the full relabeling of the 200 items for the final version.

2. What the endpoints establish. We agree and have narrowed the claims. §2.1 now states that all three endpoints are failure rates and that RI summarizes the avoidance of these three measured failures, not verified fulfillment of a role. On Court, non-adoption is mostly an explicit statement of the correct fact: of 3,832 rulings, 2,988 state the record's fact, 634 name and correct counsel's error, 111 adopt it and 99 are silent. For OpenReview, the paper-only, score-free, count-balanced and ratio controls now separate score, stance and specificity.

3. Is the specific minority review better supported (Q2)? We ran this check on 4 constructed-vote cases (2 accepted, 2 rejected): two PhD-level ML annotators read the de-identified paper and checked the four main points of the specific dissenter and of one generic majority review, shown in random order without group labels.

Each of 16 points per side was rated accurate / partly accurate / inaccurate / no specific claim to check:

Annotator 1Annotator 2
Dissenter's points13 / 1 / 1 / 111 / 5 / 0 / 0
Majority reviewer's points10 / 1 / 2 / 35 / 3 / 0 / 8
Side judged better grounded overallmajority, 4/4majority, 3/4

The annotators differ mainly on general statements, which annotator 1 mostly rated accurate and annotator 2 as having no specific claim.

Reviewer_SkEFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper defines role integrity as a senior agent’s adherence to its role when processing received materials in a hierarchical multi-agent system. It also introduces RI-Bench to evaluate role integrity. Using OpenReview, US Court, and UK Parliament materials, the paper evaluates the assessment of review arguments, verification, and preservation of evidence limitations through Weigh, Verify, and Relay. The paper evaluates eight models by modifying the materials or introducing errors and limitations. The experiments show low error adoption, majority following when the majority consists of generic reviews, and limitation loss.

Critical strengths
  • The motivation is interesting. The paper examines whether senior agents evaluate and use received materials according to their roles, beyond whether their final decisions are correct.

  • The paper builds evaluation tasks around three roles in real settings across multiple domains and provides results for eight different models alongside detailed analyses of individual cases.

Critical weaknesses
  • I think it is difficult to determine, based on Weigh alone, whether the senior agent considered the review arguments or ignored them. In the adversarial arm of the constructed-vote experiment, the majority opinion is opposite to the historical decision, making Weigh equal to the rate of agreement with that decision. An agent that reaches the same decision based only on the paper would receive the same score. Therefore, I think a comparison is needed to determine whether the reviews are actually used.

  • Regarding generated reviews, the human evaluation in Appendix D evaluates specificity and whether the review content supports the assigned score, without providing annotators with the original paper. Incorrect criticisms in generated reviews could affect the senior agent’s decision. Therefore, I would like to know whether the accuracy of the review content and the validity of the criticisms were also checked.

  • The OpenReview experiments use a fixed 3:1 split in reviewer opinions. To evaluate role integrity across other configurations, I think the authors need to check whether decisions remain consistent when the number of reviewers, the opinion ratio, or the presentation order changes while the review arguments are held fixed.

  • For reproducibility, disclose the values of hyperparameters such as temperature or top-k.

The following comment did not affect my score.

  • It would make the paper easier to read to have a separate paragraph explaining what each metric measures and what high or low values mean.
Questions for authors

Please refer to the weaknesses above. If the authors respond to these concerns, I will raise my score.

3 comments

We thank the reviewer. We have uploaded a revised PDF, and text changed in response to the reviews is in blue.

1. Does Weigh show that the reviews were used? We agree that the adversarial-arm rate alone cannot separate an agent that follows the reviews from one that decides from the paper alone. We added a paper-only control: the same agents see the same 118 papers without reviews or rebuttal. An agent that ignores the reviews would follow the wrong majority at the rate at which it disagrees with the historical decision from the paper alone.

VanillaObligation
Adversarial 3:1, follows majority0.48 [0.43, 0.54]0.28 [0.23, 0.33]
Paper only, reference0.310.34
Excess over reference, paired+0.17 [0.13, 0.21]−0.06 [−0.09, −0.03]
All scores removed, change in following−0.03 [−0.06, 0.00]−0.01 [−0.03, 0.01]

Under Vanilla the reviews pull decisions toward the majority beyond the model's own paper-only judgment, from 0.00 (Claude Opus 5) to +0.47 (GLM-5.3); under Obligation the excess disappears. Removing every numeric score barely changes following, so the pull comes from the review texts.

2. Accuracy of the generated reviews. We agree this was not checked before. We asked two PhD-level ML annotators read the de-identified paper for 4 constructed-vote cases (2 accepted, 2 rejected) and checked the four main points of the specific dissenter and of one generic majority review, shown in random order.

Each of 16 points per side was rated accurate / partly accurate / inaccurate / no specific claim to check:

Annotator 1Annotator 2
Dissenter's points13 / 1 / 1 / 111 / 5 / 0 / 0
Majority reviewer's points10 / 1 / 2 / 35 / 3 / 0 / 8
Side judged better grounded overallmajority, 4/4majority, 3/4

The annotators differ mainly on general statements, which annotator 1 mostly rated accurate and annotator 2 as having no specific claim.

3. Other reviewer counts, ratios and orders. We added a count-balanced 2:2 design (two specific against two generic reviews, all 118 papers and eight models) and ratio and order variants (30 papers, four models):

DesignVanillaObligation
2:2, specific side argues for the ICLR decision: follows specific (paper-only ref.)0.86 (0.69)0.83 (0.66)
2:2, specific side argues against: follows specific (paper-only ref.)0.51 (0.31)0.48 (0.34)
2:1 / 3:1 / 4:1, follows majority0.28 / 0.53 / 0.620.24 / 0.27 / 0.37
3:1, dissenter shown first0.450.27

With equal counts, agents follow the specific side above the paper-only reference whichever side it argues; following grows with the size of the majority and does not depend on the dissenter's position.

4. Sampling hyperparameters. The harness sets no sampling parameter: temperature, top-p and top-k are left at each provider's default, every call requests reasoning_effort: medium, and maximum output length is 32,000–64,000 tokens depending on the model. The semantic scorer runs at temperature 0. This is now stated in Appendix B.

5. Metric definitions. §2.1 now has a paragraph defining each endpoint, what it counts, and that all three are failure rates (lower is better), with RI summarizing the avoidance of these three measured failures.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer SkEF,

As the discussion period approaches its close, we would appreciate any feedback on our response, particularly whether any of your key concerns remain unresolved. We would be happy to provide further clarification on specific points.

Thank you for your time and consideration.

Thank you for the detailed responses. My concerns have been adequately addressed, and I will raise my score

Reviewer_x276OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RI-Bench, a benchmark for evaluating role integrity in multi-agent LLM systems. The core question is whether a senior agent properly fulfills its assigned responsibilities when processing information from subordinate agents, rather than simply aggregating their outputs. RI-Bench covers three real-world institutional settings: (1) an OpenReview area chair who should evaluate review arguments rather than follow majority votes; (2) a Supreme Court judge who should verify claims against case records; and (3) a UK committee chair who should preserve important caveats when reporting witness evidence. The benchmark evaluates role integrity by systematically modifying subordinate information while keeping the original cases fixed. The authors evaluate eight models, comparing standard prompts with prompts that explicitly describe the senior agent's duties. Results show that models exhibit different failure patterns across tasks. Explicit duty prompts reduce some failures but do not eliminate them. The results also suggest that role integrity is not strongly correlated with general model capabilities.

Critical strengths

This paper has the following major strengths:

  1. Well-motivated research question and realistic evaluation scenarios. The paper addresses an important limitation of existing evaluation methods: a senior agent that simply counts votes and one that carefully weighs arguments may produce the same final decision. This motivates evaluating whether agents fulfill their assigned roles by intervening on their inputs rather than relying solely on final outputs. The three selected scenarios (peer review, judicial decision-making, and parliamentary inquiry) are also well justified. Each has clearly defined responsibilities for the senior role, involves complex real-world cases, and represents a setting where LLM agents have potential applications.

  2. Detailed experimental analysis and evaluation methodology. The paper provides substantial details about its evaluation design and validation. For example, the LLM judge is required to provide verbatim quotes that are verified against the agent's output. The authors also use three blinded human annotators to calibrate the evaluation, reporting inter-annotator agreement using Cohen's κ. In addition, the paper includes a harness ablation study and acknowledges that run-to-run variance for Sol exceeds the observed differences between harnesses.

Critical weaknesses

This paper has the following critical weaknesses:

  1. Limited coverage of “hierarchical multi-agent” workflows. The paper frames its contribution as evaluating role integrity in “hierarchical multi-agent” systems (e.g. in title and intro). However, the benchmark primarily evaluates role integrity in static, one-way delegation settings rather than interactive hierarchical multi-agent workflows. In Court and Parliament, senior agents process fixed subordinate documents, while OpenReview involves a limited two-level agent hierarchy without meaningful back-and-forth interaction. Although these settings provide useful tests of how senior agents handle subordinate information, they do not capture important aspects of hierarchical collaboration, such as clarification, feedback, or iterative revision.

  2. The evaluation design for the weigh task needs further justification. The paper considers following a generic review majority (with scores of 7/7/8) against the historical ICLR decision as a failure. However, the authors acknowledge that the historical decision is a reference rather than a gold standard. Moreover, the evaluation changes review scores, stances, and specificity simultaneously, making it difficult to determine what drives the agent's decision. In the real-vote setting, where only specificity is varied, most decisions remain unchanged (691/711 pairs). Furthermore, generic reviews are less specific but not necessarily incorrect, so following their majority does not necessarily indicate a failure to weigh arguments. The authors should better justify why majority following is treated as a failure of role integrity and clarify how the evaluation distinguishes legitimate reliance on review scores from a failure to consider the quality of arguments.

Questions for authors
  1. Given that Court and Parliament involve no subordinate agents and the program chair rarely overrides the area chair (around 2.8% of adversarial episodes), how does RI-Bench measure role integrity in hierarchical multi-agent systems rather than a single agent processing adversarial inputs?

  2. In the constructed-vote setting, the majority direction, review scores, and argument specificity change together. How can the authors distinguish failure to weigh arguments from reasonable reliance on reviewer consensus?

3 comments
AuthorsOpenReview ↗

Response to Reviewer x276

We thank the reviewer. We have uploaded a revised PDF, and text changed in response to the reviews is in blue.

1. "Hierarchical" scope (Q1). We agree that RI-Bench does not test clarification, feedback or iterative revision. The revision states the scope explicitly: we test the step at which a senior agent turns subordinate material into a decision, and the material is fixed in advance so that one property can change while everything else is held constant. On the low override rate: the program chair departs from the area chair in 11/944 Vanilla and 48/939 Obligation adversarial episodes. We now report this as a finding (§4.1): what the area chair does with the majority reaches the final decision almost unchanged, even when the program chair is told to override vote tallies.

2. Failure to weigh vs. reasonable reliance on consensus (Q2). We agree that following a majority is not by itself a failure and that score, stance and specificity changed together. We added four controls on the same papers (new Table 2 in §4.1; per model in Table 13, Appendix G). Rates of program-chair decisions:

DesignMeasureVanillaObligation
Adversarial 3:1follows majority0.48 [0.43, 0.54]0.28 [0.23, 0.33]
Paper only, no reviewsreference (disagreement with ICLR)0.310.34
excess over reference, paired+0.17 [0.13, 0.21]−0.06 [−0.09, −0.03]
All scores removedchange in following−0.03 [−0.06, 0.00]−0.01 [−0.03, 0.01]
2:2, specific side argues for ICLR decisionfollows specific side (paper-only ref.)0.86 (0.69)0.83 (0.66)
2:2, specific side argues againstfollows specific side (paper-only ref.)0.51 (0.31)0.48 (0.34)
30 papers, 4 models2:13:14:13:1, dissenter first
Vanilla0.280.530.620.45
Obligation0.240.270.370.27

Per model, the Vanilla excess over the paper-only reference ranges from 0.00 (Claude Opus 5) to +0.47 (GLM-5.3).

Thank you for the detailed responses and additional experiments. I appreciate the authors' efforts to address my concerns. The additional experiments/results are helpful. However, my concern still remain. For W1, the title, abstract and figure 1 still suggest the scope is hierarchical multi-agent systems. But my understanding is that court and parliament are single-agent evaluations. Although the authors' revisions narrow down the scope, the paper's claims/research questions are still not consistent with the experiments.

For W2, the additional results/findings help to explain the decision making behavior, but I remain concerned about the validity of majority following as a measure of role-integrity failure. For example, what evidence establishes that the specific minority provides stronger or more valid arguments? The results demonstrate sensitivity to review information but do not fully distinguish reasonable reliance on consensus from a failure to weigh arguments.

AuthorsOpenReview ↗

Response to follow-up: revised title, abstract and Figure 1

Thank you for the follow-up.

W1 (scope). We agree that the title, abstract and Figure 1 still overstated the scope. In Court and Parliament the subordinate material (counsel's arguments, witness evidence) is fixed text and a single LLM plays the senior role; only OpenReview has two LLM tiers (area chair, then program chair). What all three scenarios measure is one step: how a senior agent uses the material it receives. We have uploaded a revision that brings the title, abstract and Figure 1 in line with this:

  • Title: "Do Senior Agents Do Their Job? Measuring Role Integrity in How LLM Agents Use Subordinate Material".
  • Abstract and §1: role integrity is now defined for an agent acting on its subordinates' material; "at each level of the hierarchy" and "throughout hierarchical deliberation" are removed.
  • Figure 1: the OpenReview panel shows the area-chair and program-chair agents separately, Court and Parliament one agent each, and a legend marks the LLM under test and the subordinate material fixed in advance.

§6 states that the results do not isolate hierarchy as a cause.

W2 (majority following). We do not have evidence that the specific minority's arguments are stronger. In our paper-aware check of 4 cases, both annotators judged the majority side better grounded overall (4/4 and 3/4), although the dissenter's individual points were more often rated accurate. What the controls show is narrower:

  1. Under Vanilla, agents follow the majority 0.17 more often than they reach that decision from the paper alone ([0.13, 0.21]), so the reviews move decisions away from the model's own reading of the paper.
  2. With the review arguments held fixed, following rises with the size of the majority (2:1 / 3:1 / 4:1: 0.28 / 0.53 / 0.62), so the count itself carries weight.
  3. When the program chair is given its role's duties, which include not deciding by vote tally, the excess over the paper-only reference disappears (−0.06 [−0.09, −0.03]).

Reasonable reliance on consensus would also produce (1) and (2). The measurable point is that, without the duty stated, the number of reviewers on a side moves decisions beyond the model's own reading of the paper, and that this pooled excess disappears once the duty is stated.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.