acceptodds
Under review as a conference paper at ICLR 2027

PerspectiveGap: A Benchmark for Multi-Agent Orchestration Prompting

Abstract

In a multi-agent LLM system, the main agent must write each sub-agent's prompt, yet current models struggle to determine what each sub-agent needs to know. Existing benchmarks score belief tracking or downstream task success, not the prompts themselves. We introduce PerspectiveGap, a benchmark for evaluating LLMs' ability to compose orchestration prompts for multi-agent systems. PerspectiveGap contains 110 scenarios, each evaluated through two distractor-mixed task formats: role-fragment assignment and verbatim prompt assembly, in which the model assembles each role's prompt from the given fragments. These scenarios are organized into 10 loop-centered topologies distilled from the authors' real-world engineering practice. Across 35 commercial models from 10 companies, the average combined pass rate is only 17.6%, and in role-fragment assignment, models give roles an average of 2.1 out-of-role fragments per scenario. GPT-5.5 leads by a wide margin (62.0% pass rate). These results show that current models often fail to give each sub-agent exactly the context it needs, an ability that existing benchmarks do not measure.

Primary area:
datasets and benchmarks
Keywords:
benchmark, LLM evaluation, multi-agent systems, orchestration prompting, context management, information leakage

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_cN6YOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PerspectiveGap evaluates whether an LLM can assign information to the appropriate sub-agent roles. Each scenario provides roles and information fragments, including a distractor. The model either assigns fragment IDs to roles or assembles role prompts by copying the selected fragments verbatim. The benchmark contains 110 scenarios derived from 10 topology templates and evaluates 35 models at two shuffle seeds. Under a strict criterion requiring every role’s assignment to be correct, the average combined pass rate is 17.6%, with GPT-5.5 reaching 62.0%. The paper also reports human validation, scorer audits, and prompting ablations.

Critical strengths

Testing the generated prompts directly is a useful way to examine information allocation without mixing in downstream execution failures. The authors also make a reasonable effort to validate the benchmark: five independent volunteers reproduce the reference assignments, and the containment scorer agrees closely with human judgments on 716 role prompts (Section 3; Appendix K). These checks support the consistency of the task and its scoring. I also appreciate the partial-credit metrics, which give a more informative picture than strict pass rates alone.

Critical weaknesses
  1. The connection to practical orchestration quality is not established. The benchmark measures agreement with a particular “need-only” assignment policy. This makes the task well-defined, but it does not establish that every departure from that policy is harmful in an actual workflow. Section 3’s example of withholding a reviewer’s two-hour limit from the planner illustrates the issue: such a limit could be a legitimate requirement to communicate, rather than information that should remain hidden. Confidential material, ordinary acceptance criteria, and irrelevant context have different consequences, but all extra assignments can cause a strict failure. The human checks validate agreement with the reference policy, not its downstream usefulness. A small executable study showing which scored errors actually impair coordination or violate meaningful boundaries would strengthen the paper.

  2. The task coverage is narrower than the orchestration framing suggests. Roles, workflows, and candidate fragments are already supplied; the model selects and copies information rather than designing the orchestration. Appendix E further states that the 100 generated instances preserve the structure, fragment functions, and reference mappings of the 10 templates. Domain rewrites add linguistic variety, but relatively little variety in the underlying allocation problems. The assembly format also adds copying and formatting demands that a deployment could avoid by having the model select IDs and a program concatenate the fragments. I therefore remain unsure how much the combined score measures orchestration-specific ability beyond information selection and instruction following. Synthetic construction is not itself a problem, but the paper needs stronger evidence that this particular simplification captures an important practical bottleneck.

Questions for authors
  1. Evidence for practical relevance: On a representative subset, do higher benchmark scores predict better downstream outcomes? Does correcting the scored assignment errors improve those outcomes?

  2. Reference-policy clarification: How are confidential information and ordinary shared requirements distinguished when constructing the mappings? What justifies withholding a reviewer’s acceptance criterion when it could help the actor satisfy the task?

  3. Evidence for the assembly task: What changes if role-fragment assignment is followed by deterministic prompt assembly? How much of the direct assembly failure comes from different selection decisions versus copying, formatting, or detection errors?

Reviewer_w8CvOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

PerspectiveGap evaluates context allocation before multi-agent execution. Given roles and shuffled information fragments, a model either assigns fragment identifiers to roles or assembles role prompts by copying selected fragments. The benchmark contains 110 scenarios derived from ten orchestration templates. Evaluation across 35 commercial models reveals frequent misplaced instructions, omitted shared context, and distractor inclusion. The contribution is a diagnostic benchmark with deterministic scoring.

Critical strengths
  1. Evaluating role prompts before execution isolates context-allocation errors that downstream task scores can obscure. The two output formats provide complementary views of these failures.
  2. Independent human checks support both the reference assignments and the scorer. Five volunteers reproduced all mappings, and the scorer agreed with human judgments on 250 of 253 complete responses in its audit. The broad model evaluation makes the observed difficulty informative.
Critical weaknesses
  1. The practical validity of the “need-only” policy remains unclear. Human agreement establishes that the reference assignments are recoverable, but does not establish that alternative allocations harm orchestration. Although the paper acknowledges its focus on prompt artifacts, a small execution study would help distinguish harmless extra context from consequential boundary violations.
  2. The domain instances inherit their templates’ fragment functions and reference mappings (Appendix E). Domain variety therefore adds limited structural diversity. Variants that change responsibilities, shared-context requirements, or artifact ownership would better test whether models follow information requirements beyond recurring template patterns.
Questions for authors
  1. On a small executable subset, do reference allocations outperform controlled alternatives that share selected deliverable specifications or review criteria? Can you distinguish harmless extra context from additions that cause role confusion or task failure?
  2. How does performance change on manually verified variants that retain a topology but alter responsibilities or artifact ownership, thereby changing the correct fragment assignments?
  3. What fraction of failures comes from unresolved API errors or invalid output formats, versus incorrect allocations in otherwise valid responses? Appendix F counts these outcomes as failures. Would separating them materially change the leaderboard or the interpretation of the low pass rates?
Reviewer_Fr8fOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces PerspectiveGap, a benchmark that tests whether LLMs, when writing prompts for sub-agents in a multi-agent system, can give each role only the information it needs. It contains 110 scenarios across 10 topologies, each with one distractor fragment, and two tasks: assigning fragment IDs to roles and assembling each role’s prompt verbatim. Across 35 commercial models, the average pass rate is 17.6%, with GPT-5.5 highest at 62%. The authors identify four failure modes: leaking distractors, leaking out-of-role information, confusing artifact ownership, and dropping shared context.

Critical strengths
  1. The problem is new and practical. Existing benchmarks do not directly measure this ability.

  2. The design is clean. The answers are auditable: five external volunteers independently reproduced all reference answers, and the scorer agrees with human labels 99.4% of the time.

Critical weaknesses
  1. The paper assumes that giving each sub-agent only the information it needs leads to better system performance, but this is never tested. The claim that leaking private tests causes reward hacking is also just an example, not an experiment. It would help to actually run the sub-agents on one or two topologies.

  2. The 100 instances are LLM rewrites of 10 templates, and the reference mappings are inherited directly, so there are effectively only 10 distinct structures. Also, half of the instances were generated by GPT-5.5 (Appendix E), which happens to rank first.

  3. Strict pass treats giving a role one harmless piece of extra background the same as giving the coder the private tests. The distractor is also always the same type: prompt-engineering advice.

  4. Figure 2 is too small. crp and dcrp are basically unreadable. Consider enlarging it or splitting it into separate panels.

Questions for authors
  1. Why do quite a few models score higher on the harder prompt assembly task than on role-fragment assignment? Is it reasonable to compute Combined as a simple average of the two?

  2. why does GPT-6 Astra score only 8.9% while GPT-5.5 gets 62%? Is it format errors, paraphrasing penalized by the scorer, default settings, or the fact that half the instances were generated by GPT-5.5?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.