- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
PerspectiveGap evaluates whether an LLM can assign information to the appropriate sub-agent roles. Each scenario provides roles and information fragments, including a distractor. The model either assigns fragment IDs to roles or assembles role prompts by copying the selected fragments verbatim. The benchmark contains 110 scenarios derived from 10 topology templates and evaluates 35 models at two shuffle seeds. Under a strict criterion requiring every role’s assignment to be correct, the average combined pass rate is 17.6%, with GPT-5.5 reaching 62.0%. The paper also reports human validation, scorer audits, and prompting ablations.
Critical strengths
Testing the generated prompts directly is a useful way to examine information allocation without mixing in downstream execution failures. The authors also make a reasonable effort to validate the benchmark: five independent volunteers reproduce the reference assignments, and the containment scorer agrees closely with human judgments on 716 role prompts (Section 3; Appendix K). These checks support the consistency of the task and its scoring. I also appreciate the partial-credit metrics, which give a more informative picture than strict pass rates alone.
Critical weaknesses
-
The connection to practical orchestration quality is not established. The benchmark measures agreement with a particular “need-only” assignment policy. This makes the task well-defined, but it does not establish that every departure from that policy is harmful in an actual workflow. Section 3’s example of withholding a reviewer’s two-hour limit from the planner illustrates the issue: such a limit could be a legitimate requirement to communicate, rather than information that should remain hidden. Confidential material, ordinary acceptance criteria, and irrelevant context have different consequences, but all extra assignments can cause a strict failure. The human checks validate agreement with the reference policy, not its downstream usefulness. A small executable study showing which scored errors actually impair coordination or violate meaningful boundaries would strengthen the paper.
-
The task coverage is narrower than the orchestration framing suggests. Roles, workflows, and candidate fragments are already supplied; the model selects and copies information rather than designing the orchestration. Appendix E further states that the 100 generated instances preserve the structure, fragment functions, and reference mappings of the 10 templates. Domain rewrites add linguistic variety, but relatively little variety in the underlying allocation problems. The assembly format also adds copying and formatting demands that a deployment could avoid by having the model select IDs and a program concatenate the fragments. I therefore remain unsure how much the combined score measures orchestration-specific ability beyond information selection and instruction following. Synthetic construction is not itself a problem, but the paper needs stronger evidence that this particular simplification captures an important practical bottleneck.
Questions for authors
-
Evidence for practical relevance: On a representative subset, do higher benchmark scores predict better downstream outcomes? Does correcting the scored assignment errors improve those outcomes?
-
Reference-policy clarification: How are confidential information and ordinary shared requirements distinguished when constructing the mappings? What justifies withholding a reviewer’s acceptance criterion when it could help the actor satisfy the task?
-
Evidence for the assembly task: What changes if role-fragment assignment is followed by deterministic prompt assembly? How much of the direct assembly failure comes from different selection decisions versus copying, formatting, or detection errors?