acceptodds
Under review as a conference paper at ICLR 2027

FakeWorldEval400: A Contract-Grounded Benchmark for Evaluating Multi-User Coordination in Language Agents

Abstract

Language agents increasingly operate in workflows involving multiple users, ses- sions, and evolving permissions, making reliable coordination essential. However, task completion and response quality alone cannot establish whether an agent addresses the correct recipient, respects disclosure boundaries, and follows cur- rent authority and evidence. To address this evaluation challenge, we introduce FakeWorldEval400, a contract-grounded benchmark for evaluating multi-user co- ordination in language agents. The benchmark comprises 400 manually authored tasks across 22 workflow families and formalizes coordination requirements along five axes: target, scope, authority, source, and time. We develop an architecture- agnostic action interface and a hybrid evaluation protocol that combines deter- ministic contract checks with criterion-level semantic judgments, enabling strict success assessment and trace-based failure localization. Experiments with four mixed-access baselines on 1,600 frozen actor traces yield Hybrid strict pass rates of 37.0%–53.2%. Further analysis of the 207 check-required tasks shows that 81.7%– 89.9% of Rule failures occur while the authored check criterion passes, which localizes most failures to other hard obligations. These results reveal substantial gaps in contract satisfaction within the evaluated multi-user workflows. FakeWorldEval400 provides a reproducible testbed for comparing coordination systems and diagnosing their observable failures under explicit, evolving contracts. Resources are available at https://anonymous.4open.science/r/submission-code-57FE.

Primary area:
datasets and benchmarks
Keywords:
Multi-user coordination, Language agents, Contract-grounded evaluation, Agent benchmarks, Trace-based failure localization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uT1TOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FakeWorldEval400 is a benchmark assessing whether language agents meet evolving multi-user coordination requirements. It includes 400 manually authored synthetic workflows across 22 workflow families, with explicit obligations specifying target, scope, authority, source, and time. A unified action interface records inspection, waiting, answering, and notification. Evaluation combines deterministic contract checking with criterion-by-criterion semantic judgments, requires both non-compensatory hard gates to pass, and retains trajectory evidence for diagnosis. Four language-model baselines produce 1,600 frozen actor traces, with Hybrid strict pass rates of 37.0%–53.2%. On inspection-required tasks, most Rule failures occur despite satisfaction of the prescribed inspection preconditions. The contribution is an auditable benchmark and failure-localization framework, supported by task auditing, released trajectory artifacts, and analyses by workflow type and length.

Critical strengths
  1. The benchmark targets an important gap: producing a plausible answer is not equivalent to satisfying recipient, disclosure, authorization, provenance, and timing constraints. Explicit contracts and non-compensatory hard gates provide a clear definition of success.
  2. Task-level evidence and separately reported Rule and semantic outcomes make failures inspectable. Audits of passing, near-passing, and wrong-target cases across all 400 tasks provide a useful check on the executable contracts; the released records connect tasks, trajectories, and criterion-level judgments.
Critical weaknesses
  1. Although the semantic evaluator determines Hybrid strict success, it has not been validated against independent human judgments. Both initial assessment and arbitration use the same DeepSeek model, and 37.5%–53.3% of trajectories require arbitration because of substantive disagreement.
  2. Although the benchmark aims to compare coordination architectures, its empirical baselines are only four language models under the same wrapper. Including systems with explicit authorization tracking, shared-memory governance, or controlled routing would demonstrate whether the benchmark discriminates mechanisms designed specifically for this problem.
  3. All workflows are human-authored synthetic data, spanning only one to four virtual time steps. The very low pass rate on four-step tasks is confounded with task composition, making it unclear whether difficulty stems from temporal coordination, particular constraint categories, or other design choices.
Questions for authors

See Weaknesses.

Reviewer_jdv1OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces FakeWorldEval400, a benchmark for agents that serve several users. It has 400 synthetic tasks with executable contracts over five axes. A trace passes only if both a rule track and an LLM judge accept it.

Contributions:

  1. 400 task contracts and a trace-based evaluator.
  2. A release of 1,600 frozen traces with per-criterion verdicts.
  3. Results for four actors: 37.0% to 53.2% strict pass, and 0% to 1% on four-turn tasks.
Critical strengths
  1. An inspectable release. Traces and per-criterion verdicts are all public. This is rare for agent benchmarks.

  2. A useful framing with a strict metric. The five axes give a clear vocabulary. A fluent reply cannot hide a wrong recipient.

Critical weaknesses
  1. The four-turn results seem to measure what the actor can see. In 98 of 100 four-turn tasks, the facts needed in the last turn appear only in turn 2. The released traces suggest that actors do not see earlier turns.

  2. The judge is not validated against humans. It also miscounts reply budgets, with 150 and 176 false failures for two actors. Fixing only this error turns the reported tie between Gemma4 E4B and Qwen3.5 9B into a 5.5 point gap.

Questions for authors
  1. Did actor calls include earlier turns?
  2. How well does the judge agree with humans?
  3. Can each hard criterion be met from what is visible at its turn?
Reviewer_hSBCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a benchmark called FakeWorldEval400, consisting of 400 synthetic tasks across 22 workflow families and formalizes coordination requirements along the five axes: target, scope, authority, source and time. All agents act through a shared check/wait/answer/notify interface. This interface supports both deterministic contract checks and semantic judgments. A run counts as a strict success only if all hard criterion passes in both. Four models generate 1,600 frozen traces, with Hybrid pass rates ranging from 37.0% to 53.2%. FakeWorldEval400 provides a reproducible testbed for comparing coordination systems and diagnosing their observable failures under explicit, evolving contracts.

Critical strengths
  1. The benchmark lets you check each important coordination rule separately. It does so by sorting requirements into 5 types and gives every agent the same set of actions. That way it can check who the agent replied to, whether it had permission, and whether it acted at the right time whereas a score that is only based on the final answer could miss these. I agree with the setting that requiring every key check is the condition to pass. The passing and failing examples in Appendix D (pp. 17–23) show how specific agentic actions can be traced back to the exact rules they broke.

  2. I feel this a very timely study given AI assistants increasingly act for several people at once, in shared inboxes. Sending information to the wrong person, or sharing too much or acting on outdated permissions can cause major harm. A benchmark aimed at these coordination and privacy failures is generally useful to the community.

Critical weaknesses
  1. The LLM judge has not been checked against humans. The judge decides many outcomes: it rejects 80–116 traces per model that pass the rule checks, and it lowers pass rates by 20–29 points. Since both prompt versions and the tie-breaker use the same model, so the tie-breaker settles disagreements without showing which answer is correct, and it is needed on 37–53% of traces. Without a human check, it is unclear how much of the score reflects real coordination ability and how much reflects judge strictness or noise.

  2. I am failing to understand teh setting of 4-turn tasks, 1/4th of the benchmark is passed by almost no model, even though the same models pass 31–62% of three-turn tasks. The paper doesnt show the exact cause of this drop, whether outputs failed to parse, or whether a human or a stronger model can solve these tasks. The reason could be how the tasks were built rather than from what the models capabilities.

Questions for authors
  1. Can you provide an independent human audit of a sample of frozen traces, including Rule-only passes, arbitrated cases, and four-turn workflows?
  2. For traces that pass the rules but fail the judge, which judge checks fail? Are they checks the rules already cover?
  3. Appendix B.2 describes material disagreement without specifying its exact threshold or the final-decision rule when no arbiter is called. What are these rules, and how often do the initial prompts disagree on strict success rather than only criteria or continuous scores?
  4. How much of each task's correct behavior is stated explicitly in the turn instruction and routing instruction? Do you have results on a variant with the routing hints and imperative user cues removed, so that the target and authority axes are actually inferred?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.