- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
FakeWorldEval400 is a benchmark assessing whether language agents meet evolving multi-user coordination requirements. It includes 400 manually authored synthetic workflows across 22 workflow families, with explicit obligations specifying target, scope, authority, source, and time. A unified action interface records inspection, waiting, answering, and notification. Evaluation combines deterministic contract checking with criterion-by-criterion semantic judgments, requires both non-compensatory hard gates to pass, and retains trajectory evidence for diagnosis. Four language-model baselines produce 1,600 frozen actor traces, with Hybrid strict pass rates of 37.0%–53.2%. On inspection-required tasks, most Rule failures occur despite satisfaction of the prescribed inspection preconditions. The contribution is an auditable benchmark and failure-localization framework, supported by task auditing, released trajectory artifacts, and analyses by workflow type and length.
Critical strengths
- The benchmark targets an important gap: producing a plausible answer is not equivalent to satisfying recipient, disclosure, authorization, provenance, and timing constraints. Explicit contracts and non-compensatory hard gates provide a clear definition of success.
- Task-level evidence and separately reported Rule and semantic outcomes make failures inspectable. Audits of passing, near-passing, and wrong-target cases across all 400 tasks provide a useful check on the executable contracts; the released records connect tasks, trajectories, and criterion-level judgments.
Critical weaknesses
- Although the semantic evaluator determines Hybrid strict success, it has not been validated against independent human judgments. Both initial assessment and arbitration use the same DeepSeek model, and 37.5%–53.3% of trajectories require arbitration because of substantive disagreement.
- Although the benchmark aims to compare coordination architectures, its empirical baselines are only four language models under the same wrapper. Including systems with explicit authorization tracking, shared-memory governance, or controlled routing would demonstrate whether the benchmark discriminates mechanisms designed specifically for this problem.
- All workflows are human-authored synthetic data, spanning only one to four virtual time steps. The very low pass rate on four-step tasks is confounded with task composition, making it unclear whether difficulty stems from temporal coordination, particular constraint categories, or other design choices.
Questions for authors
See Weaknesses.