- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents FATE benchmark that tests whether teams of LLM agents can complete realistic professional work together. It contains 112 tasks covering 14 domains and 129 occupations, constructed by 32 domain professionals. In each task, every agent takes an occupational role with its own responsibilities, input resources, and permissions, so the team has to communicate and hand off intermediate artifacts to produce the final deliverables. Scoring happens at two levels: individual members' artifacts (ME) and the team's final deliverables (FE), using both rule-based checks and LLM-judged criteria.
The authors evaluate eight LLMs on FATE. They also compare four communication protocols and conduct a matched-budget comparison between single-agent and multi-agent systems across three domains. The best system reaches an FE 42.0. Multi-agent systems use substantially more tokens and take longer to run, but the performance gains are relatively small. However, when compared under the same computational budget, they perform better overall than single-agent systems, except in the Research domain. Point-to-point communication performs best across the three domains tested. Models generally perform worse on multimodal tasks, and the two most common failure types are communication coordination (34%) and output-interface misalignment (32%).
Critical strengths
I like that FATE focuses on professional tasks where agents actually need to work together, rather than just assigning different roles to solve a problem. Agents have access to different files and permissions, and their work has to be passed between teammates to produce the final deliverables. The tasks also use real materials in their original formats. Having domain experts construct and review the tasks, and test the evaluators with valid and invalid outputs, makes the benchmark more convincing.
I also find the separate evaluation of individual members (ME) and final team outputs (FE) useful. For example, in Building, the average ME is 51.0 while FE is only 16.0. Although these scores use different criteria, they suggest that producing reasonable intermediate artifacts does not necessarily lead to a successful final result. Looking at both gives a better picture of how the teams perform than evaluating only the final output.
Critical weaknesses
The experimental results seem inconsistent. Table 2 and Figure 10 report quite different FE scores for GPT-5.6 Terra on the same three domains, even though both appear to use high reasoning (16.1/56.0/35.0 vs. 6.9/72.8/11.7). This makes it hard to judge how stable the results are
I have some concerns about how the evaluation scores are computed. GPT-5.6 Terra serves as both an evaluated model and the LLM judge. While the 92% human agreement is nice result, it is unclear whether this holds across different evaluated models.
Questions for authors
Given that GPT-5.6 Terra is also used as the LLM judge, I wonder if the authors have checked for potential evaluation bias?
Could the authors explain how FE is computed for tasks with different scoring rules? For example, the Tourism task uses a score cap, while the Finance task uses a continuous FinalScore. I am a little confused
4 comments
Response 2
W2 & Q1: I have some concerns about how the evaluation scores are computed. GPT-5.6 Terra serves as both an evaluated model and the LLM judge. While the 92% human agreement is nice result, it is unclear whether this holds across different evaluated models. Given that GPT-5.6 Terra is also used as the LLM judge, I wonder if the authors have checked for potential evaluation bias?
We appreciate your insightful comment regarding potential model bias and cross-model consistency, which are important considerations when using an LLM as a judge.
In FATE, the evaluation criteria are based on objective, verifiable facts, which naturally minimizes subjective bias. To further address your concerns, we conducted an additional validation study on 300 randomly sampled criterion instances. We introduced Gemini 3.8 Flash and Grok 4.6 as independent judges alongside GPT-5.6 Terra to evaluate the exact same instances:
(1) The criterion instances are not sensitive to the choice of judge. As shown in Table 2, across all 300 instances, GPT-5.6 Terra achieves 92.00% agreement with human judgments. Meanwhile, the independent judges Gemini 3.8 Flash and Grok 4.6 achieve 93.67% and 95.00% agreement, respectively. Furthermore, all four evaluators (GPT-5.6 Terra, Gemini 3.8 Flash, Grok 4.6, and Human) reach unanimous consensus on 89.67% of the instances, demonstrating that the evaluation outcomes are highly consistent and do not depend on a specific judge model.
(2) There is no evidence of self-preference bias. As shown in Table 3, in the 38 instances where GPT-5.6 Terra evaluates its own outputs, it agrees with human judgments on 92.11% of cases. In the rare 3 cases of disagreement, GPT-5.6 Terra leaned stricter toward itself (2 strict,1 lenient), showing no tendency to inflate its own scores.
(3) Evaluation consistency remains stable across evaluated models. After excluding its own outputs, GPT-5.6 Terra maintains a 91.98% agreement rate with human judgments on the 262 instances generated by other evaluated models. This aligns closely with its overall performance, confirming that the evaluation reliability of GPT-5.6 Terra holds consistently across different target models.
Table 2: Inter-Judge Consistency and Human Agreement (N = 300)
| Evaluator | Agreement with Human | Cohen's Kappa |
|---|---|---|
| GPT-5.6 Terra | 92.00% | 0.839 |
| Gemini 3.8 Flash | 93.67% | 0.873 |
| Grok 4.6 | 95.00% | 0.900 |
Table 3: Evaluation Comparison on GPT-5.6 Terra Outputs (N = 38)
| Evaluator | Agreement with Human | #More Lenient | #Stricter |
|---|---|---|---|
| GPT-5.6 Terra | 92.11% | 1 | 2 |
| Gemini 3.8 Flash | 94.74% | 2 | 0 |
| Grok 4.6 | 97.37% | 1 | 0 |
Q2:Could the authors explain how FE is computed for tasks with different scoring rules? For example, the Tourism task uses a score cap, while the Finance task uses a continuous FinalScore. I am a little confused
Thank you for your question. Although different tasks use different scoring rules, their scores are converted to the same range from 0 to 1 before the final task score is calculated.
(1) For rule-based evaluation, each item evaluates to either a binary result or a normalized continuous score. For example, in the Finance task, the raw FinalScore is calculated on a [0, 100] scale based on domain criteria and then divided by 100 to map directly to [0, 1].
(2) For LLM-based evaluation, the model delivers a 0 or 1 judgment for each criterion. The final score for that evaluation item is the average of these binary decisions.
(3) The scores are combined using the weights and rules defined by each task. Any score cap is applied after the initial overall score has been calculated. For example, in the Acadia Tourism task, if the safety score(a task-specific metric) is below 0.6, the final score cannot exceed 0.49. Other tasks may use different conditions.
In short, the process is: calculate each evaluation score → convert it to 0–1 → combine the scores → apply optional task-specific score aggregation → obtain the final task score.
Thank you for the clarifications and additional analysis. I appreciate the authors' responses to my questions. However, I will maintain my original score of 3
Response to Reviewer 4PxR
We sincerely thank the reviewer for the positive score! We will incorporate the additional analyses into the revised paper. If you have any further questions, please feel free to let us know.
Response 1
Dear Reviewer 4PxR,
Thank you for your careful and insightful review. We appreciate your positive assessment and your recognition of FATE's focus on realistic professional collaboration and its evaluation design. Below, we provide clarifications and additional evidence concerning experimental consistency and evaluation reliability. We sincerely hope these responses address your concerns!
Thank you for your question! Figure 10 in paper uses a subset of tasks from the three domains. Table 2 in paper reports results for the full task set in each domain. Because they use different sets of tasks, the FE scores can differ even when both use high reasoning effort. Table 1 of this response compares the subset and full task set FE scores for GPT-5.6 Terra across selected domains. The full task set results show consistent improvements with increasing reasoning effort across these domains. We plan to update Figure 10 using the full-task-set results reported in this table. We observe this same upward trend on both the subset and the full task set, which further shows that higher reasoning effort leads to better collaboration.
Table 1: GPT-5.6 Terra FE Scores on Task Subsets and Full Task Sets Across Selected Domains.