- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
OfficeTown is a framework for generating professional-work tasks through persistent, permissioned organizational simulation. Simulated professionals coordinate assignments, delegate execution to assistants, review artifacts, and preserve materials and unresolved matters for subsequent work. Selected task-time snapshots become executable evaluation and training resources. The paper presents OfficeTown Bench with 406 tasks and 4,836 criteria from 23 organizations, evaluates 13 models, and reports external-benchmark improvements after SFT and on-policy distillation of Qwen3.5-9B. A separate study evaluates the principal-assistant harness. The central contribution is a potentially reusable source of contextualized professional-work data, rather than a new learning algorithm.
Critical strengths
-
A coherent and practically motivated approach to task generation. Generating assignments from accumulated artifacts, decisions, and review feedback is an interesting alternative to independently synthesizing tasks. The distinction between saving, accepting, and sharing work, together with permission-filtered task-time snapshots, makes the proposed environment more concrete than an unconstrained multi-agent conversation.
-
Substantial resources and evidence beyond benchmark difficulty. The paper combines a multi-domain benchmark, expert-informed rubrics, training trajectories, and external transfer experiments. The GDPval comparison explicitly uses 79 shared tasks. The appendices provide useful accounting details: the domain task totals, displayed leaderboard aggregates, SFT filtering counts, and optimizer-step counts can be reconstructed consistently.
Critical weaknesses
-
The evidence does not isolate the contribution of persistent organizational simulation. External improvements show that training on the resulting data can be useful, but do not establish whether persistence, cross-assignment dependencies, permissions, or organizational review provides an advantage over simpler task synthesis with matched teachers, data volume, and generation/training budgets. The separate harness study does not isolate simulation either, as the paper acknowledges. Moreover, Appendix B.1 notes that task-identity separation does not establish project-level or semantic isolation; organization/project overlap and semantic decontamination results are not reported. These limitations leave the central explanation for the data's usefulness insufficiently tested.
-
Several evaluation choices prevent a fully auditable interpretation of the reported gains. For APEX-Agents and Terminal-Bench in Table 3, the exact per-stage evaluated task counts and common-task results are not supplied; Appendix E.5 refers to reporting/scored subsets. Thus, the reported increases cannot be independently separated from coverage differences. Grader failures are also assigned zero, with markedly different counts in the Flash effort comparison (34 for Low versus 4 for High), which mixes agent performance with evaluation reliability. Expert review of tasks and rubrics does not establish model-output grading accuracy, and no human-grader agreement study is reported. Finally, task/criterion selection used multi-model pilot results and reused eligible pilot judgments (Appendix B.1), so the selection and reuse procedures need an auditable account. These are comparability and evidence limitations, not evidence that the reported experiments were fabricated.
Questions for authors
-
External-evaluation coverage: For each Base/SFT/SFT+OPD result in Table 3, please provide the task IDs/counts, number of runs, failure/exclusion rules, and the intersection evaluated at all stages. Can the APEX and Terminal-Bench improvements be reproduced on that common set, with paired uncertainty estimates? GDPval's 79-task shared-set reporting is helpful; the same clarity is needed for the other benchmarks.
-
Grading reliability: Can terminal grading failures be re-evaluated, and can results be reported both under the current zero-imputation rule and a clearly specified sensitivity analysis? In particular, how does resolving the 34 Low versus 4 High Flash grading failures affect the 62.1 versus 61.9 Weighted Score comparison? Please also report a stratified human audit of output-level judgments, including disagreement and false-rejection rates.
-
Simulation-specific value: Is there a matched comparison against independently synthesized tasks or a history-disabled generator, controlling teacher, task volume, and generation/training compute? Such evidence would materially strengthen the attribution of learning gains to continuing organizational work rather than additional teacher-generated data alone.
-
Separation and selection: Please provide organization/project-level overlap and semantic-duplicate audits across benchmark, SFT, and OPD resources, along with the external-benchmark exposure policy. For pilot-based task/criterion selection, what retention rules were used, and how were reused judgments verified against the final task and rubric versions? Is an independent frozen-set evaluation available?
-
Reporting consistency: Appendix C.1, p. 27, states that Sol's Overall Weighted Score remains pending, whereas Table 8, p. 29, reports 68.5 and complete domain values. Please reconcile this status/version inconsistency. The reported aggregates themselves are arithmetically consistent within displayed rounding.
-
Resource comparability: Please supply the absolute token counts underlying the claimed 21.8% reduction versus MetaGPT and 3.49x usage versus Codex, and the exact Terminal-Bench budgets in Table 4. The retrospective ending-turn curves are explicitly disclosed, but should be interpreted as accumulated final outcomes by completion turn, not measurements of intermediate quality or fresh shorter-budget runs. Clarifying matched-resource results would determine how much of the harness advantage can be attributed to its design.
3 comments
Value of Persistent Organizational Simulation Over Static Synthesis (W1 & Q3, Appendices A.8, E.6)
We combine (1) corpus lineage (84.2% inherited reviewed deliverables, 46.1% explicit continuation links, and 93.1% prior-artifact SFT tool reads / 71.3% overall), (2) our ClawGym-inspired 2K Static-SFT control (Bai et al., 2026; near-matching 2,201 vs. 2,202 trajectories from the same 34 seeds under identical DeepSeek-V4-Flash teacher, sandbox, and 1-epoch Qwen3.5-9B SFT recipe, evaluated under the Table 3 reporting protocols), and (3) 30-day GPT-5.5 mechanism ablations (Shared Comment §3: Full sustains 91.2% multi-task dependencies and 44.1% multi-round revisions vs. 5.3% / 2.6% for no-history and 0.0% / 0.0% for no-review):
| Qwen3.5-9B Stage | GDPval | JobBench-Easy | Terminal-Bench 2.1 |
|---|---|---|---|
| Base | 65.75 | 40.41 | 42.31 |
| Static-SFT (ClawGym-inspired, 2,201 traj.) | 67.80 | 43.60 | 47.70 |
| OfficeTown SFT (Matched SFT recipe, 2,202 traj.) | 70.30 (+2.50) | 44.87 (+1.27) | 51.22 (+3.52) |
| OfficeTown SFT + OPD (additional OPD stage) | 73.65 | 51.48 | 63.64 |
Evaluating the same ablations inside OfficeTown Harness on Terminal-Bench 2.1 (Table 4, GPT-5.5 xhigh, 3-run means) confirms the same component ordering at inference time:
| Inference Harness (GPT-5.5 xhigh) | Terminal-Bench 2.1 |
|---|---|
| MetaGPT | 81.1 |
| Codex (Single-Agent) | 83.7 |
| CAMEL | 85.1 |
| OfficeTown Harness (no-history) | 86.5 |
| OfficeTown Harness (no-review) | 87.2 |
| OfficeTown Harness (Full) | 89.0† |
†Note: The submitted Table 4 contained a table transcription typo (87.4 instead of 89.0); 89.0 is computed directly from the original evaluation runs with no model reruns.
Split Separation, Provenance Scope, and Rubric Selection Stability (W2 & Q4, Appendix B.5)
- Internal split separation & shared-project scope: Across OfficeTown Bench (406 tasks, 139 projects), SFT (2,202 trajectories, 460 projects), and OPD (828 environments; 0 task-ID overlap with Bench and SFT in the released disjoint OPD manifest), exact and normalized prompt duplicates are 0. As noted in Appendix B.1, continuing organizations naturally contribute distinct tasks across splits (94 shared projects between Bench and SFT), and all post-training transfer claims in Table 3 are evaluated on independent external benchmarks.
- External provenance check: Auditing all 188 world source registries (3,903 public standard/agency URLs) found 0 benchmark-name hits, and exact-byte checks across all 23,320 unique OfficeTown input files against GDPval (509), AutomationBench (571), and JobBench (256) found 0 matches, with all benchmark evaluations executed offline without internet access.
- Rubric Pruning Rule & Post-Freeze Ranking Stability: To filter out trivial items while preserving evaluation rigor, we applied fixed domain quotas sorted by 15-model pilot pass counts (447 → 407 → 406 tasks, dropping 1 cross-domain item) and pruned the rubric from 9,472 to 4,836 criteria by retaining the harder tail half per task (rounding up on odd counts) while maintaining original ROC tier weights (98.0% combined P0+P1 share; all 5,142 pilot judgments on the 406 tasks were byte-verified). On the 15-model pilot panel, full and pruned Weighted Scores correlate at Spearman (Pass@1 shift less than 1 pp).Crucially, to verify that rubric pruning does not overfit to the pilot models, we evaluated 3 frontier models and 2 execution frameworks strictly after the 4,836-criterion mask was frozen (none of which appeared in the pilot panel). Because GPT-5.5 high grades all 9,472 draft criteria before applying the frozen mask, we can directly compare full vs. pruned scores on the exact same outputs. As expected, removing easier criteria lowers absolute Weighted Scores by 3.5 to 19.2 pp, yet induces zero rank inversions:
| Post-Freeze Model / Framework (n=406) | Full Draft (9,472) | Released (4,836) | Post-Freeze Rank |
|---|---|---|---|
| Grok-4.7 | 80.7 | 77.2 | 1st (unchanged) |
| Claude Opus 5.5 | 72.0 | 67.5 | 2nd (unchanged) |
| Gemini 3.8 Flash | 41.5 | 24.2 | 3rd (unchanged) |
| Pi Framework (Flash High) | 64.6 | 48.6 | 1st (unchanged) |
| OpenCode Framework (Flash High) | 59.9 | 40.7 | 2nd (unchanged) |
Reconciling GPT-5.6 Sol in Appendix C.1 and Appendix D.1 (W2 & Q5)
Thank you for catching this discrepancy. The 68.5 figure in Appendix C.1 was an oversight on our part; on the final 406-task benchmark, GPT-5.6 Sol's exact Overall Weighted Score is 68.42% (reported as 68.4) with a Pass@1 of 33.5% (136/406). We have updated Appendix C.1 to match Appendix D.1 (Table 13) so that both sections now consistently report 68.4.
Per-Task Token Accounting, Terminal Budgets, and Ending-Turn Curves (W1 & Q6, Appendix C.4)
- Exact ResearchCodeBench token accounting (summed over all harness roles and retries under GPT-5.5 xhigh): OfficeTown Harness outperforms MetaGPT by +5.25 pp while using 21.8% fewer tokens per task ((761,644 − 595,780) / 761,644), and outperforms single-agent Codex by +5.52 pp at 3.49× Codex's token usage (595,780 / 170,831):
| Harness (GPT-5.5 xhigh) | ResearchCodeBench (%) | Mean Tokens / Task | Mean Tokens (Passed) | Terminal-Bench 2.1 (%) |
|---|---|---|---|---|
| OfficeTown Harness | 46.40 | 595,780 | 844,955 | 89.0† |
| MetaGPT | 41.15 | 761,644 | 1,188,581 | 81.1 |
| Codex (Single-Agent) | 40.88 | 170,831 | 282,153 | 83.7 |
| CAMEL | 39.95 | 357,029 | 567,487 | 85.1 |
†Note: The submitted Table 4 contained a table transcription typo (87.4 instead of 89.0); 89.0 is computed directly from the original evaluation runs with no model reruns.
- Terminal-Bench 2.1 budgets (Table 4): External Terminal-Bench 2.1 baselines in Table 4 use their standard reported budget settings rather than uniform token caps, as noted in the Table 4 caption and Appendix C.4.
- Ending-turn curves (Figure 6): Figure 6 summarizes the final scores of runs that completed by each turn over the fixed 406-task denominator; these curves do not show intermediate deliverable quality or what score would have been achieved had execution been stopped early at shorter budgets (clarified in the Figure 6 caption).
We thank Reviewer hpdu for the rigorous, constructive questions on coverage, grading, separation, reporting consistency, and resource comparability. Please also see our Shared Comment above; below we address Q1 through Q6 and Weaknesses W1and W2 in detail.
External Evaluation Coverage, Common Intersections, and All Assigned Sensitivity (W1 & Q1, Appendix E.6)
In Table 3, OfficeTown Bench already evaluates all 406 tasks with timeouts or failures scored as 0 (1.48 → 3.45 → 4.68). Because external benchmarks exclude timeouts by default, we verify that our transfer gains are never an artifact of unequal task subsets by testing two complementary scopes:
Grader Failure Breakdown, Regrade Sensitivity, and Blinded Human Calibration (W2 & Q2, Appendix C.4)
1. Regrading the 38 Grader Failures (0 Agent Reruns): In Figure 6b, the 38 failed tasks (34 for Flash Low, 4 for Flash High) resulted from automated judge errors (API timeouts or formatting issues) rather than agent failures, though they were conservatively scored as 0. To isolate this evaluation artifact, we re-evaluated these 38 saved outputs once using the same GPT-5.5 high judge, successfully recovering 29 valid scores (while 9 remained ungraded due to persistent API rate limits). As detailed below, we conducted a sensitivity analysis across four scoring policies to verify whether the 9 remaining ungraded tasks could alter our conclusions:
Once the grader timeouts are resolved, Flash Low outperforms Flash High by +3.64 pp . Crucially, even underthe most conservative worst-case bounds for the remaining 9 ungraded tasks (assigning 0 to Flash Low and 100 to Flash High), Flash Low's lower bound (66.08) still exceeds Flash High's upper bound (62.69) by +3.39 pp, confirming that our finding is robust to grader failures.
2. Blinded output-level human calibration: While Section 3.2 reviewed task and rubric design, here we directly test whether GPT-5.5 high grades actual model outputs the same way human experts do. Four domain experts independently graded 40 non-overlapping anonymized deliverables across all 8 domains (~40 expert hours, ~1 hour/task to inspect workspace files, spreadsheet formulas, and tool logs) without seeing AI scores:
Across all 366 evaluable criterion pairs (300 both-met, 43 both-not-met, 6 AI-met/human-not-met, 17 AI-not-met/human-met), GPT-5.5 high agrees with human experts 93.72% of the time (100% on state/response checks and 93.10% on complex file deliverables), with only 1.64% false-acceptance and 4.64% slightly stricter AI rejections on wording or section completeness.