acceptodds
Under review as a conference paper at ICLR 2027

OfficeTown: Generative Organizations for Evolving Agent Intelligence

16 views

Abstract

AI agents increasingly use files and software tools to produce professional deliverables. Such work draws on materials, decisions, and feedback accumulated across assignments. We introduce OfficeTown, a generative simulation framework for professional organizations in which tasks and their context arise from continuing organizational work. From a natural-language description of an organization's purpose and activities, OfficeTown initializes an organization whose members coordinate work, delegate bounded assignments to assistant agents, and review their deliverables. Artifacts, decisions, and unresolved matters persist and shape subsequent work, while responsibilities and permissions determine what each member can see and do. We curate work requests, their context, and review feedback into executable resources for evaluation and learning. We construct OfficeTown Bench with 406 tasks and 4,836 evaluation criteria from 23 simulated organizations across eight professional domains, incorporating domain-expert feedback. Frontier-model evaluations reveal substantial remaining headroom. Supervised fine-tuning and on-policy distillation on OfficeTown data substantially improve Qwen3.5-9B on external professional-work benchmarks, including GDPval. Supplementary evaluations show that the principal–assistant harness achieves higher scores than the single- and multi-agent baselines tested. These results highlight the value of organizational simulation in generating work-grounded data and environments for the continued development of generalist agents.

Primary area:
datasets and benchmarks
Keywords:
Data Synthesis, Agent Benchmarking, Organizational Simulation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hpduOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OfficeTown is a framework for generating professional-work tasks through persistent, permissioned organizational simulation. Simulated professionals coordinate assignments, delegate execution to assistants, review artifacts, and preserve materials and unresolved matters for subsequent work. Selected task-time snapshots become executable evaluation and training resources. The paper presents OfficeTown Bench with 406 tasks and 4,836 criteria from 23 organizations, evaluates 13 models, and reports external-benchmark improvements after SFT and on-policy distillation of Qwen3.5-9B. A separate study evaluates the principal-assistant harness. The central contribution is a potentially reusable source of contextualized professional-work data, rather than a new learning algorithm.

Critical strengths
  1. A coherent and practically motivated approach to task generation. Generating assignments from accumulated artifacts, decisions, and review feedback is an interesting alternative to independently synthesizing tasks. The distinction between saving, accepting, and sharing work, together with permission-filtered task-time snapshots, makes the proposed environment more concrete than an unconstrained multi-agent conversation.

  2. Substantial resources and evidence beyond benchmark difficulty. The paper combines a multi-domain benchmark, expert-informed rubrics, training trajectories, and external transfer experiments. The GDPval comparison explicitly uses 79 shared tasks. The appendices provide useful accounting details: the domain task totals, displayed leaderboard aggregates, SFT filtering counts, and optimizer-step counts can be reconstructed consistently.

Critical weaknesses
  1. The evidence does not isolate the contribution of persistent organizational simulation. External improvements show that training on the resulting data can be useful, but do not establish whether persistence, cross-assignment dependencies, permissions, or organizational review provides an advantage over simpler task synthesis with matched teachers, data volume, and generation/training budgets. The separate harness study does not isolate simulation either, as the paper acknowledges. Moreover, Appendix B.1 notes that task-identity separation does not establish project-level or semantic isolation; organization/project overlap and semantic decontamination results are not reported. These limitations leave the central explanation for the data's usefulness insufficiently tested.

  2. Several evaluation choices prevent a fully auditable interpretation of the reported gains. For APEX-Agents and Terminal-Bench in Table 3, the exact per-stage evaluated task counts and common-task results are not supplied; Appendix E.5 refers to reporting/scored subsets. Thus, the reported increases cannot be independently separated from coverage differences. Grader failures are also assigned zero, with markedly different counts in the Flash effort comparison (34 for Low versus 4 for High), which mixes agent performance with evaluation reliability. Expert review of tasks and rubrics does not establish model-output grading accuracy, and no human-grader agreement study is reported. Finally, task/criterion selection used multi-model pilot results and reused eligible pilot judgments (Appendix B.1), so the selection and reuse procedures need an auditable account. These are comparability and evidence limitations, not evidence that the reported experiments were fabricated.

Questions for authors
  1. External-evaluation coverage: For each Base/SFT/SFT+OPD result in Table 3, please provide the task IDs/counts, number of runs, failure/exclusion rules, and the intersection evaluated at all stages. Can the APEX and Terminal-Bench improvements be reproduced on that common set, with paired uncertainty estimates? GDPval's 79-task shared-set reporting is helpful; the same clarity is needed for the other benchmarks.

  2. Grading reliability: Can terminal grading failures be re-evaluated, and can results be reported both under the current zero-imputation rule and a clearly specified sensitivity analysis? In particular, how does resolving the 34 Low versus 4 High Flash grading failures affect the 62.1 versus 61.9 Weighted Score comparison? Please also report a stratified human audit of output-level judgments, including disagreement and false-rejection rates.

  3. Simulation-specific value: Is there a matched comparison against independently synthesized tasks or a history-disabled generator, controlling teacher, task volume, and generation/training compute? Such evidence would materially strengthen the attribution of learning gains to continuing organizational work rather than additional teacher-generated data alone.

  4. Separation and selection: Please provide organization/project-level overlap and semantic-duplicate audits across benchmark, SFT, and OPD resources, along with the external-benchmark exposure policy. For pilot-based task/criterion selection, what retention rules were used, and how were reused judgments verified against the final task and rubric versions? Is an independent frozen-set evaluation available?

  5. Reporting consistency: Appendix C.1, p. 27, states that Sol's Overall Weighted Score remains pending, whereas Table 8, p. 29, reports 68.5 and complete domain values. Please reconcile this status/version inconsistency. The reported aggregates themselves are arithmetically consistent within displayed rounding.

  6. Resource comparability: Please supply the absolute token counts underlying the claimed 21.8% reduction versus MetaGPT and 3.49x usage versus Codex, and the exact Terminal-Bench budgets in Table 4. The retrospective ending-turn curves are explicitly disclosed, but should be interpreted as accumulated final outcomes by completion turn, not measurements of intermediate quality or fresh shorter-budget runs. Clarifying matched-resource results would determine how much of the harness advantage can be attributed to its design.

3 comments

We thank Reviewer hpdu for the rigorous, constructive questions on coverage, grading, separation, reporting consistency, and resource comparability. Please also see our Shared Comment above; below we address Q1 through Q6 and Weaknesses W1and W2 in detail.

External Evaluation Coverage, Common Intersections, and All Assigned Sensitivity (W1 & Q1, Appendix E.6)

In Table 3, OfficeTown Bench already evaluates all 406 tasks with timeouts or failures scored as 0 (1.48 → 3.45 → 4.68). Because external benchmarks exclude timeouts by default, we verify that our transfer gains are never an artifact of unequal task subsets by testing two complementary scopes:

  • Common (identical completed tasks): Evaluates Base, SFT, and SFT + OPD on the exact same tasks completed by all checkpoints, removing any task-selection differences.
  • All Assigned (full workload, unresolved = 0): Evaluates all assigned tasks after recovering infrastructure interruptions, penalizing every remaining timeout or unresolved run as 0.
BenchmarkScopeBaseSFTSFT + OPDBase → SFT ΔBase → OPD Δ
GDPvalCommon (n=79)65.7570.3073.65+4.55 pp+7.90 pp
GDPvalAll Assigned (n=140)60.3764.6869.71+4.31 pp+9.34 pp
JobBench-EasyCommon (n=43, all 8 runs)42.2845.6753.35+3.39 pp+11.07 pp
JobBench-EasyAll Assigned (n=63)39.3444.3750.72+5.03 pp+11.38 pp
APEX-AgentsCommon (n=198)11.1118.1819.70+7.07 pp+8.59 pp
APEX-AgentsAll Assigned (n=240)9.1715.4217.08+6.25 pp+7.92 pp

Grader Failure Breakdown, Regrade Sensitivity, and Blinded Human Calibration (W2 & Q2, Appendix C.4)

1. Regrading the 38 Grader Failures (0 Agent Reruns): In Figure 6b, the 38 failed tasks (34 for Flash Low, 4 for Flash High) resulted from automated judge errors (API timeouts or formatting issues) rather than agent failures, though they were conservatively scored as 0. To isolate this evaluation artifact, we re-evaluated these 38 saved outputs once using the same GPT-5.5 high judge, successfully recovering 29 valid scores (while 9 remained ungraded due to persistent API rate limits). As detailed below, we conducted a sensitivity analysis across four scoring policies to verify whether the 9 remaining ungraded tasks could alter our conclusions:

Scoring Policy (406 Tasks)Flash LowFlash HighLow − High Δ [95% CI / Bounds]
Original Figure 6b / Appendix D.162.0861.93+0.15 pp [−3.48, +3.83]
Single Regrading Pass66.0862.44+3.64 pp [+0.36, +7.04]
Complete-Case Intersection (n=397)67.3363.05+4.28 pp [+0.98, +7.56]
Deterministic [0, 100] Bounds[66.08, 68.05][62.44, 62.69][+3.39, +5.61 pp]

Once the grader timeouts are resolved, Flash Low outperforms Flash High by +3.64 pp . Crucially, even underthe most conservative worst-case bounds for the remaining 9 ungraded tasks (assigning 0 to Flash Low and 100 to Flash High), Flash Low's lower bound (66.08) still exceeds Flash High's upper bound (62.69) by +3.39 pp, confirming that our finding is robust to grader failures.

2. Blinded output-level human calibration: While Section 3.2 reviewed task and rubric design, here we directly test whether GPT-5.5 high grades actual model outputs the same way human experts do. Four domain experts independently graded 40 non-overlapping anonymized deliverables across all 8 domains (~40 expert hours, ~1 hour/task to inspect workspace files, spreadsheet formulas, and tool logs) without seeing AI scores:

Human-AI Calibration MetricScopeResult
Exact Criterion Agreement4 experts, 40 tasks, 366 pairs93.72% (κ = 0.75, AC1 = 0.92)
Overall False-Acceptance / Rejection6/366 AI-met / 17/366 AI-not-met1.64% / 4.64%

Across all 366 evaluable criterion pairs (300 both-met, 43 both-not-met, 6 AI-met/human-not-met, 17 AI-not-met/human-met), GPT-5.5 high agrees with human experts 93.72% of the time (100% on state/response checks and 93.10% on complex file deliverables), with only 1.64% false-acceptance and 4.64% slightly stricter AI rejections on wording or section completeness.

Value of Persistent Organizational Simulation Over Static Synthesis (W1 & Q3, Appendices A.8, E.6)

We combine (1) corpus lineage (84.2% inherited reviewed deliverables, 46.1% explicit continuation links, and 93.1% prior-artifact SFT tool reads / 71.3% overall), (2) our ClawGym-inspired 2K Static-SFT control (Bai et al., 2026; near-matching 2,201 vs. 2,202 trajectories from the same 34 seeds under identical DeepSeek-V4-Flash teacher, sandbox, and 1-epoch Qwen3.5-9B SFT recipe, evaluated under the Table 3 reporting protocols), and (3) 30-day GPT-5.5 mechanism ablations (Shared Comment §3: Full sustains 91.2% multi-task dependencies and 44.1% multi-round revisions vs. 5.3% / 2.6% for no-history and 0.0% / 0.0% for no-review):

Qwen3.5-9B StageGDPvalJobBench-EasyTerminal-Bench 2.1
Base65.7540.4142.31
Static-SFT (ClawGym-inspired, 2,201 traj.)67.8043.6047.70
OfficeTown SFT (Matched SFT recipe, 2,202 traj.)70.30 (+2.50)44.87 (+1.27)51.22 (+3.52)
OfficeTown SFT + OPD (additional OPD stage)73.6551.4863.64

Evaluating the same ablations inside OfficeTown Harness on Terminal-Bench 2.1 (Table 4, GPT-5.5 xhigh, 3-run means) confirms the same component ordering at inference time:

Inference Harness (GPT-5.5 xhigh)Terminal-Bench 2.1
MetaGPT81.1
Codex (Single-Agent)83.7
CAMEL85.1
OfficeTown Harness (no-history)86.5
OfficeTown Harness (no-review)87.2
OfficeTown Harness (Full)89.0†

†Note: The submitted Table 4 contained a table transcription typo (87.4 instead of 89.0); 89.0 is computed directly from the original evaluation runs with no model reruns.

Split Separation, Provenance Scope, and Rubric Selection Stability (W2 & Q4, Appendix B.5)

  • Internal split separation & shared-project scope: Across OfficeTown Bench (406 tasks, 139 projects), SFT (2,202 trajectories, 460 projects), and OPD (828 environments; 0 task-ID overlap with Bench and SFT in the released disjoint OPD manifest), exact and normalized prompt duplicates are 0. As noted in Appendix B.1, continuing organizations naturally contribute distinct tasks across splits (94 shared projects between Bench and SFT), and all post-training transfer claims in Table 3 are evaluated on independent external benchmarks.
  • External provenance check: Auditing all 188 world source registries (3,903 public standard/agency URLs) found 0 benchmark-name hits, and exact-byte checks across all 23,320 unique OfficeTown input files against GDPval (509), AutomationBench (571), and JobBench (256) found 0 matches, with all benchmark evaluations executed offline without internet access.
  • Rubric Pruning Rule & Post-Freeze Ranking Stability: To filter out trivial items while preserving evaluation rigor, we applied fixed domain quotas sorted by 15-model pilot pass counts (447 → 407 → 406 tasks, dropping 1 cross-domain item) and pruned the rubric from 9,472 to 4,836 criteria by retaining the harder tail half per task (rounding up on odd counts) while maintaining original ROC tier weights (98.0% combined P0+P1 share; all 5,142 pilot judgments on the 406 tasks were byte-verified). On the 15-model pilot panel, full and pruned Weighted Scores correlate at Spearman ρ=0.9964\rho = 0.9964 (Pass@1 shift less than 1 pp).Crucially, to verify that rubric pruning does not overfit to the pilot models, we evaluated 3 frontier models and 2 execution frameworks strictly after the 4,836-criterion mask was frozen (none of which appeared in the pilot panel). Because GPT-5.5 high grades all 9,472 draft criteria before applying the frozen mask, we can directly compare full vs. pruned scores on the exact same outputs. As expected, removing easier criteria lowers absolute Weighted Scores by 3.5 to 19.2 pp, yet induces zero rank inversions:
Post-Freeze Model / Framework (n=406)Full Draft (9,472)Released (4,836)Post-Freeze Rank
Grok-4.780.777.21st (unchanged)
Claude Opus 5.572.067.52nd (unchanged)
Gemini 3.8 Flash41.524.23rd (unchanged)
Pi Framework (Flash High)64.648.61st (unchanged)
OpenCode Framework (Flash High)59.940.72nd (unchanged)

Reconciling GPT-5.6 Sol in Appendix C.1 and Appendix D.1 (W2 & Q5)

Thank you for catching this discrepancy. The 68.5 figure in Appendix C.1 was an oversight on our part; on the final 406-task benchmark, GPT-5.6 Sol's exact Overall Weighted Score is 68.42% (reported as 68.4) with a Pass@1 of 33.5% (136/406). We have updated Appendix C.1 to match Appendix D.1 (Table 13) so that both sections now consistently report 68.4.

Per-Task Token Accounting, Terminal Budgets, and Ending-Turn Curves (W1 & Q6, Appendix C.4)

  • Exact ResearchCodeBench token accounting (summed over all harness roles and retries under GPT-5.5 xhigh): OfficeTown Harness outperforms MetaGPT by +5.25 pp while using 21.8% fewer tokens per task ((761,644 − 595,780) / 761,644), and outperforms single-agent Codex by +5.52 pp at 3.49× Codex's token usage (595,780 / 170,831):
Harness (GPT-5.5 xhigh)ResearchCodeBench (%)Mean Tokens / TaskMean Tokens (Passed)Terminal-Bench 2.1 (%)
OfficeTown Harness46.40595,780844,95589.0†
MetaGPT41.15761,6441,188,58181.1
Codex (Single-Agent)40.88170,831282,15383.7
CAMEL39.95357,029567,48785.1

†Note: The submitted Table 4 contained a table transcription typo (87.4 instead of 89.0); 89.0 is computed directly from the original evaluation runs with no model reruns.

  • Terminal-Bench 2.1 budgets (Table 4): External Terminal-Bench 2.1 baselines in Table 4 use their standard reported budget settings rather than uniform token caps, as noted in the Table 4 caption and Appendix C.4.
  • Ending-turn curves (Figure 6): Figure 6 summarizes the final scores of runs that completed by each turn over the fixed 406-task denominator; these curves do not show intermediate deliverable quality or what score would have been achieved had execution been stopped early at shorter budgets (clarified in the Figure 6 caption).
Reviewer_apEWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes OfficeTown, a simulated organizational environment in which roles, persistent state, collaboration, task dependencies, and feedback generate work trajectories. It constructs tasks and rubric items from these environments and uses the generated data for model training, reporting an improvement on a shared GDPval evaluation after training. The main insight is that persistent work environments may produce more contextually connected training experiences than isolated synthetic prompts. The evidence is promising, although it does not fully isolate the value of organizational simulation from the effects of additional training data and compute.

Critical strengths

Useful benchmark and training resources. OfficeTown Bench covers 406 tasks across eight professional domains, with expert-informed evaluation criteria and training trajectories. The reported improvements on external benchmarks also provide some evidence of the practical value of the generated data.

Critical weaknesses
  1. Limited novelty beyond existing benchmarks. Compared with existing professional-work benchmarks and simulated agent environments, OfficeTown follows a similar paradigm of generating tasks, simulating workflows, and evaluating professional deliverables. The main contribution appears to be a more elaborate engineering framework rather than a substantial methodological innovation. The distinction from closely related work is not sufficiently convincing.

  2. The experiments do not clearly isolate the benefits of persistent organization-level simulation from factors such as data quantity, task quality, trajectory length, and training compute. Without matched-budget comparisons against simpler synthetic task generation methods, it is difficult to determine whether the additional complexity of OfficeTown is actually necessary.

  3. Although the paper reports improvements on external benchmarks, it remains unclear whether these gains come from generalizable organizational experience or simply from additional task-specific training. The evaluation does not sufficiently demonstrate transfer to substantially different professional domains or establish how effectively the framework scales to larger and more complex organizations.

  4. OfficeTown combines persistent files, role-based permissions, cross-task dependencies, delegation, and review feedback, but their individual contributions are not clearly established. Without systematic ablations, it is difficult to identify which components are essential and whether a simpler framework could achieve comparable results.

Questions for authors

see weakness.

6 comments

We thank Reviewer apEW for the constructive review. Please also see our Shared Comment above; below we address your four questions and corresponding weaknesses one by one.

Scientific Contribution Beyond an Engineering Wrapper (W1 & Q1)

In real workplaces, professional assignments are embedded in ongoing projects rather than created from blank slates. Prior paradigms separate executable tool environments from evolving organizational state: workplace benchmarks hand-craft frozen standalone tasks, static synthesis generates isolated one-shot tasks without project memory, and social simulations lack executable tool sandboxes or weight updates. OfficeTown's distinguishing design is state-gated task extraction (save private draft ≠ accept/revise in review ≠ share across roles), an operational method converting evolving, permissioned organizational workflows (such as the 5-week Kestrel Rev B → Rev C CAD revision and active release hold in Figure 12, Appendix A.8) into leak-free, reproducible evaluation and post-training instances:

Research LinePrior WorkTool EnvAuto-GenTemporal StateTrains Weights
Professional BenchmarksTheAgentCompany, GDPvalYesNoNoNo
Static SynthesisAgent-World, EnvScaler, ClawGymYesYesNoYes
Generative SimulationsGenerative Agents, Agent HospitalNoYesYesNo
OfficeTown (Ours)State-Gated Org SimulationYesYesYes (84.2%)Yes

Controlling for Data Quantity, Seeds, Teacher, and SFT Recipe vs. Mediators (W2 & Q2, Appendix E.6)

To test the total effect of stateful organizational generation while controlling external confounders (data quantity, seed materials, rollout teacher, and SFT training recipe), we built a ClawGym-inspired static synthesis control (Static-SFT; Bai et al., 2026):

  • Matched seeds & near-matched data quantity: Both corpora start from the exact same 34 organizational seed bundles and match in scale (2,201 Static-SFT vs. 2,202 OfficeTown SFT trajectories).
  • Matched rollout teacher, sandbox & admission: Both collect trajectories with the same DeepSeek-V4-Flash teacher, Pi four-tool sandbox, turn/context limits (131K), and technical format admission without post-hoc score filtering.
  • Matched SFT recipe: Both fine-tune the same Qwen3.5-9B base checkpoint for 1 epoch under identical optimizer, batching, and learning-rate schedules.
  • Role of task complexity & trajectory depth: Under these fixed inputs and SFT settings, Static-SFT reflects one-shot task synthesis (0% cross-task state), whereas persistent simulation endogenously produces multi-round revisions and grounded prior-file tool interactions (93.1% of prior-artifact trajectories; 71.3% overall) as potential mediators of stronger downstream learning.

Across three representative external benchmarks spanning office deliverables (GDPval), grounded job workflows (JobBench-Easy), and terminal engineering (Terminal-Bench 2.1, completion-conditioned Table 3 protocol), OfficeTown SFT outperforms matched Static-SFT (OfficeTown SFT + OPD is listed separately as an additional post-training stage):

Qwen3.5-9B StageGDPvalJobBench-EasyTerminal-Bench 2.1
Base65.7540.4142.31
Static-SFT (ClawGym-inspired, 2,201 traj.)67.8043.6047.70
OfficeTown SFT (Matched SFT recipe, 2,202 traj.)70.30 (+2.50)44.87 (+1.27)51.22 (+3.52)
OfficeTown SFT + OPD (additional OPD stage)73.6551.4863.64

Cross-Sector Transfer, Provenance Checks, and Organization Scale (W3 & Q3, Appendices B.5, E.6)

  • One-sentence domain extensibility & cross-sector transfer: First, covering new industries is not a structural bottleneck for OfficeTown: as detailed in Section 3.1 and Appendix A.1, our framework automatically constructs a complete organization (departments, O*NET/ESCO-grounded roles, authority boundaries, and seed workspaces) in any target domain from a single natural-language sentence, without domain-specific engineering. Second, even without instantiating new industries, partitioning the 79-task GDPval set into the 34 tasks drawn from 4 sectors not represented by our 8 simulated domains (Wholesale, Government, Real Estate, Retail) versus the 45 tasks from 5 overlapping sectors shows positive pooled cross-sector transfer:
GDPval Sector Group (n=79)BaseOfficeTown SFTOfficeTown SFT + OPDGain
Unseen Sectors (4, n=34)70.9373.3377.70+6.77 pp
Overlapping Sectors (5, n=45)61.8368.0170.59+8.76 pp
  • Provenance check & team scale: Auditing all 188 simulated world registries (3,903 public standard/agency URLs) found 0 benchmark-name hits, and exact-byte comparisons of all 23,320 unique workspace input files against GDPval, JobBench, and AutomationBench found 0 matches, with all benchmark runs executed offline. Simulated organizations span 9 to 18 specialized roles (median 12)

Contributions of Persistent Memory and Review/Sharing Gates (W2 & Q4, Appendix A.8)

To examine what individual organizational mechanisms contribute in a controlled setting, we ablated cross-turn working memory (No-History, no-history) and the principal acceptance/sharing gate (No-Review, no-review) at both simulation time and inference time:

1. 30-day GPT-5.5 organizational simulation ablations (identical 10-role seed): Removing the review/sharing gate (no-review,) keeps outputs in private folders with 0 shared deliverables and 0% multi-task dependencies. Removing cross-turn memory (no-history,) causes principals to acknowledge 96.0% of reviewed submissions without requesting revisions (94.7% isolated tasks, max revision depth dropping from 4 rounds to 1):

Organizational Mechanism MetricFullNo-HistoryNo-Review
Multi-Task Dependency Share91.2%5.3%0.0%
Multi-Round Revision Rate44.1%2.6%0.0%
Prior-Task File Read Share100.0%2.2%0.2%
Substantive Review Critique Rate63.6%4.0%N/A (0 reviews)

2. Inference-harness component ablations on Terminal-Bench 2.1 (Table 4, GPT-5.5 xhigh, 3-run means): Ablating the same two mechanisms directly inside OfficeTown Harness confirms that both also improve downstream task execution:

Inference Harness (GPT-5.5 xhigh)Terminal-Bench 2.1
MetaGPT81.1
Codex (Single-Agent)83.7
CAMEL85.1
OfficeTown Harness (no-history)86.5
OfficeTown Harness (no-review)87.2
OfficeTown Harness (Full)89.0†

†Note: The submitted Table 4 contained a table transcription typo (87.4 instead of 89.0); 89.0 is computed directly from the original evaluation runs with no model reruns.

Dear Reviewer,

Thank you again for your thoughtful review and for the time you have spent evaluating our paper.

We have carefully addressed the concerns raised in your review and hope that our rebuttal has helped clarify the issues you pointed out.

As the discussion period is coming to an end, we would like to kindly ask if you have any remaining questions or if there is anything further we can clarify. We would be more than happy to provide additional explanations.

Thank you very much for your time and consideration.

Reviewer_4yMoOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper demonstrates how to generate professional work tasks for evaluating and training artificial intelligence (AI) agents. It introduces the OfficeTown framework, which simulates an organizational environment featuring professional roles, persistent workspaces, and access controls. The authors extract executable tasks from this simulation to construct the OfficeTown Bench—a benchmark dataset encompassing 23 simulated organizations across 8 domains, comprising 406 tasks and 4,836 evaluation criteria. Furthermore, the authors evaluate 13 different models, train the Qwen3.5-9B model using supervised fine-tuning (SFT) and on-policy distillation(OPD), and compare the "principal–assistant" architecture against other agent frameworks.

Critical strengths
  1. Task Generation from Ongoing Work.The paper generates tasks through organizational simulation rather than constructing isolated tasks. It includes role-based coordination, persistent organizational state, and review of deliverables. This provides a way to generate tasks with context from earlier work.

  2. Benchmark and Training Results.The authors build OfficeTown Bench with 406 tasks across eight professional domains, with feedback from 71 domain experts. Table 3 shows that training on OfficeTown data improves Qwen3.5-9B on external benchmarks. On GDPval, the rubric score increases from 65.75 to 73.65 after SFT and OPD.

Critical weaknesses
  1. Lack of ablation studies for core design components. The paper presents ongoing organizational work as its main difference from isolated task generation. However, Table 3 only compares training stages and does not show whether organizational simulation produces better training data than independent task generation. The paper also lacks ablations of persistent organizational state and review feedback. Figure 1 reports file accumulation, but does not quantify how often later tasks depend on earlier artifacts or decisions. It is unclear how much these components contribute to the reported results.

  2. Insufficient validation of automated scoring. The paper uses executable checks and model-based judgments to evaluate task outcomes (Section 4.1). Appendix C.1 states that GPT-5.5 high is used to judge the results in Table 2. However, the paper does not report agreement between the automated judge and human graders. The domain experts reviewed tasks and rubrics rather than model outputs (Section 3.2). This raises concerns about the reliability of the scores, especially for tasks requiring professional judgment.

Questions for authors
  1. Can the authors provide comparisons against static task-generation methods and ablation studies removing persistent organizational state or review feedback, under comparable data and computational budgets.

  2. Can the authors report the agreement between GPT-5.5 high and human graders on a sample of task outputs.

6 comments

We thank Reviewer 4yMo for the insightful review. Please also see our Shared Comment above; below we address each of your weaknesses and questions directly.

Cross-Task Dependency vs. File Accumulation (W1, Appendix A.8)

Unlike static synthesis pipelines that generate isolated single-shot tasks, OfficeTown simulates multi-role workflows where later assignments actively build on earlier reviewed work. For example, in a 5-week Kestrel Precision Engineering project (Figure 12, Appendix A.8), design, manufacturing, and quality inputs advance CAD Rev B to Rev C, record 7 unresolved blockers, and set an active release hold; a Week-5 snapshot includes only state visible at that assignment boundary, requiring the agent to inspect and update Rev C in place while honoring the active hold.

To test whether later tasks genuinely depend on prior work, the table below checks three concrete questions across our released benchmark and training data:

  • Inherited Reviewed Deliverables: When a new task starts, does the agent's workspace already contain deliverables created and approved in earlier tasks (rather than only initial seed templates)?
  • Explicit Continuation Links: Is the task itself recorded in the organizational ledger as a direct revision or follow-up dependency of an earlier assignment?
  • Active Prior-File Tool Calls (read / bash): When solving the task, does the agent actually run file or shell tools to open and read those earlier deliverables?
Cross-Task Continuity MetricCorpusVerified Share
Inherited Reviewed DeliverablesOfficeTown Bench (406 tasks)84.2% (342/406)
Explicit Continuation LinksOfficeTown Bench (406 tasks)46.1% (187/406)
Active Prior-File Tool Calls (read / bash)OfficeTown SFT (1,686 prior-file cases)93.1% (1,570/1,686)

Together, these three measurements form a direct chain rather than passive file accumulation: 84.2% of benchmark workspaces inherit deliverables that passed earlier principal review, 46.1% explicitly revise or follow up on an earlier request in the ledger, and 93.1% of prior-artifact SFT trajectories (71.3% of all 2,202 trajectories) invoke read or bash on those inherited files during execution. Component ablations below separately test their functional impact.

Static Synthesis Control and Architectural Ablations (W1 & Q1, Appendices A.8, E.6)

1. Matched-seed Static-SFT control: Near-matching trajectory count (2,201 vs. 2,202) and holding all 34 seed bundles, DeepSeek-V4-Flash rollout teacher, sandbox/turn caps (131K), format admission, and Qwen3.5-9B SFT recipe fixed, we trained our ClawGym-inspired static baseline (Static-SFT, Bai et al., 2026; 0% cross-task state) and evaluated it across three representative external benchmarks spanning office deliverables (GDPval), grounded job workflows (JobBench-Easy), and terminal engineering (Terminal-Bench 2.1):

Qwen3.5-9B StageGDPvalJobBench-EasyTerminal-Bench 2.1
Base65.7540.4142.31
Static-SFT (ClawGym-inspired, 2,201 traj.)67.8043.6047.70
OfficeTown SFT (Matched SFT recipe, 2,202 traj.)70.30 (+2.50)44.87 (+1.27)51.22 (+3.52)
OfficeTown SFT + OPD (additional OPD stage)73.6551.4863.64

2. 30-day simulation ablations: On an identical 10-role seed, we ran 30-day GPT-5.5 simulations comparing Full against No-History (removing cross-turn principal memory while keeping workspaces/review,) and No-Review (removing principal acceptance review and cross-role publication).

  • Multi-Task Dependency Share: Share of tasks consuming prior deliverables or continuing revisions (100% minus isolated single-shot tasks).
  • Multi-Round Revision Rate: Share of tasks undergoing ≥1 principal-requested revision loop.
  • Prior-Task File Read Share: Mean share of accessed non-seed workspace inputs created by earlier tasks.
  • Substantive Review Critique Rate: Share of review events requesting concrete fixes or missing inputs rather than immediate approval.
Organizational Mechanism MetricFullNo-HistoryNo-Review
Multi-Task Dependency Share91.2%5.3%0.0%
Multi-Round Revision Rate44.1%2.6%0.0%
Prior-Task File Read Share100.0%2.2%0.2%
Substantive Review Critique Rate63.6%4.0%N/A (0 reviews)

Without the review/sharing gate (no-review), 0 deliverables are shared across roles and all tasks remain isolated. Without cross-turn memory (no-history), principals acknowledge 96.0% of reviewed submissions without requesting revisions (94.7% isolated tasks).

3. Inference-harness ablations on Terminal-Bench 2.1 (Table 4, GPT-5.5 xhigh, 3-run means): Evaluating both ablations directly inside OfficeTown Harness confirms that both components also improve downstream execution:

Inference Harness (GPT-5.5 xhigh)Terminal-Bench 2.1
MetaGPT81.1
Codex (Single-Agent)83.7
CAMEL85.1
OfficeTown Harness (no-history)86.5
OfficeTown Harness (no-review)87.2
OfficeTown Harness (Full)89.0†

†Note: The submitted Table 4 contained a table transcription typo (87.4 instead of 89.0); 89.0 is computed directly from the original evaluation runs with no model reruns.

Human Expert Calibration (W2 & Q2, Appendix C.4)

GPT-5.5 high follows an evidence-linked verification protocol decomposing the 4,836 released criteria into 9,715 binary file, formula, code, and state sub-checks. In a 40-expert-hour blinded calibration (~1 hour per task), 4 domain experts independently evaluated 40 non-overlapping anonymized GPT-5.6 Terra High deliverables (10 tasks per expert across all 8 domains; 366 binary evaluable criterion pairs out of 371 total items, excluding 4 N/A and 1 unassessable item) blind to AI labels. Confusion-matrix counts are 300 both-met, 43 both-not-met, 6 AI-met/human-not-met (1.64% overall false-acceptance), and 17 AI-not-met/human-met (4.64% overall stricter rejection), yielding 93.72% exact agreement (Cohen's κ = 0.75, Gwet's AC1 = 0.92).

Calibration / Regrade MetricScopeValue
Blinded Human-AI Agreement4 experts, 40 tasks, 366 pairs93.72% (κ = 0.75, AC1 = 0.92)
Overall False-Acceptance / Rejection6/366 AI-met / 17/366 AI-not-met1.64% / 4.64%

Dear Reviewer,

Thank you again for your thoughtful review and for the time you have spent evaluating our paper.

We have carefully addressed the concerns raised in your review and hope that our rebuttal has helped clarify the issues you pointed out.

As the discussion period is coming to an end, we would like to kindly ask if you have any remaining questions or if there is anything further we can clarify. We would be more than happy to provide additional explanations.

Thank you very much for your time and consideration.

Comments on the paper

3 comments

We thank the Area Chair and Reviewers (4yMo, apEW, hpdu) for their constructive reviews and for recognizing OfficeTown's practical motivation, multi-domain benchmark scale, and external post-training gains. Below is an executive overview of our new controlled experiments, component ablations, human expert calibration, and coverage audits, all incorporated in blue in the revised manuscript (Figure 12 and Appendices A.8, B.5, C.4, D.1, and E.6).

1. Core Contribution Beyond Existing Benchmarks and Synthesis

Real-world professional work is not isolated single-turn Q&A: agents must use multi-step tools inside persistent workspaces shaped by multi-stage review processes, cross-role interactions, and temporal dependencies (save private draft ≠ accept/revise in review ≠ share across roles). Prior lines capture only part of this loop, whereas OfficeTown unifies all four elements:

Research ParadigmRepresentative WorkTool EnvAuto-GenTemporal StateTrains Weights
Professional BenchmarksTheAgentCompany, OSWorld, GDPvalYesNoNoNo
Static SynthesisAgent-World, EnvScaler, ClawGymYesYesNoYes
Generative SimulationsGenerative Agents, Agent HospitalNoYesYesNo
OfficeTown (Ours)State-Gated Org SimulationYesYesYes (84.2%)Yes

For example, across a 5-week Kestrel Precision Engineering project (Figure 12, Appendix A.8), mechanical, manufacturing, and quality roles introduce competing physical constraints, supersede CAD Rev B with Rev C, and place an active release hold due to missing inputs; freezing Week 5 yields a leak-free mid-project task requiring multi-step file and shell tool calls over Rev C while honoring the hold. Across our corpora, 84.2% of benchmark workspaces inherit reviewed files from earlier tasks, 46.1% directly revise or follow up on a prior work request, and 93.1% of prior-artifact SFT trajectories (71.3% of all 2,202 SFT trajectories) invoke file/shell tools to read those inherited deliverables during execution.

2. Controlling for Seeds, Data Quantity, Teacher, and SFT Recipe: Static-SFT vs. OfficeTown SFT (Appendix E.6)

To test whether external gains come from stateful organizational simulation rather than raw data volume, teacher capability, or SFT recipe, we followed workspace-grounded static synthesis (ClawGym, Bai et al., 2026, arXiv:2604.26904) to build a ClawGym-inspired static control (Static-SFT: 2,201 standalone tasks from the exact same 34 organizational seeds with 0% cross-task state). We near-matched trajectory count (2,201 vs. 2,202) and held seed documents, rollout teacher (DeepSeek-V4-Flash), Pi four-tool sandbox and turn/context limits (131K), technical format admission, and Qwen3.5-9B SFT hyperparameters fixed; richer multi-step revisions and trajectory depth thus arise naturally from stateful simulation rather than external confounders. Across three representative external benchmarks spanning multi-sector office deliverables (GDPval), grounded job workflows (JobBench-Easy), and terminal engineering (Terminal-Bench 2.1, completion-conditioned Table 3 protocol), OfficeTown SFT outperforms matched Static-SFT on all three evaluations (with OfficeTown SFT + OPD shown below as an additional post-training stage):

Qwen3.5-9B StageGDPvalJobBench-EasyTerminal-Bench 2.1
Base65.7540.4142.31
Static-SFT (ClawGym-inspired, 2,201 traj.)67.8043.6047.70
OfficeTown SFT (Matched SFT recipe, 2,202 traj.)70.30 (+2.50)44.87 (+1.27)51.22 (+3.52)
OfficeTown SFT + OPD (additional OPD stage)73.6551.4863.64

(Parentheses on OfficeTown SFT show direct percentage-point gains over matched Static-SFT.)

3. Contributions of Persistent Memory and Principal Review/Sharing Gates (Appendix A.8)

To examine how each organizational component shapes task structure, we ran controlled 30-day GPT-5.5 simulations on an identical 10-role design organization seed under three variants: Full (complete architecture,), No-History (no-history: removing cross-turn principal memory while keeping workspaces and review,), and No-Review/Sharing Gate (no-review: removing principal acceptance review and cross-role publication so outputs remain saved in private folders,). Four ledger metrics quantify interdependent work: Multi-Task Dependency Share (tasks consuming prior deliverables or continuing revisions; 100% minus isolated single-shot tasks), Multi-Round Revision Rate (tasks with ≥1 principal revision loop), Prior-Task File Read Share (mean share of non-seed workspace inputs created by prior tasks), and Substantive Review Critique Rate (share of review events requesting revisions or missing inputs vs. immediate acknowledgment):

Organizational Mechanism MetricFullNo-HistoryNo-Review
Multi-Task Dependency Share91.2%5.3%0.0%
Multi-Round Revision Rate44.1%2.6%0.0%
Prior-Task File Read Share100.0%2.2%0.2%
Substantive Review Critique Rate63.6%4.0%N/A (0 reviews)

Without the review/sharing gate (no-review), 0 deliverables are shared across roles and all 49 assignments remain isolated single-shot tasks (with only 0.2% incidental local path reads). Without cross-turn memory (no-history), principals lose prior constraint context and acknowledge 96.0% (24/25) of reviewed submissions without requesting revisions (94.7% isolated tasks, max revision depth 4 → 1). Testing the same two ablations directly inside OfficeTown Harness on Terminal-Bench 2.1 (GPT-5.5 xhigh, 3-run means matching Table 4) confirms that both modules also improve downstream execution:

Inference Harness (GPT-5.5 xhigh)Terminal-Bench 2.1
MetaGPT81.1
Codex (Single-Agent)83.7
CAMEL85.1
OfficeTown Harness (no-history)86.5
OfficeTown Harness (no-review)87.2
OfficeTown Harness (Full)89.0†

†Note: The submitted Table 4 contained a table transcription typo (87.4 instead of 89.0); 89.0 is computed directly from the original evaluation runs with no model reruns.

4. Transfer Scope, Unseen Sectors, and Automated Judge Reliability (Appendices B.5, C.4, E.6)

  • Same-task (Common) and zero-imputed (All Assigned) transfer: OfficeTown Bench (406 tasks) already scores all timeouts/failures as 0 (1.48 → 3.45 → 4.68). Across GDPval, JobBench-Easy, and APEX-Agents, evaluating both identical jointly scored task intersections (Common) and complete assigned workloads after recovering historical infrastructure interruptions with any remaining timeouts/unresolved cases scored as 0 (All Assigned) confirms consistent positive gains:
BenchmarkScopeBaseSFTSFT + OPDGain over Base
GDPvalCommon (n=79)65.7570.3073.65+7.90 pp
GDPvalAll Assigned (n=140)60.3764.6869.71+9.34 pp
JobBench-EasyCommon (n=43,)42.2845.6753.35+11.07 pp
JobBench-EasyAll Assigned (n=63)39.3444.3750.72+11.38 pp
APEX-AgentsCommon (n=198)11.1118.1819.70+8.59 pp
APEX-AgentsAll Assigned (n=240)9.1715.4217.08+7.92 pp
  • One-sentence domain extensibility & zero-shot unseen-sector transfer: By design, OfficeTown instantiates a complete, role-permissioned organization in any new domain from a single natural-language sentence (Section 3.1, Appendix A.1) without domain-specific engineering. Moreover, even zero-shot without generating new organizations, partitioning the 79-task GDPval set into the 34 tasks from 4 sectors not represented by our 8 simulated domains (Wholesale, Government, Real Estate, Retail) versus the 45 tasks from 5 overlapping sectors shows positive cross-sector transfer:
GDPval Sector Group (n=79)BaseOfficeTown SFTOfficeTown SFT + OPDGain [95% CI]
Unseen Sectors (4, n=34)70.9373.3377.70+6.77 pp [+0.52, +13.68]
Overlapping Sectors (5, n=45)61.8368.0170.59+8.76 pp [+3.56, +14.04]
  • Blinded human calibration & rubric stability: In a 40-expert-hour blinded calibration (~1 hour per task to audit multi-file workspace deliverables, spreadsheet formulas, and tool logs), 4 domain experts independently graded 40 non-overlapping anonymized tasks (10 tasks per expert across all 8 domains; 366 binary evaluable criterion pairs out of 371 total items) without access to AI scores. Experts confirmed that item-level criteria map cleanly to concrete file/cell evidence, with 93.72% exact agreement (343/366; 1.64% [6/366] false-acceptance and 4.64% [17/366] stricter AI rejection, summing to 100.00%):
Verification DimensionScopeKey Metric
Human-AI Exact Agreement4 experts, 40 tasks, 366 pairs93.72% (κ = 0.75, AC1 = 0.92)
Overall False-Acceptance / Rejection6/366 AI-met / 17/366 AI-not-met1.64% / 4.64%

We sincerely thank Reviewer hpdu for the careful re-evaluation and for confirming that all concerns have been addressed.

Warm regards,

Submission 30948 Authors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.