- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Graph World Models (GWMs), an external graph memory built from enterprise-agent interaction histories. It captures workflow states, transitions, outcomes, and supporting examples, then uses this evidence to guide a frozen policy or rank candidate actions without retraining. Evaluations on CRMArena-Pro and EnterpriseOps-Gym show improved success rates, including gains of 12.33 percentage points for Gemma and 6.33 points for Qwen on CRM. The paper also explores cross-model reuse, shows that larger graphs can improve performance, and finds that candidate availability is a bigger source of failure than misranking. However, graph-free methods such as majority voting also help, making it important to separate the gains from graph memory itself from those of multi-candidate selection. The authors additionally report the inference overhead of GWM guidance.
Critical strengths
-
The paper separates failures caused by information loss during abstraction, missing successful candidates, and incorrect ranking. The results support this distinction: at K = 10, 114 of 156 replay failures stem from having no successful candidate, while only 42 are due to misranking. This provides a more useful analysis than success rates alone.
-
The authors compare against random and majority selection using the same candidate pools, run fixed-candidate replays, report paired rescues and regressions, and include task-clustered confidence intervals and inference costs. They also openly acknowledge limitations, including ambiguous action labels and unadjusted confidence intervals.
Critical weaknesses
-
The reported gains combine candidate generation, advice, and selection, while graph-free baselines recover much of the improvement. At K = 10, majority voting improves CRM success by 2.67 pp, and on the validation split, graph-free selection slightly outperforms GWM (48.86% vs. 48.04%). Fixed-candidate replays also show negligible gains from graph evidence for Qwen. These results support the overall harness but do not establish that workflow-graph memory is the main source of improvement.
-
The evaluation uses relatively small cohorts (100 CRM and 80 EOPS tasks), with some comparisons relying on earlier runs and unadjusted confidence intervals. Several gains are marginal or vary across configurations. The larger Gemma improvement (+12.33 pp) comes from a separate run, making direct comparison with earlier results difficult. Cross-benchmark reuse is promising, but in-domain graphs perform better in most matched comparisons, and the discovery-fraction results are non-monotonic. Overall, the findings are encouraging, but stronger matched experiments and statistical controls are needed to establish the robustness and source of the gains.
Questions for authors
-
Fixed-candidate replay shows almost no benefit from graph evidence for Qwen, while graph-free selection matches GWM on the validation split. Can you provide an ablation that holds candidate generation and advice constant while removing only graph retrieval, with adjusted confidence intervals? This would help establish the graph’s contribution beyond candidate selection.
-
The GWM-S/L results come from a separate run with active guidance, whereas the earlier Table 8 results use graph fallbacks. Could you report matched comparisons using the same tasks, candidates, and selector to isolate the effects of graph size, source policy, and active selection? Please also clarify in the main text which Gemma results use fallback versus active guidance.
-
Appendix E.4 shows that shifting action labels changes 39–50% of edge labels and substantially alters successor distributions, yet both Qwen graph constructions achieve 52.33% online success. Which convention is used for the headline results, and how should the graph be interpreted as an action-conditioned workflow model when performance appears insensitive to this choice?
-
Experiment A takes roughly 11.6× the baseline wall time, yielding about 4.6 successful executions per run-hour versus 46.8 for the baseline. Could you compare GWM against alternatives under the same inference budget, such as generating more candidates, using a simpler selector, or making fewer guidance calls? Evaluation on larger task sets and additional benchmarks would also strengthen the generality claim.
7 comments
Authors Response to Reviewer 593s (Part 2)
593s — W2.B — Benchmark realism and evaluation workload
The evaluation uses relatively small cohorts (100 CRM and 80 EOPS tasks),
Thank you for highlighting the need for stronger matched experiments and statistical controls. We would also like to put the cohort sizes in context: the 100 CRM and 80 EOPS tasks simulate practical enterprise workflows, and evaluating them requires substantial agent interaction, candidate generation, and outcome verification.
The CRM cohort spans 22 recorded workflow categories, including case routing, lead qualification, quote approval, sales analysis, configuration checking, and confidentiality. EOPS samples ten tasks from each of eight enterprise domains, covering Calendar, customer service management, Drive, Email, HR, IT service management, Teams, and Hybrid workflows (Sections 1 and 5; Appendix G, Table 26).
These environments reproduce practical requirements of enterprise work: querying organizational records, resolving dependencies, supplying correct tool arguments, and verifying outcomes. For example, the recorded Drive task requires recovering from a shared-drive permission error: the baseline leaves access ungranted and satisfies four of five checks, whereas the corrected execution satisfies all five (Appendix I.2). EOPS evaluates the resulting operational state through reported SQL checks.
The recorded interactions show the work involved. Appendix F.5 reports 1,948 EOPS decision steps across 240 task executions, averaging 8.12 steps per execution. In CRM, 100 tasks evaluated three times with eight complete candidates entail 2,400 executed candidate rollouts per configuration (Section 5; Tables 2 and 16). Historical CRM experiment A at averages 11.42 hours per 100-task run, including candidate generation and guidance (Appendix E.7, Table 15).
The post-submission Qwen3.6-27B experiment underlying G3 (matched EOPS selection) further quantifies this workload. Its 80 tasks, three repeats, and eight conditions produced 1,920 task executions, 17,939 logged tool calls, and 116,807 server-recorded model requests. Within the graph-informed condition, the 240 executions averaged 9.11 tool calls and 5.73 minutes per execution, with a 95th-percentile duration of 15.13 minutes.
This G3 Qwen experiment used three nodes with eight AMD MI300X GPUs each, consuming 55.64 allocated GPU-hours, equivalent to 6.96 eight-GPU node-hours of allocated capacity. This accounting includes model loading, service preparation, and idle capacity. These hardware and workload measurements are additional post-submission evidence.
The 100/80-task cohorts cover diverse enterprise workflows with operational dependencies and verifiable outcomes, and their evaluation entails thousands of trajectories and many thousands of tool interactions. This substantial workload establishes the practical scope of the evaluation; the matched experiments and statistical controls address the robustness and source of the gains.
Authors Response to Reviewer 593s (Part 3)
593s — Q1 — Retrieval-only ablation with adjusted intervals**
Can you provide an ablation that holds candidate generation and advice constant while removing only graph retrieval, with adjusted confidence intervals? This would help establish the graph’s contribution beyond candidate selection.
Table 20 in the submitted manuscript already contains the requested fixed-candidate ablation. We provide retrospective adjusted intervals and an additional post-submission Gemma pool replay.
In experiments 1–3 below, we remove the retrieved historical-evidence block while preserving task and candidate information and the scoring procedure. Within each experiment, evidence conditions share the same saved candidates, judge, decoding settings, candidate order, and tie rule. Reusing candidates also preserves each pool’s generation-time advice exposure. These controls apply separately within each replay; models or task populations differ across experiments.
1. Historical Qwen replay—Submitted manuscript, Table 20. We replayed 1,479 CRM task pools with Qwen3.6-27B to measure the contribution of graph evidence during ranking. The graph-minus-no-evidence difference is −0.07 pp [−1.90, +1.77].
2. Historical Gemma replay—Submitted manuscript, Table 20. We tested the same intervention using Gemma on a 600-task subset. The difference is +1.50 pp [−0.33, +3.33], recording a positive observed increment with uncertainty spanning zero.
3. Additional Gemma pool replay—post-submission. We conducted this replay to test retrieval on candidates generated by the evaluated Gemma actor. Gemma acts and judges with ; the source runs cross advice on/off with selection-time graph evidence on/off, with three repeats per configuration. Each saved pool is then rescored under matched evidence conditions. On the primary exact-match/privacy-rejection subset—180 tasks and 2,160 pools—full graph evidence gives +0.51 pp [−0.79, +1.81] over no historical evidence. This supplementary replay was also discussed in our earlier bJA4 Q1 response and is separate from G3 and G4.
Adjusted uncertainty. Table 20 originally reported unadjusted intervals. The intervals above are retrospective, approximate simultaneous 95% intervals, computed using 20,000 joint task-cluster bootstrap resamples and maximum absolute centered errors. Adjustment covers the 12 Table 20 contrasts and, separately, the five pool contrasts, with simultaneous coverage within each family. The interval calculation required no new model calls.
4. Complementary offline ranking—vfr3 Q1.A. We conducted an exploratory post-submission experiment to test whether learned workflow outcomes improve ranking without an LLM judge. An outcome-conditioned transition/tool-error selector ranks completed trajectories, with settings chosen on 744 development tasks and evaluated on 735 held-out CRM pools. Its one-state and retrieval controls do not establish an advantage for graph structure or latent states. However, successful selection reaches 34.56% versus 29.93% for the fewest-tool-errors heuristic: +4.63 pp [pointwise 95% CI: +2.31, +6.94], with Holm-adjusted p=0.0006 across eight contrasts.
Overall, these studies quantify the incremental effect of graph retrieval and demonstrate that learned workflow outcome information can improve offline candidate selection over a simple tool-error heuristic on held-out CRM pools, without relying on an LLM judge.
Authors Response to Reviewer 593s (Part 1)
Thank you,
593s, for your time of reviewing our work and engaging so thoughtfully with our analysis of abstraction loss, missing successful candidates, and ranking errors. Your acknowledgment of the matched controls and candid reporting of uncertainty, costs, and limitations is especially encouraging.593s —W1 — Graph-free explanations of the gains
Thank you for distinguishing the effectiveness. The results support describing the headline improvement as a complete-harness result. We also noted in Section 8 (Attributing improvements, Line 469-470) of the initial reviewed manuscript:"Online gains concern combined memory, candidate generation, advice, and selection with fixed policy weights. The contribution is the GWM harness...". Our additional studies address this distinction by separating four questions: whether the complete method improves performance beyond the original evaluation, whether stored workflow experience contains useful predictive information, whether supplying workflow evidence improves decisions under matched controls, and how the method affects inference and execution costs.
G1 study addresses the reliability and scope of the complete-system result. Full GWM improves family-weighted success from 34.76% for single generation to 40.71%, a gain of +5.95 pp with an adjusted interval of [+3.65, +8.27]. This strengthens the evidence that the combined configuration remains useful under family exclusion within CRM.
G2 study addresses whether stored experience provides useful information independently of the LLM selection process. GWM predicts externally recorded tool errors on held-out EOPS tasks with an AUROC of 0.9174, exceeding raw retrieval at 0.8142 and all eight alternatives in the exploratory expanded comparison after multiplicity adjustment. Because this evaluation does not use an LLM judge to select candidates, it demonstrates predictive usefulness outside the selection harness. The original per-tool control reaches 0.9187, which shows that the result supports the value of experience-derived outcome information.
G3 and G4 studies address your attribution concern more directly through matched online comparisons. G3 study holds the actor, judge, candidate-generation settings, candidate count, and advice setting constant while changing the workflow evidence supplied to the judge. This tests whether evidence adds value to a procedure that already includes multiple candidates and LLM selection. The Gemma EOPS comparison yields an observed +3.33 pp improvement under both success definitions. G4 provides analogous graph-informed and graph-free next-message comparisons in CRM. These controls make the incremental contribution assessable.
G4 and G5 studies also examine practical costs that success rates alone do not capture. In G4, selecting a next message before execution reduces executed tool calls by approximately 88% relative to executing complete candidate episodes. This is a benefit of the execution strategy, which graph-free selection can also use. G5’s original budget-controlled comparison achieves 31.0% lower inference-token use within the declared 3 pp quality margin under its primary analysis.
Together, these studies strengthen the evidence for GWM as an integrated harness and for the predictive value of stored workflow experience.
Please also see our responses to bJA4-W1, bJA4-Q1.B, and vfr3-Q1.
593s —W2.A — Robustness and contemporaneous Gemma baselines
G1 evaluated 600 CRM tasks across all 22 task families after fixing the configuration, using family-excluded memories and fresh matched runs. GWM improves family-weighted success over single generation by +5.95 pp (adjusted interval [+3.65, +8.27]); whole-family sensitivity also remains positive [+2.00, +10.41].
Separately from the numbered G studies, we conducted post-submission Gemma controls to address stored-baseline comparability. We freshly evaluated single generation, graph-free selection, and GWM-L on the original 100-task CRM cohort and a 204-task validation cohort, with three repeats per condition and K=8 for both selectors. GWM-L gains over single generation are +14.67 pp [+9.33, +20.33] and +7.35 pp [+3.92, +10.78], respectively; differences from graph-free selection are 0.00 pp [$-4.00,+3.67]and+0.16pp[-$2.45, +2.78]. These are pointwise paired 95% task-bootstrap intervals; the comparisons are distinct from the original +12.33 pp stored-baseline result.
The fresh controls reproduce GWM-L’s gains over single generation on both the original 100-task cohort (+14.67 pp) and the separate 204-task validation cohort (+7.35 pp). This directly addresses the stored-baseline comparability concern and strengthens the evidence for repeatable complete-system improvements within CRM.
Please see bJA4-W2, vfr3-W2, vfr3-Q2.