acceptodds
Under review as a conference paper at ICLR 2027

Latent Workflow Discovery for Enterprise World Models: Grounding General-Purpose Agents in Organizational Context

Abstract

Enterprise agents must connect user goals to organization-specific records, tool dependencies, and workflow prerequisites. We formulate experience reuse as latent workflow discovery and propose Graph World Models (GWMs), combining a workflow graph, language model, and software harness. GWM builds a world model for a target enterprise from interaction histories, storing inferred workflow states, observed successors, outcomes, support counts, and examples. The harness retrieves evidence for model calls that advise the policy or select its candidates. We evaluate within-benchmark, cross-benchmark, and cross-model reuse on customer-relationship-management tasks in CRMArena-Pro and enterprise-operation tasks in EnterpriseOps-Gym. By combining graph-structured workflow memory, advice, and candidate selection, GWM improves CRM task success over direct generation for both Qwen and Gemma without policy retraining. Qwen-derived experience supports same-model and cross-model guidance, and the large-graph configuration further improves Gemma success over the small-graph configuration. Controlled comparisons characterize the incremental effects of historical evidence and candidate selection. Our benchmark-agnostic analysis separates abstraction loss, candidate availability, and selection error; in historical CRM pools, missing successful candidates account for more failures than misranking available ones. We release GWM as a vLLM extension with loadable workflow-graph adapters, supporting graph construction and online/offline inference at https://github.com/iclr-gwm/gwm

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Enterprise agents, Graph World Models, latent workflow discovery, workflow memory, organizational context, inference-time guidance, candidate selection, cross-model reuse

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_593sOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Graph World Models (GWMs), an external graph memory built from enterprise-agent interaction histories. It captures workflow states, transitions, outcomes, and supporting examples, then uses this evidence to guide a frozen policy or rank candidate actions without retraining. Evaluations on CRMArena-Pro and EnterpriseOps-Gym show improved success rates, including gains of 12.33 percentage points for Gemma and 6.33 points for Qwen on CRM. The paper also explores cross-model reuse, shows that larger graphs can improve performance, and finds that candidate availability is a bigger source of failure than misranking. However, graph-free methods such as majority voting also help, making it important to separate the gains from graph memory itself from those of multi-candidate selection. The authors additionally report the inference overhead of GWM guidance.

Critical strengths
  1. The paper separates failures caused by information loss during abstraction, missing successful candidates, and incorrect ranking. The results support this distinction: at K = 10, 114 of 156 replay failures stem from having no successful candidate, while only 42 are due to misranking. This provides a more useful analysis than success rates alone.

  2. The authors compare against random and majority selection using the same candidate pools, run fixed-candidate replays, report paired rescues and regressions, and include task-clustered confidence intervals and inference costs. They also openly acknowledge limitations, including ambiguous action labels and unadjusted confidence intervals.

Critical weaknesses
  1. The reported gains combine candidate generation, advice, and selection, while graph-free baselines recover much of the improvement. At K = 10, majority voting improves CRM success by 2.67 pp, and on the validation split, graph-free selection slightly outperforms GWM (48.86% vs. 48.04%). Fixed-candidate replays also show negligible gains from graph evidence for Qwen. These results support the overall harness but do not establish that workflow-graph memory is the main source of improvement.

  2. The evaluation uses relatively small cohorts (100 CRM and 80 EOPS tasks), with some comparisons relying on earlier runs and unadjusted confidence intervals. Several gains are marginal or vary across configurations. The larger Gemma improvement (+12.33 pp) comes from a separate run, making direct comparison with earlier results difficult. Cross-benchmark reuse is promising, but in-domain graphs perform better in most matched comparisons, and the discovery-fraction results are non-monotonic. Overall, the findings are encouraging, but stronger matched experiments and statistical controls are needed to establish the robustness and source of the gains.

Questions for authors
  1. Fixed-candidate replay shows almost no benefit from graph evidence for Qwen, while graph-free selection matches GWM on the validation split. Can you provide an ablation that holds candidate generation and advice constant while removing only graph retrieval, with adjusted confidence intervals? This would help establish the graph’s contribution beyond candidate selection.

  2. The GWM-S/L results come from a separate run with active guidance, whereas the earlier Table 8 results use graph fallbacks. Could you report matched comparisons using the same tasks, candidates, and selector to isolate the effects of graph size, source policy, and active selection? Please also clarify in the main text which Gemma results use fallback versus active guidance.

  3. Appendix E.4 shows that shifting action labels changes 39–50% of edge labels and substantially alters successor distributions, yet both Qwen graph constructions achieve 52.33% online success. Which convention is used for the headline results, and how should the graph be interpreted as an action-conditioned workflow model when performance appears insensitive to this choice?

  4. Experiment A takes roughly 11.6× the baseline wall time, yielding about 4.6 successful executions per run-hour versus 46.8 for the baseline. Could you compare GWM against alternatives under the same inference budget, such as generating more candidates, using a simpler selector, or making fewer guidance calls? Evaluation on larger task sets and additional benchmarks would also strengthen the generality claim.

7 comments
AuthorsOpenReview ↗

Authors Response to Reviewer 593s (Part 1)

Thank you, 593s, for your time of reviewing our work and engaging so thoughtfully with our analysis of abstraction loss, missing successful candidates, and ranking errors. Your acknowledgment of the matched controls and candid reporting of uncertainty, costs, and limitations is especially encouraging.

 

593s —W1 — Graph-free explanations of the gains

Thank you for distinguishing the effectiveness. The results support describing the headline improvement as a complete-harness result. We also noted in Section 8 (Attributing improvements, Line 469-470) of the initial reviewed manuscript:"Online gains concern combined memory, candidate generation, advice, and selection with fixed policy weights. The contribution is the GWM harness...". Our additional studies address this distinction by separating four questions: whether the complete method improves performance beyond the original evaluation, whether stored workflow experience contains useful predictive information, whether supplying workflow evidence improves decisions under matched controls, and how the method affects inference and execution costs.

G1 study addresses the reliability and scope of the complete-system result. Full GWM improves family-weighted success from 34.76% for single generation to 40.71%, a gain of +5.95 pp with an adjusted interval of [+3.65, +8.27]. This strengthens the evidence that the combined configuration remains useful under family exclusion within CRM.

G2 study addresses whether stored experience provides useful information independently of the LLM selection process. GWM predicts externally recorded tool errors on held-out EOPS tasks with an AUROC of 0.9174, exceeding raw retrieval at 0.8142 and all eight alternatives in the exploratory expanded comparison after multiplicity adjustment. Because this evaluation does not use an LLM judge to select candidates, it demonstrates predictive usefulness outside the selection harness. The original per-tool control reaches 0.9187, which shows that the result supports the value of experience-derived outcome information.

G3 and G4 studies address your attribution concern more directly through matched online comparisons. G3 study holds the actor, judge, candidate-generation settings, candidate count, and advice setting constant while changing the workflow evidence supplied to the judge. This tests whether evidence adds value to a procedure that already includes multiple candidates and LLM selection. The Gemma EOPS comparison yields an observed +3.33 pp improvement under both success definitions. G4 provides analogous graph-informed and graph-free next-message comparisons in CRM. These controls make the incremental contribution assessable.

G4 and G5 studies also examine practical costs that success rates alone do not capture. In G4, selecting a next message before execution reduces executed tool calls by approximately 88% relative to executing complete candidate episodes. This is a benefit of the execution strategy, which graph-free selection can also use. G5’s original budget-controlled comparison achieves 31.0% lower inference-token use within the declared 3 pp quality margin under its primary analysis.

Together, these studies strengthen the evidence for GWM as an integrated harness and for the predictive value of stored workflow experience.

Please also see our responses to bJA4-W1, bJA4-Q1.B, and vfr3-Q1.

 

593s —W2.A — Robustness and contemporaneous Gemma baselines

G1 evaluated 600 CRM tasks across all 22 task families after fixing the configuration, using family-excluded memories and fresh matched runs. GWM improves family-weighted success over single generation by +5.95 pp (adjusted interval [+3.65, +8.27]); whole-family sensitivity also remains positive [+2.00, +10.41].

Separately from the numbered G studies, we conducted post-submission Gemma controls to address stored-baseline comparability. We freshly evaluated single generation, graph-free selection, and GWM-L on the original 100-task CRM cohort and a 204-task validation cohort, with three repeats per condition and K=8 for both selectors. GWM-L gains over single generation are +14.67 pp [+9.33, +20.33] and +7.35 pp [+3.92, +10.78], respectively; differences from graph-free selection are 0.00 pp [$-4.00,+3.67]and+0.16pp[4.00, +3.67] and +0.16 pp [-$2.45, +2.78]. These are pointwise paired 95% task-bootstrap intervals; the comparisons are distinct from the original +12.33 pp stored-baseline result.

The fresh controls reproduce GWM-L’s gains over single generation on both the original 100-task cohort (+14.67 pp) and the separate 204-task validation cohort (+7.35 pp). This directly addresses the stored-baseline comparability concern and strengthens the evidence for repeatable complete-system improvements within CRM.

Please see bJA4-W2, vfr3-W2, vfr3-Q2.

AuthorsOpenReview ↗

Authors Response to Reviewer 593s (Part 2)

593s — W2.B — Benchmark realism and evaluation workload

The evaluation uses relatively small cohorts (100 CRM and 80 EOPS tasks),

Thank you for highlighting the need for stronger matched experiments and statistical controls. We would also like to put the cohort sizes in context: the 100 CRM and 80 EOPS tasks simulate practical enterprise workflows, and evaluating them requires substantial agent interaction, candidate generation, and outcome verification.

The CRM cohort spans 22 recorded workflow categories, including case routing, lead qualification, quote approval, sales analysis, configuration checking, and confidentiality. EOPS samples ten tasks from each of eight enterprise domains, covering Calendar, customer service management, Drive, Email, HR, IT service management, Teams, and Hybrid workflows (Sections 1 and 5; Appendix G, Table 26).

These environments reproduce practical requirements of enterprise work: querying organizational records, resolving dependencies, supplying correct tool arguments, and verifying outcomes. For example, the recorded Drive task requires recovering from a shared-drive permission error: the baseline leaves access ungranted and satisfies four of five checks, whereas the corrected execution satisfies all five (Appendix I.2). EOPS evaluates the resulting operational state through reported SQL checks.

The recorded interactions show the work involved. Appendix F.5 reports 1,948 EOPS decision steps across 240 task executions, averaging 8.12 steps per execution. In CRM, 100 tasks evaluated three times with eight complete candidates entail 2,400 executed candidate rollouts per configuration (Section 5; Tables 2 and 16). Historical CRM experiment A at K=10K=10 averages 11.42 hours per 100-task run, including candidate generation and guidance (Appendix E.7, Table 15).

The post-submission Qwen3.6-27B experiment underlying G3 (matched EOPS selection) further quantifies this workload. Its 80 tasks, three repeats, and eight conditions produced 1,920 task executions, 17,939 logged tool calls, and 116,807 server-recorded model requests. Within the graph-informed condition, the 240 executions averaged 9.11 tool calls and 5.73 minutes per execution, with a 95th-percentile duration of 15.13 minutes.

This G3 Qwen experiment used three nodes with eight AMD MI300X GPUs each, consuming 55.64 allocated GPU-hours, equivalent to 6.96 eight-GPU node-hours of allocated capacity. This accounting includes model loading, service preparation, and idle capacity. These hardware and workload measurements are additional post-submission evidence.

The 100/80-task cohorts cover diverse enterprise workflows with operational dependencies and verifiable outcomes, and their evaluation entails thousands of trajectories and many thousands of tool interactions. This substantial workload establishes the practical scope of the evaluation; the matched experiments and statistical controls address the robustness and source of the gains.

AuthorsOpenReview ↗

Authors Response to Reviewer 593s (Part 3)

593s — Q1 — Retrieval-only ablation with adjusted intervals**

Can you provide an ablation that holds candidate generation and advice constant while removing only graph retrieval, with adjusted confidence intervals? This would help establish the graph’s contribution beyond candidate selection.

Table 20 in the submitted manuscript already contains the requested fixed-candidate ablation. We provide retrospective adjusted intervals and an additional post-submission Gemma pool replay.

In experiments 1–3 below, we remove the retrieved historical-evidence block while preserving task and candidate information and the scoring procedure. Within each experiment, evidence conditions share the same saved candidates, judge, decoding settings, candidate order, and tie rule. Reusing candidates also preserves each pool’s generation-time advice exposure. These controls apply separately within each replay; models or task populations differ across experiments.

1. Historical Qwen replay—Submitted manuscript, Table 20. We replayed 1,479 CRM task pools with Qwen3.6-27B to measure the contribution of graph evidence during ranking. The graph-minus-no-evidence difference is −0.07 pp [−1.90, +1.77].

2. Historical Gemma replay—Submitted manuscript, Table 20. We tested the same intervention using Gemma on a 600-task subset. The difference is +1.50 pp [−0.33, +3.33], recording a positive observed increment with uncertainty spanning zero.

3. Additional Gemma pool replay—post-submission. We conducted this replay to test retrieval on candidates generated by the evaluated Gemma actor. Gemma acts and judges with K=8K=8; the source runs cross advice on/off with selection-time graph evidence on/off, with three repeats per configuration. Each saved pool is then rescored under matched evidence conditions. On the primary exact-match/privacy-rejection subset—180 tasks and 2,160 pools—full graph evidence gives +0.51 pp [−0.79, +1.81] over no historical evidence. This supplementary replay was also discussed in our earlier bJA4 Q1 response and is separate from G3 and G4.

Adjusted uncertainty. Table 20 originally reported unadjusted intervals. The intervals above are retrospective, approximate simultaneous 95% intervals, computed using 20,000 joint task-cluster bootstrap resamples and maximum absolute centered errors. Adjustment covers the 12 Table 20 contrasts and, separately, the five pool contrasts, with simultaneous coverage within each family. The interval calculation required no new model calls.

4. Complementary offline ranking—vfr3 Q1.A. We conducted an exploratory post-submission experiment to test whether learned workflow outcomes improve ranking without an LLM judge. An outcome-conditioned transition/tool-error selector ranks completed trajectories, with settings chosen on 744 development tasks and evaluated on 735 held-out CRM pools. Its one-state and retrieval controls do not establish an advantage for graph structure or latent states. However, successful selection reaches 34.56% versus 29.93% for the fewest-tool-errors heuristic: +4.63 pp [pointwise 95% CI: +2.31, +6.94], with Holm-adjusted p=0.0006 across eight contrasts.

Overall, these studies quantify the incremental effect of graph retrieval and demonstrate that learned workflow outcome information can improve offline candidate selection over a simple tool-error heuristic on held-out CRM pools, without relying on an LLM judge.

Reviewer_vfr3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Graph World Models (GWMs) for enterprise tool-using agents. Offline, it embeds interaction-history prefixes from past rollouts, clusters them into workflow states, and stores observed successors, toolset and outcome statistics, and examples in a graph. At inference, the current history is matched to a state. The retrieved evidence can trigger advice to the acting policy, and an LLM selector uses it to pick one of K candidates: complete executed rollouts on CRMArena-Pro, or next messages on EnterpriseOps-Gym (EOPS). Policy weights stay fixed. With Qwen3.6-27B and Gemma-4-26B, the complete configurations improve success over single generation by 6.33 to 12.33 points on 100 CRM tasks and by 5.42 points on 80 EOPS tasks. The paper also reports cross-benchmark and cross-model reuse, fixed-candidate replays, and a decomposition of selection failures into missing and misranked candidates.

Critical strengths
  1. Thorough controls, reported candidly. The paper runs the experiments needed to test its own claims: fixed-candidate replays, graph-free selection, matched random and majority controls, a serialization control, rescue and regression counts, and cost accounting. It also reports the results that go against it. The coverage decomposition (Eq. 10) is a useful diagnostic for best-of-K agents: at K = 10, 114 of 156 replay failures have no successful candidate in the pool (App. E.3).

  2. A relevant setting and released artifacts. Reusing enterprise interaction logs without retraining is a practical problem, the two benchmarks are realistic, and the vLLM extension and all trajectories are released.

Critical weaknesses
  1. The paper's own controls show that the workflow graph adds little; the gains come from best-of-K selection. On the 204-task validation split, GWM advice and selection reach 48.04% and graph-free selection with the same K reaches 48.86%, against 41.67% for single generation (Table 19). With candidates held fixed, graph evidence changes selection success by −0.07 points for Qwen and +1.50 for Gemma (Table 20), and by 0.00 for both in a second replay (Table 23). Relabeling the transitions changes 945 of 964 state–toolset keys yet leaves online success unchanged (App. F.5). As a world model, the graph predicts the next toolset worse than simply repeating the current action (38.92% vs. 64.44% top-1, App. F.2). Table 2 compares GWM only with single generation and with random or majority choice on the same pools, not with the same LLM selector without the graph, which is the control that isolates the paper's contribution. The evidence therefore supports a known result, that best-of-K selection with an LLM judge helps, rather than the value of latent workflow discovery. This drives my rating.

  2. The headline gains are selected among many configurations and come at a large cost. The Qwen CRM gain of +6.33 points is K = 10 of experiment A. Of the six K values tested, only K = 10 and K = 15 have intervals that exclude zero. Rerunning K = 10 (experiment B) gives +2.00 [−3.67, +7.67], and pooling both runs gives +4.17 [−0.83, +9.17] (Table 10). On EOPS, the main text reports +5.42 from one execution, while the other EOPS executions give +0.42, −1.25, +1.25 and +0.83 (Tables 10 and 2b). The Gemma gain falls from +12.33 to +8.00 under the archived original-evaluator output (Table 18). The cost is large: K = 10 takes 11.62 times the wall time of the baseline, and successful executions per run-hour drop from 46.80 to 4.58 (Table 15). On CRM, the candidates are complete executed rollouts, so a deployment would run every state-changing tool call K times before choosing one answer.

Questions for authors

For each headline configuration, can you report GWM against the same LLM selector, with the same K, but without graph evidence or advice? On the validation split this control matches GWM (Table 19). If that holds generally, what does the workflow graph contribute?

How were the reported configurations chosen (K, discovery fraction, experiment A versus B, which EOPS execution)? Please identify a pre-specified configuration or correct for selection, since the pooled K = 10 interval includes zero.

Whole-episode selection on CRM executes K complete rollouts, including state-changing tool calls. How would this be deployed in a real enterprise, and how does GWM perform on CRM with next-message selection only?

4 comments
AuthorsOpenReview ↗

Authors Response to Reviewer vfr3 (Part 1):

We thank reviewer vfr3 for the careful review and for recognizing our thorough controls, transparent reporting of positive and negative results, and the diagnostic value of the candidate-coverage decomposition. We also appreciate your recognition of the practical relevance of reusing enterprise interaction logs without retraining, the realistic benchmarks, and the released vLLM extension and trajectories. In response to your feedback, we have added the requested same-judge comparisons, a prospectively specified confirmation, an enforced-budget study, and a next-message experiment.

 

vfr3 — W1 — workflow discovery

Following your concern, we ran offline ranking and tool-error prediction experiments without an LLM judge to clarify the contribution of workflow discovery (G2). We tested whether workflow experience carries useful outcome information independently of generating and judging more candidates. An outcome-conditioned transition selector, without an LLM judge, selects successful candidates in 34.56% of held-out CRM pools, versus 31.56% for development-tuned dense retrieval and 29.93% for selecting the candidate with the fewest tool errors. Workflow memory also predicts next-step tool errors with an AUROC of 0.908 on CRM and 0.9174 on EOPS.

The CRM next-toolset prediction experiment in Appendix F.2 tests whether the set of tools called at the next step can be predicted from an observed trajectory prefix. On 50,680 held-out prefixes covered by both methods, the graph achieves 38.92% top-1 accuracy, compared with 64.44% for a toolset-transition frequency baseline. This baseline, called a toolset bigram, predicts the toolset that most frequently follows the current toolset in the discovery trajectories; it does not simply repeat the current toolset. In the separate CRM next-state analysis (distinct from the EOPS next-state results in Q1), the targets are inferred clusters. GWM reduces task-averaged log loss by 0.071, 0.342, and 0.116 nats against Markov, per-tool, and raw-retrieval predictors. Raw retrieval also has higher eventual-success AUROC (0.780 versus 0.751–0.768). These added experiments support outcome-informed offline ranking and tool-error prediction independently of an LLM judge. Our diagnostic analysis of candidate coverage and ranking (Eq. 10 and Appendix E.3) separates failures caused by missing successful candidates from those caused by misranking, which call for different interventions in generation and selection.

 

vfr3 —W2 — Configuration selection, variability and cost

The headline gains are selected among many configurations and come at a large cost...

First, we should emphasize that we report both positive and negative results (as you noted in the review) and distinguish exploratory studies from the prospectively specified confirmation (Q2). In addition, to address your concern, we make the study identities more explicit in the revised manuscript.

The Qwen headline (Table 2(a); Table 16, “separate GWM”) is a separate K=10 campaign with the full (100%-discovery) graph, not experiment A (Table 10). Both improve by +6.33 pp, with repeat success counts of 53/53/51 and 53/52/52, respectively. The reported EOPS +5.42 pp averages three repeats within one study. The EOPS next-message selection experiments at K=2, 4, and 8 (Table 10) give +0.42/$-1.25/+1.25pp,respectively.TheseparateK=8studywiththinkingdisabled(Table2(b);AppendixF.8,Table24)gives+0.83ppwhenitsthreeexecutionswithoutarecordedoutcomecountasfailures,sothatestimateisalowerbound.ThesehistoricalEOPSstudiesaredistinctfromG3.ThehistoricalGemmagainis+12.33ppunderpositiverewardand+8.00pp.Weretainthesedistinctionsandthehistorical11.621.25/+1.25 pp, respectively. The separate K=8 study with thinking disabled (Table 2(b); Appendix F.8, Table 24) gives +0.83 pp when its three executions without a recorded outcome count as failures, so that estimate is a lower bound. These historical EOPS studies are distinct from G3. The historical Gemma gain is +12.33 pp under positive reward and +8.00 pp. We retain these distinctions and the historical 11.62\times$ wall-time overhead, which reduced CRM throughput from 46.80 to 4.58 successful executions per run-hour. The enforced-budget CRM study (G5) reports lower inference-token use for the GWM policy with early stopping, while the CRM next-message selection study (G4; Q3) reports fewer executed tool calls than whole-episode selection.

We ran the rebuilt-memory EOPS study (G3) with all eight conditions and three repeat labels, retaining all 1,920 executions on 80 tasks without outcome-selected reruns. Gemma’s graph-informed minus graph-free differences are +5.00/+5.00/0.00 pp by repeat under both outcome definitions.

We froze the confirmation configuration before execution to address selection (G1); Q2 below gives both prespecified comparisons. We also conducted an enforced-budget study (G5). Mean token use is 104,823 with 31.0% lower usage, with family-weighted success of 39.26% versus 39.86%. We added this discussion summary and its conclusions to the revised manuscript to improve clarity.

Prespecified confirmation and complete repeat reporting address configuration selection and variability, while G5 quantifies the cost trade-off: 31.0% fewer tokens with a 0.60 pp decrease in success.

AuthorsOpenReview ↗

Authors Response to Reviewer vfr3 (Part 2):

vfr3 — Q1 — Same-LLM controls and the contribution of workflow memory

can you report GWM against the same LLM selector, with the same K, but without graph....

We address this question with three complementary experiments. G3 compares graph-informed and graph-free online selection with the same actor, LLM judge, and candidate count, with advice disabled. Subsections A and B then examine memory independently of LLM judging: A tests offline ranking of already executed CRM candidates, while B tests next-step CRM tool-error prediction from the preceding history and proposed toolset.

We ran the fresh EOPS comparison in G3 with the actor,judge, and candidate count matched and with advice off to test the incremental evidence bundle.

EOPS benchmark success uses the benchmark's original evaluator, which aggregates verifier results by check name; repeated names can overwrite earlier results. Every-defined-check success requires every defined verifier to pass without this overwriting.All eight conditions are shown below,with 240 executions each.

ConditionEOPS benchmark success (%) ↑\uparrowEvery-defined-check success (%) ↑\uparrow
Single-candidate agent30.4230.42
Random candidate selector30.0030.00
Graph-free LLM judge30.0028.75
State-aware transition scorer29.1729.17
Pooled-state transition scorer32.0832.08
Action-frequency scorer30.4230.42
History-retrieval scorer27.5027.50
Graph-informed Selector33.3332.08

Adding state summaries, transition/action statistics, and trajectory examples gives +3.33 pp under both outcomes.

We now examine whether workflow memory supports candidate ranking and error prediction without relying on the LLM judge.

  A. Judge-free offline candidate selection. We conducted an exploratory post-submission experiment with an outcome-conditioned transition selector. We fit separate transition and tool-error models to successful and failed discovery trajectories, then rank each already executed candidate by the mean success-versus-failure log-likelihood ratio of its observed transitions and tool-error outcomes, without an LLM judge. We chose settings on 744 development tasks and evaluated them on 735 held-out task pools. Higher successful-selection rates are better; differences are GWM minus the comparator.

SelectorSuccessful selection (%) ↑\uparrowGWM advantage (pp), 95% CI ↑\uparrow
GWM · outcome-conditioned transition model34.56—
Dense kNN · dev-tuned, same discovery data31.56+2.99 [+0.54, +5.44]
Fewest tool errors29.93+4.63 [+2.31, +6.94]
KNNraw · untuned raw embedding retrieval27.48+7.08 [+4.22, +9.93]
Uniform random choice · expected success24.58+9.97 [+7.84, +12.17]
First candidate · single-generation proxy24.49+10.07 [+7.35, +12.93]
MBR / medoid · soft vote over the pool24.08+10.48 [+7.75, +13.33]
Majority vote over final answers19.73+14.83 [+11.70, +17.96]

KNNraw is untuned embedding retrieval; dense kNN tunes retrieval on development data. MBR (minimum Bayes risk)/medoid selects the candidate most similar to the rest of its pool. The intervals are pointwise paired task-bootstrap intervals from 20,000 resamples. The fewest-errors comparison survives Holm adjustment over eight contrasts (p = 0.0006); dense kNN has an adjusted p = 0.0618 over four retrieval contrasts.

In this offline comparison, learned workflow outcomes improve candidate selection over simply choosing the candidate with the fewest tool errors.

  B. Tool-error prediction without an LLM judge. A 93-state memory built from 357 CRM discovery tasks predicts the observed next-step tool-error outcome using the preceding history and proposed toolset. We evaluated it on 77,804 transitions from 1,479 held-out tasks; an AUROC of 0.5 corresponds to chance.

PredictorError AUROCGWM advantage, 95% CI
GWM0.908—
KNNraw · same discovery data0.876+0.031 [+0.029, +0.034]
Per-tool error rate0.847+0.060 [+0.057, +0.064]
Domain error rate0.706+0.201 [+0.193, +0.208]

These are pointwise paired task-bootstrap intervals from 2,000 resamples.

Workflow memory predicts next-step tool errors on held-out CRM tasks with higher AUROC than the retrieval and error-rate baselines, without an LLM judge.

In conclusion, G3 shows a positive observed gain under matched online selection settings, while A and B demonstrate useful offline ranking and tool-error prediction without an LLM judge. We also include these results in the revised version of the manuscript. We appreciate your feedback, which helped us explore this direction.

AuthorsOpenReview ↗

Authors Response to Reviewer vfr3 (Part 3):

vfr3 — Q2 — How configurations were chosen

How were the reported configurations chosen (K, discovery fraction, experiment A versus B, which EOPS execution)? Please identify a pre-specified configuration or correct for selection, since the pooled K = 10 interval includes zero.

The historical experiments were designed as a practical, industry-based, exploratory evaluation of configuration choices, rather than a single pre-specified confirmatory test. We report both positive and negative findings (as you noted) to make the observed variation across configurations transparent. We intentionally retain the historical K sweep as an exploratory analysis without selection correction; its intervals should therefore be interpreted as pointwise, descriptive estimates, not as selection-adjusted evidence that the chosen configuration is superior. We will clarify this distinction in the revised manuscript, and the original Qwen K=10 headline remains exploratory.

For the subsequent confirmation experiment, we fixed the model, candidate count, advice policy, decoding settings, tie-breaking rule, and comparison conditions before the new executions began. This separates exploratory configuration development from prospective evaluation and prevents choosing a configuration based on the confirmation outcomes.

G1 addresses the request for a prespecified configuration by fixing its settings before evaluation; the historical K sweep remains exploratory.

 

vfr3 — Q3 — Enterprise effects and CRM next-message selection

Whole-episode selection on CRM executes K complete rollouts, including state-changing tool calls.  

Thank you for raising this distinction. Whole-episode selection requires separating candidate evaluation from production effects. We outline eight deployment and evaluation options, including possible extensions:

1. Select before execution. Generate K next-message candidates, rank them with GWM, and dispatch only the selected message after application checks.

2. Isolated execution. Evaluate complete candidate episodes in independent sandbox snapshots with external effects blocked or simulated, then revalidate the selected workflow against current production state before applying it.

3. Simulation or digital twins. GWM could guide exploration while a simulator supplies outcomes; CRMArena-Pro [1] illustrates a Salesforce test environment with synthetic data.

4. Read-only or advisory use. Compare analyses or recommendations without changing business state, handling any subsequent write separately.

5. Shadow evaluation. Log GWM recommendations alongside a production agent for comparison while only the existing agent executes actions.

6. Human-reviewed execution. Present the selected action and supporting workflow evidence for approval or revision before dispatch (tool-approval example).

7. Offline regression testing. Replay historical tasks in isolated environments to compare model, prompt, or workflow-memory changes before deployment (sandbox testing example).

8. Imagination World (proposed extension). GWM could guide policy exploration within a learned world model that predicts action consequences without real-world execution, inspired by V-JEPA 2-AC [2] and Dreamer [3].

how does GWM perform on CRM with next-message selection only?

In G4, we evaluated selection before dispatch on 204 CRM validation tasks in a read-only workload, with three repeats, K=8, and advice disabled. Compared with whole-episode GWM selection, executed tool calls fell by approximately 88% (87.51% for Gemma; 88.11% for Qwen).

Evaluating multiple candidates need not multiply production effects: GWM can select before dispatch or evaluate alternatives outside production, with only the chosen actions applied after validation. We will include a summary of this discussion as a dedicated paragraph on enterprise deployment in the revised manuscript.

 

References

[1] CRMArena-Pro: Holistic Assessment of LLM Agents Across Diverse Business Scenarios and Interactions.

[2] V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning.

[3] Dream to Control: Learning Behaviors by Latent Imagination.

AuthorsOpenReview ↗

Reviewer vfr3: Follow-Up on Workflow Memory Beyond Best-of-K Selection

Dear Reviewer vfr3,

We understand that you have many demands on your time, so we will keep this brief. Before the discussion closes, we wanted to highlight our responses to your central concern: what workflow memory contributes beyond best-of-K selection with the same LLM judge. Alongside matched graph-free controls, our additional experiments demonstrate useful held-out ranking and tool-error prediction without an LLM judge. We also addressed configuration selection and CRM execution costs. We would appreciate any feedback you can share on these additions or remaining concerns.

Reviewer_bJA4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Graph World Models (GWMs), a framework for improving enterprise-oriented language-model agents by reusing historical interaction trajectories without updating the acting policy's parameters. The framework constructs a graph of latent workflow states by clustering interaction-history prefixes and aggregating observed transitions, outcome statistics, support counts, and representative examples.

At inference time, GWM retrieves relevant workflow evidence to provide guidance to the acting agent and to select among candidate continuations. The authors evaluate the framework on CRMArena-Pro and EnterpriseOps-Gym, considering within-benchmark performance, cross-benchmark transfer, and cross-model reuse.

The experiments demonstrate improvements over direct generation across several configurations. In particular, the Gemma large-graph configuration improves CRM success from 40.00% to 52.33%, while the Qwen CRM configuration improves from 46.00% to 52.33%. Additional analyses examine workflow-state informativeness, candidate selection, evidence representations, and computational costs.

Overall, the paper presents a practically motivated framework with encouraging empirical results and an extensive evaluation.

Critical strengths
  1. A practically useful framework for reusing enterprise workflow experience.

The paper addresses an important challenge in enterprise agent systems: how to reuse historical organizational experience to improve decision-making without retraining the underlying language model.

The proposed integration of workflow-state representations, historical outcomes, retrieval, advice, and candidate selection provides a coherent approach to this problem. Unlike approaches that primarily reuse individual examples or textual instructions, GWM explicitly organizes historical experience through state-level associations and observed continuations.

The framework is also compatible with different acting models, making its potential applicability beyond a single language-model architecture particularly appealing.

  1. Encouraging empirical improvements and comprehensive diagnostic evaluation.

The experimental results provide evidence that complete GWM configurations can improve task success. The Gemma GWM-L configuration achieves a 12.33 percentage-point improvement over its recorded baseline, with a positive paired confidence interval. The Qwen CRM configuration also shows an improvement over direct generation.

The evaluation goes beyond reporting aggregate success rates. The authors investigate cross-model and cross-benchmark reuse, candidate selection mechanisms, workflow-state outcome information, and computational overhead.

The workflow-state outcome statistics achieve an AUROC of 0.745 in the reported diagnostic experiment, suggesting that the learned representations retain useful information about eventual task outcomes.

I also appreciate the inclusion of controlled comparisons, uncertainty estimates, and negative or inconclusive results. These analyses make the empirical conclusions more informative and transparent.

Critical weaknesses
  1. The incremental contribution of graph-structured evidence is not consistently established.

Although the complete GWM configurations achieve encouraging improvements over direct generation, it remains difficult to determine how much of the improvement comes specifically from the graph representation rather than additional candidate generation, advice, and language-model-based selection.

For example, the separate CRM validation experiment reports comparable performance between GWM guidance and graph-free selection (Table 19). The fixed-candidate comparisons in Table 20 show mixed graph-evidence effects, while the common-cohort experiment in Table 23 reports no improvement from adding workflow-state evidence.

These results suggest that the graph-specific contribution is less consistently supported than the effectiveness of the overall guidance framework.

Nevertheless, I view this primarily as a limitation in isolating the mechanisms behind the observed improvements, rather than evidence that the complete framework is ineffective. A clearer separation of the individual contributions would further strengthen the paper.

  1. Computational overhead and generalization remain important considerations.

The framework introduces additional costs through candidate generation, language-model-based selection, retrieval, and offline workflow discovery. In particular, CRM whole-episode selection requires multiple executed candidate trajectories, which makes comparisons with single-generation baselines difficult to interpret purely in terms of computational efficiency.

Although the authors report resource measurements, the practical effectiveness-cost trade-off would benefit from a clearer analysis under matched inference budgets.

Furthermore, the evaluation is concentrated on two enterprise-agent benchmarks, and the comparison with existing workflow-memory approaches largely uses adapted evidence formats rather than complete competing systems.

These limitations leave some uncertainty about the framework's advantages in broader deployment settings, although the cross-benchmark and cross-model experiments provide useful initial evidence of transferability.

Questions for authors

Q1. How do the authors interpret the inconsistent incremental effects of graph evidence across Tables 19, 20, and 23? Which graph components contribute most to the improvements observed in the complete online configurations?

Q2. Can the authors clarify the practical effectiveness-cost trade-off when comparing GWM with graph-free selection or other inference-time strategies under similar computational budgets?

Q3. To what extent do the discovered workflow states and transitions capture transferable organizational procedures rather than task-specific historical correlations? Can the authors provide additional insight into the role of the learned transition structure during candidate selection?

5 comments
AuthorsOpenReview ↗

Response to Reviewer bJA4 (Part 1):

We thank the reviewer bJA4 for recognizing GWM’s practical value in reusing enterprise workflow experience, as well as its compatibility with different acting models. We also appreciate the reviewer’s positive assessment of the encouraging empirical improvements and comprehensive diagnostic evaluation, including our controlled comparisons, uncertainty estimates, and transparent reporting of negative or inconclusive results.

 

bJA4 — W1 — Incremental graph contribution

 

W1: The incremental contribution of graph-structured evidence is not consistently established. Although the complete GWM configurations achieve encouraging improvements over direct generation, ....  

We agree that a clearer presentation of Tables 19, 20, and 23 is needed. We have therefore clarified these comparisons in the revised manuscript and provide further details in Q1 below. However, please note that GWM is presented as an integrated harness combining workflow memory, advice, and a guided selector, making whole-system evaluation a primary objective alongside component analysis.

As detailed in our general response, G1 tests the robustness of complete-system gains over single generation, G2 evaluates predictive information independently of LLM judging, and G3–G4 assess incremental evidence effects under matched selection procedures. G5 examines system efficiency, while G6 helps interpret candidate-coverage and ranking limitations. Together, these results demonstrate gains from the complete GWM configuration, an observed benefit from workflow evidence in matched selection, and useful predictive information in the stored experience.

 

bJA4 — W2 — Cost, baselines and generality

 

the practical effectiveness-cost trade-off would benefit from a clearer analysis under matched inference budgets...  

Thank you for highlighting the effectiveness–cost trade-off and the scope of the generalization evidence.

G5 addresses matched-budget comparability. Its primary task-level analysis finds 31.0% lower token use within that margin for the integrated harness and stopping policy. G1 addresses configuration selection and within-CRM reuse by freezing the configuration before evaluating 600 historical CRM tasks across all 22 task families, with each task’s family fold and audited duplicate/template groups excluded from its memory. It confirms a +5.95 pp gain over single generation. G4 separately measures executed tool work. Our current competing-memory comparisons isolate evidence formats under a shared selector.

In summary, G5 quantifies the inference cost–quality trade-off under a common budget, while G1 confirms complete-system gains.

AuthorsOpenReview ↗

Response to Reviewer bJA4 (Part 2):

bJA4 — Q1.A — Interpreting Tables 19, 20 and 23.

How do the authors interpret the inconsistent incremental effects of graph evidence across Tables 19, 20, and 23?

 

These tables evaluate three different interventions. Table 19 compares complete online configurations, in which advice can influence candidate generation. Table 20 holds the candidates fixed while varying the evidence used to rank them. Table 23 varies state labels and statistics on a common 600-task cohort.

The replay results support the following interpretation, without identifying the causal contribution of each limitation.

  1. Information overlap. In the graph-free prompts, the task and bounded trajectory context are exposed as prompt inputs; similarly, next-action prompts also expose candidate arguments (see the templates below).

    Each next-action candidate exposes tools=[...], args=[...], Candidate message text: ....

    Graph mode uses state statistics and optional exemplars for next-action selection and/or whole-rollout scoring.

    Exposed prompt templates. These condensed templates retain the input fields of the inspected source.

    Whole-rollout scoring (CRM).

    Estimate whether the rollout's final answer is correct/complete.
    [Task excerpt] {task_goal}
    [Domain / workflow state] {domain} / {state}
    {state_stats}
    {tool_docs_block}{examples_block}
    [Trajectory tail] {flow_tail}
    Return {"score": <0--1>, "reason": "<one line>"}.
    

    Next-action comparison (EOPS).

    Compare {n} candidates by argument correctness and task progress.
    [Task excerpt] {task_goal}
    [Domain / workflow state] {domain} / {state}
    {state_stats}
    [Ranked next actions] {next_actions}
    {tool_docs_block}{examples_block}
    [Recent activity] {flow_tail}
    [Candidates] {candidates}
    Return {"compare": "...", "best": <index>, "scores": [...]}.
    
  2. Abstraction limits. The action key ψ(a)\psi(a) keeps sorted, deduplicated tool names, omitting arguments, within-step order, and multiplicity. A lookup qtoolset(s,a)q_{\mathrm{toolset}}(s,a) returns one stored success value per state and key. At a common retrieved state ss,

    ψ(a1)=ψ(a2)⟹qtoolset(s,a1)=qtoolset(s,a2),\psi(a_1)=\psi(a_2) \quad\Longrightarrow\quad q_{\mathrm{toolset}}(s,a_1)=q_{\mathrm{toolset}}(s,a_2),

    provided the lookup exists with support from at least three rollouts.

  3. Fixed-pool coverage. Replay generates no new candidates. The guided primary pools have the following coverage limitations:

    • Gemma test: 9494 tasks ×,3\times\\,3 repeats =282=282 pools, each with 88 candidates. In 9494 pools (33.3333.33\\%), all 88 candidates failed.

    • Gemma validation: 180180 tasks ×,3\times\\,3 repeats =540=540 pools, each with 88 candidates. In 236236 pools (43.7043.70\\%), all 88 candidates failed.

    • Qwen validation: 180180 tasks ×,3\times\\,3 repeats =540=540 pools, each with 1010 candidates. In 202202 pools (37.4137.41\\%), all 1010 candidates failed.

    For example, if a Gemma pool contains eight incorrect solutions, adding graph evidence to the selector cannot produce a successful selection(no successful candidate).

    These percentages are not the selector’s total failure rate: a selector can also fail by choosing an unsuccessful candidate from a successful pool. Indeed, graph selection remains 14.8914.89, 14.4414.44, and 17.0417.04 pp below the respective oracle ceilings.

    All-failure pools force both selectors to receive a failure outcome, reducing the opportunities to demonstrate better selection.

    The oracle success rate is the percentage of pools containing at least one successful candidate. The saved rates show slightly higher coverage for unguided pools on Gemma, with the opposite direction on Qwen:

    • Gemma test: unguided 68.4468.44\\%, guided 66.6766.67\\%; unguided higher by 1.771.77 pp.

    • Gemma validation: unguided 57.0457.04\\%, guided 56.3056.30\\%; unguided higher by 0.740.74 pp.

    • Qwen validation: unguided 61.8561.85\\%, guided 62.5962.59\\%; guided higher by 0.740.74 pp.

Scope. The primary exact-match/privacy-rejection subsets contain 94, 180, and 180 tasks, respectively, with three repeats (K=8K=8 for Gemma; K=10K=10 for Qwen). Each contrast holds the selector and candidates fixed, varying graph evidence; guided/unguided denotes pool origin. Intervals use 20,000 task-cluster bootstrap resamples with approximate simultaneous 95% coverage via Bonferroni adjustment over five contrasts per cohort.

These diagnostics help interpret the mixed graph-evidence results by separating candidate-generation effects from ranking effects and identifying where information overlap, coarse action representations, and missing successful candidates can limit selection gains.

AuthorsOpenReview ↗

Response to Reviewer bJA4 (Part 3):

bJA4 — Q1.B — GWM component contributions

Which graph components contribute most to the improvements observed in the complete online configurations?

Thank you for suggesting a clearer separation of the individual contributions. We report new online component-removal experiments and complementary fixed-pool analyses in G7 — Component contributions within the complete GWM harness.

On Gemma CRM, the full configuration achieves the highest observed success among the five conditions (40.50%), and all four component estimates favor retention; explicit successor information has the largest point estimate (+1.24 pp). However, the intervals do not establish an individual effect or component ordering, and EOPS estimates are mixed. G7 provides the estimates and uncertainty, and explains why these conditional removals cannot allocate separate shares of the complete-harness gain.

The confirmed complete-system gain supports the integrated GWM harness; identifying a dominant individual contributor remains unresolved.

 

bJA4 — Q2 — Effectiveness at similar computational budgets

Can the authors clarify the practical effectiveness-cost trade-off when comparing GWM with graph-free selection or other inference-time strategies under similar computational budgets?

  To answer this question, we conducted an enforced-budget experiment (G5) on 600 historical CRM tasks with three repeats, giving GWM and the graph-free control the same 200,000-token ceiling per task (as you sugessted) across actor, judge, and advisor. We measured actual token use and tested whether any reduction in success remained within a predefined 3 pp noninferiority margin. The selected GWM policy with early stopping used 31.0% fewer tokens, with 39.26% versus 39.86% success, meeting the joint quality–efficiency criterion under the primary task-level analysis. This directly addresses the requested trade-off by comparing inference-token use at a declared quality tolerance. The result applies to the combined harness and stopping policy and excludes offline discovery/build costs.

Please note that Table 2 of the initially reviewed manuscript already includes uniform random selection and majority-vote selection under matched candidate-generation budgets (lines 319–323) for a fair comparison.

We separately measured executed tool work in G4 by choosing a proposal before dispatch, which avoids executing every candidate episode. By comparing task performance and tool-call counts, G4 evaluates whether reducing executed work preserves quality, directly addressing the practical cost of multiple candidate trajectories and complementing G5’s inference-token comparison.

In summary, G5 supports lower inference-token use within the declared quality margin for the integrated harness and stopping policy under the primary analysis. Table 2 and G4 study complement this result with matched candidate-generation comparisons and a separate assessment of the quality–tool-work trade-off.

Comments on the paper

6 comments
AuthorsOpenReview ↗

Additional experimental evidence on GWM effectiveness, prediction, and execution cost

General response — Part 3

G5. Enforced-budget inference efficiency of the GWM harness.

The primary study supports 31.0% lower inference-token use within the predeclared 3-pp quality margin under the registered analysis. We evaluated Gemma-4-26B-A4B-it on 600 historical CRM tasks with three repeated executions per condition. We enforced a common 200,000-token ceiling per task across actor, judge and advisor. Both methods received the same excluded development allowance; the frozen policies used early stopping for full GWM and up to eight candidates for the graph-free control.

MeasureFull GWM harnessGraph-free control
Task-family-weighted success39.26%39.86%
Mean model tokens per task, across all roles104,823151,823
Token reduction31.0%—

The success difference was −0.59 pp, with a multiplicity-adjusted 98.75% interval of [−2.69, +1.46]; the full/control token ratio was 0.690 [0.669, 0.712] under the same adjustment. This meets the predeclared criterion of at least 20% lower inference-token use with our analysis of a 3-pp noninferiority margin in the registered task-stratified analysis. The result supports the integrated harness and its stopping policy. Although the whole-family sensitivity did not meet the joint criterion, limiting generalization across task families, it is successfully demonstrate the token reduction within configuration. Please note that offline discovery/build costs are outside this token metric.

G6. Key contributions and strengths recognized by the reviewers.

We thank the reviewers for recognizing:

  • Practical harness: “A practically useful framework for reusing enterprise workflow experience” and and “framework is also compatible with different acting models, making its potential applicability beyond a single language-model architecture particularly appealing” (bJA4, Strength 1) , “the two benchmarks are realistic, and the vLLM extension and all trajectories are released.“ (vfr3, Strength 2)

  • Empirical evidence: “Encouraging empirical improvements and comprehensive diagnostic evaluation” (bJA4, Strength 2).

  • Failure diagnostics: “a useful diagnostic for best-of-K agents” (vfr3, Strength 1) and “a more useful analysis than success rates alone” (593s, Strength 1).

  • Transparency: “Thorough controls, reported candidly” (vfr3, Strength 1), alongside matched comparisons, uncertainty estimates and acknowledged limitations (593s, Strength 2).

AuthorsOpenReview ↗

Additional experimental evidence on GWM effectiveness, prediction, and execution cost

General response — Part 2

G3. Observed gains with the same EOPS actor and judge. (bJA4: W1, Q1, Q2; vfr3: W1, W2, Q1; 593s: W1) We added another experiment to test whether GWM evidence improves next-message selection when Gemma serves as both actor and judge. We ran fresh executions on the same 80-task EOPS cohort used in the earlier Qwen study, with three repeats, K=8K = 8 next-message proposals and advice off. We report both outcome definitions and the observed resource use below. Original benchmark success uses the original name-keyed verifier aggregation, in which a repeated check name can overwrite an earlier result. For every-defined-check success, we require all defined checks to pass without that overwriting; the memory's archived discovery labels remain unchanged.

MeasureGraph-free LLM judgeGraph-informed LLM judgeDifference
Original benchmark success ↑72/240 (30.00%)80/240 (33.33%)+3.33 pp [0.00, +6.67]
Every-defined-check success ↑69/240 (28.75%)77/240 (32.08%)+3.33 pp [−0.42, +7.08]
Total model tokens (millions)171.346170.196−0.67%
Median task runtime (seconds)27.030.3+3.3 s

Each success endpoint uses its own five-contrast simultaneous 95% task-cluster interval; these are not joint intervals across both families. Please note for the transparency both include zero and the predefined +5 pp useful-gain criterion is unmet but useful-gain criterion of +3 pp is met. Resource differences are descriptive, not evidence of an enforced-budget advantage. We retain all eight conditions in vfr3 Q1, with repeat results and the earlier Qwen comparison in vfr3 W2. This is an observed gain for the combined evidence bundle; topology is not isolated.

G4. CRM selection before execution: measured quality and work. (bJA4: W2, Q2; vfr3: W2, Q3; 593s: Q4) We evaluated all four conditions (direct generation, graph-free next-message selection, GWM next-message selection, and GWM whole-episode selection) on 204 reused validation tasks with three repeats, giving 612 executions per condition and model. In the selection conditions, we use K=8K = 8 with advice off and the same named model for acting and judging. In the two next-message conditions, we dispatch only the selected message; in the whole-episode reference, we execute all candidate rollouts before selection.

ModelConditionNonzero reward (%) ↑Full credit (%) ↑Mean reward (0–1) ↑Calls / executionTokens / executionp95 latency (s)
Gemma-4-26B-A4B-itDirect generation34.9729.250.31014.2924,31034.97
Gemma-4-26B-A4B-itGraph-free next message37.9131.860.33584.26216,011310.82
Gemma-4-26B-A4B-itGWM next message38.0732.190.33734.19210,045296.00
Gemma-4-26B-A4B-itGWM whole episode41.0135.290.371433.57195,093254.58
════════════════════════════════════════════════════════════════════
Qwen3.6-27BDirect generation41.0135.290.37154.7628,04958.73
Qwen3.6-27BGraph-free next message41.8336.110.37974.54216,457421.75
Qwen3.6-27BGWM next message43.6338.240.40144.57225,030426.55
Qwen3.6-27BGWM whole episode46.2440.360.425938.46226,265384.69
Executed-call reduction: whole episode → next messageReduction (%)Adjusted 99.375% interval (%)
Gemma-4-26B-A4B-it87.51[86.85, 88.16]
Qwen3.6-27B88.11[87.69, 88.54]

vfr3 Q3 gives the primary contrasts and intervals. We define nonzero reward to include partial credit and use the unchanged fuzzy-match evaluator for full credit and mean reward. We specified the supplementary full-credit and mean-reward analyses after reviewing Gemma's results, but before reviewing and providing Qwen's accuracy results (Qwen’s accuracy results” refers broadly to its measured task-performance). They summarize the same recorded outcomes and do not replace the predefined nonzero-reward measure. The table's emphasis identifies the proposed next-message configuration, not a winner on every measure.

These results shows that GWM next-message selection reduces executed tool calls by about 88% compared with whole-episode selection

AuthorsOpenReview ↗

Additional experimental evidence on GWM effectiveness, prediction, and execution cost

General response — Part 1

Dear Reviewers,

We warmly appreciate your time for reviewing our work and offering insightful constructive feedback. We take advantage of your raised concerns/feedbacks to strength the quality of the manuscript by addressing all your concerns in the revised version of the manuscript.

We have completed additional studies addressing configuration robustness, same-judge controls, executed tool work and predictive information. We summarize the findings of these studies in the tables below and address their controls and interpretation in the reviewer-specific answers. We keep model weights fixed throughout. We will post a revised version of the manuscript with additional details shortly.

G1. Prospective confirmation against a single-attempt baseline. (bJA4: W2; vfr3: W2, Q2; 593s: W2, Q4) We froze the configuration before confirmation execution on 600 CRM tasks covering all 22 task-type families, with three repeated executions per condition. Full GWM uses Gemma, K=8K = 8, memory evidence and advice. We grouped the task families into five folds and built a memory for each fold. Each task uses the memory that excludes its entire fold and audited connected duplicate/template groups; we also exclude the 40 pilot tasks.

ComparisonBaseline success (%) ↑GWM success (%) ↑GWM gain (pp)Adjusted interval (pp)
Full GWM versus baseline without GWM (single attempt)34.76 *40.71+5.95[+3.65, +8.27]

* The Gemma baselines come from separate CRM evaluations: 40.0 on the paper’s 100-task cohort and 34.76 on the new 600-task, 22-family confirmation, with different aggregation and success definitions.\scriptsize\text{* The Gemma baselines come from separate CRM evaluations: 40.0 on the paper’s 100-task cohort and 34.76 on the new 600-task, 22-family confirmation, with different aggregation and success definitions.}

We average success rates across task families using fixed weights that reflect each family's share of the original task corpus. Throughout these responses, we calculate differences before rounding the displayed values. The predefined criterion—at least +5 pp as the point estimate and a positive adjusted lower bound, is met. The interval retains the original adjustment for both co-primary comparisons: a 97.5% individual interval for approximate 95% joint coverage, using 20,000 paired within-family task-bootstrap draws. This supports the combined configuration under within-CRM family exclusion on historical tasks. Please note that it does not isolate the effect of memory or equalize inference cost. We extend the report in vfr3 Q2.

G2. Predicting external tool errors without an LLM judge. ( bJA4: W1, Q1, Q3; vfr3: W1, Q1; 593s: W1, Q3) We conducted an external tool-error prediction study with a baseline suite expanded to eight alternatives, using the same 412 held-out EOPS tasks, 16,851 rollouts and 148,319 transitions. Here and below, corrected or rebuilt EOPS memory refers to reconstruction after repairing rollout identities; G3 separately explains the success-scoring definitions. GWM improves error AUROC over all eight expanded alternatives after adjustment across 24 comparator-by-metric endpoints. Further details of the expanded comparison will be added to the revised manuscript.

PredictorError AUROC ↑Brier ↓ECE ↓
Global prior0.50000.05380.0038
Domain prior0.79450.05040.0066
Last-error persistence0.71390.06160.0616
Error-history Markov0.85170.04400.0102
State-only0.74610.05390.0343
Text history linear0.79620.04910.0213
Text and action linear0.80940.04820.0206
Raw kNN0.81420.04620.0323
GWM0.91740.04030.0157

We observed an AUROC advantage of +0.0656 [+0.0226, +0.1082] over the strongest expanded alternative, error-history Markov, which conditions on the domain and previous error flag. We weight transitions equally and preserve within-task dependence with 20,000 whole-task bootstrap draws. In this experiment, we show that GWM achieves the lowest observed Brier score in this suite, with adjusted improvements over seven alternatives; calibration is mixed. This exploratory expansion was specified after the original readout and before computing the new scores. We report the original required per-tool comparison and the reason for stopping before the online study. The separate CRM prediction and selection results appear in vfr3 Q1.

Models:

  • Gemma-4-26B-A4B-it refers as Gemma
  • Qwen3.6-27B refers as Qwen

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.