- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces Procedural Graphs (PGs), an explicit representation of procedural knowledge intended to improve action selection in LLM agents. The graph represents actions, reasoning steps, and task states as nodes, with attributed edges encoding transition conditions, guidance, and pitfalls in natural language. During inference, the framework localizes the agent's current state, retrieves a local graph neighbourhood, and uses an LLM to generate situational guidance for the next action.
There is an offline self-evolution mechanism that updates graph nodes, edges, and attributes based on successful and failed execution trajectories. Candidate updates are evaluated on a held-out validation set and accepted if performance does not decrease and rejected modifications are retained as negative feedback.
Evaluated on seven benchmarks and four LLMs. They compare against several memory-based and workflow-based baselines. The results show improvements across many settings including long-horizon decision-making.
Critical strengths
-
The method is practical for maintaining and improving PGs. Explicit procedural representation, step-level guidance, and feedback-driven refinement are useful contribution. Constructing graphs from a minimal initialization and recovering from poorly designed expert priors reduces dependence on manually specified workflows. The paper also provides useful ablations comparing graph construction strategies and methods of incorporating graph information during inference.
-
Promising empirical results. Broad evaluations across models, benchmarks and baselines. The results are competitive. The EnterpriseArena runs shows changes are associated with agent behaviour, such as anticipating funding requirements and avoiding redundant tool calls, rather than simply increasing the number of actions.
Critical weaknesses
- The contribution of the graph representation is not sufficiently isolated.
The proposed framework combines multiple components: structured procedural knowledge, an additional LLM guidance call at each execution step, and iterative refinement with validation-based selection. The paper does not clearly establish how much of the improvement comes specifically from the graph representation.
A flat collection of conditional guidelines could potentially be refined using the same execution feedback and validation procedure, and then converted into situational guidance using the same LLM. Similarly, an additional planning or reflection step might improve a standard ReAct agent without requiring a procedural graph. The ablations in Table 3 compare graph injection and localization strategies, but do not isolate these alternative explanations. This limits the strength of the paper's central claim that the explicit graph structure is responsible for the observed improvements.
- Evidence on reliable self-evolution is a bit limited.
The design choices make their generality difficult to assess. In the EnterpriseArena evolution experiment, candidate graphs are selected using only 20 validation episodes, and the same validation set is consulted repeatedly across rounds. Moreover, the acceptance criterion allows updates with equal measured validation performance, even when the evaluation is noisy.
In Table 11 Round 7 achieves 80% validation survival and 95% test survival, whereas Round 8 achieves 90% validation survival but only 85% test survival. This does not invalidate the method, especially given the small sample sizes, but it raises questions about the reliability of checkpoint selection.
- The improvements on EnterpriseArena may partly reflect learning benchmark-specific heuristics rather than generalizable long-horizon planning. The evolved graph mostly learns a simple sequence of checking cash, forecasting liquidity, and requesting funding early. It is unclear whether similar gains could be achieved by providing the baseline agent with explicit instructions about liquidity buffers and fundraising delays. A comparison against such a baseline would help establish whether the proposed framework offers benefits beyond discovering effective domain-specific rules.
Questions for authors
-
Have the authors considered comparing PG against a flat collection of conditional procedural guidelines that undergoes the same iterative refinement and validation process? Would help establish whether explicit graph connectivity provides an advantage beyond learning and updating natural-language instructions.
-
Since PG introduces an additional guidance LLM call at each decision step, how would the method compare against a compute-matched ReAct agent that uses an additional planning or reflection step without a graph?
-
In EnterpriseArena, how much of the improvement can be attributed to discovering relatively simple domain-specific strategies, such as maintaining a liquidity buffer and initiating fundraising early?
Thank you for distinguishing the performance of the full system from the contribution of its components. We address the new ablation, validation protocol, and EnterpriseArena interpretation below.
1. Contributions of the graph, localization, and guidance
The new local/raw condition directly supplies the localized subgraph without an auxiliary guidance-model call. It uses the same settings as Table 3.
These comparisons show gains without the extra guidance call, benefits from localization, and the incremental benefit of generative guidance, respectively. Table 2 separately evaluates refinement: evolution raises the expert-initialized graph's MultiChallenge success rate from 58.93% to 92.86%.
We will add these results to Table 3. They isolate components within PG, but do not replace a compute-matched, graph-free planning or reflection baseline.
2. Comparison with conditional guidelines
AutoGuide retrieves rules from a condition-indexed pool and is the closest existing baseline to the proposed conditional guidelines. ExpeL stores natural-language insights; KnowAgent represents action and transition rules in text. Table 1 gives the following point-estimate comparisons across 24 model–benchmark settings. These results support PG relative to the evaluated methods.
3. Validation reuse, equal-score updates, and checkpoint selection
For EnterpriseArena, the selection metric is full-horizon survival rate on a fixed set of 20 validation episodes. Training, validation, and test seed sets are disjoint. We accept candidates whose validation survival improves or ties the retained graph, preferring the newer candidate on a tie.
A fixed validation set keeps evaluation cases consistent, but repeated selection can still adapt to that set. One success changes the rate by five percentage points, limiting the gate's resolution. The tie rule retains revisions based on new training feedback when there is no measured validation drop; it does not establish better expected test performance.
From Round 7 to Round 8, validation survival rises from 80% to 90%, while test survival falls from 95% to 85%. Each change corresponds to two episodes. Sampling variability or adaptation to the validation set could explain this reversal; these rounds alone cannot distinguish them. Repeated selection on finite validation data can overfit the selection criterion (Cawley and Talbot, 2010).
The returned graph is selected by validation survival and the stated tie rule. Its test survival is 85%; the intermediate 95% test peak was not used for selection. We will clarify this protocol and avoid implying that every accepted update improves test performance.
4. EnterpriseArena and domain-specific rules
Learning domain-specific procedures is part of PG's intended role. EnterpriseArena requires applying them over up to 132 months, including crises at months 32, 59, and 112, financing delays of one to six months, changing capital limits, repayments, losses, and operating costs. “Raise funds early” still leaves state-dependent decisions about timing, amount, and financing type. Held-out seeds vary stochastic realizations within this environment; the crisis schedule is shared.
Table 8 reports full-horizon survival:
PG improves survival for three models; all methods remain at zero for Gemini 3.5 Flash. These comparisons support gains over the evaluated baselines, but do not determine whether explicitly supplying the same liquidity and fundraising rules would achieve similar results. That control remains untested.
We will frame this evidence as learning useful procedures for long-horizon execution within this environment. The separate contributions of rule content and its organization remain unresolved.
Summary
The new ablation shows gains without an auxiliary guidance call, benefits from localization, and further gains from generative guidance on all three benchmarks. Together with the refinement results in Table 2 and the guideline comparisons in Table 1, these findings support the effectiveness of learning, refining, and locally retrieving procedural knowledge. We also clarify that final graph selection uses validation survival, without selecting on the intermediate test peak.