acceptodds
Under review as a conference paper at ICLR 2027

Evidence-Guided Updates for Reusable Agent Workflows

Abstract

Language-model agents reuse executable workflows to make recurring work efficient, but changing user requirements demand updates that preserve existing responsibilities. Correct examples can identify a new decision without specifying when it should apply, allowing an otherwise successful change to disrupt other uses of the workflow. Starting from an existing program and a small set of correct public execution examples, we propose evidence-guided local updates that combine execution constraints with inherited applicability conditions. The program's conditions are bound across function calls before input adaptation, preserving the behavior of uses unaffected by the new requirement. To reduce the cost of selecting reliable changes, we replay captured decisions to screen bounded candidates and use native execution to accept the selected program. On authored reconciliation workflows, inherited conditions improve evidence-registration success from 60% to 100%. For identical final programs, replay reduces incremental acquisition time by 61.7% relative to native interpreter reuse with early stopping.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
language-model agents, continual capability acquisition, executable workflows, program refinement, execution-guided learning, decision replay

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WoSQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies how to update a reusable agent workflow program when requirements change, without breaking its other uses. The method starts from the existing program and two correct executions. It records the inputs of a pure decision function and enumerates candidate input adapters from a small typed grammar. It then restricts each adapter with conditions taken from the existing program's branches and selects the first candidate that reproduces the target outputs when the decision function is replayed. A final run in the real application environment confirms the choice. On reconciliation and maintenance workflows written by the authors in AppWorld, the method solves all relational updates in about 11–12 seconds, versus 480–630 seconds for LLM-based selection. The inherited conditions raise evidence-registration success from 45/75 to 75/75. When both select the same final program, replay is 61.7% faster than validating in the real environment.

Critical strengths

S1. The problem is well motivated and under-studied (Fig. 1). Using the parent program's own conditions to decide where a change applies is an appealing idea. Table 7 isolates its effect cleanly: when only the condition is removed, all teaching and audit checks still pass, but evidence registration falls from 75/75 to 45/75.

S2. The experimental accounting is careful and transparent. All draws are reported, including failures, costs are itemized, and frozen programs are re-evaluated on fresh instances.

Critical weaknesses

W1. The evaluation is built around the method, so its generality is not established.

  • The tasks, parent programs and grammar are all written by the authors. The three relational updates map one-to-one onto the grammar's three adapter families. The parent programs also encode their different uses in exactly the form the method extracts.
  • When the needed change falls outside the grammar, the method does not help: in THOR, none of 54 fresh acquisition paths yields an accepted update (Table 37).
  • Because inherited-scope candidates are always tried first, the method prefers the conservative scope. Yet no experiment includes an update that should apply inside the bypassed branch.
  • Each event has only five underlying cases, each viewed in three ways.

W2. The baselines are weak, and the presentation obscures the contribution.

  • The strongest control is an LLM choosing among the same candidates without executing them. It already matches replay's quality in most cases (Table 1), so the gain is mainly cost. Fairer comparisons would be LLM editing combined with execution-based repair, or a stronger model.
  • The 61.7% saving comes from replaying a pure function instead of running the full environment, which is an expected gain.
  • The core method uses no learned model, so its relation to program repair and regression-test selection should be discussed.
  • The main text is very compressed and full of jargon, and neither the grammar nor a worked example appears in it.
Questions for authors
  • Can the method be evaluated on change requests that were not written together with the grammar? How often does the needed change fall within the grammar?
  • What happens when the correct update should also apply inside the bypassed branch?
  • How sensitive is the condition inheritance to how the parent program is structured, e.g., dispatch tables or separate functions?
  • How does LLM editing with execution-based repair compare when it uses the same teaching executions and retention tasks?
Reviewer_VtFSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses updating an already-acquired executable workflow from two correct public executions without disturbing its other uses. The method captures a pure decision function's inputs during teaching runs, aligns them with rows of the correct report to obtain targets, enumerates a finite grammar of adapter candidates (scalar filters, string-map normalization, identifier lookup, set-membership enrichment), attaches an "inherited" applicability condition lifted from the parent's early returns, screens candidates by replay, and accepts the first consistent one after a native check. On an authored AppWorld reconciliation workflow (five model-acquired parents), an authored maintenance-scheduling workflow, ALFWorld, THOR and Minecraft, it reports 15/15 on relational updates (Table 1), evidence-registration preservation of 100% vs. 60% without the inherited condition (Fig. 4a, Table 7, Table 13), and 61.7% less validation time than native early stopping. A 150-file supplement with raw records and replay code is provided.

Critical strengths

The reported numbers are exceptionally well supported. Every cell of Table 1 reproduces from Tables 4–6; Tables 2, 8, 12–17, 25–27 and Fig. 9b reproduce cell-by-cell from reproducibility/data/records/*.json; derived figures (2,448 checks; 27.58 / 371.31 s; 61.7%; 17.1%) recompute exactly; the manifest verifies 149/149 files; the replay code contains no hard-coded outcomes or special cases; failed proposals and GPU noise are reported rather than dropped. This means my concerns are entirely about interpretation, which the authors can address.

Critical weaknesses
  1. The headline 60%→100% result is produced by a fixed candidate ordering, not by the teaching evidence. Sec. 3.3 (p.4) states that "inherited-use variants precede unrestricted variants in the fixed order," and the code confirms this is the only mechanism: src/relational_adapters.py L66–67 places existing_mode before all_modes, and reproduce.py L53 selects min(valid). In both maintenance replay inputs (data/replay/maintenance_*.json) all 12 teaching constraints are schedule-view, and archive_check.checks shows the scoped candidate 4 and the unscoped candidate 5 both accepted 12/12. The evidence cannot distinguish them; the 25/25 vs 15/25 and 12/12 vs 8/12 differences follow from an inductive bias that happens to agree with the authors' labelling rule, and the case where the prior is wrong is never examined. The title, abstract and Fig. 4a present this as an evidence-driven improvement. Since this is the paper's main empirical claim, it determines my recommendation.

  2. Generality is unestablished: the "transfer" study uses author-written parents pre-shaped to the grammar, and the main events map one-to-one onto the adapter types. In native/maintenance_spec.py, the keyword parent is the positional parent with five str.replace edits (L137–141); both contain if view == 'evidence': return windows (L114–115), exactly the early-return lifted_early_returns needs; the authored reference() (L153–155) is syntactically the grammar's scoped candidate. Table 13's 8/12→12/12 therefore tests whether the grammar finds a target placed inside it. The three relational events (App. B) correspond one-to-one to the three adapter types (Sec. 3.2), and the LLM "choice" baselines select from the same candidate pool (Sec. 4), so 15/15 for both says little beyond grammar coverage. Secondary: the abstract's 61.7% is the most favourable comparison (all 20 stages select candidate 31 of 32; for delegated approval replay is slower than early stop in 5/5 parents, 12.22 vs 11.46 s, described as "similar time"), and Table 1's replay times labelled "mean" are P1-only values (five-parent means 11.13/10.57/11.79 s).

Questions for authors
  1. Is there any experiment in which the scoped vs. unscoped choice was decided by constraints rather than by order? If not, would you reframe the abstract and Sec. 5.2 accordingly, and can you add a case where the inherited condition is wrong?
  2. Were the candidate grammar and the three relational events designed independently? Is there an evaluated update the grammar does not contain but a model baseline solves?
  3. How were the two maintenance parents written, and in what sense do they have "different decision structure" given maintenance_spec.py L137–141? Can transfer be shown on a parent the authors did not write?
  4. data/records/maintenance.json records "thinking": false, "request_temperature_zero": true (L6112, L6119), whereas Table 13 and App. G say thinking at temperature 0.6. Which was in effect?
Reviewer_RtsyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how to update reusable LLM agent workflows in response to new requirements while preserving their existing functionality. The authors identify a key limitation of example-driven updates: correct execution examples can constrain the desired behavior on demonstrated inputs without specifying the applicability conditions under which a change should be applied. To address this issue, the paper proposes an evidence-guided update framework with two main components. First, it constructs bounded candidate modifications to existing decision routines and combines execution-derived constraints with applicability conditions inherited from the original program. Second, it captures decision inputs from teaching executions and uses pure decision replay to evaluate candidate programs before accepting the selected update through native execution. The authors evaluate their approach primarily on authored business-reconciliation and maintenance-scheduling workflows, with additional studies on workflow reuse and embodied environments.

Critical strengths
  1. Identifying an important problem in maintaining reusable agent workflows. The paper correctly distinguishes between satisfying new execution examples and preserving the semantics of existing workflow uses. Its observation that changing shared inputs can silently affect downstream consumers, even when their code remains unchanged, is practically relevant. The explicit treatment of inherited applicability conditions provides a useful perspective on workflow maintenance beyond simply generating a replacement program or skill.

  2. A concrete and interpretable update mechanism with encouraging controlled results. The combination of bounded program modifications, inherited conditions, execution-based candidate screening, and native acceptance offers a relatively transparent alternative to unrestricted LLM-driven rewriting. The ablation studies provide evidence that applicability conditions contribute to behavioral preservation, while the matched validation experiments isolate the computational benefit of replay. In particular, the reported reduction in validation time against native execution with early stopping supports the efficiency motivation under the evaluated settings.

Critical weaknesses
  1. The presentation makes it unnecessarily difficult to assess the method and its contribution. The core algorithm is described through dense terminology, multiple interacting program transformations, and several specialized experimental branches, while the relationship between the main algorithmic components and the different evaluation settings is not consistently clear. In particular, the paper moves between candidate construction, inherited-condition binding, historical-rule refinement, and validation-cost accounting without giving readers a sufficiently simple end-to-end example that connects these pieces. This weakens the paper’s accessibility and makes it difficult to determine which components are essential, which assumptions are required, and how the method should be applied in practice.

  2. The method relies on restrictive structural assumptions, and its general applicability remains unclear. The framework requires the existing workflow to contain an identifiable decision boundary that can be extracted and replayed safely. It further relies on supported program structures, a finite hand-designed adapter grammar, and applicability conditions that can be recovered from eligible conditional returns. These assumptions substantially constrain the class of updates the method can handle. It is unclear how often realistic agent workflows satisfy these requirements, or how the approach handles more complex control flow, stateful decision logic, dynamically constructed arguments, and dependencies between multiple decision points. Consequently, the proposed framework currently appears closer to a specialized program-update procedure than a general solution for maintaining reusable LLM agent workflows.

  3. The evaluation provides limited evidence of generalization beyond the authors’ constructed settings. The main results rely heavily on a small collection of authored workflows, a limited number of parent programs, and relatively small evaluation sets. Multiple task views and repeated evaluations of the same underlying batches provide useful diagnostic evidence, but do not necessarily constitute independent evidence of generalization. Moreover, the embodied experiments primarily evaluate particular execution or interface corrections, and some results rely on supplied programs or manually constructed components. These experiments therefore provide limited additional evidence that the central evidence-guided update mechanism generalizes to substantially different agent domains. Broader evaluation on independently developed workflows and more diverse program structures would be needed to substantiate the paper’s general claims.

  4. The contribution relative to existing program synthesis and execution-guided repair techniques is not sufficiently isolated. The framework combines several established ideas, including finite-space program modification, execution-based candidate filtering, regression-style behavioral preservation, and reuse of intermediate execution results. The paper demonstrates that this combination works in its chosen setting, but does not fully establish which aspects constitute a substantive methodological advance over existing constrained program synthesis and repair approaches. In particular, the replay component primarily changes how candidate programs are evaluated, whereas the update component depends on a manually specified transformation space and extracted conditions. The experiments would be more convincing if they established clearer boundaries between the benefits of these individual design choices and the benefits obtainable from simpler program-specific repair or synthesis procedures.

Questions for authors
  1. What is the practical failure behavior when an update cannot be safely applied? The framework retains the original program when no candidate passes acceptance. In a deployed agent, however, retaining the parent may mean that the newly requested functionality remains unavailable. How do the authors envision handling this trade-off between preserving established behavior and implementing the new requirement? Could the framework expose unresolved ambiguity or request a discriminating execution before deciding whether to install an update?

  2. How should the reported efficiency gains be interpreted in terms of end-to-end deployment cost? The replay experiments show reduced incremental validation time for identical final programs. Beyond the measured update-time savings, how much preprocessing and engineering effort is required to extract the decision boundary, define the supported adapters, and recover inherited conditions for a new workflow? Can the authors quantify these setup costs separately and explain when they are amortized over repeated updates?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.