- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
The paper studies how to update a reusable agent workflow program when requirements change, without breaking its other uses. The method starts from the existing program and two correct executions. It records the inputs of a pure decision function and enumerates candidate input adapters from a small typed grammar. It then restricts each adapter with conditions taken from the existing program's branches and selects the first candidate that reproduces the target outputs when the decision function is replayed. A final run in the real application environment confirms the choice. On reconciliation and maintenance workflows written by the authors in AppWorld, the method solves all relational updates in about 11–12 seconds, versus 480–630 seconds for LLM-based selection. The inherited conditions raise evidence-registration success from 45/75 to 75/75. When both select the same final program, replay is 61.7% faster than validating in the real environment.
Critical strengths
S1. The problem is well motivated and under-studied (Fig. 1). Using the parent program's own conditions to decide where a change applies is an appealing idea. Table 7 isolates its effect cleanly: when only the condition is removed, all teaching and audit checks still pass, but evidence registration falls from 75/75 to 45/75.
S2. The experimental accounting is careful and transparent. All draws are reported, including failures, costs are itemized, and frozen programs are re-evaluated on fresh instances.
Critical weaknesses
W1. The evaluation is built around the method, so its generality is not established.
- The tasks, parent programs and grammar are all written by the authors. The three relational updates map one-to-one onto the grammar's three adapter families. The parent programs also encode their different uses in exactly the form the method extracts.
- When the needed change falls outside the grammar, the method does not help: in THOR, none of 54 fresh acquisition paths yields an accepted update (Table 37).
- Because inherited-scope candidates are always tried first, the method prefers the conservative scope. Yet no experiment includes an update that should apply inside the bypassed branch.
- Each event has only five underlying cases, each viewed in three ways.
W2. The baselines are weak, and the presentation obscures the contribution.
- The strongest control is an LLM choosing among the same candidates without executing them. It already matches replay's quality in most cases (Table 1), so the gain is mainly cost. Fairer comparisons would be LLM editing combined with execution-based repair, or a stronger model.
- The 61.7% saving comes from replaying a pure function instead of running the full environment, which is an expected gain.
- The core method uses no learned model, so its relation to program repair and regression-test selection should be discussed.
- The main text is very compressed and full of jargon, and neither the grammar nor a worked example appears in it.
Questions for authors
- Can the method be evaluated on change requests that were not written together with the grammar? How often does the needed change fall within the grammar?
- What happens when the correct update should also apply inside the bypassed branch?
- How sensitive is the condition inheritance to how the parent program is structured, e.g., dispatch tables or separate functions?
- How does LLM editing with execution-based repair compare when it uses the same teaching executions and retention tasks?