- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The system translates agent execution trajectories into reusable programs, where code manages the stable routines and neural models manage GUI-grounding and state-checking. The NSPI uses task-completion and step-level judgments, agent execution following failed roll-outs, and code repair to incrementally improve the programs. The experiments investigate the reliability of the repeated execution and number of executions necessary to pay off the construction costs on OSWorld-Verified and ScienceBoard. They also investigate the generalization capability from one instance of the workflow to another within OSWorld task families. Other experiments evaluate the pre-action verification and longer workflows on OSWorld-V2.
Critical strengths
-
Evaluation deals with practical needs of recurring workflows. In four benchmark-model pairs, NSPI enhances Pass^3 over the baseline agent by 3.6 to 15.8 percentage points. Its slight disadvantage in terms of Pass^1 in both OSWorld settings is presented as well. I appreciate such differentiation of single-run performance and stability in repetitive runs (Table 1).
-
Expenses and failure cases are explicitly mentioned. Table 3 presents construction and execution expenses; monetary cost-break-even points are about 3 to 27 runs. Construction failure stays in the denominator. Failure recovery, transfer between workflow instances, and decreased Pass^1 rate in OSWorld-V2 reveal areas of application of this technique.
Critical weaknesses
-
Contribution of individual components and budget comparability is not clear yet. Batch2Code enables iterative improvement overall but not for ablation study of dual judges and agent continuity; judge correctness in Section 4.7 cannot serve this purpose. Moreover, in A.3 the number of steps is equal for rollout of code and the baseline agent but Table 8 mentions the limit of 5×/16× for ScienceBoard. The scope and units of measurement of their budgets have to be clarified before budget equality assessment.
-
Benefits of full deployment are not yet verified. In Section 4.6, two verifiers miss 17/24 and 6/24 actions which require blockage and blockage of the correct workflow. The evaluation includes intention comments and changes in rubric. It requires end-to-end success, cost, and latency with verification, false blockage, and fallback before extending the efficiency claims to deployment under guard.
Questions for authors
-
Are the 5×/16× bounds from Table 8 applicable to final evaluation? In the affirmative, please give the matched-budget figures; otherwise, resolve the A.3. Can you also prune each judge and agent continuations given the matched budget construction?
-
Is the verification process included in the primary execution costs? What calibration data was used for θ=0.78, and what data was used for the rubric revision? Overall success, cost, latency, and break-even counts, including comments generation, verification, false blocks, and fallback.
-
The variation generator takes the evaluation scripts into account, and the families chosen should have at least one base-agent success. How do these restrictions affect the generalizability of the findings? Please list the variants and confidence intervals, as estimated at the task family level. The interpretation of the AutoRPA results is, thus, a comparison of these particular workflow variants.
2 comments
Response to Reviewer 7epF comments (Part 2)
How do variant generation and family selection affect the generalization findings? (Q3)
Evaluation-script access. The variant generator receives the original evaluation script to keep each variant’s specification and evaluator consistent, but this doesn’t impact generalization results as both methods don't use the evaluation script during exploration and refinement. After construction on the generated variants, the policies are frozen and tested on the held-out original task using its benchmark evaluator.
Task-family selection. We focus on families whose original benchmark task was solved at least once by a base agent to help disentangle policy generalization from underlying task difficulty. If the base agent cannot solve the original task, failures of policies seeded from that agent may reflect its limitations rather than failures to transfer across variants. This setup also accommodates AutoRPA’s reliance on a successful seed trajectory, whereas a successful seed is not required for NSPI. Both methods conduct their own exploration on the generated variants and are not given the successful original-task trajectory.
Task-family-level confidence intervals. NSPI’s advantage is statistically supported on parameter variations, while the initial-state estimate favors NSPI by 8.6 percentage points but has greater uncertainty (95% CI: −8.6 to 25.7). Motivated by the judge-annotation analysis in Section 4.7, we also repeated policy construction after a simple change to the step judge’s inputs: providing program return values. This raises Pass^3 from 55.0% to 57.5% on parameter variations and from 60.0% to 71.4% on initial-state variations. We will add the full variant list and confidence-interval estimation details to the appendix.
| Variation (families) | AutoRPA | Original NSPI | Difference [95% CI] | NSPI (judge with return values) | Difference [95% CI] |
|---|---|---|---|---|---|
| Parameter (40) | 37.5 | 55.0 | +17.5 [5.0, 32.5] | 57.5 | +20.0 [5.0, 35.0] |
| Initial state (35) | 51.4 | 60.0 | +8.6 [−8.6, 25.7] | 71.4 | +20.0 [0.0, 40.0] |
Response to Reviewer 7epF comments (Part 1)
We thank the reviewer for the insightful comments and questions. We have addressed all your comments below.
How are execution budgets matched, and what does each refinement component contribute? (W1/W2/Q1)
On ScienceBoard, we use the 5×/16× limits to match the code replay budget to the base agent’s execution budget. ScienceBoard limits the base agent in planning calls, and each call can issue several primitive actions. For a task allowing n planning calls, we cap each code rollout at 5 × n state-changing primitive actions for Terra and 16 × n for Opus 5 based on inspecting the model baseline runs (Table 8). The same replay cap applies during refinement and final evaluation. We will clarify the two units in Appendix A.3 and Table 8.
Contribution of each component. We conducted component-wise ablations on 60 OSWorld tasks with Opus 5. Starting from the initial neuro-symbolic policy, we measured the gains from adding judge-guided refinement and then agent continuation during refinement. Both components improve reliability, with judge-guided refinement providing the larger Pass^3 gain. The optional pre-action verifier was disabled for the main results, and we discuss its deployment role and costs below.
What does pre-action verification add at deployment, and what do the reported costs include? (W3/Q2)
The role of the pre-action verifier in practical deployment. We introduced the pre-action verifier as an optional check for practitioners who need to guard a neuro-symbolic policy against drifts in environment dynamics. It can block a state-changing action if the current screen no longer matches the action’s preconditions and hand control to an agent or a human. Section 4.6 evaluates its accuracy separately in an offline setting because it is a separate deploy-time decision for practitioners to make. We ran all main results, including the reported costs, with verification disabled because OSWorld and ScienceBoard do not exhibit environment nonstationarity, and we will make that distinction explicit in the manuscript. The θ=0.78 was calibrated by sweeping θ for wrong-block rate ≤5% on 387 human-labeled steps from out-of-distribution agent runs: OSWorld-style Chrome tasks on live webpages on a Windows operating system. We also utilize this offline set for informed rubric revisions.
What does a verifier block imply for performance and cost? A verify pass lets the policy execute the same action it would take during ordinary replay. A block hands the remaining task to the base agent or a human. After an agent handoff, success depends on whether the agent can complete the task from the current state. Useful progress made before the block can reduce the remaining agent work. Guarded replay retains a per-run cost advantage when the policy steps already executed, verifier calls, and any fallback together cost less on average than running the base agent from the start.