- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Summary
This paper proposes WebAct, a framework that enables web agents to learn reusable action interfaces from interaction traces without updating model weights. Each learned operation contains a parameterized plan, target-resolution rules, invocation conditions, and checks on the intended effect. Successful attempts contribute reusable operations, while failed attempts provide constraints that inform subsequent action selection. The method is evaluated on held-out WebArena tasks and additional web settings, with results showing improved task completion and reduced token usage.
Critical strengths
Strengths
-
The paper identifies an important limitation of existing skill-learning approaches. Learning a reusable procedure is not sufficient if the agent cannot determine when the procedure applies or whether its execution produces the desired outcome. WebAct addresses these issues by integrating invocation conditions and effect checks into the learned action representation. This makes the proposed interface more operationally explicit than a collection of unqualified reusable routines.
-
The framework combines several learning signals into a coherent test-time learning loop. The method uses successful traces to expand the action interface, failed attempts to derive constraints, and invocation outcomes to update action preferences. The resulting system learns both what operations are available and when they should be selected, while keeping the underlying model weights fixed. The component ablations provide evidence that invocation conditions and effect checks can materially affect task performance.
-
The empirical evaluation includes both performance and efficiency measurements. On 613 held-out WebArena tasks, WebAct improves task success from 20.39% to 27.08% while reducing reported action calls and tokens per task. The additional evaluations on WebVoyager and the analysis of incorrect skill invocation help establish that the method's benefits are not limited to a single aggregate benchmark result.
Critical weaknesses
Weaknesses
-
The distinction from prior skill-learning and tool-learning approaches could be more sharply established. The paper discusses existing methods that learn executable skills, validate routines, or filter them using preconditions. WebAct's contribution appears to lie in the particular combination of parameterized plans, applicability conditions, effect checks, and feedback-driven interface updates. The paper would benefit from a more direct comparison that isolates which part of this combination provides a capability that previous approaches cannot achieve, beyond an improvement from integrating several established mechanisms.
-
The strongest mechanism-level evidence is concentrated in a limited setting. The Reddit ablations show large effects when invocation conditions or effect checks are removed, but such results do not necessarily establish the same mechanism's contribution across different applications. More systematic component ablations across multiple websites would clarify whether the observed effects are general properties of the design or are partly driven by particular application interfaces and task distributions.
-
The reliability of learned conditions under environment changes remains an important open question. The learned invocation and effect predicates are derived from observed interaction traces. Although the paper provides evidence of within-application transfer, predicates inferred from a limited set of source interactions may become overly specific, stale, or misleading when page structure and behavior change. Stronger evaluation under substantial interface changes, unseen layouts, and misleading or incomplete effect signals would help establish the robustness of the learned interface.
Questions for authors
-
Clarification In the frozen-interface evaluation, the paper states that induction stops while confidence updates and operation retirement remain active. Could the authors clarify whether task outcomes from the held-out evaluation stream are used to update action preferences or retire operations, and when this feedback becomes available to the agent? This distinction matters for interpreting the results as frozen-interface transfer versus online adaptation during evaluation.
-
Assessment-critical evidence WebAct's central contribution is the combination of reusable plans, invocation conditions, effect checks, and failure-derived constraints. Could the authors clarify how much of the improvement over prior skill-learning methods remains when the compared systems use the same source traces, warm-up tasks, backbone, and evaluation budget? In particular, which controlled results most directly isolate the contribution of the verifiable interface design, rather than the benefits of learning reusable operations more generally?
6 comments
W1 & Q2. Controlled evidence isolating the verifiable interface design
Direct Answer:
To isolate the contribution of our verifiable interface (Pre+Post+Locate) from general skill reuse or extra exploration, we provide two strictly controlled comparisons:
-
Identical Source Traces, Backbone, and Budget (WebVoyager, Table 1): Using the exact same exploration traces generated by SkillWeaver, the same Gemini-2.5-Flash backbone, and the same 15-decision budget, WebAct induces 6
ActSpecs and solves 46/129 (35.7%) tasks vs. SkillWeaver's 26/129 (20.2%) (+20 tasks / +15.50 pp). This holds exploration experience constant and directly isolates the benefit of structuredActSpeccontracts. -
Controlled Gate Ablations Retaining Identical Executable Routines (GitLab & Shopping): To isolate
PreandPostchecks from the benefit of reusable macro-actions (Plan+Locate), we evaluated variants starting from the exact same warm-up library, backbone, and budget:Gate Configuration GitLab (144 tasks) Shopping (150 tasks) Full ActSpec(Pre+Post)54 (37.50%) 42 (28.00%) Disable Pre(Postonly)25 (-29 tasks / -20.14 pp) 27 (-15 tasks / -10.00 pp) Disable Post(Preonly)47 (-7 tasks / -4.86 pp) 23 (-19 tasks / -12.67 pp) Disable Both ( Plan+Locateonly)22 (-32 tasks / -22.22 pp) 18 (-24 tasks / -16.00 pp) Primitive-Only Baseline (AgentOccam) 38 (26.39%) 25 (16.67%) Critically, disabling both gates performs worse than the primitive baseline on both apps (22 vs. 38; 18 vs. 25), proving that reusable macro-actions alone are insufficient—and even harmful—without runtime applicability and effect verification.
W2. Mechanism evidence beyond Reddit
We completed full Pre/Post ablations on GitLab (code collaboration) and Shopping (e-commerce), which together account for 33 of the 41 net task gains in Table 3. As shown in the table above, both domains confirm the exact same mechanism pattern observed on Reddit: the full contract outperforms every ablated variant, while the relative impact of Pre vs. Post reflects domain structure (Pre prevents cross-view misinvocations in GitLab, while Post verifies search/filter state transitions in Shopping).
Response to Reviewer rPDE
We thank you for your insightful review and for highlighting the core distinction between learning how to execute a routine and learning when to apply and verify it. Below, we directly address your questions and provide the requested cross-application evidence.