- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
HxAgent executes a natural-language web task step by step, using the current page, executable elements, recent action history, and prior experience to produce a script that can be replayed without an LLM. It achieves 97.4% Exact-Match on the shared set of 975 MiniWoB++ instances. When both generation and successful replay are required, end-to-end success is 78.9% on the real-world dataset and 38.2% on OnlineMind2Web
Critical strengths
- The paper measures both action-sequence accuracy and script replay, exposing the gap between generating a plausible sequence and obtaining a reusable script.
- Stepwise state checks, feasible-action constraints, and memory have clear roles; ablations show lower accuracy when memory or iterative planning is removed.
- Evaluation includes controlled and real websites, with concrete analyses of failures on dynamic interfaces and date pickers.
Critical weaknesses
- The output is an action script without test assertions or an independent test oracle. Completing a workflow does not yet demonstrate defect detection.
- XPath locators can break under layout changes or loading delays; OnlineMind2Web end-to-end success is 38.2%.
Questions for authors
- How much of HxAgent’s advantage remains when the main baselines use a shared candidate-element set?
- How could the system add checkable assertions, and what are its defect-detection rate and script survival rate across application versions?
- Please clarify how the MiniWoB++ sets of 975 and 1,100 instances relate, and reconcile the stated gains over WALT with the table values.
Response to Reviewer xpco
Q1: We thank the reviewer for raising this important question. HxAgent's effectiveness stems from two complementary mechanisms: (1) extracting and grounding actions in executable UI elements to reduce invalid action selection, and (2) iterative, memory-aware planning that reasons over evolving webpage states to handle multi-step interactions.
We acknowledge that the existing baselines use their respective candidate-element extraction mechanisms, which may contribute to the observed performance differences. However, our ablation study in Table 6 provides evidence that HxAgent's improvements are not solely attributable to candidate extraction. All ablation variants retain the same candidate-element extractor, yet removing iterative planning reduces Exact-Match from 97% to 75%, removing experience reduces it to 88%, and removing both short-term memory and experience reduces it to 47%. These results demonstrate the independent importance of iterative planning and memory under a fixed extraction mechanism.
Q2: We thank the reviewer for raising these important questions. As acknowledged in Appendix A (Limitations, lines 660–667), HxAgent currently focuses on generating executable, replayable action sequences and test scripts rather than test script robustness, independent assertions or test oracles.
Checkable assertions and defect detection. HxAgent could be extended to generate assertions from the task description and observed webpage states, such as verifying expected text, element visibility, navigation outcomes, or application-state changes after specific actions. These assertions could be inserted into the generated Selenium scripts and checked automatically during replay. However, the current implementation does not generate such assertions, and we therefore do not report a defect-detection rate. We will clarify that successful workflow execution should not be interpreted as evidence of defect-detection effectiveness.
Script survival across application versions. As discussed in lines 665–667, HxAgent currently relies on XPath locators, which may become invalid under layout changes, dynamic content, or loading delays. Our replay evaluation (Table 5) reports success rates of 100%, 90.8%, and 77.3% on MiniWoB++, Real-world, and OnlineMind2Web, respectively, for successfully generated action sequences. However, these results measure replay reliability in the evaluated environments, not survival across application versions. We have not conducted a longitudinal cross-version study and therefore cannot report a script survival rate across versions. Existing robust locator-generation techniques (Leotta et al., 2016; Nguyen et al., 2021; Nass et al., 2023) could be integrated to improve script resilience.
We will strengthen the limitations discussion to distinguish action-sequence generation, assertion-based defect detection, replay reliability, and cross-version script robustness.
Q3: Our MiniWoB++ evaluation comprises 1,100 instances from 44 tasks, with 25 instances per task, divided into two disjoint subsets.
First, Table 3 reports results on 975 instances from 39 tasks, representing the shared evaluation subset used for comparison with the principal baselines. On this subset, HxAgent achieves 97.4% Exact-Match, compared with 86.9% for WALT, corresponding to an improvement of 10.5 percentage points.
Second, Table 4 evaluates an additional 125 challenging instances from five tasks involving complex UI interactions, such as date pickers, flight booking, and autocomplete fields. These tasks require navigation across multiple screens and longer action sequences. HxAgent achieves 81.6% Exact-Match, whereas WALT achieves 0%, demonstrating a substantial performance difference on these challenging interactions.
Thus, the 975 and 125 instances constitute the complete set of 1,100 MiniWoB++ evaluation instances, and the reported 10.5-point improvement over WALT specifically refers to the shared 975-instance subset. We will clarify this distinction in the experimental methodology and revise the corresponding performance discussion to avoid ambiguity.