acceptodds
Under review as a conference paper at ICLR 2027

HxAgent: Iterative LLM Agent Planning for Generating Replayable Web Test Scripts from Natural Language

Abstract

Turning natural-language functionality descriptions into executable test scripts is a labor-intensive step of web UI testing. We present HxAgent, an iterative LLM-based planning agent that executes a described task in a browser and emits a replayable, locator-based test script that runs without any LLM at replay time. After every action, a dedicated state evaluator reassesses progress and can stop a diverging trajectory, and the next action is chosen only among executable elements, conditioned on (1) the current observation, (2) a short-term memory of prior state–action pairs, and (3) examples and textual rules accumulated in context from HxAgent’s own earlier successful and failed attempts; no model parameters are updated. HxAgent reaches 97.4% Exact-Match on MiniWoB++ (10.5% above a state-of-the-art WALT), 83.8% on 350 real-world tasks (13.4% above WALT), and 50.7% on OnlineMind2Web (13.5% above WALT). Counting only scripts that are generated and successfully replayed, it succeeds end-to-end on 78.9% of real-world and 38.2% of OnlineMind2Web tasks.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Web Testing, Test Script Generation

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_xpcoOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HxAgent executes a natural-language web task step by step, using the current page, executable elements, recent action history, and prior experience to produce a script that can be replayed without an LLM. It achieves 97.4% Exact-Match on the shared set of 975 MiniWoB++ instances. When both generation and successful replay are required, end-to-end success is 78.9% on the real-world dataset and 38.2% on OnlineMind2Web

Critical strengths
  • The paper measures both action-sequence accuracy and script replay, exposing the gap between generating a plausible sequence and obtaining a reusable script.
  • Stepwise state checks, feasible-action constraints, and memory have clear roles; ablations show lower accuracy when memory or iterative planning is removed.
  • Evaluation includes controlled and real websites, with concrete analyses of failures on dynamic interfaces and date pickers.
Critical weaknesses
  • The output is an action script without test assertions or an independent test oracle. Completing a workflow does not yet demonstrate defect detection.
  • XPath locators can break under layout changes or loading delays; OnlineMind2Web end-to-end success is 38.2%.
Questions for authors
  1. How much of HxAgent’s advantage remains when the main baselines use a shared candidate-element set?
  2. How could the system add checkable assertions, and what are its defect-detection rate and script survival rate across application versions?
  3. Please clarify how the MiniWoB++ sets of 975 and 1,100 instances relate, and reconcile the stated gains over WALT with the table values.
1 comment
AuthorsOpenReview ↗

Response to Reviewer xpco

Q1: We thank the reviewer for raising this important question. HxAgent's effectiveness stems from two complementary mechanisms: (1) extracting and grounding actions in executable UI elements to reduce invalid action selection, and (2) iterative, memory-aware planning that reasons over evolving webpage states to handle multi-step interactions.

We acknowledge that the existing baselines use their respective candidate-element extraction mechanisms, which may contribute to the observed performance differences. However, our ablation study in Table 6 provides evidence that HxAgent's improvements are not solely attributable to candidate extraction. All ablation variants retain the same candidate-element extractor, yet removing iterative planning reduces Exact-Match from 97% to 75%, removing experience reduces it to 88%, and removing both short-term memory and experience reduces it to 47%. These results demonstrate the independent importance of iterative planning and memory under a fixed extraction mechanism.

Q2: We thank the reviewer for raising these important questions. As acknowledged in Appendix A (Limitations, lines 660–667), HxAgent currently focuses on generating executable, replayable action sequences and test scripts rather than test script robustness, independent assertions or test oracles.

Checkable assertions and defect detection. HxAgent could be extended to generate assertions from the task description and observed webpage states, such as verifying expected text, element visibility, navigation outcomes, or application-state changes after specific actions. These assertions could be inserted into the generated Selenium scripts and checked automatically during replay. However, the current implementation does not generate such assertions, and we therefore do not report a defect-detection rate. We will clarify that successful workflow execution should not be interpreted as evidence of defect-detection effectiveness.

Script survival across application versions. As discussed in lines 665–667, HxAgent currently relies on XPath locators, which may become invalid under layout changes, dynamic content, or loading delays. Our replay evaluation (Table 5) reports success rates of 100%, 90.8%, and 77.3% on MiniWoB++, Real-world, and OnlineMind2Web, respectively, for successfully generated action sequences. However, these results measure replay reliability in the evaluated environments, not survival across application versions. We have not conducted a longitudinal cross-version study and therefore cannot report a script survival rate across versions. Existing robust locator-generation techniques (Leotta et al., 2016; Nguyen et al., 2021; Nass et al., 2023) could be integrated to improve script resilience.

We will strengthen the limitations discussion to distinguish action-sequence generation, assertion-based defect detection, replay reliability, and cross-version script robustness.

Q3: Our MiniWoB++ evaluation comprises 1,100 instances from 44 tasks, with 25 instances per task, divided into two disjoint subsets.

First, Table 3 reports results on 975 instances from 39 tasks, representing the shared evaluation subset used for comparison with the principal baselines. On this subset, HxAgent achieves 97.4% Exact-Match, compared with 86.9% for WALT, corresponding to an improvement of 10.5 percentage points.

Second, Table 4 evaluates an additional 125 challenging instances from five tasks involving complex UI interactions, such as date pickers, flight booking, and autocomplete fields. These tasks require navigation across multiple screens and longer action sequences. HxAgent achieves 81.6% Exact-Match, whereas WALT achieves 0%, demonstrating a substantial performance difference on these challenging interactions.

Thus, the 975 and 125 instances constitute the complete set of 1,100 MiniWoB++ evaluation instances, and the reported 10.5-point improvement over WALT specifically refers to the shared 975-instance subset. We will clarify this distinction in the experimental methodology and revise the corresponding performance discussion to avoid ambiguity.

Reviewer_ZSFeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents HxAgent, an iterative LLM agent that converts natural-language web tasks into replayable, locator-based test scripts requiring no LLM during replay. It combines Sequence Action Memory, long-term experience containing successful trajectories and textual rules from prior attempts, and a dedicated State Evaluator, without updating model parameters. The paper also introduces a dataset of web tasks and action sequences. Experiments on MiniWoB++, real-world websites, and OnlineMind2Web report improved Exact-Match and evaluate script replay, with component ablations and token-consumption analysis.

Critical strengths
  • The paper evaluates executable script replay without an LLM, rather than task completion alone. The reported end-to-end success rates of 78.9% on real-world tasks and 38.2% on OnlineMind2Web provide direct evidence of practical utility for repeated web execution.

  • The ablations show lower Exact-Match after removing experience or iterative planning. These results provide initial support for the integrated design beyond aggregate baseline comparisons.

Critical weaknesses
  1. Consistency of the reported results.

    • Table 2's task counts and Exact-Match scores imply approximately 49.5%, rather than 50.7%, consistent with the 141/285 successful SoAs in Table 5.
    • In Table 9, Li et al.'s 949/975 correct runs imply 97.3%, rather than 91.7%.
    • These discrepancies need clarification before the reported improvements can be assessed reliably.
  2. Technical contribution and supporting ablations.

    • Distinction from prior work: The differences between HxAgent's iterative planning and experience reinforcement and prior methods such as AdaPlanner and Li et al. need clearer explanation.
    • Component contributions: Table 6 does not separately ablate the State Evaluator or vision enhancement, or distinguish textual rules from successful SoA exemplars.
    • Proactive correction: Algorithm 1 shows early termination of diverging trajectories, but its relationship to "proactive correction" remains unclear. Evaluating incorrect early termination would help assess the benefit of this mechanism.
Questions for authors
  1. Could you reconcile the Exact-Match percentages and successful-run counts in Tables 2, 5, and 9, and clarify whether this changes the reported baseline comparisons?

  2. What specifically distinguishes HxAgent's iterative planning and experience reinforcement from AdaPlanner and Li et al.?

  3. How does "proactive correction" operate beyond early termination in Algorithm 1, and how often does the State Evaluator incorrectly terminate a task?

  4. What evidence separates the contributions of the State Evaluator, vision enhancement, and textual rules from those of successful SoA exemplars?

2 comments
AuthorsOpenReview ↗

Response to Reviewer ZSFe

Q1: We have re-examined the results and will correct the affected tables and texts.

Tables 2 and 5 (OnlineMind2Web). The reported 50.7% Exact-Match in Table 2 is the macro-average across three difficulty groups: Easy (81.3%), Medium (40.7%), and Hard (30.0%). Because these groups contain different numbers of tasks (80, 135, and 70, respectively), the instance-weighted average is approximately 49.5%, consistent with the 141/285 successful trajectories used for replay evaluation in Table 5. We will report both macro- and instance-weighted averages and clarify that Table 5 evaluates script replay on successfully generated trajectories, rather than averaging accuracy across difficulty groups.

Table 9 (MiniWoB++): We acknowledge inconsistencies. Using the stated denominator of 975, Li et al.'s 949 successful runs correspond to 97.3%, rather than 91.7%, while Synapse's 958 successful runs correspond to 98.3%, rather than 97.3%.

Impact on baseline comparisons: These corrections do not change the reported HxAgent results, but its advantage over Li et al. on MiniWoB++ becomes smaller, and Synapse achieves a slightly higher overall accuracy. We will revise the corresponding discussion.

Q2: Compared with AdaPlanner, which adaptively refines plans through in-plan and out-of-plan feedback, HxAgent selects each action from executable UI elements based on the updated webpage state. HxAgent also employs a dedicated State Evaluator, separate from action grounding, to assess task completion and trajectory divergence after each action. While AdaPlanner reuses successful plans as skills or exemplars, HxAgent extracts reusable behavioral rules from both successful and failed trajectories, combining them with successful exemplars and short-term state–action memory.

Compared with Li et al., which uses compact screen representations, staged planning, and structured reflection, HxAgent separates intermediate-state evaluation from action selection. It also combines current state–action history with reusable rules derived from previous successes and failures to guide subsequent actions.

Finally, unlike these approaches, which primarily focus on task completion, HxAgent generates executable, locator-based test scripts that can be replayed without further LLM calls. Thus, our contribution lies in integrating dedicated state evaluation, experience-derived rules, and executable action grounding for replayable web-script generation.

Q3: We will change “proactive correction” into “proactive divergence detection and early termination”.

In HxAgent, the mechanism operates at two levels.

First, within an execution, the dedicated State Evaluator checks the evolving state–action trajectory after each interaction for task completion or significant divergence. Diverging attempts are terminated early, while viable executions continue with actions selected based on the updated webpage state and short-term memory.

Second, across attempts during experience construction, unsuccessful trajectories are analyzed to extract behavioral rules that guide subsequent attempts and help avoid repeated mistakes. During evaluation, the experience base remains frozen.

To assess the State Evaluator's reliability, we additionally analyzed its early-termination decisions to identify prematurely terminated but recoverable trajectories. Across 100 evaluation trajectories, the State Evaluator triggered 25 early terminations, of which 24 were identified as incorrect because the tasks remained recoverable. This corresponds to a false early-termination rate of 96% among early-terminated trajectories (24/25), or 24% of all evaluated trajectories (24/100). These results quantify the risk of prematurely terminating recoverable executions.

Q4: Our ablation study demonstrates the contributions of experience and iterative planning, but does not fully isolate the effects of individual mechanisms. Thus, we conducted additional controlled ablations using the same LLM, action extractor, and interaction-step budget:

Successful exemplars only: Retaining successful SoA exemplars while removing extracted textual rules yields 90.2% Exact-Match, compared with 97% for the full framework. This isolates the incremental contribution of textual rules beyond successful exemplars.

Completion-only evaluator: Replacing the full State Evaluator with one that checks only task completion, without detecting divergence, yields 92% Exact-Match. This measures the contribution of proactive divergence detection and early termination.

Without vision enhancement: Replace dedicated evaluator in HxAgent with a naive evaluation preserving iterative planning and memory and removing vision yields 78.6% Exact-Match, isolating the contribution of vision.

Together, these ablations distinguish the contributions of textual rules, intermediate-state evaluation, divergence detection, and vision enhancement from those of successful SoA exemplars.

Thank you for the response. The added ablations (Q4) addressed my question. However, Q1 and Q3 raised further concerns, and I keep my score.

Q1. The corrected numbers changed the MiniWoB++ conclusion rather than leaving the comparison intact. With the stated denominator of 975, Li et al.'s 949 runs give 97.3% against HxAgent's 950 (97.4%), a difference of one run rather than the 5.7 points stated in the abstract, and Synapse's 958 give 98.3%, above HxAgent. As corrected, HxAgent neither outperformed Li et al. nor matched the best demonstration-based method on MiniWoB++, and I would ask how the claims in the abstract and Section 5.1 will be revised. The same check also applied to rows the response did not mention: RCI's 926 runs give 95.0% rather than 89.5%, and WebNT5's 532 give 54.6% rather than 59.1%. In Table 5, 304 successful Real-world trials over 350 tasks give 86.9% rather than the 83.8% of Table 1, and 1,048 over 1,100 give 95.3% rather than the 97.4% of Table 3. Given this pattern, I would ask for a single table of raw counts from which all reported percentages are derived.

Q3. The reported figures did not appear consistent with the main results. If 24 of 100 evaluation trajectories were terminated prematurely while remaining recoverable, this is difficult to reconcile with 97.4% Exact-Match on MiniWoB++ and 83.8% on the Real-world dataset, and the response did not state which dataset these trajectories came from or whether they were retried. This also bore on the central claim: proactive correction was presented as the main distinction from reactive reflection methods, yet a 96% false early-termination rate indicated the mechanism rarely fired correctly. The new ablation pointed the same way, since removing divergence detection and early termination cost 5 points (97% to 92%), less than removing vision (78.6%).

Reviewer_dBk9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents HxAgent, which converts natural-language descriptions of web tasks into executable scripts that can be replayed without further LLM calls. The method uses GPT-4o to make decisions based on step-by-step state evaluation, executable action candidates, short-term interaction history, and examples and textual rules extracted from training trajectories. The authors evaluate the method on MiniWoB++, a self constructed real world website task set, and OnlineMind2Web.

Critical strengths

The research objective is valuable. Converting an initial agent execution into a script that can subsequently run without model involvement can reduce reliance on language models for repeated execution. The evaluation covers both controlled environments and real-world websites and includes an analysis of replay failures. This design helps identify practical deployment challenges, such as dynamic web content and changes in element locators.

Critical weaknesses

There are inconsistencies in key experimental results, as well as potential concerns about the fairness of some comparisons. In Table 9, the Li baseline’s 949/975 successful runs correspond to 97.3%, but the reported percentage is 91.7%; Synapse’s 958/975 successful runs correspond to 98.3%, but the reported percentage is 97.3%. If the success counts are correct, HxAgent’s 950 successful runs exceed Li’s by only one and fall below Synapse’s (pp. 19–21). In anonymous implementation, HxAgent counts a displayed MiniWoB reward of ≥0 as success, whereas WALT requires a raw reward of ≥1.

Inaccuracies in the descriptions of related work and baselines affect the assessment of the method’s contribution and comparative results. The paper describes Li et al. as using full HTML, the original method uses a compact screen representation. In addition, the arXiv identifier listed for Gur et al. (2023b) actually points to Llama 2. These descriptions, references require verification (pp. 5, 10, and 18).

Questions for authors
  1. Please explain the inconsistencies between success counts and percentages in Table 9 and specify the corresponding evaluation task sets and denominators. Is the reported 50.7% on OnlineMind2Web an equally weighted average across difficulty groups?
  2. How are experience construction and evaluation separated for the real-world website tasks and OnlineMind2Web? Please confirm whether the experience remains frozen throughout evaluation and clarify how access to historical experience and attempt budgets are matched across methods.
  3. I could not find the AI Use Statement required by ICLR 2027 in the current manuscript. Please provide this statement, correct the reference for Gur et al. (2023b), and clarify the description of the Li et al. baseline
1 comment
AuthorsOpenReview ↗

Response to Reviewer dBk9

Q1: We have re-examined the reported results and identified two issues concerning numerical consistency and aggregation.

First, Table 9 contains inconsistencies between successful-run counts and reported percentages. For example, Li et al.'s 949/975 corresponds to 97.3%, rather than 91.7%, while Synapse's 958/975 corresponds to 98.3%, rather than 97.3%. We will verify the underlying results and evaluation denominators for all baselines and correct Table 9 accordingly. We will also revise the corresponding comparisons and conclusions to reflect the corrected values.

Second, the 50.7% Exact-Match reported for OnlineMind2Web in Table 2 is an equally weighted average across the three difficulty groups, namely Easy (81.3%), Medium (40.7%), and Hard (30.0%). Since these groups contain different numbers of instances (80, 135, and 70, respectively), the instance-weighted average is approximately 49.5%. Thus, the difference arises from macro- versus instance-weighted aggregation, rather than an arithmetic error in the reported macro-average.

To improve clarity and consistency, we will explicitly distinguish macro-averaged and instance-weighted results, verify the underlying counts and denominators, and update the affected tables and accompanying discussion.

Q2: We thank the reviewer for requesting clarification. For both the Real-world and OnlineMind2Web datasets, HxAgent separates experience construction from evaluation into two phases. During experience construction, the experience base is dynamically updated with successful action sequences and behavioral rules extracted from previous successful and failed attempts. After this phase, the constructed experience is selected and frozen throughout evaluation. The agent can retrieve and use this historical experience but cannot update it using evaluation outcomes. Only the short-term state–action memory is updated during each evaluation trajectory, (Appendix D).

Experience construction is specific to HxAgent; the baseline methods follow their respective original designs and are not provided with HxAgent's historical experience. For a consistent interaction budget, all methods are limited to a maximum of 30 browser interaction steps per attempt, including HxAgent. However, the number of LLM calls may differ because of their distinct planning and evaluation mechanisms.

We will clarify the separation between experience construction and evaluation, the experience-access policies, and the common interaction-step budget in the revised manuscript.

Q3: We will add the required AI Use Statement to the manuscript, clarifying that generative AI tools were used to improve the clarity and readability of the writing. All AI-assisted text was reviewed and validated by the authors, who take full responsibility for the manuscript. On lines 442-443 in the paper, our intention actually was to refer to the fact that the general LLMs including Llamma 2 (2023) and CLaude 3.5 Sonnet have been used in Computer Use. Unfortunately, we made an editorial error in the bibtex entry for Llamma 2. We will correct this citation.

Li et al. baseline: We acknowledge that our description of Li et al. (2023) as using full HTML was inaccurate. Their approach employs a compact screen representation derived from HTML, retaining task-relevant element attributes to reduce input complexity. We will correct this description and clarify that the main distinction from HxAgent lies in the use of a dedicated intermediate-state evaluator and reusable behavioral rules extracted from prior trajectories, rather than in the use of full versus compact HTML representations.

Reviewer_NJpZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HxAgent turns natural-language web tasks into executable, locator-based scripts that replay without an LLM. Its focus is automating UI workflows, rather than generating a complete test suite with assertions.

Contributions

  • Memory-aware, stepwise planning: After each browser action, it observes the new page and selects the next action from extracted interactable elements, using DOM context and screenshots to resolve ambiguity.
  • Separate state evaluation: An LLM evaluator checks completion or divergence at every step and can terminate unsuccessful trajectories early.
  • Experience reuse: Current-run state–action history is combined with successful examples and textual rules distilled from earlier successes and failures. This is in-context adaptation—not parameter training or reinforcement learning.
  • Replayable output and a dataset: Successful trajectories become Selenium/XPath scripts; the authors also contribute 350 task instances across seven real-world websites.
  • The evidence (MiniWoB++, Real-World dataset on 350 tasks, OnlineMind2Web) suppoerts a useful integration for workflow-script generation.
Critical strengths
  1. Iterative planning informed by past successes and failures. HxAgent combines short-term state–action memory with long-term examples and rules extracted from earlier attempts to guide stepwise decisions. This integration helps maintain task context and avoid repeating mistakes; the ablations provide evidence that these mechanisms meaningfully improve performance.
  2. Broad evaluation with detailed analysis. The paper evaluates the approach across three datasets spanning controlled interfaces and real-world websites, supplemented by replay experiments, component ablations, task-complexity analysis, and token-cost comparisons. This breadth strengthens the empirical case by examining not only overall accuracy, but also where the approach helps, where it fails, and whether its generated scripts remain executable.
Critical weaknesses
  1. Limited experimental detail and coverage. Experiments use only GPT-4o, with insufficient detail about the exact model version, inference settings, and execution budgets to assess reproducibility. Evaluation with additional LLMs would clarify whether the gains generalize beyond this model. The cost comparison is also limited to WALT on selected tasks, rather than all principal baselines under comparable conditions. Consequently, the evidence for model-independent effectiveness and cost efficiency remains incomplete.

  2. Limited demonstrated methodological novelty. The method is primarily a prompting and memory-management framework combining established iterative planning, action history, and reflection-derived experience. The paper does not sufficiently establish what is fundamentally new beyond this integration or isolate the benefits of its distinctive mechanisms. This weakens the methodological contribution, despite the useful application and empirical results.

Questions for authors

Clarification requests that could materially affect my assessment:

  1. Experimental protocol: Which exact GPT-4o version, sampling settings, prompt configurations, and action/LLM-call budgets were used? How were experience-building instances separated from evaluation instances, and was experience frozen during evaluation?
  2. Cost comparison: Does the reported token cost include experience building, failed attempts, state evaluation, and visual inputs? Were HxAgent and WALT evaluated under comparable budgets and caching assumptions, and are equivalent measurements available for Li et al. and SeeAct?
  3. Distinctive contribution: Beyond combining established planning and memory mechanisms, what is the principal methodological novelty? What evidence isolates the benefit of the dedicated evaluator and extracted rules, rather than successful exemplars, additional prompting, or differences in action extraction?
  4. Consistency of results: Can you reconcile the sequence Exact-Match rates with the successful-trajectory counts used for replay evaluation—for example, 83.8% versus 304/350 on the real-world dataset, and 50.7% versus 141/285 on OnlineMind2Web? Are these different aggregations or separate experimental runs?
  5. Future-work suggestion, not a clarification requirement: Evaluation with additional LLMs would strengthen the generality claim; its absence alone does not invalidate the GPT-4o results.
1 comment
AuthorsOpenReview ↗

Response to Reviewer NJpZ

Q1: We clarify that all experiments use gpt-4o-2024-08-06 (GPT-4o), with temperature = 0 and top-p = 1. The prompt configurations, including action grounding, state evaluation, short-term memory, and experience-based guidance, are provided in our project repository. We separate experience construction from evaluation. During experience construction, the agent executes task instances iteratively, accumulating successful action sequences and textual rules derived from previous successes and failures. For MiniWoB++, we use 20 training instances and retain up to eight successful examples, together with the extracted rules (Section 4 and Appendix D). The resulting experience is selected and frozen before evaluation. During evaluation, the agent can retrieve this experience but cannot modify it or incorporate evaluation outcomes; only the short-term state-action memory is updated within each execution. We impose a maximum of 30 browser interaction steps per execution. Each step invokes the state evaluator and, if execution continues, the action-grounding LLM; additional calls may be required for visual disambiguation or input generation. Thus, the interaction-step limit is distinct from the number of LLM calls.

Q2: Figure 9 reports token consumption during evaluation only, excluding the cost of constructing the experience base. The reported evaluation cost includes all calls from successful and failed attempts, state evaluation, action grounding, and visual inputs are included. Experience construction incurs an additional cost of 34,089 input tokens and 1125 output tokens on average, which is paid before evaluation and can be amortized over subsequent task executions that reuse the frozen experience. For the comparison with WALT, Figure 9 reports prompt, cached-prompt, and completion tokens separately for representative MiniWoB++ tasks. However, the methods were not necessarily evaluated under identical execution budgets or caching conditions. Thus, the comparison reflects observed token consumption rather than a strictly controlled, end-to-end cost comparison. Equivalent token measurements for Li et al. and SeeAct are unavailable.

Q3: The principal methodological contribution of HxAgent is not the introduction of planning or memory individually, but an experience-guided, proactively evaluated action-generation framework that combines two mechanisms: (1) a dedicated state evaluator that independently assesses task completion and trajectory divergence after each action, enabling early termination of unsuccessful executions; and (2) an experience-abstraction mechanism that extracts reusable behavioral rules from successful and failed trajectories to guide subsequent action selection, beyond simply reusing successful exemplars. Our ablation study (Table 6) demonstrates the importance of the overall framework: removing iterative planning reduces Exact-Match from 97% to 75%, while removing experience reduces it to 88%. However, these results do not isolate the contributions of the evaluator and extracted rules individually. To address this, we conducted additional controlled ablations using the same LLM, action extractor, and interaction budget:

  1. Successful exemplars only: Retaining successful exemplars while removing extracted rules yields 90.2% Exact-Match, compared with 97% for the full framework, demonstrating the incremental contribution of rule abstraction.
  2. Completion-only evaluator: Replacing the full evaluator with one that checks only task completion, without detecting divergence, yields 92% Exact-Match, quantifying the benefit of proactive failure detection.
  3. Naive evaluator: Replace dedicated evaluator in HxAgent with a naive evaluation preserving iterative planning and memory and removing vision yields 78.6% Exact-Match, isolating the contribution of vision. These experiments complement Table 6 by disentangling the benefits of rule abstraction, dedicated evaluation, and iterative planning.

Q4: The difference comes from different aggregation procedures, rather than inconsistent results. The Exact-Match rates reported in Tables 1 and 2 are averaged over five independent runs to account for variability in action-sequence generation. In contrast, the successful-trajectory counts in Table 5 correspond to the specific sets of correctly generated action sequences used for script generation and replay evaluation. For the Real-world dataset, 304 of 350 trajectories (86.9%) were selected for replay, compared with the five-run average Exact-Match of 83.8%. Similarly, for OnlineMind2Web, 141 of 285 trajectories (49.5%) were selected, compared with the five-run average of 50.7%. Thus, the two sets of results measure different stages of the evaluation: Tables 1–2 report average action-sequence accuracy across five runs, whereas Table 5 reports replay success on a specific set of successful trajectories.

Q5: Thank you. We will evaluate with an additional LLM in the revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.