- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the gap between performing actions and achieving goals in long web tasks. It proposes HORIZON, which connects user requirements, interface actions, and observed evidence through task completion memory, a DOM translation layer, and experience retrieval based on the current state. It also separates past suggestions from evidence for the current task. Comparisons of full agent configurations on WebArena show that HORIZON improves task success rates and supports tasks across websites. Example execution traces further show its role in locating targets, checking results based on requirements, and adapting action paths.
Critical strengths
- Task progress modeling is flexible and easy to trace. The framework organizes action records and evidence around requirements, supports multiple valid paths, and clearly tracks unfinished items.
- Actions and verification form a complete loop. The two way DOM interface keeps actions and result checks focused on the same target. It also links interaction results to updates in requirement status.
- The experience reuse mechanism is carefully designed. It uses interaction states and comparison results to produce guidance, keeps evidence and verification status, and separates experience based suggestions from verification of the current task.
Critical weaknesses
- The evaluation has limited coverage. The paper evaluates the framework only on a limited set of benchmarks, such as WebArena. It lacks experiments on other independent benchmarks, unseen websites, and conditions involving interface changes. The results therefore do not provide enough evidence that the framework generalizes to different website structures and task distributions.
- The comparison with external methods in Table 2 lacks fairness. Different methods or frameworks do not use the same models and test sets. For example, Agent-as-Annotators with A3-Qwen3.5-9B uses a subset, while HORIZON with A3-Qwen3.5-9B uses the full test set.
- The paper lacks comparisons with relevant baselines. It cites related work such as WebRider, ContractSkill, and Agent Workflow Memory, which share similar core ideas, but does not compare HORIZON with these methods in the experiments.
- The comparison with existing work needs more detail. The core ideas are similar to those of WebRider, ContractSkill, and Agent Workflow Memory, but the paper does not clearly explain the differences. A comparison table can help make these differences clearer.
- The paper lacks strict comparisons under the same budget. It does not provide complete statistics on token usage, model calls, actual execution time, and cost per task. This makes it difficult to assess the extra cost of verification and memory maintenance.
Questions for authors
See above.
3 comments
W2. Table 2 comparisons are not on the same models and test sets
We agree. Table 2 will be split into a same-model controlled block and a block of published results for reference, and each row will state its task set and number of attempts. The controlled block is the following. Each row is a separate controlled comparison: within a row, both systems use the same model, the same tasks and the same browser-step limit, and only the harness differs. Rows use different models and task sets, so numbers should be compared within a row, not across rows. "Without HORIZON" is the official baseline agent from the WebArena repository (a chain-of-thought prompt over the accessibility tree).
| Benchmark | Model | Tasks | Without HORIZON | With HORIZON | Gain (points) | Paired test |
|---|---|---|---|---|---|---|
| WebArena (paper setting) | Qwen3.8-27B | 475 | 35.6% | 45.1% | +9.5 | p = 0.0003 |
| WebArena subset (new) | gpt-6-luna | 51 | 11.1% | 49.7% | +38.6 | p = 1e-5 |
| WebArena subset (new) | Claude Opus 5.5 | 51 | 33.3% | 66.7% | +33.3 | p = 1.5e-5 |
| Online-Mind2Web (new) | gpt-6-luna | 300 | 13.3% | 36.3% | +23.0 | p < 1e-13 |
Each task is run once, except in the gpt-6-luna WebArena row, which is the mean of 3 runs per task. Run-to-run variation is small relative to the gain: over those 3 runs, HORIZON solves 25, 25 and 26 of the 51 tasks, and the baseline agent 6, 4 and 7.
- The WebArena subset has 51 tasks (Shopping 20, Shopping Admin 20, Reddit 11), drawn with a fixed seed from the single-site tasks that are scored by string or URL match.
- The row for Agent-as-Annotators (A3) will be reported on the same 381-task split that A3 uses. A3's model is fine-tuned for its own agent's prompt and action format. Inside HORIZON both change, so that row measures the model outside its training format and not the effect of the harness.
W3. No experimental comparison with WebRider, ContractSkill, Agent Workflow Memory
Agent Workflow Memory can be compared directly on WebArena, and we ran it. All rows use gpt-6-luna on the same 51 WebArena tasks.
| System | Tasks solved (of 51) | Success rate |
|---|---|---|
| WebArena baseline agent (without HORIZON) | 5.7 | 11.1% |
| Agent Workflow Memory, its default 10-step limit | 14 | 27.5% |
| Agent Workflow Memory, 30-step limit | 15 | 29.4% |
| HORIZON | 25.3 | 49.7% |
HORIZON and the baseline agent are means of 3 runs per task; Agent Workflow Memory was run once per task.
- Agent Workflow Memory solves 14 of 51 tasks with its default step limit and 15 with a 30-step limit; HORIZON solves 25 to 26 per run. In both settings the difference is significant against each of the three HORIZON runs (sign test, p ≤ 0.041).
- We used its official code and the workflow files released in its repository, without online updates. Its default step limit is 10, while HORIZON and the baseline agent use 30. For the 30-step row, we reran with a 30-step limit the 16 tasks on which it had run out of steps; one more task was solved. Its released workflows were induced on WebArena test tasks and include some of our 51 tasks.
- WebRider is evaluated only on its own RiderBench (information seeking and recommendation; purchases and irreversible actions are excluded). It releases a dataset but no runnable code.
- ContractSkill is evaluated on a 100-task VisualWebArena subset and on MiniWoB. Its released code supports only those two benchmarks.
W4. Differences from prior work
The table compares the four systems on what they track and what they accept as evidence.
| What is tracked | Who writes it | Bound to application objects? | What counts as evidence | State-changing tasks? | Evaluated on | |
|---|---|---|---|---|---|---|
| WebRider | Intent contract and evidence obligations | Written in advance, fixed during the run | No | Page facts that must be seen before answering | No (explicitly excluded) | RiderBench |
| ContractSkill | Pre- and post-conditions of each skill | Generated from the task | Belongs to the skill, not the task | Skill-level checks | — | VisualWebArena subset, MiniWoB |
| Agent Workflow Memory | Reusable workflows | Induced from past trajectories | Not applicable | Not applicable | Yes | WebArena, Mind2Web |
| HORIZON | Task-level requirements and their status | Compiled at run time from the user request | Yes, to specific objects or collections | Read-backs of saved state in the current task, with validity; stale evidence reopens the requirement | Yes | WebArena, Online-Mind2Web |
Task Completion Memory (TCM) represents the task state as : the task specification, the object bindings, the requirement states, the evidence with its validity, and the open obligations. The key distinction from the three other systems is that TCM tracks what must hold, independently of the sequence of operations used to achieve it. A requirement can stay open after a locally successful action, and it reopens when its evidence becomes stale. Past experience is guidance and is never counted as evidence for the current task.
W5. Same-budget comparison and cost statistics
HORIZON uses more model calls and tokens per task than the baseline agent, because it plans and checks evidence. Four measurements on the 51-task WebArena subset put this overhead in context.
- Same budget. Both systems have the same limit of 30 browser steps. With gpt-6-luna, the baseline agent used it up in only 24 of its 102 runs; the other 78 runs ended on their own, by answering or by stopping early. A larger budget would therefore not close the gap.
- Browser steps and time. HORIZON takes fewer browser steps per task (2.5 against 14.2), because it reads pages through the DOM instead of scrolling through them. Its median time per task is about the same (172 s against 188 s). Most of its input is a repeated prompt prefix: 86% of its input tokens are cache reads.
- A small model with the harness against a frontier model without it. gpt-6-luna with HORIZON solves 49.7% of the tasks at about $0.05 per task. Claude Opus 5.5 without a harness solves 33.3% at more than four times that cost.
- The overhead shrinks with a stronger model. With Claude Opus 5.5, HORIZON needs 13.9 model calls per task against 11.7 for the baseline agent, and it solves 66.7% of the tasks against 33.3%.
We thank the reviewer for recognizing the requirement-level task modeling, the action–verification loop, and the experience design. We answer each weakness below, with new experiments for W1, W2, W3 and W5.
W1. Limited coverage: other benchmarks, unseen websites, interface changes
Other benchmarks and unseen websites. We added Online-Mind2Web: 300 tasks on live public websites, none of which is a WebArena site. "Without HORIZON" is the official baseline agent from the WebArena repository (a chain-of-thought prompt over the accessibility tree), run with the same model (gpt-6-luna) and one run per task. The harness ran on these sites with no site-specific code or prompts.
Interface changes. Three properties of the design address this.