acceptodds
Under review as a conference paper at ICLR 2027

HORIZON: Evidence-Grounded Task Memory, Interface Translation, and Experience for Long-Horizon Web Agents

Abstract

Long-horizon web tasks require agents to connect user requirements to the effects of actions across changing interfaces. An executed action can leave the requested outcome unresolved: a click may activate the wrong control, a value may remain unsaved, or a returned answer may lack supporting observations. We introduce HORIZON, a framework that makes these connections explicit through evidence-grounded task state. Task Completion Memory tracks requirements, object bindings, progress, and supporting observations. A Document Object Model (DOM) Translation Layer links structured interface views and semantic operations to their source elements. State-conditioned experience retrieves prior task records as advice and provides a formulation for contrasting successful and unsuccessful interactions while keeping historical advice separate from current-task evidence. On WebArena, HORIZON raises gpt-5.6-luna's overall success rate from 15.4% to 40.3%, a gain of 24.9 percentage points. With Qwen3.8-27B, it achieves 60.1% across all 812 tasks, including 66.7% on cross-site tasks, under a protocol with up to two attempts on four single-site domains. These results compare complete agent configurations. Selected trajectories illustrate source-linked targeting, requirement-directed readback, and archived-route adaptation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Web Agent, Long-horizon, Memory, DOM Manipulation, Long-horizon planning, Evidence-grounded reasoning, Semantic interface translation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_KsasOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the gap between performing actions and achieving goals in long web tasks. It proposes HORIZON, which connects user requirements, interface actions, and observed evidence through task completion memory, a DOM translation layer, and experience retrieval based on the current state. It also separates past suggestions from evidence for the current task. Comparisons of full agent configurations on WebArena show that HORIZON improves task success rates and supports tasks across websites. Example execution traces further show its role in locating targets, checking results based on requirements, and adapting action paths.

Critical strengths
  1. Task progress modeling is flexible and easy to trace. The framework organizes action records and evidence around requirements, supports multiple valid paths, and clearly tracks unfinished items.
  2. Actions and verification form a complete loop. The two way DOM interface keeps actions and result checks focused on the same target. It also links interaction results to updates in requirement status.
  3. The experience reuse mechanism is carefully designed. It uses interaction states and comparison results to produce guidance, keeps evidence and verification status, and separates experience based suggestions from verification of the current task.
Critical weaknesses
  1. The evaluation has limited coverage. The paper evaluates the framework only on a limited set of benchmarks, such as WebArena. It lacks experiments on other independent benchmarks, unseen websites, and conditions involving interface changes. The results therefore do not provide enough evidence that the framework generalizes to different website structures and task distributions.
  2. The comparison with external methods in Table 2 lacks fairness. Different methods or frameworks do not use the same models and test sets. For example, Agent-as-Annotators with A3-Qwen3.5-9B uses a subset, while HORIZON with A3-Qwen3.5-9B uses the full test set.
  3. The paper lacks comparisons with relevant baselines. It cites related work such as WebRider, ContractSkill, and Agent Workflow Memory, which share similar core ideas, but does not compare HORIZON with these methods in the experiments.
  4. The comparison with existing work needs more detail. The core ideas are similar to those of WebRider, ContractSkill, and Agent Workflow Memory, but the paper does not clearly explain the differences. A comparison table can help make these differences clearer.
  5. The paper lacks strict comparisons under the same budget. It does not provide complete statistics on token usage, model calls, actual execution time, and cost per task. This makes it difficult to assess the extra cost of verification and memory maintenance.
Questions for authors

See above.

3 comments

We thank the reviewer for recognizing the requirement-level task modeling, the action–verification loop, and the experience design. We answer each weakness below, with new experiments for W1, W2, W3 and W5.

W1. Limited coverage: other benchmarks, unseen websites, interface changes

Other benchmarks and unseen websites. We added Online-Mind2Web: 300 tasks on live public websites, none of which is a WebArena site. "Without HORIZON" is the official baseline agent from the WebArena repository (a chain-of-thought prompt over the accessibility tree), run with the same model (gpt-6-luna) and one run per task. The harness ran on these sites with no site-specific code or prompts.

Success rate on the 300 tasks, as scored byWithout HORIZONWith HORIZONExact McNemar test
WebJudge, the benchmark's official automatic judge15.7% (47/300)28.3% (85/300)p = 1.5e-5
Our review of each run's full record (answer, page text read, screenshots)13.3% (40/300)36.3% (109/300)p < 1e-13
  • Denominator. No task is excluded: the success rate is over all 300 tasks. On 41 of them the website refuses automated browsers, so both systems fail there (each solved 1 of the 41). Another 8 need login or purchases; we did not run them for safety reasons, and they count as failures for both systems.
  • WebJudge is the benchmark's official automatic judge, run with the backbone its maintainers currently recommend.
  • Why we add a second score. WebJudge is an automatic judge and it makes errors: its maintainers report 87.4% agreement with human labels. It sees only the screenshots and the action list. In our runs it failed correct runs whose answer was read from page content that is not on a screenshot, and it passed runs whose screenshots are all "access denied" pages.
  • Our review of the full record. We therefore also checked every run against its full record (the answer, the page text the agent read, and the screenshots), using an LLM reviewer (Claude Opus 5.5). Runs where this check and WebJudge gave different verdicts were judged again by two independent LLM reviewers that could not see which system had produced the run. Whenever those two disagreed, we counted the run in the way that is worse for HORIZON. Even under this conservative counting, HORIZON solves 36.3% of the 300 tasks, against 13.3% without it (second row of the table).
  • How this compares with published results. The top of the Online-Mind2Web leaderboard (above 90%) is held by dedicated computer-use products and by models trained for web tasks. Our claim is a different one: the gain from the harness when the model is held fixed. The closest published reference is therefore a general-purpose LLM in an open harness. Such agents score 18–42% across 12 models in the independent HAL runs, and text-only accessibility-tree agents with a 30-step limit score 34.8% (Gemini 3 Flash) and 41.9% (GPT-5). With gpt-6-luna, a low-cost model that receives no screenshots, HORIZON reaches 28.3% under WebJudge (36.3% under our review) at about $0.06 per task, up from 15.7% without the harness. These numbers come from different judges, task versions and browser setups, and 41 of our 300 tasks are on sites we could not reach, so they are context and not a controlled comparison.

Interface changes. Three properties of the design address this.

  • The agent re-reads the current page at every step. After a page update, old element references are invalid and must be reacquired ("stale targets require reacquisition").
  • Past experience is used as guidance only, never as evidence. If the interface has changed, an old experience cannot count toward completing the current task.
  • Task 802 is an example. The retrieved path belonged to a different object; the agent adapted it to the new object and confirmed the result by reading the current page back.

W2. Table 2 comparisons are not on the same models and test sets

We agree. Table 2 will be split into a same-model controlled block and a block of published results for reference, and each row will state its task set and number of attempts. The controlled block is the following. Each row is a separate controlled comparison: within a row, both systems use the same model, the same tasks and the same browser-step limit, and only the harness differs. Rows use different models and task sets, so numbers should be compared within a row, not across rows. "Without HORIZON" is the official baseline agent from the WebArena repository (a chain-of-thought prompt over the accessibility tree).

BenchmarkModelTasksWithout HORIZONWith HORIZONGain (points)Paired test
WebArena (paper setting)Qwen3.8-27B47535.6%45.1%+9.5p = 0.0003
WebArena subset (new)gpt-6-luna5111.1%49.7%+38.6p = 1e-5
WebArena subset (new)Claude Opus 5.55133.3%66.7%+33.3p = 1.5e-5
Online-Mind2Web (new)gpt-6-luna30013.3%36.3%+23.0p < 1e-13

Each task is run once, except in the gpt-6-luna WebArena row, which is the mean of 3 runs per task. Run-to-run variation is small relative to the gain: over those 3 runs, HORIZON solves 25, 25 and 26 of the 51 tasks, and the baseline agent 6, 4 and 7.

  • The WebArena subset has 51 tasks (Shopping 20, Shopping Admin 20, Reddit 11), drawn with a fixed seed from the single-site tasks that are scored by string or URL match.
  • The row for Agent-as-Annotators (A3) will be reported on the same 381-task split that A3 uses. A3's model is fine-tuned for its own agent's prompt and action format. Inside HORIZON both change, so that row measures the model outside its training format and not the effect of the harness.

W3. No experimental comparison with WebRider, ContractSkill, Agent Workflow Memory

Agent Workflow Memory can be compared directly on WebArena, and we ran it. All rows use gpt-6-luna on the same 51 WebArena tasks.

SystemTasks solved (of 51)Success rate
WebArena baseline agent (without HORIZON)5.711.1%
Agent Workflow Memory, its default 10-step limit1427.5%
Agent Workflow Memory, 30-step limit1529.4%
HORIZON25.349.7%

HORIZON and the baseline agent are means of 3 runs per task; Agent Workflow Memory was run once per task.

  • Agent Workflow Memory solves 14 of 51 tasks with its default step limit and 15 with a 30-step limit; HORIZON solves 25 to 26 per run. In both settings the difference is significant against each of the three HORIZON runs (sign test, p ≤ 0.041).
  • We used its official code and the workflow files released in its repository, without online updates. Its default step limit is 10, while HORIZON and the baseline agent use 30. For the 30-step row, we reran with a 30-step limit the 16 tasks on which it had run out of steps; one more task was solved. Its released workflows were induced on WebArena test tasks and include some of our 51 tasks.
  • WebRider is evaluated only on its own RiderBench (information seeking and recommendation; purchases and irreversible actions are excluded). It releases a dataset but no runnable code.
  • ContractSkill is evaluated on a 100-task VisualWebArena subset and on MiniWoB. Its released code supports only those two benchmarks.

W4. Differences from prior work

The table compares the four systems on what they track and what they accept as evidence.

What is trackedWho writes itBound to application objects?What counts as evidenceState-changing tasks?Evaluated on
WebRiderIntent contract and evidence obligationsWritten in advance, fixed during the runNoPage facts that must be seen before answeringNo (explicitly excluded)RiderBench
ContractSkillPre- and post-conditions of each skillGenerated from the taskBelongs to the skill, not the taskSkill-level checks—VisualWebArena subset, MiniWoB
Agent Workflow MemoryReusable workflowsInduced from past trajectoriesNot applicableNot applicableYesWebArena, Mind2Web
HORIZONTask-level requirements and their statusCompiled at run time from the user requestYes, to specific objects or collectionsRead-backs of saved state in the current task, with validity; stale evidence reopens the requirementYesWebArena, Online-Mind2Web

Task Completion Memory (TCM) represents the task state as Mt=(St,Bt,Zt,Et,Ot)M_t = (S_t, B_t, Z_t, E_t, O_t): the task specification, the object bindings, the requirement states, the evidence with its validity, and the open obligations. The key distinction from the three other systems is that TCM tracks what must hold, independently of the sequence of operations used to achieve it. A requirement can stay open after a locally successful action, and it reopens when its evidence becomes stale. Past experience is guidance and is never counted as evidence for the current task.

W5. Same-budget comparison and cost statistics

HORIZON uses more model calls and tokens per task than the baseline agent, because it plans and checks evidence. Four measurements on the 51-task WebArena subset put this overhead in context.

  • Same budget. Both systems have the same limit of 30 browser steps. With gpt-6-luna, the baseline agent used it up in only 24 of its 102 runs; the other 78 runs ended on their own, by answering or by stopping early. A larger budget would therefore not close the gap.
  • Browser steps and time. HORIZON takes fewer browser steps per task (2.5 against 14.2), because it reads pages through the DOM instead of scrolling through them. Its median time per task is about the same (172 s against 188 s). Most of its input is a repeated prompt prefix: 86% of its input tokens are cache reads.
  • A small model with the harness against a frontier model without it. gpt-6-luna with HORIZON solves 49.7% of the tasks at about $0.05 per task. Claude Opus 5.5 without a harness solves 33.3% at more than four times that cost.
  • The overhead shrinks with a stronger model. With Claude Opus 5.5, HORIZON needs 13.9 model calls per task against 11.7 for the baseline agent, and it solves 66.7% of the tasks against 33.3%.
Reviewer_bAG8OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces HORIZON, a harness that connects action outcomes with user requirements. The proposed harness has several modules: (1) a memory module for tracking requirements, progress, and supporting evidence, (2) a DOM translation layer that converts an interface into structured data, (3) a module that retrieves prior trajectories to act as advice. The paper demonstrates that HORIZON improves upon the gpt-5.6-luna and Qwen3.8-27B baseline models.

Critical strengths
  • Structured harnesses have shown to be beneficial for CUAs and web navigation models. This paper explores one version of such a harness which makes the outcomes of actions explicit through several modules (memory, DOM translation, experience).
Critical weaknesses
  • Weak baselines: gpt-5.6-luna and Qwen3.8-27B are not frontier models. Show the benefit of HORIZON on an actual SOTA model (e.g., GPT-6 Astra) is necessary to justify the additional engineering and infra cost of the harness. If cost is in issue, showing it even on a subset of tasks (ideally past statistical significance) is still necessary.
  • The proposed method involves many handcrafted stages, and it is not convincing that these are necessary, scalable, and general modules. For example, DOM does not generalize to general computer use, and most frontier CUAs (which also achieve SOTA on web navigation) do not generally use DOM structures. They are primarily pixel based approaches that take in screenshots.
  • The baselines do not appear very fair. gpt-5.6-luna and Qwen3.8-27B are trained as models that take in screenshots as inputs, but it appears that the baselines reported here use DOM? It would be more fair to run and compare the models exactly as recommended according to the official documents (which is screenshot as input, and actions as output). The fair comparison here would also be to include other harness-based methods, such as running ColorBrowserAgent, OpAgent, Agent-as-Annotators, Agent Workflow Memory with the same gpt-5.6-luna and Qwen3.8-27B models. At the moment, these are omitted from the second part of Table 2.
  • Limited evaluation: the paper benchmarks only on WebArena, which is a fairly constrained synthetic environment (and somewhat outdated today). It would be ideal to show results on other web navigation benchmarks, such as Online-M2W or Odysseys.
Questions for authors
  • How does your method performance against the other harness-based methods, such as ColorBrowserAgent, OpAgent, Agent-as-Annotators, Agent Workflow Memory?
  • How does HORIZON perform on other web navigation benchmarks?
3 comments

We thank the reviewer for the detailed assessment. We ran new experiments for each weakness; the results are below.

W1. Backbones are not frontier models

We now have same-model comparisons on three models, including a frontier model. Each row of the table is a separate controlled comparison: within a row, both systems use the same model, the same tasks and the same browser-step limit, and only the harness differs. Rows use different models and task sets, so numbers should be compared within a row, not across rows. "Without HORIZON" is the official baseline agent from the WebArena repository (a chain-of-thought prompt over the accessibility tree).

BenchmarkModelTasksWithout HORIZONWith HORIZONGain (points)Paired test
WebArena (paper setting)Qwen3.8-27B47535.6%45.1%+9.5p = 0.0003
WebArena subset (new)gpt-6-luna5111.1%49.7%+38.6p = 1e-5
WebArena subset (new)Claude Opus 5.55133.3%66.7%+33.3p = 1.5e-5
Online-Mind2Web (new)gpt-6-luna30013.3%36.3%+23.0p < 1e-13

Each task is run once, except in the gpt-6-luna WebArena row, which is the mean of 3 runs per task. Run-to-run variation is small relative to the gain: over those 3 runs, HORIZON solves 25, 25 and 26 of the 51 tasks, and the baseline agent 6, 4 and 7.

  • The WebArena subset has 51 tasks (Shopping 20, Shopping Admin 20, Reddit 11), drawn with a fixed seed from the single-site tasks that are scored by string or URL match. Every run starts from a site restored to its initial state.
  • We chose Qwen3.8-27B for the paper because it is open-weight, runs locally, and can be reproduced by others. Prior harness papers also did not use the newest model of their time: Agent Workflow Memory (ICML 2025) uses GPT-4 "following baseline approaches", and AgentOccam (ICLR 2025) uses GPT-4-turbo.
  • On whether the harness justifies its cost: on the 51-task subset, gpt-6-luna with HORIZON solves 49.7% at about $0.05 per task, while Claude Opus 5.5 without a harness solves 33.3% at about $0.22 per task. With Claude Opus 5.5 itself, HORIZON solves 34 of the 51 tasks and the baseline agent 17, and every task the baseline agent solves is also solved with HORIZON.

W2. Many handcrafted stages; DOM does not generalize

  • Scope. HORIZON targets web tasks, as the title and abstract state. We do not claim applicability to desktop applications and will say so in the limitations.
  • What is web-specific. Only the DOM layer is. Task Completion Memory (TCM) and experience retrieval operate on requirements and evidence and do not depend on the DOM. In principle they can be paired with a screenshot-based executor.
  • Handcrafted rules. TCM's rules are general rules about what counts as evidence. They are not written per website.
  • Why a DOM layer. It addresses a problem that pixel-based agents share: the target is known but cannot be hit by coordinates. In Figure 3, five rating buttons share one 1×1-pixel box. In the paper's targeted study, element-level execution recovered 23 of 25 such failed tasks.
  • Use of page structure is standard for web agents. AgentOccam is text-only, the official WebChoreArena agent uses the accessibility tree, and ColorBrowserAgent uses screenshots plus the accessibility tree.
  • Evidence on necessity. With the same model (gpt-6-luna) on the 51-task subset, the harness raises the success rate from 11.1% to 49.7%. Removing only DOM-based page reading, the read direction of the DOM Translation Layer, lowers it to 37.3% (sign test, p = 0.007). Element-level execution stays on in that ablation.
  • Evidence on generality and scalability. The same harness, with no site-specific code or prompts, raises the success rate on the 300 live-website tasks of Online-Mind2Web from 13.3% to 36.3% (see W4 in Part 3). Running on these new websites required no additional engineering.

W3. Fairness of baselines

(a) Observation modality.

  • In the paper's comparison, the baseline is the official agent from the WebArena repository with its official prompts. Both systems receive the text accessibility tree and the URL, and neither receives screenshots. Only the harness differs.
  • Text observation is the standard WebArena setting. The official agent, AgentOccam, Agent Workflow Memory and the official WebChoreArena agent all use it.
  • Under this matched setting, the result is 169 vs. 214 of 475 tasks (p = 0.0003). Without the 62 tasks that are scored by an LLM judge, it is 149 vs. 182 (p = 0.0046).

(b) Other harness methods with the same model. Agent Workflow Memory is the one method whose backbone can be swapped directly, and we ran it. All rows use gpt-6-luna on the same 51 WebArena tasks.

SystemTasks solved (of 51)Success rate
WebArena baseline agent (without HORIZON)5.711.1%
Agent Workflow Memory, its default 10-step limit14.027.5%
Agent Workflow Memory, 30-step limit15.029.4%
HORIZON25.349.7%

HORIZON and the baseline agent are means of 3 runs per task; Agent Workflow Memory was run once per task.

  • Agent Workflow Memory solves 14 of 51 tasks with its default step limit and 15 with a 30-step limit; HORIZON solves 25 to 26 per run. In both settings the difference is significant against each of the three HORIZON runs (sign test, p ≤ 0.041).
  • We used its official code and the workflow files released in its repository, without online updates. Its default step limit is 10, while HORIZON and the baseline agent use 30. For the 30-step row, we reran with a 30-step limit the 16 tasks on which it had run out of steps; one more task was solved. Its released workflows were induced on WebArena test tasks and include some of our 51 tasks.
  • Agent-as-Annotators is a training method, not a harness. A same-model comparison would require fine-tuning again.
  • OpAgent depends on its own fine-tuned grounding model and on per-site guidance.
  • ColorBrowserAgent's reported result uses 52 human-written hints derived from its failures on WebArena tasks.
  • We will split Table 2 of the paper into "same-model controlled comparison" and "published results for reference".

W4. Evaluation only on WebArena

  • WebArena's sites are local deployments of real open-source applications, not synthetic pages. The best published results are 71–74%, so the benchmark is not saturated.
  • We added Online-Mind2Web, one of the two benchmarks the reviewer named: 300 tasks on live websites, the same model (gpt-6-luna) for both systems, one run per task.
Success rate on the 300 tasks, as scored byWithout HORIZONWith HORIZONExact McNemar test
WebJudge, the benchmark's official automatic judge15.7% (47/300)28.3% (85/300)p = 1.5e-5
Our review of each run's full record (answer, page text read, screenshots)13.3% (40/300)36.3% (109/300)p < 1e-13
  • Denominator. No task is excluded: the success rate is over all 300 tasks. On 41 of them the website refuses automated browsers, so both systems fail there (each solved 1 of the 41). Another 8 need login or purchases; we did not run them for safety reasons, and they count as failures for both systems.
  • WebJudge is the benchmark's official automatic judge, run with the backbone its maintainers currently recommend.
  • Why we add a second score. WebJudge is an automatic judge and it makes errors: its maintainers report 87.4% agreement with human labels. It sees only the screenshots and the action list. In our runs it failed correct runs whose answer was read from page content that is not on a screenshot, and it passed runs whose screenshots are all "access denied" pages.
  • Our review of the full record. We therefore also checked every run against its full record (the answer, the page text the agent read, and the screenshots), using an LLM reviewer (Claude Opus 5.5). Runs where this check and WebJudge gave different verdicts were judged again by two independent LLM reviewers that could not see which system had produced the run. Whenever those two disagreed, we counted the run in the way that is worse for HORIZON. Even under this conservative counting, HORIZON solves 36.3% of the 300 tasks, against 13.3% without it (second row of the table).
  • How this compares with published results. The top of the Online-Mind2Web leaderboard (above 90%) is held by dedicated computer-use products and by models trained for web tasks. Our claim is a different one: the gain from the harness when the model is held fixed. The closest published reference is therefore a general-purpose LLM in an open harness. Such agents score 18–42% across 12 models in the independent HAL runs, and text-only accessibility-tree agents with a 30-step limit score 34.8% (Gemini 3 Flash) and 41.9% (GPT-5). With gpt-6-luna, a low-cost model that receives no screenshots, HORIZON reaches 28.3% under WebJudge (36.3% under our review) at about $0.06 per task, up from 15.7% without the harness. These numbers come from different judges, task versions and browser setups, and 41 of our 300 tasks are on sites we could not reach, so they are context and not a controlled comparison.
  • We do not report Odysseys. Its official setup operates a desktop virtual machine, which is outside the scope of a web harness.

Reviewer_Yx1JOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces HORIZON, a framework designed to improve the performance of web agents on complex, long-horizon tasks. The three primary contributions are (A) Task Completion Memory that maintains a requirement-level progress tracker, (B) DOM Translation Layer to preserve identity of DOM target elements and (C) State-Conditioned Experience to use prior episodes for guiding future actions. Overall these contributions lead to big gains on WebArena for a variety of backbone LLMs.

Critical strengths

The task completion memory setup makes sense, as binding requirements to specific object state prevents agents from prematurely concluding that a task is complete. The performance gains from this approach are impressive, with a boost of +24.9% (from 15.4% to 40.3% using gpt-5.6-luna).

Critical weaknesses

The primary problem with this work is the limited evaluation setup which involves only WebArena. The authors should include other long horizon web tasks in the evaluation.

The methodology section is too long and hard to follow. The authors should instead compress the method section to expand the experiments section to include more ablations / analysis.

Questions for authors

How does the system handle cases where TCM’s internal verification disagrees with the benchmark evaluator?

2 comments

We thank the reviewer for the constructive feedback. Since submission we ran new controlled experiments; the results are under W1 and W2.

W1. Evaluation limited to WebArena

We added a second benchmark, Online-Mind2Web (300 tasks on live public websites). "Without HORIZON" is the official baseline agent from the WebArena repository (a chain-of-thought prompt over the accessibility tree), run with the same model. Both systems use gpt-6-luna and the same browser setting (a headed browser where a site refuses headless ones), with one run per task.

Success rate on the 300 tasks, as scored byWithout HORIZONWith HORIZONExact McNemar test
WebJudge, the benchmark's official automatic judge15.7% (47/300)28.3% (85/300)p = 1.5e-5
Our review of each run's full record (answer, page text read, screenshots)13.3% (40/300)36.3% (109/300)p < 1e-13
  • Denominator. No task is excluded: the success rate is over all 300 tasks. On 41 of them the website refuses automated browsers, so both systems fail there (each solved 1 of the 41). Another 8 need login or purchases; we did not run them for safety reasons, and they count as failures for both systems.
  • WebJudge is the benchmark's official automatic judge, run with the backbone its maintainers currently recommend.
  • Why we add a second score. WebJudge is an automatic judge and it makes errors: its maintainers report 87.4% agreement with human labels. It sees only the screenshots and the action list. In our runs it failed correct runs whose answer was read from page content that is not on a screenshot, and it passed runs whose screenshots are all "access denied" pages.
  • Our review of the full record. We therefore also checked every run against its full record (the answer, the page text the agent read, and the screenshots), using an LLM reviewer (Claude Opus 5.5). Runs where this check and WebJudge gave different verdicts were judged again by two independent LLM reviewers that could not see which system had produced the run. Whenever those two disagreed, we counted the run in the way that is worse for HORIZON. Even under this conservative counting, HORIZON solves 36.3% of the 300 tasks, against 13.3% without it (second row of the table).
  • How this compares with published results. The top of the Online-Mind2Web leaderboard (above 90%) is held by dedicated computer-use products and by models trained for web tasks. Our claim is a different one: the gain from the harness when the model is held fixed. The closest published reference is therefore a general-purpose LLM in an open harness. Such agents score 18–42% across 12 models in the independent HAL runs, and text-only accessibility-tree agents with a 30-step limit score 34.8% (Gemini 3 Flash) and 41.9% (GPT-5). With gpt-6-luna, a low-cost model that receives no screenshots, HORIZON reaches 28.3% under WebJudge (36.3% under our review) at about $0.06 per task, up from 15.7% without the harness. These numbers come from different judges, task versions and browser setups, and 41 of our 300 tasks are on sites we could not reach, so they are context and not a controlled comparison.

We chose WebArena for the paper because it scores the final application state, its sites can be reset, and it contains many state-changing tasks. Verifying such tasks is what HORIZON is built for.

W2. Method section too long; more ablations and analysis

We will rewrite Section 3 around the requirement–action–evidence loop. The contrastive-induction formulas for experience and the field-level details of the DOM interface will move to the appendix. The space goes to the two analyses below.

Same-model comparison in the paper's setting. With Qwen3.8-27B and a single attempt on 475 WebArena tasks, the baseline agent solves 169 tasks and HORIZON solves 214 (McNemar p = 0.0003). The two systems use the same model and the same browser-step limit.

New ablation on a WebArena subset, all with gpt-6-luna. The subset has 51 tasks (Shopping 20, Shopping Admin 20, Reddit 11), drawn with a fixed seed from the single-site tasks that are scored by string or URL match. Before every run the site is restored to its initial state and checked against a checksum of its database.

SystemTasks solved (of 51)Success rate
WebArena baseline agent (without HORIZON)5.711.1%
HORIZON without DOM-based page reading19.037.3%
HORIZON25.349.7%

All numbers are means of 3 runs per task. Run-to-run variation is small: HORIZON solves 25, 25 and 26 tasks in the 3 runs, and the baseline agent 6, 4 and 7. The harness raises the success rate more than fourfold. DOM-based page reading contributes about six tasks per run (sign test, p = 0.007). This is the read direction of the DOM Translation Layer (Section 3.3): the planner's read-only tools return the content of a page, of a whole paged table, or of a list from the DOM, without a browser action. Element-level execution, the other direction of the layer, stays on in this ablation.

Q1. What happens when TCM's verification disagrees with the benchmark evaluator?

  • Reported success is decided only by the benchmark evaluator.
  • During an attempt, the agent never sees any evaluator output.
  • The readiness assessment of Task Completion Memory (TCM) is internal to the agent. A disagreement with the evaluator therefore never changes a reported number.
  • TCM acts on the planning side: requirements that are not yet supported by evidence return to the planner as open obligations. In task 755, the "private" requirement was still open after the project was created, so the agent went back and read the saved visibility setting.
  • We do not claim that TCM is a calibrated completion classifier. Section 4.4 says so, and the revision will make it more explicit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.