acceptodds
Under review as a conference paper at ICLR 2027

The Agent Is in the Details: What Matters in Web Agent Harness Design?

Abstract

A web agent harness determines what the model sees, what it retains across steps, and how model outputs become browser actions. Harness designs are often introduced inside complete systems with their own models and setups, so reported gains do not show which design to adopt. We build a configurable framework on GenericAgent from AgentLab, implement 30 configurations in six functional categories, and compare each with its setting-specific baseline on WebArena-Lite with Qwen3.5-9B and OpenApps with Qwen3.5-27B. The two highest-scoring configurations in the reported catalog results are multi-turn histories in both settings: full observation history keeps earlier page bodies, whereas observation masking replaces them with placeholders; both retain past model responses. Full history and masking reach success rates of 26.06% and 23.64% against a 15.76% baseline on WebArena-Lite, and 57% and 31% against 19% on OpenApps. On WebArena-Lite, masking trails full history by 2.42 percentage points and uses about a third fewer action-model tokens than the baseline; on OpenApps, it trails by 26 points. Only one tested combination outscores the constituent configurations alone: full history with execution-error feedback on OpenApps. We also evaluate one fixed workflow library, distilled from successful trajectories on the evaluated task set, across five WebArena-Lite model settings. Ordered by baseline success, the settings show declining workflow increments, from +7.88 points with Qwen3.5-4B to −1.82 with GPT-5.6 Sol. These results favor evaluating multi-turn history first, comparing combinations with the constituent configurations alone and the unchanged baseline when selecting by success rate, and evaluating this library on the target model. We will release the framework code and configurations as an extensible platform for web agent harness research, supporting new designs and reuse of existing comparisons.

Primary area:
applications to robotics, autonomy, planning
Keywords:
web agents, agent harnesses

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AhbcOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper aims to comprehensively compare 30 agent harness designs, inspired from various existing works like AgentOccam, WebDreamer, AgentLab/BrowserGym, AWM, etc for web navigation tasks. The evaluate these 30 agent harness designs on 2 benchmarks: WebArena-Lite w/ Qwen3.5-9B and OpenApps (100 tasks) w/ Qwen3.5-27B. These 30 designs are mapped to 6 functional categories that common web agents design for according to the survey conducted by (Guo et. al, 2026). The paper mainly finds: 1. most configurations score below baseline 2. Two history related configurations (include full trajectory history, and include masked trajectory history) and “Parse-error early stop” improve performance over baseline for both OpenApps and WebArena 3. Based on experiments with configurations that supply in context examples to the agent (called Offline workflow memory, they find that the returns from supplying examples diminishes as model intelligence improves.

All agent configurations are built upon the GenericAgent created by ServiceNow (https://github.com/ServiceNow/AgentLab). The original GenericAgent already sets up the prompting pipeline, keeps observation preprocessor, observation history, prior actions, internal state containing plan, memories, thoughts, retry loops etc - thus containing some of the agent configurations already. However, to use this framework as a baseline, the authors create a stripped down version of GenericAgent, keeping only the goal, the current AXTree, and past actions. Meanwhile, screenshots, HTML, past error logs, past reasoning, planning, memory, and feedback are all turned off in the baseline setting.

Critical strengths
  1. The paper does a controlled comparison of multiple harness designs, on a shared baseline agent framework, backbone model and evaluation tasks. This provides a useful measure for readers to choose which agent design categories to incorporate when building their web navigation agents.
  2. Such a study can help show what agent harness settings might not bring useful gains to end-to-end performance. Table 4 also conducts a useful study for a specific harness configuration, to show that gains from these designs can vary significantly across different model capability tiers.
Critical weaknesses
  1. Lack of insight into negative results: The paper does not provide an insight into why configurations like planning, observation reductions etc. underperform. Thus while we can see relative performances on WebArena and OpenApps on these 30 settings, there is little guidance that the community can get from why certain configurations can score lower. It is unclear why Subgoal planning or Parser error retries cause a significant hit to the performance, and can indicate potential implementation bugs in the absence of accompanying error analysis.
  2. Potential lack of generalizability: The study is conducted using one model, one environment combinations. All harness categories are tested against a single baseline agentic framework. Thus the trends seen across the harness configurations might be confounded. The study is most useful only if a reader can walk away with generalizable takeaways and guidances on good agentic designs.
  3. Incremental findings: The main finding from the paper is that adding history of the trajectory helps, but cited works by Enomoto et al[1] already show that result. Another finding is that masking history reduces token usage but at the expense of performance. Both these findings are incremental over existing works on web agents.

[1] Masafumi Enomoto, Ryoma Obara, Haochen Zhang, and Masafumi Oyamada. Read More, Think More: Revisiting Observation Reduction for Web Agents. arXiv preprint arXiv:2604.01535, 2026. URL https://arxiv.org/abs/2604.01535v1

Questions for authors

Questions:

  1. L319 say: "Relative to baseline, both history configurations also change message organization and parsing, as § 2.2 specifies, so their differences from baseline cannot be attributed to retained history alone." This refutes the main finding of the paper. Can the authors clarify?

Suggestions:

  1. L112: The concept of workflow library can be introduced more clearly.
  2. The shorthand FMT for the baseline agent is never explained. The baseline agent setting being a stripped down version of GenericAgent, is also only clarified in Appendix A.2.
  3. Statistical significance concerns: The agent runs are conducted with temp=1.0, and with single runs per setting. Without proper confidence intervals, it is hard to rank these configurations and figure out what results are within noise limits from each other.
Reviewer_qYJuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper builds a configurable framework on AgentLab’s GenericAgent and implements 30 web-agent harness configurations in six categories: observation representation, context management, state and memory, planning and control, action interface, and verification and recovery. Three of them are content controls that inject irrelevant notes, plans or feedback. Each configuration is compared with a baseline in two settings: WebArena-Lite (165 tasks, Qwen3.5-9B) and OpenApps (100 composite tasks, Qwen3.5-27B).

The main findings:

  1. Full observation history and observation masking, both multi-turn variants that retain past model responses, score highest in both settings. On WebArena-Lite they reach 26.06% and 23.64% against a 15.76% baseline, and on OpenApps 57% and 31% against 19%. Masking uses about a third fewer action-model tokens than the baseline on WebArena-Lite.
  2. 21 of 30 configurations fall below baseline in each setting.
  3. Only one tested combination (full history plus execution-error feedback on OpenApps, 63% vs. 57%) beats its constituents alone. A fixed workflow library distilled from successful trajectories gives a declining increment across five model settings, from +7.88 pp with Qwen3.5-4B to −1.82 pp with GPT-5.6 Sol.

The authors derive adoption guidance from these results and promise to release the framework.

Critical strengths
  1. The comparison design is useful. All 30 configurations are compared against one baseline per environment, with the same tasks and action model. This is more informative than the usual practice of reporting a design only inside a complete system. The catalog (Table A.1, Figure A.1) states each configuration’s operation and design source, which makes the comparison auditable and reusable. The negative result is also valuable: most designs, including several popular ones, do not beat a simple baseline.

  2. The three content controls (irrelevant notes, irrelevant plan, fixed irrelevant feedback) are a good methodological idea. On OpenApps they match or beat their real counterparts (irrelevant plan 24% vs. 17–18% for planning configurations, irrelevant notes 17% vs. in-task notes 14%). This is a useful warning for the field that gains from “memory” or “planning” modules may not come from the claimed mechanism.

  3. The analysis goes beyond headline success rates. It compares combinations with the constituent configurations alone and not only with the start (Table 3), and it reports tokens and steps per task (Table 2). The step counts show that masking cuts tokens on WebArena-Lite but increases steps (10.04 vs. 8.83). The paper also reports per-task overlap of solved tasks and a post-hoc audit of the WebArena success labels.

Critical weaknesses
  1. Single runs and no uncertainty estimates, while the conclusions rest on small differences. Each configuration appears to be one run at temperature 1.0 on 165 tasks (one task = 0.61 pp) or 100 tasks (one task = 1 pp). No seeds, confidence intervals or paired tests are reported. The paper itself notes that 16 WebArena configurations are within four tasks of baseline. Several of its own results show how large the noise floor is. “Default AXTree text,” which should be near-equivalent to the baseline, differs by one task on WebArena-Lite. Control configurations beat real ones on OpenApps by a few tasks. The key combination claim (63% vs. 57%, six tasks of 100) and the reading that self-feedback “beats memory alone but not feedback alone” are treated as findings, and the guidance sections (“evaluate X first,” “choose the feedback type on the target start”) are built on them. The very large effects (history vs. baseline, +10 pp on WebArena-Lite and +38 pp on OpenApps) are probably real, but most of the rest of the catalog ranking, the combination analysis and the Table 4 trend are not statistically supported. Repeated runs with CIs, or at least bootstrap or McNemar-style paired comparisons over tasks, are needed.

  2. The headline history result is confounded and not resource-matched. The paper acknowledges that both history configurations also change the message template and parser (WebAgent-R1) relative to the baseline, so the gain “cannot be attributed to retained history alone.” No control isolates this, for example the baseline with the WebAgent-R1 template and parser but no old observations. Both history configurations also bypass the baseline input-fitting target (40K on WebArena, 24K on OpenApps). Full history uses a 50K estimated budget, which the appendix says does not bound the final request, and uses 2.4× to 5× the tokens. Masking trades tokens for extra steps and, on OpenApps, uses about the same tokens as the baseline while gaining 12 pp. The baseline-fitting policy also affects other configurations: 19 of 165 tasks were rejected for input length in “recent-observation append,” which then trails the history configurations. That comparison therefore mixes design quality with a context-overflow artifact. Most catalog entries are also single, untuned “adaptations” of published designs, with no per-configuration tuning (e.g., parse-error retry at 1.21%, which looks like an implementation or interaction problem more than a property of the idea). “No configuration of category X beats baseline” should be read as a statement about these implementations only.

  3. The workflow-memory evaluation is contaminated, and the evaluation protocol is not uniform. The library of 25 workflows was distilled by GPT-4o from 69 successful trajectories on the same WebArena-Lite tasks used for evaluation. The paper concedes that Table 4 “does not test transfer to unseen tasks.” The positive increments (+7.88, +4.85, +2.42, +1.21 pp) and the 20.61% that makes workflow memory one of the better WebArena configurations in Table 1 are therefore upper bounds. The declining-increment trend rests on five points from one run each, with a different thinking mode and a different API model, so it cannot be separated from the model-specific limits of this library. The OpenApps “offline workflow memory” is a different resource (eight hand-written workflows) under the same name and appears in the same table rows. Separately, the WebArena success labels are post-hoc adjudicated with model assistance (Claude, then a second GPT-5.6 review of a “designated subset”), and the paper does not make clear whether every configuration and the baseline received identical treatment or what the label changes were. OpenApps has no such adjudication. Because many differences are a few tasks, adjudication choices could shift rankings, and the paper should report both raw and adjudicated numbers per configuration.

Minor points:

  1. The two settings differ in both environment and model (9B vs. 27B), so the paper cannot say whether the history gap (2.42 pp vs. 26 pp) is due to the environment or the model, and the text says so only in a caption.

  2. The guidance (“evaluate full history first”) largely repeats prior findings on history and observation masking (Enomoto et al., 2026; Lindenbauer et al., 2025), so the novelty lies mostly in the breadth of the catalog.

  3. Only text/AXTree observations and small open models (plus one API model on one task suite) are covered.

Questions for authors
  1. Can you add run-to-run variance? At minimum, rerun the baseline, the two histories and a few borderline configurations with several seeds, and report CIs or paired significance tests per task. How many of the 30 configurations, and which of the combination conclusions in Table 3, remain distinguishable from the baseline once this is done?

  2. Can you run controls that separate the history effect from the template and parser change and from the context budget? For example: (a) baseline with the WebAgent-R1 template and parser but no old page bodies, (b) full history under the same token cap as the baseline, and (c) token- and step-matched comparisons for the configurations that make extra model calls (model-based planning, critics, line retrieval). Without them, “multi-turn history” is not clearly the cause of the gain.

  3. For the workflow library, can you build it from a disjoint set of tasks (or by task split) and re-run Table 4? And was the post-hoc adjudication applied identically to the baseline and all 30 configurations? Please report raw (evaluator-only) and adjudicated success per configuration, and say how many labels changed.

Reviewer_7EGQOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper studies which web-agent harness choices are worth adopting. It implements 30 configurations across six functional categories on a GenericAgent-based framework and evaluates them on WebArena-Lite with Qwen3.5-9B and OpenApps with Qwen3.5-27B. Full observation history and observation masking lead both catalogs, with substantial differences in success and token use. Selected combinations are also tested. A separate five-setting model comparison finds decreasing gains from one fixed workflow library as baseline success rises. The main contribution is the comparative framework and empirical guidance.

Critical strengths
  1. The shared framework makes a broad set of practical harness choices easier to compare. Including irrelevant-content controls and comparing combinations against both individual components are useful design decisions.

  2. The OpenApps history comparison is informative: full history solves 57 tasks versus 31 for masking, with much higher token use. Reporting shared and exclusive successes makes this trade-off more useful than an aggregate score alone.

Critical weaknesses
  1. The history recommendation needs a cleaner control. CTX1/CTX4 also change the message template, parser, and input handling. History benefits were already shown by Enomoto et al.; a matched comparison is important to establish what this broader catalog adds beyond that finding.

  2. The full 30-configuration study is limited to Qwen models. GPT-5.6 Sol receives only a small comparison of workflow memory and self-feedback, while Claude is not evaluated. Neither addition improves GPT’s baseline in Table 4 (52.73%, 50.91%, and 52.73%). This raises concerns that the evaluation findings can't work on stronger models. A comparably broad evaluation on stronger models is needed to establish whether the main harness recommendations generalize.

  3. The finer comparisons are difficult to assess. WebArena scores include selected post-hoc adjudications, but correction counts and original scores are not reported. This matters for conclusions such as the three-task loss from workflow memory on GPT and the six-task gain from adding error feedback on OpenApps.

Questions for authors
  1. Can you compare no history, response-only history, and full history using the same template, parser, and enforced input limit? Report success, tokens, and input rejections. Does the history ranking also hold for GPT-5.6 Sol or Claude?

  2. For each WebArena configuration, how many labels changed in each direction after post-checking? Please report original scores and the criteria for selecting cases for each review stage.

  3. Why does parse-error retry reduce success from 15.76% to 1.21% on WebArena? Does ACT1 change anything besides retrying failed parses, and what causes the additional failures?

  4. Figure 1 seems to drawn by AI, but not disclosed in the Section AI USE STATEMENT.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.