- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper aims to comprehensively compare 30 agent harness designs, inspired from various existing works like AgentOccam, WebDreamer, AgentLab/BrowserGym, AWM, etc for web navigation tasks. The evaluate these 30 agent harness designs on 2 benchmarks: WebArena-Lite w/ Qwen3.5-9B and OpenApps (100 tasks) w/ Qwen3.5-27B. These 30 designs are mapped to 6 functional categories that common web agents design for according to the survey conducted by (Guo et. al, 2026). The paper mainly finds: 1. most configurations score below baseline 2. Two history related configurations (include full trajectory history, and include masked trajectory history) and “Parse-error early stop” improve performance over baseline for both OpenApps and WebArena 3. Based on experiments with configurations that supply in context examples to the agent (called Offline workflow memory, they find that the returns from supplying examples diminishes as model intelligence improves.
All agent configurations are built upon the GenericAgent created by ServiceNow (https://github.com/ServiceNow/AgentLab). The original GenericAgent already sets up the prompting pipeline, keeps observation preprocessor, observation history, prior actions, internal state containing plan, memories, thoughts, retry loops etc - thus containing some of the agent configurations already. However, to use this framework as a baseline, the authors create a stripped down version of GenericAgent, keeping only the goal, the current AXTree, and past actions. Meanwhile, screenshots, HTML, past error logs, past reasoning, planning, memory, and feedback are all turned off in the baseline setting.
Critical strengths
- The paper does a controlled comparison of multiple harness designs, on a shared baseline agent framework, backbone model and evaluation tasks. This provides a useful measure for readers to choose which agent design categories to incorporate when building their web navigation agents.
- Such a study can help show what agent harness settings might not bring useful gains to end-to-end performance. Table 4 also conducts a useful study for a specific harness configuration, to show that gains from these designs can vary significantly across different model capability tiers.
Critical weaknesses
- Lack of insight into negative results: The paper does not provide an insight into why configurations like planning, observation reductions etc. underperform. Thus while we can see relative performances on WebArena and OpenApps on these 30 settings, there is little guidance that the community can get from why certain configurations can score lower. It is unclear why Subgoal planning or Parser error retries cause a significant hit to the performance, and can indicate potential implementation bugs in the absence of accompanying error analysis.
- Potential lack of generalizability: The study is conducted using one model, one environment combinations. All harness categories are tested against a single baseline agentic framework. Thus the trends seen across the harness configurations might be confounded. The study is most useful only if a reader can walk away with generalizable takeaways and guidances on good agentic designs.
- Incremental findings: The main finding from the paper is that adding history of the trajectory helps, but cited works by Enomoto et al[1] already show that result. Another finding is that masking history reduces token usage but at the expense of performance. Both these findings are incremental over existing works on web agents.
[1] Masafumi Enomoto, Ryoma Obara, Haochen Zhang, and Masafumi Oyamada. Read More, Think More: Revisiting Observation Reduction for Web Agents. arXiv preprint arXiv:2604.01535, 2026. URL https://arxiv.org/abs/2604.01535v1
Questions for authors
Questions:
- L319 say: "Relative to baseline, both history configurations also change message organization and parsing, as § 2.2 specifies, so their differences from baseline cannot be attributed to retained history alone." This refutes the main finding of the paper. Can the authors clarify?
Suggestions:
- L112: The concept of workflow library can be introduced more clearly.
- The shorthand FMT for the baseline agent is never explained. The baseline agent setting being a stripped down version of GenericAgent, is also only clarified in Appendix A.2.
- Statistical significance concerns: The agent runs are conducted with temp=1.0, and with single runs per setting. Without proper confidence intervals, it is hard to rank these configurations and figure out what results are within noise limits from each other.