acceptodds
Under review as a conference paper at ICLR 2027

LongWebBench: Fine-Grained Evaluation of Agents on Long-Horizon Web Tasks

Abstract

Agents powered by large language models are increasingly asked to complete long-horizon tasks that span hundreds of dependent interactions. Evaluating them calls for tasks that are long by construction, an environment that is controllable yet disrupts the agent as real runs are disrupted, and an evaluation that shows where a run falls short, but existing benchmarks rarely provide all three. To fill this gap, we introduce LongWebBench, a benchmark of long-horizon web data collection on generated, self-hosted websites. Each of its 85 tasks, spanning five difficulty levels, asks the agent to collect a complete dataset, such as every product of an online store with all its attributes, amounting to several hundred to a few thousand items spread over listing, detail, and login-protected pages. During collection, verification gates, expiring sessions, truncated responses, and obfuscated pages disrupt the run. Because the server records what it delivers, LongWebBench compares what a task requires, what the server delivered, and what the agent submitted, and it diagnoses each run at four stages, namely Interface Access, State Control, Data Acquisition, and Result Assembly. We evaluate ten agent systems on the benchmark and find that even the strongest remains far from complete collection and that rankings on the easiest level do not carry over to the hardest. The diagnosis further shows that agents fail mainly by omission, since most of the missing information was never delivered to them rather than misread, and that the backbone model shapes what is collected far more than the agent harness does. A stage-wise evaluation grounded in delivery records thus reveals what a single score hides and where agents need to improve.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, Long-horizon tasks, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ujN1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a benchmark called LongWebBench, a large-scale web data collection consisting of multiple pages. It comprises 85 automatically generated tasks. To avoid contamination from prior knowledge and to conduct evaluations in a controllable environment, the authors host the website themselves and perform the tasks within it. Furthermore, by conducting detailed evaluations at each stage, rather than relying solely on the final score, they make it easier to analyze the causes of the agent’s failures. The experimental results show that even the latest agents still have room for improvement.

Critical strengths
  • S1. I believe that developing tasks that utilize long-term memory and short-term memory is a timely and important area of research.

  • S2. Few benchmarks conduct stage-wise evaluation, and I consider this aspect to be beneficial.

  • S3. The fact that they built their own website can be highly evaluated. While questions remain regarding how closely it resembles real-world websites, this approach clearly eliminates issues such as contamination from pre-training knowledge and makes the environment controllable.

Critical weaknesses
  • W1. Similarity to existing benchmarks: Tasks such as “collecting a complete dataset, such as every product from an online store” are similar to the “Massive Memory” tasks in WebChoreArena.

  • W2. Narrow scope: This benchmark is specialized solely in web data collection and has only 85 tasks. Therefore, it seems to lack comprehensiveness and variety in the tasks.

  • W3. The 85 tasks in LongWebBench are automatically generated by a program. It would be necessary to analyze whether these tasks were all manually checked and, if solved manually, what the accuracy rate is or whether there are any tendencies to make mistakes. For example, adding a “Human Baseline” to Table 2 could be beneficial.

  • W4. It is difficult to interpret the results based solely on the weighted Overall Score. The authors should report the “Strict Task Success Rate.” This would provide readers with results that are easier to interpret, showing the current percentage of tasks that can actually be executed.

Questions for authors

For more details, please refer to “Critical Weaknesses.” In particular, I would like the authors to clarify the similarity to existing benchmarks (W1) and the validity of a narrow scope (W2), and task validity (W3).

5 comments
AuthorsOpenReview ↗

Response to Reviewer ujN1 (1/3)

We sincerely thank you for your thoughtful review and for the time you devoted to our paper. We are encouraged that you view the topic as timely, the stage-wise evaluation as beneficial, and the self-built websites as free of contamination and controllable. Your questions on similarity, scope and task validity prompted new analyses that we believe have improved the paper, and we are grateful for them. We respond to each of your concerns below.

W1. Similarity to existing benchmarks

The tasks, such as collecting every product from an online store, are similar to the Massive Memory tasks in WebChoreArena.

R1: Thank you for this comment. We agree that the task format resembles the Massive Memory tasks, and we have reframed our claim to make the difference explicit. In short, LongWebBench tests whether an agent can complete an exhaustive collection without being told the global target count. We develop this answer in three steps below.

1. The difference from Massive Memory. A Massive Memory task states the slice to collect, such as the top 50 items, so the agent knows when it is done, and the difficulty lies in retaining what it has read. In contrast, our tasks ask for every record, hide the size of the set and place part of it outside the default listing. The agent therefore has to judge by itself when the set is complete.

2. This judgment is where agents lose. Across 24 full L3 runs, field accuracy stays flat along the listing, from 0.593 in the first quarter to 0.589 in the last, so agents do not forget as the run grows longer. Moreover, doubling the time limit gives no gain in Overall. Instead, the losses come from a wrong belief that the task is finished. Among the 57 archived runs that missed records, 46 report the set as complete in their final message, and such runs stop after about 38% of their time. They stop early while the global target count remains undisclosed and completing the default listing need not establish global completeness.

To test whether this belief causes the losses, we supplied the missing knowledge in the prompt and observed whether the losses disappeared. We ran this test on the 20 L2 tasks with eight systems in four harnesses. Each system received the task in up to four forms, and Table 1 reports the outcome.

  • No addition. The original prompt.
  • Check instruction. An instruction to verify completeness before submitting, without giving any number.
  • Record count. The exact number of target records.
  • Count and location. The count plus where the hidden records are.

Table 1: Complete submissions out of 20 L2 tasks, with Overall in parentheses.

SystemNo additionCheck instructionRecord countCount and location
Claude Opus 5, Claude Code1 (76.6)1 (76.3)20 (82.9)20 (82.9)
GPT-5.6 Sol, Codex0 (76.0)0 (76.1)20 (83.6)20 (83.3)
DeepSeek V4.1 Flash, Claude Code14 (81.6)14 (80.8)19 (81.2)20 (83.6)
GPT-6 Astra, Codex4 (80.3)7 (81.4)20 (86.2)20 (85.4)
DeepSeek V4.1 Flash, Pi11 (80.1)13 (81.7)19 (80.6)20 (83.1)
DeepSeek V4.1 Flash, Codex1 (73.5)1 (73.3)19 (78.6)20 (82.6)
Gemini 3.8 Flash, Antigravity2 (74.9)7 (73.4)18 (79.3)18 (75.4)
GLM-5.3, Claude Code9 (73.5)9 (73.2)15 (66.0)16 (71.0)

The results lead to three conclusions.

  • The count alone is sufficient. With the count, every system submits the complete set on at least 15 of the 20 tasks, and six of the eight systems do so on 19 or 20. Without it, Opus 5 and Sol do so on only 1 and 0 tasks.
  • An instruction to check helps little. The number of complete sets stays unchanged or rises only slightly, and no system gains significantly in Overall. Hence the agents are not careless. They lack a number to check against.
  • The location adds little. Once an agent knows how many records exist, it finds the hidden ones by itself, and adding their location changes at most one task per system.

Taken together, the losses trace back to one missing piece of knowledge, namely the size of the target. The task requires record completeness and timely submission: an agent must determine whether it has collected the full target set and save a usable answer within the budget. Unlike Massive Memory, determining completeness here requires establishing the collection scope, beyond retaining what the agent has read.

AuthorsOpenReview ↗

Response to Reviewer ujN1 (2/3)

R1 (continued): 3. What the three properties enable together. LongWebBench combines controlled target sets with delivery-based evaluation to study completeness and locate omissions in the same environment.

  • Long-horizon task with a full record set defines the collection objective. The global target count is undisclosed, and from L2 onward, completing the default listing does not establish global completeness.
  • Controlled environment enables the disclosure intervention. Because we generate and host the sites, the true record set is known exactly, so we can disclose or withhold its count while holding the site content fixed.
  • Delivery-based evaluation locates omissions relative to recorded delivery. The server logs distinguish pre-delivery coverage gaps from post-delivery output losses. Transcripts and prompt interventions then help investigate why collection stops short.

Table 2 compares the completion criteria and omission scoring of earlier benchmarks with this combined measurement.

Table 2: Completion signal and omission scoring in earlier benchmarks.

BenchmarkDone whenOmission seen as
WebArena, WorkArena++The target state is reachedNot applicable, single end state
WebChoreArena Massive MemoryThe stated slice is collected, such as top 50A wrong final answer
Open CaptchaWorldOne CAPTCHA is solvedNot applicable, pass or fail
Fayolle et al. (2026)One form is submitted behind bot controlsNot applicable, submission logged
ContractBenchThe stateful API calls succeedNot applicable, server-log reward
WAREXThe original task is done under injected faultsThe original benchmark's score
WideSearchOpen-ended collectionMissing rows in the final output, on the live web
LongWebBenchGlobal target count undisclosedMissing records and fields, before or after recorded delivery

The first six rows evaluate a stated collection slice, a target state, or a gate or contract. WideSearch evaluates open-ended collection against curated gold tables on the live web. Our generated environment fixes the target set and links record- and field-level delivery to the submission, enabling target-count interventions and diagnosis of omissions before or after delivery.

Revision. We revised the abstract and the introduction to state this capability. Table 2 above has been added to Section 2, where our tasks are also contrasted with Massive Memory, and the experiment is reported in the new Section 4.4 and Appendix H.

AuthorsOpenReview ↗

Response to Reviewer ujN1 (3/3)

W2. Narrow scope

The benchmark covers only web data collection and has only 85 tasks, so it seems to lack comprehensiveness and variety.

R2: Thank you for your insightful comment. The focus on collection is deliberate, because attributing each missing record requires a task family whose full answer key the environment holds. Within this family, the 85 tasks cover 20 archetypes, five levels and several site types. Moreover, since tasks are generated from seeds, the number of tasks can grow beyond 85. However, each task is already long-horizon and costly. For example, running all 85 tasks with GPT-5.6 Sol costs over $400 and takes about 24 hours. We therefore keep the released set at 85 tasks. We also agree that our conclusions should not be generalized beyond collection tasks, and we plan to expand the diversity of tasks in future updates.

Revision. This scope and our plan to extend it are now stated in the limitations of Appendix K.

W3. Task validity / human baseline

The 85 tasks are generated by a program. Were they all manually checked, and what accuracy do humans reach on them? A human baseline in the leaderboard table would help.

R3: Thank you for your detailed comment. We answer the two parts of this concern in turn.

Are the tasks correct and solvable? Because the generator writes both the site and the ground truth, the tasks are correct by construction. To check solvability, we ran a scripted reference crawl that completes each task through the same interfaces as the agents. The scripts retrieve the full results on all L1–L3 tasks. For L4 and L5, a simple script is not enough to validate solvability. We therefore gave GPT-6 Astra computer-use capability, a more concrete prompt about the data range and an unlimited budget. It retrieves 97.4% of the results at L4 and 75% at L5, which shows that most of the tasks are solvable.

What do humans achieve? Because human experts are costly, we do not benchmark human performance directly. Instead, a human expert solved the tasks jointly with Opus 5 in Claude Code. Table 3 reports the results.

Table 3: Overall of Claude Opus 5 with human interaction.

MethodL1L2L3L4L5
Claude Opus 577.278.070.266.549.5
Claude Opus 5 + human interaction96.189.688.780.466.3

As Table 3 shows, Opus 5 improves substantially with human guidance. Our analysis of the trajectories attributes the gain to a more efficient collection strategy and a more thorough search for hidden records. This result is consistent with our response to W1. Meanwhile, even a frontier model with human interaction leaves large room for improvement at the hardest levels.

Revision. The solvability check is now reported in Appendix B.5, and we added Table 3 above to Appendix F.7.

W4. Strict task success rate

The weighted Overall score alone is hard to interpret. Please report the strict task success rate.

R4: Thank you for your constructive comment. We agree that metrics that directly reflect the final results are beneficial. We therefore report record recall, field accuracy, precision and strict success rate in Table 4. Here, a task counts as a strict success only when its recall, accuracy and precision all exceed 0.999.

Table 4: Submission metrics on all 85 tasks.

ModelRecord recallField accuracyPrecisionStrict success
Claude Opus 584.263.195.01.2 (1/85)
DeepSeek V4.1 Flash77.957.684.311.8 (10/85)
GLM-5.376.051.088.42.4 (2/85)
GPT-5.6 Luna73.247.288.90
GPT-5.6 Sol73.151.996.50
GPT-5.6 Terra65.843.786.30
Sonnet 4.6, low effort53.031.068.20
Qwen3.8-Flash48.126.957.80
GLM-5.3-Flash28.316.934.30

As Table 4 shows, the models rarely achieve strict success, and record recall and field accuracy, not precision, are the main bottlenecks.

Revision. The strict success rate is now cited in Section 4.2, and we added Table 4 above to Appendix F.5.

Questions

Please clarify the similarity to existing benchmarks (W1), the narrow scope (W2) and the task validity (W3).

R5: Thank you for your questions. Please refer to our responses R1–R3 above.

Reviewer_ZNx1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

LongWebBench is a benchmark of 85 web data-collection tasks on generated, self-hosted websites. Each task asks an agent to export a complete dataset of several hundred to a few thousand records, while login walls, verification checks, expiring sessions, and obfuscated pages disrupt the run. Its main strength is the evaluation. Because the server logs what it delivers, the benchmark compares what the task required, what the agent was served, and what it submitted, and splits each run into four stages (Access, Control, Acquire, Assemble). This separates failure modes a single score hides - never reaching a page, losing a session, opening a page without extracting its fields, or dropping collected data from the submission - and it is fully automated. The resulting findings are informative: agents fail mainly by missing data rather than misreading it, Control is the weakest stage, and the best collector and the best assembler are different systems. The benchmark and evaluator code are released.

The main weaknesses are in how well the evidence supports the claims. The stage labels are tested only on hand-constructed runs, never checked against human review of real runs (W1). The detailed analysis of where data is lost excludes six of the ten systems, including the two strongest (W3). The claim that rankings change from easy to hard levels rests on one run per task and small per-level samples, with no run-to-run variance reported (W4). The “long by construction” claim is weaker than presented: most tasks are long in volume rather than in sequential steps, length is never compared with other benchmarks, and the evidence sits in the appendix (W5). Finally, the lower levels already score high, so the benchmark's ability to rank systems rests mostly on its 27 hardest tasks (W2).

Current assessment: weak accept

Critical strengths
  • The stage-wise diagnosis is a genuinely insightful tool. Splitting each run into Interface Access, State Control, Data Acquisition, and Result Assembly reveals what a single score hides, and Table 2 yields several non-obvious findings: (i) Access is near ceiling for every system, so the losses arise later in the run, not at the point of entering the site; (ii) Control has the lowest values of any stage (pass rates 8–32%). Agents struggle to sustain through authentication, cookie chains, and URL session identifiers (iii) despite near-perfect Access, Acquire stays at 59–75, so reaching the site does not mean collecting it. On L1–L3 agents reach ≥85% of default-listing records but receive only 35–70% of detail fields (§4.3); (iv) Assemble varies most across systems (51–82), so agents often fail to carry data they were served into the final submission, partly through runs that hit the time limit before submitting (§4.4); and (v) the best Acquire and the best Assemble belong to different systems (DeepSeek V4.1 Flash and Claude Opus 5), which Overall alone would hide.
  • Deceptively simple, yet far from solved. The task sounds simple - export every record of a website and yet current agents do poorly on it. The best system scores 70.1% overall (Table 2) i.e. no system acquires more than three quarters of the required records and fields (§4.2), and on the hardest level, the best of the four systems in Figure 3 scores only about 49%. The authors have found a useful spot: a task that is easy to state and easy to check, but that today's strongest agents still cannot complete.
Critical weaknesses
  • The room left for improvement sits mostly in the hardest levels; the lower levels already score high and will saturate quickly.

    • Lower levels barely separate systems. At L1 the four systems in Figure 3 score 77–84 and lie within seven points of each other (§4.2). On the 58 L1–L3 tasks, DeepSeek V4.1 Flash already reaches Acquire scores of 88.8–90.0 in every harness (Table 3).
    • Ranking power rests on a minority of tasks. The spread between systems grows with level, from within seven points at L1 to over 25 points at L5 (§4.2), and L5 alone orders the four systems exactly as Overall does. The ranking therefore rests mainly on the 27 L4–L5 tasks out of 85. As models improve, L1–L3 will stop being informative well before the benchmark as a whole saturates.
    • Suggested resolution: Since the sites are generated from a specification and seed (App. B.2), discuss how difficulty can be scaled beyond L5 (record volume, discovery depth, challenge density) and whether the authors plan to refresh or extend the levels — this would also help their case for the benchmark's longevity.
  • All results rest on a single reported run per task, so run-to-run variation is never measured.

    • §4.1 states: “we report the selected run for each task rather than the best of several attempts.” Only one run per task enters the results, so there is no pass@k and no mean ± standard deviation across repeated runs. “Selected” is also never defined - if several attempts existed, it is unclear how the reported one was chosen.
    • Why it matters. Long agent runs are noisy: the same system on the same task can succeed in one run and time out in the next. The confidence intervals the paper does report (Tables 2-3) is over tasks, not repeated runs, so they do not show how much a system's score would change if it were run again. Without repeated runs, it is unclear which of the reported differences between systems are real. For example, whether the systems in the middle of Table 2, whose intervals already overlap, are actually distinguishable.
    • Suggested resolution. Run each system several times per task (e.g. three runs), report the mean ± standard deviation or pass@k, and state how many attempts were made and how the reported run was selected (see Q4).
  • The claim that tasks are long by construction is under-supported: Most tasks are long in volume rather than in sequential decisions, and length is never compared quantitatively with prior benchmarks.

    • Most tasks are wide rather than deep. By the paper's own dependency-depth measure (Appendix B.4, Table 8), 68 of the 85 tasks have a depth of only 4–10: with numbered or next-link pagination, every listing page can be requested in parallel once the first page reveals the page count, which the authors themselves call wide rather than deep. Only the 17 cursor-paginated tasks (five at L4 and all twelve at L5) require long chains of dependent requests (99–146).
    • Site requests is a bad length metric because the tasks are automatable. Agents write automation/collection code that issues many HTTP requests per tool call. A 10-line scraper loop produces thousands of requests in a single turn. So the 842–5,953 requests overstate the horizon, and the agent's actual decision horizon is the turn count.
    • "Long by construction is the first of their three pillars, yet the only quantitative evidence for it sits in the appendix. Moving Table 8 (or a compact version) into §3.1 would let readers check the claim directly. It would also make the length picture visible: L1–L2 runs are short (38–52 turns, 7–9 minutes), and real depth exists only in the 17 cursor tasks.
    • Suggested resolution. Add a quantitative length comparison with prior benchmarks, report the agent's decision horizon (turns) rather than site requests as the primary length measure, and either scope the long-horizon claim to the cursor-paginated tasks or add more tasks whose length comes from sequential dependence rather than volume.
  • Several parts of the claimed novelty are overstated. What is new is the combination and the record-level diagnosis, not the individual ingredients.

    • The perturbations are standard web-navigation problems. Login walls, CAPTCHA-style challenges, timing checks that wait for human-like interaction, browser checks, and expiring sessions (Fig. 2b; §3.2) are well-known obstacles in web navigation, and the paper itself cites prior work that tests them (Open CaptchaWorld, ContractBench, WAREX). The authors state that the challenges simulate common web mechanisms, so building them into the benchmark is not itself a contribution, yet disruption is presented as one of the benchmark's three pillars. Moreover, their effect is never isolated: no ablation removes them.
    • Server logs as evidence are not new. Using environment-side logs to judge what an agent did is established practice, as the paper's own related work shows: Fayolle et al. (2026) use server logs to check whether an agent submits a form, and ContractBench scores agents with server-log rewards. The authors concede that prior work already covers set-valued outputs, access barriers, state requirements, and process-level evidence. The genuinely new part is narrower: linking delivery logs to every record and field of a large submitted dataset.
    • Suggested resolution. Reframe the contribution around what is new. The combination of large multi-record tasks with record + field-level diagnosis which is grounded in server delivery. Present the perturbations, server logging, and task length as supporting design choices rather than contributions in their own right.
Questions for authors

Q1. What purpose does the Access diagnostic serve? Access passes as soon as a single page with data is reached. Unsurprisingly, it is near ceiling for every system and therefore separates no systems and says little about whether agents get through the benchmark's challenges. Why report it as one of the four headline stages? Would it be better to drop Access be from the headline diagnostics?

Q2. Why don't agents open detail pages? Access passes for nearly every run, so agents do reach the site, yet Acquire stays at 59–75. The paper locates the loss: "Before delivery, agents mostly leave detail pages unopened or never reach a record" (§4.3). On L1–L3 agents reach at least 85% of the default-listing records but receive only 35–70% of the detail fields. Once a detail page is fetched, its values are rarely misread, though some come back in the wrong format. So the problem is that agents do not open these pages, not that they misread them. The paper does not explain why. What makes detail pages different from listing pages? Is it their cost (one request per record), the opaque identifiers that force agents to follow each link rather than construct URLs, or the time limits?

Q3. How was the selected run in results table chosen? §4.1 says "we report the selected run for each task rather than the best of several attempts." Why report a single selected run instead of the standard practice of repeated runs with mean ± std or pass@k? How many attempts were made per task, and what were the selection criteria?

7 comments
AuthorsOpenReview ↗

Response to Reviewer ZNx1 (1/5)

We are sincerely grateful for your thorough and constructive review, and for the effort you put into suggesting a concrete resolution for every weakness. We especially appreciate your recognition that the stage-wise diagnosis reveals what a single score hides and that the task, though easy to state and check, remains far from solved. Your suggestions led to new experiments and to more careful claims, and the paper is stronger for them. We respond to each concern below.

W1. Lower levels saturate

The lower levels already score high and will saturate quickly. At L1 the systems lie within seven points of each other, and the ranking rests mainly on the 27 L4–L5 tasks. Please discuss how difficulty can scale beyond L5 and whether the levels will be refreshed or extended.

R1: Thank you for this comment. We agree that the current ranking power among the strongest systems sits in L4–L5. To obtain a more comprehensive understanding of the current results, we conducted the following experiments.

The lower levels still separate weaker models. Our submitted results focused on frontier models and harnesses, which are highly capable and do well on the lower-level tasks. However, our additional experiments in Table 1 show that the lower levels remain challenging for smaller models.

Table 1: Overall of weaker models on L1–L3.

SystemL1L2L3
Qwen3.5-35B-A3B30.429.08.8
GLM-4.5-Flash46.226.66.5
Qwen3.6-Flash74.449.225.2
Sonnet 4.6, low effort68.756.839.6
Qwen3.6-27B71.769.842.3
Strong systems (DeepSeek V4.1 Flash, GLM-5.3, GPT-5.6 Sol, Luna, Terra)77.8–84.070.9–81.759.3–69.3

Specifically, Qwen3.5-35B-A3B reaches only 8.8 at L3, and the strict success rate at L1 is still 0% for every system. Thus, the lower levels remain far from solved and stay informative for smaller, edge or on-device models.

Harder levels follow from the same specification. More records alone do not raise difficulty, since halving the record count changes DeepSeek V4.1 Flash by under 3 points. Our new experiments instead show that agents lose records when the target extends beyond what they have seen, so a harder level scales the parameters that hide the target. Because every site is generated from a specification and a seed, three such parameters can grow beyond L5 without new engineering. First, the share of records absent from the default listing, which rises from 0% at L1 to 45% at L5, can grow further and spread over discovery routes that only open from other detail pages. Second, the sequential depth can grow by using single-use cursors on every route and shortening their lifetime below the current 20 seconds. Third, the challenge density can grow by placing expiring state, obfuscation and decoys on more pages of each task. Since new instances are drawn from unreleased seeds, such levels can be added without exposing the released tasks. We also plan versioned refreshes, in which a new level is released once the strongest system saturates the current hardest one.

Revision. We revised Section 4.2 to note that the lower levels still separate weaker models. Table 1 above has been added to Appendix F.6, and the three scaling axes and the refresh plan are described in Appendix B.2.

W2. Single run per task

All results rest on a single run per task, so run-to-run variation is never measured and it is unclear which differences between systems are real. Please run each system several times, report mean ± std or pass@k, and state how the reported run was selected (see Q3).

R2: Thank you for this constructive comment. We reran three systems on all tasks to evaluate run-to-run variance.

Table 2: Overall over three runs, mean ± std.

SystemL1L2L3L4L5
DeepSeek V4.1 Flash80.2 ± 0.681.9 ± 1.465.0 ± 1.341.1 ± 2.441.7 ± 3.5
GPT-5.6 Sol84.3 ± 0.275.7 ± 0.460.4 ± 1.960.8 ± 2.144.9 ± 3.0
Claude Opus 577.5 ± 1.077.8 ± 0.671.4 ± 2.066.8 ± 1.949.8 ± 2.2

As Table 2 shows, the systems are stable at the lower levels, and the variance grows with the length of the tasks. Moreover, at the harder levels, stronger models vary less between runs, and the systems remain separable under this variance.

Revision. The repeated runs are now summarized in Section 4.1, and we added Table 2 above to Appendix F.4.

AuthorsOpenReview ↗

Response to Reviewer ZNx1 (2/5)

W3. "Long by construction" is under-supported

The claim that tasks are long by construction is under-supported. Most tasks are wide rather than deep, site requests overstate the horizon because agents automate them, the quantitative evidence sits in the appendix, and length is never compared with prior benchmarks. Please report turns as the primary length measure, add a comparison with prior benchmarks and scope the claim.

R3: Thank you for this careful analysis. We agree with it and have narrowed the claim accordingly.

  • Turns as the length measure. Section 3.1 now reports length in LLM turns as the main measure, with site requests as a secondary one, and it points to the horizon table in Appendix B.4.
  • Scope of the claim. We now state that L1–L2 runs are short at 38–52 turns, that 68 of the 85 tasks are wide rather than deep, and that sequential depth comes from the 17 cursor-paginated tasks.
  • Comparison with prior work. The revision adds Table 3 below, and Section 3.1 summarizes it.

Table 3: Steps per task across benchmarks.

BenchmarkReported mean steps / LLM calls per task
WebArena6.8
VisualWebArena4.7
WorkArena L19.0
WorkArena++ L2 / L333.8 / 25.0
AssistantBench7.5
Odysseys41.7–86.4
TheAgentCompany14.55–39.85
LongWebBench L1 / L238 / 52
LongWebBench L3–L5175–214

The first five rows use the mean steps per episode that BrowserGym reports for GenericAgent with Claude 3.5 Sonnet, with budgets of 30 steps for WebArena, VisualWebArena, WorkArena L1 and AssistantBench, and 50 for WorkArena++ L2–L3. The other comparison rows use ranges of model-level means from Odysseys (Table 2), under a 100-step budget, and TheAgentCompany. Both define steps in terms of LLM calls. These are observed execution lengths under different models, harnesses and budgets, not minimum steps required for successful completion.

Revision. We revised Section 3.1 to narrow the claim. The horizon measurements are given in Appendix B.4, where Table 3 above has been added.

AuthorsOpenReview ↗

Response to Reviewer ZNx1 (3/5)

W4. Novelty is overstated

The novelty is overstated. The perturbations are standard web-navigation obstacles and no ablation isolates their effect, and server logs as evidence are not new. What is new is the combination of large multi-record tasks with record-level and field-level diagnosis grounded in server delivery. Please reframe the contribution around this.

R4: Thank you for this detailed comment. We agree with it and have reframed our claim accordingly. We address your points in turn.

1. Perturbations, server logs and length are supporting design choices. We agree that prior benchmarks already test login walls, CAPTCHA-style challenges and expiring sessions, and that prior work already uses server logs to judge what an agent did. The revised paper therefore presents them as supporting design choices, which keep a single scripted loop from completing the collection. We also ran an additional ablation that turns the access gates off on the 18 L3 tasks, and the Overall of Opus 5 rises by only 2.7 points. The gates therefore act like a supporting design choice and not the main source of difficulty.

2. What the record-level linkage enables. The linkage that you identify as new matters because it lets us ask why records go missing, which answer-only scoring cannot. With it, LongWebBench tests whether an agent can complete an exhaustive collection without being told the global target count. Among the 57 archived runs that missed records, 46 report the set as complete in their final message. To test whether this belief causes the losses, we gave eight systems in four harnesses the 20 L2 tasks in up to four forms, namely the original prompt, an instruction to verify completeness without any number, the exact number of target records, and the count plus the location of the hidden records.

Table 4: Complete submissions out of 20 L2 tasks, with Overall in parentheses.

SystemNo additionCheck instructionRecord countCount and location
Claude Opus 5, Claude Code1 (76.6)1 (76.3)20 (82.9)20 (82.9)
GPT-5.6 Sol, Codex0 (76.0)0 (76.1)20 (83.6)20 (83.3)
DeepSeek V4.1 Flash, Claude Code14 (81.6)14 (80.8)19 (81.2)20 (83.6)
GPT-6 Astra, Codex4 (80.3)7 (81.4)20 (86.2)20 (85.4)
DeepSeek V4.1 Flash, Pi11 (80.1)13 (81.7)19 (80.6)20 (83.1)
DeepSeek V4.1 Flash, Codex1 (73.5)1 (73.3)19 (78.6)20 (82.6)
Gemini 3.8 Flash, Antigravity2 (74.9)7 (73.4)18 (79.3)18 (75.4)
GLM-5.3, Claude Code9 (73.5)9 (73.2)15 (66.0)16 (71.0)

With the count alone, every system submits the complete set on at least 15 of the 20 tasks, and Opus 5 and Sol move from only 1 and 0 tasks to 20. In contrast, an instruction to check helps little, and adding the location changes at most one task per system. The losses thus trace back to one missing piece of knowledge, namely the size of the target. This test needs both halves of the combination. The generated sites fix the true record set, so its count can be disclosed while the site content stays the same, and the delivery records show whether a missing record was lost before or after delivery.

Reviewer_A5njOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces LongWebBench, a benchmark for long-horizon web data collection. Agents are asked to collect complete datasets spread over listing / login-protected pages, while the environment throws in "disruptions" e.g. verification gates or expiring sessions. Through experiments the authors find that even the best agents cannot really handle complete collection, that they mostly fail by omission.

Critical strengths
  • I like the delivery-record idea. Logging what the server actually served to separate "never fetched" from "fetched but dropped" is a neat diagnostic design.

  • Some findings are interesting and actionable, e.g., agents mostly skip pages rather & rankings on easy tasks do not generalize to hard ones, etc.

  • Self-hosting with seeded generation makes sense for reproducibility and avoiding pretraining contamination.

Critical weaknesses
  • Framing clarification. In general I'm still not fully sure what capability this benchmark tests beyond existing ones. Based on table 1 "long" seems to be describing the scale on the number of records and pages agents are required to visit, but it's still a bit fuzzy to me what additional challenges agents would run into that does not exist when their task is shorter / the definition of "complete" is less extensive. Is it more of drift? Persistence? What I find most interesting is (again) the delivery-based diagnosis, but in any case arguing more clarly for all the different properties covered in this benchmark & why the combination of them is necessary might be helpful.

  • Writing. The paper could be made easier to follow, especially in 4.3. The figures pack many categories and configurations together and the text moves quickly between them. I would recommend explicating 1-2 takeaways per paragraph/subsection and simplifying the main-text figures, leaving the full breakdowns to the appendix.

  • Ethics. I feel given the task objective this paper could deserve an ethical statement. Many of the challenges (CAPTCHA-style gates, browser checks, login walls) mirror mechanisms that sites use to say "please don't scrape this", and the benchmark essentially rewards getting past them. I worry that optimizing agents against this benchmark means optimizing them to circumvent access controls. It would help to discuss this explicitly, frame the setting as authorized collection, and maybe even include cases where the right behavior is to stop or ask, and score that.

Questions for authors

In your view, what's the most important capability getting tested in this benchmark? Why is this something that cannot be covered in prior benchmarks that have a subset of dimensions available?

5 comments
AuthorsOpenReview ↗

Response to Reviewer A5nj (1/3)

We sincerely thank you for the time and care you put into reviewing our paper. We are glad that you find the delivery-based diagnosis neat, the findings actionable, and the self-hosted design sound for reproducibility. Your comments on the framing, the writing and the ethics have made the paper considerably clearer, and we are grateful for them. We address each of your concerns below.

W1. Framing clarification

It is unclear what capability the benchmark tests beyond existing ones. Long seems to describe the scale of records and pages, but which additional challenge does it create, drift or persistence? Please argue more clearly for each property and for why their combination is necessary.

R1: Thank you for this constructive comment. It led us to reframe what the benchmark tests. In short, LongWebBench tests whether an agent can complete an exhaustive collection without being told the global target count. We develop this answer in three steps below.

1. Is it drift or persistence? We conducted experiments to test both, and neither explains the losses.

  • Drift. Across 24 full L3 runs, field accuracy stays flat along the listing, from 0.593 in the first quarter to 0.589 in the last. Thus, agents do not degrade as the run grows longer.
  • Persistence. Doubling the time limit gives no gain in Overall, and telling agents that they may exceed the request budget even lowers the scores through timeouts. Thus, time and cost are not the main obstacles.

2. The losses come from a wrong belief that the task is finished. Among the 57 archived runs that missed records, 46 report the set as complete in their final message, and such runs stop after about 38% of their time. The agents therefore do not run out of capacity. They stop early while the global target count remains undisclosed and completing the default listing need not establish global completeness.

To test whether this belief causes the losses, we supplied the missing knowledge in the prompt and observed whether the losses disappeared. We ran this test on the 20 L2 tasks with eight systems in four harnesses. Each system received the task in up to four forms.

  • No addition. The original prompt.
  • Check instruction. An instruction to verify completeness before submitting, without giving any number.
  • Record count. The exact number of target records.
  • Count and location. The count plus where the hidden records are.

Table 1: Complete submissions out of 20 L2 tasks, with Overall in parentheses.

SystemNo additionCheck instructionRecord countCount and location
Claude Opus 5, Claude Code1 (76.6)1 (76.3)20 (82.9)20 (82.9)
GPT-5.6 Sol, Codex0 (76.0)0 (76.1)20 (83.6)20 (83.3)
DeepSeek V4.1 Flash, Claude Code14 (81.6)14 (80.8)19 (81.2)20 (83.6)
GPT-6 Astra, Codex4 (80.3)7 (81.4)20 (86.2)20 (85.4)
DeepSeek V4.1 Flash, Pi11 (80.1)13 (81.7)19 (80.6)20 (83.1)
DeepSeek V4.1 Flash, Codex1 (73.5)1 (73.3)19 (78.6)20 (82.6)
Gemini 3.8 Flash, Antigravity2 (74.9)7 (73.4)18 (79.3)18 (75.4)
GLM-5.3, Claude Code9 (73.5)9 (73.2)15 (66.0)16 (71.0)

The results lead to three conclusions.

  • The count alone is sufficient. With the count, every system submits the complete set on at least 15 of the 20 tasks, and six of the eight systems do so on 19 or 20. Without it, Opus 5 and Sol do so on only 1 and 0 tasks. Accordingly, Overall rises by 6.2 points for Opus 5, by 7.5 for Sol and by 5.9 for Astra.
  • An instruction to check helps little. The number of complete sets stays unchanged or rises only slightly, and no system gains significantly in Overall. Hence carelessness explains little of the loss. The agents lack a number to check against.
  • The location adds little. Once an agent knows how many records exist, it finds the hidden ones by itself, and adding their location changes at most one task per system. Navigation is therefore not the obstacle.

Taken together, the losses trace back to one missing piece of knowledge, namely the size of the target, and this holds across model vendors and harnesses. The task requires record completeness and timely submission: an agent must determine whether it has collected the full target set and save a usable answer within the budget.

AuthorsOpenReview ↗

Response to Reviewer A5nj (2/3)

R1 (continued): 3. What the three properties enable together. LongWebBench combines controlled target sets with delivery-based evaluation to study completeness and locate omissions in the same environment.

  • Long-horizon task with a full record set defines the collection objective. The global target count is undisclosed, and from L2 onward, completing the default listing does not establish global completeness.
  • Controlled environment enables the disclosure intervention. Because we generate and host the sites, the true record set is known exactly, so we can disclose or withhold its count while holding the site content fixed.
  • Delivery-based evaluation locates omissions relative to recorded delivery. The server logs distinguish pre-delivery coverage gaps from post-delivery output losses. Transcripts and prompt interventions then help investigate why collection stops short.

Table 2 compares the completion criteria and omission scoring of earlier benchmarks with this combined measurement.

Table 2: Completion signal and omission scoring in earlier benchmarks.

BenchmarkDone whenOmission seen as
WebArena, WorkArena++The target state is reachedNot applicable, single end state
WebChoreArena Massive MemoryThe stated slice is collected, such as top 50A wrong final answer
Open CaptchaWorldOne CAPTCHA is solvedNot applicable, pass or fail
Fayolle et al. (2026)One form is submitted behind bot controlsNot applicable, submission logged
ContractBenchThe stateful API calls succeedNot applicable, server-log reward
WAREXThe original task is done under injected faultsThe original benchmark's score
WideSearchOpen-ended collectionMissing rows in the final output, on the live web
LongWebBenchGlobal target count undisclosedMissing records and fields, before or after recorded delivery

The first six rows evaluate a stated collection slice, a target state, or a gate or contract. WideSearch evaluates open-ended collection against curated gold tables on the live web. Our generated environment fixes the target set and links record- and field-level delivery to the submission, enabling target-count interventions and diagnosis of omissions before or after delivery.

Revision. We revised the abstract and the introduction to state this capability. The comparison above is now Table 2 of the paper, in Section 2. The experiments above are reported in a new Section 4.4, with the disclosure test as Table 5 of the paper, and the details are given in Appendix H.

AuthorsOpenReview ↗

Response to Reviewer A5nj (3/3)

W2. Writing

The paper is hard to follow, especially Section 4.3. Please give one or two takeaways per paragraph, simplify the main-text figures and leave the full breakdowns to the appendix.

R2: Thank you for this detailed suggestion. We agree and have rewritten the stage-wise analysis, which was Section 4.3 in the submission, along the three lines you recommend.

  • One takeaway per paragraph. Each paragraph now states its takeaway in its heading and then gives the evidence for it. The section has three takeaways. First, agents fail by omission and not by misreading. Second, as horizons grow, the bottleneck moves from fields to records. Third, pass rates misstate what the challenges cost.
  • Simpler main-text figures. The main figure on information loss now shows only the outcome of each required record–field pair, namely delivered and returned correctly, never delivered, and lost after delivery, with a separate share for runs without a submission.
  • Full breakdowns in the appendix. The full loss partition, the sub-components by level and the per-family results for state protocols now appear in the appendix, and the main text points to them where needed.

We applied the same rule to the rest of the experiments section, so every paragraph heading there now states a finding.

Revision. We rewrote Section 4.3 along these lines. The full breakdowns have been moved to Appendix G, with the sub-components in G.1, the verification gates in G.2, the protocol families in G.3 and the breakdown by level in G.4.

W3. Ethics (also the Ethics Flag)

The paper deserves an ethics statement. The challenges mirror mechanisms that sites use against scraping, so optimizing agents for the benchmark may optimize them to circumvent access controls. Please discuss this, frame the setting as authorized collection, and consider scoring cases where the right behavior is to stop or ask.

R3: Thank you for bringing this to our attention. We agree and have added an ethics statement to the paper. In our benchmark, all sites are synthetic and self-hosted, the task supplies the credentials, and the gates model steps that an authorized user also takes. This design lets us observe how agents pass the gates in a synthetic environment without causing any harm, and these observations can in turn help strengthen defense mechanisms. The statement also stresses that the benchmark should not be used to tune agents against real access controls.

On the other hand, we show that the benchmark can also test whether an agent respects such controls. We added a robots.txt file to the sites of six L3 tasks. As Table 3 shows, Opus 5 made no disallowed fetches, which lowered its score by 12.4 points on average, whereas the score of DeepSeek V4.1 Flash remained almost unchanged. Opus 5 therefore behaved correctly, and the score change can serve as a measure of whether an agent respects the site's rules.

Table 3: Effect of robots.txt on six L3 tasks.

SystemDisallowed fetchesScore change
Claude Opus 50 in 6 runs−12.4
DeepSeek V4.1 FlashUnchanged+0.4

Revision. An Ethics Statement has been added after the conclusion, and we report the robots.txt experiment in the new Appendix J.

Q1

What is the most important capability tested, and why can prior benchmarks that cover a subset of the dimensions not test it?

R4: Thank you for this constructive question. We have reframed our contribution as measuring whether an agent can complete an exhaustive collection without being told the global target count. The combination supports target-count interventions and record- and field-level omission diagnosis within the same environment. Please refer to our response to W1 for details.

Revision. The changes are the same as for W1. We revised the abstract and the introduction, and Table 2 of the paper and the new Section 4.4 have been added.

Comments on the paper

1 comment

Dear Reviewers,

We sincerely thank all of you for the careful reading and the helpful suggestions. For your convenience, this comment gives an overview of our main responses, and the details are provided in the individual replies. A revised manuscript that reflects these changes has also been uploaded.

  1. Framing, positioning and root cause. We reframe the contribution as testing whether an agent can complete an exhaustive collection without being told the global target count. Our new experiments show that neither drift nor persistence explains the losses, since field accuracy stays flat along the listing and doubling the time limit gives no gain. Instead, the losses come from a wrong belief that the task is finished, as 46 of the 57 runs that missed records report the set as complete. A disclosure test on the 20 L2 tasks with eight systems in four harnesses confirms this. The count alone is sufficient, with every system submitting the complete set on at least 15 tasks, whereas an instruction to check and the location of the hidden records add little. We therefore present the perturbations and server logs as supporting design choices, and compare the completion signal and omission scoring with earlier benchmarks, including WebChoreArena Massive Memory and WideSearch. (Reviewer A5nj W1, Q1; Reviewer ZNx1 W4, Q2; Reviewer ujN1 W1)

  2. Run-to-run variance and strict success rate. We reran three systems three times on all tasks and report mean ± std. The systems are stable at the lower levels and remain separable under the variance at the harder ones. We also report record recall, field accuracy, precision and strict success rate on all 85 tasks, where recall and field accuracy, not precision, are the main bottlenecks. In addition, we clarify that the paper reports one randomly selected run per task. (Reviewer ZNx1 W2, Q3; Reviewer ujN1 W4)

  3. Difficulty, horizon and scope. Additional experiments show that the lower levels still separate weaker models, and harder levels follow from the same specification by scaling the parameters that hide the target. We narrow the long-by-construction claim, report length in LLM turns as the main measure, and add a comparison of steps per task across benchmarks. Moreover, Access is now reported as a gate, and we state that the focus on collection is deliberate. (Reviewer ZNx1 W1, W3, Q1; Reviewer ujN1 W2)

  4. Task validity and human reference. The tasks are correct by construction, because the generator writes both the site and the ground truth. For solvability, a scripted reference crawl retrieves the full results on all L1–L3 tasks, and GPT-6 Astra with computer use and an unlimited budget retrieves 97.4% at L4 and 75% at L5. Furthermore, Opus 5 improves substantially with human interaction at every level, while large room for improvement remains at the hardest levels. (Reviewer ujN1 W3)

  5. Ethics and writing. We added an Ethics Statement, which notes that all sites are synthetic and self-hosted and that the gates model steps an authorized user also takes. A robots.txt experiment further shows that the benchmark can test whether an agent respects such controls. We also rewrote the stage-wise analysis so that each paragraph states one takeaway, simplified the main-text figures, and moved the full breakdowns to the appendix. (Reviewer A5nj W2, W3)

We would greatly appreciate it if you could take a look at our responses and the revised manuscript, and we are happy to address any remaining concerns. Thank you again for your time and consideration.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.