- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents a benchmark called LongWebBench, a large-scale web data collection consisting of multiple pages. It comprises 85 automatically generated tasks. To avoid contamination from prior knowledge and to conduct evaluations in a controllable environment, the authors host the website themselves and perform the tasks within it. Furthermore, by conducting detailed evaluations at each stage, rather than relying solely on the final score, they make it easier to analyze the causes of the agent’s failures. The experimental results show that even the latest agents still have room for improvement.
Critical strengths
-
S1. I believe that developing tasks that utilize long-term memory and short-term memory is a timely and important area of research.
-
S2. Few benchmarks conduct stage-wise evaluation, and I consider this aspect to be beneficial.
-
S3. The fact that they built their own website can be highly evaluated. While questions remain regarding how closely it resembles real-world websites, this approach clearly eliminates issues such as contamination from pre-training knowledge and makes the environment controllable.
Critical weaknesses
-
W1. Similarity to existing benchmarks: Tasks such as “collecting a complete dataset, such as every product from an online store” are similar to the “Massive Memory” tasks in WebChoreArena.
-
W2. Narrow scope: This benchmark is specialized solely in web data collection and has only 85 tasks. Therefore, it seems to lack comprehensiveness and variety in the tasks.
-
W3. The 85 tasks in LongWebBench are automatically generated by a program. It would be necessary to analyze whether these tasks were all manually checked and, if solved manually, what the accuracy rate is or whether there are any tendencies to make mistakes. For example, adding a “Human Baseline” to Table 2 could be beneficial.
-
W4. It is difficult to interpret the results based solely on the weighted Overall Score. The authors should report the “Strict Task Success Rate.” This would provide readers with results that are easier to interpret, showing the current percentage of tasks that can actually be executed.
Questions for authors
For more details, please refer to “Critical Weaknesses.” In particular, I would like the authors to clarify the similarity to existing benchmarks (W1) and the validity of a narrow scope (W2), and task validity (W3).
5 comments
Response to Reviewer ujN1 (2/3)
R1 (continued): 3. What the three properties enable together. LongWebBench combines controlled target sets with delivery-based evaluation to study completeness and locate omissions in the same environment.
- Long-horizon task with a full record set defines the collection objective. The global target count is undisclosed, and from L2 onward, completing the default listing does not establish global completeness.
- Controlled environment enables the disclosure intervention. Because we generate and host the sites, the true record set is known exactly, so we can disclose or withhold its count while holding the site content fixed.
- Delivery-based evaluation locates omissions relative to recorded delivery. The server logs distinguish pre-delivery coverage gaps from post-delivery output losses. Transcripts and prompt interventions then help investigate why collection stops short.
Table 2 compares the completion criteria and omission scoring of earlier benchmarks with this combined measurement.
Table 2: Completion signal and omission scoring in earlier benchmarks.
| Benchmark | Done when | Omission seen as |
|---|---|---|
| WebArena, WorkArena++ | The target state is reached | Not applicable, single end state |
| WebChoreArena Massive Memory | The stated slice is collected, such as top 50 | A wrong final answer |
| Open CaptchaWorld | One CAPTCHA is solved | Not applicable, pass or fail |
| Fayolle et al. (2026) | One form is submitted behind bot controls | Not applicable, submission logged |
| ContractBench | The stateful API calls succeed | Not applicable, server-log reward |
| WAREX | The original task is done under injected faults | The original benchmark's score |
| WideSearch | Open-ended collection | Missing rows in the final output, on the live web |
| LongWebBench | Global target count undisclosed | Missing records and fields, before or after recorded delivery |
The first six rows evaluate a stated collection slice, a target state, or a gate or contract. WideSearch evaluates open-ended collection against curated gold tables on the live web. Our generated environment fixes the target set and links record- and field-level delivery to the submission, enabling target-count interventions and diagnosis of omissions before or after delivery.
Revision. We revised the abstract and the introduction to state this capability. Table 2 above has been added to Section 2, where our tasks are also contrasted with Massive Memory, and the experiment is reported in the new Section 4.4 and Appendix H.
Response to Reviewer ujN1 (3/3)
W2. Narrow scope
The benchmark covers only web data collection and has only 85 tasks, so it seems to lack comprehensiveness and variety.
R2: Thank you for your insightful comment. The focus on collection is deliberate, because attributing each missing record requires a task family whose full answer key the environment holds. Within this family, the 85 tasks cover 20 archetypes, five levels and several site types. Moreover, since tasks are generated from seeds, the number of tasks can grow beyond 85. However, each task is already long-horizon and costly. For example, running all 85 tasks with GPT-5.6 Sol costs over $400 and takes about 24 hours. We therefore keep the released set at 85 tasks. We also agree that our conclusions should not be generalized beyond collection tasks, and we plan to expand the diversity of tasks in future updates.
Revision. This scope and our plan to extend it are now stated in the limitations of Appendix K.
W3. Task validity / human baseline
The 85 tasks are generated by a program. Were they all manually checked, and what accuracy do humans reach on them? A human baseline in the leaderboard table would help.
R3: Thank you for your detailed comment. We answer the two parts of this concern in turn.
Are the tasks correct and solvable? Because the generator writes both the site and the ground truth, the tasks are correct by construction. To check solvability, we ran a scripted reference crawl that completes each task through the same interfaces as the agents. The scripts retrieve the full results on all L1–L3 tasks. For L4 and L5, a simple script is not enough to validate solvability. We therefore gave GPT-6 Astra computer-use capability, a more concrete prompt about the data range and an unlimited budget. It retrieves 97.4% of the results at L4 and 75% at L5, which shows that most of the tasks are solvable.
What do humans achieve? Because human experts are costly, we do not benchmark human performance directly. Instead, a human expert solved the tasks jointly with Opus 5 in Claude Code. Table 3 reports the results.
Table 3: Overall of Claude Opus 5 with human interaction.
| Method | L1 | L2 | L3 | L4 | L5 |
|---|---|---|---|---|---|
| Claude Opus 5 | 77.2 | 78.0 | 70.2 | 66.5 | 49.5 |
| Claude Opus 5 + human interaction | 96.1 | 89.6 | 88.7 | 80.4 | 66.3 |
As Table 3 shows, Opus 5 improves substantially with human guidance. Our analysis of the trajectories attributes the gain to a more efficient collection strategy and a more thorough search for hidden records. This result is consistent with our response to W1. Meanwhile, even a frontier model with human interaction leaves large room for improvement at the hardest levels.
Revision. The solvability check is now reported in Appendix B.5, and we added Table 3 above to Appendix F.7.
W4. Strict task success rate
The weighted Overall score alone is hard to interpret. Please report the strict task success rate.
R4: Thank you for your constructive comment. We agree that metrics that directly reflect the final results are beneficial. We therefore report record recall, field accuracy, precision and strict success rate in Table 4. Here, a task counts as a strict success only when its recall, accuracy and precision all exceed 0.999.
Table 4: Submission metrics on all 85 tasks.
| Model | Record recall | Field accuracy | Precision | Strict success |
|---|---|---|---|---|
| Claude Opus 5 | 84.2 | 63.1 | 95.0 | 1.2 (1/85) |
| DeepSeek V4.1 Flash | 77.9 | 57.6 | 84.3 | 11.8 (10/85) |
| GLM-5.3 | 76.0 | 51.0 | 88.4 | 2.4 (2/85) |
| GPT-5.6 Luna | 73.2 | 47.2 | 88.9 | 0 |
| GPT-5.6 Sol | 73.1 | 51.9 | 96.5 | 0 |
| GPT-5.6 Terra | 65.8 | 43.7 | 86.3 | 0 |
| Sonnet 4.6, low effort | 53.0 | 31.0 | 68.2 | 0 |
| Qwen3.8-Flash | 48.1 | 26.9 | 57.8 | 0 |
| GLM-5.3-Flash | 28.3 | 16.9 | 34.3 | 0 |
As Table 4 shows, the models rarely achieve strict success, and record recall and field accuracy, not precision, are the main bottlenecks.
Revision. The strict success rate is now cited in Section 4.2, and we added Table 4 above to Appendix F.5.
Questions
Please clarify the similarity to existing benchmarks (W1), the narrow scope (W2) and the task validity (W3).
R5: Thank you for your questions. Please refer to our responses R1–R3 above.
Response to Reviewer ujN1 (1/3)
We sincerely thank you for your thoughtful review and for the time you devoted to our paper. We are encouraged that you view the topic as timely, the stage-wise evaluation as beneficial, and the self-built websites as free of contamination and controllable. Your questions on similarity, scope and task validity prompted new analyses that we believe have improved the paper, and we are grateful for them. We respond to each of your concerns below.
W1. Similarity to existing benchmarks
R1: Thank you for this comment. We agree that the task format resembles the Massive Memory tasks, and we have reframed our claim to make the difference explicit. In short, LongWebBench tests whether an agent can complete an exhaustive collection without being told the global target count. We develop this answer in three steps below.
1. The difference from Massive Memory. A Massive Memory task states the slice to collect, such as the top 50 items, so the agent knows when it is done, and the difficulty lies in retaining what it has read. In contrast, our tasks ask for every record, hide the size of the set and place part of it outside the default listing. The agent therefore has to judge by itself when the set is complete.
2. This judgment is where agents lose. Across 24 full L3 runs, field accuracy stays flat along the listing, from 0.593 in the first quarter to 0.589 in the last, so agents do not forget as the run grows longer. Moreover, doubling the time limit gives no gain in Overall. Instead, the losses come from a wrong belief that the task is finished. Among the 57 archived runs that missed records, 46 report the set as complete in their final message, and such runs stop after about 38% of their time. They stop early while the global target count remains undisclosed and completing the default listing need not establish global completeness.
To test whether this belief causes the losses, we supplied the missing knowledge in the prompt and observed whether the losses disappeared. We ran this test on the 20 L2 tasks with eight systems in four harnesses. Each system received the task in up to four forms, and Table 1 reports the outcome.
Table 1: Complete submissions out of 20 L2 tasks, with Overall in parentheses.
The results lead to three conclusions.
Taken together, the losses trace back to one missing piece of knowledge, namely the size of the target. The task requires record completeness and timely submission: an agent must determine whether it has collected the full target set and save a usable answer within the budget. Unlike Massive Memory, determining completeness here requires establishing the collection scope, beyond retaining what the agent has read.