- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes the Wiki Live Challenge (WLC), a continuously updated benchmark that uses Wikipedia Good Articles as a reference for evaluating Deep Research Agents. The author proposed the Wiki Eval evaluation framework, covering 39 writing evaluation criteria, as well as factual coverage and citation accuracy indicators. Experiments on multiple open source and proprietary agents, combined with human assessment consistency studies and cross-temporal evaluations, show that current agents still lag significantly behind Wikipedia reference entries in fact coverage.
Critical strengths
The paper leverages Wikipedia's existing manual review process to reduce the cost of building high-quality reference content while supporting the continuous updating of benchmarks. The authors evaluated writing quality, factual coverage, and citation reliability, and supported the evaluation through multi-model comparison and manual evaluation consistency studies.
Critical weaknesses
W1. Wikipedia article writing focuses primarily on retrieving, organizing, and presenting existing knowledge, while Deep Research also involves complex reasoning, multi-source evidence synthesis, and in-depth analysis. The paper does not fully prove that Wikipedia entry generation can represent general Deep Research capabilities. W2. The Wikipedia entry cannot cover all the facts. An agent may receive a lower Cov. Wiki score even if it provides accurate and valuable information because the information is not in the reference entry. In addition, the longer the report, the easier it is usually to cover more reference facts. Without controlling for output length, it is difficult to tell whether differences in scores result from research ability or report length.
Questions for authors
Q1. Can the authors provide evidence that WLC reflects general Deep Research capabilities, not just Wikipedia-style article generation capabilities?
Q2. How does the evaluation deal with correct facts generated by the agent that are not included in the Wikipedia reference entry? Can Cov. Wiki's correlation with actual research quality be verified through independent human evaluation? Does the output length of different agents significantly affect Cov. Wiki? Can the authors provide a correlation analysis and comparison for the same output budget?
4 comments
Response to Reviewer jTrx (2/2)
Weakness 2 and Question 2
The Wikipedia entry cannot cover all the facts. An agent may receive a lower Cov. Wiki score even if it provides accurate and valuable information because the information is not in the reference entry. In addition, the longer the report, the easier it is usually to cover more reference facts. Without controlling for output length, it is difficult to tell whether differences in scores result from research ability or report length.
How does the evaluation deal with correct facts generated by the agent that are not included in the Wikipedia reference entry? Can Cov. Wiki's correlation with actual research quality be verified through independent human evaluation? Does the output length of different agents significantly affect Cov. Wiki? Can the authors provide a correlation analysis and comparison for the same output budget?
Correct facts outside the reference. As stated in Appendix K, Cov. Wiki is a recall metric that does not penalize correct information absent from the Wikipedia fact list. Such information is assessed by Ref. Acc., which checks every generated statement against the sources it cites, and Appendix K reports that 53.5% of the checkable statements outside Wikipedia are supported by their sources. In a new importance evaluation by three judges from different model families, generated statements consistent with Wikipedia are rated as core knowledge about twice as often as correct statements outside Wikipedia. Cov. Wiki therefore targets the knowledge most essential to a topic, and Ref. Acc. accounts for the additional correct information that agents find. The importance evaluation is added to Section 5.1.2 and Appendix K of the revised paper.
Validation of Cov. Wiki. Cov. Wiki is an objective recall metric: for each fact in the expert reference, it checks whether the report states it. Its validity therefore rests on whether this check is accurate. Fact extraction covers 91.7% of the 3,333 citation-backed passages in the Wikipedia articles, as reported in Appendix B.1. The verifier agrees with the majority of three human annotators on 92.0% of 100 sampled verdicts, as reported in Appendix B.2. Applied to DeepResearch Bench II reports, the same matching procedure agrees with that benchmark's official information-recall scores, with a Spearman correlation of 0.90 across 68 reports. This last check is added to Appendix Q of the revised paper.
Output length. Appendix I of the original paper reports the correlation between output length and each score. For the comparison under the same output budget, we compared systems whose reports have similar lengths. Eleven systems write reports with a median length between 1,300 and 2,000 words, and their Cov. Wiki ranges from 14.1% to 31.7%; within this group, length and Cov. Wiki are almost uncorrelated, with a Spearman correlation of 0.13. The two systems with the longest reports, Qwen-3-max Deep Research and Gemini-2.5-pro Deep Research, have median lengths of about 3,900 and 4,300 words, and their Cov. Wiki is 22.2% and 30.8%. Differences in Cov. Wiki therefore reflect how much expert knowledge an agent gathers rather than how long it writes. This comparison is added to Appendix I of the revised paper.
Follow-up on Our Rebuttal Response
Dear Reviewer jTrx,
We would like to kindly follow up on our response to your review. In our rebuttal, we compared WLC with a deep research benchmark, DeepResearch Bench II, analyzed how agents use the knowledge they retrieve, and examined the effect of output length on Cov. Wiki. As the discussion period is approaching its end, we would greatly appreciate it if you could take a look at our rebuttal and let us know whether our responses have addressed your concerns.
Thank you for your time and consideration.
Response
Some of my concerns have been addressed, so I am willing to keep my original score. I still have reservations regarding the level of innovation, but if the paper is accepted, I have no objection.
Response to Reviewer jTrx (1/2)
Thank you for the suggestions. Following them, we conducted further experiments and address each point below.
Weakness 1 and Question 1
WLC asks an agent to research a topic on the open web, gather the knowledge that experts consider necessary from many sources, and synthesize it into a structured article. The reference articles are selected by their number of reference URLs and structural depth, as described in Section 3.1, and link to 75 external pages on average, so covering them requires finding and combining evidence from dozens of sources. We provide three pieces of evidence that this is a general deep research capability.
Agents find knowledge but fail to organize it. For 6 backbones in the shared harness and 20 articles, we recorded the evidence each agent retrieved, the notes it wrote and its final report, and checked at each stage which reference facts were present. Many retrieved facts never reach the report, and the loss grows with how deep the fact sits in the expert article: of the facts an agent retrieved from the lead section, 74.2% appear in its report, compared with 43.6% for the first half of the article body and 36.5% for the second half. All six backbones show this decline. Because the reference here is the agent's own retrieved evidence, this finding does not depend on whether Wikipedia is complete. The retrieval and organization of knowledge measured here are stages of any deep research task.
The same loss appears on a deep research benchmark. We compared WLC with DeepResearch Bench II, whose analytical tasks are scored with rubrics derived from expert reports. We ran four representative backbones of the same harness, GPT-5, Kimi-K2.5, Gemini-2.5-pro and DeepSeek-V3.2, on 20 of its tasks and applied the same stage tracing, with the benchmark's information-recall rubric items in place of Wikipedia facts. Our tracing agrees with the official DeepResearch Bench II information-recall scores, with a Spearman correlation of 0.90 across 68 reports. On DeepResearch Bench II, 56.8% of the key information that agents retrieved reaches their reports, and on WLC 38.1%. Losing retrieved knowledge before writing is therefore a general property of current agents, and WLC places stronger demands on this stage.
WLC rankings agree with the deep research benchmark. On the eight systems evaluated by both benchmarks, WLC's Wiki Writing and the published DeepResearch Bench II results place the same three systems at the top and the same system at the bottom, and 21 of the 28 system pairs are ordered the same way. Within model families with a clear generational gap, WLC also ranks the newer model higher: GPT-5 above GPT-4.1 in both collection periods, and DeepSeek-V4 above DeepSeek-V3.2 in the second period. Because DeepResearch Bench II scores information recall, analysis and presentation together, this agreement indicates that the differences WLC measures extend to the analytical side of deep research.
In the revised paper, the trajectory analysis forms a new Section 5.2.4 with details in Appendix F, the comparison with DeepResearch Bench II is given in Appendix Q, and the introduction states the capability that WLC measures.