acceptodds
Under review as a conference paper at ICLR 2027

Wiki Live Challenge: Challenging Deep Research Agents with Expert-Level Wikipedia Articles

Abstract

Deep Research Agents (DRAs) show great potential in assisting humans with complex research tasks. Yet existing DRA evaluations are limited by scalability, as models iterate rapidly, benchmarks require continuous updates, while high-quality references and criteria depend on substantial human effort that cannot scale at low cost. Existing benchmarks have yet to achieve both reference quality and scalability simultaneously. To bridge this gap, we introduce Wiki Live Challenge (WLC), a scalable, live benchmark that leverages the newest Wikipedia Good Articles (GAs) as expert-level references. The Wikipedia community continuously produces expert-reviewed GAs, enabling WLC to acquire high-quality new tasks at minimal cost. Grounded in Wikipedia's GA criteria, we collect 100 recent GAs and propose Wiki Eval, a comprehensive evaluation framework comprising Wiki Writing, a fine-grained writing assessment with 39 criteria, and Wiki Fact, rigorous metrics for factual verifiability. Extensive experiments on various DRA systems demonstrate a significant gap between current DRAs and human expert-level Wikipedia articles.

Primary area:
datasets and benchmarks
Keywords:
Deep Research Agent, Large Language Model, Evaluation Benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jTrxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the Wiki Live Challenge (WLC), a continuously updated benchmark that uses Wikipedia Good Articles as a reference for evaluating Deep Research Agents. The author proposed the Wiki Eval evaluation framework, covering 39 writing evaluation criteria, as well as factual coverage and citation accuracy indicators. Experiments on multiple open source and proprietary agents, combined with human assessment consistency studies and cross-temporal evaluations, show that current agents still lag significantly behind Wikipedia reference entries in fact coverage.

Critical strengths

The paper leverages Wikipedia's existing manual review process to reduce the cost of building high-quality reference content while supporting the continuous updating of benchmarks. The authors evaluated writing quality, factual coverage, and citation reliability, and supported the evaluation through multi-model comparison and manual evaluation consistency studies.

Critical weaknesses

W1. Wikipedia article writing focuses primarily on retrieving, organizing, and presenting existing knowledge, while Deep Research also involves complex reasoning, multi-source evidence synthesis, and in-depth analysis. The paper does not fully prove that Wikipedia entry generation can represent general Deep Research capabilities. W2. The Wikipedia entry cannot cover all the facts. An agent may receive a lower Cov. Wiki score even if it provides accurate and valuable information because the information is not in the reference entry. In addition, the longer the report, the easier it is usually to cover more reference facts. Without controlling for output length, it is difficult to tell whether differences in scores result from research ability or report length.

Questions for authors

Q1. Can the authors provide evidence that WLC reflects general Deep Research capabilities, not just Wikipedia-style article generation capabilities?

Q2. How does the evaluation deal with correct facts generated by the agent that are not included in the Wikipedia reference entry? Can Cov. Wiki's correlation with actual research quality be verified through independent human evaluation? Does the output length of different agents significantly affect Cov. Wiki? Can the authors provide a correlation analysis and comparison for the same output budget?

4 comments
AuthorsOpenReview ↗

Response to Reviewer jTrx (1/2)

Thank you for the suggestions. Following them, we conducted further experiments and address each point below.

Weakness 1 and Question 1

Wikipedia article writing focuses primarily on retrieving, organizing, and presenting existing knowledge, while Deep Research also involves complex reasoning, multi-source evidence synthesis, and in-depth analysis. The paper does not fully prove that Wikipedia entry generation can represent general Deep Research capabilities.

Can the authors provide evidence that WLC reflects general Deep Research capabilities, not just Wikipedia-style article generation capabilities?

WLC asks an agent to research a topic on the open web, gather the knowledge that experts consider necessary from many sources, and synthesize it into a structured article. The reference articles are selected by their number of reference URLs and structural depth, as described in Section 3.1, and link to 75 external pages on average, so covering them requires finding and combining evidence from dozens of sources. We provide three pieces of evidence that this is a general deep research capability.

  1. Agents find knowledge but fail to organize it. For 6 backbones in the shared harness and 20 articles, we recorded the evidence each agent retrieved, the notes it wrote and its final report, and checked at each stage which reference facts were present. Many retrieved facts never reach the report, and the loss grows with how deep the fact sits in the expert article: of the facts an agent retrieved from the lead section, 74.2% appear in its report, compared with 43.6% for the first half of the article body and 36.5% for the second half. All six backbones show this decline. Because the reference here is the agent's own retrieved evidence, this finding does not depend on whether Wikipedia is complete. The retrieval and organization of knowledge measured here are stages of any deep research task.

  2. The same loss appears on a deep research benchmark. We compared WLC with DeepResearch Bench II, whose analytical tasks are scored with rubrics derived from expert reports. We ran four representative backbones of the same harness, GPT-5, Kimi-K2.5, Gemini-2.5-pro and DeepSeek-V3.2, on 20 of its tasks and applied the same stage tracing, with the benchmark's information-recall rubric items in place of Wikipedia facts. Our tracing agrees with the official DeepResearch Bench II information-recall scores, with a Spearman correlation of 0.90 across 68 reports. On DeepResearch Bench II, 56.8% of the key information that agents retrieved reaches their reports, and on WLC 38.1%. Losing retrieved knowledge before writing is therefore a general property of current agents, and WLC places stronger demands on this stage.

  3. WLC rankings agree with the deep research benchmark. On the eight systems evaluated by both benchmarks, WLC's Wiki Writing and the published DeepResearch Bench II results place the same three systems at the top and the same system at the bottom, and 21 of the 28 system pairs are ordered the same way. Within model families with a clear generational gap, WLC also ranks the newer model higher: GPT-5 above GPT-4.1 in both collection periods, and DeepSeek-V4 above DeepSeek-V3.2 in the second period. Because DeepResearch Bench II scores information recall, analysis and presentation together, this agreement indicates that the differences WLC measures extend to the analytical side of deep research.

In the revised paper, the trajectory analysis forms a new Section 5.2.4 with details in Appendix F, the comparison with DeepResearch Bench II is given in Appendix Q, and the introduction states the capability that WLC measures.

AuthorsOpenReview ↗

Response to Reviewer jTrx (2/2)

Weakness 2 and Question 2

The Wikipedia entry cannot cover all the facts. An agent may receive a lower Cov. Wiki score even if it provides accurate and valuable information because the information is not in the reference entry. In addition, the longer the report, the easier it is usually to cover more reference facts. Without controlling for output length, it is difficult to tell whether differences in scores result from research ability or report length.

How does the evaluation deal with correct facts generated by the agent that are not included in the Wikipedia reference entry? Can Cov. Wiki's correlation with actual research quality be verified through independent human evaluation? Does the output length of different agents significantly affect Cov. Wiki? Can the authors provide a correlation analysis and comparison for the same output budget?

Correct facts outside the reference. As stated in Appendix K, Cov. Wiki is a recall metric that does not penalize correct information absent from the Wikipedia fact list. Such information is assessed by Ref. Acc., which checks every generated statement against the sources it cites, and Appendix K reports that 53.5% of the checkable statements outside Wikipedia are supported by their sources. In a new importance evaluation by three judges from different model families, generated statements consistent with Wikipedia are rated as core knowledge about twice as often as correct statements outside Wikipedia. Cov. Wiki therefore targets the knowledge most essential to a topic, and Ref. Acc. accounts for the additional correct information that agents find. The importance evaluation is added to Section 5.1.2 and Appendix K of the revised paper.

Validation of Cov. Wiki. Cov. Wiki is an objective recall metric: for each fact in the expert reference, it checks whether the report states it. Its validity therefore rests on whether this check is accurate. Fact extraction covers 91.7% of the 3,333 citation-backed passages in the Wikipedia articles, as reported in Appendix B.1. The verifier agrees with the majority of three human annotators on 92.0% of 100 sampled verdicts, as reported in Appendix B.2. Applied to DeepResearch Bench II reports, the same matching procedure agrees with that benchmark's official information-recall scores, with a Spearman correlation of 0.90 across 68 reports. This last check is added to Appendix Q of the revised paper.

Output length. Appendix I of the original paper reports the correlation between output length and each score. For the comparison under the same output budget, we compared systems whose reports have similar lengths. Eleven systems write reports with a median length between 1,300 and 2,000 words, and their Cov. Wiki ranges from 14.1% to 31.7%; within this group, length and Cov. Wiki are almost uncorrelated, with a Spearman correlation of 0.13. The two systems with the longest reports, Qwen-3-max Deep Research and Gemini-2.5-pro Deep Research, have median lengths of about 3,900 and 4,300 words, and their Cov. Wiki is 22.2% and 30.8%. Differences in Cov. Wiki therefore reflect how much expert knowledge an agent gathers rather than how long it writes. This comparison is added to Appendix I of the revised paper.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal Response

Dear Reviewer jTrx,

We would like to kindly follow up on our response to your review. In our rebuttal, we compared WLC with a deep research benchmark, DeepResearch Bench II, analyzed how agents use the knowledge they retrieve, and examined the effect of output length on Cov. Wiki. As the discussion period is approaching its end, we would greatly appreciate it if you could take a look at our rebuttal and let us know whether our responses have addressed your concerns.

Thank you for your time and consideration.

Reviewer_jTrxOpenReview ↗

Response

Some of my concerns have been addressed, so I am willing to keep my original score. I still have reservations regarding the level of innovation, but if the paper is accepted, I have no objection.

Reviewer_j6oDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces WLC, a renewable benchmark for Deep Research Agents built from newly created Wikipedia Good Articles used as expert references. The first collection contains 100 articles across 15 domains, and a second collection demonstrates an update of the pipeline. Wiki Eval combines Wiki Writing, a comparison over 39 criteria derived from the Good Article guidelines, with Wiki Fact. The latter measures coverage of reference-article facts and citation support (Ref. Acc.), whose denominator is extracted Statement–URL pairs for which the cited page can be retrieved; it does not measure accuracy of all generated content. Gemini-2.5-Pro is selected as the writing judge, with 82.13% criterion-level agreement with humans, the highest among six candidates. The main table evaluates 16 system configurations: two open-source agents, seven backbones in a shared harness, and seven proprietary agents. The highest first-period factual coverage is 31.71%. The paper also reports cross-judge, graded-scoring, presentation-order, fact-verification, and residual Wikipedia-leakage checks.

Critical strengths
  1. A demonstrated route to replenishing expert-reference tasks. Using newly created Good Articles allows the benchmark to be refreshed while reusing an established editorial process. Two collection periods demonstrate the collection and evaluation pipeline. Article freshness can reduce exposure to the exact reference in pretraining when the relevant cutoff precedes article creation, although it does not guarantee that underlying facts or overlapping sources are unseen.
  2. Several checks support the evaluation at an aggregate level. In addition to the selected judge's 82.13% human agreement, rankings correlate across judges (Spearman 0.876, Kendall 0.750), binary versus graded scoring (Spearman 0.941), and swapped presentation order (Spearman 0.9455). Fact-extraction coverage is checked against 3,333 citation-backed spans. Appendix B.2 separately reports a three-model majority check on 600 verdicts and 92% agreement with a three-human majority on 100 verdicts; the model panel includes the verifier itself, so these are distinct forms of validation. The three-signal leakage audit also makes an important limitation visible.
Critical weaknesses
  1. Residual target-page exposure complicates some comparisons. Citation leakage ranges from 0.09% to 40.45%, and near-copy rates reach 7.87%. Removing statements that directly cite the target page from coverage calculations cannot remove information already acquired through that page or mirrors. Thus, the complete leaderboard cannot be interpreted unconditionally as research ability under equally enforced Wikipedia-free access. This concern is strongest for comparisons involving substantially different exposure signals; it does not invalidate every comparison or the benchmark's value as a task-output evaluation. A sensitivity analysis should establish which central comparisons survive restricting attention to lower-exposure systems or examples. Citation and overlap signals are imperfect proxies for actual access.
  2. Aggregate robustness does not establish the resolution of small score differences. The cross-judge, graded-scoring, and order-swap checks support broad ranking stability, but do not establish that a gap such as 40.92 versus 40.77 is distinguishable from evaluation variation. Article-level paired confidence intervals and repeated judging would clarify the precision of system differences. Human–human criterion agreement would be a useful additional reference, but neither that agreement nor the 82.13% judge–human figure substitutes for uncertainty estimates on aggregate comparisons.
Questions for authors
  1. Which main conclusions survive analyses restricted to lower-leakage systems and examples across several thresholds? A 5% citation threshold could be one sensitivity setting, but should not be treated as a validated contamination boundary. Please distinguish observable access evidence from inferred exposure.
  2. Can you provide article-level paired confidence intervals for nearby system scores, repeated-judge variability, and human–human criterion agreement to contextualize the reported judge–human agreement?
  3. How is the Wikipedia prohibition enforced for closed services, and what evidence is available about target-page or mirror access beyond citations and text overlap?
  4. How sensitive are rankings to changes in the presentation or detail of the Good Article instructions? This would help separate instruction sensitivity from stable differences in research and writing performance, both of which are relevant to the task.
  5. Appendix O, Table 19 already reports account/API and human collection costs per system. Can you add the complete per-period evaluation cost, including Wiki Writing judging, fact extraction and verification, unavailable-URL handling, local computation, and human validation? Please distinguish the $1/article collection payment from the $10/hour expert-validation work in Appendix P.
3 comments
AuthorsOpenReview ↗

Response to Reviewer j6oD (1/2)

Thank you for the suggestions. Following them, we conducted further experiments and address each question below.

Question 1

Which main conclusions survive analyses restricted to lower-leakage systems and examples across several thresholds? … Please distinguish observable access evidence from inferred exposure.

We restricted the evaluation at two levels. At the system level, we retained only systems below citation-leakage thresholds of 1%, 2%, 5% and 10%, each combined with thresholds on near-copied sentences. At the example level, we retained only the articles in which a system neither cites the target page nor produces near-copied sentences. The main conclusions hold in all settings: the large gap in factual coverage between agents and Wikipedia, the leading systems in Wiki Writing, and the ranking of backbones within the shared harness. At the example level, the Wiki Writing ranking has a Spearman correlation of 0.971 with Table 1, meaning that systems are ordered almost identically whether or not the articles with leaked content are counted. The rankings therefore reflect what agents find and write through their own research, not access to the Wikipedia reference.

Regarding observable access evidence and inferred exposure: for the agents run in our shared harness, the whole research process is recorded and the search layer filters Wikipedia, so access is directly observed and controlled. For closed services, only the outputs are visible, so exposure is measured through two observable signals, citations of the target page and near-copied text. The restricted analyses above exclude both signals, and the main conclusions still hold. These analyses are added to Appendix N of the revised paper.

Question 2

Can you provide article-level paired confidence intervals for nearby system scores, repeated-judge variability, and human–human criterion agreement to contextualize the reported judge–human agreement?

  • Article-level paired comparisons. We resampled the 100 articles 10,000 times and tested the score difference of every pair of systems on the same articles. Differences as small as 40.92 versus 40.77 are not significant, whereas larger gaps are robust: on Wiki Writing, 104 of the 120 system pairs differ significantly at the 0.05 level, and on Cov. Wiki, 95 of the 120 pairs do. Across the resamples, the top system in Wiki Writing stays first 93.1% of the time.
  • Judge variability. Re-running the judge with the order of the two articles swapped gives a ranking whose Spearman correlation with the original is 0.946, and replacing the judge with GPT-5 gives a correlation of 0.876.
  • Human–human agreement. We invited two annotators who, for 20 randomly sampled article pairs, independently labeled all 39 criteria, judging for each criterion whether the generated article wins or loses against the Wikipedia article. The two annotators agree on 75.51% of their judgments.

These results are added to Appendices G and P of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer j6oD (2/2)

Question 3

How is the Wikipedia prohibition enforced for closed services, and what evidence is available about target-page or mirror access beyond citations and text overlap?

For closed web services, the task prompt prohibits the use of Wikipedia, and when a service asks a follow-up question, the operator replies with a fixed instruction to exclude Wikipedia, as described in Appendix A.2.1. These services do not expose their browsing logs, so citations and text overlap are the evidence available for them. For the open-source harness, access can be restricted in the search layer itself: wikipedia.org and 13 known mirror sites are removed both from the search request and from the returned results, and WLC uses this filter by default. Because the harness is open, its full research trajectory is also recorded, so whether an agent accessed Wikipedia can be checked directly rather than inferred from its output. This white-box setting is an advantage of evaluating backbones within a shared harness. The filter is added to the evaluation protocol in Section 4.1 and Appendix A.2 of the revised paper.

Question 4

How sensitive are rankings to changes in the presentation or detail of the Good Article instructions? This would help separate instruction sensitivity from stable differences in research and writing performance …

We examined this with DeepResearch Bench II, whose tasks contain no Good Article instructions. On the eight systems evaluated by both benchmarks, the published DeepResearch Bench II results rank the same three systems at the top and the same system at the bottom as WLC's Wiki Writing, and 21 of the 28 system pairs are ordered the same way. Our agents also show the same loss of retrieved knowledge before writing on these tasks. The differences WLC measures therefore persist without the Good Article instructions, which indicates that they reflect stable differences in research and writing ability. The comparison is added to Appendix Q of the revised paper.

Question 5

Can you add the complete per-period evaluation cost, including Wiki Writing judging, fact extraction and verification, unavailable-URL handling, local computation, and human validation?

The evaluation cost has two parts.

  • Building a new collection period requires no annotation. The Good Articles are collected with a public query on Wikipedia's database replica and filtered and parsed automatically, and their reference facts are extracted once and reused for every system evaluated in that period.
  • Evaluating one system on 100 articles. Wiki Writing judging with Gemini-2.5-pro costs $13.2, or $0.132 per article. The main cost of Wiki Fact is the verification for Cov. Wiki: in the second collection period, it used about 20,000 calls to Gemini-2.5-Flash per system, with about 15 million input and 14 million output tokens, which is about $40 at list prices. Ref. Acc. verifies all statements that share a citation in a single call, which reduces its number of calls by about 90%. With 20 parallel requests, one system is evaluated in about a day.

The cost of generating the reports themselves is reported in Appendix O, and this evaluation cost table is added there in the revised paper.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal Response

Dear Reviewer j6oD,

We would like to kindly follow up on our response to your review. In our rebuttal, we added sensitivity analyses restricted to lower-leakage systems and articles, article-level paired significance tests for system differences, human–human agreement, and the complete evaluation cost. As the discussion period is approaching its end, we would greatly appreciate it if you could take a look at our rebuttal and let us know whether our responses have addressed your concerns.

Thank you for your time and consideration.

Reviewer_HjPBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Wiki Live Challenge, a benchmark for evaluating Deep Research Agents using Wikipedia Good Articles (GAs) recently created as reference documents. It also introduces an evaluation framework, Wiki Eval, comprising two components:

  1. Wiki Writing: Evaluates generated articles against Wikipedia references using 39 criteria covering writing quality, neutrality, and breadth of coverage.
  2. Wiki Fact: Evaluates factual coverage relative to Wikipedia and the extent to which generated statements are supported by their cited sources.

The authors evaluate a diverse set of proprietary and open-source research agents, including experiments using a shared LangChain research scaffold with different underlying language models. The paper also investigates evaluation reliability, model-judge agreement, information leakage, and performance differences across domains.

Critical strengths

The paper shows a scalable renewable reference construction. The paper uses recently created Wikipedia Good Articles as human-reviewed references, reducing the cost of manually constructing benchmark datasets. The authors also demonstrate the feasibility of updating the benchmark with a second collection period in the appendix. This is a practical approach to keeping benchmarks fresh as research agents improve.

The paper shows well-defined evaluation criteria and extensive experiments. it evaluates a diverse set of research agents, including comparisons using the same agent framework with different LLM backbones.

Critical weaknesses
  1. Limited validity and generalizability of Wikipedia-based factual coverage. My primary concern is whether factual coverage relative to a Wikipedia reference accurately measures the underlying research capabilities of an agent. The equation (3) measures how many reference facts appear in the generated article, but missing a fact from Wikipedia does not necessarily indicate a failure in information retrieval or research reasoning. The paper treats Wikipedia's fact list as the complete truth. By the paper's own table 14, 48% of generated statements fall outside WIKI's coverage and about half of those are supported by their sources. This suggests that Wikipedia references do not fully capture the factual information produced by research agents. While Wikipedia Good Articles provide high-quality references, they are not necessarily exhaustive representations of a topic. The benchmark’s generalizability beyond Wikipedia-style article generation therefore remains unclear.

  2. Potential contamination of Wiki Writing scores due to Wikipedia leakage. The paper acknowledges that some agents access the target Wikipedia articles, even though the task instructions explicitly prohibit this. The authors remove statements citing these pages from Cov. Wiki, but it is unclear whether the same is done for Wiki Writing. This raises my concerns about the writing scores. The authors report leakage statistics but do not show whether the writing rankings remain consistent after controlling for leakage.

Questions for authors
  1. What are the Wiki Writing scores after removing content that cites or copies the target page or its mirrors? Do rankings change?
2 comments
AuthorsOpenReview ↗

Response to Reviewer HjPB

Thank you for the suggestions. Following them, we conducted further experiments and address each point below.

Weakness 1

Missing a fact from Wikipedia does not necessarily indicate a failure in information retrieval or research reasoning. The paper treats Wikipedia's fact list as the complete truth. … The benchmark's generalizability beyond Wikipedia-style article generation therefore remains unclear.

WLC does not treat the Wikipedia fact list as the complete truth. As stated in Appendix K, Cov. Wiki is a recall metric that does not penalize correct information absent from the Wikipedia fact list. Such information is assessed by Ref. Acc., which checks every generated statement against the sources it cites, and Appendix K reports that 53.5% of the checkable statements outside Wikipedia are supported by their sources.

To examine what Cov. Wiki rewards, we compared the importance of generated statements consistent with Wikipedia with that of correct statements outside Wikipedia, sampling 781 such statements from all 16 systems. Three judges from different model families, GPT-5.5, Claude Opus 4.6 and DeepSeek-V4-Pro, rated each statement as core knowledge, a supporting detail, or peripheral information for understanding the topic. The judges saw only the topic and the statement, and we use the label given by at least two of the three judges. Statements consistent with Wikipedia are rated as core knowledge about twice as often as correct statements outside Wikipedia, and only 2.3% of them are rated as peripheral, compared with 14.6% outside Wikipedia. Cov. Wiki therefore measures whether an agent recovers the knowledge most essential to a topic, while Ref. Acc. accounts for the additional correct information that the agent finds.

Regarding generalizability beyond Wikipedia-style articles, we traced the research trajectories of 6 backbones in the shared harness on 20 articles, recording the evidence each agent retrieved, the notes it wrote and its final report. Many retrieved facts never reach the report: of the facts an agent retrieved from the lead section, 74.2% appear in its report, compared with 43.6% for the first half of the article body and 36.5% for the second half. Because the reference here is the agent's own retrieved evidence, this finding does not depend on whether Wikipedia is complete. We then compared WLC with DeepResearch Bench II, a deep research benchmark whose analytical tasks are scored with rubrics derived from expert reports, by running four of these backbones on 20 of its tasks with the same tracing. There, 56.8% of the key information that agents retrieved reaches their reports, compared with 38.1% on WLC. Losing retrieved knowledge before writing is therefore a general property of current agents, and WLC places stronger demands on this stage of deep research. In addition, on the eight systems evaluated by both benchmarks, WLC's Wiki Writing and the published DeepResearch Bench II results place the same three systems at the top and the same system at the bottom, and 21 of the 28 system pairs are ordered the same way. In the revised paper, the importance evaluation is added to Section 5.1.2 and Appendix K, the trajectory analysis forms a new Section 5.2.4 with details in Appendix F, and the comparison with DeepResearch Bench II is given in Appendix Q.

Weakness 2 and Question 1

Potential contamination of Wiki Writing scores due to Wikipedia leakage. … The authors remove statements citing these pages from Cov. Wiki, but it is unclear whether the same is done for Wiki Writing. … The authors report leakage statistics but do not show whether the writing rankings remain consistent after controlling for leakage.

What are the Wiki Writing scores after removing content that cites or copies the target page or its mirrors? Do rankings change?

Wiki Writing judges each article as a whole against the Wikipedia article, so instead of removing individual statements, we recomputed Wiki Writing for every system using only the articles in which the system neither cites the target Wikipedia page or its mirrors nor contains sentences nearly copied from Wikipedia. Because the retained articles differ across systems, we adjusted each score by the article's average score over all 16 systems. The ranking remains essentially unchanged: its Spearman correlation with the ranking in Table 1 is 0.971, the top four systems keep their positions, and Gemini-3-pro Deep Research remains first. The Wiki Writing ranking therefore reflects the agents' own research and writing rather than content taken from Wikipedia. These results are added to Appendix N of the revised paper.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal Response

Dear Reviewer HjPB,

We would like to kindly follow up on our response to your review. In our rebuttal, we recomputed Wiki Writing on articles without any Wikipedia leakage and examined what Cov. Wiki rewards through an importance evaluation of covered facts. As the discussion period is approaching its end, we would greatly appreciate it if you could take a look at our rebuttal and let us know whether our responses have addressed your concerns.

Thank you for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Responses

We thank all reviewers for their constructive suggestions. Following them, we conducted further experiments. We compared WLC with a deep research benchmark, DeepResearch Bench II; we traced how agents use the knowledge they retrieve during research; and we evaluated how essential the reference facts are. Three findings emerge.

  1. Agents lose expert knowledge after finding it. Many facts that agents retrieve during research never reach their final reports, and this loss grows for the deeper, section-level knowledge of a topic. The same loss appears on DeepResearch Bench II. WLC therefore measures how agents explore a topic and organize expert knowledge, a stage shared by deep research tasks in general.
  2. Fresh articles separate research from recall. On WLC, about 4% of the content credited in reports cannot be traced to the agent's own research, compared with about 11% on DeepResearch Bench II, whose tasks are built from older documents. Because WLC articles are created after the models' training cutoffs, WLC scores reflect what the agent found rather than what it remembered.
  3. Cov. Wiki rewards the most essential facts. Statements consistent with Wikipedia carry more essential knowledge than correct statements outside it: in our importance evaluation, the former are rated as core knowledge about twice as often as the latter.

The details are given below in our response to each reviewer. We have also updated the paper, with all revisions marked in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.