- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CogExtract, a method for generating XPath extraction rules that keep working across pages of the same website. It generates several candidate rules and selects one by executing them on a seed page and an unlabeled verification page. When all candidates fail, an LLM diagnoses the failure type from the execution results and new candidates are generated. On LiveWeb-IE with 11 backbone models, CogExtract improves over the VGS baseline by 14.5 F1 on average with 13% more model calls.
Critical strengths
- The code is publicly released, and the paper provides the prompt templates for all compared methods, which supports reproducibility.
- The method is evaluated on 11 closed- and open-source backbones of different sizes with additional thinking variants.
Critical weaknesses
- Hallucinated references
- Two cited papers could not be found. No paper titled "EVAPORATE: Code generation as a prompt learning problem" (Lockard et al., VLDB 2020) appears on arXiv, Crossref or OpenAlex. Its four listed authors and year match ZeroShotCeres [1], while EVAPORATE is a system from a different paper by Arora et al. [2]. "Revisiting information extraction from structured data with language models" (Aman Wadhwa et al., EMNLP 2023) appears neither in the EMNLP 2023 proceedings nor on arXiv, Crossref or OpenAlex.
- Four entries list authors who did not write the cited paper. LEVER (arXiv:2302.08468) lists five authors, none of whom is among its seven actual authors led by Ansong Ni. Self-Debug (arXiv:2304.05128) lists "Xiang Chen" and "Mingda Lin" in place of Xinyun Chen and Maxwell Lin. Khandelwal et al. (arXiv:2508.17959) lists "Karthik Murugesan", "Mark Campbell" and "Rayid Horesh" in place of Keerthiram Murugesan, Murray Campbell and Lior Horesh. WebLists (arXiv:2504.12682) gives its first author as "Ankit Bohra" instead of Arth Bohra.
- These errors suggest that parts of the bibliography were generated without verification. All references should be checked against the original papers.
- Dataset statistics contradict the released data and the LiveWeb-IE paper
- The paper states that the evaluation covers 14 websites, 312 page groups and 50,524 URLs (Abstract, Section 3, Appendix E.1, Table 7, Conclusion). For these 14 sites, the released dataset contains 41 page groups, 312 queries and 4,690 unique URLs. Thus 312 is the number of queries, and 50,524 is the number of pairs obtained by matching each query with every URL listed in its page group, which Appendix E.1 itself counts as extraction instances. Every row of Table 7 (Appendix J.1) has the same mix-up. For example, books.toscrape.com has 3 groups and 1,100 URLs, not 32 and 16,800.
- The excluded website is named "dp.art" (Appendix E.1, Table 7) and labeled "Academic" (Table 7). No such site appears in the released data or in [3]. The excluded site is dp.la (DPLA), which [3] places in the Library domain. Its 30 queries and 2,008 query and URL pairs match the "dp.art" row of Table 7.
- Table 8 reports 65, 66, 97 and 84 groups for the four task types. Without dp.la, the released data gives 61, 94, 86 and 71 queries. The values 65, 97 and 84 are the counts for all 15 sites including dp.la, where the type 2 count is 96, not 66.
- Limited novelty over AutoScraper
- Layer 1 of CogExtract (DCG and EGV) generates several candidate XPaths, executes them on two pages of the same page group, and keeps the highest-scoring one that returns results on both. This closely resembles the synthesis module of AutoScraper [4], which generates an XPath action sequence on each of several seed pages, executes all of them on all seed pages, and has an LLM choose the one that extracts the target information across these pages. AutoScraper also revises failed XPaths using their execution results, although on a single page and without an explicit failure-mode classification that selects a repair strategy.
- The paper does not cite AutoScraper, even though LiveWeb-IE [3] uses it as a baseline. The differences are not discussed, and CogExtract is never compared with AutoScraper.
- Evaluation on a single benchmark with a narrow set of baselines
- All experiments use only LiveWeb-IE. The LiveWeb-IE paper [3] also evaluated VGS, whose front end CogExtract reuses, on SWDE [5] and Expanded SWDE [6], and AutoScraper [4] also used both. The released code includes scripts that run CogExtract, VGS, CoT and Reflexion on SWDE, but the paper reports no SWDE results and gives no reason.
- Appendix E.5 mentions a human baseline as a reference upper bound, but no human-baseline score appears in any table or figure.
References
[1] ZeroShotCeres: Zero-Shot Relation Extraction from Semi-Structured Webpages, Lockard et al., ACL 2020
[2] Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes, Arora et al., PVLDB 2023
[3] LiveWeb-IE: A Benchmark for Online Web Information Extraction, Yang et al., ICLR 2026
[4] AutoScraper: A Progressive Understanding Web Agent for Web Scraper Generation, Huang et al., EMNLP 2024
[5] From One Tree to a Forest: A Unified Solution for Structured Web Data Extraction, Hao et al., SIGIR 2011
[6] OpenCeres: When Open Information Extraction Meets the Semi-Structured Web, Lockard et al., NAACL 2019
Questions for authors
Please check the above weaknesses.