acceptodds
Under review as a conference paper at ICLR 2027

CogExtract: Test-Time Search with Execution-Grounded Verification for Web Information Extraction on Dynamic Pages

Abstract

Web information extraction systems must generate reusable rules that remain reliable across pages with locally varying structures. Existing approaches typically generate a single rule and reuse it without external verification, making errors difficult to detect and repair. We introduce CogExtract, a test-time search framework that couples three mechanisms: Diversity-Conditioned Generation constructs structurally distinct XPath candidates; Execution-Grounded Verification ranks them using deterministic execution signals from a seed page and a sibling verification page; and Failure-Conditioned Reflection uses the resulting failure evidence to guide targeted regeneration when no candidate passes verification. We evaluate CogExtract on LiveWeb-IE, covering 14 websites, 312 page groups, and 50,524 URLs, with 11 vision-language model backbones. CogExtract achieves the best F1 for every backbone and improves over VGS by 14.5 points on average, while increasing model calls by only 13%; GPT-5 reaches the highest overall F1 of 76.6. Mechanism-level analyses show that execution-grounded scoring outperforms LLM self-evaluation in candidate discrimination (AUC 0.840 vs. 0.640; best-selection P@1 96.2% vs. 77.9%). Within the audited a/c failure subset, structured execution-grounded diagnosis also achieves 91.0% accuracy, compared with 59.0% for LLM judgment. These results demonstrate that external execution evidence provides an effective basis for both rule selection and targeted repair on dynamic web pages.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
web information extraction, wrapper induction, vision-language models, test-time search, execution-grounded verification, failure-conditioned reflection, visual grounding, XPath generation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_qB9vOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CogExtract, a method for generating XPath extraction rules that keep working across pages of the same website. It generates several candidate rules and selects one by executing them on a seed page and an unlabeled verification page. When all candidates fail, an LLM diagnoses the failure type from the execution results and new candidates are generated. On LiveWeb-IE with 11 backbone models, CogExtract improves over the VGS baseline by 14.5 F1 on average with 13% more model calls.

Critical strengths
  • The code is publicly released, and the paper provides the prompt templates for all compared methods, which supports reproducibility.
  • The method is evaluated on 11 closed- and open-source backbones of different sizes with additional thinking variants.
Critical weaknesses
  1. Hallucinated references
  • Two cited papers could not be found. No paper titled "EVAPORATE: Code generation as a prompt learning problem" (Lockard et al., VLDB 2020) appears on arXiv, Crossref or OpenAlex. Its four listed authors and year match ZeroShotCeres [1], while EVAPORATE is a system from a different paper by Arora et al. [2]. "Revisiting information extraction from structured data with language models" (Aman Wadhwa et al., EMNLP 2023) appears neither in the EMNLP 2023 proceedings nor on arXiv, Crossref or OpenAlex.
  • Four entries list authors who did not write the cited paper. LEVER (arXiv:2302.08468) lists five authors, none of whom is among its seven actual authors led by Ansong Ni. Self-Debug (arXiv:2304.05128) lists "Xiang Chen" and "Mingda Lin" in place of Xinyun Chen and Maxwell Lin. Khandelwal et al. (arXiv:2508.17959) lists "Karthik Murugesan", "Mark Campbell" and "Rayid Horesh" in place of Keerthiram Murugesan, Murray Campbell and Lior Horesh. WebLists (arXiv:2504.12682) gives its first author as "Ankit Bohra" instead of Arth Bohra.
  • These errors suggest that parts of the bibliography were generated without verification. All references should be checked against the original papers.
  1. Dataset statistics contradict the released data and the LiveWeb-IE paper
  • The paper states that the evaluation covers 14 websites, 312 page groups and 50,524 URLs (Abstract, Section 3, Appendix E.1, Table 7, Conclusion). For these 14 sites, the released dataset contains 41 page groups, 312 queries and 4,690 unique URLs. Thus 312 is the number of queries, and 50,524 is the number of pairs obtained by matching each query with every URL listed in its page group, which Appendix E.1 itself counts as extraction instances. Every row of Table 7 (Appendix J.1) has the same mix-up. For example, books.toscrape.com has 3 groups and 1,100 URLs, not 32 and 16,800.
  • The excluded website is named "dp.art" (Appendix E.1, Table 7) and labeled "Academic" (Table 7). No such site appears in the released data or in [3]. The excluded site is dp.la (DPLA), which [3] places in the Library domain. Its 30 queries and 2,008 query and URL pairs match the "dp.art" row of Table 7.
  • Table 8 reports 65, 66, 97 and 84 groups for the four task types. Without dp.la, the released data gives 61, 94, 86 and 71 queries. The values 65, 97 and 84 are the counts for all 15 sites including dp.la, where the type 2 count is 96, not 66.
  1. Limited novelty over AutoScraper
  • Layer 1 of CogExtract (DCG and EGV) generates several candidate XPaths, executes them on two pages of the same page group, and keeps the highest-scoring one that returns results on both. This closely resembles the synthesis module of AutoScraper [4], which generates an XPath action sequence on each of several seed pages, executes all of them on all seed pages, and has an LLM choose the one that extracts the target information across these pages. AutoScraper also revises failed XPaths using their execution results, although on a single page and without an explicit failure-mode classification that selects a repair strategy.
  • The paper does not cite AutoScraper, even though LiveWeb-IE [3] uses it as a baseline. The differences are not discussed, and CogExtract is never compared with AutoScraper.
  1. Evaluation on a single benchmark with a narrow set of baselines
  • All experiments use only LiveWeb-IE. The LiveWeb-IE paper [3] also evaluated VGS, whose front end CogExtract reuses, on SWDE [5] and Expanded SWDE [6], and AutoScraper [4] also used both. The released code includes scripts that run CogExtract, VGS, CoT and Reflexion on SWDE, but the paper reports no SWDE results and gives no reason.
  • Appendix E.5 mentions a human baseline as a reference upper bound, but no human-baseline score appears in any table or figure.

References

[1] ZeroShotCeres: Zero-Shot Relation Extraction from Semi-Structured Webpages, Lockard et al., ACL 2020

[2] Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes, Arora et al., PVLDB 2023

[3] LiveWeb-IE: A Benchmark for Online Web Information Extraction, Yang et al., ICLR 2026

[4] AutoScraper: A Progressive Understanding Web Agent for Web Scraper Generation, Huang et al., EMNLP 2024

[5] From One Tree to a Forest: A Unified Solution for Structured Web Data Extraction, Hao et al., SIGIR 2011

[6] OpenCeres: When Open Information Extraction Meets the Semi-Structured Web, Lockard et al., NAACL 2019

Questions for authors

Please check the above weaknesses.

Reviewer_HmvnOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper tackles the web information extraction task, where extraction rules are generated to turn unstructured web content to structured data. Existing methods are either unable to detect cross-page failures or utilize the detection for subsequent candidate search. The paper proposes CogExtract, which generates diverse XPath candidates, verifies them by executing on a seed page and a verification page, and regenerate based on system diagnosis when no candidate passes the verification. Evaluation on LiveWeb-IE shows that CogExtract improves F1 score by 14.5 over VGS on average across 11 model backbones.

Critical strengths
  • Substantial and consistent performance gains. CogExtract is evaluated across 11 models, and it shows a consistent, significant gain over VGS baseline.
Critical weaknesses
  • The methodology is not clearly described (verification step and FCR repair step). In Section 2.4 and Section 2.5, FCR is bypassed if a candidate has nonempty extraction on both pages. However, in the case study in Appendix H, it seems that FCR is triggered when "XPath extracts an inconsistent number of elements" (nonempty). The ranking scorer details are not fully specified (Sec 2.4).
  • Correctness is not guranteed. In methodology, it is described that eligibility requires only nonempty outputs on both verification pages. If one candidate qualifies, the FCR repair step is not triggered. It is possible that every candidate extracts the incorrect content (wrong extraction). The verification step has no access to the ground-truth, and thus the ranking may not gurantee correctness.
  • Limited novelty. The components such as diverse candidate generation, selection filtering, and repair based on execution feedback are well-established ideas. The paper needs more evidence to confirm the value of the combination in web extraction task.
  • Evaluation is not rigorous enough. There is a single benchmark with 14 websites. The seed and sibling pages are selected in advance, and there is no reported variance / uncertainty for alternative selection of the two pages. There is no repeated sampling analysis, which limits reproducibility.
Questions for authors
  • When are FCR repair triggered? Is it triggered only when zero eligible candidates (non-empty extraction results on both pages)? Can wrong extraction trigger the repair (e.g. both pages are extracted, but wrong)?
  • How sensitive are the method and results to the seed page and verification page selection?
  • How often does the FCR repair yields no eligible candidate? What is the fallback strategy?
  • What does correct-repair rate refer to in Section 3.5? It is claimed that 94.2% generate new candidate that passes EGV, but only 19.5% represent correct repair. Is a correct repair means it matches the ground truth? Is correctness measured over the whole set?
Reviewer_qjJuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper identifies cross-page failure as a key limitation in web information extraction. Building on this, the paper introduces CogExtract, a framework that leverages cross-page execution feedback to improve wrapper generation. The paper reports experimental results on LiveWeb-IE, showing improved performance compared to baselines.

Critical strengths
  • The paper is well-motivated.
  • The proposed framework outperforms baselines while remaining computationally efficient.
Critical weaknesses
  • The core conceptual contribution of the paper (i.e., identifying cross-page execution feedback for reusable rule generation) is already established in prior script-based extraction work [1][2], with a similar component-wise ablation in [2].

[1] SCRIBES: Web-Scale Script-Based Semi-Structured Data Extraction with Reinforcement Learning

[2] Knowledge Extraction on Semi-Structured Content: Does It Remain Relevant for Question Answering in the Era of LLMs

Questions for authors
  • Compared to prior script-based extraction work, which specific element of CogExtract is conceptually new?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.