- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces TCFDNav, a benchmark and a three stage reference framework designed to evaluate how well AI systems can interpret and follow human-readable disclosure indexes in corporate annual reports. Instead of relying on traditional similarity-based retrieval, this work focuses on index-guided navigation, which follows explicit topic-to-location mappings declared by the company itself.
Critical strengths
-
Comprehensive evaluation of the navigation pipeline with informative diagnostics: The TCFDNav benchmark is meticulously constructed from 100 UK FTSE 100 annual reports and 17 external PDFs. It includes layered, source-checked annotations covering the entire navigation chain (index discovery, mapping recovery, and target resolution), including 30 NO_INDEX reports to test rejection capabilities.
-
Innovative "Sequential VLM" Calibration: A standout technical contribution is the Stage 3 "Sequential VLM" approach for page-number calibration. By checking neighboring pages to verify pagination rules (and reading a third page if there is a disagreement), it achieved 100% accuracy in recovering correct local page-mapping rules across 83 target PDFs, significantly outperforming single-page VLMs or text-layer parsers.
Critical weaknesses
- Conditional, Not Fully Autonomous Evaluation: The evaluation relies on supplied "Gold" or auxiliary inputs for certain stages (e.g., target registry, document aliases, and pagination probe positions). The authors acknowledge in the conclusion that this "conditional evaluation does not establish autonomous target-file selection," meaning the system is not yet a fully end-to-end autonomous agent in real-world, messy conditions.
- Narrow Task Scope (Navigation vs. Verification): The benchmark strictly evaluates whether a system can follow the index to the declared page. It does not evaluate whether the content on that page is actually relevant, accurate, complete, or compliant with TCFD standards. If a company’s index is misleading or points to an irrelevant section, the system will still be rewarded for navigating there correctly.
Questions for authors
-
How does the proposed framework compare with adapting existing disclosure-index extraction approaches, particularly ESG-CID and Gupta et al. (2025), to the TCFDNav task? Is there evidence that the three-stage framework provides measurable advantages over a simpler pipeline combining existing table/index extraction methods with deterministic reference resolution?
-
Can the authors provide the archived model predictions and exact scoring procedure that reproduce the 47/70 navigation-map exact match and 93.01% physical-address F1? The available evaluator scores 4,881 heterogeneous relations in its conditional-address diagnostic, rather than the paper's 3,407 physical PDF addresses.
5 comments
W2. Navigation vs verification
W2. Navigation vs verification
Navigation. Our focus on following the index to the declared pages comes from practice: when we sought for TCFD-related information manually, these indexes and tables appeared all the time. This raised a simple question: why not use them directly? That question motivates the whole study. When we checked the content of the declared pages, we found only one company whose index declared wrong pages, and only for 2 page relations, a tiny mistake. We therefore assume in this work that the indexes declare correct pages.
Verification. Our framework stops at locating the TCFD-related pages because classifying their content is already well studied: prior work listed below such as ClimateBERT and fine-tuned LLMs labels paragraphs with TCFD pillars. Combined with these classifiers, our framework directly supports checking whether the content on a declared page is relevant, accurate, complete, or compliant with TCFD standards.
Recent examples include:
- What Transfers Under Source Shift? Definitions, Examples, and Fine-Tuning for Climate Disclosure Classification (Li et al., 2026, EMNLP)
- How Cheap Talk in Climate Disclosures Relates to Climate Initiatives, Corporate Emissions, and Reputation Risk (Bingler et al., 2024, Journal of Banking & Finance)
Q1. Comparisons
Q1. Comparisons
Comparison with ESG-CID, Gupta et al. (2025) and a simpler pipeline. Neither work releases code, so we re-implemented both and tested them on TCFDNav. For Gupta et al., we follow their table extraction (Table Transformer + an LLM) on the Gold index pages and add our deterministic page resolution. For ESG-CID, we train its retrievers on its public GRI data and apply them to our 70 reports. Both fall far behind our framework.
Stage 2: Gupta-style pipeline (Gold index pages, 70 reports)
| Method | Mapping F1 | Navigation F1 | Address F1 | Reports fully correct |
|---|---|---|---|---|
| Gupta-style + Llama-3.1-8B | 64.3 | 68.2 | 64.0 | 13/70 |
| Gupta-style + Qwen3.5-27B | 75.5 | 78.1 | 75.5 | 19/70 |
| TCFDNav Stage 2 | 94.1 | 94.6 | 94.1 | 49/70 |
| 60 reports the 8B pipeline completed | Mapping F1 | Reports fully correct |
|---|---|---|
| Gupta-style + Llama-3.1-8B | 70.1 | 13/60 |
| Gupta-style + Qwen3.5-27B | 79.1 | 19/60 |
| TCFDNav Stage 2 | 94.9 | 43/60 |
Page localisation: ESG-CID retrievers (70 reports, 11 TCFD recommendations)
| Method | Rec. F1 | Pillar F1 | Recall@10 | MRR@50 |
|---|---|---|---|---|
| bge-base (no fine-tuning) | 26.7 | 32.3 | 57.7 | 47.7 |
| ESG-CID FT (GRI-trained, 5 seeds) | 28.3 ± 1.5 | 35.2 ± 1.3 | 56.5 ± 1.8 | 48.7 ± 3.4 |
| ESG-CID FTLLM (GRI-trained, 5 seeds) | 30.9 ± 0.6 | 37.4 ± 0.8 | 59.9 ± 1.2 | 56.3 ± 0.6 |
| TCFDNav (index-guided) | 93.9 | 97.5 | – | – |
We see two reasons. First, table extraction does not fit index pages well: Table Transformer finds no table in 5 reports and misaligns rows in others, and a larger LLM closes only part of the gap. Second, ESG-CID needs training data, and its GRI-trained retrievers transfer poorly to TCFD; suitable TCFD training data is hard to obtain.
TCFDNav is training-free, so no training set is needed. More importantly, it mirrors how a person reads an index, with a VLM in place of the reader: find the index (Stage 1), read which topic points to which pages (Stage 2), and turn to those pages in the right document (Stage 3). This is why we consider the three-stage design well grounded. It also works better than traditional non-VLM approaches, such as keyword rules, similarity retrieval and table extraction.
W1. From conditional to end-to-end evaluation
We thank the reviewer for the careful reading and constructive questions. Below we address each point
W1. From conditional to end-to-end evaluation
Stages 1 and 2 are fully autonomous. In Stage 3, the only supplied input was a target registry for the external documents identified by Stage 2, and even this registry was largely automatic: 19/20 of its files had been found by an agent. In the revised manuscript, Stage 3 is automated as well, so no manual step is involved and Stages 1–3 run fully end to end. A web agent added in Stage 3 searches for and downloads each external document named by Stage 2, and page calibration starts from the index page found by Stage 1 or the first page that Stage 2 cites. Even so, our framework reaches 93.16 address F1 with 49/70 exactly correct reports, versus 93.19 and 50/70 in the conditional setting.