acceptodds
Under review as a conference paper at ICLR 2027

TCFDNav: Benchmarking Index-Guided Navigation from Annual Reports

Abstract

When a PDF provides a human-readable index, its mapping table should be used to locate information. Disclosure indexes based on the Task Force on Climate-related Financial Disclosures (TCFD) provide a representative example. However, existing approaches largely rely on similarity-based retrieval, while index-guided navigation remains underexplored. We therefore introduce TCFDNav, a benchmark for evaluating the recovery and execution of index-declared navigation relations in the TCFD setting. Rather than focusing on an isolated step, we address the complete navigation process, from an input annual report to the TCFD-related pages referenced by its index. Starting with 100 UK annual reports and extending to 17 external PDFs, we construct an annotated resource covering the full navigation chain. To support benchmark evaluation, we propose a three-stage reference framework that discovers complete indexes, recovers their declared relations, and resolves target addresses to produce traceable navigation maps. From a deployment perspective, Qwen3.5-27B performs competitively with leading proprietary models, showing the practical potential of open-weight models for this task.

Primary area:
datasets and benchmarks
Keywords:
Multimodal Document Understanding, Document Navigation, Benchmarking and Evaluation, Climate Disclosure

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_voKhOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TCFDNav, a benchmark and a three stage reference framework designed to evaluate how well AI systems can interpret and follow human-readable disclosure indexes in corporate annual reports. Instead of relying on traditional similarity-based retrieval, this work focuses on index-guided navigation, which follows explicit topic-to-location mappings declared by the company itself.

Critical strengths
  1. Comprehensive evaluation of the navigation pipeline with informative diagnostics: The TCFDNav benchmark is meticulously constructed from 100 UK FTSE 100 annual reports and 17 external PDFs. It includes layered, source-checked annotations covering the entire navigation chain (index discovery, mapping recovery, and target resolution), including 30 NO_INDEX reports to test rejection capabilities.

  2. Innovative "Sequential VLM" Calibration: A standout technical contribution is the Stage 3 "Sequential VLM" approach for page-number calibration. By checking neighboring pages to verify pagination rules (and reading a third page if there is a disagreement), it achieved 100% accuracy in recovering correct local page-mapping rules across 83 target PDFs, significantly outperforming single-page VLMs or text-layer parsers.

Critical weaknesses
  1. Conditional, Not Fully Autonomous Evaluation: The evaluation relies on supplied "Gold" or auxiliary inputs for certain stages (e.g., target registry, document aliases, and pagination probe positions). The authors acknowledge in the conclusion that this "conditional evaluation does not establish autonomous target-file selection," meaning the system is not yet a fully end-to-end autonomous agent in real-world, messy conditions.
  2. Narrow Task Scope (Navigation vs. Verification): The benchmark strictly evaluates whether a system can follow the index to the declared page. It does not evaluate whether the content on that page is actually relevant, accurate, complete, or compliant with TCFD standards. If a company’s index is misleading or points to an irrelevant section, the system will still be rewarded for navigating there correctly.
Questions for authors
  1. How does the proposed framework compare with adapting existing disclosure-index extraction approaches, particularly ESG-CID and Gupta et al. (2025), to the TCFDNav task? Is there evidence that the three-stage framework provides measurable advantages over a simpler pipeline combining existing table/index extraction methods with deterministic reference resolution?

  2. Can the authors provide the archived model predictions and exact scoring procedure that reproduce the 47/70 navigation-map exact match and 93.01% physical-address F1? The available evaluator scores 4,881 heterogeneous relations in its conditional-address diagnostic, rather than the paper's 3,407 physical PDF addresses.

5 comments
AuthorsOpenReview ↗

W1. From conditional to end-to-end evaluation

We thank the reviewer for the careful reading and constructive questions. Below we address each point

W1. From conditional to end-to-end evaluation

Stages 1 and 2 are fully autonomous. In Stage 3, the only supplied input was a target registry for the external documents identified by Stage 2, and even this registry was largely automatic: 19/20 of its files had been found by an agent. In the revised manuscript, Stage 3 is automated as well, so no manual step is involved and Stages 1–3 run fully end to end. A web agent added in Stage 3 searches for and downloads each external document named by Stage 2, and page calibration starts from the index page found by Stage 1 or the first page that Stage 2 cites. Even so, our framework reaches 93.16 address F1 with 49/70 exactly correct reports, versus 93.19 and 50/70 in the conditional setting.

Setting (70 reports)External documentsCalibration pagesNavigation F1 / exactAddress F1 / exact
Conditionalsupplied registrysupplied93.39 / 4693.19 / 50
Automatic documents onlyagent (16/20 obtained)supplied93.35 / 4593.16 / 49
Fully end-to-endagent (16/20 obtained)from Stages 1–293.35 / 4593.16 / 49
AuthorsOpenReview ↗

W2. Navigation vs verification

W2. Navigation vs verification

Navigation. Our focus on following the index to the declared pages comes from practice: when we sought for TCFD-related information manually, these indexes and tables appeared all the time. This raised a simple question: why not use them directly? That question motivates the whole study. When we checked the content of the declared pages, we found only one company whose index declared wrong pages, and only for 2 page relations, a tiny mistake. We therefore assume in this work that the indexes declare correct pages.

Verification. Our framework stops at locating the TCFD-related pages because classifying their content is already well studied: prior work listed below such as ClimateBERT and fine-tuned LLMs labels paragraphs with TCFD pillars. Combined with these classifiers, our framework directly supports checking whether the content on a declared page is relevant, accurate, complete, or compliant with TCFD standards.

Recent examples include:

  • What Transfers Under Source Shift? Definitions, Examples, and Fine-Tuning for Climate Disclosure Classification (Li et al., 2026, EMNLP)
  • How Cheap Talk in Climate Disclosures Relates to Climate Initiatives, Corporate Emissions, and Reputation Risk (Bingler et al., 2024, Journal of Banking & Finance)
AuthorsOpenReview ↗

Q1. Comparisons

Q1. Comparisons

Comparison with ESG-CID, Gupta et al. (2025) and a simpler pipeline. Neither work releases code, so we re-implemented both and tested them on TCFDNav. For Gupta et al., we follow their table extraction (Table Transformer + an LLM) on the Gold index pages and add our deterministic page resolution. For ESG-CID, we train its retrievers on its public GRI data and apply them to our 70 reports. Both fall far behind our framework.

Stage 2: Gupta-style pipeline (Gold index pages, 70 reports)

MethodMapping F1Navigation F1Address F1Reports fully correct
Gupta-style + Llama-3.1-8B64.368.264.013/70
Gupta-style + Qwen3.5-27B75.578.175.519/70
TCFDNav Stage 294.194.694.149/70
60 reports the 8B pipeline completedMapping F1Reports fully correct
Gupta-style + Llama-3.1-8B70.113/60
Gupta-style + Qwen3.5-27B79.119/60
TCFDNav Stage 294.943/60

Page localisation: ESG-CID retrievers (70 reports, 11 TCFD recommendations)

MethodRec. F1Pillar F1Recall@10MRR@50
bge-base (no fine-tuning)26.732.357.747.7
ESG-CID FT (GRI-trained, 5 seeds)28.3 ± 1.535.2 ± 1.356.5 ± 1.848.7 ± 3.4
ESG-CID FTLLM (GRI-trained, 5 seeds)30.9 ± 0.637.4 ± 0.859.9 ± 1.256.3 ± 0.6
TCFDNav (index-guided)93.997.5––

We see two reasons. First, table extraction does not fit index pages well: Table Transformer finds no table in 5 reports and misaligns rows in others, and a larger LLM closes only part of the gap. Second, ESG-CID needs training data, and its GRI-trained retrievers transfer poorly to TCFD; suitable TCFD training data is hard to obtain.

TCFDNav is training-free, so no training set is needed. More importantly, it mirrors how a person reads an index, with a VLM in place of the reader: find the index (Stage 1), read which topic points to which pages (Stage 2), and turn to those pages in the right document (Stage 3). This is why we consider the three-stage design well grounded. It also works better than traditional non-VLM approaches, such as keyword rules, similarity retrieval and table extraction.

Reviewer_nFGKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TCFDNav, a benchmark for evaluating whether models can locate and follow TCFD disclosure indexes in corporate annual reports. The task requires identifying the index (or determining its absence), extracting the declared mappings between disclosure topics and document locations, and resolving these references to actual PDF pages, rather than directly retrieving semantically relevant content. The authors construct a dataset of 100 UK annual reports and propose a three-stage framework covering index discovery, mapping extraction, and target resolution. Experiments with multiple multimodal LLMs achieve 93.01% address-level F1, but complete navigation-map accuracy on only 47 of 70 reports containing indexes.

Critical strengths
  1. The benchmark uses real corporate annual reports and includes challenging cases such as multi-page indexes, cross-document references, and reports without indexes.
  2. The dataset provides detailed annotations covering index locations, topic–document–page mappings, and physical PDF addresses, supporting evaluation at different stages.
  3. The stage-wise experiments and multiple evaluation metrics help identify performance bottlenecks, particularly the gap between relation-level accuracy and report-level completeness.
Critical weaknesses
  1. Insufficient motivation for index-guided navigation. The paper does not sufficiently justify why index-guided navigation is preferable to directly retrieving TCFD-related disclosures. Since TCFD follows a relatively standardized disclosure framework, existing retrieval methods could potentially locate relevant content without first identifying and parsing an index. This concern is particularly relevant because 30% of the reports lack a qualifying index, for which the proposed approach simply returns NO_INDEX. Although index-guided navigation is a well-defined task, the paper does not demonstrate its incremental value for practical disclosure retrieval.
  2. Limited novelty and insufficient justification of the new task. The proposed benchmark largely combines three established document-processing tasks: index localization, structured information extraction, and page-number alignment. In particular, Stage 2 closely resembles existing work on extracting disclosure-to-page mappings from ESG indexes. While integrating these components into an end-to-end navigation benchmark is useful, the paper does not clearly establish what fundamentally new challenges emerge from their combination. The authors should better distinguish TCFDNav from existing document information extraction and cross-reference resolution tasks, and demonstrate why the combined task warrants a dedicated benchmark.
  3. Insufficient baselines and error analysis for the main performance bottleneck. Stage 2 appears to be the primary performance bottleneck: even with gold index spans, Gemini achieves only 49/70 report-level exact matches despite a relation-level F1 of 94.14%. However, the paper provides limited analysis of the errors responsible for these failures. Given that the input indexes already contain explicit topic–document–page mappings, it remains unclear whether the difficulties arise from complex layouts, incorrect topic–page associations, missing entries, or implementation-related issues. Furthermore, the paper lacks comparisons with conventional PDF/table parsing combined with rule-based extraction. Such baselines and a detailed error breakdown are necessary to establish whether Stage 2 presents substantial challenges beyond existing structured information extraction methods.
Questions for authors

What are the main causes of mapping extraction failures, even with gold index spans? Could the authors provide an error breakdown and representative examples to clarify whether these failures arise from complex document structures or ordinary extraction errors?

6 comments
AuthorsOpenReview ↗

Index-guided navigation complements retrieval

We thank the reviewer for the careful reading and constructive comments, which helped us substantially strengthen the baselines and the error analysis.

W1 — Index-guided navigation complements retrieval

In practice, the two approaches work together. For the 30% of reports without a qualifying index, NO_INDEX signals that the system should fall back to existing retrieval methods; for reports that do have an index, TCFDNav uses it to navigate directly to the declared disclosures. To demonstrate its incremental value, we ran two standard retrievers, BM25 and bge, on the 70 indexed reports, using the official TCFD recommendation texts as queries. Compared with the Gold pages, the retrievers reach at most about 31% F1 at every level, against 92–98% for index-guided navigation, so an available index gives a much more precise route to the declared disclosure pages.

Micro F1 (%) between retrieved and Gold PDF pages on the 70 indexed reports. Retrieved chunks are mapped to their pages, and each retriever returns as many pages as the Gold set; pillar-level rankings fuse the pillar's recommendation queries by reciprocal-rank fusion.

LevelTCFDNavBM25bge
Recommendation93.922.726.8
Pillar, with recommendations97.529.831.4
Pillar only91.527.430.9
AuthorsOpenReview ↗

W2 — Novelty of the task and framework

W2 — Novelty of the task and framework

Problem formulation. TCFDNav goes beyond combining existing document-processing components by defining a task with its own objective, evaluation criteria, and practical need. Unlike prior work focusing on individual components, TCFDNav aims to recover and execute complete company-declared navigation relations, from annual reports to physical PDF addresses. It evaluates not only individual relations but also the completeness of the entire navigation map at the report level. This reflects practical disclosure review and auditing needs, where accurate, complete, and traceable references are essential.

Methodological contribution. We introduce a three-stage framework that places VLM-based document understanding at the core of index-guided navigation. Rather than relying solely on conventional text extraction and rule-based parsing, TCFDNav uses VLMs to identify and recover complete indexes across complex document structures and multi-page layouts. Comparisons with existing approaches support this design. TCFDNav reaches 91.5–97.5% page-level F1, against at most 31.4% for conventional retrieval (table under W1). In Stage 2, it improves mapping F1 from 75.5% to 94.1% over the strongest conventional pipeline and raises fully correct reports from 19/70 to 49/70 (table under W3).

New challenges. The central challenge is achieving complete and reliable navigation across interconnected stages, rather than high accuracy on isolated components. This is particularly important for regulatory and auditing applications, where even a few missing or incorrectly resolved references can undermine the evidence used for regulatory assessments and audit judgments. Despite achieving 94.1% mapping F1 with Gold index spans, only 49/70 reports have completely correct mappings. By defining this task and establishing a benchmark, TCFDNav provides a foundation for future research on index-guided navigation across different document structures, layouts, and domains.

AuthorsOpenReview ↗

W3 and Q1 — Comparison with table parsing and rule-based extraction

W3 and Q1 — Comparison with table parsing and rule-based extraction

We added three conventional pipelines in the revised manuscript. All of them receive the same input as our Stage 2 in the oracle setting, namely the Gold index pages, and are scored with the same evaluator.

The first uses only rules. PyMuPDF groups the words on each index page into rows. Lexical rules then read the pillar names, the official recommendation wording, the a)/b)/c) labels, the page numbers and the names of external documents. The second applies the same rules to the cell grid produced by Table Transformer. The third follows Gupta et al. (2025): the Table Transformer cells are given to Llama-3.1-8B, and in a second variant to Qwen3.5-27B. The rules were fixed before evaluation and checked only for bugs on 10 development reports.

Mapping F1 (%) and fully correct reports, with Gold index pages as input.

MethodParsingExtractionMapping F1 (%)Fully correct reports
RulesPyMuPDF rowsRules57.04/70
Table + rulesTable TransformerRules69.411/70
Gupta-style + Llama-3.1-8BTable TransformerLlama-3.1-8B64.313/70
Gupta-style + Qwen3.5-27BTable TransformerQwen3.5-27B75.519/70
TCFDNav Stage 2Page imageVLM94.149/70

The best conventional pipeline remains far below our Stage 2, at 75.5 against 94.1 F1 and 19/70 against 49/70 fully correct reports.

Q1: Error analysis. Appendix C.4 shows representative failures with page images. In all of them the page numbers are printed legibly; the difficulty lies in interpreting the index. Rentokil illustrates this most clearly. Its index has only seven lines and a simple layout, but apart from Governance every line carries a thematic label ("Risks and opportunities", "Plan and targets", "Metrics", …) rather than a TCFD pillar name. Parsing the lines is therefore not enough; each label must be mapped back to its pillar, which requires knowledge of the TCFD framework. The selected model extracts only the Governance line, whose label matches the pillar name, and omits the other six; only one of the seven models recovers the whole report.

Reviewer_s9XBOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how to enable AI to automatically understand and follow the disclosure indexes (defined by Task Force on Climaterelated Financial Disclosures, TCFD) in enterprise annual reports. By following the indexes, an AI agent may accurately locate the occurance of given requested information. Unlike the traditional similarity-based information retrieval, this manuscript focuses on recovering and executing the topic–document–page relationships in PDF annual reports. The authors propose TCFDNav Benchmark, a benchmark suite constructed from 100 annual reports of British companies. They construct this benchmark through three steps: they (1) locate TCFD indexes through long annual reports, (2) use multimodal LLM to extract the relationships between disclosure topics, referenced documents, and printed page numbers, and (3) resolve cross-document references and map printed page numbers to actual PDF locations. Experimental results show that existing LLMs can achieve strong performance on individual subtasks, with open-weight models such as Qwen3.5-27B demonstrating competitive performance against proprietary models.

Critical strengths
  • The paper works on a very clear and meaningful task. If an analyst simply asks a LLM to summarize an annual report, the result could mix up information from multiple sources with potential hallucination. This work, however, judges whether the output faithfully reflect the navigation information of the original annual report. As a benchmark, such a clearly-defined and easy-to-evaluate task is necessary.

  • The construction of the benchmark makes sense. Instead of using AI-generated labels, the authors construct a multi-layer label system to match the navigation process, which enables clear identification of failures.

Critical weaknesses

While I understand that this task is real and worth exploration, I have one serious concern: is this task really that hard? Given the fast iteration of recent LLMs, and the length of annual reports (say, a few hundred pages), I suspect this task can already be well processed by existing LLMs and agents. What if I directly instruct an agent to "find the TCFD index, extract all actual occurances according to the index, and output in JSON"? Admittedly, hallucination may be a key issue. However, today's frontier models are RL-trained exactly to resist hallucination, and they do give a satisfactory outcome. Furthermore, In Appendix Table 12, even a 9B model can achieve 100% accuracy. In Table 6, 98.8% -> 100% means only one sample among 83. I would suggest the authors try to prove the difficulty of this task by, for example, using a mainstream LLM agent to discover bad cases. Through analysis of these bad cases, the readers will understand why this task is difficult and what can be done to achieve better results.

The writing of this manuscript may also require improvement. For example, the first sentence in the abstract reads "When a PDF provides a human-readable index, its mapping table should be used to locate information". This is a bit confusing and less informative. " However, existing approaches largely rely on similarity-based retrieval", I would appreciate explaining the problem itself first so that the readers understand the topic better.

Questions for authors

As given in Critical Weaknesses, have you tried existing LLM agents powered by mainstream models? If so, the results could significantly improve this manuscript's soundness.

2 comments

Thanks for recognising this as a clear and meaningful task and for the thoughtful suggestions. Below we first clarify our motivation, then report mainstream LLM agent performance, and finally respond to the comments on Tables 6/12 and the writing.

1. Motivation: index-guided navigation, not task difficulty

In designing this task, our core contribution is to define a new navigation setting for index-guided documents, rather than to deliberately make the task harder.

For information retrieval, existing work almost by default relies on text-similarity retrieval (e.g., RAG). But when a long document already provides a clear index, is similarity-based retrieval still the most reasonable solution? Clearly, this is not how humans proceed.

When consulting a long corporate annual report, people do not blindly read the whole text. They follow a natural process: first find the index, then read the relations it declares, and finally navigate directly to the target pages. Our task design therefore follows this reading logic closely and decomposes it into three stages (Stage 1 index discovery → Stage 2 mapping recovery → Stage 3 target resolution and page calibration).

2. Can a mainstream LLM agent solve the task?

We added an experiment with two mainstream agents, Codex CLI (GPT-6.1 Sol, medium reasoning) and Claude Code (Claude Opus 4.8), on all 100 reports. Each agent received only the raw PDFs and one well-designed instruction, and was scored with the paper's evaluator and Gold.

StageMetricCodexClaude CodeOur framework
Stage 1: index discoveryComplete index span (70 with an index)67/7061/7065/70
Stage 1: index discoveryCorrect NO_INDEX rejection (30 without)26/3026/3030/30
Stage 2: mapping recoveryComplete-binding F1 (P / R)93.8 (92.9 / 94.7)93.7 (95.8 / 91.7)93.2 (96.4 / 90.1)
Stage 2: mapping recoveryMapping report EM (70 with an index)45/7048/7047/70
Stage 3: target resolutionCorrect local rules80/8072/7283/83
FinalNavigation-map report EM (100)71/10074/10077/100

Frontier agents already match our framework on aggregate accuracy, but their errors differ. Both agents built navigation maps for the same 4 of the 30 reports without an index, so this over-generation is systematic rather than specific to one model. Codex also over-reports relations (precision 92.9 vs. 96.4), and Claude Code misses more index spans (61/70). Our framework under-reports (recall 90.1 vs. 94.7): it missed the index span in 5 reports.

Report-level comparison on all 100 reports (a report counts as correct only if it is fully correct):

Correct byReports
All three61
Our framework and one agent9
Our framework only7 (4 of them without an index)
Agents only (one or both)8
None of the three15

The reports on which all three fail share two types of bad case, shown with page images in Appendix X of the revised PDF: (1) complex table structure, where page numbers must be assigned to the right row across multi-line cells and continuation pages (e.g. Standard Chartered); and (2) narrative indexes, where navigation links must be told apart from page numbers mentioned in passing (e.g. IMI).

3. Why TCFDNav instead of just unconstrained agents?

At comparable accuracy, our advantages are cost and auditability.

  1. Cost, covering both token consumption and the price of the model.
    • Tokens: We compare the model input tokens per indexed report across all three stages, as billed by each API, including image tokens. The agents use about 239K (Codex) and 347K (Claude Code) on average; our framework uses about 19K, or about 39K when image tokens are counted with the agent's tokenizer, a gap of roughly 6–18 times.
    • Model: The agents rely on the frontier models GPT-6.1 Sol and Claude Opus 4.8, whereas our framework achieves comparable results with the lightweight Gemini 3.8 Flash and the open-weight Qwen3.5 27B; the strongest model is not required.
  2. Auditability over black-box trajectories: Compliance and financial auditing demand verifiable evidence chains. Our staged task decomposition produces inspectable, deterministic intermediate artifacts (index spans, declared mappings, pagination rules) that can be audited separately.

4. Tables 6 and 12: Stage 3 is one subtask

Tables 6 and 12 both evaluate Stage 3, one subtask of the benchmark: resolving printed page numbers to physical PDF pages. Our VLM-based approach solves this subtask well: Sequential VLM reaches 100% where conventional methods reach 7.2–92.8%, and the paper shows hard cases such as National Grid, where an unnumbered panel resembles a two-page spread (Figure 7). The task as a whole, however, is far from solved: even with Gold spans, the best model reaches a mapping report exact match of only 70.00% in Stage 2 (Table 5), and the agents fully solve only 45 and 48 of 70 reports.

5. Writing: revised abstract

We revise the opening of the abstract as follows:

Document retrieval largely relies on semantic similarity between a query and text chunks. Yet many long documents already declare where key topics are located; for instance, corporate annual reports often provide disclosure indexes based on the Task Force on Climate-related Financial Disclosures (TCFD). While human readers naturally follow such indexes directly to the declared pages, automated index-guided navigation remains largely underexplored. We therefore introduce TCFDNav, a benchmark for evaluating the recovery and execution of index-declared navigation relations in the TCFD setting. …

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.