- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces query-conditioned table selection and reconstruction, together with PCRC: paragraph retrieval, progressive evidence incorporation, and KV-cache reuse. Three benchmarks cover financial, scientific, and synthetic degraded documents. Without task-specific finetuning, relevant context improves selection and HTML reconstruction while caching reduces repeated prefill.
Critical strengths
-
The problem is practical and goes beyond table retrieval. The output includes reconstructed HTML from rendered documents, not merely relevance decisions. Visual reconstruction and document context provide a clear extension to natural-language-conditioned table discovery.
-
The empirical investigation is substantial. Composite F1 improves from Qwen's 0.597/0.601/0.644 to 0.723/0.756/0.794 (Table 3). Ablations cover random/all/semantic/spatial context, shared/exclusive evidence, stopping, single-judgment inference with identical selected evidence, and another backbone. The strong generic-prefix-cache control in Table 14 is particularly useful.
-
Annotation and evaluation boundaries are documented carefully. A roughly 5% condition audit covers 217,229 condition–table pairs, with blinded checking, double annotation, and adjudication; gains persist on human-checked labels (§A.1.6, Tables 8 and 12). The paper explicitly reports candidate-conditional quality and identical quality with/without cache reuse.
Critical weaknesses
-
End-to-end robustness needs stronger evidence. Table 9 excludes unmatched source tables from recall, leaving missed detections unpenalized. Incorrect early rejection also ends evidence collection and prevents extraction (Algorithm 1, lines 3–7). Table 14’s “No reliability gating” control reports only aggregate scores, without measuring erroneous early stops or how often later evidence could correct them. The overall robustness claim in §A.4.9 (lines 1720–1723) therefore needs targeted validation.
-
The composite metric can reward missing difficult tables. In §A.3 (lines 1251–1258), F1-TEDS = F1 × EQ uses cell-weighted EQ only over true positives. With two equally sized positives, TEDS 1.0 and 0.1, and no false positives, selecting both scores 0.55; missing the second scores 0.667. Higher scores need not mean more complete extraction. Add a metric with a fixed denominator over all positives, zero credit for misses, and a false-positive penalty. This counterexample does not establish incorrect method rankings.
-
Evidence for gains over a simpler baseline remains limited. With identical evidence, single judgment plus generic prefix caching nearly matches PCRC’s quality (Table 14); generic caching also recovers most prefill savings under the same schedule (§A.4.5). Across 20 documents per dataset, PCRC is only about 3–5% faster, excluding detection, OCR, and NV-Embed indexing (Table 16). Context selection has clearer support; assessing progressive inference and custom caching requires total costs and timing variability. The 3.4–4.1-fold prefill saving over uncached execution does not imply equivalent acceleration over the simpler baseline.
Questions for authors
-
Does PCRC’s robustness advantage survive counting upstream misses? Report erroneous early-stop rates and the fraction correctable by later evidence (Table 9, Algorithm 1, Table 14).
-
Do the rankings and gains in Table 3 persist with a fixed denominator over all positives, zero credit for misses, and a false-positive penalty?
-
After including or amortizing detection, OCR, and NV-Embed indexing, how much speedup remains over single judgment with identical evidence and generic prefix caching? Report paired time differences and their variability (Tables 14 and 16).
9 comments
2. Fine-grained comparison of No reliability gating and PCRC
Table 14’s “No reliability gating” control reports only aggregate scores, without measuring erroneous early stops or how often later evidence could correct them.
We want to clarify that the original Table 14 already reports judgment F1, composite F1, prefill consumption, and R.R. for both variants. To show how continued evidence collection affects precision and recall, Table R3c adds the previously unreported judgment P / R and composite Pc / Rc for No reliability gating (marked †), alongside the corresponding PCRC scores from the original Tables 11 and 3.
Both variants use the same inputs, selected evidence pool, and cache-reuse policy under the original Table 14 protocol. No reliability gating continues through all selected exclusive paragraphs regardless of an earlier definite judgment.
Table R3c. Fine-grained comparison of PCRC and No reliability gating (No gating). † marks newly reported values; all other entries are reproduced from the original manuscript. P, R, and F1 are judgment-only micro scores; Pc, Rc, and F1c are composite scores. Quality scores are displayed to three decimal places; differences below are calculated from the displayed values. Prefill is measured per condition in thousands of tokens.
| Metric | FIN: PCRC | FIN: No gating | PUB: PCRC | PUB: No gating | DEG: PCRC | DEG: No gating |
|---|---|---|---|---|---|---|
| 0.888 | 0.889† | 0.884 | 0.898† | 0.899 | 0.891† | |
| 0.914 | 0.925† | 0.919 | 0.935† | 0.906 | 0.905† | |
| 0.901 | 0.907 | 0.901 | 0.916 | 0.902 | 0.898 | |
| 0.713 | 0.716† | 0.741 | 0.744† | 0.791 | 0.780† | |
| 0.734 | 0.745† | 0.771 | 0.775† | 0.797 | 0.792† | |
| (F1-TEDS) | 0.723 | 0.731 | 0.756 | 0.759 | 0.794 | 0.786 |
| Prefill tokens (k) | 32.95 | 40.19 | 13.43 | 17.45 | 19.57 | 23.96 |
The following changes are No reliability gating minus PCRC, with quality differences expressed in percentage points:
- FIN: additional evidence gives a small benefit, mainly in recall. Judgment P / R / F1 increase by 0.1 / 1.1 / 0.6 points, and composite Pc / Rc / F1c increase by 0.3 / 1.1 / 0.8 points. Continuing evidence collection therefore improves aggregate coverage, with a 0.8-point gain in composite F1.
- PUB: judgment improves more than the joint judgment-and-reconstruction score. Judgment P / R / F1 increase by 1.4 / 1.6 / 1.5 points, while composite Pc / Rc / F1c increase by only 0.3 / 0.4 / 0.3 points. The additional evidence improves final judgment, but its net benefit to composite F1 is only 0.3 points.
- DEG: consuming all remaining evidence lowers all six quality scores. Judgment P / R / F1 decrease by 0.8 / 0.1 / 0.4 points, and composite Pc / Rc / F1c decrease by 1.1 / 0.5 / 0.8 points. Continued evidence collection therefore provides no aggregate quality gain on this dataset.
- The additional evidence has a consistent token cost. Removing gating increases prefill from 32.95 / 13.43 / 19.57k to 40.19 / 17.45 / 23.96k on FIN / PUB / DEG: an extra 7.24 / 4.02 / 4.39k tokens, or 22.0% / 29.9% / 22.4%. This additional processing yields composite F1 changes of +0.8 / +0.3 / -0.8 points, respectively.
PCRC offers a favorable quality-cost trade-off: it saves 18.0% / 23.0% / 18.3% of prefill relative to No reliability gating, with composite F1 differences within 0.8 percentage points and higher quality on DEG.
Insisting on using all selected exclusive related paragraphs as evidence for judgment and reconstruction incurs additional prefill-token consumption: full evidence consumption and the savings from skipping later evidence cannot be obtained simultaneously. PCRC does not force the model to make an early binary decision; the uncertain option allows it to progressively request more evidence before committing. Our experiments demonstrate that this design maintains comparable task quality while avoiding the additional prefill required to consume all remaining evidence unconditionally.
3. The counterexample does not contradict our composite evaluation
The composite metric can reward missing difficult tables. In §A.3 (lines 1251–1258), F1-TEDS = F1 × EQ uses cell-weighted EQ only over true positives. With two equally sized positives, TEDS 1.0 and 0.1, and no false positives, selecting both scores 0.55; missing the second scores 0.667. Higher scores need not mean more complete extraction.
We want to clarify that our composite evaluation comprises three jointly reported scores: composite precision , composite recall , and composite F1 . To assess the combined outcome of condition judgment and table reconstruction, these three scores should be considered together. Table 3 in the paper follows this reporting protocol; the evaluation is not based on alone.
In your example, the two relevant tables have equal numbers of cells and TEDS scores of 1.0 and 0.1. Recovering both tables gives a lower than returning only the easier table (0.550 versus 0.667), but a higher (0.550 versus 0.500). The benefit of recovering the difficult table therefore remains visible within the three composite metrics themselves.
Table R3d. Composite scores for the reviewer's two equally sized positive tables, with TEDS scores of 1.0 and 0.1 and no false positives. Values are rounded to three decimals.
| Returned tables | |||
|---|---|---|---|
| Both tables | 0.550 | 0.550 | 0.550 |
| Only the perfect table | 1.000 | 0.500 | 0.667 |
Reading the three composite scores together does not support treating the output that misses the difficult table as uniformly better: its higher and are accompanied by lower . Thus, although the scalar reverses the ordering in this example, the composite evaluation still identifies the coverage advantage of recovering both relevant tables. This is why joint judgment-and-reconstruction quality should be assessed through , , and together.
We thank the reviewer for recognizing the importance of our research. We address the concerns below.
1. Evidence of end-to-end robustness
To address this concern, we conducted condition judgment and HTML reconstruction using table regions cropped directly from the datasets' annotated bounding boxes, rather than detector-generated proposals. This eliminates the influence of upstream table detection errors, allowing us to assess whether PCRC's quality and token-saving advantages persist when ground-truth table regions are provided.
Table R3a. Composite quality with source-annotated boxes (main-paper Table 3 layout). TEDS is the extraction metric. FIN, PUB, and DEG denote FINCONTAB, PUBCONTAB, and DEGCONTAB.
Table R3b. Per-condition VLM token consumption with source-annotated boxes (main-paper Table 4 layout). Pre., Dec., and Tot. are prefill, decode, and total tokens in thousands; R.R. is the recomputation ratio in percent. FIN, PUB, and DEG denote FINCONTAB, PUBCONTAB, and DEGCONTAB.
PCRC therefore retains both its quality lead and substantial cache savings when source-annotated boxes replace detector proposals. These advantages are not confined to the detector-generated candidate set used in the original experiments.