- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
A table RAG benchmark is constructed, featuring "sibling" tables as hard negatives to challenge table retrieval.
Critical strengths
The idea of synthesizing highly similar tables as hard negatives to challenge table retrieval is a very interesting idea. To my knowledge, current there is indeed a lack of such benchmarks.
Critical weaknesses
- Positioning of the paper is under-developed.
The paper argues that "existing evaluations largely emphasize semantic relevance rather thanwhether a retrieved table is uniquely sufficient to answer the query". This is not true. The examples given are table retrieval benchmarks, which unsurprisingly focus on relevance. The authors completely ignored open-domain table QA datasets, such as OTT-QA and NQ-TABLES, which exactly focus on answerability. This significantly weakens the novelty and contribution of this work.
- Evaluation protocol is overly simple.
To reveal the "semantic-answerability gap", a variety of retrievers are evaluated. However, it is unsurprising that only using retrievers is insufficient for identifying the answer. Today, there have been many RAG frameworks. They should be evaluted as they consider answerability. For example, the authors included reranker, answerability classifier, and LLM to perform query-conditioned answerability assessment, and obtained significantly improved results. One would expect that if RAG frameworks were used, better results would be obtained to close the gap. This reveals a critical shortcoming in evaluation design.
Questions for authors
Q1. Could you compare your benchmark with open-domain table QA datasets?
Q2. Have you tried using any RAG frameworks?
Q3. Who generated the queries in your benchmark? Humans or LLMs? I did not find relevant description in Appendix A.
Q4. Do you believe your sibling distractor generation can represent real-world cases? Is it often to see such highly similar tables in practice?
7 comments
Response to Reviewer fyCm (2/2)
W2: “Evaluate RAG frameworks”
The framework evaluation raised by the reviewer is indeed important: multi-stage RAG that couples retrieval with answerability-oriented scoring is a direct path to easing the gap. That is why the main text already reports a standard two-stage retrieve→rerank setup (dense→AAR-CE); the revision completes downstream QA for that setup and for CRAFT under the same generator, and adds a SAG2-style pipeline.
Concretely, the main text already includes dense→AAR-CE as a two-stage instance (R@1 0.182→0.574, DS@1 to 0.750), to see whether explicit answerability modeling eases within-group discrimination; the appendix already included CRAFT as a cascaded setting. A fuller multi-hop / multi-round system evaluation was not expanded at submission; the revision now reports QA for both under the same generator, and adds a SAG2-style pipeline (BM25 → sufficiency decision → expansion re-query, up to 2 rounds; SAG2 [1] is not our SAG). Numbers below.
| setting | R@1 | DS@1 | QA Overall F1 (top-1 / top-5) |
|---|---|---|---|
| Two-stage RAG (dense + AAR-CE numbers;) | 0.574 | 0.750 | 0.442 / 0.399 |
| CRAFT (appendix; Simplified / +LLM aug.) | 0.215 / 0.196 | 0.313 / 0.293 | 0.171 / 0.303; 0.196 / 0.290 |
| SAG2-style (BM25 + Reranker-0.6B / Decider-2B, up to 2 rounds; new) | 0.306 / 0.316 | 0.366 / 0.647 | 0.259 / 0.405; 0.232 / 0.296 |
Reference: Target-only Oracle 0.755; Dense E4B top-1/5 = 0.113 / 0.329.
Using methods that can ease the gap to raise retrieval and QA performance is only a small part of the paper: the main line also isolates SAG with siblings into measurable GR / R / DS, diagnoses first-stage failure via SenSS / RID / geometry, and probes whether embeddings can be reshaped under in-distribution answerability supervision. Multi-stage results show the gap can be eased; these diagnostics make the failure mode measurable and comparable, and provide a basis for later method design.
Q1 / Q2
OTT/NQ comparison: W1. Two-stage / CRAFT / SAG2-style retrieve→QA: W2.
Q3: Who generated the queries?
Queries are first built from multiple fixed templates for EM/SF/SA, then paraphrased with GPT-OSS-120B, and finally checked by humans (~5% corrected).
The appendix already described this pipeline, but it was mixed into a longer construction section and easy to miss; the revision states the template → GPT → human steps more clearly in one place.
Q4: Are Sibling Distractor Tables realistic?
Yes. Highly similar tables do arise in practice, often in messier forms than our controlled setting, and the setting itself is motivated by needs observed with an industry partner.
The introduction already notes that tables are widespread in enterprise databases, data lakes, and web resources [2], where semantically similar tables may still differ in content. Multiple near-duplicate versions of the same table after edits (truncation, partial rewrites, and the like) do occur in data repositories [3], especially in organizations or settings with incomplete data management; monthly/quarterly, store/warehouse, and fiscal-year slices of the same topic also coexist. Synthetic Siblings only isolate “neighbor ≠ answerable” more cleanly; real deployments are usually more complex. Real-TCR uses partner tables already partitioned by time/org unit (not answerability edits) and still shows high GR, low R (e.g., E4B GR@1=0.909, DS@1=0.043).
Revision already done
This response reports concise results; full protocols, hyperparameters, and additional numbers are in the revised manuscript at the locations below.
- Positioning vs OTT-QA / NQ-Tables: Introduction; Related Work.
- Same-encoder open-domain contrast: Appendix Open-Domain Neighborhood Geometry (Appendix E); Real-TCR in Appendix Real-TCR: Natural-Negative Validation (Appendix F).
- RAG pipelines and QA: §Modeling Answerability (Query-Conditioned Answerability Assessment); Appendix RAG-Style Pipelines Beyond Single-Vector Retrieval (Appendix D; CRAFT QA and SAG2-style).
- Query-generation pipeline: Appendix Benchmark Construction Details (Appendix A) → Source Tables and Query Generation (and Manual Validation).
We welcome a read of the revised manuscript alongside this response, and are happy to clarify further if anything remains unclear.
Thank you again for the comments.
References
[1] WU Y. SAG2: Embedding-Free Multi-Hop Retrieval with BM25 and JEV[EB/OL]. arXiv:2610.22588, 2026. https://arxiv.org/abs/2610.22588.
[2] CAFARELLA M J, et al. WebTables: Exploring the power of tables on the web[J]. Proc. VLDB Endow., 2008, 1(1): 538-549.
[3] LOU Y, et al. Datalore: Can a large language model find all lost scrolls in a data repository?[C]//ICDE. IEEE, 2024: 5170-5176.
Follow-up on response to W1
The rebuttal argued that "they (OTT-QA / NQ-Tables) ask whether the answerable table can be retrieved from a heterogeneous corpus; TCR asks which table among near-duplicates can actually answer." Frankly speaking, I did not see any significant difference between these two statements. Both are formulated as a query-oriented table ranking problem.
I fully understand your contribution. You focus on near-duplicate (but synthetic) tables, while OTT-QA / NQ-Tables rely on real (but often dissimilar) tables. It is true that your setting is more challenging, but one may question whether those synthetic near-duplicate tables are common in practice. The error patterns found on OTT-QA / NQ-Tables could be more useful for improving a real-world table retrieval system.
Response to Reviewer fyCm (1/2)
We thank Reviewer fyCm for the review. We especially appreciate the recognition that synthesizing highly similar tables as hard negatives to challenge table retrieval is a valuable direction, and that such benchmarks are currently scarce. We clarify positioning, evaluation, and the questions below.
W1: Positioning vs OTT-QA / NQ-Tables
OTT-QA and NQ-Tables do focus on answer support; the original claim that “existing evaluations largely emphasize semantic relevance” was too broad. The revision discusses both alongside other Table RAG work in the Introduction and Related Work, and narrows the claim to: existing evaluations largely do not distinguish semantic relevance from answerability, so a semantically related table is often treated as an answerable one. TCR-Bench supplies the controlled near-duplicate diagnosis that separates the two; it is not another open-domain TableQA leaderboard.
Our title, Semantically Similar, Yet Not Answerable, already states the focus: semantically close need not mean answerable. Target and Sibling Distractors are both derived from the same source table by controlled edits, e.g., dropping a condition or target column, or deleting/resampling rows that remove answer-bearing records, together with schema and surface variation. The aim is to keep substantial semantic and structural similarity while changing answerability. Fig.1 is typical: nearly matching schemas, yet a wrong time window ranks higher because its values sit close to the query phrasing. Embedding geometry likewise shows Siblings sit genuinely close to the Target (Qwen3-E4B: median within-group distance ≈0.08–0.11, corpus ≈0.40–0.55): retrieval often reaches the neighborhood (GR@1=0.684) yet fails to select the uniquely answerable table (R@1=0.172; 51.2% neighborhood hit without Target).
The contrast with OTT-QA / NQ-Tables is not about who scores higher, but about a different evaluation question: they ask whether the answerable table can be retrieved from a heterogeneous corpus; TCR asks which table among near-duplicates can actually answer. Table RAG evaluation overall remains limited, and different settings typically target different failure modes; answer support under open-domain heterogeneous retrieval is not the same test as answerability among near-duplicate candidates. Under Qwen3-E4B (same family as the main dense results; E0.6B shows the same pattern), we measure how often a non-answer table is closer to the query than the gold table: about 9.6% on OTT-QA and 12.6–13.2% on NQ-Tables, while gold R@1 stays high (≈0.86–0.90). That 9.6% is a lower bound under heterogeneous candidates; within same-source near-duplicate sibling groups on TCR-Bench, the neighborhood-hit-without-Target rate is 51.2% (GR@1=0.684, R@1=0.172). OTT / NQ use heterogeneous candidates with roughly one gold answerable table per query and score whether that table is retrieved. They have no near-duplicate Sibling groups derived from the same source table, so they cannot report our GR@1 (did retrieval reach the right group?) or DS@1 (within the group, was the uniquely answerable table selected?). The open-domain rates only show occasional ranking errors among dissimilar tables; those setups do not isolate TCR’s near-duplicate contrast. Real-TCR (natural partitions, not answerability edits) still shows high GR with low DS (E4B: GR@1=0.909, DS@1=0.043). This contrast is recorded in Appendix Open-Domain Neighborhood Geometry.