acceptodds
Under review as a conference paper at ICLR 2027

Semantically Similar, Yet Not Answerable: Diagnosing the Semantic-Answerability Gap in Table RAG

Abstract

In retrieval-augmented generation (RAG), semantic relevance asks whether a source matches a query in meaning, while answerability asks whether it contains sufficient information to answer the query. A Semantic-Answerability Gap (SAG) may arise in retrieval when a retriever can reach semantically relevant sources yet fail to identify those that are uniquely answerable. We uncover this gap using tables as a controlled setting, where shared schemas and entities provide strong semantic signals while localized content and row-column bindings distinguish answerable from non-answerable sources. Using TCR-Bench, a controlled sibling-table benchmark, we find that dense retrievers achieve only 18.2% top-1 target retrieval, reducing QA F1 from 0.755 with the oracle table to 0.330 with retrieved top-5 tables. Controlled diagnostics show that retrievers favor semantic volume over sufficiency, respond weakly to row-column binding disruptions, and struggle to distinguish Targets from Siblings. Explicit answerability assessment substantially improves target identification, while fine-tuning shows that answerability is learnable but difficult to transfer without compromising broad semantic retrieval.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Retrieval-Augmented Generation, Information Retrieval, Answerability, Dense Retrieval, Semantic Relevance, Table Retrieval

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fyCmOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

A table RAG benchmark is constructed, featuring "sibling" tables as hard negatives to challenge table retrieval.

Critical strengths

The idea of synthesizing highly similar tables as hard negatives to challenge table retrieval is a very interesting idea. To my knowledge, current there is indeed a lack of such benchmarks.

Critical weaknesses
  1. Positioning of the paper is under-developed.

The paper argues that "existing evaluations largely emphasize semantic relevance rather thanwhether a retrieved table is uniquely sufficient to answer the query". This is not true. The examples given are table retrieval benchmarks, which unsurprisingly focus on relevance. The authors completely ignored open-domain table QA datasets, such as OTT-QA and NQ-TABLES, which exactly focus on answerability. This significantly weakens the novelty and contribution of this work.

  1. Evaluation protocol is overly simple.

To reveal the "semantic-answerability gap", a variety of retrievers are evaluated. However, it is unsurprising that only using retrievers is insufficient for identifying the answer. Today, there have been many RAG frameworks. They should be evaluted as they consider answerability. For example, the authors included reranker, answerability classifier, and LLM to perform query-conditioned answerability assessment, and obtained significantly improved results. One would expect that if RAG frameworks were used, better results would be obtained to close the gap. This reveals a critical shortcoming in evaluation design.

Questions for authors

Q1. Could you compare your benchmark with open-domain table QA datasets?

Q2. Have you tried using any RAG frameworks?

Q3. Who generated the queries in your benchmark? Humans or LLMs? I did not find relevant description in Appendix A.

Q4. Do you believe your sibling distractor generation can represent real-world cases? Is it often to see such highly similar tables in practice?

7 comments
AuthorsOpenReview ↗

Response to Reviewer fyCm (1/2)

We thank Reviewer fyCm for the review. We especially appreciate the recognition that synthesizing highly similar tables as hard negatives to challenge table retrieval is a valuable direction, and that such benchmarks are currently scarce. We clarify positioning, evaluation, and the questions below.

W1: Positioning vs OTT-QA / NQ-Tables

OTT-QA and NQ-Tables do focus on answer support; the original claim that “existing evaluations largely emphasize semantic relevance” was too broad. The revision discusses both alongside other Table RAG work in the Introduction and Related Work, and narrows the claim to: existing evaluations largely do not distinguish semantic relevance from answerability, so a semantically related table is often treated as an answerable one. TCR-Bench supplies the controlled near-duplicate diagnosis that separates the two; it is not another open-domain TableQA leaderboard.

Our title, Semantically Similar, Yet Not Answerable, already states the focus: semantically close need not mean answerable. Target and Sibling Distractors are both derived from the same source table by controlled edits, e.g., dropping a condition or target column, or deleting/resampling rows that remove answer-bearing records, together with schema and surface variation. The aim is to keep substantial semantic and structural similarity while changing answerability. Fig.1 is typical: nearly matching schemas, yet a wrong time window ranks higher because its values sit close to the query phrasing. Embedding geometry likewise shows Siblings sit genuinely close to the Target (Qwen3-E4B: median within-group distance ≈0.08–0.11, corpus ≈0.40–0.55): retrieval often reaches the neighborhood (GR@1=0.684) yet fails to select the uniquely answerable table (R@1=0.172; 51.2% neighborhood hit without Target).

The contrast with OTT-QA / NQ-Tables is not about who scores higher, but about a different evaluation question: they ask whether the answerable table can be retrieved from a heterogeneous corpus; TCR asks which table among near-duplicates can actually answer. Table RAG evaluation overall remains limited, and different settings typically target different failure modes; answer support under open-domain heterogeneous retrieval is not the same test as answerability among near-duplicate candidates. Under Qwen3-E4B (same family as the main dense results; E0.6B shows the same pattern), we measure how often a non-answer table is closer to the query than the gold table: about 9.6% on OTT-QA and 12.6–13.2% on NQ-Tables, while gold R@1 stays high (≈0.86–0.90). That 9.6% is a lower bound under heterogeneous candidates; within same-source near-duplicate sibling groups on TCR-Bench, the neighborhood-hit-without-Target rate is 51.2% (GR@1=0.684, R@1=0.172). OTT / NQ use heterogeneous candidates with roughly one gold answerable table per query and score whether that table is retrieved. They have no near-duplicate Sibling groups derived from the same source table, so they cannot report our GR@1 (did retrieval reach the right group?) or DS@1 (within the group, was the uniquely answerable table selected?). The open-domain rates only show occasional ranking errors among dissimilar tables; those setups do not isolate TCR’s near-duplicate contrast. Real-TCR (natural partitions, not answerability edits) still shows high GR with low DS (E4B: GR@1=0.909, DS@1=0.043). This contrast is recorded in Appendix Open-Domain Neighborhood Geometry.

AuthorsOpenReview ↗

Response to Reviewer fyCm (2/2)

W2: “Evaluate RAG frameworks”

The framework evaluation raised by the reviewer is indeed important: multi-stage RAG that couples retrieval with answerability-oriented scoring is a direct path to easing the gap. That is why the main text already reports a standard two-stage retrieve→rerank setup (dense→AAR-CE); the revision completes downstream QA for that setup and for CRAFT under the same generator, and adds a SAG2-style pipeline.

Concretely, the main text already includes dense→AAR-CE as a two-stage instance (R@1 0.182→0.574, DS@1 to 0.750), to see whether explicit answerability modeling eases within-group discrimination; the appendix already included CRAFT as a cascaded setting. A fuller multi-hop / multi-round system evaluation was not expanded at submission; the revision now reports QA for both under the same generator, and adds a SAG2-style pipeline (BM25 → sufficiency decision → expansion re-query, up to 2 rounds; SAG2 [1] is not our SAG). Numbers below.

settingR@1DS@1QA Overall F1 (top-1 / top-5)
Two-stage RAG (dense + AAR-CE numbers;)0.5740.7500.442 / 0.399
CRAFT (appendix; Simplified / +LLM aug.)0.215 / 0.1960.313 / 0.2930.171 / 0.303; 0.196 / 0.290
SAG2-style (BM25 + Reranker-0.6B / Decider-2B, up to 2 rounds; new)0.306 / 0.3160.366 / 0.6470.259 / 0.405; 0.232 / 0.296

Reference: Target-only Oracle 0.755; Dense E4B top-1/5 = 0.113 / 0.329.

Using methods that can ease the gap to raise retrieval and QA performance is only a small part of the paper: the main line also isolates SAG with siblings into measurable GR / R / DS, diagnoses first-stage failure via SenSS / RID / geometry, and probes whether embeddings can be reshaped under in-distribution answerability supervision. Multi-stage results show the gap can be eased; these diagnostics make the failure mode measurable and comparable, and provide a basis for later method design.

Q1 / Q2

OTT/NQ comparison: W1. Two-stage / CRAFT / SAG2-style retrieve→QA: W2.

Q3: Who generated the queries?

Queries are first built from multiple fixed templates for EM/SF/SA, then paraphrased with GPT-OSS-120B, and finally checked by humans (~5% corrected).

The appendix already described this pipeline, but it was mixed into a longer construction section and easy to miss; the revision states the template → GPT → human steps more clearly in one place.

Q4: Are Sibling Distractor Tables realistic?

Yes. Highly similar tables do arise in practice, often in messier forms than our controlled setting, and the setting itself is motivated by needs observed with an industry partner.

The introduction already notes that tables are widespread in enterprise databases, data lakes, and web resources [2], where semantically similar tables may still differ in content. Multiple near-duplicate versions of the same table after edits (truncation, partial rewrites, and the like) do occur in data repositories [3], especially in organizations or settings with incomplete data management; monthly/quarterly, store/warehouse, and fiscal-year slices of the same topic also coexist. Synthetic Siblings only isolate “neighbor ≠ answerable” more cleanly; real deployments are usually more complex. Real-TCR uses partner tables already partitioned by time/org unit (not answerability edits) and still shows high GR, low R (e.g., E4B GR@1=0.909, DS@1=0.043).

Revision already done

This response reports concise results; full protocols, hyperparameters, and additional numbers are in the revised manuscript at the locations below.

  • Positioning vs OTT-QA / NQ-Tables: Introduction; Related Work.
  • Same-encoder open-domain contrast: Appendix Open-Domain Neighborhood Geometry (Appendix E); Real-TCR in Appendix Real-TCR: Natural-Negative Validation (Appendix F).
  • RAG pipelines and QA: §Modeling Answerability (Query-Conditioned Answerability Assessment); Appendix RAG-Style Pipelines Beyond Single-Vector Retrieval (Appendix D; CRAFT QA and SAG2-style).
  • Query-generation pipeline: Appendix Benchmark Construction Details (Appendix A) → Source Tables and Query Generation (and Manual Validation).

We welcome a read of the revised manuscript alongside this response, and are happy to clarify further if anything remains unclear.

Thank you again for the comments.

References

[1] WU Y. SAG2: Embedding-Free Multi-Hop Retrieval with BM25 and JEV[EB/OL]. arXiv:2610.22588, 2026. https://arxiv.org/abs/2610.22588.
[2] CAFARELLA M J, et al. WebTables: Exploring the power of tables on the web[J]. Proc. VLDB Endow., 2008, 1(1): 538-549.
[3] LOU Y, et al. Datalore: Can a large language model find all lost scrolls in a data repository?[C]//ICDE. IEEE, 2024: 5170-5176.

Reviewer_fyCmOpenReview ↗

Follow-up on response to W1

The rebuttal argued that "they (OTT-QA / NQ-Tables) ask whether the answerable table can be retrieved from a heterogeneous corpus; TCR asks which table among near-duplicates can actually answer." Frankly speaking, I did not see any significant difference between these two statements. Both are formulated as a query-oriented table ranking problem.

I fully understand your contribution. You focus on near-duplicate (but synthetic) tables, while OTT-QA / NQ-Tables rely on real (but often dissimilar) tables. It is true that your setting is more challenging, but one may question whether those synthetic near-duplicate tables are common in practice. The error patterns found on OTT-QA / NQ-Tables could be more useful for improving a real-world table retrieval system.

Reviewer_B6niOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper says that a RAG retrieval should be evaluated on answerability, which means whether the source contains enough information to answer the query and not just semantic similarity. It coins the mismatch as the semantic answerability gap and builds a dataset called TCR bench to study it. Here, each query is paired with one answerable table and several similar sibling tables that do not contain the answer. They find that the best density reaches high group recall, but low recall. When trying to diagnose the gap, the paper lands on three things; first being sensitivity to the volume of the semantic content rather than the sufficiency . Second is weak sensitivity to row-column bindings. The third is the compact sibling clusters in embedding space where the target is usually not nearest to the query. They find that answerability-aware re-ranking raises recall. Fine-tuning an embedder on sibling negatives raises in-domain recall, but hurts generalizability and degrades general retrieval.

Critical strengths

The paper provides a more accurate, reliable split of retrieval failure into stages. This gap between the recall@1 and group recall@1 is insightful and draws the community's attention to the fact that retrievers land in the right group but discriminate poorly.

The paper also does a good job of ruling out alternative explanations. Several dense embedders were tested. They also vary serialization (formatting), query phrasing, and compare an embedder against a re-ranker, etc. All of this makes the main claim robust within the paper's settings.

Critical weaknesses

Some findings of the paper are expected. For embeddings, for example, one vector trying to summarize the whole table does not perform well in very similar-looking tables. But if we give the model the query and the table together, it does much better at a much higher cost. The paper's main value seems to be the benchmark and the diagnostic contributions; the gap itself is not fully new insight.

If I am understanding correctly, the bench has no training/test split. The fine-tuned model was trained and evaluated on the same queries. So, does not completely establish that answerability is learnable.

Questions for authors

Can authors report the fine-tuning results with a held-out split? Perhaps this helps us better assess "answerability is learnable" claim.

Eff@1 exceeds 1 sometimes; what does it signify? Are there some guarantees that sibling tables never answer the query?

2 comments
AuthorsOpenReview ↗

Response to Reviewer B6ni (1/2)

We thank Reviewer B6ni for the constructive review. We especially appreciate the recognition that the paper cleanly separates stages of retrieval failure, that the GR@1 versus R@1 gap is insightful, and that the controls over embedders, serialization, and reranking strengthen the main conclusions. We address the concerns and questions below.

W1: “Single-vector limits are unsurprising”

The SAG we propose is not merely the qualitative intuition that “single-vector retrieval is limited,” but a more quantitative finding; the controlled benchmark and later diagnostics serve to measure and disentangle that gap. Contribution (1) is exactly this: under the sibling setting, dense retrieval can reach the right semantic neighborhood yet fail to select the uniquely answerable table. The benchmark and diagnostic parts then further separate the signals: Contribution (2) is an in-distribution dense-retrieval probe under the sibling setting, and Contribution (3) asks whether matched supervision can reshape embeddings.

As community intuition, single-vector limits on near-duplicate tables, and stronger but costlier query-conditioned scoring, are not surprising; that also matches the paper’s reranker / AAR results (e.g., dense→AAR-CE: R@1 0.182→0.574). The Semantic-Answerability Gap (SAG) gives a checkable quantitative reading: under the Sibling Distractor setting, group hit can be high (best dense GR@1 0.670), while selecting the uniquely answerable table within the group stays near chance (DS@1 near the 0.298 baseline; conditional random still ≈0.22). We separately report GR@1, R@1, and DS@1, and use interventions plus geometry to locate the associated signals; the diagnostic focus remains the first stage, isolating group hit versus uniquely answerable selection. We start from a simple, clear-boundary, single-table, single-turn sibling setting precisely to measure that quantitative gap cleanly before discussing richer RAG settings.

W2 / Q1: Held-out and “learnability”

“Learnable” is only a weak, in-distribution diagnostic claim: whether matched supervision can reshape embeddings, not whether the model has learned generalizable answerability rules. The same-distribution held-outs and the paper’s cross-distribution results further support that reading; the revision rewrites the third contribution and the fine-tuning paragraph as an in-distribution diagnostic probe.

The original setting does train and evaluate on the same queries, so in-distribution gains alone cannot establish transferable “learnability.” By “learnable” we meant only the weak diagnostic claim: if matched supervision cannot raise Target Table versus Sibling Distractor Table discrimination (R@1 / DS@1), single-vector retrieval cannot access answerability; if it can, the embedding space is at least reshapeable under that signal, rather than implying transferable answerability rules. In-distribution gains should therefore be read as a probe result, not as extrapolation evidence.

To answer the held-out request directly, we add two same-distribution held-outs under one shared 37-setting sweep (Appendix M) and report counts on the test set:

(1) DB-disjoint (147/62; table overlap 0). Frozen R@1 is 15/62, and the paper-best recipe peaks at only 7/62. Under a limited schedule we first run a compact lr×temp core, then modestly expand along negatives, warmup/weight decay, and a longer schedule; a few peaks exceed frozen (best 20/62), mostly early spikes that collapse, and one run ends at 17/62.

(2) DB-overlap query-stratified (146/63; db overlap 46.2%). Frozen R@1 is 13/63; only 4 peaks exceed frozen (best 17/63), all early, and all four finish below frozen.

At this test size, proportion confidence intervals show these gaps are not significant: there is no stable held-out improvement, nor statistically supported degradation relative to frozen. The paper’s weak cross-distribution TCR→Real-TCR transfer contrasts with these held-outs: a TCR-Bench-trained checkpoint reaches only about 0.065 R@1 on Real-TCR, versus 0.442 under matched Real-TCR training. Same-distribution extrapolation is already hard; cross-distribution transfer is weaker still. That is consistent with reading in-distribution gains as largely overfitting, and again indicates that “learnable” here means reshapeable under matched supervision, not generalizable answerability rules. The revision rewrites the third contribution and the fine-tuning paragraph in those terms.

AuthorsOpenReview ↗

Response to Reviewer B6ni (2/2)

Q2: Eff@1 > 1; can Sibling Distractor Tables ever answer?

Eff@1>1 follows from using a global Oracle average in the denominator: when conditional QA on queries where the Target Table ranks first is slightly above that global average, the ratio can exceed 1 (values 1.046 / 1.014). Sibling Distractor Tables are logically non-answerable by construction; wrong-table top-1 answers are rare.

Concretely, Eff@k = (QA@k / Oracle) / R@k. Oracle averages over all queries, while the numerator is measured on the subset where the Target Table ranks first. If that subset is slightly easier, Eff@1 can overshoot 1. That is an expected fluctuation. Sibling Distractor Tables remove answering evidence and are thus logically non-answerable. Reported Eff@1 values are only slightly above 1; if wrong tables at top-1 were often answered correctly, overall QA@1 would rise and Eff@1 would be much larger. On dense top-1 misses the mean F1 is only about 0.006–0.018 (only a few nonzero cases out of 209); under AAR-CE the corresponding miss-subset mean F1 is about 0.023. The main-line guarantee is therefore logical non-answerability; generation may still show small residual error.

Revision already done

This response reports concise results; full protocols, hyperparameters, and additional numbers are in the revised manuscript at the locations below.

  • Third contribution / “learnable”: Abstract; Introduction contribution (3); §Modeling Answerability (Learning Answerability in Embeddings); Appendix Held-out splits (DB-disjoint + DB-overlap; Appendix M).
  • Eff@1 and sibling false negatives: §Downstream QA Evaluation; Appendix Sibling Distractor Generation (Appendix A); Appendix Sibling Interference and Controlled Downstream Packing (miss subset).
  • Diagnostic narrative (isolate GR vs. answerable-table selection): Introduction; Related Work; §A Controlled Setting for Studying Answerability.

We welcome a read of the revised manuscript alongside this response, and are happy to clarify further if anything remains unclear.

Thank you again for the comments.

Reviewer_cwRxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the Semantic-Answerability Gap (SAG) in table retrieval-augmented generation: retrievers can identify tables with similar topics, schemas, or entities, yet may fail to identify those containing sufficient evidence to answer the query.

The authors construct TCR-Bench, which provides an answerable target table for each query alongside semantically and structurally similar distractor tables that lack the necessary evidence. The benchmark contains 209 queries and 637 tables, covering exact matching, conditional filtering, and conditional aggregation. Experiments show that Qwen3-Embedding-8B achieves a group-level recall GR@1 of 67.0%, but a target-table recall R@1 of only 18.2%. Downstream question-answering F1 drops from 0.755 with the ground-truth target table to 0.330 with the top five retrieved tables.

The paper further investigates this phenomenon through experiments on serialization, content removal, cell arrangement, and embedding geometry, with supplementary validation on Real-TCR, a collection of real-world business tables. Query-conditioned reranking improves R@1 to 57.4%. Fine-tuning experiments suggest that models can improve target-table identification on training examples, but the tested configurations show limited cross-distribution transfer and retention of general retrieval capabilities.

Critical strengths
  1. The benchmark exposes retrieval failures that conventional relevance evaluations can obscure.

By constructing candidate tables that share substantial semantic and structural information, TCR-Bench requires retrievers to identify answer sources satisfying specific query conditions. Jointly reporting GR@1 and R@1 provides a more informative diagnosis of where retrieval fails than reporting recall alone.

The authors also conduct answerability checks across the full candidate pool, screen for potential false negatives, and perform manual validation. These measures are particularly important for a diagnostic benchmark, as they reduce the risk of incorrectly labeling alternative valid answer sources as negatives.

  1. The core phenomenon is evaluated broadly, with meaningful positive controls.

The paper evaluates dense retrievers of multiple sizes, alongside sparse, hybrid, multi-vector, and table-specific methods, and examines serialization, table length, and query type. Real-TCR provides supplementary evidence that the phenomenon does not depend entirely on the construction of synthetic distractor tables.

Critical weaknesses
  1. The fine-tuning experiments primarily demonstrate that training examples can be fitted, rather than sufficiently supporting stronger interpretations about learnability and transfer trade-offs.

Appendix L explicitly states that TCR-Bench and Real-TCR do not have separate training splits. The improvements in R@1 from 0.182 to 0.493 on TCR-Bench, and to 0.442 on Real-TCR, are measured on data used for training. These results therefore cannot distinguish whether the model learns generalizable answerability rules or memorizes particular queries, tables, or distractor patterns.

The authors’ explicit acknowledgment of the exploratory nature of these experiments is appreciated. However, without an independent in-distribution evaluation, subsequent poor cross-dataset performance is insufficient to attribute the problem to difficulty in transferring answerability capabilities. Training-set overfitting and ordinary domain shift remain plausible explanations.

Furthermore, the maximum passage length used for answerability fine-tuning is 512 tokens, whereas tables in the main benchmark are approximately 6,500–7,800 tokens long. The paper needs to explain how answer evidence is retained during training; otherwise, the signal the model actually learns remains uncertain. These issues directly affect the credibility of the third main contribution.

  1. Some core diagnostic experiments do not sufficiently isolate variables and therefore do not directly support the current mechanistic interpretations.

First, RID+Shuf in Appendix G.2 retains the original row IDs attached to the corresponding values when shuffling cell positions. Matching IDs can still be used to reconstruct the original logical records. Thus, the intervention changes spatial arrangement without necessarily disrupting the logical associations required to answer the query. Performance changes may reflect the model’s ability to process dispersed records and unusual serialization, rather than directly indicating insensitivity to row–column binding.

Second, the decrease in table-level recall after SenSS removes redundant attributes may also result from losing information useful for identifying the semantic group. In Table 2, Qwen3-E4B’s GR@1 decreases from 0.987 with Sen to 0.865 with SenSS, while DS@1 increases from 0.235 to 0.261. Thus, “more semantic information improves overall retrieval” and “the model favors semantic content over answerability” are different conclusions, and the current analysis does not sufficiently distinguish them.

Because explaining retrieval failures is a main contribution of the paper, these issues affect the diagnostic conclusions themselves, beyond their wording.

Questions for authors

The following questions are intended to clarify the current conclusions, and the answers may change my assessment.

  1. Independent generalization and input length: Could you split training and test data by source table or database to evaluate answerability gains on unseen queries and unseen tables? How are answer-relevant rows retained within the 512-token fine-tuning input? Have you measured the proportion of examples in which truncation removes the training signal?
  2. Interpretation of interventions: Since RID+Shuf retains the original row IDs, the original records remain recoverable. Is the experiment intended to disrupt spatial adjacency or logical binding? Also, how do you explain the decrease in group-level recall alongside the slight increase in within-group discrimination for Qwen3-E4B under SenSS?
  3. Random baselines and statistical accounting: The main text reports two random DS@1 baselines, 0.221 and 0.298. Which candidate pools do Figure 3 and Appendix M correspond to? Since DS@1 is conditional on the model retrieving from the correct group, have you considered calculating a corresponding random baseline using the queries actually retrieved into the correct group by each model and their group sizes? Should queries sharing source or candidate tables be bootstrapped at the group level?
  4. Downstream interference: Eff@k is normalized by the overall Oracle F1, but the set of queries for which the target table is successfully retrieved differs across k. Could you evaluate the same queries while fixing the target table and its position, varying only the number of added distractor tables? This would more clearly distinguish contextual interference from the effects of query difficulty and changes in target position.
4 comments
AuthorsOpenReview ↗

Response to Reviewer cwRx (3/3)

Q3: Random DS@1 and statistics

The two random baselines are defined by query sets, not by “one candidate pool for Figure 3 and another for Appendix M.” 0.221 is E[1/∣Gi∣]E[1/|G_i|] on all 209 queries; 0.298 is the same expectation after excluding column-deletion variants. Figure 3, the main-text comparisons, and Appendix M (significance; now Appendix N) all use 0.298; Appendix M (now Appendix N) adds confidence intervals on that same main comparison, so the estimates can also be compared with the full-set 0.221. Conditional random stays ≈0.22; models are not clearly above chance.

We also average 1/∣Gi∣1/|G_i| only over queries with GR@1=1 (i.e., those that already land in the correct sibling group). The table below also reports source-table cluster-bootstrap 95% confidence intervals.

ModelDS@1conditional randomgapsource-table cluster-bootstrap 95% CI
Qwen3-E4B0.2520.221+0.031[0.185, 0.333]
Jina0.2540.222+0.032[0.183, 0.324]
Stella0.1820.224−0.042[0.111, 0.261]

Intervals typically cover 0.221 and often cover 0.298. The revision states the Figure 3 / Appendix M (now Appendix N) mapping to the two baselines and adds this table.

Q4: Position and interference (fixed-Target packing)

We fix the uniquely answerable Target Table and its rank, and vary only the number and source of distractors, so that context interference is separated from query difficulty and Target-position changes.

Downstream generation uses the same Qwen3-30B setup as the main paper, reported as Overall F1. In the main-text dense rankings, when the Target already ranks first, the remaining top-5 slots still hold 2.27 Sibling Distractor Tables on average. The controlled packs follow from that: we first fix the Target at rank 1 and vary only the remaining context among Sibling Distractor Tables or Non-Sibling Tables; two further packs place the Target at rank 3 / 5 together with up to 4 Sibling Distractor Tables in the same context, to isolate rank itself.

settingmeaningOverall F1
Target onlyTarget Table alone, no distractors0.755
Target@1 + 2 / + 4 Sibling Distractors5-table context; Target fixed at rank 1; other slots get 2 or 4 Siblings (or all available)0.566 / 0.516
Target at rank 3 / 55-table context; Target fixed at rank 3 / 5; other 4 slots filled with Siblings (or all available)0.461 / 0.551
Target@1 + 2 / + 4 Non-Sibling; 2 Sibling + 2 Non-SiblingTarget@1; remaining slots Non-Sibling, or a Sibling/Non-Sibling mix0.692 / 0.679 / 0.571

Relative to Target-only (0.755), adding distractors or moving the Target later all lower Overall F1. Sibling Distractor Tables hurt more than Non-Sibling Tables; first-stage hit quality still dominates.

Revision already done

This response reports concise results; full protocols, hyperparameters, and additional numbers are in the revised manuscript at the locations below.

  • Third contribution / “learnable”: Abstract; Introduction contribution (3); §Modeling Answerability (Learning Answerability in Embeddings); Appendix Held-out splits ((1) DB-disjoint + (2) DB-overlap; Appendix M).
  • Positive answer-row fronting + 512 truncation audit: §Modeling Answerability (Learning Answerability in Embeddings); Appendix Truncation audit (Appendix M).
  • RID / SenSS mechanistic wording: §Sensitivity to Semantic and Structural Information (Semantic volume; Logical row binding); Appendix Dense Retrieval Diagnostics.
  • Random DS@1 and statistics: Figure 3; Appendix Significance (now Appendix N; conditional random and cluster bootstrap).
  • Eff@1, sibling false negatives, and fixed-Target packing: §Downstream QA Evaluation; Appendix Sibling Distractor Generation (Appendix A); Appendix Sibling Interference and Controlled Downstream Packing (miss subset and packing).

We welcome a read of the revised manuscript alongside this response, and are happy to clarify further if anything remains unclear.

Thank you again for the comments.

AuthorsOpenReview ↗

Response to Reviewer cwRx (2/3)

W2: RID+Shuf and SenSS

RID+Shuf is intended to disrupt spatial adjacency while leaving logical rows recoverable via RID, not to sever logical binding. SenSS keeps logically sufficient information and drops redundant attributes. Each asks whether the model uses those logical cues.

Logical records remain recoverable after RID+Shuf, matching the setting: we first attach a row index (RID) to each cell, then shuffle values within columns; spatial adjacency changes, but the same RID can still reassemble logical rows. The table below reports full-pool R@1 for Qwen3-E4B.

settingmeaningR@1
Oritable without row IDs0.443
RIDeach cell prefixed with its row ID0.386
RID+Shufattach RID, then shuffle within columns0.314

RID+Shuf is weaker than both Ori and RID (0.314 vs 0.443/0.386). If the concern is that the drop comes from uncommon serialization, Ori vs RID is the check: across the full appendix table the two are broadly similar with mixed wins (clearer on Stella, where RID is often slightly better or close to Ori), and RID+Shuf rarely beats both. RID and RID+Shuf share the same row-ID writing and differ mainly by within-column shuffle; relative to RID, RID+Shuf still drops clearly while logical rows remain recoverable via RID, so the model does not make effective use of that recoverable cue. The revision tightens the wording to: sensitive to spatial disruption (dispersed records), failing to use RID to rebuild logical rows, and remaining more tied to surface layout.

Under SenSS, GR↓ and the slight DS↑ split: the group-recall drop corresponds to losing cues for the semantic group; the within-group rise stays near chance, so a sufficiency subset does not improve answerability discrimination. SenSS is already logically sufficient; if ranking followed sufficiency it should be close to Sen/SenS, but performance tracks semantic volume, so retrieval is driven more by semantic volume than by logical sufficiency.

SenSS keeps only a logically sufficient subset for answering. If logical sufficiency dominated ranking, an already-sufficient SenSS should not be systematically weaker than Sen/SenS. What we observe is that Sen/SenS raises R and GR over SenSS, so more semantic information helps overall retrieval; via this observation, ranking does not follow sufficiency, and scores move with semantic volume. The submitted manuscript’s favor semantic volume over (information) sufficiency refers to this performance dependence (wording: sufficiency, not answerability), not a subjective preference.

On the inter-table setting for Qwen3-E4B, Sen→SenSS moves GR@1 0.987→0.865, R@1 0.232→0.226, and DS@1 0.235→0.261. The drop in group-level recall corresponds to also losing cues useful for identifying the semantic group when redundant attributes are removed; the slight rise in within-group discrimination must be read with R@1 (appendix DTR is the case: R@1 only 0.005/0.010, yet DS@1 can report 1.000/0.250), and after that rise E4B DS@1 remains near chance, so within-group answerability discrimination does not meaningfully improve. Under the same Sen→SenSS contrast, Stella moves R@1 0.213→0.148 and DS@1 0.214→0.187; applying the same sufficiency-only subset in intra-table row-level retrieval, Qwen3-E4B R@1 moves 0.871→0.619. Taken together, the sufficiency subset neither lifts E4B DS@1 off chance nor raises Stella DS@1, and the intra-table primary metric likewise falls with less semantic volume. The revision revises the favor wording to logical sufficiency and tightens the claim to this counterfactual reading.

The revision tightens the mechanistic claims for RID / SenSS; the sibling isolation and GR / R / DS diagnostic main line are unchanged.

Q1 / Q2

Held-out and 512 evidence: see W1. RID+Shuf intent (spatial adjacency) and the SenSS GR/DS reading: see W2.

AuthorsOpenReview ↗

Response to Reviewer cwRx (1/3)

We thank Reviewer cwRx for the careful review. We especially appreciate the recognition that TCR-Bench exposes retrieval failures often masked by relevance-only evaluation, that jointly reporting GR@1 and R@1 is diagnostically useful, and that answerability checks plus Real-TCR provide meaningful positive controls. We address the concerns and questions below.

W1: Fine-tuning without held-out; 512 truncation; “learnability”

“Learnable” is only a weak, in-distribution diagnostic claim: whether matched supervision can reshape embeddings, not whether the model has learned generalizable answerability rules.

By “learnable” we meant only the weak diagnostic claim: if matched supervision cannot raise Target Table versus Sibling Distractor Table discrimination (R@1 / DS@1), single-vector retrieval cannot access answerability; if it can, the embedding space is at least reshapeable under that signal, rather than encoding transferable answerability rules. The revision revises the third contribution and the fine-tuning paragraph to an in-distribution diagnostic probe so they are not read as a transfer claim.

To test whether that signal extrapolates, we add two same-distribution held-outs under one shared 37-setting grid (Appendix M), reported in absolute test counts:

(1) DB-disjoint (147/62; table overlap 0). Frozen R@1 is 15/62; the paper-best recipe peaks at only 7/62. A limited sweep (compact lr×temp core, then modest expansion along negatives, warmup/weight decay, and longer schedules) finds occasional peaks above frozen (best 20/62), mostly early spikes that collapse; one run ends at 17/62.

(2) DB-overlap, query-stratified (146/63; db overlap 46.2%). Frozen 13/63; only 4 peaks exceed frozen (best 17/63), all early, and all four finish below frozen.

At this test size, proportion confidence intervals show these gaps are not significant: there is no stable held-out improvement, and no statistically supported degradation relative to frozen. The paper’s weak TCR→Real-TCR transfer is cross-distribution, and contrasts with these held-outs: stable extrapolation is already hard without leaving the benchmark, and weaker still across distributions. That reading is consistent with the concern that the original matched gain may largely reflect in-distribution overfitting, and it again marks our “learnable” claim as in-distribution reshapeability under supervision, not that generalizable answerability rules were learned.

On 512: when building training positives, we attempt to move answer-bearing rows to the front of the table, then truncate to 512 tokens; the held-out train/eval positives use the same serialization. The revision states this explicitly. Auditing those same answer-row positions, 208/209 ≈ 99.5% still show answer-row evidence in the retained prefix (EM 98/98, SF 55/55, SA 55/56); the remaining 1 is an extremely wide table whose serialized header alone already exceeds 512 tokens, so answer rows never enter the retained prefix. The table below breaks down truncation retention against the paper-best fine-tune (Qwen3-E0.6B; overall R@1 0.182→0.493, DS@1 0.290→0.691). DS@1 is the within-group discriminative score: whether the uniquely answerable table is selected inside the correct sibling group.

typeanswer-row evidence kept after trunc.frozen R@1FT R@1frozen DS@1FT DS@1
EM (Exact Match, 98)98/98 = 100%0.2450.3670.3240.581
SF (Selective Filtering, 55)55/55 = 100%0.1270.6000.2920.805
SA (Selective Aggregation, 56)55/56 ≈ 98.2%0.1250.6070.2120.739

So under the 512 setting, answer-row evidence almost always remains in the prefix; in-distribution fine-tuning lifts EM/SF/SA alike, rather than fitting only one type. The main 512 setting was forced by submission-time compute: full-parameter hard-negative fine-tuning with many negatives and long serialized tables simply could not afford longer context then. That does not mean we think long context is unimportant. A preliminary length-extension sweep at 1024/2048 peaks at R@1 0.440 / 0.349 versus ≈0.493 at 512; none beat the main setting in our runs. Under the current setup, lengthening is not an easy free gain.

AuthorsOpenReview ↗

Follow-up to Reviewer cwRx

Dear Reviewer cwRx,

As the discussion period approaches its end, we wanted to gently follow up. The detailed responses are in our earlier reply and the revised manuscript; here is only a brief summary. As requested, we added held-out splits, a truncation audit, tighter random/bootstrap baselines, and controlled packing. Based on those results and analyses, we narrow the fine-tuning claim to an in-distribution diagnostic probe and tighten the RID+Shuf and SenSS readings. The main findings remain unchanged: matched gains should not be treated as transferable learnability, within-group answerability selection stays near chance under stricter conditional baselines and statistical controls, and sibling context still hurts downstream QA when the Target is held fixed. Please take another look when convenient. If any concern remains unresolved, we would be happy to clarify further.

Thank you again for your careful review.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author Rebuttal Summary

We briefly summarize the discussion-stage updates for reviewers and the Area Chair. Full details remain in the per-reviewer replies and in the revised manuscript.

1. Stronger checks added in the revision. Following requests for tighter evaluation, we (i) train and test the answerability fine-tuning probe on held-out splits (no shared databases, or shared tables but held-out queries), and find no stable gain over the frozen model at the available test size; (ii) check that answer evidence still appears in the short training context for almost all positives (208/209); (iii) compare within-group selection against chance and report confidence intervals that respect shared source tables, which still place discrimination near chance; (iv) clarify two diagnostics: shuffling cell layout while leaving row IDs recoverable, and keeping only a logically sufficient content subset; and (v) fix the answerable target’s position in the QA context while varying only nearby vs.\ unrelated distractors. Together, these support reading matched fine-tuning as an in-distribution probe rather than transferable learnability, and keep the main gap reading intact.

2. Research positioning. A central discussion point is how this work relates to open-domain table QA retrieval (e.g., OTT-QA / NQ-Tables). Sharing a ranking interface does not make the evaluation questions the same. Open-domain settings mainly ask whether an answerable table can be found among many dissimilar candidates. Our study focuses on a closed inventory of near-duplicate tables: once candidates already look alike in meaning and structure, can retrieval still select the one source that can actually answer? Open-domain retrieval also exhibits near-neighbor confusion; near-duplicate candidates make neighborhood hit and uniquely answerable selection easier to report separately. Real-TCR provides a complementary check in a real enterprise setting, using naturally partitioned tables (by time or organization) rather than synthetic siblings. It is not the same constructed near-duplicate protocol as TCR-Bench, but it still shows a separation between high neighborhood hit and weak within-group answerability selection. Related document/page work shows a similar relevance vs.\ sufficiency tension outside tables; the revised manuscript states this framing in the Introduction, Related Work, and Conclusion.

3. RAG-style pipeline checks. Beyond plain dense retrieval, on the general side the paper already included retrieve-then-rerank with explicit answerability scoring, and the revision adds its QA results; we further add query rewriting before retrieval and a corrective second pass on low-confidence first-pass cases. On the table-oriented side, a cascaded setup was already present and now has QA, and we add another multi-step cascade. These are supporting mitigation checks: making answerability more explicit can ease the gap relative to dense-only retrieval. The main contribution remains controlled diagnosis and quantification of this selection failure, not a claim to solve general RAG answerability.

We thank all reviewers for their time and constructive comments. We hope the Area Chair will consider the full author responses, the additional experiments, and the revised manuscript together when forming the final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.