acceptodds
Under review as a conference paper at ICLR 2027

Right Values, Wrong Records: Benchmarking and Improving Record Formation in Prompt-to-Data Systems

Abstract

Prompt-to-data systems can recover correct values but assign them to wrong records without shared source identifiers. End-to-end evaluation combines this decision with extraction and structural generation. RecordBench isolates record formation using controlled source-table candidates and document extractions: 2,382 episodes from 807 tables, including explicit-identity controls. Reference-Guided Assembly (RGA) uses a frozen tabular foundation model to infer candidate compatibility from reference records in context, then forms records by constrained global assignment, preserving observed values under complete or partial matching. Component-macro target-cell accuracy reaches 76.81% on controlled and 68.94% on extracted candidates; controlled RGA exceeds Qwen3.5-27B's executed tools by 18.23 percentage points. Same-backbone scoring ablations and separate assignment interventions establish the roles of reference-conditioned predictive distributions, cross-target conditioning and global competition in RGA.

Primary area:
datasets and benchmarks
Keywords:
Prompt-to-Data, Structured Data Extraction, Tabular Foundation Models, Benchmarking

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DEcfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the problem of record formation in Prompt-to-Data systems, where information extracted from documents may be correct but assigned to the wrong record. The authors propose RecordBench, a benchmark specifically designed to evaluate a system’s ability to correctly match extracted data to the corresponding records. Additionally, the authors introduce RGA, which uses a pre-trained tabular model and reference data to calculate matching scores, and then performs record assembly using a global matching algorithm. Experiments show that RGA outperforms some existing LLMs and data processing methods.

Critical strengths
  1. The research question is fairly well-defined and has practical significance. Existing Prompt-to-Data systems typically focus on whether information has been extracted correctly, but correct extraction does not necessarily mean that the records match correctly. This paper examines these two issues separately, and I believe this approach has value.

  2. The benchmark is well design. The authors constructed two test setups: one ensures that the extracted values are completely correct, specifically to test matching capabilities; the other uses results generated by real-world extraction processes, which more closely resemble practical applications. It can help people locate the errors. and the experiments are comprehensive. The authors compared various LLMs, traditional machine learning methods, and table-based models, and also conducted ablation experiments and statistical analyses. These experiments help elucidate the roles of different components within RGA.

Critical weaknesses
  1. The proposed method RGA primarily uses the existing tabular foundation model LimiX-2 to calculate matching scores, and then employs an existing global matching algorithm to perform record linkage. Although this combination is reasonable, I believe the technical novelty is rather limited. The authors need to explain more clearly what exactly RGA brings to the table compared to existing record linkage or probabilistic matching methods.
  1. Although RGA outperforms methods such as Qwen by a considerable margin, compared to TabPFN-3.5, it achieved improvements of only 1.42 and 0.57 percentage points in the Controlled and Extracted settings, respectively, and neither of these improvements is statistically significant. Furthermore, in the Text2DB experiment, RGA’s results are nearly identical to those of its Marginal Density version. Therefore, I believe the current experiments are insufficient to demonstrate that the full RGA method has a clear advantage over existing strong baselines.

Given the strengths and the weaknesses, I feel the score should be 2.5 to me, but I personally like the task and the benchmark they proposed, so I choose to give 3 despite the weaknesses of the method itself.

Questions for authors

see weaknesses.

2 comments

Thank you for recognizing the value of the task and benchmark. We have revised the contribution statements and added targeted comparisons with both strong marginal-density baselines.

Q1. Contribution beyond an existing tabular model and global matching

Thank you for highlighting this ambiguity. Our presentation did not clearly enough separate the proposed record-formation approach from its established implementation ingredients. We have clarified this distinction in §1, §3.1 and §5. RGA builds forward and reverse conditional scoring on top of an existing tabular model. Given complete reference records and query attributes, it evaluates candidate target values sequentially in both forward and reverse target orders, then averages the two directional joint compatibilities and takes the negative logarithm to obtain the matching cost. Each step conditions on the preceding target values within the same candidate. In multi-target settings, this allows RGA to use dependencies between targets to distinguish candidates whose values are individually plausible but jointly inconsistent. Relative to marginal scoring that evaluates each target independently, this adds a consistency assessment of the complete candidate value group, which then informs global one-to-one matching. This joint-scoring mechanism is RGA's methodological contribution beyond the existing tabular model and global matching solver.

Q2. Added evidence for when joint scoring helps

To identify conditions in which RGA provides additional gains, we added target-count and matching-size experiments comparing RGA directly with LimiX-2 marginal and TabPFN-3.5 marginal. All three methods use identical inputs and assignment constraints within each condition.

Number of targetsRGALimiX-2 marginalTabPFN-3.5 marginal
167.19%67.19%65.23%
281.25%76.76%76.56%
490.04%86.33%84.47%
This diagnostic uses 32 source groups, 128 references and eight queries/candidates per task. With two targets, the gains are 4.49 pp [0.78, 8.98] over LimiX marginal and 4.69 pp [0.39, 9.38] over TabPFN marginal. With four targets, the respective gains are 3.71 pp [0.00, 7.81] and 5.57 pp [0.78, 10.64]. The results and fixed-first-target control are in §4.6, Table 6; Appendix S5.1, Table S10.
For larger matching problems, we hold 128 references fixed and increase both queries and candidates from 8 to 32 to 128, scoring the same eight focal queries throughout. At 128 candidates, RGA obtains 53.13%, versus 48.83% for LimiX marginal and 45.31% for TabPFN marginal. The paired gains are 4.30 pp [0.78, 8.59] and 7.81 pp [0.00, 16.02]. §4.7 and Appendix S6, Tables S12–S13, report the complete candidate-count grid.
Text2DB. Of the 23 test tasks, 21 have one numeric target, where joint scoring reduces to marginal scoring. Each of the other two tasks has one query record and one candidate group in the saved inventories, leaving no choice between competing candidates; the two scorers also produce identical saved matching/rejection outcomes. The scores therefore coincide before rounding. We added this explanation in §4.3 beside Table 4, with inventory details in Appendix S9, Table S18.

Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to address my concerns. While some concerns have been addressed, others remain unresolved. Therefore, I will maintain my original rating of 3.

Reviewer_Y9hYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the problem of record formation, i.e., correctly associating values across structured records when shared identifier keys are unavailable. This is a challenging problem with connections to database joins, entity resolution, record linkage, and matching under unknown correspondences.

The authors introduce RecordBench, a benchmark designed to isolate record formation from upstream information extraction. The proposed method, Reference Guided Assembly (RGA), assumes access to a set of reference records containing the underlying relationships between attributes. It uses off-the-shelf pretrained tabular foundation models (TFMs) off the shelf to estimate the compatibility between candidate records and anchors. Rather than generating new values, the method scores candidate assignments using reference conditioned predictive distributions (like marginal density and conditional joint density) and subsequently performs global assignment matching to obtain the final record associations.

Critical strengths

The paper’s experimental rigor, extensive baselines and same-backbone ablations is a strength. This provides good evidence that tabular FMs outperformed off-the-shelf LLMs (like Qwen) on this specific task even with various settings to allow them statistical tool usage.

Furthermore, RecordBench comes through as a useful controlled evaluation framework for testing record matching independently of end-to-end methods that extract documents and populate databases, specially in the setup that the authors' setup.

The authors also transparently acknowledge in L265-266 transparently about TabPFN-marginal being similar in performance to their RGA which is appreciated.

Critical weaknesses
  1. RGA combines a frozen tabular FM with conditional joint density (forward reverse factorizations) and an off the shelf assignment solver. While a combination of tricks could still be an important insight for the community, the results when compare to TabPFN and even LimiX-2 marginal show modest improvements.

  2. TabPFN outperforms Qwen but Qwen was given specific statistical tools. It would be a better claim to say tabular FMs outperform LLMs in general, IF Qwen was given open ended code interpreter type tools to execute code. Inherently, while TabPFN is made for tabular tasks, open-ended LLMs are more likely to do well by getting the flexibility to reason and code on the fly.

  3. RecordBench assumes the existence of reference and query records from the same table. Currently, the benchmark only focusses on the numeric targets with fixed candidate bundles (8 anchors, 8 candidates, 2 targets). While record matching is directionally a practical problem, this current setup does not seem representative of broader data integration workloads. It is fair to isolate the problem for focussed analysis and progress compared to end-to-end document to database setups, but the range seems restrictive to be a useful resource driving impact in the community. Mixed-types, missingness in references and candidate shortlisting problems would have made this problem more nuanced and reusability of the benchmark better.

Questions for authors

Impactful for assessment:

  1. Is it possible to give the LLM baseline a pathway with a more open-ended code interpreter and some domain knowledge of the challenges of record matching? That would help in assessing the LLM competitiveness a bit more.
2 comments

Thank you for suggesting an open code interpreter with record-matching guidance. We have implemented and evaluated this pathway, and added its protocol and results to Appendix S8.1, Table S17. The open agent uses the same Qwen3.5-27B model as our original fixed-tool baseline. In a sandbox, it follows a Python/ReAct loop: write code, execute it, observe the output, revise the analysis and submit the final mapping. Each task allows at most eight model responses, including the final submission. The original baseline used one fixed-tool interaction round: invoking predefined Ridge, kNN or CatBoost tools and using the returned cost matrices to produce the final matching. We also supplied the requested domain knowledge in the prompt: candidate IDs and list order do not reveal the correct pairing; observed values must be preserved; assignments must respect one-to-one use; complete reference rows are learning examples; validation must use references only; dependencies between targets may help distinguish candidates; and SciPy provides a minimum-cost assignment solver.

MethodControlled
Qwen3.5-27B, fixed tools, matched seed58.36%
Qwen3.5-27B, open Python/ReAct52.14%
RGA76.81%
On Controlled, RGA exceeds the open agent by 24.67 percentage points. On Controlled, open execution versus the matched fixed-tool pathway gives −6.22 pp, with a paired 95% source-group bootstrap interval of [−10.00, −2.77].
One possible explanation is that the open workflow places greater demands on the base model's planning, coding and evaluation abilities. The fixed-tool workflow supplies ready-to-use modeling procedures, whereas the open workflow requires the same Qwen3.5-27B to construct and debug the analysis itself. Under the current response budget, these additional demands may prevent greater freedom from translating into better matching results. Full evaluation protocols and results are provided in Appendix S8.1, Table S17.
Reviewer_Y9hYOpenReview ↗

Response to rebuttal

Thank you for adding the additional experiment. I am not sure why the open Python/ReAct agent could not atleast be prompted in a way (or even with prompt optimization methods like GEPA) that it mimics the closed tools. Something there seems off, but it may be (as the authors say) to do with the current response budget and how much exploration the open ended code interpreter is demanding with its current prompting.

Given that other aspects of the weaknesses are also there beyond code interpreter, I will preserve my original rating.

Reviewer_o9PWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies "record formation": after facts are extracted from documents that share no common key (e.g. a facility report and a separate measurement report), which value bundle belongs to which entity? It proposes RecordBench, built from TabLib tables (99.85% from GitHub): each table is split into an "anchor" report and a shuffled "target bundle" report rendered with text templates, with 128 other rows of the same table given as reference records. There are 768 controlled episodes (128 tables, source values kept), 1,358 episodes using a shared DocETL extraction (679 tables), and 256 controls where explicit identifiers are present. Standard episodes are 8 anchors x 8 bundles with two numeric targets. The method, Reference-Guided Assembly (RGA), uses a frozen tabular foundation model (LimiX-2) conditioned on the reference rows to score how likely each bundle is for each anchor (averaging forward and reverse factorizations over the two targets), then solves a linear assignment. RGA reaches 76.81% target-cell accuracy on controlled and 68.94% on extracted candidates, vs 58.58% for Qwen3.5-27B with executed tools, 64.53% for Pair-CatBoost, and 75.39% for TabPFN-3.5 marginal density. Ablations cover mean vs distributional scoring, cross-target conditioning, global vs row-wise assignment and shuffled references; a small Text2DB study tests partial assignment.

Critical strengths
  1. Isolating record formation from extraction is a useful idea, and the benchmark is built with real care: disjoint source components across splits, fixed manifests, random-assignment baselines and gold-informed ceilings per setting (Table S3), failures kept in the denominator, and component-level paired bootstrap intervals. The identity controls (96% accuracy for LLMs when keys exist) make clear exactly which situation the benchmark targets.

  2. The ablations are clean and convincing about where the signal comes from. Shuffling bundles across reference anchors drops accuracy from 77.4% to 22.3% (near random) while reordering whole rows keeps 77.0%, so the method really uses attribute co-occurrence in the references. Global assignment beats row-wise selection by 5-6 points, and distributional scoring beats point-prediction scoring by 6-8 points with the backbone fixed.

Critical weaknesses
  1. The task setting is artificial and its link to real prompt-to-data use is not shown. Every episode assumes the system has 128 reference rows from the very same source table, with the same columns and distribution as the queries, and that the only missing link is the row pairing. "Documents" are template renderings of GitHub tables (field-ledger / attribute-sentence templates), not real reports, and semantic domains were not annotated. In practice the motivating case (facility reports without shared IDs) rarely comes with a large clean table of correctly linked rows from the same source. The authors acknowledge this is "a same-source association setting, not an independently sourced transfer test" (S1.2), but then the claim that RGA "improves prompt-to-data systems" is not supported. The only end-to-end database test (Text2DB, Table 4) is tiny (22 test components), absolute F1 is 18-29%, and all methods are almost identical (RGA = its marginal variant exactly; differences of ~0.1 to TabPFN/CatBoost), so it shows no practical benefit.

  2. The method's contribution over a simple strong baseline is small and not significant. "Use a tabular foundation model's predictive likelihood as the cost, then run Hungarian assignment" is already what the TabPFN-3.5 marginal-density baseline does, and RGA's gain over it is 1.42 [-0.07, 2.87] controlled and 0.57 [-0.19, 1.32] extracted, which the paper itself says is not significant. The parts unique to RGA (bidirectional cross-target factorization) add 0.8 points over its own marginal variant. So most of the gain comes from using a strong TFM, not from the proposed method. Meanwhile, the large headline gaps (+18 to +52 points) are against LLM pathways that are asked to infer numeric statistical relations across 128 rows from text in one shot, which is a poor fit for LLMs by design; the 18.23-point claim in the abstract compares against this weak family rather than the strongest baseline. Finally, the paper is very hard to read: dense shorthand (L/K/H/S7, anchors, bundles, components, cohorts), many key definitions only in a long appendix, and some sentences that are almost telegraphic.

Questions for authors

Questions that could change my assessment:

  1. Can you show a real (non-template) setting where reports lack shared identifiers but a clean set of correctly linked reference rows from the same source exists? How does RGA behave when references come from a different source or time period (distribution shift), which you list as untested?
  2. What does RGA add beyond "TabPFN-3.5 marginal density + Hungarian"? Is there any setting (more targets, more candidates, fewer references) where the gap becomes significant?
  3. Was the LLM "executed tools" pathway allowed to fit any model it wanted (e.g. call TabPFN or CatBoost)? If an agent can write that code, the comparison measures tool choice, not a capability gap.
  4. For Text2DB, why are RGA and its marginal variant identical to 4 decimal places in every cell, and why do all methods differ by ~0.1? Does record formation matter at all in this cohort, or is the score limited by extraction?
  5. TabLib tables are scraped from GitHub, and S1.4 says pretraining overlap was not checked. Could LimiX-2 / TabPFN have seen these tables (or their row pairings) during training?

Clarifications: please define L/K/H/S7, anchors, bundles and components in the main text; report how many episodes contain equal-valued bundles where value credit is given without the right identity; Table 4 reports 4 decimal places, which suggests more precision than 22 components support.

1 comment

Thank you for the questions. We clarified the writing and added experiments; citations refer to the revised paper.

Q1. Real reports and reference shift

We clarified the reporting scenario in S11, Table S22: NF Energy's real, non-template disclosures have unreliable shared customer labels and correctly linked historical records from the same company. S10, Tables S20–S21 report cross-source/time cases. Methods share inputs and constraints; query entities are excluded from references. Cell accuracy (%) averages ten sampling seeds.

Queries ← referencesRGALimiX-2 marginalTabPFN-3.5 marginal
Australia ← US + India56.2546.2543.75
India ← Australia + US48.7535.0038.75
US ← Australia + India43.7537.5037.50
Universities 2015 ← 201445.0037.5045.00
The WRI power-plant cases use 128 references; the university case uses a Text2DB source database and 64 references.

Q2. Contribution and conditions for joint gains

We clarified the scoring contribution in §1, §3.1 and §5. Both score candidates before global assignment; marginal scoring evaluates targets separately. RGA uses LimiX-2 to evaluate candidate values in forward/reverse order, conditioning each target on preceding values in the same candidate. It averages the two joint compatibilities, then takes the negative logarithm. This adds cross-target consistency to record matching, distinguishing individually plausible but jointly inconsistent values. The backbone and solver are existing components. The target-count diagnostic fixes 128 references and eight queries/candidates across 32 source groups. Cell accuracies (%) are:

TargetsRGALimiX-2 marginalTabPFN-3.5 marginal
167.1967.1965.23
281.2576.7676.56
490.0486.3384.47
With two targets, paired gains are 4.49 pp [0.78, 8.98] over LimiX marginal and 4.69 pp [0.39, 9.38] over TabPFN marginal (95% intervals). §4.6, Table 6; S5.1, Table S10 include a fixed-first-target control.
On 16 source groups, we increase queries/candidates from 8 to 32 to 128, fixing 128 references and scoring the same eight focal queries. At 128 candidates, RGA scores 53.13%, versus 48.83%/45.31% for LimiX/TabPFN marginal. Gains are 4.30 pp [0.78, 8.59] / 7.81 pp [0.00, 16.02] (§4.7; S6, Tables S12–S13).

Q3. LLM tool freedom

We clarified the original interface. One fixed-tool round covered: choose Ridge/kNN/CatBoost, inspect reference-only validation, then select, combine or replace the cost matrix. Model fitting was restricted to those tools. We added an open Python/ReAct agent using the same Qwen3.5-27B: write code, execute, inspect and revise. The budget is eight model responses including final submission, with 4,096 output tokens per response. It scores 52.14%/48.61% on Controlled/Extracted, versus RGA's 76.81%/68.94%. Matched-seed fixed-tool 27B scores 58.36% on Controlled. Autonomous planning, coding and debugging may tax the same base model more than fixed tools within this budget. S8.1, Table S17 provide the protocol, guidance, outcomes and timing.

Q4. Text2DB equality and extraction

The saved results coincide before rounding. Of 23 test tasks, 21 have one target, where joint reduces to marginal. Each remaining task has one query and one candidate; both scorers produce identical saved matching/rejection outcomes. Even the three-extraction union has multiple queries and candidates in only 3/23 tasks. These structures restrict candidate competition, while database-update F1 also depends on extracted-fact coverage and partial assignment. Details: §4.3, Table 4; S9, Table S18.

Q5. Pretraining overlap

The LimiX-2 report and TabPFN-3.5 model card describe exclusively synthetic pretraining. Based on these descriptions, we believe RecordBench does not overlap with these models' pretraining data in the way raised here.

Q6. Terminology, equal values and precision

We revised the terminology: L/K/H/S7 mean Controlled/Identity controls/Extracted/64-source-group extension; anchors are query attributes; bundles are candidate value groups; components are source groups used for splitting, aggregation and bootstrap; cohort denotes an evaluation set or row group. Equal target values receive correct-value credit regardless of candidate source identity: exchanging identical values leaves the reconstructed table unchanged. Candidate indices constrain one-to-one use. S1.5 specifies scoring; S1.7 reports the episode counts. Table 4 now uses two decimal places. Its identical scores reflect the task structure and matching/rejection outcomes explained in Q4, rather than rounding.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.