- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper investigates the problem of record formation in Prompt-to-Data systems, where information extracted from documents may be correct but assigned to the wrong record. The authors propose RecordBench, a benchmark specifically designed to evaluate a system’s ability to correctly match extracted data to the corresponding records. Additionally, the authors introduce RGA, which uses a pre-trained tabular model and reference data to calculate matching scores, and then performs record assembly using a global matching algorithm. Experiments show that RGA outperforms some existing LLMs and data processing methods.
Critical strengths
-
The research question is fairly well-defined and has practical significance. Existing Prompt-to-Data systems typically focus on whether information has been extracted correctly, but correct extraction does not necessarily mean that the records match correctly. This paper examines these two issues separately, and I believe this approach has value.
-
The benchmark is well design. The authors constructed two test setups: one ensures that the extracted values are completely correct, specifically to test matching capabilities; the other uses results generated by real-world extraction processes, which more closely resemble practical applications. It can help people locate the errors. and the experiments are comprehensive. The authors compared various LLMs, traditional machine learning methods, and table-based models, and also conducted ablation experiments and statistical analyses. These experiments help elucidate the roles of different components within RGA.
Critical weaknesses
- The proposed method RGA primarily uses the existing tabular foundation model LimiX-2 to calculate matching scores, and then employs an existing global matching algorithm to perform record linkage. Although this combination is reasonable, I believe the technical novelty is rather limited. The authors need to explain more clearly what exactly RGA brings to the table compared to existing record linkage or probabilistic matching methods.
- Although RGA outperforms methods such as Qwen by a considerable margin, compared to TabPFN-3.5, it achieved improvements of only 1.42 and 0.57 percentage points in the Controlled and Extracted settings, respectively, and neither of these improvements is statistically significant. Furthermore, in the Text2DB experiment, RGA’s results are nearly identical to those of its Marginal Density version. Therefore, I believe the current experiments are insufficient to demonstrate that the full RGA method has a clear advantage over existing strong baselines.
Given the strengths and the weaknesses, I feel the score should be 2.5 to me, but I personally like the task and the benchmark they proposed, so I choose to give 3 despite the weaknesses of the method itself.
Questions for authors
see weaknesses.
2 comments
Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to address my concerns. While some concerns have been addressed, others remain unresolved. Therefore, I will maintain my original rating of 3.
Thank you for recognizing the value of the task and benchmark. We have revised the contribution statements and added targeted comparisons with both strong marginal-density baselines.
Q1. Contribution beyond an existing tabular model and global matching
Thank you for highlighting this ambiguity. Our presentation did not clearly enough separate the proposed record-formation approach from its established implementation ingredients. We have clarified this distinction in §1, §3.1 and §5. RGA builds forward and reverse conditional scoring on top of an existing tabular model. Given complete reference records and query attributes, it evaluates candidate target values sequentially in both forward and reverse target orders, then averages the two directional joint compatibilities and takes the negative logarithm to obtain the matching cost. Each step conditions on the preceding target values within the same candidate. In multi-target settings, this allows RGA to use dependencies between targets to distinguish candidates whose values are individually plausible but jointly inconsistent. Relative to marginal scoring that evaluates each target independently, this adds a consistency assessment of the complete candidate value group, which then informs global one-to-one matching. This joint-scoring mechanism is RGA's methodological contribution beyond the existing tabular model and global matching solver.
Q2. Added evidence for when joint scoring helps
To identify conditions in which RGA provides additional gains, we added target-count and matching-size experiments comparing RGA directly with LimiX-2 marginal and TabPFN-3.5 marginal. All three methods use identical inputs and assignment constraints within each condition.