- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces MapForge, a benchmark for generating virtual knowledge graph (VKG) mappings from relational databases, a target ontology, and optional domain documentation. The benchmark is organized around three practical requirements: federation across multiple databases, transformation of source values, and the use of domain documentation. It comprises 51 tasks over 84 databases, drawn from text-to-SQL benchmarks, RODI scenarios, and public VKGs, together with 9,780 evaluation queries that are generated automatically from the gold VKGs and cover every populated class and property. Following RODI, a generated mapping is evaluated by comparing its query answers with those of the gold mapping using multiset F1.
The authors evaluate six mapping-generation systems, five of which are LLM-based and are run with four open-weight LLMs. Even the best configuration reaches only 61.1 F1, and performance drops further on queries that require value transformations and, for most systems, on queries that span multiple sources. The experiments also show that domain documentation does not consistently improve mapping quality, as its effect depends on both the system and the LLM. An additional study examines how mapping quality changes when only part of the target ontology is visible during generation.
Critical strengths
-
A practically motivated benchmark for an under-evaluated setting. Generating deployable VKG mappings often requires combining several databases, computing target values, and following domain conventions, which goes well beyond the schema matching emphasized by existing benchmarks. MapForge brings these requirements together in a single execution-based testbed built from public VKGs, RODI scenarios, and newly annotated tasks, and it evaluates direct-mapping, schema-matching, and end-to-end generation approaches across four LLMs. The resulting comparisons already offer useful reference points, such as the contrast between LLM4VKG, which remains comparatively stable across LLMs, and the vanilla prompting baseline, whose performance depends strongly on the LLM. This is the main source of the paper's value: with a validated evaluation, the benchmark could become a reusable resource for studying LLMs in data integration.
-
Careful scoring and transparent reporting. The evaluation compares the answer multisets of deployed mappings, keeps deployment failures, query errors, and timeouts in the averages as zero scores, and reports deployment success separately. Several analyses are designed to avoid obvious confounds, for example by comparing federated and non-federated queries within the same tasks and by separating deployment failures from accuracy changes in the documentation analysis. The gold mappings are validated through expert review and automated checks, and the appendices discuss limitations openly. Together with the supplied tasks, evaluator, and run artifacts, this transparency made it possible to check many of the paper's claims in detail, and it is the main reason I regard the problems below as fixable rather than fatal.
Critical weaknesses
-
Insufficient evidence that the query suites detect incorrect mappings. Reliable evaluation is the benchmark's central contribution, yet the query suites are supported only by a coverage guarantee and by a check that the gold mappings score perfectly, which establish coverage and consistency rather than sensitivity to errors. As described in Appendix A.2, the generator omits member enumeration for classes without a key, so the data properties of such classes are checked only through value lists and aggregate counts, and values attached to the wrong instances can go undetected. In a query-level check on the supplied trains task, all 47 queries still receive perfect scores when the length values of two cars are swapped, and even when each attribute of the 63 cars, including the train a car belongs to, is shuffled independently so that no car keeps its original combination of values. Keyless classes occur in five of the seven supplied tasks, and the blind spot also extends to the transformation subset: in the jukebox task, the computed billing-place strings of sales are checked only as a collection, so correctly formatted strings attached to the wrong sales would still pass. The supplementary code contains a mutation routine that deletes a single mapping assertion, but the paper reports no results from it, and deletions are precisely the errors that coverage-based queries already catch. Without sensitivity tests on realistic errors, such as permuted values and incorrect joins, it is unclear to what extent the reported F1 scores reflect mapping correctness.
-
The documentation analysis does not isolate the contribution of domain knowledge. The paper presents domain documentation as the source of domain conventions and transformation rules. In the five supplied tasks with newly authored ontologies, however, the target ontology, which every system receives with or without the document, already describes much of this knowledge in its annotations. In the two transformation-focused tasks, every transformation rule is written in the ontology together with a worked example, and the documents explicitly refer the reader to the ontology for these rules. The ontologies also contain notes on the benchmark design, such as which errors a task is meant to expose and how they are detected, and among the saved prompts only the vanilla prompting baseline receives these notes verbatim. The with/without-document comparison therefore measures the marginal effect of a partly redundant document rather than the value of domain knowledge, so the conclusion that domain knowledge alone does not ensure better mappings is not supported as stated. The per-origin results in Table 8 are consistent with this concern: the 12.4-point documentation gain of Vanilla LLM on Qwen-3.8-27B comes entirely from the RODI tasks, whose ontologies were not authored for the benchmark, while its performance slightly decreases on the text-to-SQL tasks. Since documentation is one of the three core features of the benchmark, a controlled comparison using ontologies that neither restate the documents nor contain design notes is needed to support this finding.
Questions for authors
Q1. Sensitivity of the query suites. How many classes, properties, and queries across the 51 tasks concern classes without a key? More importantly, could the authors evaluate the suites against controlled errors, such as permuted attribute values, incorrect joins, and wrong IRI templates, particularly for keyless classes and transformed properties? It would also help to know whether adding queries that combine several values of the same instance, or link values to a related keyed entity, changes the rankings.
Q2. Placement of domain knowledge. For each task, are the conventions and transformation rules stated in the document, in the ontology annotations, or in both? Could the authors repeat the documentation comparison, at least for the strongest systems, using ontologies that do not restate the documents and contain no design notes? Please also clarify whether the vanilla prompt used without documentation still tells the model that a document is provided, as it does in the supplied example.
Q3. Foreign-key information in the baseline inputs. In the supplied bookshelf run, the prompts given to Vanilla LLM and Magneto-LLM describe the foreign keys of the first catalogue as pointing to tables in the second catalogue, whereas the database declares local references. Under the advertised references, 7,314 of the 11,432 book-author links would match no book. The LAMP prompt shows the correct references, yet the final LAMP and Magneto-LLM mappings link the first catalogue's book-author rows to authors from the second catalogue, which suggests a shared extraction or assembly step. In the saved vanilla run, the error happens to have no visible effect. Which tasks and systems are affected, for instance other federated tasks with identically named tables, and do corrected runs change the main results?
Q4. Attribution of the direct-mapping improvement. The paper attributes the improvement of DM+LLM over DM+LogMap to replacing LogMap with an LLM, but the two pipelines start from different bootstrapped mappings: DM+LogMap creates separate entities for each source database, whereas DM+LLM merges entities that share an identifier across databases. On the bookshelf task, this difference alone determines whether duplicated authors are merged, which is exactly what one of the task's queries checks, independently of the aligner. Could the authors compare the two aligners on the same bootstrapped mapping, or revise this interpretation?
Q5. Repair procedure and release. Please describe the repair applied to Vanilla LLM mappings that Ontop rejects, including whether it is automated, what information and budget it uses, and how often it changes deployment success. The reproducibility statement promises the dataset and code upon acceptance; will this include the baseline implementations and per-task outputs? The current supplement contains only seven tasks and selected run artifacts, which limits independent verification.
No rebuttal has been submitted by the authors so far. Hence, I stay with my initial assessment.