acceptodds
Under review as a conference paper at ICLR 2027

MapForge: A Benchmark for Virtual Knowledge Graph Construction with Federation, Transformation and Documentation

Abstract

Virtual knowledge graphs (VKGs) unify access to relational data through an ontology, while keeping the data in the underlying sources. In practice, generating mappings for VKGs may require combining data from multiple databases, transforming source values, or referring to domain documentation. However, current benchmarks do not adequately evaluate these requirements. To fill this gap, we present MAPFORGE, a benchmark for VKG construction with three features: (1) Federation combining data from multiple databases; (2) Transformation converting source values to match target property requirements; (3) Documentation providing domain knowledge for mapping construction. To evaluate these capabilities, MAPFORGE provides 51 mapping generation tasks across 84 databases and 9,780 queries to assess the quality of generated mappings. Our evaluation of six baselines across four open-weight large language models (LLMs) shows that even the best-performing baseline achieves an F1 score of only 61.1%, with further performance drops on the transformation and federation subsets. Interestingly, we find that providing documentation does not always improve performance. These findings highlight the challenges of VKG mapping generation in more realistic settings and provide directions for future research.

Primary area:
datasets and benchmarks
Keywords:
Virtual Knowledge Graphs, Ontology-Based Data Access, Relational-to-Ontology Mapping Generation, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hraEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MapForge, a benchmark for generating virtual knowledge graph (VKG) mappings from relational databases, a target ontology, and optional domain documentation. The benchmark is organized around three practical requirements: federation across multiple databases, transformation of source values, and the use of domain documentation. It comprises 51 tasks over 84 databases, drawn from text-to-SQL benchmarks, RODI scenarios, and public VKGs, together with 9,780 evaluation queries that are generated automatically from the gold VKGs and cover every populated class and property. Following RODI, a generated mapping is evaluated by comparing its query answers with those of the gold mapping using multiset F1.

The authors evaluate six mapping-generation systems, five of which are LLM-based and are run with four open-weight LLMs. Even the best configuration reaches only 61.1 F1, and performance drops further on queries that require value transformations and, for most systems, on queries that span multiple sources. The experiments also show that domain documentation does not consistently improve mapping quality, as its effect depends on both the system and the LLM. An additional study examines how mapping quality changes when only part of the target ontology is visible during generation.

Critical strengths
  1. A practically motivated benchmark for an under-evaluated setting. Generating deployable VKG mappings often requires combining several databases, computing target values, and following domain conventions, which goes well beyond the schema matching emphasized by existing benchmarks. MapForge brings these requirements together in a single execution-based testbed built from public VKGs, RODI scenarios, and newly annotated tasks, and it evaluates direct-mapping, schema-matching, and end-to-end generation approaches across four LLMs. The resulting comparisons already offer useful reference points, such as the contrast between LLM4VKG, which remains comparatively stable across LLMs, and the vanilla prompting baseline, whose performance depends strongly on the LLM. This is the main source of the paper's value: with a validated evaluation, the benchmark could become a reusable resource for studying LLMs in data integration.

  2. Careful scoring and transparent reporting. The evaluation compares the answer multisets of deployed mappings, keeps deployment failures, query errors, and timeouts in the averages as zero scores, and reports deployment success separately. Several analyses are designed to avoid obvious confounds, for example by comparing federated and non-federated queries within the same tasks and by separating deployment failures from accuracy changes in the documentation analysis. The gold mappings are validated through expert review and automated checks, and the appendices discuss limitations openly. Together with the supplied tasks, evaluator, and run artifacts, this transparency made it possible to check many of the paper's claims in detail, and it is the main reason I regard the problems below as fixable rather than fatal.

Critical weaknesses
  1. Insufficient evidence that the query suites detect incorrect mappings. Reliable evaluation is the benchmark's central contribution, yet the query suites are supported only by a coverage guarantee and by a check that the gold mappings score perfectly, which establish coverage and consistency rather than sensitivity to errors. As described in Appendix A.2, the generator omits member enumeration for classes without a key, so the data properties of such classes are checked only through value lists and aggregate counts, and values attached to the wrong instances can go undetected. In a query-level check on the supplied trains task, all 47 queries still receive perfect scores when the length values of two cars are swapped, and even when each attribute of the 63 cars, including the train a car belongs to, is shuffled independently so that no car keeps its original combination of values. Keyless classes occur in five of the seven supplied tasks, and the blind spot also extends to the transformation subset: in the jukebox task, the computed billing-place strings of sales are checked only as a collection, so correctly formatted strings attached to the wrong sales would still pass. The supplementary code contains a mutation routine that deletes a single mapping assertion, but the paper reports no results from it, and deletions are precisely the errors that coverage-based queries already catch. Without sensitivity tests on realistic errors, such as permuted values and incorrect joins, it is unclear to what extent the reported F1 scores reflect mapping correctness.

  2. The documentation analysis does not isolate the contribution of domain knowledge. The paper presents domain documentation as the source of domain conventions and transformation rules. In the five supplied tasks with newly authored ontologies, however, the target ontology, which every system receives with or without the document, already describes much of this knowledge in its annotations. In the two transformation-focused tasks, every transformation rule is written in the ontology together with a worked example, and the documents explicitly refer the reader to the ontology for these rules. The ontologies also contain notes on the benchmark design, such as which errors a task is meant to expose and how they are detected, and among the saved prompts only the vanilla prompting baseline receives these notes verbatim. The with/without-document comparison therefore measures the marginal effect of a partly redundant document rather than the value of domain knowledge, so the conclusion that domain knowledge alone does not ensure better mappings is not supported as stated. The per-origin results in Table 8 are consistent with this concern: the 12.4-point documentation gain of Vanilla LLM on Qwen-3.8-27B comes entirely from the RODI tasks, whose ontologies were not authored for the benchmark, while its performance slightly decreases on the text-to-SQL tasks. Since documentation is one of the three core features of the benchmark, a controlled comparison using ontologies that neither restate the documents nor contain design notes is needed to support this finding.

Questions for authors

Q1. Sensitivity of the query suites. How many classes, properties, and queries across the 51 tasks concern classes without a key? More importantly, could the authors evaluate the suites against controlled errors, such as permuted attribute values, incorrect joins, and wrong IRI templates, particularly for keyless classes and transformed properties? It would also help to know whether adding queries that combine several values of the same instance, or link values to a related keyed entity, changes the rankings.

Q2. Placement of domain knowledge. For each task, are the conventions and transformation rules stated in the document, in the ontology annotations, or in both? Could the authors repeat the documentation comparison, at least for the strongest systems, using ontologies that do not restate the documents and contain no design notes? Please also clarify whether the vanilla prompt used without documentation still tells the model that a document is provided, as it does in the supplied example.

Q3. Foreign-key information in the baseline inputs. In the supplied bookshelf run, the prompts given to Vanilla LLM and Magneto-LLM describe the foreign keys of the first catalogue as pointing to tables in the second catalogue, whereas the database declares local references. Under the advertised references, 7,314 of the 11,432 book-author links would match no book. The LAMP prompt shows the correct references, yet the final LAMP and Magneto-LLM mappings link the first catalogue's book-author rows to authors from the second catalogue, which suggests a shared extraction or assembly step. In the saved vanilla run, the error happens to have no visible effect. Which tasks and systems are affected, for instance other federated tasks with identically named tables, and do corrected runs change the main results?

Q4. Attribution of the direct-mapping improvement. The paper attributes the improvement of DM+LLM over DM+LogMap to replacing LogMap with an LLM, but the two pipelines start from different bootstrapped mappings: DM+LogMap creates separate entities for each source database, whereas DM+LLM merges entities that share an identifier across databases. On the bookshelf task, this difference alone determines whether duplicated authors are merged, which is exactly what one of the task's queries checks, independently of the aligner. Could the authors compare the two aligners on the same bootstrapped mapping, or revise this interpretation?

Q5. Repair procedure and release. Please describe the repair applied to Vanilla LLM mappings that Ontop rejects, including whether it is automated, what information and budget it uses, and how often it changes deployment success. The reproducibility statement promises the dataset and code upon acceptance; will this include the baseline implementations and per-task outputs? The current supplement contains only seven tasks and selected run artifacts, which limits independent verification.

Reviewer_1ffXOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MapForge, a benchmark for virtual knowledge graph (VKG) mapping generation that extends conventional relational-to-ontology mapping evaluation along three dimensions: multi-source federation, source-value transformation, and the use of domain documentation. MapForge comprises 51 mapping-generation tasks over 84 relational databases, accompanied by 9,780 SPARQL evaluation queries. Its construction combines existing public VKGs, RODI scenarios, and databases from text-to-SQL benchmarks. The authors additionally propose an automated query-generation procedure based on the populated classes and properties of a gold VKG, allowing generated mappings to be evaluated by comparing their query answers against reference results. The paper evaluates six mapping-generation baselines, including schema-matching and end-to-end generation approaches, using four open-weight language models. The strongest configuration achieves an overall F1 score of 61.1. The experiments reveal lower performance on transformation-related queries, generally reduced performance on the federation subset, and inconsistent effects from supplying domain documentation. Overall, MapForge addresses a relevant gap in VKG benchmarking and provides a useful evaluation resource. However, concerns remain regarding the completeness of its gold-derived query evaluation and the extent to which its feature-specific subsets establish general conclusions about federation, transformation, and documentation-aware mapping generation.

Critical strengths

S1. A relevant and practically motivated extension of existing VKG mapping benchmarks. The paper identifies a meaningful gap between conventional schema-to-ontology mapping benchmarks and practical VKG construction, where mappings may require integrating multiple databases, transforming source values, and interpreting domain-specific documentation. Unlike approaches that evaluate only schema correspondences or the textual form of generated mappings, MapForge evaluates executable mappings through their effect on query answering. This is particularly valuable because a syntactically plausible mapping does not necessarily produce correct RDF instances, values, or relationships. The integration of federation, transformation, and documentation into a unified benchmark therefore represents a useful contribution for studying mapping-generation systems under more demanding conditions. S2. A relatively comprehensive evaluation pipeline with useful empirical findings. MapForge contains 51 tasks, 84 relational databases, and 9,780 task-query evaluation pairs. Its query suite covers several categories, including class instances, property values, direct and multi-hop relationships, filtering, aggregation, and class hierarchies. The construction process combines expert review with automated checks. Appendix A.1 further reports validation of source constraints and the gold mappings. Importantly, the scoring procedure assigns zero scores to queries that fail or time out and to mappings rejected by Ontop, instead of excluding unsuccessful tasks from the evaluation. The experiments also compare several substantially different mapping-generation approaches across four open-weight LLMs. The reported variation across methods, transformation types, and documentation conditions provides useful evidence of current limitations and motivates further research.

Critical weaknesses

W1. The gold-derived query-generation procedure does not establish complete detection of incorrect or extraneous mappings. Evidence: Section 3.3, Appendix A.2 (Algorithm 1 and Table 5), and Appendix B.2. This is my primary concern regarding evaluation validity. In Section 3.3, the authors construct an active ontology from populated classes and properties of the gold VKG. Evaluation queries are instantiated from this active ontology, and queries returning empty gold answers are discarded. This procedure guarantees coverage of populated elements under predefined query templates, but it does not establish that the resulting query suite detects all semantically incorrect mappings. For example, suppose an ontology property is not populated in the gold VKG. If a generated mapping incorrectly introduces triples for that property, queries derived exclusively from the gold active ontology may not test it. Such assertions could remain unpenalized unless they indirectly change the answers to another evaluated query. A similar problem arises for unpopulated gold classes that a generated mapping wrongly populates. Evaluating whether required knowledge is reconstructed is not equivalent to testing whether incorrect knowledge is introduced. The validation in Appendix A.1 is valuable, but having gold mappings score 100 F1 against reference answers computed from those same mappings mainly establishes internal consistency; it does not independently show that the query suite discriminates erroneous mappings. Systematic mapping-perturbation tests covering extraneous class assertions, spurious properties, incorrect entity links, and plausible but wrong transformations would clarify this limitation. Without such analysis, F1 should be interpreted as performance on the selected positive query workload rather than a comprehensive measure of mapping correctness. W2. Federation and transformation subsets offer limited evidence for general capability-level claims; documentation effects are partially confounded by deployment failures. Evidence: Sections 4, 5.2-5.4, Appendix B.3, and Appendix B.5. Regarding federation, Appendix B.3 explicitly defines membership through gold-mapping coverage of at least two data sources and acknowledges that the label does not establish whether the generated mapping must reconcile entities across sources. Multi-source coverage is a reasonable benchmark feature, but is broader than evaluating cross-source entity reconciliation or join correctness. Thus, lower federation F1 does not by itself isolate reconciliation failures; task complexity and ontology structure may also contribute. Regarding transformation, only 260 of 9,780 evaluation queries belong to this subset, distributed across 12 tasks; 215 concern property-value retrieval or equality filtering. Appendix B.3 further shows that date extraction comprises 14 queries from one task, arithmetic or unit conversion 15 queries from three tasks, and mixed transformations only eight queries from two tasks. These are useful diagnostic workloads but do not support sweeping generalizations across SQL transformation families or application domains. Regarding documentation, Vanilla LLM with Qwen-3.8-27B improves in overall F1 from 48.7 to 61.1 when documentation is supplied, yet successfully deployed VKGs decline from 40 to 35 of 51 tasks and the executed-query rate decreases from 75.1% to 53.6% (Appendix B.5). The authors acknowledge this trade-off, which is commendable, but headline F1 improvements do not establish uniformly improved deployment reliability. Paired task-level analyses, deployment-conditioned accuracy, and explicit failure-mode decompositions should be more prominent. The representativeness of domain documents independently authored in practice also deserves clarification because Section 3.2 constructs them within the expert-supervised gold-mapping workflow. Overall, the results reveal useful difficulties on selected workloads, but stronger feature-specific controls and broader evaluation would be needed to substantiate general claims about federation, transformation, and documentation-aware mapping generation.

Questions for authors

Q1. How well does the query suite detect incorrect assertions outside the populated gold ontology? Since Algorithm 1 generates queries from gold-populated ontology elements and discards queries with empty gold answers, how does MapForge penalize a system that produces additional triples for otherwise unpopulated classes or properties? Can the authors provide controlled perturbation experiments measuring sensitivity to spurious triples, incorrect object-property links, and extraneous class assertions? This would materially affect my assessment of evaluation completeness. Q2. How many federation queries genuinely require cross-source entity reconciliation or cross-source joins? Appendix B.3 distinguishes multi-source coverage from entity reconciliation. Could the authors classify queries into independent source contributions, unions, cross-source joins, and entity-identity reconciliation, and report performance for each? Could performance be compared with matched single-source queries within the same tasks? This would clarify the actual federation capabilities tested. Q3. How representative and statistically stable are the reported transformation results? The transformation subset contains 260 queries across 12 tasks, with some operation categories appearing in only one to three tasks. Can the authors provide per-category query counts, task-level scores, and uncertainty estimates? Could additional independently constructed tasks be evaluated, particularly for numeric operations, date processing, and mixed transformations? Q4. Does documentation improve mapping correctness independently of its effect on deployment success? With Vanilla LLM and Qwen-3.8-27B, documentation improves overall F1 but reduces deployed VKGs and executed queries. Can the authors provide paired results for tasks deploying in both conditions, separately accounting for answer accuracy, mapping parse errors, deployment failure, and query execution failure? Please clarify the mapping-repair procedure and whether applicable baselines receive comparable repair opportunities. Q5. How sensitive are conclusions about domain documentation to document origin and quality? Section 3.2 indicates that domain documents are constructed under expert supervision alongside the gold ontologies and mappings. How do the authors ensure that these represent independently available realistic domain knowledge rather than benchmark-specific explanations closely aligned with gold mapping design? Would the results persist with independently authored, irrelevant, incomplete, or ambiguous documents?

Reviewer_tnL9OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a benchmark for the generation of VKG mappings that focuses on challenges that arise in federated settings or if data has to be transformed to fit the vocabulary. For that, the authors use public benchmarks like RODI, text-to-SQL benchmarks and public VKGs for constructing their benchmark. In addition, they pay a particular attention to the role of textual data documentation in the context of the mapping generation quality. To evaluate the benchmark, the authors additionally present a query generation algorithm that automatically generates SPARQL queries that cover all populated ontology class and properties based on the gold VKG. Their findings show that the four used open-weight LLMs have difficulties with mappings that involve federation or transformation but cannot show consistent improvement if domain documentation is added as additional information.

Critical strengths

In the context of real-world settings and dataspaces, the topic of automatic mapping construction itself is very timely and relevant. In addition, we still lack realistic benchmarks that cover all the complex tasks, especially in federation settings.

The methodical approach to the generation of the experiment is clear and understandable. However, in order to understand the paper in detail, it is really required to take a look at the Appendix.

Critical weaknesses
  1. Due to the fact that no state-of-the art frontier LLM was used, it is difficult to judge whether the research gap is as prominent as described. The failure of the NPD task raises concerns about the ability of the generation pipeline to handle large, realistic scenarios, i.e., it fails at the most realistic parts of the benchmark. Here, it would have been especially interesting to see if these errors still hold for SOTA LLMs. In addition, the authors don’t justify why the selected exactly those four open-weight LLMs. Why not using DeepSeek for instance?

  2. The results of the main experiments are based on single-LLM runs as far as I understood and therefore are not corrected for non-deterministic LLM variance weakening the statistic interpretation of the experiment even further. Although the authors claim that they set the temperature to 0 (in the appendix), this is usually not sufficient to get deterministic results. In addition, the authors did not explain how the other remaining parameters for the LLMs were set. For instance, it is unclear whether QWEN 3.8 was used in Instruct or in Reasoning mode? In addition, the authors did not evaluate the role of the prompt.

  3. During the generation of the gold VKGs I have two concerns: First, the public VKGs might be part of the LLMs training data and might bias the VKG generation results. Second, generating the remaining Gold VKGs with LLMs even if checked by experts might bias the results as LLM-generated VKGs are likely more similar to what other LLMs will generate compared to human generated VKGS

  4. The transformation subset is underrepresented in the benchmark even though it is prominently described. In my opinion, a realistic benchmark dataset for federated settings will definitely require a broader set of transformations. In addition, it is not explained in detail in the paper itself which transformation tasks the authors exactly examine (only Table 3 lists them and only the Appendix contains information about them). This makes the paper very hard to understand. In addition, some transformation tasks might be much easier for an LLM than others. This needs to be examined in more detail with a broader set of transformation tasks.

  5. Figures and Tables of the main experiment do not report error margins on the results. Therefore, it is difficult to judge how reliable the results are.

  6. Related work about other approaches, that already rely on textual data documentation for mapping generation, is missing

  7. Finally, the paper wants to much. It tries to make to many different contributions in a single paper. This results in the fact, that each of the contributions is only evaluated in a superficial way. The paper would benefit from the fact to focus on a single aspect and evaluate this one in detail. For instance, the paper could focus more on the fact why a documentation does not improve the mapping task or if there would be other metadata that could be used for improving the mapping (that also exist in real-world settings, like dataspaces).

Questions for authors

Why generate OnTop mappings and not R2RML as recommended by the W3C?

Did you address the above mentioned possible biases in some way?

1 comment

No rebuttal has been submitted by the authors so far. Hence, I stay with my initial assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.