MapForge: A Benchmark for Virtual Knowledge Graph Construction with Federation, Transformation and Documentation

Virtual knowledge graphs (VKGs) unify access to relational data through an ontology, while keeping the data in the underlying sources. In practice, generating mappings for VKGs may require combining data from multiple databases, transforming source values, or referring to domain documentation. However, current benchmarks do not adequately evaluate these requirements. To fill this gap, we present MAPFORGE, a benchmark for VKG construction with three features: (1) Federation combining data from multiple databases; (2) Transformation converting source values to match target property requirements; (3) Documentation providing domain knowledge for mapping construction. To evaluate these capabilities, MAPFORGE provides 51 mapping generation tasks across 84 databases and 9,780 queries to assess the quality of generated mappings. Our evaluation of six baselines across four open-weight large language models (LLMs) shows that even the best-performing baseline achieves an F1 score of only 61.1%, with further performance drops on the transformation and federation subsets. Interestingly, we find that providing documentation does not always improve performance. These findings highlight the challenges of VKG mapping generation in more realistic settings and provide directions for future research.