acceptodds
Under review as a conference paper at ICLR 2027

From Knowledge Gaps to Better Generation: Auditing and Completing World Knowledge in Text-to-Image Training Data

Abstract

Text-to-image (T2I) models achieve high visual quality but remain inconsistent in depicting concepts grounded in world knowledge, particularly those sparsely represented in training data. Because datasets are typically organized by source, concept coverage gaps are difficult to identify and address. We present CGTR, a framework that maps large scale image–text data to a hierarchical world knowledge graph for coverage analysis and targeted training-data refinement. Auditing 370M annotation records from publicly available T2I data reveals missing and underrepresented concepts across semantic levels. These statistics guide the construction of KnowGapBench, which covers 3,286 underrepresented concepts and evaluates concept identity separately from compositional compliance. CGTR independently audits each target training dataset to identify supplementation targets. World knowledge and diverse scene graph templates guide caption construction, while curated reference images guide appearance during image synthesis. Caption consistency and image–text alignment checks filter the resulting training pairs. The hierarchy organizes original and supplementary data into semantic groups, and concept-aware sampling regulates their exposure during training. We apply CGTR to Fine-T2I and fine-tune UniWorld-V1 with the refined data under its original generative objective. The overall Knowledge score on KnowGapBench increases from 15.3 to 22.5, while performance on familiar concepts outside the supplementation target set is preserved.

Primary area:
datasets and benchmarks
Keywords:
Text-to-image generation, World knowledge, Knowledge graphs, Data curation, Evaluation of generative models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Ldx4OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

CGTR aligns image-text data to a concept hierarchy built from Wikipedia and Wikidata. It estimates coverage and flags concepts that are missing or underrepresented. Using an audit of around 370 million annotations, the authors construct KnowGapBench, which covers 3,286 sparse concepts; identity and compositional understanding get scored separately. To boost Fine-T2I, they generate extra training pairs targeted at dataset-specific gaps, drawing on scene graphs, reference appearances, and filtering rules. They then resample data with concept-aware weighting and fine-tune UniWorld-V1. On KnowGapBench, its knowledge score rises from 15.3 to 22.5, while performance on CommonConceptBench holds steady without degradation. Thirteen existing T2I models are also evaluated on this benchmark.

Critical strengths

Coverage here is actionable, not just for analysis. These coverage numbers directly decide which concepts get augmented data and adjust the sampling weights. That makes this work more practical than many papers that merely demonstrate poor performance on rare concepts.

KnowGapBench shares the same measurement framework used to set training targets. Identity assessments can rely on references, while compositional evaluation remains decoupled. The UniWorld result, paired with the check showing no collapse on unrelated concepts, gives solid grounds to trust their approach.

Critical weaknesses

“Uncovered in annotations” does not equal a true lack of visual supervision. Aliasing errors, misassigned category links, or text-only mentions can create false gaps. On the flip side, a handful of nearly identical images might register as covered yet contribute almost no useful learning signal. Without human auditing to measure the precision and recall of these detected gaps, the headline coverage numbers are less robust than they appear.

The training evidence also feels lightweight relative to the scope of their claim. They only test one backbone, swap in roughly 5% of the data, and run no ablations isolating sampling weighting alone, synthetic data alone, or filtering. It is impossible to tell which component delivered that +7.2 gain on the Knowledge metric.

Questions for authors

We ask for four additional results and clarifications. First, for a random sample of 200 concepts, report human agreement against the automatically assigned Uncovered and Underrepresented labels. Second, using the same UniWorld setup, provide KnowGapBench and CommonConceptBench scores for three separate variants: concept-aware sampling alone, synthetic augmented data alone, and the full CGTR pipeline. Third, describe the safeguards used to prevent synthetic images from closely replicating the reference appearances in KnowGapBench, such as perceptual hashing or retrieval-based filtering. Fourth, frame the related-work discussion to distinguish this data-gap completion approach from methods that use fine-grained feedback on the reward side; Unicorn, SRUM and UniRL are good points of comparison.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Ldx4

We thank the reviewer for constructive feedback and encouraging comments on our work. Below, we address the questions and suggestions raised in the review.

CGTR provides data infrastructure for large scale T2I training. It audits concept coverage, supplements missing visual supervision, and adjusts sampling to improve basic knowledge before post training.

Q1. Reliability of Coverage Gaps

Our coverage labels measure support in the audited annotation pool, rather than proving that visual evidence is absent from every image. We use hierarchical concept extraction with structured outputs and allow multiple concepts per image. KNN and BM25 link concepts to the knowledge graph, and a VLM checks uncertain matches. We manually examined 200 low frequency training samples to assess extraction and linking:

StageOverall P/RSupport=0 P/RSupport=1–10 P/R
Extraction95.4/94.396.2/95.494.6/93.2
Linking98.1/94.097.4/94.498.8/93.7

We also sampled 100 concepts, 50 labeled Uncovered and 50 Underrepresented, and checked caption retrieval results manually:

LabelConceptsConfirmed
Uncovered5049 (98%)
Underrepresented5045 (90%)

Many retrieved captions contain related terms or mention the target only as a secondary subject. Such matches do not necessarily provide useful visual supervision. The two audits assess different issues: the 200 examples test extraction and linking, while the 100 concepts test gap labels. Together, they support our concept coverage and support estimates for the audited corpus.

Q2. Training Evidence and Component Contributions

We use about 6M Fine-T2I samples to study data refinement at a scale relevant to T2I pretraining. The original Baseline and CGTR process the same number of training samples, but use global batch sizes of 64 and 256. A full training run costs about 627 GPU-hours, so repeating full runs with matched batch sizes is costly. We instead ran four controlled experiments at batch size 64, with matched optimization settings and about one quarter of the original training volume per run.

Table. Controlled ablations on KnowGapBench and CommonConceptBench.

SettingBatch sizeKnowGapBenchCommonConceptBench
Baseline6414.762.3
Concept-aware sampling6415.162.1
Synthetic data6416.864.2
Full CGTR6418.963.6

With matched settings, sampling alone raises KnowGapBench Knowledge by 0.4 points, synthesis alone by 2.1, and full CGTR by 4.2. Sampling has limited effect when long tail concepts have few examples, and cannot supply missing visual supervision. Synthetic data provides this supervision, while concept aware sampling increases exposure to these concepts. On CommonConceptBench, the four settings remain close (62.1 to 64.2). Full CGTR scores 63.6 against the baseline's 62.3, showing that performance on familiar concepts is preserved while KnowGapBench improves. In the original full runs, KnowGapBench improves from 15.3 to 22.5 (+7.2), and CommonConceptBench from 62.1 to 65.9 (+3.8).

Q3. Similarity to Benchmark References

We already applied hash deduplication during reference collection and selection. Synthesis reference images were collected from different sources than KnowGapBench evaluation references. Our scene graph prompts vary auxiliary objects, scenes, and styles while keeping the target concept fixed.

These measures reduce near duplicate risk. We will add a cross-set pHash check between synthesis references, final synthetic images, and KnowGapBench references, with manual review of potential matches, and report the results in the appendix.

Q4. Related Work

We will expand the Related Work section in the main paper. UniCorn uses self generated supervision through Proposer, Solver, and Judge roles. SRUM uses global and local reward feedback from a unified model's understanding component. UniRL combines generated data with SFT and reinforcement learning to improve generation.

CGTR instead focuses on training data. It identifies missing and underrepresented concepts, guides targeted supplementation, and adjusts sampling during training. When visual evidence for a concept is scarce, feedback alone may not provide sufficient supervision for its appearance. CGTR is therefore complementary to these methods.

Thanks for fast reply, I recommand you to show these changes and additional discussions in paper using different color. It will be clearer for AC and us to check!

I think knowledge is an important topic in UMMs, if you authors can give us revisied manuscript with exact improvements, I tend to accept this paper.

We sincerely thank the reviewer for the thoughtful questions and constructive feedback on our work. We have carefully addressed the concerns raised and provided corresponding explanations and additional clarifications in our response. We have also incorporated the relevant revisions and supplementary details into the revised manuscript. For ease of review, the corresponding changes are highlighted in different colors to help identify the revisions and clarify the points raised in the original review. We hope that our responses and revisions adequately address your concerns and provide further clarity on our work. We would greatly appreciate any further feedback.

Reviewer_GqENOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Coverage-Guided Training Refinement (CGTR), a framework for improving the generation of underrepresented world-knowledge concepts in text-to-image models. The authors first construct a hierarchical knowledge graph using Wikipedia and Wikidata, then map approximately 370 million image-text records to canonical concepts to analyze their coverage and identify knowledge gaps. Based on this analysis, they introduce KnowGapBench, a benchmark comprising 3,286 concepts and 3,698 prompts across four evaluation tasks. To address the identified gaps, CGTR generates supplementary training data using knowledge-guided scene graphs and reference images, followed by caption consistency checks, image-text alignment filtering, and category-aware resampling. The authors evaluate 13 text-to-image models on KnowGapBench and further assess the effectiveness of CGTR by fine-tuning UniWorld-V1 on the refined Fine-T2I dataset. The results show that CGTR improves the overall Knowledge score from 15.3 to 22.5 on KnowGapBench, while also improving performance on a separate benchmark of familiar concepts.

Critical strengths
  1. Meaningful problem formulation. The paper addresses an important challenge in text-to-image generation: producing accurate representations of fine-grained world-knowledge concepts. By mapping large-scale training data to a hierarchical knowledge graph, the authors provide a systematic approach to identifying underrepresented concepts and analyzing knowledge gaps.

  2. Well-structured evaluation framework. KnowGapBench evaluates concept accuracy and compositional ability separately, making it easier to identify different types of generation errors. Its broad concept coverage and evaluation of 13 text-to-image models further demonstrate the benchmark's practical value.

Critical weaknesses

1.The baseline and CGTR use different global batch sizes (64 vs. 256), resulting in substantially different numbers of optimizer updates (96,521 vs. 24,131). Although both settings process the same number of training samples, these differences may affect the optimization process and introduce a potential confounding factor. A controlled comparison under identical training configurations would help clarify how much of the reported improvement can be attributed to the proposed data refinement strategy.

  1. CGTR combines several components, including coverage-guided concept selection, synthetic data generation, reference-image guidance, and category-aware resampling. However, the current experiments do not clearly isolate their individual contributions. In particular, comparing coverage-guided supplementation with random concept selection under the same synthetic data budget would provide stronger evidence for the effectiveness of the coverage-guided strategy.

  2. The coverage analysis relies on automatically generated visual annotations and concept linking. Although the authors employ several filtering and verification steps, the accuracy of the resulting coverage estimates is not directly established through human-verified annotation and linking measurements. Rare concepts may be missed or mapped incorrectly, potentially affecting gap identification. A targeted analysis of annotation precision and recall would help assess the reliability of the identified coverage gaps.

  3. The authors compare several automatic evaluators against human judgments on 200 samples, which provides useful initial validation. However, given the diversity of KnowGapBench, it remains unclear whether the reported improvements are consistent across different evaluation methods and semantic categories. Additional human evaluation or cross-evaluator analysis of the baseline and CGTR results would strengthen confidence in the reported gains.

Questions for authors
  1. Training fairness: Could the authors provide a controlled comparison where the baseline and CGTR use identical batch sizes and optimization settings? Do the reported improvements remain consistent under these conditions?

  2. Could the authors further clarify the individual contributions of synthetic data supplementation and category-aware resampling? In particular, how does coverage-guided concept selection compare with random concept selection under the same synthetic data budget?

  3. Have the authors evaluated the precision and recall of the concept annotation and linking pipeline, particularly for rare concepts? How sensitive are the identified coverage gaps to potential annotation or linking errors?

  4. Could the authors verify whether the improvements from CGTR remain consistent under human evaluation or alternative automatic evaluators? It would also be helpful to report confidence intervals for the main performance gains.

2 comments
AuthorsOpenReview ↗

Response to Reviewer GqEN

We thank the reviewer for constructive feedback and comments on our work. Below, we address the questions and suggestions raised in the review.

Our work builds data infrastructure for million-scale generative model pretraining, filling concept gaps to improve basic knowledge before post-training.

Q1. Training Fairness

We use 6M-scale SFT to approximate learning in T2I pretraining. The original Baseline and CGTR process the same total number of samples, which is our main basis for comparing data refinement, but use global batch sizes of 64 and 256. We expect limited batch-size effects at this scale. A full run costs 627 GPU-hours, making full reruns costly.

We therefore ran four controlled experiments at the same global batch size of 64 and with matched optimization settings. Each used about one quarter of the original training data budget. Table 1 shows that full CGTR improves Overall Knowledge from 14.7 to 18.9 (+4.2 points). This gain suggests batch size is not the source of improvement. Detailed configurations will be presented in the appendix.

Q2. Component Contributions

Table 1. Ablations under the same global batch size.

SettingGlobal batch sizeOverall
Baseline6414.7
CGTR-concept-aware sampling6415.1
CGTR-synth data6416.8
CGTR-full6418.9

The components serve different roles. Long-tail concepts occupy only a small fraction of large training corpora. Resampling has limited benefits when their support is very sparse, and it cannot provide supervision for concepts absent from the data. Synthetic supplementation adds such visual supervision, while concept-aware sampling increases the exposure of rare and newly added examples. Accordingly, sampling alone offers a small gain (+0.4), synthetic data gives +2.1, and their combination reaches +4.2. These results support their complementary roles.

Q3. Concept Extraction and Linking Reliability

For coverage auditing, we use the language model's world knowledge, an upper-level hierarchy, and structured JSON output. It first selects a broad category, then identifies specific concepts, and allows multiple concepts per image to reduce omissions. Linking combines KNN embedding retrieval and BM25 matching over names and aliases, with VLM verification of uncertain matches. We manually assessed 200 random low-frequency examples to test extraction and linking.

Table 2. Human evaluation of concept extraction and linking.

StageMetricOverallSupport=0Support=1–10
ExtractionPrecision95.40%96.20%94.60%
Recall94.30%95.40%93.20%
LinkingPrecision98.10%97.40%98.80%
Recall94.00%94.40%93.70%

We also sampled 100 gap concepts, 50 from each category, retrieved their matches from the original corpus captions, and manually checked the evidence.

Table 3. Manual audit of gap concepts.

Pipeline labelSampled conceptsConfirmed conceptsConfirmation rate
Uncovered504998.00%
Underrepresented504590.00%

Many caption matches refer to related terms rather than the target concept, or mention it without making it the main visual subject. Such matches are not necessarily valid visual supervision. The audit confirms 49/50 Uncovered and 45/50 Underrepresented labels (98% and 90%). Together with the extraction and linking results, this supports the reliability of our coverage estimates and concept support statistics in the audited corpus.

Q4. Robustness Across Evaluators

We evaluated the same full-training Baseline and CGTR outputs with three additional evaluators: GPT-5.6-sol, Gemini 3.1 Pro, and GLM-5.3-Flash. GPT-5.6-sol had the strongest human alignment in our prior validation.

Table 4. Overall Knowledge across evaluators.

SettingOriginal evaluatorGPT-5.6-solGemini 3.1 ProGLM-5.3-Flash
Baseline15.312.614.211.3
CGTR22.520.921.919.8
Improvement7.28.37.78.5

All four evaluators consistently show CGTR gains of 7.2–8.5 points. Despite score differences, gains remain stable across evaluators, rather than reflecting one judge's preferences. Improved samples largely overlap, with limited differences in ambiguous style cases. This consistency also supports KnowGapBench's structured evaluation protocol, which assesses concept realization through Knowledge scores and uses dependency constraints for Atomic judgments. The complete evaluation results and metrics will be presented and discussed in the appendix.

We sincerely thank the reviewer for the thoughtful questions and constructive feedback on our work. We have carefully addressed the concerns raised and provided corresponding explanations and additional clarifications in our response. We have also incorporated the relevant revisions and supplementary details into the revised manuscript. For ease of review, the corresponding changes are highlighted in different colors to help identify the revisions and clarify the points raised in the original review. We hope that our responses and revisions adequately address your concerns and provide further clarity on our work. We would greatly appreciate any further feedback.

Reviewer_B6X1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper constructs a hierarchy of 250k+ concept to audit 370M existing annotated image-text records to investigate training data of text-to-image models. They propose a framework to support the learning of concepts that are not well represented in the training sets and finally they introduce a benchmark to evaluate the correctedness of the concepts in different t2i generative models.

Critical strengths

The paper tackles an important research question and provides three-fold contribution to investigate the effect and even complete "world knowledge" in text-to-image training data. The experiment are exhaustive and consider 13 t2i models, highlighting different behaviors across them.

Critical weaknesses

I find the paper generally hard to follow, as the writing is quite dense. Despite this, both section 2 and in particular table 1 don't clarify how this paper contributes to the literature differently or "better" than previous work. Maybe the columns "source", "selection basis" and "score" could become a different table than "cat.", "concepts" "joint ref." and "freq." where the goal of comparing to existing literature is more clear.

Questions for authors

The paper does not address limitations nor future work directions. It would be insightful to understand what the authors think that their concept hierarchy, framework and benchmark might have missed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.