acceptodds
Under review as a conference paper at ICLR 2027

JIG: JSON-to-Image Generation in Pixel and Latent Spaces

Abstract

Text conditioning is fundamental to image generation and has evolved from short alt-text to dense synthetic captions and, more recently, structured JSON, with caption lengths growing from tens to thousands of tokens. However, the large-scale JSON-annotated corpora used by recent generators remain unreleased, limiting reproducible studies of how annotation content, textual format, and generator design contribute to performance. In this paper, we present JIG, an open model family and empirical study of JSON-to-image generation in pixel and latent spaces. First, we introduce JIG-14M, a dataset of 13.7M images paired with rich structured annotations that capture global scene information, visual styles, and spatially grounded element details. Second, we investigate the influence of annotation content and textual format. Both JSON and natural-language prose realizations of the structured annotations outperform conventional dense captions, suggesting that richer annotation content is a key contributor to improved generation quality across textual formats. Third, we train and compare four JIG models spanning pixel and latent spaces using decoupled diffusion and unified Transfusion-style architectures. Under the baseline training setup, latent-space models outperform their pixel-space counterparts, while differences between architectures are comparatively small. We further analyze this gap and develop a pixel-specific training recipe that substantially narrows it in both architectures. With this improvement, pixel models approach their latent counterparts in prompt following, with the largest remaining gap in aesthetic quality. With image-generation training on JIG-14M, the resulting systems also achieve competitive performance against compact open-source generators, with strong compositional generation and prompt alignment. We will release the dataset JIG-14M and all four JIG models to support the research community.

Primary area:
datasets and benchmarks
Keywords:
structured caption, image generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_z8DUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces a dataset of 13.7M images with detailed structured annotations and compares generation performance with dense-caption baselines. They also compare pixel-space and latent-space generators and investigate training techniques to reduce their performance gap.

Critical strengths

They construct a large dataset with detailed annotations for 13.7M images, which could be useful for further research.

There are nice ablations both on the generative model side and on the text format (e.g. comparing JSON and prose while keeping the annotation content the same).

Critical weaknesses
  • Although the dataset could be useful, the method's novelty seems limited. The annotation scheme closely follows existing work, and the improvements to pixel-space generation combine known techniques. Structured captions being useful has also been studied before. Thus, the additional insights beyond existing work seem limited to me.

  • Table 1 reports low score for some aspects like object coverage. This could be concerning about the quality of the annotations. It would be useful to understand how this is affecting training and the resulting models.

Questions for authors

Aside from showing that richer annotations are helpful, which components contribute the most to the superior performance (bounding boxes, separate records for objects, ...)? Also, compared with the dense-caption version, what do the authors believe is the reason behind the boost in some aspects and not the others in Table 14?

3 comments
AuthorsOpenReview ↗

Response to Reviewer z8DU: Critical Weaknesses W1–W2

Dear Reviewer z8DU

Thank you for your careful review and for recognizing the value of our dataset and controlled studies. Below, we clarify the contributions beyond prior work and discuss how annotation quality relates to generation performance.

W1. Novelty and additional insights

The annotation scheme closely follows existing work, the improvements to pixel-space generation combine known techniques. Structured captions being useful has also been studied before. Thus, the additional insights beyond existing work seem limited.

JIG's primary contribution lies in the open data foundation and empirical findings about structured image generation. Structured JSON conditioning has been widely adopted by leading image generators, including FLUX.3, MOSS-Image, FIBO2, etc. Yet few large-scale datasets with structured image annotations are publicly available, limiting the community’s ability to conduct further research under this paradigm. JIG-14M addresses this gap with high-quality structured annotations for 13.7M images, providing a shared data foundation for open research.

Beyond data release, we also provide a systematic empirical study of structured supervision. FIBO compares long structured captions with short captions, while ContextScaling examines caption content and organization jointly. Compared with these prior studies, we further distinguish annotation content from its serialization, a distinction that a dense-versus-structured comparison alone cannot establish. We vary the format and ordering of the same finalized records while preserving their information and element associations. As shown below, the gains over dense captions persist across formats and record orders. In addition, the field-level ablations in Q1 further examine the roles of different components, including element records, bounding boxes, and regional descriptions.

SupervisionGenEval2 ↑PRISM ↑
Dense36.265.79
JSON62.986.29
Shuffled JSON63.826.16
Prose63.236.29

Regarding the pixel-space improvements, we acknowledge that they are built on established techniques. Our contribution here lies in evaluating these techniques under structured supervision using four generator baselines spanning pixel and latent representations and diffusion and Transfusion architectures. The baseline pixel–latent gap exceeds the gap between the two architectures, and pixel-adapted training narrows it in both. These results show that pixel-space training strategies developed for text-to-image generation remain effective in the JSON-conditioned setting. Together with the latent baselines, these models and their training recipes provide a common reference for future JSON-to-image research.

W2. Annotation quality and training impact

Table 1 reports low score for some aspects like object coverage. [...] It would be useful to understand how this is affecting training and the resulting models.

Thank you for highlighting the room for improvement in annotation quality, particularly object coverage. Our assessment shows that Ideogram 4-style structured annotations provide broader object coverage and more accurate descriptions than dense captions. Models trained with these annotations also yield substantially better generation performance compared with their dense-caption counterparts. We additionally provide Reve 2-style annotations with better quality, especially higher object coverage. Across these annotation variants, higher annotation quality is accompanied by stronger generation performance, supporting the value of improving structured supervision.

AnnotationObject coverage (%) ↑Description accuracy (%) ↑GenEval2 ↑PRISM ↑
Dense caption54.586.836.265.79
Ideogram 4-style JSON61.4895.9362.986.29
Reve 2-style JSON81.1596.6773.827.42
AuthorsOpenReview ↗

Response to Reviewer z8DU: Questions for Authors Q1–Q2

Dear Reviewer z8DU

Thank you for your questions about which annotation components drive the gains and why improvements vary across generation skills. We address Q1 and Q2 below through the field ablations and an analysis of the skill-specific results.

Q1. Contributions of individual components

Aside from showing that richer annotations are helpful, which components contribute the most to the superior performance (bounding boxes, separate records for objects, ...)

To identify which annotation components contribute most to the generation capability, we compare Full JSON with three field-ablation variants. Global-only removes all element records, retaining only global scene and style descriptions. w/o Box removes element coordinates. w/o Region-desc removes element descriptions while retaining boxes, types, colors, and literal text.

AnnotationGenEval2 ↑PRISM Total ↑Alignment ↑Aesthetic ↑
Full JSON62.986.297.175.42
w/o Box50.206.126.875.37
w/o Region-desc37.005.506.264.74
Global-only31.945.376.054.69

Removing regional descriptions has a larger effect than removing boxes: both PRISM alignment and aesthetics decrease, and GenEval2 performance approaches Global-only. The retained boxes, types, colors, and literal text do not substitute for detailed local descriptions. This points to element descriptions as an important source of supervision for both content fidelity and visual quality.

Removing boxes has a more selective effect: alignment decreases while aesthetics remains close to Full JSON. The GenEval2 breakdown further shows a larger change in Count, from 78.95 to 65.78, than in Position, from 81.56 to 78.88, while object and attribute scores remain comparable. Our interpretation is that explicit boxes associate element records with distinct image regions, helping the model distinguish individual instances and render the requested number of objects.

Q2. Uneven gains across GenEval2 skills

Compared with the dense-caption version, what do the authors believe is the reason behind the boost in some aspects and not the others in Table 14?

Thank you for highlighting the uneven gains across the skills in Table 14. We believe these differences reflect the additional information each skill needs from structured annotations. Dense captions already convey object identities, attributes, and actions, whereas precise layout and counting benefit more from explicit instance-level detail.

Training annotationEvaluation inputObject ↑Attribute ↑Count ↑Position ↑Verb ↑
DenseDense desc.93.7890.5856.1469.3132.87
DenseRecord desc.92.2191.2661.5181.5738.96
JSONJSON90.0591.2478.9581.5635.78
ProseProse90.1291.4378.7683.2334.58

Positioning benefits largely from richer inference descriptions. With the Dense model unchanged, providing record-based descriptions brings its Position performance close to that of JSON and prose. This suggests that clearer layout information in the input explains much of the positioning gain.

Counting, however, benefits additionally from training with structured annotations. Even with record-based inputs, the Dense model retains a substantial counting gap relative to models trained on JSON or prose. We interpret this as a benefit of learning from separate element records: each record associates an individual instance with its description and location, helping the model distinguish instances and render the requested number of objects.

I thank the authors for their comprehensive responses, especially the ablations. Although novelty remains a concern for me, the responses address my main questions and help clarify the value of the dataset for open research. I will therefore revise my score to weak acceptance.

Reviewer_GyfsOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes JIG, an open model family coupled with an emprical study focusing on the impact of the structured captions on image generation models. The proposed study investigates caption structure differences (Dense vs. JSON vs. Prose), architecture differences with these captions (DiT vs. Transfusion) and predicition targets (latent vs. pixel). Overall, the presented experiments and ablations clarify the impact of the detailed captions, and the type of details that actually help the model to improve its generation performance. The authors also propose an open dataset JIG-14M, which involves 13.7M grounded image-annotation pairs. The paper also provides qualitative and quantitative assessments on the impact of differet captions along with their coupling with different modelling choices.

Critical strengths
  • The paper provides an image-to-annotation dataset with 14M samples, which corresponds to a dataset that is not easily available and can facilitate further research, which is a meaningful contribution.
  • The paper provides an emprical study that invesitgates different modelling choices in combination with the change in caption structure. Furthermore, the impact of the details of the captions are investigated in a transparent and detailed way, which shows detailed findings. Especially the finding that providing layout elements in a precise way but not in a JSON format, see Fig. 8, is a useful.
  • The quantitative and qualitative experiments are shown to be detailed and the effectiveness of the curated dataset along with the presented findings are clear.
Critical weaknesses
  • One major concern with the proposed dataset is the diversity of the samples which may create issues in the generated samples by the models trained with the dataset. Specifically, detailed structures such as portraits, and settings that require abilities such as counting (e.g. Tabletop & Food) are not included as a majority. Furthermore it is a concern that the trained models may be skewed through the training data and improving the diversity of the dataset would be helpful.
  • While in terms of scale the dataset JIG is a valuale contribution, there are also publicly available datasets such as [1], which is composed of 380K samples for image-annotation pairs. The paper does not compare with such open datasets in terms of model performance, which can be useful for understanding if layout guided samples can learn with less amount of samples or do we need as many samples as regular text-to-image training in the annotated setup too.

[1] Zhang, Hui, et al. "Creatidesign: A unified multi-conditional diffusion transformer for creative graphic design." International Conference on Learning Representations. Vol. 2026. 2026.

Questions for authors
  • Can the authors provide generation examples from domains that the curated dataset have minority of samples? As mentioned in the weaknesses section, these can be portrait images, or samples that require counting.
  • How does the method compare with the existing datasets focused on annotations? Also considering the scale, did the authors observe any patterns in terms of utilizing different amount of training samples for the sake of generalization, as now the samples include interpreted information with layout settings?
1 comment
AuthorsOpenReview ↗

Response to Reviewer Gyfs

Dear Reviewer Gyfs,

Thank you for recognizing JIG-14M’s value for open research and the insights provided by our controlled experiments.

W1 / Q1. Dataset diversity and generation in less represented domains

One major concern with the proposed dataset is the diversity of the samples which may create issues in the generated samples by the models trained with the dataset.

Can the authors provide generation examples from the domains that the curated dataset have minority of samples? […] these can be portrait images, or samples that require counting.

Thank you for raising the concern about how dataset composition affects generation in less represented domains. Portraits and tabletop/food account for 5.17% and 7.25% of JIG-14M, respectively, as shown in the statistics in Appendix A.4, corresponding to approximately 0.71M and 0.99M images.

We have added twelve examples generated by our latent-diffusion model trained on JIG-14M to Appendix F, Figure 15 (p. 28) of the revised manuscript, covering portraits, tabletop/food scenes, and counting. The examples show detailed facial features and multi-object scenes with the requested counts, including three ears of corn, five tulips, and three magpies.

For counting, our caption study also provides quantitative evidence: JSON and Prose both improve GenEval2 counting performance over Dense variant, demonstrating the benefit of richer annotation content for this capability.

AnnotationGenEval2 Count ↑
Dense56.14
JSON78.95
Prose78.76

W2 / Q2. Comparison with existing datasets and training-data scale

The paper does not compare with such open datasets in terms of model performance.

Thank you for pointing us to CreatiDesign. It provides structured annotations for approximately 393K synthetic graphic design images. JIG-14M provides annotations for 13.7M images—approximately 35 times as many—with broader coverage spanning both photographic scenes and graphic designs.

We compare latent-diffusion models trained on CreatiDesign and an equally sized subset of JIG-14M, using the same architecture and 393K images each. Each run uses eight H200 GPUs with shared initialization and optimization settings and is trained for 30k steps. As shown below, the JIG JSON model achieves better generation performance on both GenEval2 and PRISM than the CreatiDesign JSON model.

Training dataAnnotationImagesGenEval2 ↑PRISM ↑
Small-data comparisons
CreatiDesignJSON393K17.983.93
JIG subsetDense393K14.043.43
JIG subsetJSON393K23.504.25
Full-data references
JIG-14MDense13.7M36.265.79
JIG-14MJSON13.7M62.986.29

The PRISM advantage over CreatiDesign is mainly reflected in prompt alignment (5.15 vs. 4.58), while aesthetic scores remain close (3.36 vs. 3.28). The higher GenEval2 score is also consistent with stronger compositional generation. JIG’s detailed element-level annotations explicitly associate individual elements with their descriptions, attributes, and spatial locations. This rich supervision may help the model bind attributes to the correct elements and arrange multiple elements into the requested composition.

did the authors observe any patterns in terms of utilizing different amount of training samples for the sake of generalization

To examine whether the benefits of structured supervision persist across training-data scales, we compare JSON and Dense supervision on the 393K-image subset and the full 13.7M-image corpus. JSON outperforms Dense on both benchmarks at both scales, showing that the benefit of structured supervision is already visible in the smaller-data setting. The full-data models, trained with substantially more images and updates, also achieve higher absolute scores for both annotation types.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Response to Reviewer DKMG: Critical Weaknesses W1–W2

Dear Reviewer DKMG,

Thank you for recognizing the value of our dataset and controlled empirical study, and for your constructive feedback. Below, we address your concerns about which properties of structured supervision produce the gains and the reliability of our annotation quality assessment.

W1. Which supervision properties produce the gain?

The caption experiments isolate serialization better than they isolate which supervision properties produce the gain.

We examine two complementary questions: whether the gains depend on serialization format or record order, and which information within the annotations contributes to them.

To distinguish annotation content from serialization, we compare JSON with prose and shuffled JSON derived from the same finalized records. These variants preserve field values and element associations while changing the textual format or record order.

SupervisionGenEval2 ↑PRISM ↑
Dense36.265.79
JSON62.986.29
Shuffled JSON63.826.16
Prose63.236.29

The gains over Dense persist across all three structured variants. These results show that the benefit of the richer annotation content is retained when expressed as prose or when the records are reordered.

To examine the contribution of specific information groups within a common annotation pipeline, we train and evaluate models on field subsets of the same finalized records. Global-only removes all element records, retaining global scene and style descriptions. w/o Box removes element coordinates. w/o Region-desc removes regional descriptions while retaining boxes, types, colors, and literal text.

AnnotationGenEval2 ↑PRISM Total ↑Alignment ↑Aesthetic ↑
Full JSON62.986.297.175.42
w/o Box50.206.126.875.37
w/o Region-desc37.005.506.264.74
Global-only31.945.376.054.69

Removing regional descriptions has a larger effect than removing boxes: both PRISM alignment and aesthetics decrease, and GenEval2 performance approaches Global-only. The retained boxes, types, colors, and literal text do not substitute for detailed regional descriptions. This points to regional descriptions as an important source of supervision for both content fidelity and visual quality.

Removing boxes has a more selective effect: alignment decreases while aesthetics remains close to Full JSON. The GenEval2 breakdown shows a larger change in Count, from 78.95 to 65.78, than in Position, from 81.56 to 78.88, while object and attribute scores remain comparable. Our interpretation is that explicit boxes associate element records with distinct image regions, helping the model distinguish individual instances and render the requested number of objects.

W2. Annotation quality and reliability

The annotation audit is informative but relies on an automated judge without reported human calibration.

To calibrate the automated assessment, we invited 10 human annotators to independently review a source-stratified subset of 200 images and averaged their individual results. Annotators evaluated the same description claims and box–instance associations without access to the VLM’s judgments, and independently identified objects and text for coverage.

AssessorDescription accuracy (%)Box–instance match (%)Object coverage (%)Text coverage (%)
GPT-5.6-Luna96.8698.5061.3875.90
Human annotators91.7197.9763.5968.78

The human scores support the high accuracy of recorded descriptions and their instance associations. Human and automated judgments agree in 93.25% of description assessments and 98.59% of box–instance assessments. Independently constructed object and text inventories also provide a human reference for the coverage estimates.

AuthorsOpenReview ↗

Response to Reviewer DKMG: Questions for Authors Q1–Q2

Dear Reviewer DKMG,

Thank you for your thoughtful questions and suggestions for strengthening our evaluation. Below, we address input truncation across caption formats and annotation schemas, uncertainty in the pixel–latent comparisons, computational costs, and the resources we will release for reproducibility.

Q1. Truncation by format and schema

What fraction of examples is truncated under each caption format and schema, and which information is lost?

We measured input length and content truncation under a common 2,048-token cutoff by caption format (Dense, JSON, Prose) and annotation schema (Ideogram 4-style, Reve 2-style). We applied this cutoff to inputs from paired images after the corresponding preprocessing and tokenization.

AnnotationMean tokensTruncated inputs (%)
Dense232.50
Ideogram 4-style JSON375.10.057
Ideogram 4-style Prose413.60.155
Reve 2-style JSON925.60.567

For 99.85% of Ideogram 4-style JSON/prose pairs, both formats retain all preprocessed content at the cutoff. Thus, the content matching underlying the comparison in W1 is preserved at the model input for almost all pairs. This supports interpreting their similar performance as evidence that the annotation content remains useful across serialization formats.

The longer Reve 2-style JSON inputs also retain their full content in 99.43% of cases. When truncation occurs, it removes trailing element properties.

Q2. Uncertainty, compute costs, and artifact release

quantify uncertainty for the closest pixel/latent comparisons

We evaluated the latent and full-recipe pixel models with five shared generation seeds and computed 95% confidence intervals by paired resampling of prompts and seeds. The table reports latent-minus-pixel differences.

ArchitectureGenEval2 gap [95% CI]PRISM gap [95% CI]
Diffusion1.15 [-0.64, 2.97]0.48 [0.38, 0.60]
Transfusion1.07 [-0.62, 2.75]0.45 [0.33, 0.57]

Averaged over five generation seeds, the GenEval2 gap remains about one point in both architectures, with 95% confidence intervals that include zero. Latent models retain higher PRISM scores in both architectures.

Please also report the relevant compute costs

ModelTraining cost (H200 GPU-h)Inference time (s/image)
Latent Diffusion1,23311.64
Pixel Diffusion baseline1,05111.70
Pixel Diffusion + full recipe1,19510.91
Latent Transfusion1,1403.99
Pixel Transfusion baseline7343.76
Pixel Transfusion + full recipe1,3474.89

Training costs are estimated for 150k steps. Inference latency is computed over 100 images on one H200 at 512 × 512 resolution and 100 sampling steps. It is observed that both pixel baselines have lower estimated training cost than their latent counterparts. For the full recipes used in the closest pixel–latent comparisons, the cost relationship depends on architecture: full-recipe pixel Diffusion has slightly lower estimated training cost and inference latency than latent Diffusion, whereas full-recipe pixel Transfusion has higher costs on both measures.

clarify which dataset annotations, source mappings, and training artifacts will be available for reproduction.

Upon acceptance, we will release JIG-14M with its Ideogram 4-style annotations and source-image mappings, together with checkpoints and training configurations for all four baseline generators.

Dear Reviewers,

Thank you again for your thoughtful reviews. The authors have now posted their responses, and I would greatly appreciate it if you could take a moment to read them and share your thoughts. Even a brief note on whether your concerns have been addressed would be very helpful. If anything remains unclear, please feel free to follow up with the authors directly in the discussion thread.

Best regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.