acceptodds
Withdrawn from ICLR 2027

LayoutLattice: Pairwise Attention Biases for Contextual Layout Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We present a simple, yet powerful mechanism to inject pairwise context into layout generation models via attention biases. We target relation-conditioned layout generation (the structure-preserving layout-variation setting), which produces alternative arrangements of page elements while preserving given pairwise relations such as which elements overlap and their reading order. Recent approaches based on transformers and continuous diffusion have advanced the field, yet they struggle to capture the pairwise relationships between page elements. Rather than relying solely on per-element features, our approach augments attention with an N x N bias matrix that injects pairwise priors (overlap/IoU and, optionally, reading order) computed from a reference layout and supplied as a conditioning signal, where N is the number of page elements. The mechanism is drop-in: binary masks and continuous priors are stacked as channels and applied as additive attention biases to a subset of attention heads, while the remaining heads stay unmasked. Across two regimes (max 25 and max 50 page items), our approach improves both realism and structure, e.g., reducing FID from  11.6 to  2.7 in the 50-item regime while increasing MaxIoU. Crucially, supplying the same priors as plain per-token inputs instead degrades the model, showing that the gains stem from the interface rather than mere access to the priors. We further demonstrate generating variations in different canvas sizes by using a relative-area (RA) representation of canvas and page items. Our results suggest that the N x N attention map is the natural representation for pairwise relational structure: a robust, reusable, parameter-free interface for structure-aware layout generation.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
attention, deep learning, nxn matrix, bias

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_JquWOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes LayoutLattice, which injects pairwise structural priors (IoU and reading order) into Transformer attention through head-specific additive biases. Combined with an overlap-consistency loss and a relative-area representation, the method achieves substantial improvements on RICO and PubLayNet, particularly for layouts containing up to 50 elements.

Critical strengths

1.Simple and efficient design. The proposed attention-bias mechanism is easy to implement, introduces no additional trainable parameters, and can be applied to different backbones.

2.Strong empirical results. The method achieves substantial improvements in FID and Max. IoU across multiple layout generation tasks.

Critical weaknesses

W1. Unfair comparison due to additional ground-truth conditions. The proposed method receives pairwise IoU and reading-order information derived from the ground-truth test layout, whereas conventional baselines do not. Although this is valid for relation-conditioned generation, the reported improvements cannot be directly attributed to the proposed mechanism. Moreover, using the target layout itself as the reference does not adequately evaluate genuine layout variation.

W2. Insufficient comparison with existing methods. Layout generation has been extensively studied, including recent LLM-based approaches such as LayoutPrompter, LayoutNUWA, and PosterLLaMA. These directions are insufficiently discussed. More importantly, the only same-condition baseline flattens pairwise relations into token features, which is a relatively weak way to represent relational information. Stronger relation-aware attention, LLM-based structured generation, or constraint-optimization baselines would make the comparison more convincing.

W3. Limited methodological novelty and insufficient ablations. Additive attention biases and structural attention masks are well-established techniques. The paper does not sufficiently disentangle the effects of hard binary masking, continuous IoU biases, and overlap-consistency supervision.

Questions for authors

See weakness.

Reviewer_F5NQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LayoutLattice for relation-conditioned layout generation. Pairwise IoU and optional reading-order relations extracted from a reference layout are injected as fixed additive biases into selected attention heads, while other heads remain unrestricted. The method combines this interface with an overlap-consistency loss and a relative-area representation. Experiments evaluate layout quality on RICO and PubLayNet, including layouts with up to 50 elements, and explore transfer to another layout backbone and human pose prediction.

Critical strengths
  1. Preserving intentional overlaps is a practically relevant objective for UI and document layout variation.
  2. The attention-bias interface is simple and introduces no additional trainable parameters.
  3. Experiments cover two layout datasets, two backbones, and denser layouts with up to 50 elements.
Critical weaknesses
  1. The evaluation does not adequately separate the proposed interface from access to additional conditioning information. Most baselines do not receive the reference-derived pairwise relations available to LayoutLattice. The same-condition flatten controls perform poorly, but this establishes superiority over the tested concatenation scheme rather than over competitive relational encodings. A competitive relation-conditioned baseline is needed to substantiate the central representation claim.

  2. The contribution of continuous IoU biases is not isolated. A small positive additive bias does not inherently imply negligible attention, contrary to the explanation in §3.3; zero-IoU pairs are explicitly blocked only in the IoU-biased heads. Without comparing binary overlap masks against continuous IoU biases under the same overlap-consistency loss, it remains unclear whether the gains arise from relational connectivity, graded values, or supervision.

  3. The metrics do not directly establish structure-preserving variation. FID and Max. IoU measure distributional quality and geometric similarity, but do not directly quantify pairwise overlap preservation, reading-order violations, or conditional diversity. Consequently, the central claim of generating varied layouts while preserving relational structure remains incompletely evaluated.

  4. The relative-area formulation contains unresolved inconsistencies. Replacing width with square-root area is a reparameterization, not a reduction in degrees of freedom. Moreover, the page token’s area coordinate of 1 appears inconsistent with the stated normalization and width-decoding rule for non-square canvases, unless a separate convention is intended for the page token. These issues require clarification to establish the correctness and reproducibility of the aspect-ratio formulation.

Questions for authors
  1. Can you compare against a competitive relation-aware encoding under identical pairwise conditions, beyond the existing flatten controls?
  2. How does a binary overlap mask perform relative to continuous IoU biases when the loss and remaining settings are fixed?
  3. Can you report relation-preservation errors, reading-order violations, and diversity across multiple samples from the same reference?
  4. How is area normalized for ordinary elements versus the page token, and in what sense does the representation reduce degrees of freedom?
Reviewer_DjYPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper focuses on the relation-conditioned layout generation task. Based on the existing LACE (Chen et al., 2024) model, a parameter-free representation of pairwise structure in the attention map is used to inject pairwise and global priors into a selected subset of attention heads. A relative-area representation stabilizes geometry and supports page-size variation. The same mechanism is further applied to another task, i.e., human pose prediction, to verify its cross-domain usefulness.

Critical strengths

The idea of injecting pairwise priors with contextual attention masks is interesting and reasonable. The experiments demonstrate its effectiveness as a domain-agnostic mechanism.

Critical weaknesses
  1. The problem formulation is questionable. Standard conditional layout generation tasks typically use element classes, sizes, or partial elements as inputs to generate plausible layouts, which aligns with real-world designer workflows. However, this paper extracts pairwise relations as additional inputs, making it impossible to conduct fair comparisons against existing baselines. Furthermore, it remains unclear how to adopt such a setup in practical design applications.
  2. The related work discussion fails to reflect the current state of the field. All references cited in the paper were published on or before 2024. A comprehensive discussion comparing the proposed approach with recent layout generation frameworks is necessary.
  3. The technical contribution is relatively limited. The proposed method can be considered an incremental work of LACE (Chen et al., 2024) without significant improvement.
  4. The experimental evaluation is not thorough. Only two metrics are used for quantitative evaluation, completely omitting standard structural metrics used in existing layout works. In addition, recent layout generation methods and LLM/LVM-based generative models should also be included in the experiments.
Questions for authors
  1. Are there any references or evidence showing that existing layout generation works only evaluated layouts with at most 25 page items?
  2. How should the subset of masked heads and unconstrained free heads be selected in practice? What is the exact head allocation ratio used in current experiments, and how sensitive is model performance by varying this ratio?
  3. The current evaluated relations only contain reading order and overlapping (IoU) constraints. How does the framework handle other essential constraints like alignment and symmetry?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.