acceptodds
Under review as a conference paper at ICLR 2027

AnyLayer: Order-Agnostic Layer Decomposition with Heterogeneous Queries

Abstract

Recent approaches to image layer decomposition have made substantial progress. However, existing methods typically support limited query modalities or process targets independently, making it difficult to jointly handle heterogeneous queries while maintaining stable query–layer correspondence. We present AnyLayer, a unified framework that maps heterogeneous text, point, box, and mask queries to their corresponding RGBA foreground layers and a background image within a single diffusion process. First, AnyLayer provides a unified prompting paradigm for jointly processing heterogeneous queries. Second, it introduces an order-agnostic layer representation that removes ordinal foreground identities while preserving explicit query–layer correspondence. Third, Hybrid Layer Attention improves multi-layer generation efficiency by reducing redundant cross-layer computation. Experiments demonstrate effective mixed-modality control, stable performance with increasing layer cardinality, high-quality foreground recovery, occlusion-completion, and background results, and efficient training and inference.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
image layer decomposition

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bQQROpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This article proposes AnyLayer, a unified framework for multi-layer image decomposition. Unlike existing methods that mainly focus on single-target decomposition or limited query modalities, AnyLayer supports heterogeneous queries such as text, point, box, and mask queries, and generates corresponding RGBA foreground layers and a background image simultaneously in a diffusion process. In order to solve the query–layer correspondence problem in multi-layer generation, the author proposes Layer-aligned Shared Position Encoding and Prompt-conditioned Attention Routing to reduce the impact of query order. The authors also design Hybrid Layer Attention to reduce the computational overhead of multi-layer inference. The experiment was conducted on RevealLayerBench and AnyLayerBench constructed by the author, and the results showed that the method can handle different query forms and maintain good performance and efficiency in high-cardinality layer decomposition scenarios.

Critical strengths
  1. A targeted design has been proposed for the query–layer correspondence problem in multi-query and multi-layer generation. Existing methods typically focus on single-target decomposition, while AnyLayer studies how to maintain the correspondence between target queries and generated layers when multiple heterogeneous queries are input simultaneously. The combination of LSPE and PAR can alleviate the problem of target mixing caused by changes in query order, and relevant ablation experiments have also verified the effectiveness of these designs.

  2. The experimental coverage is relatively rich, considering various query forms and high-cardinality scenarios. In addition to box/mask queries, the author also evaluated text, point, and mixed queries, and tested the performance of 5-15 layers. At the same time, the paper analyzed the inference efficiency and the role of different modules, and the overall experimental design was relatively complete.

Critical weaknesses
  1. The scale and reliability of the benchmark still need further verification. AnyLayerBench was constructed by the author themselves, with a test set size of 200 samples, and occluded regions and backgrounds were completed through GPT-Image. Although the author conducted manual screening, this ground truth constructed based on generative models may affect the reliability of the evaluation. For a task that emphasizes complex occlusion restoration and generalization ability, more validation on real-world layered datasets or manually annotated layer data is still needed.
  2. Lack of comparison with the most relevant baselines makes it difficult to fully demonstrate the advantages of joint generation. The paper emphasizes better efficiency and consistency compared to target-by-target processing methods, but does not compare sequential inference with existing single-target layer decomposition methods. For example, applying methods such as RLD/RefLayer to multiple queries separately and then combining them for output has not been validated for differences in quality, background consistency, and inference time compared to AnyLayer. In addition, there is currently a lack of sufficient competitive methods for comparing text/point/mixed queries.
Questions for authors
  1. Invisible areas and backgrounds in AnyLayerBench are completed by GPT-Image. How can the author verify that this benchmark does not introduce evaluation bias related to generative models? Is there manual evaluation, real layer annotation data, or validation on other independent benchmarks? Additionally, are 200 test images sufficient to represent model performance for high complexity generation tasks?

  2. The paper proposes that joint multi-layer generation has advantages over single-target decomposition methods, but why is there no sequential inference comparison with existing single-target decomposition methods? For example, how does RLD/RefLayer compare to AnyLayer in terms of decomposition quality, background consistency, and inference efficiency when combined after multiple runs?

  3. What are the essential differences between LSPE, PAR, and HLA and existing shared position encoding, attention routing, and sparse/local attention methods? How do the contributions brought by these modules differ from the combination of existing methods? Suggest the author further discuss this in the supplementary materials.

  4. "Order-agnostic" is mainly validated through permutation experiments. Can the author further analyze whether this property has structural permutation equivariance, rather than just empirical robustness obtained through training?

  5. Modern generative image editing models can now perform image editing directly through text instructions. What is the practical necessity of explicit layer decomposition in this context? Has the author conducted experiments to verify that the layer representation generated by AnyLayer can bring practical advantages in subsequent tasks such as multi-round editing, asset reuse, and editing consistency? In addition, the current model still relies heavily on a large diffusion backbone. Does the computational cost match its actual benefits?

  6. This paper mainly reports carefully selected successful cases, lacking a systematic analysis of failure cases and performance limitations. The qualitative results in the paper and supplementary materials mainly show visually favorable cases, but provide limited information about AnyLayer failure modes. For challenging generation tasks such as multi-layer image decomposition, reporting only successful examples is not sufficient to evaluate the actual robustness and performance boundaries of the proposed method. The authors should provide more challenging cases and analyze failure scenarios, such as severe occlusion, blurred object boundaries, similar objects, or complex transparency. For these situations, the performance lower bound is more convincing. Understanding when and why the model fails is crucial for evaluating the actual reliability of AnyLayer.

Reviewer_WNQkOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes AnyLayer, a framework that can decompose an image to a background and multiple foreground layers at the same time, under heterogeneous queries (text, point, mask, and box queries). To achieve this, AnyLayer designs a unified interface for jointly processing different types of queries. It also proposes layer-aligned shared position encoding and prompt-conditioned attention routing to preserve layer-query association while eliminating the dependency on ordering. In addition, the authors propose hybrid layer attention to accelerate the inference efficiency. Evaluation shows that AnyLayer achieves competitive layer quality in multi-layer decomposition, and is robust to query permutation.

Critical strengths
  • I found the problem very interesting and practical. The simultaneous heterogeneous-query layer decomposition problem is well motivated.
  • The empirical results are promising. Specifically, AnyLayer outperforms Qwen-Image-Layered and RevealLayer in terms of decomposition quality while being more efficient when the number of foreground layers scales up.
Critical weaknesses
  • It is unclear to me how robust AnyLayer is when handling challenging scenarios, such as highly overlapping objects or ambiguous targets. Although Figure 4 presents two qualitative examples involving overlapping objects, a more systematic and quantitative evaluation of AnyLayer's performance in complex scenes would help better understand its limitations and robustness.
  • The paper lacks sufficient analysis on the RP (region predictor): how does it perform, how does it influence the final decomposition quality. I'd like to better understand how accurately RP predicts the foreground regions and how region prediction errors affect the final decomposition quality.
  • The paper does not compare AnyLayer with query-modality-specific decomposition baselines. It remains unclear whether AnyLayer achieves competitive performance against methods specifically designed for individual query modalities. I understand that directly comparable baselines may not be available, and I would appreciate clarification from the authors on this point. Otherwise, a comparison would help clarify the trade-offs between modality-specific and unified decomposition approaches.
Questions for authors

Please see Critical Weaknesses

Reviewer_PkdPOpenReview ↗
Rating
3(+1)
Confidence
1(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes AnyLayer, a unified diffusion-based framework for decomposing an RGB image into multiple RGBA foreground layers and a shared background using heterogeneous queries, including text, points, boxes, and masks. This approach differs from preivous approaches that process individual objects separately, it jointly generates multiple layers in a single diffusion process while maintaining explicit query–layer correspondence. The method introduces Point-Grounded Text RoPE for spatial grounding, Layer-aligned Shared Position Encoding (LSPE) and Prompt-conditioned Attention Routing (PAR) for order-agnostic query–layer association, and Hybrid Layer Attention (HLA) to reduce computational costs. A lightweight query-region predictor also improves efficiency for text and point queries. Experiments on RevealLayerBench and AnyLayerBench show competitive reconstruction quality, improved robustness to query permutations and increasing numbers of layers, and faster inference compared with existing methods.

Critical strengths
  • Order-agnostic query–layer association: The combination of LSPE and PAR addresses an important challenge in simultaneous query-conditioned multi-layer generation. The ablation studies demonstrate improved query–layer correspondence and robustness to query reordering.
  • Efficient joint inference: Generating multiple foreground layers and a shared background in a single diffusion process reduces repeated computation, avoids sequential error accumulation, and enables cross-layer information sharing.
  • Scalability: The query-region predictor and HLA significantly reduce computational costs, particularly when handling many foreground layers.
  • Experimental validation: The paper provides convincing ablations for its main components, along with experiments evaluating query permutations, increasing layer counts, and inference efficiency.
Critical weaknesses
  • Limited methodological novelty: Several contributions build on existing work. Heterogeneous prompting was already explored in RefLayer, while joint multi-layer generation and attention-based layer interactions have been investigated previously. The novelty mainly lies in combining these techniques for order-agnostic, query-conditioned generation.
  • Missing comparisons with closely related methods: RefLayer and MRT are excluded due to unavailable implementations. Additionally, comparison with Qwen-Image-Layered is not entirely fair, as it receives only the number of layers, whereas AnyLayer receives explicit target queries.
  • Modest reconstruction improvements: Despite the additional architectural components, foreground-quality improvements over RevealLayer are relatively modest.
  • Limited training data: The model uses approximately 180K curated and synthetic layered samples, considerably fewer than the datasets reported by RefLayer (1.11M triplets) and MRT (over 10M design samples), raising questions about generalization to diverse real-world scenes.
  • Limited robustness to severe occlusion: The authors acknowledge that performance degrades under complex occlusion, although recovering hidden object regions is a central objective of layer decomposition.
  • High absolute inference latency: Despite substantial relative speedups, inference still takes 91.9 seconds for 16 mixed-query layers on an H800 GPU, limiting its practicality for interactive image editing.
Questions for authors
  • Comparison with prior work: Can the authors provide a comparison with RefLayer using the same prompts, evaluating reconstruction quality, inference time, and scalability?
  • Novelty: Can the authors clarify the methodological novelty of LSPE and PAR relative to existing positional encoding and attention-routing techniques?
  • Occlusion robustness: How does performance vary across different occlusion levels (0–20%, 20–40%, 40–60%, and over 60%)?
  • Ambiguous queries: How does the model handle ambiguous or overlapping prompts, such as "the two people," "all flowers," or "the chair and table," given the assumption that each query identifies a distinct foreground layer?
  • Generalization: How well does the model generalize to complex real-world scenes beyond the curated and synthetic training distribution?
  • Practical efficiency: Can the authors provide inference latency for typical interactive editing scenarios involving 2–5 foreground layers?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.