acceptodds
Under review as a conference paper at ICLR 2027

LayerBench: Benchmarking Layer Decomposition Beyond Reconstruction

Abstract

Layer decomposition is fundamental to editable image manipulation, yet its evaluation remains misaligned with its purpose: a flat copy can achieve near-perfect reconstruction while being useless for editing. Existing protocols prioritize pixel fidelity, are tied to narrow distributions, and rarely validate model-based scores against human preferences. We introduce LayerBench, a comprehensive benchmark for layer decomposition that explicitly aligns evaluation with human preferences. LayerBench comprises 1.9K curated images spanning diverse domains, layer cardinalities, occlusion patterns, transparency effects, and semantic structures, together with a preference corpus of 13,979 expert pairwise comparisons used to train LayerReward. Its evaluation suite combines rule-based metrics for reconstruction, alignment, text preservation, and operational editability; model-based metrics from pretrained large vision-language models; and LayerReward, a five-dimensional evaluator that exceeds 80% overall pairwise accuracy and generalizes to out-of-distribution shadow separation. Beyond evaluation, LayerReward also serves as a reward signal for LayerCompass, a reward-guided post-training method included as part of LayerBench. Experiments show that reconstruction fidelity is necessary but insufficient for editability and human preference.

Primary area:
datasets and benchmarks
Keywords:
Layer Decomposition, Benchmarking, Human Preference Alignment

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_opQFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents LayerBench for evaluating image layer decomposition, LayerReward for scoring decompositions using expert preferences, and LayerCompass for improving a decomposition model through RL.

Critical strengths

The editing tests measure whether the layers support operations such as moving, deleting, and recoloring objects.

The expert annotations and blind human studies provide useful evidence beyond reconstruction scores.

Critical weaknesses
  1. The examples cover only simple layer structures. Figure 4 shows two inputs without GT. Figures 14 and 15 show GT stacks with only two or three layers. These examples leave complex PSD structures largely unexamined. Complete GT and predicted RGBA files are needed to inspect layers.

  2. Several dataset sources and filtering rules are missing. The paper does not specify the Online PSD filtering criteria or the exact sources of Text-rich and Editability. Photorealistic examples are foreground and background composites, and 68.6% of the benchmark has only two or three layers.

  3. The RL training data is not specified. Appendix A.8 gives hardware and training settings but does not identify the source images, dataset size, etc.

  4. The final ranking is not validated by the human study. SynLayers ranks second in the final score but fourth in the human ranking. Table 12 also shows that SynLayers ranks first when the LayerReward weight is 0.33, while LayerCompass ranks first at 0.5. The human study supports the LayerReward ranking, but it does not establish the choice of weights for the final score.

  5. The rules for Visual Plane Grouping are not clearly defined. Appendix A.3 mentions a top-down scan, but does not explain how it determines that layers belong to the same visual plane, or when a new group starts. These choices affect DTW matching and the alignment scores.

Questions for authors
  1. Can the authors provide complete GT and predicted RGBA files for complex PSDs?

  2. What are the sources and filtering rules for Online PSD, Text-rich, and Editability?

  3. What data is used for LayerCompass RL training?

  4. Why use the current score weights given the ranking differences in Tables 2, 12, and 13?

  5. How is Visual Plane Grouping implemented? For example, how layers are assigned to the same plane, and when a new group starts.

4 comments

W1 & Q1. Complex PSD examples. Due to space and layout constraints, the main paper includes only a limited number of complex decompositions with many layers. To address your concern, we now provide additional examples in the supplementary material, showing the input images, ground-truth RGBA layers, and predicted RGBA layers from all five evaluated models.

W2 & Q2. Dataset sources and filtering. We will clarify the dataset sources and filtering procedures in the revision.

  • Online PSD (500): PSD design templates collected by crawling publicly accessible design-asset repositories. We remove empty layers, group compatible elements into ordered RGBA layers, and retain designs with at least two effective layers and faithful recomposition, followed by manual screening.
  • Text-rich (200): the OCR subset combines 50 programmatically typeset posters and 50 Online PSD posters retaining their original text. The Tiny Text subset adds separately rendered small-text layers to 50 text-removed PSD designs and 50 MAGICK/Pexels composites, providing explicit text-layer references.
  • Editability (100): 80 programmatically designed backgrounds and 20 MAGICK/Pexels composites, each with a separately rendered object and title. These provide known editing targets and background references for assessing changes after editing.

For the photorealistic subset, we composite foreground objects onto photographic backgrounds to obtain reliable reference layers. We also explored object-removal models, but their recovered backgrounds could alter originally visible regions, introducing errors into reference-based evaluation. Compositing preserves known background content and makes the reference layers and resulting evaluation more reliable.

We consider the 68.6% proportion of two- or three-layer samples reasonable for four reasons. First, current models still exhibit substantial errors even on two- or three-layer decompositions, making low-layer-count cases informative for distinguishing model performance. Second, these cases can also test occlusion completion, shadow ownership, text preservation, and alignment with human preferences. Third, constructing reliable ground-truth layers for the Editability, Shadow, and Photorealistic subsets is challenging; controlled, low-layer-count settings enable reliable references while broadening evaluation scope. Finally, they help keep evaluation across the dataset and multiple models computationally practical. As the first comprehensive layer decomposition benchmark, LayerBench does not exhaust all layer structures. We will acknowledge this limitation and identify broader coverage as future work.

W3 & Q3. RL training data. We recognize that Appendix A.8 does not clearly identify the post-training images. LayerCompass is post-trained using 1600 RGB images from GenPoster100K, a separate dataset with no image overlap with either the LayerReward preference dataset or the LayerBench evaluation set.

W4 & Q4. Ranking differences and weights. Thank you for highlighting this distinction. Table 13 validates LayerReward as a measure of overall human preference; Final additionally considers fidelity, reference alignment, and editing capability. These objectives are complementary, so Final is a default summary rather than a requirement to reproduce the preference-only ranking. We report component scores and radar plots to make these tradeoffs explicit. The default w_LR = 0.5 gives LayerReward the largest individual weight while retaining the other assessments.

SynLayers illustrates why the rankings can differ. Its finer-grained outputs can increase the burden of inspecting and coordinating layers, while providing more precise control over individual elements. To examine overall versus task-specific preferences, we additionally conduct a user study with five raters on 20 source images under five conditions. Without a specified task, participants judge all recovered layers; for each specified task, they judge only the relevant layers. We use dense ranking, allowing ties without gaps. Average Rank is the equally weighted mean across the five conditions.

ModelNone (No specified task) ↓Background Extraction & Replacement ↓Foreground Object Movement ↓All-Text Removal ↓Local Text Editing ↓Average Rank ↓
LayerCompass1.551.101.301.301.301.31
Stable-Layers1.751.301.351.401.701.50
Qwen-Image-Layered2.301.201.401.401.601.58
SynLayers2.351.401.301.101.301.49
LayerD2.852.002.002.001.802.13

The no-task ordering matches Table 13. However, SynLayers achieves the best mean rank for all-text removal, illustrating that lower preference for the complete decomposition does not imply lower suitability for every task. The ordering averaged across the five conditions matches Final.

The weighting conclusions are also stable around our default: the complete Final ranking remains unchanged throughout w_LR ∈ [0.40, 0.60] under the weighting protocol in Appendix A.11. SynLayers ranks first at 0.33 because that setting gives greater weight to its stronger Rule and VLM editing scores. This reflects a different balance of objectives rather than a contradiction with the human-preference results. More broadly, Final is a convenient default summary whose weights express the relative emphasis placed on Rule, VLM Editability, and LayerReward; it is not the sole basis for interpreting LayerBench results. Our default emphasizes human preference while also accounting for fidelity, reference alignment, and editing capability. We report the total scores for all three components together with their detailed breakdowns and radar plots, and encourage users with different task priorities to rely on the dimensions most relevant to their intended applications.

W5 & Q5. Visual Plane Grouping. We group predicted and reference stacks independently by alpha-mask overlap, after matching canvas sizes and resizing to a 512 px long side. Masks include all pixels with alpha > 0, including the background.

The topmost unassigned layer seeds a plane. We scan remaining layers top-down, comparing each mask M with the current plane’s mask union U. A layer is deferred only if |M ∩ U| ≥ 4 pixels and |M ∩ U| / min(|M|, |U|) > 0.001; otherwise it joins the plane and updates U. After a full pass, the topmost remaining layer seeds the next plane. Thus, disjoint layers can share a plane even if an intervening layer is deferred.

Members are composited in their original order, and DTW aligns the plane sequences top-down. Separately, up to three greedy adjacent-layer merges are tried on predictions, regrouping and realigning candidates and retaining a merge only if it reduces mean DTW path cost. Thank you for your feedback. We will clarify this in the revision.

Dear Reviewer opQF,

Thank you again for your time and constructive feedback. As the discussion period draws to a close, we would appreciate hearing whether our responses and additional results have addressed your concerns. If any questions remain, we would be happy to clarify them during the remaining time. Thank you for your consideration and effort in reviewing our work.

Thank you for the rebuttal and clarifications, which have addressed some of my concerns. However, I still have reservations about the limited coverage of complex layer structures in the benchmark and the validity of the proposed evaluation metrics. Therefore, I will maintain my original score.

Reviewer_xvUpOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

LayerBench evaluates image-to-layer decomposition beyond pixel reconstruction. It contains 1,902 images across Crello, online PSDs, composited photorealistic scenes, text-rich images, shadows, and editability probes. Evaluation combines rule-based reconstruction, DTW layer alignment, text preservation, and editing metrics; a Qwen3-VL-8B editability judge; and LayerReward, a preference-trained evaluator using the same backbone family. The preference corpus contains 13,979 expert-annotated pairs, split into 12,979 training and 1,000 image-disjoint test pairs; LayerReward reports 84.24% overall test accuracy.

LayerReward also guides DiffusionNFT post-training of Qwen-Image-Layered, producing LayerCompass. Its final benchmark score rises from 0.592 to 0.703. A blind A/B study reports a 76.4% preference share with ties counted as half votes, and a separate five-model human study ranks LayerCompass first.

Critical strengths

(1) A clear diagnosis with concrete evaluation safeguards. A flat copy can reconstruct the input while providing no useful decomposition. The paper explicitly assigns zero reconstruction and alignment scores to a single predicted layer that nearly reconstructs the input (mean pixel error at most 0.01) when the reference is multilayered (Sec. 3.2; Appendix A.3). Applying actual scaling, translation/rotation, deletion, and recoloring also makes editability operational. These are useful steps toward evaluating what a decomposition enables, rather than reconstruction alone.

(2) Shadow ownership is a useful problem to expose. The shadow subset makes explicit how assigning a cast shadow affects object movement or removal (Appendix A.1). LayerReward's 71.4% preference for the shadow-aware pipeline on 500 pairs without shadow-domain training is an interesting generalization probe (Table 6). This supports the value of the problem framing, although that figure is a reward-model win rate, not independently quantified agreement with human shadow judgments.

Critical weaknesses

(1) The composite score is insufficiently validated, and part of the post-training evaluation reuses the optimized signal. LayerCompass is trained against LayerReward, which contributes 50% of the final score; the paper acknowledges this (p. 7, lines 348-350). The generic VLM judge shares its Qwen3-VL-8B backbone, so correlated evaluation preferences remain possible. Appendix A.12 explains the deployment rationale and the different prompting versus preference-training procedures; sharing a backbone does not by itself prove biased results. The rule-based improvement and blind human studies are meaningful additional evidence for improvement over the base model, but they do not independently validate the complete aggregate.

(2) In particular, Table 13's human ordering is LayerCompass > Stable-Layers > Qwen-Image-Layered > SynLayers > LayerD, whereas Table 2's final score places SynLayers second. LayerReward alone matches the human ordering. SynLayers also leads the rule aggregate, 0.760 versus 0.734. The authors give a design rationale for balancing complementary objectives, and Appendix A.11 shows LayerCompass remains first for LayerReward weights 0.4-0.6; nevertheless, SynLayers becomes first at 0.33. Different objectives can legitimately yield different rankings, but the paper has not sufficiently validated what the combined ranking represents or empirically justified the chosen weights as a measure of overall quality. This is my main reason for a negative assessment.

Component validity and transfer across benchmark conventions remain insufficiently established. Text-metric agreement with human majority votes is 54.7% over 86 pairs, with a 95% CI of [44.2%, 65.1%] (Table 4). This does not establish above-chance agreement, yet text receives 30% of the rule aggregate, equal to alignment and editability, whose agreement is substantially stronger. Reconstruction agreement is also comparatively weak at 63.1%, although its reported interval [50.8%, 75.4%] excludes 50%. The validation uses 65-97 retained pairs per dimension; reconstruction and alignment reuse inputs, as the paper acknowledges. The report does not establish how pooled uncertainty accounts for that dependence, and per-subset uncertainty for the main comparisons is absent.

Broader coverage also does not fully resolve reference-convention dependence. Photorealistic examples use composited mattes, shadow references prescribe an explicit black RGBA layer, and editability probes use a fixed three-layer structure. DTW, coplanar grouping, and limited merging accommodate some alternatives, but do not establish acceptance of all functionally valid decompositions. LayerReward's 84.24% accuracy is measured on an image-disjoint, consensus-filtered test split from the same PrismLayersPro source as its training data. The human ranking and shadow study provide useful additional evidence, but do not characterize evaluator accuracy across the six LayerBench domains. These limitations matter for a benchmark intended to generalize beyond a narrow annotation convention.

Questions for authors

(1) Clarification about training/evaluation separation: Which models and inference configurations generated the preference candidates, and in what proportions? Which images were used to train LayerCompass, and how were they separated from LayerBench evaluation images? Were the A/B and five-model study raters disjoint from the preference-data annotators, and what qualifications beyond the stated expert status did they have?

(2) Decision-relevant validation request: Beyond the deployment rationale in Appendix A.12, what evidence shows that the VLM editability gains persist with an evaluator from a different model family? Shared architecture raises a possible correlation in evaluation preferences; it does not establish that the gains are artifacts.

(3) Clarification about the aggregate: What empirically supports the 1:3:3:3 rule weights and 50% LayerReward weight? How should users interpret the mismatch with the human ranking and the winner change at weight 0.33? For the claimed use case, what justifies prioritizing the composite ranking over the component tradeoffs already reported?

(4) Decision-relevant statistical request: Can you provide stronger validation of the text metric or justify its weight despite the current uncertainty? Please clarify the resampling unit for pooled human agreement and provide uncertainty for the subset comparisons that determine the headline rankings, accounting for shared images where applicable.

(5) Clarification about scope and reproducibility: How are functionally valid alternatives, such as a shadow grouped with its object, treated beyond the existing grouping/merging rules? What is the source of the text-rich subset, and how will the proposed source-URL and processing-code release allow durable reconstruction of the exact online-PSD subset?

(6) Related-work and operational-scope clarification: Please cite and discuss Improving Editability in Image Generation with Layer-wise Memory (CVPR 2025) and Multi-Edit Bench, distinguishing isolated transformations of recovered RGBA stacks from consistency across sequential edits with retained history. Which capabilities does LayerBench assess beyond that prior editing evaluation, and which remain outside its scope? Could compatible editing tasks provide independent validation of the benchmark scores, or what input/output mismatch prevents this? This asks for substantive positioning and functional validation, rather than an unmatched comparison to a history-dependent generator.

5 comments

Q1. Data provenance and rater independence. Preference candidates come from LayerD and Qwen-Image-Layered on PrismLayersPro, with varied hyperparameters and seeds and approximately balanced contributions across configurations. LayerCompass uses 1,600 RGB images from GenPoster100K, with no LayerBench overlap or ground-truth layer supervision. The A/B and five-model studies use separate rater groups comprising computer-vision researchers of different genders with annotation experience; some have direct layer-decomposition research experience.

W1 & Q2. Independent VLM evaluation. LayerCompass improves all four rule-based dimensions over its base (Table 2). We also replace only the VLM judge with Kimi K3.1 Pro, which was not used in training, keeping inputs, edited outputs, and the evaluation protocol fixed.

JudgeBase ↑LayerCompass ↑Gain
Qwen3-VL-8B0.31650.5289+0.2124
Kimi K3.1 Pro0.52340.6283+0.1049

The editability gain persists beyond the shared Qwen evaluator family.

W2 & Q3. Composite score and task preferences. Final combines complementary assessments: Rule checks reconstruction, reference alignment, text preservation, and editing behavior; VLM Editability judges edited outputs; LayerReward captures expert preferences over the complete decomposition without reference layers. Their rankings can differ: subtle reconstruction errors may be overlooked, while finer layers offer local control but increase organizational burden. As with reference-based and no-reference super-resolution metrics, different criteria answer different questions.

Final is a default summary, not a universal ranking or a replacement for component scores and radar plots. The largest individual weight, w_LR = 0.5, emphasizes human preference while retaining fidelity, reference alignment, and editing assessments. Users can prioritize components according to their applications.

We initially observed that overall decomposition preferences can differ from preferences considering downstream tasks. To test this, we conduct a user study with five raters on 20 images. Participants view all layers without a task, or only relevant layers for each specified task. We use dense ranking with ties; Avg. equally weights the five conditions.

ModelNo task ↓Background extraction/replacement ↓Object movement ↓All-text removal ↓Local text editing ↓Avg. ↓
LayerCompass1.551.101.301.301.301.31
Stable-Layers1.751.301.351.401.701.50
Qwen-Image-Layered2.301.201.401.401.601.58
SynLayers2.351.401.301.101.301.49
LayerD2.852.002.002.001.802.13

No-task ranks match Table 13; average ranks match Final. SynLayers ranks best for all-text removal despite lower overall preference.

Both Rule and Final rankings remain unchanged under rule weights 1:2:2:2, 1:3:3:3, and 1:4:4:4. Uniform weights also preserve Final and largely preserve Rule. Reconstruction receives less weight because trivial decompositions can reconstruct accurately without meaningful layer separation.

The complete Final ranking remains unchanged for w_LR ∈ [0.40, 0.60] under Appendix A.11’s protocol. SynLayers becomes first at 0.33 as its stronger Rule and VLM scores receive greater weight, reflecting a different objective balance. The default thus reflects our priorities, not uniquely optimal weights. LayerCompass also improves every aggregate component and all four rule dimensions over its base under any nonnegative, normalized weighting.

W2 & Q4. Text metric and statistical uncertainty. Humans may tolerate small-character errors penalized by OCR, explaining part of the text disagreement; lower Tiny Text agreement supports this interpretation. Rule metrics need not mirror perception. Reducing Text’s rule weight from 3 to 2 or 1, with other weights fixed, preserves both Rule and Final rankings.

We retain explicit preferences shared by ≥2/3 raters. Accounting for shared source images, the updated source-cluster bootstrap (100,000 resamples), grouping shared reconstruction/alignment judgments, yields the same pooled agreement of 75.2%, with a 95% CI of [70.3%, 80.0%].

Paired source-cluster bootstrap yields the following score differences. Positive differences favor the first model. Intervals are two-sided 95% percentile CIs, without multiplicity adjustment.

ComparisonMetricDifference95% CI
LayerCompass − Qwen-Image-LayeredText Overall+0.0912[0.0541, 0.1280]
LayerCompass − Qwen-Image-LayeredTiny Text+0.0114[−0.0297, 0.0524]
LayerCompass − SynLayersFinal+0.0285[0.0167, 0.0404]
SynLayers − Stable-LayersFinal+0.0391[0.0266, 0.0513]

These intervals support the reported Final-score ordering. Text Overall improves over the base, while the Tiny Text improvement remains uncertain.

W2 & Q5. Valid alternatives and reproducibility. LayerBench is designed not only to distinguish valid from invalid decompositions, but also to assess quality differences among valid alternatives. Grouping, DTW, and adjacent-layer merging accommodate alternative organizations. Shadows grouped with objects are evaluated normally, with preference and alignment differences reflected in their scores. Alignment is only one criterion alongside fidelity, human preference, text preservation, and editability.

Text-rich uses typeset posters, PSD designs, and MAGICK/Pexels composites with rendered text. We will release the exact RGB inputs, reference RGBA layers, source URLs, and processing code.

LayerReward matches human model rankings on independently sourced LayerBench data (Table 13).

Q6. Related work and editing scope. We will cite and discuss Prompting Editability in Image Generation with Layer-wise Memory and its proposed Multi-Edit Bench. Our editing probes test whether existing image elements can be independently and precisely moved, scaled, deleted, or recolored through recovered RGBA layers, with pixel-level reference targets and without generating new content. They target a key benefit of explicit layers: precise pixel-level control. Specified operations yield pixel-level ground-truth composites against which the edited outputs are evaluated. Multi-Edit Bench instead primarily assesses sequential prompt- and mask-conditioned generation and editing, so its different inputs, outputs, and evaluation granularity require adaptation. Explicit layers may also preserve untouched content across successive edits; we will discuss Multi-Edit Bench as an additional downstream setting for testing this potential benefit.

Dear Reviewer xvUp,

Thank you again for your time and constructive feedback. As the discussion period draws to a close, we would appreciate hearing whether our responses and additional results have addressed your concerns. If any questions remain, we would be happy to clarify them during the remaining time. Thank you for your consideration and effort in reviewing our work.

Reviewer_Yqa5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Paper Summary

LayerBench evaluates image layer decomposition beyond reconstruction fidelity, focusing on whether the recovered layers are useful for editing. It introduces 1,902 evaluation images, rule-based editing probes, a VLM judge, and LayerReward, an evaluator trained on 13,979 expert preference pairs. The authors also use LayerReward to post-train Qwen-Image-Layered, producing LayerCompass. Validation includes comparisons across decomposition methods, held-out preference prediction, ablations, and blind human studies. The results highlight differences between reconstruction quality, operational editability, and overall human preference.

Critical strengths

Strengths

  1. The motivation is clear and practically relevant. A decomposition can reconstruct the input accurately while remaining difficult to edit, making reconstruction alone an insufficient evaluation target.

  2. The benchmark combines diverse images with concrete editing operations, including scaling, movement, deletion, and recoloring. These probes provide interpretable tests of whether the layers support their intended use.

  3. The paper includes meaningful human validation. In particular, the blind comparison reports a 76.4% preference share for LayerCompass over its base model, providing evidence beyond improvements in the reward being optimized.

Critical weaknesses

Weaknesses

  1. LayerReward’s advantage over existing evaluators is not fully established. Tables 8–9 compare backbone choices and training-data sizes, but do not compare LayerReward with an untuned VLM judge on the same held-out preference set. The treatment of ties in the reported accuracy is also unclear. These details matter because human-aligned evaluation is a central contribution.

  2. The separation between post-training and evaluation data is unclear. Although the preference dataset has an image-disjoint training/test split, Section 3.5 and Appendix A.8 do not clearly identify the images used for LayerCompass post-training or their overlap with LayerBench. This makes it difficult to assess whether the improvements generalize to unseen images.

Questions for authors

Questions for Authors

  1. How does LayerReward compare with an untuned VLM on the same preference test set, and how are ties scored?

  2. Which images were used for LayerCompass post-training, and are they separate from the benchmark and held-out preference test images?

3 comments

W1 & Q1. Untuned VLM and ties. Thank you for your feedback. We additionally compare LayerReward with the untuned Qwen3-VL-8B-Instruct backbone on the same held-out preference test set. LayerReward substantially outperforms the untuned VLM across all dimensions, achieving a 32.86 percentage-point improvement in overall accuracy.

DimensionUntuned VLMLayerReward
Decomposition Correctness50.00%83.69%
Layer Quality46.96%88.18%
Occlusion Completion57.35%90.61%
Recomposition Consistency51.56%74.61%
Overall Quality51.38%84.24%

Regarding ties, following ImageReward[1], we exclude tied comparisons when computing accuracy and evaluate only pairs with explicit preferences. We will clarify this protocol and include the additional comparison in the revision.

W2 & Q2. Training data separation. Thank you for pointing this out. LayerReward is trained on preference pairs constructed from PrismLayersPro, with image-disjoint training and test splits. LayerCompass is post-trained using GenPoster100K[2], a separate dataset with no image overlap with either the LayerReward preference dataset or the LayerBench evaluation set.

Importantly, LayerCompass is post-trained for only 200 steps using 1,600 distinct source images. The observed improvements are achieved through LayerReward-guided adaptation on a small set of source images, without overlap with either evaluation set, rather than through large-scale additional training or exposure to benchmark images.

We will explicitly document these dataset sources, their separation, and the actual number of source images used during post-training in the revision.

[1] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. ImageReward: Learning and evaluating human preferences for text-to-image generation. In NeurIPS, 2023.

[2] Haoran Wang, Bo Zhao, Jinghui Wang, Hanzhang Wang, Huan Yang, Wei Ji, Hao Liu, and Xinyan Xiao. SEGA: A Stepwise Evolution Paradigm for Content-Aware Layout Generation with Design Prior. In ICCV, 2025.

Dear Reviewer Yqa5,

Thank you again for your time and constructive feedback. As the discussion period draws to a close, we would appreciate hearing whether our responses and additional results have addressed your concerns. If any questions remain, we would be happy to clarify them during the remaining time. Thank you for your consideration and effort in reviewing our work.

Thank you for the response. The additional results and clarifications resolve all of my concerns.

Reviewer_QYj7OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper is about how to fairly measure layer decomposition, which is the task of turning one flattened picture back into a stack of separate, editable image layers. The authors argue that current ways of judging this task are off, because a flat copy of the input can score well on reconstruction while being useless for editing, and because different prior works use narrow, inconsistent setups. Their answer is a benchmark built to match how humans think about a good decomposition. It brings together a set of curated images covering many styles and hard cases, a group of clear rule-based measures that check reconstruction, layer alignment, text preservation and editability, some model-based scores for finer visual quality, and a learned scorer trained on a large set of expert pairwise comparisons. This scorer is also reused as a reward to improve a decomposition model, and the improved model is checked against human preference. The overall message is that reconstruction quality is necessary but not enough, and a good decomposition should be judged by alignment, text, editability and human preference together.

Critical strengths

The first strength is that it fixes a real blind spot in how this field measures progress. Reconstruction alone can rank a useless flat answer above a genuinely good layered answer. By tying the benchmark to alignment, text preservation, editability and human preference, the paper makes progress on this task mean something closer to what users actually care about. For a benchmark, this alignment to human judgment is the most important quality, and it is what carries my assessment.

The second strength is that the evaluation also becomes a usable training signal, and that the whole loop is checked. The scorer is learned from expert comparisons rather than hand rules, holds up on data it was not trained on, and when used as a reward it improves a model in a way that is then confirmed by human preference. This connects evaluation to real improvement rather than leaving it as a passive scoreboard.

Critical weaknesses

The first weakness is that the contribution is mainly a tool and an evaluation, not a new capability, and it is somewhat narrow in scope. The ideas behind each piece are reasonable but not deep, and the whole thing is tied to the specific open models used for data and for the reward experiment, so it is hard to know how far the gains travel beyond that setting.

The second weakness is that the improvement story is not fully clean. The model that is improved and some of the comparison baselines are closely related and start from the same base, so part of the gain may come from a strong base model and careful fine-tuning rather than from the scorer or the reward signal itself. A more separate, matched comparison would make the claim stronger.

Questions for authors

First, how much of the improvement over the base model comes from the reward signal itself rather than from extra fine-tuning on a strong base? A budget-matched comparison without the reward would help.

Second, how well does the learned scorer transfer to decomposition models that are quite different from the ones its training data came from? More detail on this would show whether it is a general evaluator or tied to one family.

Third, the rules and the learned scorer are combined by hand-set weights into one final number; how sensitive is the conclusion to those weights?

2 comments

W1. Contribution and scope. We appreciate the reviewer’s recognition of the importance of human-aligned evaluation and human-validated improvement. LayerBench aims to guide layer decomposition toward practical outcomes through a comprehensive benchmark combining reconstruction, alignment, text preservation, editability, and expert preferences.

Our contributions include the benchmark, an expert preference dataset, LayerReward, and LayerCompass. We agree that we do not introduce a new decomposition algorithm, but the resulting improvement extends to practical capabilities: Figure 4 shows cleaner shadow separation after LayerReward-guided post-training. Our additional evaluation on an unseen decomposition model further addresses generalization (Q2).

W2 & Q1. comparison. Thanks for raising this question. Supervised fine-tuning can improve decomposition, but requires additional image–layer pairs. LayerCompass instead aligns decompositions with human preferences through LayerReward-guided RL using only RGB inputs, without additional layer annotations.

To examine whether the gains can be attributed simply to extra fine-tuning, we add an SFT control using image–layer pairs constructed from the same 1,600 GenPoster100K source images. Both conditions start from the same Qwen-Image-Layered checkpoint and use the same 1,600 source images, number of optimization steps, and effective source-image batch size, matching the total source-image training exposure.

Training conditionRule ↑VLM Editability ↑LayerReward ↑
Base model0.6710.3170.690
Supervised fine-tuning0.6720.3830.691
LayerCompass0.7340.5290.775

SFT yields limited improvements, whereas LayerCompass achieves larger gains across all three components. This comparison supports the benefit of preference-guided post-training beyond additional supervised fine-tuning on the same source images. The improvements in Rule and VLM Editability further show that the gains extend beyond increasing the optimized reward.

Q2. Unseen-model transfer. Thank you for this suggestion. LayerReward’s training candidates already span different model families, including LayerD and Qwen-Image-Layered. We further evaluate transfer to the recently open-sourced Ming-Image-0.1-Design-Layer, which was not used in preference-data construction.

We proportionally sample 100 LayerBench images and generate two decompositions per image using different random seeds. Without retraining or adaptation, LayerReward predicts preferences for these 100 pairs. We report agreement with expert preferences, excluding human-tied pairs separately for each dimension.

DimensionNon-tied pairsAccuracy ↑
Decomposition Correctness7880.77%
Layer Quality7684.21%
Occlusion Completion7087.14%
Recomposition Consistency1968.42%
Overall Quality8381.93%

Agreement exceeds 80% in four dimensions, supporting transfer beyond the training model families. We will include these cross-model validation results in the revision and benchmark release.

Q3. Weight sensitivity. Thank you for this question.

  1. Improvement over the base model is weight-independent. LayerCompass improves all four rule-based dimensions, VLM Editability, and LayerReward. Any nonnegative, normalized weighting of these components therefore preserves this improvement.

  2. Rule weights have limited impact on ranking. Both Rule and Final rankings remain unchanged under 1:2:2:2, 1:3:3:3, and 1:4:4:4. With uniform weights, the Rule ranking is largely unchanged and the Final ranking remains identical. We downweight reconstruction because trivial decompositions can reconstruct accurately without meaningful layer separation.

  3. The complete Final ranking remains unchanged for w_LR ∈ [0.40, 0.60]. Our default of 0.5 emphasizes human preference while accounting for fidelity, reference alignment, and editing capability. Final summarizes these complementary objectives rather than reproducing preference alone.

We additionally evaluate human preferences with five raters on 20 images, showing all layers without a task and only relevant layers for each specified task. We use dense ranking with ties; the last column averages the five conditions.

ModelNo task ↓Background extraction/replacement ↓Object movement ↓All-text removal ↓Local text editing ↓Avg. ↓
LayerCompass1.551.101.301.301.301.31
Stable-Layers1.751.301.351.401.701.50
Qwen-Image-Layered2.301.201.401.401.601.58
SynLayers2.351.401.301.101.301.49
LayerD2.852.002.002.001.802.13

The no-task ordering matches Table 13, whereas the average ordering matches Final, illustrating the distinction between overall preference and task-specific suitability.

Dear Reviewer QYj7,

Thank you again for your time and constructive feedback. As the discussion period draws to a close, we would appreciate hearing whether our responses and additional results have addressed your concerns. If any questions remain, we would be happy to clarify them during the remaining time. Thank you for your consideration and effort in reviewing our work.

Comments on the paper

2 comments

Discussion Summary

We thank all reviewers, the Area Chair, and the Senior Area Chair for their time and feedback during the review process. We are especially grateful to Reviewer Yqa5 for confirming that our responses resolved all of their concerns and to Reviewer opQF for acknowledging that our clarifications resolved several of their questions. Below, we summarize our main clarifications and additional experiments across all reviews.

1. Robustness to aggregation weights (Refer: Reviewer QYj7, Q3):

Both Rule and Final rankings remain unchanged under rule weights of 1:2:2:2, 1:3:3:3, and 1:4:4:4, and the complete Final ranking remains unchanged for wLR∈[0.40,0.60]w_{\mathrm{LR}}\in[0.40,0.60]. Moreover, LayerCompass improves over its base model across all three aggregate components and all four rule-based dimensions, independently of the aggregation weights.

2. SynLayers’ ranking and task-specific editing utility (Refer: Reviewer opQF, W4 & Q4):

Our additional 20-image user study with five raters examines the distinction between preferences over complete decompositions and preferences under specific editing tasks. SynLayers’ finer decomposition granularity can provide finer editing control, although inspecting and managing more layers may reduce preference for complete decompositions. SynLayers ranks first on all-text removal. The ranking averaged across all five conditions matches the Final ranking, whereas the no-task ranking matches the original human-preference ranking. These results provide supporting evidence for evaluating complementary aspects of decomposition quality.

3. Interpretation of the Final score (Refer: Reviewer xvUp, W2 & Q3):

LayerBench evaluates layer decomposition across four complementary dimensions: reconstruction fidelity, reference alignment, editability, and human preference. Human preference is central, but viewers may tolerate subtle color discrepancies that violate reconstruction fidelity or prefer coarser decompositions to reduce visual inspection effort.

Different aggregation weights reflect different evaluation priorities. Our default Final score emphasizes human preference (wLR=0.5w_{\mathrm{LR}}=0.5) while retaining fidelity, reference alignment, and editability. We report all individual component scores and radar plots so users can prioritize dimensions suited to their downstream needs. The user study supports distinguishing overall preference from task-specific utility, while the sensitivity analysis shows that the reported ranking does not depend on choosing exactly wLR=0.5w_{\mathrm{LR}}=0.5.

4. Improvements beyond supervised fine-tuning (Refer: Reviewer QYj7, W2 & Q1):

To test whether additional standard supervised fine-tuning alone explains the gains, we added an SFT control using image–layer pairs constructed from the same 1,600 GenPoster100K source images, starting from the same base model:

Training conditionRule ↑VLM Editability ↑LayerReward ↑
Base model0.6710.3170.690
Supervised fine-tuning0.6720.3830.691
LayerCompass0.7340.5290.775

LayerCompass outperforms this SFT control across all three components, supporting the benefit of LayerReward-guided post-training beyond this supervised fine-tuning baseline.

5. Scorer generalization to unseen models and comparison with untuned VLMs (Refer: Reviewer QYj7, Q2; Reviewer Yqa5, W1 & Q1; Reviewer xvUp, W2):

  • Unseen-model transfer: Evaluated without additional training on 100 candidate pairs generated by the recently open-sourced Ming-Image-0.1-Design-Layer, which was not used to construct the preference-training data, LayerReward achieves 81.93% overall-quality agreement with expert preferences on the 83 non-tied pairs, exceeding 80% agreement in four of the five dimensions.
  • Untuned VLM comparison: On the held-out preference test set, LayerReward outperforms the untuned Qwen3-VL-8B-Instruct backbone by 32.86 percentage points in overall accuracy (84.24% vs. 51.38%), with ties excluded following standard protocol.

6. Data separation, construction rules, reproducibility, and complex examples (Refer: Reviewer Yqa5, W2 & Q2; Reviewer opQF, W1–W3, W5 & Q1–Q3, Q5; Reviewer xvUp, Q1 & Q5):

We clarified the separation among the LayerReward preference dataset, the LayerCompass post-training set, and the LayerBench evaluation set. LayerReward’s preference data are constructed from PrismLayersPro with image-disjoint training and test splits. LayerCompass uses 1,600 RGB images from GenPoster100K for post-training, with no image overlap with either the LayerReward preference dataset or the LayerBench evaluation set.

We also detailed the filtering rules for each subset and the alpha-overlap algorithm for Visual Plane Grouping. We will release the processed RGB inputs, reference RGBA layers, source URLs, and processing code. We have also added complete multi-layer PSD examples—inputs, reference layers, and predictions from all five models—to the supplementary material.

Regarding complex layer structures, layer count alone does not determine decomposition difficulty. Two- or three-layer cases still test occlusion completion, shadow ownership, text preservation, and editing behavior, and current models exhibit substantial errors even in these settings. These samples therefore remain informative for distinguishing model capabilities. The additional multi-layer PSD examples further illustrate evaluation behavior on more complex structures. We will clarify the benchmark’s coverage and identify broader structural diversity as an important direction for extension.

We appreciate Reviewer opQF’s acknowledgment that our responses addressed some concerns, and recognize the remaining reservations about complex-structure coverage and metric validity. We hope this summary helps clarify both the additional evidence and the remaining limitations, and thank all reviewers and the AC/SAC for considering the full discussion in their final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.