- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents LayerBench for evaluating image layer decomposition, LayerReward for scoring decompositions using expert preferences, and LayerCompass for improving a decomposition model through RL.
Critical strengths
The editing tests measure whether the layers support operations such as moving, deleting, and recoloring objects.
The expert annotations and blind human studies provide useful evidence beyond reconstruction scores.
Critical weaknesses
-
The examples cover only simple layer structures. Figure 4 shows two inputs without GT. Figures 14 and 15 show GT stacks with only two or three layers. These examples leave complex PSD structures largely unexamined. Complete GT and predicted RGBA files are needed to inspect layers.
-
Several dataset sources and filtering rules are missing. The paper does not specify the Online PSD filtering criteria or the exact sources of Text-rich and Editability. Photorealistic examples are foreground and background composites, and 68.6% of the benchmark has only two or three layers.
-
The RL training data is not specified. Appendix A.8 gives hardware and training settings but does not identify the source images, dataset size, etc.
-
The final ranking is not validated by the human study. SynLayers ranks second in the final score but fourth in the human ranking. Table 12 also shows that SynLayers ranks first when the LayerReward weight is 0.33, while LayerCompass ranks first at 0.5. The human study supports the LayerReward ranking, but it does not establish the choice of weights for the final score.
-
The rules for Visual Plane Grouping are not clearly defined. Appendix A.3 mentions a top-down scan, but does not explain how it determines that layers belong to the same visual plane, or when a new group starts. These choices affect DTW matching and the alignment scores.
Questions for authors
-
Can the authors provide complete GT and predicted RGBA files for complex PSDs?
-
What are the sources and filtering rules for Online PSD, Text-rich, and Editability?
-
What data is used for LayerCompass RL training?
-
Why use the current score weights given the ranking differences in Tables 2, 12, and 13?
-
How is Visual Plane Grouping implemented? For example, how layers are assigned to the same plane, and when a new group starts.
4 comments
W4 & Q4. Ranking differences and weights. Thank you for highlighting this distinction. Table 13 validates LayerReward as a measure of overall human preference; Final additionally considers fidelity, reference alignment, and editing capability. These objectives are complementary, so Final is a default summary rather than a requirement to reproduce the preference-only ranking. We report component scores and radar plots to make these tradeoffs explicit. The default w_LR = 0.5 gives LayerReward the largest individual weight while retaining the other assessments.
SynLayers illustrates why the rankings can differ. Its finer-grained outputs can increase the burden of inspecting and coordinating layers, while providing more precise control over individual elements. To examine overall versus task-specific preferences, we additionally conduct a user study with five raters on 20 source images under five conditions. Without a specified task, participants judge all recovered layers; for each specified task, they judge only the relevant layers. We use dense ranking, allowing ties without gaps. Average Rank is the equally weighted mean across the five conditions.
| Model | None (No specified task) ↓ | Background Extraction & Replacement ↓ | Foreground Object Movement ↓ | All-Text Removal ↓ | Local Text Editing ↓ | Average Rank ↓ |
|---|---|---|---|---|---|---|
| LayerCompass | 1.55 | 1.10 | 1.30 | 1.30 | 1.30 | 1.31 |
| Stable-Layers | 1.75 | 1.30 | 1.35 | 1.40 | 1.70 | 1.50 |
| Qwen-Image-Layered | 2.30 | 1.20 | 1.40 | 1.40 | 1.60 | 1.58 |
| SynLayers | 2.35 | 1.40 | 1.30 | 1.10 | 1.30 | 1.49 |
| LayerD | 2.85 | 2.00 | 2.00 | 2.00 | 1.80 | 2.13 |
The no-task ordering matches Table 13. However, SynLayers achieves the best mean rank for all-text removal, illustrating that lower preference for the complete decomposition does not imply lower suitability for every task. The ordering averaged across the five conditions matches Final.
The weighting conclusions are also stable around our default: the complete Final ranking remains unchanged throughout w_LR ∈ [0.40, 0.60] under the weighting protocol in Appendix A.11. SynLayers ranks first at 0.33 because that setting gives greater weight to its stronger Rule and VLM editing scores. This reflects a different balance of objectives rather than a contradiction with the human-preference results. More broadly, Final is a convenient default summary whose weights express the relative emphasis placed on Rule, VLM Editability, and LayerReward; it is not the sole basis for interpreting LayerBench results. Our default emphasizes human preference while also accounting for fidelity, reference alignment, and editing capability. We report the total scores for all three components together with their detailed breakdowns and radar plots, and encourage users with different task priorities to rely on the dimensions most relevant to their intended applications.
W5 & Q5. Visual Plane Grouping. We group predicted and reference stacks independently by alpha-mask overlap, after matching canvas sizes and resizing to a 512 px long side. Masks include all pixels with alpha > 0, including the background.
The topmost unassigned layer seeds a plane. We scan remaining layers top-down, comparing each mask M with the current plane’s mask union U. A layer is deferred only if |M ∩ U| ≥ 4 pixels and |M ∩ U| / min(|M|, |U|) > 0.001; otherwise it joins the plane and updates U. After a full pass, the topmost remaining layer seeds the next plane. Thus, disjoint layers can share a plane even if an intervening layer is deferred.
Members are composited in their original order, and DTW aligns the plane sequences top-down. Separately, up to three greedy adjacent-layer merges are tried on predictions, regrouping and realigning candidates and retaining a merge only if it reduces mean DTW path cost. Thank you for your feedback. We will clarify this in the revision.
Dear Reviewer opQF,
Thank you again for your time and constructive feedback. As the discussion period draws to a close, we would appreciate hearing whether our responses and additional results have addressed your concerns. If any questions remain, we would be happy to clarify them during the remaining time. Thank you for your consideration and effort in reviewing our work.
Thank you for the rebuttal and clarifications, which have addressed some of my concerns. However, I still have reservations about the limited coverage of complex layer structures in the benchmark and the validity of the proposed evaluation metrics. Therefore, I will maintain my original score.
W1 & Q1. Complex PSD examples. Due to space and layout constraints, the main paper includes only a limited number of complex decompositions with many layers. To address your concern, we now provide additional examples in the supplementary material, showing the input images, ground-truth RGBA layers, and predicted RGBA layers from all five evaluated models.
W2 & Q2. Dataset sources and filtering. We will clarify the dataset sources and filtering procedures in the revision.
For the photorealistic subset, we composite foreground objects onto photographic backgrounds to obtain reliable reference layers. We also explored object-removal models, but their recovered backgrounds could alter originally visible regions, introducing errors into reference-based evaluation. Compositing preserves known background content and makes the reference layers and resulting evaluation more reliable.
We consider the 68.6% proportion of two- or three-layer samples reasonable for four reasons. First, current models still exhibit substantial errors even on two- or three-layer decompositions, making low-layer-count cases informative for distinguishing model performance. Second, these cases can also test occlusion completion, shadow ownership, text preservation, and alignment with human preferences. Third, constructing reliable ground-truth layers for the Editability, Shadow, and Photorealistic subsets is challenging; controlled, low-layer-count settings enable reliable references while broadening evaluation scope. Finally, they help keep evaluation across the dataset and multiple models computationally practical. As the first comprehensive layer decomposition benchmark, LayerBench does not exhaust all layer structures. We will acknowledge this limitation and identify broader coverage as future work.
W3 & Q3. RL training data. We recognize that Appendix A.8 does not clearly identify the post-training images. LayerCompass is post-trained using 1600 RGB images from GenPoster100K, a separate dataset with no image overlap with either the LayerReward preference dataset or the LayerBench evaluation set.