acceptodds
Under review as a conference paper at ICLR 2027

Appearance Pointers: Multimodal Region Control of Diffusion Transformers

Abstract

Creative professionals rarely want a single global description of an image; they want region-level control over the material, identity, and placement of individual objects, which text prompting alone cannot reliably deliver. Diffusion Transformers (DiTs) natively ingest heterogeneous text and image tokens, yet provide no mechanism to decide where and how each token should shape the output. We introduce appearance pointers, compact tokens that route a frozen DiT to the correct appearance cue at the correct region by aligning text or reference-image inputs with user-specified masks. A region correspondence network produces these pointers, and a depth-wise aggregation step fuses many regions into a single canvas, so multiple heterogeneous regions are generated in one denoising pass without inflating the token count. Unlike prior region-control methods — tied to a single modality or one region at a time — AppearancePointers is the first to condition the same region on image and text simultaneously, with one set of weights and only 7.5% added parameters. Across text- and image-conditioned benchmarks, and zero-shot on real-world data, our single model matches or surpasses modality-specific state-of-the-art methods, offering a simple, extensible interface for precise multimodal region control.

Primary area:
generative models
Keywords:
Text+Image+Region Control, Generative Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_eU8UOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Appearance Pointers for region-controlled generation and editing with FLUX Kontext. A region correspondence transformer links VAE-encoded masks and image references with regional text, and spatial aggregation compresses the resulting features into two conditioning canvases; contour guidance and LoRA provide additional adaptation. The authors construct AppearancePointers-37K, evaluate text-conditioned and image-conditioned generation on synthetic data and SACap-eval, and demonstrate mixed-modality generation and insertion qualitatively.

Critical strengths
  1. A single set of weights supports text, image, and combined regional inputs, with missing modalities handled within the same correspondence module. The examples of category instructions combined with reference materials, together with heterogeneous conditions across regions, illustrate a useful interface beyond either text-only layout control or image-only personalization. The separation of region encoding from denoising is also a sensible design for reusing conditions across sampling steps.

  2. The image-conditioned results provide concrete evidence of useful identity control. In Table 3, DINO-I is 69.31 compared with 64.20 for DreamRenderer and 45.61 for MS-Diffusion. On the real-data bounding-box setting in Table 4, it is 72.16 compared with 46.86 for MS-Diffusion, so the gains are not confined to providing finer masks. Table 5 also shows that aggregation improves DINO-I from 54.47 to 69.31, providing support for this component beyond the qualitative examples.

Critical weaknesses
  1. The experiments do not sufficiently establish the central claim of joint text-and-image control within the same region, or isolate the proposed routing mechanism from the surrounding training setup. The 500-image benchmark in Section 5.1 and Appendix F evaluates text-only and image-only regional conditions separately, and Table 5 only ablates image-conditioned generation. The mixed-modality galleries are promising, but do not measure whether complementary instructions from both modalities are satisfied simultaneously or whether changing one condition leaves the other intact. Furthermore, removing aggregation retains the region correspondence network; it does not compare the complete learned pointer mechanism with simpler regional conditioning under the same FLUX Kontext backbone, training data, contour guidance, and LoRA budget. Such a control and a focused mixed-modality evaluation are important because unification and learned routing are the principal contributions. The current evidence supports useful unimodal performance more strongly than it supports the claimed multimodal advance.

  2. The claim of negligible inference overhead is not established by the complexity analysis. Equations 4 and 5 still concatenate the individual regional image and text prompts into the backbone at every denoising step, so the total sequence length can grow with the number of regions even though the aggregated pointer canvases have fixed spatial size. The aggregation transformer also performs attention over regions at each spatial position. Computing the pointers once removes repeated correspondence-network computation, but does not remove these costs. The model adds about 400 million correspondence/aggregation parameters and 500 million LoRA parameters, yet no end-to-end latency or peak-memory measurements are reported.

Questions for authors
  1. How are the 500 synthetic evaluation cases separated from the 37K training data? In particular, are source scenes, object identities, and all derived pose/material variants kept in the same split?
3 comments

Since the authors did not submit a rebuttal, I will keep my original rating.

AuthorsOpenReview ↗

Author response to Reviewer eU8U (Phase 1)

We thank reviewer eU8U for following up and apologize for the delayed response. Our experiments are unfortunately taking longer than anticipated. We start with phase 1 rebuttal and will soon follow with numbers for phase 2.

W1. Joint text-and-image per region

Contradictory conditions (red-car image + "blue car" text) are out of the training regime, and the model pays more attention to the image in these cases. We do not have region-wise editing data that teaches modification of a reference from local text; generating such data and retraining the same appearance pointer formulation is a natural extension the architecture already supports, but out of scope here.

Within-region image+text is trained as composition: image carries geometry, identity, and texture; text adds a complementary attribute the image does not fix. Per-modality adherence is measured separately in Table 2 (text) and Table 3 (image). Compositional cases are qualitative (Fig. 1(d); supp. Figs. 10, 11 explicitly show material + text compositions where both signals are followed).

Phase 2 adds: (i) heterogeneous-across-regions (one scene with text-only, image-only, image+text regions, per-region metrics); (ii) chained-specialist baseline (Seg2Any then iterative InsertAnything) using two models to attempt the same heterogeneous control.

W2. Inference overhead

The reviewer is correct that sequence length grows with local reference and prompt tokens. The last paragraph in §3.2 states only the Appearance Pointers are computed once per generation.

The reference image and prompt tokens are necessary for the base model's conditioning budget, not overhead introduced by AppearancePointers.

  • Pointer tokens add negligible overhead. ΦRC\Phi_{RC} and ΦA\Phi_A run once per generation, not per denoising step.
  • Pointer canvases IAP,TAP{}^{I}\mathcal{AP}, {}^{T}\mathcal{AP} are of fixed spatial size independent of RR.
  • Aggregation cuts the image-stream token count by roughly 2×2\times: without it, each of RR regions contributes its own mask-token stack along with its conditions. With aggregation all region mask tokens collapse into one fixed canvas per stream.

Train / eval split

500 synthetic eval scenes held out by generation seed and shifted along five axes:

  1. Indoor and outdoor scenes not seen at train time.
  2. New object categories, large (cars, washing machines, trucks, tables, etc.) and small (coins, pens, books, etc.).
  3. New human-profession prompts (doctors, lawyers, etc.).
  4. More regions per scene than training.
  5. Different generation seeds.

Source scenes, identities, and all pose/material variants are held out together; no identity, scene, or seed appears in both splits. This is also what enables zero-shot transfer to real SACap-eval (Fig. 17). We will state it in the appendix.

Isolating the pointer

  • No-pointer / attention-manipulation baseline. The simplest no-learned-linking regional conditioning is training-free attention manipulation that restricts conditions to certain regions. DreamRenderer does exactly this with a different backbone per task: InstanceDiffusion + FLUX.1-Dev-Depth (DreamRenderer, Tab. 2) and Depth Anything v2 + FLUX.1-Dev + FLUX.1-Redux-Dev (DreamRenderer*, Tab. 3). Seg2Any is the trained at-scale (SACap-1M) version of the same mechanism. Tables 2 and 3 (AP-37K) and Table 4 (SACap-eval zero-shot) compare directly against these baselines. On AP-37K text-region, our pointer wins region CLIP-I, DINO-I, MIoU and global CLIP-IQA against Seg2Any; on real SACap-eval, we lead on CLIP-IQA zero-shot and on global alignment after fine-tuning on 2.3% of the real data.
  • Within-pointer isolation. Table 5's w/o Region Aggregation row removes aggregation while keeping the correspondence transformer, isolating aggregation's contribution: +14.8+14.8 DINO-I (54.47 → 69.31) and +3.36+3.36 CLIP-I (89.93 → 93.29) at 0.50.5 MIoU cost. Aggregation is also what lets the pointer scale past ~7 regions without OOM.

Phase 2 (over the next 2 days)

  1. LAMIC & SIGMA-Gen on our image-region benchmark and SACap-eval.
  2. Latency of appearance pointers for varying regions.
  3. Heterogeneous-across-regions benchmark with per-region metrics along with chained-specialist baseline (Seg2Any then iterative InsertAnything).
AuthorsOpenReview ↗

Author response to Reviewer eU8U (Phase 2)

Thank you for being patient! Phase 2 numbers and paper updates are posted below (all revisions highlighted in purple in the updated PDF).

1. Joint same-region image+text evaluation (W1, Table 4 and Figure 8 in Appendix F)

To directly measure joint text-and-image control within the same region, we evaluate on 200 images where each region is conditioned on both an image reference and a text prompt.

MethodRegion CLIP-IRegion CLIP-TRegion DINO-IRegion MIoUGlobal CLIP-TGlobal CLIP-IQA
MS-Diffusion89.2425.8948.9921.8829.8584.64
LAMIC84.7024.1332.6522.9328.6993.82
Ours94.0427.0373.6143.0429.2689.45

Our method leads on all four region metrics with substantial margins on DINO-I (+24.6) and MIoU (+20.1). Region CLIP-I measures image-reference fidelity and region CLIP-T measures text-adherence per region, so both signals are reported jointly for the same scene; we outperform all baselines on both. Figure 8 in Appendix F gives side-by-side qualitative comparisons.

2. Isolation of pointer routing (W1)

LAMIC is the matched-backbone isolation control. LAMIC is training-free attention masking on the identical FLUX Kontext backbone we use — "simpler regional conditioning" at the exact backbone and zero-extra-parameter setting. Our Phase 2 numbers show LAMIC fails to preserve fine region details (Table 3 region DINO-I 33.21 vs our 69.31; Table 4 region CLIP-I 84.70 vs our 94.04). Seg2Any and DreamRenderer are the trained at-scale variants of this masked-control family and we outperform them on region fidelity in our setting.

Table 5 "w/o region aggregation" is the matched in-model isolation. It keeps the correspondence network and concatenates per-region mask tokens sequence-wise into the prompt stream instead of collapsing them through depth-wise aggregation, under the same FLUX Kontext backbone, rank-128 LoRA, contour guidance, and AP-37K training data.

3. Latency and peak memory (W2, Appendix E, Table 12)

Measured on a single A100 SXM4 40GB GPU at 1024×1024 with 18 flow-matching steps. Each region contributes 1024 tokens if image-conditioned and 512 tokens if text-conditioned (image+text regions contribute both), so latency scales roughly linearly with region count.

Generation time (seconds):

Regionsimage+text (same region)image-onlytext-only
122.920.819.6
333.827.623.7
544.934.627.9
756.241.532.4

Peak GPU memory reaches ~39 GB at R=7 with image+text per region.

4. Data split (Q)

Addressed in our Phase 1 reply: the 500 evaluation images are held out from AP-37K training with no shared source scenes, object identities, or pose/material variants.

5. Heterogeneous conditioning vs chained-specialist baseline (Table 5 and Figure 9 in Appendix F)

As committed in Phase 1, we construct a chained-specialist baseline: one pass of Seg2Any (text regions) followed by iterative InsertAnything passes (image references). 200 scenes mix image-only and text-only regions in a single diffusion pass.

MethodRegion CLIP-IRegion CLIP-TRegion DINO-IRegion MIoUGlobal CLIP-TGlobal CLIP-IQA
Seg2Any + InsertAnything91.4527.1262.9837.4528.5185.23
Ours93.4926.9571.6244.1629.4289.46

Our method leads on five of the six metrics, with gains of +8.6 on DINO-I and +6.7 on MIoU. Figure 9 in Appendix F shows the qualitative comparison.

6. Other paper updates

  • LAMIC and SIGMA-Gen on image-region benchmarks (Tables 3 and 6): added as additional baselines.
  • Abstract and intro rescoped to "flexible heterogeneous region conditioning in a single diffusion pass."
  • CFG scale 4 noted in §3.2 Implementation Details.
  • Capability Table 1 updated with LAMIC. Related work (§2) mentions LAMIC and SIGMA-Gen.
Reviewer_CCEiOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes AppearancePointers, a framework for multimodal region control of a DiT based on FLUX Kontext. A region correspondence network fuses the text or reference image of each region with its mask, and a region aggregation transformer collapses all regions into a single canvas, producing image and text appearance pointers that are passed to the DiT together with region contour guidance. This allows regions to be described by text, an image, or both within a single denoising pass using one set of weights, with about 7.5% added parameters. The authors also build AppearancePointers-37K, a synthetic dataset with region captions, novel-view references, and material edits, and evaluate the method on text and image region benchmarks and on the real-world SACap-eval.

Critical strengths
  • S1. The paper addresses a practical need by offering a single interface for region control with text, images, or both, and it supports a variable number of regions in one pass.

  • S2. The design is efficient in structure. Region correspondence is computed once per generation and independently per region, and the aggregation step compresses multiple regions into a single canvas.

  • S3. In the image-conditioned setting, the method shows strong region fidelity and identity preservation (CLIP-I 93.29 and DINO-I 69.31 in Table 3), and identity remains stable as the number of regions grows and masks become coarser (Tables 8 and 9).

  • S4. The dataset pipeline is described in detail with prompts and filtering criteria in the appendix, which supports reproducibility and further work on region-level multi-modal control.

Critical weaknesses

Concerns about novelty and problem definition

  • W1. Scope of editing The editing examples only change the attributes or materials of existing objects or insert objects into empty regions. Can the method replace an object with one of a different class under fine and coarse masks, and is background preservation measured?

  • W2. Difference from multi-modal layout-to-image generation Most components follow prior work, including contour guidance (DreamRenderer, Seg2Any), RoPE assignment (OminiControl, SIGMA-Gen), the LoRA setup (OminiControl), and the data pipeline (SIGMA-Gen). (The authors also mention about this in this paper.) The new elements are region correspondence and region aggregation, and the latter lowers MIoU (41.49 to 40.97, Table 5). Beyond combining text and mask methods (e.g., Seg2Any, EliGen) with image and layout methods (e.g., MS-Diffusion, SIGMA-Gen), what is the methodological contribution?

  • W3. Scope of the "first" claim The italicized "same" and "and" in the abstract narrow the "first" claim, since prior work already supports multiple modalities in one model or across regions, as Table 1 shows for DreamRenderer. This narrowed capability is not evaluated quantitatively.

  • W4. Vanilla baseline in the introduction figure (No number figure) The vanilla baseline in the unnumbered introduction figure (lines 099 to 110) passes reference tokens without any mask information, so its failure is expected and does not justify the pointers. Does the pointer design outperform simple mask token concatenation or regional attention masking (e.g., Seg2Any, EliGen) under the same backbone and LoRA setting?

  • W5. Primary goal and missing DiT baselines The headline multi-modal capability is not evaluated quantitatively, so the primary goal of the paper is unclear, and key baselines are missing either way. EliGen (ACM MM Asia 2025) and SIGMA-Gen (ICLR 2026) are cited as arXiv preprints and not compared, although SIGMA-Gen is listed in Table 1 and followed for the data pipeline and RoPE design. DiT-based multi-condition methods such as OminiControl, UniCombine, and EasyControl are also missing.

Concerns about experimental evaluation

  • W6. Inconsistent comparison settings The competitors and inputs differ between the qualitative and quantitative comparisons. Fig. 8(in appendix) uses DreamRenderer* with depth while Table 2 uses DreamRenderer without it, Nano-Banana appears only in Fig. 18, and the inputs range from bounding boxes for MS-Diffusion to depth for DreamRenderer* and fine masks for the proposed method, which makes a fair comparison difficult.

  • W7. Meaning of zero-shot It is unclear why zero-shot transfer from AP-37K supports the contribution. If AP-37K covered real-world data, the upper section of Table 4 should already be strong, yet the method trails Seg2Any, and if it does not, training on each dataset would be the natural choice. The image task compares only with MS-Diffusion, so generalization is not clearly demonstrated.

Questions for authors

Please refer to Critical Weaknesses section.

3 comments
AuthorsOpenReview ↗

Author response to Reviewer CCEi (Phase 1)

We thank the reviewer CCEi for the detailed analysis. Our experiments are taking longer than anticipated. We address points below and will post phase 2 numbers in 1-2 days.

Novelty scope (W2, W3)

We do not claim novelty on contour guidance, RoPE positional IDs, or the LoRA setup (all cited). The novel pieces are ΦRC\Phi_{RC} (region-mask routing, once per generation rather than per timestep) and per-patch ΦA\Phi_A (per-region identity routing via fixed-size canvas aggregation); neither alone is prior-art-equivalent in the multimodal setting. Their combination enables per-region modality heterogeneity in a single pass: within one scene, some regions are image-only, some text-only, and some image+text, all generated by a single model.

Methodimage-onlytext-onlyimage+text (same region)heterogeneous across regions
SIGMA-Gen✓✗✗ (text only at scene level)✗
Seg2Any / EliGen✗✓✗✗
MS-Diffusion✓✗✗✗
DreamRenderer (InstanceDiffusion + FLUX.1-Dev-Depth) in Tab. 2✗✓✗✗
DreamRenderer* (Depth Anything v2 + FLUX.1-Dev + FLUX.1-Redux-Dev) in Tab. 3✓ (via Redux)✗✗✗
Ours (single model)✓✓✓✓

DreamRenderer uses two different backbones for the image+region task (Tab. 3) and the text+region task (Tab. 2), and cannot do both in one diffusion pass. The last column has no prior baseline — we construct one by stacking Seg2Any (text regions) with InsertAnything (image references) in phase 2 of the rebuttal. Fig. 1(d) and supp. Figs. 10-13 show our one-pass heterogeneous results.

Pointer vs. simpler baselines (W4)

The two controlled comparisons the reviewer proposes already exist as full systems:

Pointer vs. mask-token concatenation. We take the mask tokens IM,TM{}^{I}\mathcal{M}, {}^{T}\mathcal{M} from the correspondence network and append them to the condition tokens, with no aggregation. This is Table 5's "w/o region aggregation" row — same backbone, LoRA, contour, and data as the full model. The two components do different jobs. Contour keeps content inside the mask: without it, MIoU drops from 40.97 to 35.81. Aggregation routes identity: without it, DINO-I falls to 54.47 and MIoU stays near 41.49. Aggregation raises DINO-I to 69.31, CLIP-I to 93.29, CLIP-IQA to 95.57. Concatenation also doubles DiT tokens and OOMs at higher region counts.

Pointer vs. regional attention masking. Seg2Any is the strongest at-scale realization and is already compared. Table 2 (AP-37K): we win on region CLIP-I, DINO-I, MIoU, global CLIP-IQA; Seg2Any wins region CLIP-T, global CLIP-T. Table 4 (SACap-eval, synthetic+2.3% real): we win on global CLIP-T, CLIP-IQA; Seg2Any wins region CLIP-T, MIoU (in-distribution SACap-1M training). We are also comparing against LAMIC (training-free attention masking on FLUX Kontext, our backbone); numbers in phase 2.

The unnumbered intro figure is for motivation — reference tokens concatenated with the global prompt, no per-region routing.

Missing baselines (W5)

  • OminiControl, UniCombine, EasyControl: each conditions on a single controlled region per generation. Extending them to heterogeneous edits (different modalities per region) is non-trivial.
  • EliGen: bbox regions only; Seg2Any (bbox + finer masks, stronger) is already compared in Table 2.
  • SIGMA-Gen and LAMIC: evaluations in phase 2 on region benchmark + SACap-eval.

Comparison settings (W6)

Each baseline is evaluated in its own supported setting (standard practice in Seg2Any, MS-Diffusion, InstanceDiffusion). Protocol per method:

MethodRegion inputRegion conditionBackboneIn
InstanceDiffusionbboxtextSD1.5 (native)Tab. 2
Seg2Anyfine masktextFLUX.1-DevTab. 2
DreamRenderermasktextInstanceDiffusion + FLUX.1-Dev-DepthTab. 2
DreamRenderer*mask + depthimage (via Redux)Depth Anything v2 + FLUX.1-Dev + FLUX.1-Redux-DevTab. 3
MS-Diffusionbbox + entity textimageSDXLTab. 3
Oursfine masktext / image / image+textFLUX KontextTab. 2, 3

Zero-shot evaluation (W7)

We train on synthetic AP-37K, so SACap-eval measures transfer. Seg2Any trains on SACap-1M (in-distribution upper bound). Fine-tuning on 2.3% real data narrows the region CLIP-T gap: 25.57→25.78 vs 26.24 (Table 4).

Editing scope (W1)

Class-level replacement under fine and coarse masks is supported by treating the background image as a region condition (supp. §C). Tables 2 and 3 follow prior works' generation protocol with no input background image, so background preservation is not applicable — it applies only to the insertion/editing setting.

Phase 2 (over the next 2 days)

  1. LAMIC & SIGMA-Gen on our image-region benchmark and SACap-eval.
  2. Heterogeneous-across-regions benchmark vs stacked Seg2Any + InsertAnything.
  3. Latency across varying number of regions.

Thank you for the clarification. My remaining points on W3 are as follows.

  • DreamRenderer. I agree that DreamRenderer relies on two separate configurations and does not condition the same region on image and text in a single pass.

  • Missing quantitative evaluation. My main concern in W3 was the lack of a quantitative evaluation of the same-region image and text setting. The response provides only the capability table and qualitative results (Fig. 1(d) and supp. Figs. 10 to 13). The planned phase 2 benchmark targets heterogeneous modalities across regions, which is a different setting. While the abstract claims to be the first to condition the same region on image and text, the response shifts the framing to heterogeneity across regions.

  • Distinction from MS-Diffusion. The capability table marks MS-Diffusion as not supporting image and text for the same region. However, MS-Diffusion conditions each subject on its reference image together with an entity phrase and a box, and the rebuttal itself lists its region input as “bbox + entity text”. The difference is that the entity phrase in MS-Diffusion is a label for the subject in the image, whereas the material case of Appearance Pointers combines an image and a text that carry different information, such as a material image with an object category. This difference is the distinctive contribution of the paper, yet it is not verified quantitatively.

Based on the comments above, I additionally request a quantitative evaluation of this setting in phase 2.

AuthorsOpenReview ↗

Author response to Reviewer CCEi (Phase 2)

Thank you for the feedback and for being patient. Phase 2 numbers and paper updates are posted below, with all revisions highlighted in purple in the updated PDF.

1. Same-region image+text conditioning vs MS-Diffusion and LAMIC (Table 4 and Figure 8 in Appendix F)

To address your request for a quantitative evaluation of same-region image+text conditioning, we evaluate on 200 images where each region is conditioned on both an image reference and a text prompt.

MethodRegion CLIP-IRegion CLIP-TRegion DINO-IRegion MIoUGlobal CLIP-TGlobal CLIP-IQA
MS-Diffusion89.2425.8948.9921.8829.8584.64
LAMIC84.7024.1332.6522.9328.6993.82
Ours94.0427.0373.6143.0429.2689.45

Our method leads on all four region metrics with substantial margins on DINO-I (+24.6) and MIoU (+20.1). MS-Diffusion leads global CLIP-T and LAMIC leads global CLIP-IQA, with ours close second on both.

2. Distinction from MS-Diffusion

Beyond the numbers in Table 4, our method extends MS-Diffusion along four capability axes:

  1. Image+text per region. Table 4 (and Figure 8 in Appendix F) shows we outperform MS-Diffusion and LAMIC on all four region metrics for this setting.
  2. Fine-grained region shapes. We support fine segmentation masks, while MS-Diffusion is restricted to bounding boxes.
  3. Heterogeneous region conditioning. We mix image-only and text-only regions within a single scene in one diffusion pass (Table 5, Figure 9); MS-Diffusion does not support this setting.
  4. Insertion into existing scenes. We support object insertion into an existing background with qualitative and quantitative comparisons against InsertAnything in Appendix C; MS-Diffusion does not support insertion.

3. Heterogeneous conditioning vs stacked Seg2Any + InsertAnything baseline (Table 5 and Figure 9 in Appendix F)

We evaluate on 200 scenes where image-only and text-only regions coexist in a single diffusion pass. Since no prior method supports this heterogeneous setting natively, we construct a chained-specialist baseline: one pass of Seg2Any for the text regions followed by iterative InsertAnything passes for the image references.

MethodRegion CLIP-IRegion CLIP-TRegion DINO-IRegion MIoUGlobal CLIP-TGlobal CLIP-IQA
Seg2Any + InsertAnything91.4527.1262.9837.4528.5185.23
Ours93.4926.9571.6244.1629.4289.46

Our method leads on five of the six metrics, with gains of +8.6 on DINO-I and +6.7 on MIoU. Figure 9 in Appendix F shows the corresponding qualitative comparison.

4. LAMIC and SIGMA-Gen on image-region (Tables 3 and 6)

AP-37K image-region (Table 3):

MethodRegion CLIP-IRegion CLIP-TRegion DINO-IRegion MIoUGlobal CLIP-TGlobal CLIP-IQA
LAMIC (training-free, FLUX Kontext)84.5124.2733.2123.4428.6093.04
SIGMA-Gen (image-region specialist, SIGMA-27K)93.4527.1770.5141.4629.5290.92
Ours93.2927.2569.3140.9729.6295.57

SIGMA-Gen marginally leads region fidelity (DINO-I +1.2, CLIP-I +0.2, MIoU +0.5) as expected from a specialist dedicated to image-region conditioning, but supports only image-conditioned regions. Ours wins global CLIP-IQA by +4.65 and covers three settings SIGMA-Gen does not support: text-only, image+text per region, and heterogeneous across regions.

On SACap-eval Region+Image (Table 6), LAMIC and SIGMA-Gen are added; Ours (Fine-Grained) wins global CLIP-T (33.02) and is second-best on region fidelity.

5. Latency and peak memory (Appendix E, Table 12)

Measured on a single A100 SXM4 40GB GPU at 1024×1024 with 18 flow-matching steps. Each region contributes 1024 tokens if image-conditioned and 512 if text-conditioned (image+text regions contribute both), so latency scales roughly linearly with region count.

Generation time (seconds):

Regionsimage+text (same region)image-onlytext-only
122.920.819.6
333.827.623.7
544.934.627.9
756.241.532.4

Peak GPU memory reaches roughly 39 GB at the largest configuration (R=7 with image+text per region).

6. Abstract rescope and other paper updates

  • Abstract: rescoped to "flexible heterogeneous region conditioning in a single diffusion pass, where image-only, text-only, and image+text regions coexist within one scene", removing the earlier priority claim on same-region image+text.
  • Intro: aligned with the same scope.
  • Capability Table 1: LAMIC row added.
  • Related work (§2): LAMIC and SIGMA-Gen are now briefly characterized.
  • CFG scale 4 stated in §3.2 Implementation Details.
  • Fig. 4 (depth-wise aggregation) explicitly referenced in §3.1.
Reviewer_upCXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes AppearancePointers, a lightweight module that enables multi-region controllable generation and editing. Each region is specified by a mask and described by text, a reference image, or both. A region correspondence network and a region aggregation network convert these inputs into compact pointer tokens, which guide the DiT to the correct appearance cue in each region within a single denoising pass. The authors also introduce a synthetic dataset, AppearancePointers-37K.

Critical strengths
  1. Well structured. The paper is well organized, with a logical flow between sections that makes the method and its motivation easy to follow. The explanations are clear and the overall presentation is easy to understand.
  2. Extensive qualitative results. The paper presents a wide range of qualitative examples across several tasks. These results provide a comprehensive of the performance.
Critical weaknesses
  1. Limited novelty, and the gain comes largely from prior work. a. The paper claims to be the first to condition the same region on both image and text. However, LAMIC [1] already pairs each reference image with a text description and a target region. The authors should clarify what distinguishes their approach or tone down this claim.
    b. The paper’s backbone, in-context reference conditioning, positional ID scheme and data pipeline closely follow SIGMA-Gen [2], yet SIGMA-Gen appears only in Tab. 1 and is not discussed in the related work. The authors should clarify their contribution relative to [1-3]. c. The Region Contour Guidance is essentially the Entity Contour Map of Seg2Any [3]. The ablation in Table 5 suggests this Region Contour Guidance component accounts for most of the improvement, removing it lowers metrics significantly.
  2. Missing baselines. SIGMA-Gen [2] and LAMIC [1] are never evaluated or compared with the proposed method.
  3. Missing and unfair evaluation. Tables 2-3 are in-domain for AppearancePointers and out-of-domain for all baselines. The ground truth is also FLUX-generated, which biases ground-truth similarity metrics toward FLUX-based models. On real data, Seg2Any remains better in region adherence. I suggest an evaluation that is out-of-domain for all methods. Masks and prompts could come from an existing real dataset that none of the methods was trained on, such as COCO or ADE20K. I also suggest reporting FID/KID on that dataset, following [3].

[1] "LAMIC: Layout-Aware Multi-Image Composition via Scalability of Multimodal Diffusion Transformer", Chen et al., AAAI 2026.
[2] "SIGMA-Gen: Structure and Identity Guided Multi-Subject Assembly for Image Generation", Saha et al., ICLR 2026.
[3] "Seg2Any: Open-set Segmentation-Mask-to-Image Generation with Precise Shape and Semantic Control", Li et al., NeurIPS 2025.

Questions for authors
  1. Coordinate-augmented mask. L232 refers to the supplement for detail of the coordinate augmentation, yet I could not find them. Could authors point to where they are, and if ablation of the design have been done?
  2. “Depth-wise” processing (L283). This depth-wise processing is never mentioned in the later part of manuscript. Could authors explain what this operation is?
  3. Region CLIP-T. AppearancePointers shows lower region CLIP-T than the baselines, both in Tab. 2 and on SACap-eval (Tab. 4). Could the authors analyze the reason behind this?
  4. CFG configuration. Please report the guidance configuration for AppearancePointers and all baselines: the guidance type and scale, which conditions are dropped for the unconditional branch.
  5. Computational efficiency. Could the authors report inference time and peak memory compared with the baselines, and how they scale with the number of regions?

Minor comments

  1. Inconsistent notation makes the paper hard to follow.
    1. The binary mask is defined as R_i in L228 but later appears as {}^{R}P.
    2. The Region Aggregation Transformer is Φ_A in the text but Φ_K in Fig. 4.
    3. The learnable [CLS] token is a calligraphic (\mathcal) A in L281 but a plain A in L291.
    4. The appearance pointer is a calligraphic AP in the text but a plain AP in Fig. 4.
    5. {}^{G}P in Eq. 5 is never defined.
    6. In L328, R and T are both defined as “the number of regions.” Are they the same quantity?
    7. The method and dataset names are used inconsistently. The method appears as “AppearancePointers,” “Appearance Pointers,” “AppearancePointer,” and “Appearance Pointer.” The dataset appears as “AppearancePointers-37K,” “Appearance Pointers-37K,” “Appearance-37K,” and “AppearancePointers-27K.” The baseline appears as both “MS-Diffusion” and “MSDiffusion.”
  2. Typos
    1. L294: “resulting better” → “resulting in better”
    2. L328: “ant T is the number or regions” → “and T is the number of regions”
    3. L443 (Tab. 4 caption): “adherance” → “adherence”
    4. L948 (Fig. 7 caption): “we perform multi subject insertion” → “We perform multi-subject insertion”
    5. L1018: “Aggregation:.” → “Aggregation:”
    6. L1061: “implementing attention pointer” → “implementing appearance pointers”
    7. L1665: “upto” → “up to”
    8. L1682: “highlights the region of correct region of interest” → “highlights the correct region of interest”
    9. L1036, L1048: “table” → “Table”. L1066: “fig. 9” → “Fig. 9”. Lowercase “fig.” and “table” appear throughout the manuscript, so the capitalization should be made consistent.
    10. For consistency with the other captions, I suggest making “Quantitative comparison on SACap-eval dataset (real-world images)” bold in the Tab. 4 caption.
2 comments
AuthorsOpenReview ↗

Author response to Reviewer upCX (Phase 1)

We thank the reviewer upCX for the careful read. Our experiments are unfortunately taking longer than anticipated. We address points below and will post phase 2 numbers in 1-2 days.

Novelty scope (W1, W2)

We do not claim novelty on contour guidance, RoPE positional IDs, or the LoRA setup (prior works were already cited where necessary). Our novelty is per-region modality heterogeneity in a single pass: within one scene, some regions are image-only, some text-only, and some image+text, all generated by a single model. Prior works each target a subset of this space:

Methodimage-onlytext-onlyimage+text (same region)heterogeneous across regions
SIGMA-Gen✓✗✗ (text only at scene level)✗
Seg2Any✗✓✗✗
MS-Diffusion✓✗✗✗
DreamRenderer (InstanceDiffusion + FLUX.1-Dev-Depth) in Tab. 2✗✓✗✗
DreamRenderer* (Depth Anything v2 + FLUX.1-Dev + FLUX.1-Redux-Dev) in Tab. 3✓ (via Redux)✗✗✗
Ours (single model)✓✓✓✓

DreamRenderer and DreamRenderer* use different backbones per task. The last column has no prior baseline: chaining Seg2Any (text) then InsertAnything (image refs) breaks when a new text-only region is added, as re-running Seg2Any overwrites the earlier inserts. Fig. 1(d) and supp. Figs. 10-13 show our one-pass heterogeneous results.

LAMIC's paper evaluates image+text per-region conditioning (text-only regions are not demonstrated); we are evaluating it under different conditional settings in phase 2.

Contour vs. pointer contribution

Table 5 shows both components are necessary. Removing contour lets content bleed outside the region; the GT-mask crop then misses it, hurting both DINO-I (−27.7) and MIoU (40.97 → 35.81). Keeping contour but removing aggregation recovers MIoU to 41.49 yet DINO-I stays at 54.47 because identity is not routed. Aggregation supplies this routing, raising DINO-I by +14.8 and CLIP-I by +3.4 to 69.31 and 93.29.

Out-of-distribution evaluation (W3)

AppearancePointers-37K is purely synthetic. SACap-eval is OOD for us (no real training images) but in-distribution for Seg2Any (trained on SACap-1M). COCO/ADE20K are not held out either: Seg2Any ships dedicated ADE20K and COCO-Stuff checkpoints (HF 0xLDF/Seg2Any), so FID/KID there is in-distribution too. On the FLUX-GT bias concern: Seg2Any and DreamRenderer are themselves FLUX-based, so FLUX-generated GT does not favor us over them. Our headline metrics (DINO-I, MIoU, CLIP-IQA) are backbone-agnostic and measured against the user mask and region prompt, not FLUX pixel distributions.

Q1 — Coordinate-augmented mask

Given a binary mask RiR_i, we form a 3-channel map where in-mask pixels are [h,w,1][h,w,1] and out-of-mask pixels are [0,0,0][0,0,0], then encode it with the FLUX VAE like an image prompt. We add this formula explicitly in appendix §B.1 of the revised paper.

Q2 — Depth-wise aggregation

Depth-wise is exactly what Fig. 4 illustrates: at each patch, our region-aggregation transformer runs self-attention across all regions of IM,TM{}^{I}\mathcal{M}, {}^{T}\mathcal{M} (region axis as sequence), with a learnable [CLS][\mathrm{CLS}] producing IAP{}^{I}\mathcal{AP} and TAP{}^{T}\mathcal{AP}. Each patch aggregates along the patches of all regions of M\mathcal{M}. We will rewrite §4.1.3 and add a Fig. 4 pointer.

Q3 — Region CLIP-T is lower

We agree Seg2Any has better region CLIP-T (27.67 vs 27.24): it is trained on SACap-1M, orders of magnitude more than our 37K synthetic set. On SACap-eval (Table 4), fine-tuning on 2.3% real SACap data narrows the region CLIP-T gap (25.57 → 25.78 vs Seg2Any's 26.24). On the same split, our global CLIP-T moves from 32.17 to 32.99, above Seg2Any's 32.30. The region gap is consistent with the 27x data-scale difference. Table 9 shows our CLIP-T is stable across mask granularities (28.60-28.93).

Q4 — CFG configuration

We use CFG scale 4 in our model and use the settings provided by prior works for the baselines.

Q5 — Inference time and peak memory

Our pointer canvases are of fixed spatial size independent of RR, and ΦRC\Phi_{RC} runs once per generation, giving a per-step cost of O(R(Nreg/k)2)\mathcal{O}(R(N_{\mathrm{reg}}/k)^2) instead of O(T(R⋅Nreg)2)\mathcal{O}(T(R \cdot N_{\mathrm{reg}})^2). We will report wall-clock latency and peak memory in phase 2.

Notation, typos, naming

We thank the reviewer for catching these. All notation and typos are fixed in the updated PDF (purple): RPi{}^{R}\mathcal{P}_i = VAE-encoded RiR_i; GP{}^{G}\mathcal{P} = global prompt; TT = denoising timesteps; [CLS] is calligraphic TA,IA{}^{T}\mathcal{A}, {}^{I}\mathcal{A} throughout; MS-Diffusion naming consistent; all listed typos corrected.

Phase 2 (over the next 2 days)

  1. LAMIC & SIGMA-Gen on our image-region benchmark and SACap-eval.
  2. Heterogeneous-across-regions benchmark with per-region metrics.
  3. Latency for varying regions.
AuthorsOpenReview ↗

Author response to Reviewer upCX (Phase 2)

Phase 2 numbers and paper updates are posted below (all revisions highlighted in purple in the updated PDF).

1. LAMIC and SIGMA-Gen on image-region benchmark (Table 3, AP-37K)

MethodRegion CLIP-IRegion CLIP-TRegion DINO-IRegion MIoUGlobal CLIP-TGlobal CLIP-IQA
LAMIC (training-free, FLUX Kontext)84.5124.2733.2123.4428.6093.04
SIGMA-Gen (image-region specialist, SIGMA-27K)93.4527.1770.5141.4629.5290.92
Ours93.2927.2569.3140.9729.6295.57

SIGMA-Gen marginally leads region fidelity (DINO-I +1.2, CLIP-I +0.2, MIoU +0.5) as expected from a dedicated image-region model. Ours wins global CLIP-IQA by +4.65 and covers three settings SIGMA-Gen does not support (text-only, image+text per region, heterogeneous across regions).

2. SACap-eval Region+Image (Table 6)

LAMIC and SIGMA-Gen added. Ours (Fine-Grained) wins global CLIP-T (33.02 vs 33.00 LAMIC, 32.92 SIGMA-Gen) and remains silver on region fidelity. SIGMA-Gen leads region metrics on its native image-region setting.

3. Same-region image+text conditioning (Table 4 and Figure 8 in Appendix F)

Each region conditions on an image reference and a text prompt.

MethodRegion CLIP-IRegion CLIP-TRegion DINO-IRegion MIoUGlobal CLIP-TGlobal CLIP-IQA
MS-Diffusion89.2425.8948.9921.8829.8584.64
LAMIC84.7024.1332.6522.9328.6993.82
Ours94.0427.0373.6143.0429.2689.45

Ours wins all 4 region metrics with DINO-I +24.6 and MIoU +20.1 over the best baseline. MS-Diffusion leads global CLIP-T, LAMIC leads global CLIP-IQA, Ours is second-best on both. Figure 8 (Appendix F) shows side-by-side qualitative comparisons against MS-Diffusion and LAMIC.

4. Heterogeneous conditioning (Table 5 and Figure 9 in Appendix F)

200 scenes mixing image-only and text-only regions in one pass. Chained-specialist baseline: one pass of Seg2Any (text regions) followed by iterative InsertAnything (image references).

MethodRegion CLIP-IRegion CLIP-TRegion DINO-IRegion MIoUGlobal CLIP-TGlobal CLIP-IQA
Seg2Any + InsertAnything91.4527.1262.9837.4528.5185.23
Ours93.4926.9571.6244.1629.4289.46

Ours wins 5/6 metrics (DINO-I +8.6, MIoU +6.7, CLIP-I +2.0, global CLIP-T +0.9, CLIP-IQA +4.2). Figure 9 (Appendix F) shows the visual comparison.

5. Latency and peak memory (Appendix E, Table 12)

Measured on a single A100 SXM4 40GB GPU at 1024×1024 resolution with 18 flow-matching steps. Each region contributes 1024 tokens if image-conditioned and 512 tokens if text-conditioned; image+text regions contribute both. Latency scales roughly linearly with region count.

Generation time (seconds):

Regionsimage+text (same region)image-onlytext-only
122.920.819.6
333.827.623.7
544.934.627.9
756.241.532.4

Peak GPU memory reaches ~39 GB at the largest configuration (R=7 with image+text per region).

6. On FLUX-GT bias (W3 follow-up)

All strong baselines in our comparison are FLUX-based: Seg2Any (FLUX.1-Dev), DreamRenderer (InstanceDiffusion + FLUX.1-Dev-Depth), DreamRenderer* (Depth Anything v2 + FLUX.1-Dev + FLUX.1-Redux-Dev), LAMIC (training-free on FLUX Kontext), and SIGMA-Gen (FLUX-based). FLUX-generated GT therefore cannot privilege us over them. On SACap-eval (real images, not FLUX-generated), Seg2Any is in-distribution (trained on SACap-1M) and still only wins region CLIP-T by 0.46 after 2.3% fine-tune.

7. Other paper updates

  • Q3 region CLIP-T: we agree Seg2Any has better region CLIP-T (27.67 vs 27.24); it is trained on SACap-1M, orders of magnitude more than our 37K synthetic set. On SACap-eval (Table 6), fine-tuning on 2.3% real SACap data narrows the region CLIP-T gap (25.57 → 25.78 vs Seg2Any's 26.24). On the same split, our global CLIP-T moves from 32.17 to 32.99, above Seg2Any's 32.30. The region gap is consistent with the 27x data-scale difference. Table 11 shows our CLIP-T is stable across mask granularities (28.60-28.93).
  • CFG scale 4 stated in §3.2 Implementation Details. Baselines use released checkpoints with recommended CFG settings.
  • Abstract and intro rescoped to claim flexible heterogeneous region conditioning in a single diffusion pass.
  • Fig. 4 reference added to the depth-wise aggregation description in §3.1.
  • Capability Table 1 now includes LAMIC. Related work (§2) mentions LAMIC and SIGMA-Gen.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.