- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Appearance Pointers for region-controlled generation and editing with FLUX Kontext. A region correspondence transformer links VAE-encoded masks and image references with regional text, and spatial aggregation compresses the resulting features into two conditioning canvases; contour guidance and LoRA provide additional adaptation. The authors construct AppearancePointers-37K, evaluate text-conditioned and image-conditioned generation on synthetic data and SACap-eval, and demonstrate mixed-modality generation and insertion qualitatively.
Critical strengths
-
A single set of weights supports text, image, and combined regional inputs, with missing modalities handled within the same correspondence module. The examples of category instructions combined with reference materials, together with heterogeneous conditions across regions, illustrate a useful interface beyond either text-only layout control or image-only personalization. The separation of region encoding from denoising is also a sensible design for reusing conditions across sampling steps.
-
The image-conditioned results provide concrete evidence of useful identity control. In Table 3, DINO-I is 69.31 compared with 64.20 for DreamRenderer and 45.61 for MS-Diffusion. On the real-data bounding-box setting in Table 4, it is 72.16 compared with 46.86 for MS-Diffusion, so the gains are not confined to providing finer masks. Table 5 also shows that aggregation improves DINO-I from 54.47 to 69.31, providing support for this component beyond the qualitative examples.
Critical weaknesses
-
The experiments do not sufficiently establish the central claim of joint text-and-image control within the same region, or isolate the proposed routing mechanism from the surrounding training setup. The 500-image benchmark in Section 5.1 and Appendix F evaluates text-only and image-only regional conditions separately, and Table 5 only ablates image-conditioned generation. The mixed-modality galleries are promising, but do not measure whether complementary instructions from both modalities are satisfied simultaneously or whether changing one condition leaves the other intact. Furthermore, removing aggregation retains the region correspondence network; it does not compare the complete learned pointer mechanism with simpler regional conditioning under the same FLUX Kontext backbone, training data, contour guidance, and LoRA budget. Such a control and a focused mixed-modality evaluation are important because unification and learned routing are the principal contributions. The current evidence supports useful unimodal performance more strongly than it supports the claimed multimodal advance.
-
The claim of negligible inference overhead is not established by the complexity analysis. Equations 4 and 5 still concatenate the individual regional image and text prompts into the backbone at every denoising step, so the total sequence length can grow with the number of regions even though the aggregated pointer canvases have fixed spatial size. The aggregation transformer also performs attention over regions at each spatial position. Computing the pointers once removes repeated correspondence-network computation, but does not remove these costs. The model adds about 400 million correspondence/aggregation parameters and 500 million LoRA parameters, yet no end-to-end latency or peak-memory measurements are reported.
Questions for authors
- How are the 500 synthetic evaluation cases separated from the 37K training data? In particular, are source scenes, object identities, and all derived pose/material variants kept in the same split?
3 comments
Author response to Reviewer eU8U (Phase 1)
We thank reviewer eU8U for following up and apologize for the delayed response. Our experiments are unfortunately taking longer than anticipated. We start with phase 1 rebuttal and will soon follow with numbers for phase 2.
W1. Joint text-and-image per region
Contradictory conditions (red-car image + "blue car" text) are out of the training regime, and the model pays more attention to the image in these cases. We do not have region-wise editing data that teaches modification of a reference from local text; generating such data and retraining the same appearance pointer formulation is a natural extension the architecture already supports, but out of scope here.
Within-region image+text is trained as composition: image carries geometry, identity, and texture; text adds a complementary attribute the image does not fix. Per-modality adherence is measured separately in Table 2 (text) and Table 3 (image). Compositional cases are qualitative (Fig. 1(d); supp. Figs. 10, 11 explicitly show material + text compositions where both signals are followed).
Phase 2 adds: (i) heterogeneous-across-regions (one scene with text-only, image-only, image+text regions, per-region metrics); (ii) chained-specialist baseline (Seg2Any then iterative InsertAnything) using two models to attempt the same heterogeneous control.
W2. Inference overhead
The reviewer is correct that sequence length grows with local reference and prompt tokens. The last paragraph in §3.2 states only the Appearance Pointers are computed once per generation.
The reference image and prompt tokens are necessary for the base model's conditioning budget, not overhead introduced by AppearancePointers.
- Pointer tokens add negligible overhead. and run once per generation, not per denoising step.
- Pointer canvases are of fixed spatial size independent of .
- Aggregation cuts the image-stream token count by roughly : without it, each of regions contributes its own mask-token stack along with its conditions. With aggregation all region mask tokens collapse into one fixed canvas per stream.
Train / eval split
500 synthetic eval scenes held out by generation seed and shifted along five axes:
- Indoor and outdoor scenes not seen at train time.
- New object categories, large (cars, washing machines, trucks, tables, etc.) and small (coins, pens, books, etc.).
- New human-profession prompts (doctors, lawyers, etc.).
- More regions per scene than training.
- Different generation seeds.
Source scenes, identities, and all pose/material variants are held out together; no identity, scene, or seed appears in both splits. This is also what enables zero-shot transfer to real SACap-eval (Fig. 17). We will state it in the appendix.
Isolating the pointer
- No-pointer / attention-manipulation baseline. The simplest no-learned-linking regional conditioning is training-free attention manipulation that restricts conditions to certain regions. DreamRenderer does exactly this with a different backbone per task: InstanceDiffusion + FLUX.1-Dev-Depth (DreamRenderer, Tab. 2) and Depth Anything v2 + FLUX.1-Dev + FLUX.1-Redux-Dev (DreamRenderer*, Tab. 3). Seg2Any is the trained at-scale (SACap-1M) version of the same mechanism. Tables 2 and 3 (AP-37K) and Table 4 (SACap-eval zero-shot) compare directly against these baselines. On AP-37K text-region, our pointer wins region CLIP-I, DINO-I, MIoU and global CLIP-IQA against Seg2Any; on real SACap-eval, we lead on CLIP-IQA zero-shot and on global alignment after fine-tuning on 2.3% of the real data.
- Within-pointer isolation. Table 5's w/o Region Aggregation row removes aggregation while keeping the correspondence transformer, isolating aggregation's contribution: DINO-I (54.47 → 69.31) and CLIP-I (89.93 → 93.29) at MIoU cost. Aggregation is also what lets the pointer scale past ~7 regions without OOM.
Phase 2 (over the next 2 days)
- LAMIC & SIGMA-Gen on our image-region benchmark and SACap-eval.
- Latency of appearance pointers for varying regions.
- Heterogeneous-across-regions benchmark with per-region metrics along with chained-specialist baseline (Seg2Any then iterative InsertAnything).
Author response to Reviewer eU8U (Phase 2)
Thank you for being patient! Phase 2 numbers and paper updates are posted below (all revisions highlighted in purple in the updated PDF).
1. Joint same-region image+text evaluation (W1, Table 4 and Figure 8 in Appendix F)
To directly measure joint text-and-image control within the same region, we evaluate on 200 images where each region is conditioned on both an image reference and a text prompt.
| Method | Region CLIP-I | Region CLIP-T | Region DINO-I | Region MIoU | Global CLIP-T | Global CLIP-IQA |
|---|---|---|---|---|---|---|
| MS-Diffusion | 89.24 | 25.89 | 48.99 | 21.88 | 29.85 | 84.64 |
| LAMIC | 84.70 | 24.13 | 32.65 | 22.93 | 28.69 | 93.82 |
| Ours | 94.04 | 27.03 | 73.61 | 43.04 | 29.26 | 89.45 |
Our method leads on all four region metrics with substantial margins on DINO-I (+24.6) and MIoU (+20.1). Region CLIP-I measures image-reference fidelity and region CLIP-T measures text-adherence per region, so both signals are reported jointly for the same scene; we outperform all baselines on both. Figure 8 in Appendix F gives side-by-side qualitative comparisons.
2. Isolation of pointer routing (W1)
LAMIC is the matched-backbone isolation control. LAMIC is training-free attention masking on the identical FLUX Kontext backbone we use — "simpler regional conditioning" at the exact backbone and zero-extra-parameter setting. Our Phase 2 numbers show LAMIC fails to preserve fine region details (Table 3 region DINO-I 33.21 vs our 69.31; Table 4 region CLIP-I 84.70 vs our 94.04). Seg2Any and DreamRenderer are the trained at-scale variants of this masked-control family and we outperform them on region fidelity in our setting.
Table 5 "w/o region aggregation" is the matched in-model isolation. It keeps the correspondence network and concatenates per-region mask tokens sequence-wise into the prompt stream instead of collapsing them through depth-wise aggregation, under the same FLUX Kontext backbone, rank-128 LoRA, contour guidance, and AP-37K training data.
3. Latency and peak memory (W2, Appendix E, Table 12)
Measured on a single A100 SXM4 40GB GPU at 1024×1024 with 18 flow-matching steps. Each region contributes 1024 tokens if image-conditioned and 512 tokens if text-conditioned (image+text regions contribute both), so latency scales roughly linearly with region count.
Generation time (seconds):
| Regions | image+text (same region) | image-only | text-only |
|---|---|---|---|
| 1 | 22.9 | 20.8 | 19.6 |
| 3 | 33.8 | 27.6 | 23.7 |
| 5 | 44.9 | 34.6 | 27.9 |
| 7 | 56.2 | 41.5 | 32.4 |
Peak GPU memory reaches ~39 GB at R=7 with image+text per region.
4. Data split (Q)
Addressed in our Phase 1 reply: the 500 evaluation images are held out from AP-37K training with no shared source scenes, object identities, or pose/material variants.
5. Heterogeneous conditioning vs chained-specialist baseline (Table 5 and Figure 9 in Appendix F)
As committed in Phase 1, we construct a chained-specialist baseline: one pass of Seg2Any (text regions) followed by iterative InsertAnything passes (image references). 200 scenes mix image-only and text-only regions in a single diffusion pass.
| Method | Region CLIP-I | Region CLIP-T | Region DINO-I | Region MIoU | Global CLIP-T | Global CLIP-IQA |
|---|---|---|---|---|---|---|
| Seg2Any + InsertAnything | 91.45 | 27.12 | 62.98 | 37.45 | 28.51 | 85.23 |
| Ours | 93.49 | 26.95 | 71.62 | 44.16 | 29.42 | 89.46 |
Our method leads on five of the six metrics, with gains of +8.6 on DINO-I and +6.7 on MIoU. Figure 9 in Appendix F shows the qualitative comparison.
6. Other paper updates
- LAMIC and SIGMA-Gen on image-region benchmarks (Tables 3 and 6): added as additional baselines.
- Abstract and intro rescoped to "flexible heterogeneous region conditioning in a single diffusion pass."
- CFG scale 4 noted in §3.2 Implementation Details.
- Capability Table 1 updated with LAMIC. Related work (§2) mentions LAMIC and SIGMA-Gen.
Since the authors did not submit a rebuttal, I will keep my original rating.