acceptodds
Under review as a conference paper at ICLR 2027

ChiaroRetouch: Recoverable Color Actions for Stagewise Local Photo Retouching

Abstract

Reasoning photo retouching translates abstract user intent into precise tone and color adjustments. Generative editors regenerate pixels and reach impressive quality at the cost of slow inference and capped output resolution, whereas driving a differentiable executor with compact editing parameters is efficient and resolution-independent but has largely been limited to global operations. Extending such efficient models to local retouching requires explicitly modeling *where* to adjust and *what* action to apply. From a per-pixel perspective, we factorize region control along chromatic, luminance, and semantic dimensions, and build a six-stage process over Hue, three luminance bands (Shadows, Midtones, and Highlights), and Subject supports, together with a Global stage. Learning such structured local execution, however, requires stagewise trajectories with spatial supervision, which are difficult to collect at scale. We therefore treat recoverability as a design constraint and propose ChiaroRetouch, whose fixed-basis Gaussian executor stays nonlinear in color yet linear in its executable action coefficients, so every masked state transition determines its action by a deterministic support-weighted solve. This property lets MaskGrade synthesize and numerically verify local stagewise trajectories at scale via controlled degradation. Experiments show that ChiaroRetouch achieves a favorable trade-off between reference fidelity and instruction adherence across multiple expert retouching benchmarks; its closed-form recovery is over 94x faster than iterative fitting, and it processes a 1K image in 0.35 s while supporting local retouching at original resoluti

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
photo retouching, vision-language models, masked image editing, color correction

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_wWEpOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ChiaroRetouch, a stagewise local photo retouching framework that decomposes instruction-guided retouching into spatial supports and continuous color actions. The method uses a fixed-basis Gaussian color executor that is nonlinear in color but linear in its action coefficients, enabling closed-form recovery of color actions from masked state transitions. The authors use this recoverability property to construct synthetic local stagewise trajectories through the proposed MaskGrade pipeline. Experiments show competitive fidelity and instruction-adherence trade-offs, as well as strong inference efficiency.

Critical strengths
  • Clear motivation for local retouching. The reviewer agrees that modeling both what color action to apply and where to apply is an important practical problem for photo retouching.
  • Interesting recoverability formulation. The fixed-basis Gaussian executor is a useful design choice: by keeping the action coefficients linear, the paper obtains a closed-form solution for recovering color actions from masked transitions. This is technically clean and helps avoid slow iterative fitting.
  • Good computational efficiency.
Critical weaknesses
  • The novelty of the paper is somewhat incremental. The paper is well engineered and the closed-form recovery mechanism is useful. However, the overall framework appears to be an incremental extension of prior efficient executor-based retouching methods to local regions using masks and a fixed stagewise decomposition. The six-stage structure is practical, but largely hand-designed from standard photo-editing operations. The paper would be stronger if it more clearly separated what is fundamentally new from what is an adaptation of existing retouching executors.

  • The paper states that learning such structured local execution requires step-by-step trajectories with spatial supports and color actions, which are difficult to collect from experts at scale. The reviewer agrees that this is indeed a key issue. However, the proposed solution relies on synthetic MaskGrade trajectories generated through controlled degradation. It is not fully convincing that these synthetic trajectories accurately reflect real expert local retouching workflows. The recovered trajectories may be numerically consistent with the constructed degradations, but that does not necessarily imply that they correspond to human editing processes.

  • Evaluation and human validation are limited. The work relies heavily on synthetic trajectories and model-generated instructions, while instruction adherence is also judged by a VLM-based metric, raising concerns about bias or circularity. The scale of the user study is small, with only ten participants, and their backgrounds are not reported. Since retouching quality often requires expertise to assess, the human evaluation is not sufficiently convincing.

  • Can the method support multiple semantic objects or multiple subject masks?

Questions for authors

See weaknesses above.

6 comments
AuthorsOpenReview ↗

Response to Reviewer wWEp (1/3)

Dear Reviewer wWEp,

Thank you for recognizing the motivation, the closed-form recovery, and the efficiency of our method. Our responses follow.

Q1: Positioning and novelty

Local retouching is not global retouching plus a mask: a global edit has one action, which an input–target pair supervises directly, while a local edit adjusts several regions differently and the target only constrains their combined effect. Step-by-step supervision is therefore needed.

Such supervision is scarce, as experts usually leave only the input and the final image; our core contribution addresses this. We make each executor action uniquely recoverable from two adjacent masked states; given an input, a target, and preset region masks, we can jointly solve for any trajectory that explains the change. With this, MaskGrade builds local stagewise trajectories at scale for stagewise process supervision.

VeraRetouch [1] also drives a differentiable executor with a VLM, but it targets global retouching and needs a pretrained renderer. Our VLM instead regresses color coefficients that are recovered in closed form for a fixed-basis executor, needs no separate renderer pretraining, and edits region by region through spatial supports.

We keep the Gaussian color representation of GLUT [2]. GLUT optimizes the Gaussian geometry together with the transforms, so recovering an action is non-convex, and only 21–42% of samples pass our recovery check. Freezing the geometry gives a unique closed-form solution, and the pass rate reaches 60–100%.

JarvisArt [3], JarvisEvo [4], and MonetGPT [5] edit with mature retouching software and are easy to interpret, but they are trained only with operation text or trajectory-level rewards as indirect supervision. We supervise every step: the intermediate state, spatial support, and color action of each stage. This raises stage alignment from 0.250 (endpoint-only supervision) to 0.281.

Per pixel, local adjustments fall into hue, luminance, and semantic types, which with the global stage give six stages. This division is not unique (see Point 2).


References

[1] VeraRetouch: A Lightweight Fully Differentiable Framework for Multi-Task Reasoning Photo Retouching

[2] GLUT: 3D Gaussian Lookup Table for Continuous Color Transformation

[3] JarvisArt: Liberating Human Artistic Creativity via an Intelligent Photo Retouching Agent

[4] JarvisEvo: Towards a Self-Evolving Photo Editing Agent with Synergistic Editor-Evaluator Optimization

[5] MonetGPT: Solving Puzzles Enhances MLLMs' Image Retouching Skills

AuthorsOpenReview ↗

Response to Reviewer wWEp (2/3)

Response to Reviewer wWEp (2/3)

Q2: Do synthetic trajectories reflect real expert retouching?

We do not claim that our trajectories replicate a photographer's workflow. We show instead that real expert local edits can be explained by them, and that what the model learns from them transfers to real expert edits.

To learn local retouching, a model first has to learn to use local retouching tools, which requires step-by-step trajectories. Our tools are the ones photographers use, such as the Color Range and Luminance Range masks in Lightroom and the Luma Range in Capture One [6][7]. Since real step-by-step records are scarce, MaskGrade builds, for each input–target pair, one executable explanation of the change with these tools. It need not follow a photographer's actual order; what matters is that every step is a real tool action and that the steps together reproduce the expert result.

Real expert edits can be explained this way, not only our own degradations. We collected real RAW photos edited by several experts from the Play Raw community on PIXLS.US [8] and solved a stagewise trajectory for each pair, and evaluated on pixels held out from fitting. On 615 pairs, the mean absolute error is 7.63/255 and the PSNR is 28.25 dB. On the 201 pairs with real local edits (96 source images), our stagewise recovery explains the expert result better than a single global fit:

SettingMean absolute error (0–255) ↓Mean PSNR ↑
Single global fit9.3126.69 dB
Our stagewise local recovery8.4227.36 dB

What the model learns transfers to real expert edits: on MIT-Adobe FiveK and PPR10K, the six-stage model is best on 9 of 10 reference fidelity metrics (Table 2).

The explanation is not unique, so the pipeline can be chosen: reversing the hue and luminance stages keeps the endpoint PSNR at 35.50 dB, a three-step Global → Midtones → Subject pipeline lowers the L1L_1 error to the target from 0.078 to 0.014 on 2,927 local samples, and a learned tool-selection policy chooses its own pipeline with 4.74 steps on average (Appendix D.2).

Q3: Reliability of the evaluation

We enlarged the user study from 10 to 20 participants and report their backgrounds (Figure 4, Appendix H.1):

  • Participants: 3 professional photographers, 7 computer vision researchers, and 10 photography enthusiasts.
  • Protocol: 50 ArtEdit requests that call for local edits; five methods shown anonymously in shuffled order, without the reference or method names; the best output chosen for instruction following, fidelity, and visual quality, with 1,000 choices per criterion.
  • Results: ours receives the most votes for visual quality (45.3%) and fidelity (32.7%), and the most for instruction following among retouching methods (15.1%); 19 of 20 participants preferred ours to Qwen-Image-Edit in visual quality.

We agree that a vision-language judge has its own preferences. However, our experiments show that the judges used here are reliable in two ways:

  1. They agree with human judgments. With the user-study inputs and criteria, five judges picked the best instruction-following output, 5 rounds per request. Their vote shares correlate positively with those of the 20 participants over 45 requests × 5 methods (Spearman 0.645–0.760; 0.744 for the main-table judge Gemini 3.1 Pro):
JudgeCorrelation with humans (average of 5 rounds)Single-round correlation (mean ± standard deviation)
Gemini 3.1 Pro (main-table judge)0.7440.653 ± 0.016
GPT-6-Astra0.7600.609 ± 0.028
GPT-6.1-Sol0.6960.550 ± 0.038
Opus-5.50.7400.694 ± 0.164
Gemini-4-Argon0.6450.590 ± 0.029
  1. Their scores are stable. Different judges rank the methods consistently: re-scored with GPT-6.1-Sol from another model family, each baseline's SC is 0.14–0.52 lower than with Gemini 3.1 Pro, but the rankings agree (Spearman 0.91; 25 of 28 method pairs in the same order; Appendix H.2, Table H1), and both rank Qwen-Image-Edit first, matching its 70.7% of instruction-following votes. Repeated runs also vary little: except for Opus-5.5, the standard deviation across rounds is 0.016–0.038, and averaging rounds raises every correlation.

References

[6] Adobe: Masking Tool in Lightroom Classic

[7] Capture One: Luma Range Overview

[8] PIXLS.US Play Raw Community

AuthorsOpenReview ↗

Response to Reviewer wWEp (3/3)

Q4: Multiple semantic objects and multiple subject masks

Our Subject head supports multi-subject masks. The Subject data include couples and two-person portraits whose reference mask is the union of all subjects, so when an instruction edits several subjects as one region (for example, "brighten the faces and clothes of the two men"), the head predicts their joint mask and one color action adjusts them together (Dice 93.7% on validation and 92.9% on test; Appendix D.4, Figure 17).

Our method is limited when different objects need different edits: each named object must then be segmented separately, and the original head returns the same mask for prompts naming different objects in one image. We trained an instruction-conditioned multi-target head with 75K prompts naming an object, a position, or a box. It switches targets with the prompt and outputs one mask per target in a single pass, but its Dice on compound prompts is only 45.36%:

Segmentation head evaluationSamplesResult
All validation prompts (IoU / Dice)4,23272.86% / 79.32%
Single-target / compound prompts (Dice)4,23280.53% / 45.36%
Target switching in the same image12 images / 64 pairsprobability difference 0.113; strict switching 21/64
Two targets in one pass (IoU)12 two-person images49.19% / 44.67%

We see two likely reasons:

  • Data scale: work on the same Qwen3-VL-4B usually uses large datasets; Qwen3-VL-Seg [9] trains its mask decoder on about 3 million samples built from 2 million SA-1B images, while our head uses 75K.
  • External segmentation prior: with similar or smaller data, such work often relies on a pretrained segmenter; GenSeg-R1 [10] uses about 7K samples but a frozen SAM 2, while our head learns from the vision-language model's features alone.

Even so, parametric masks already allow different adjustments in different regions, for example darkening a background in shadow while brightening a subject in highlight, or cooling a blue sky while warming green plants. They correspond to the Luminance Range / Color Range masks in Lightroom and the Luma Range in Capture One [6][7]. Unlike semantic masks, they follow pixel values and are self-feathering [11], need no object boundary, and avoid hard edges and artifacts. In future work, we will build trajectories with several semantic objects and study applying different edits to each.

Finally, we sincerely thank you for your careful and detailed review. Your comments are very constructive and helped us state the contribution and scope of this paper more clearly.


References

[6] Adobe: Masking Tool in Lightroom Classic

[7] Capture One: Luma Range Overview

[9] Qwen3-VL-Seg: Unlocking Open-World Referring Segmentation with Vision-Language Grounding

[10] GenSeg-R1: RL-Driven Vision-Language Grounding for Fine-Grained Referring Segmentation

[11] Luminosity Masks (T. Kuyper)

Reviewer_k6CkOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes ChiaroRetouch, a six-stage framework for instruction-guided photo retouching. A VLM predicts color-action parameters for Global, Hue, Shadows, Midtones, Highlights, and Subject. A compact executor applies these actions within the corresponding spatial supports to perform local edits at native resolution. The paper also introduces MaskGrade, a data construction pipeline that constructs and verifies stagewise trajectories with aligned intermediate images, spatial supports, and recoverable color actions. Experiments on multiple retouching benchmarks demonstrate the method’s effectiveness.

Critical strengths
  1. The paper provides an explicit retouching framework by separating spatial supports from color actions. This makes edits inspectable and enables intermediate supervision. The compact executor supports native-resolution editing.

  2. The paper also presents a stage-wise, scalable data construction pipeline for retouching. MaskGrade aligns intermediate images, spatial supports, and recoverable actions. The construction and verification pipeline provides structured training data and enables analysis of individual editing stages.

Critical weaknesses

Using the same checkpoint, single-stage execution outperforms all reference-fidelity metrics PQ, Q-Align, and DeQA, in Table 1. 6-stage execution improves instruction adherence and human preference. The paper provides limited analysis of these mixed results and when the additional stages are beneficial.

Questions for authors
  1. In Table 1, the 1-stage generally achieves better performance than the 6-stage. Do more stages introduce more errors in this benchmark? A stagewise analysis with examples would help clarify this.

  2. Could the authors explain why this particular six-stage order was chosen?

  3. Could smaller VLMs achieve similar results? If prompted similarly, how does the current model perform compared to affordable proprietary models like gpt 6 Luna?

5 comments
AuthorsOpenReview ↗

Response to Reviewer k6Ck (1/2)

Dear Reviewer k6Ck,

Thank you for recognizing how our method separates spatial supports from color actions, edits at native resolution, and builds stagewise trajectories with MaskGrade. For the three questions, we add an amplitude-matched comparison, a stagewise analysis with examples, order and tool-selection experiments, and results for two smaller VLMs and GPT-6 Luna.

Q1: Why does 1-stage score better in Table 1? Do more stages add errors, and when do they help?

More stages do not accumulate errors. 1-stage scores better on the reference metrics mainly because it changes the image less. The no-reference quality metrics differ only slightly (PQ −0.09-0.09, Q-Align −0.05-0.05, DeQA −0.04-0.04, ArtiMuse +0.23+0.23), and the extra stages help most on instructions with a local target.

We split each edit into a direction (where colors move) and an amplitude (how far). Under mean squared error, if the cosine between our edit and the reference edit is cos⁡θ\cos\theta, the error is smallest when our amplitude is cos⁡θ\cos\theta times the reference amplitude. One instruction allows many valid edits and the reference is only one of them, so this cosine is only about 0.48, and smaller edits tend to score better. For the main-table checkpoint, 1-stage and 6-stage edit in almost the same direction (cosine 0.927) and agree with the reference equally (0.480 vs. 0.471); 6-stage simply changes more to complete the local requests (15.71/255 vs. 13.09/255). Scaled to the same amplitude, the gap almost disappears:

Setting (ArtEdit-400, 391 input groups)ΔSC\Delta\text{SC} (6-stage −- 1-stage)95% intervalL1L_1 gap (×10−2\times10^{-2})
Original outputs+0.678+0.678[+0.517,+0.841][+0.517, +0.841]+0.144+0.144
Amplitude-matched−0.039-0.039[−0.212,+0.135][-0.212, +0.135]+0.056+0.056

So the gap comes from how much the image changes, not from errors added by more stages. On MIT-Adobe FiveK and PPR10K, with expert edits as reference, 6-stage is best on 9 of 10 fidelity metrics (Table 2).

Errors also do not build up over the stages. On the real intermediate images of the Full model (Appendix D.1, Figure 13), the five local stages lower the reference L1L_1 by 0.26×10−20.26\times10^{-2} (95% interval [0.02,0.50][0.02, 0.50]).

To see when the extra stages help, we split ArtEdit-400 by instruction text into local or mixed requests (270) and global-only requests (128). For the Full model, the local stages clearly lower L1L_1 on the former and leave the latter unchanged:

Instruction typeRequestsΔL1\Delta L_1 (×10−2\times10^{-2}) ↓95% interval
Global only128+0.082+0.082[−0.354,+0.524][-0.354, +0.524]
Local and mixed270−0.419-0.419[−0.697,−0.148][-0.697, -0.148]

Figure 6 of the revised paper shows stagewise examples from the main-table checkpoint. Their reference L1L_1 (×10−2\times10^{-2}, change from the previous stage in parentheses):

SampleGlobal+Hue+Shadows+Midtones+Highlights+Subject
Swan (blue-green tone, darker corners)4.924.36 (−0.56-0.56)4.71 (+0.35+0.35)5.95 (+1.24+1.24)8.24 (+2.29+2.29)8.00 (−0.24-0.24)
Frog (deeper shadows, frog stands out)9.459.88 (+0.43+0.43)13.91 (+4.03+4.03)17.84 (+3.93+3.93)22.06 (+4.22+4.22)20.96 (−1.10-1.10)
Leaves (warm tone, yellower leaves, more contrast)4.464.68 (+0.22+0.22)5.90 (+1.22+1.22)6.95 (+1.05+1.05)7.44 (+0.49+0.49)7.23 (−0.21-0.21)

Global sets the overall tone, and the local stages then carry out the regional requests. L1L_1 rises because of the requested changes: the frog instruction asks for deeper shadows, so the image keeps darkening from Shadows on. The final Subject stage lowers L1L_1 in all three samples, so it corrects earlier edits rather than adding errors.

AuthorsOpenReview ↗

Response to Reviewer k6Ck (2/2)

Q2: Why these six stages and this order?

Our executor applies a per-pixel color transform (Eq. 1). Per pixel, besides the Global stage, local adjustments fall into three types, hue, luminance, and semantics, matching the Hue, Shadows/Midtones/Highlights, and Subject supports. These are standard local tools: the Color Range, Luminance Range, and Select Subject masks in Lightroom, and the Luma Range in Capture One, which selects shadows, midtones, and highlights [1][2].

The order follows common practice: global tone first, then tonal ranges, then the subject; Adobe's Develop workflow also applies global adjustments before local ones [3]. On 64 validation pairs with the same closed-form fitter, reversing the hue and luminance stages keeps the endpoint PSNR at 35.50 dB, and swapping Shadows and Highlights changes SC by only −0.05-0.05 (Figure 9, Table C1). Moving Subject to the second position lowers PSNR by 0.66 dB, because later large-area adjustments change the edited subject again. So Subject goes last, and the order of the other stages matters little.

A fixed order is also not required. The same pipeline builds a three-step Global → Midtones → Subject trajectory, lowering the L1L_1 to the target from 7.79 to 1.41 (×10−2\times10^{-2}) on 2,927 local validation samples. We also trained a tool-selection policy that chooses the tools and the number of steps from the image and instruction. It uses 4.74 steps on average and reaches SC 7.69, above 6.74 for the fixed order with the same weights, though still slightly below the model trained with fixed stages (7.83), since it must also learn to choose supports.

Q3: Do smaller VLMs work? How does the model compare with GPT-6 Luna?

We trained Qwen3.5-2B and Qwen3.5-0.8B with the same recipe and inference protocol as the 4B model. We also gave GPT-6 Luna (low reasoning) white-box tools with global sliders and linear, radial, luminance-range, and color-range masks; each round it sees the input, the instruction, and the current rendering, and outputs all tool parameters, for up to 4 rounds. All results are on ArtEdit-400; SC (instruction adherence), PQ (content consistency), and O (their geometric mean) are scored by GPT-6.1-Sol.

First, replacing the Qwen3-VL-4B backbone with the newer and smaller Qwen3.5-2B:

ModelSC ↑PQ ↑O ↑PSNR (dB) ↑L1L_1 (×10−2\times10^{-2}) ↓ΔE00\Delta E_{00} ↓
Ours (Qwen3-VL-4B)7.839.778.6820.408.399.55
Ours (Qwen3.5-2B)7.899.898.7920.398.429.59

The 2B model matches 4B in PSNR and is slightly higher in SC, PQ, and O, so the newer backbone does not hurt.

Next, shrinking Qwen3.5-2B to 0.8B within the same family:

ModelSC ↑PQ ↑O ↑PSNR (dB) ↑L1L_1 (×10−2\times10^{-2}) ↓ΔE00\Delta E_{00} ↓
Ours (Qwen3.5-2B)7.899.898.7920.398.429.59
Ours (Qwen3.5-0.8B)7.759.878.6820.078.639.78

PSNR drops by 0.32 dB and SC and O by 0.15 and 0.11: some loss with less than half the parameters, but the metrics stay close.

Finally, the 4B model against GPT-6 Luna:

ModelInferenceSC ↑PQ ↑O ↑PSNR (dB) ↑L1L_1 (×10−2\times10^{-2}) ↓ΔE00\Delta E_{00} ↓
Ours (Qwen3-VL-4B)one pass7.839.778.6820.408.399.55
GPT-6 Luna + white-box tools≤4 rounds7.999.808.7621.217.779.04

With a single forward pass, our 4B model is 0.16 lower in SC and 0.08 lower in O than GPT-6 Luna with up to 4 rounds of feedback, with similar PQ (9.77 vs. 9.80); GPT-6 Luna has better reference fidelity (PSNR 21.21 dB). Our model is small, runs locally, and edits a 1K image end to end in about 0.35 s.

Best regards,The Authors


References

[1] Adobe: Masking Tool in Lightroom Classic

[2] Capture One: Luma Range Overview

[3] Adobe: Editing in the Develop Module in Lightroom Classic

AuthorsOpenReview ↗

Looking Forward to Your Feedback

Dear Reviewer k6Ck,

Thank you again for the time and care you put into reviewing our paper, and for your encouraging assessment of our work. Your questions led us to a much clearer analysis of the stagewise design.

In our response, we added an amplitude-matched comparison between 1-stage and 6-stage execution, a stagewise analysis with examples, experiments on the stage order, and results for two smaller VLMs and GPT-6 Luna. We would be truly grateful if you could let us know whether our response has answered your questions. If anything remains unclear, we would be glad to provide further clarification.

We understand how busy this period is, and we sincerely appreciate your time and thoughtful input.

Best regards, The Authors

Reviewer_qy4WOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ChiaroRetouch, a VLM-based framework for instruction-guided local photo retouching. By introducing a fixed-basis Gaussian color executor with closed-form action recovery, the method enables scalable synthesis of stage-wise editing trajectories and achieves competitive retouching quality with efficient, resolution-independent inference.

Critical strengths
  1. The paper decouples Gaussian basis learning from color coefficient prediction, enabling the VLM to predict color adjustments directly from an input image and text instruction.
  2. By fixing the pre-trained Gaussian basis, the color coefficients WW can be directly recovered from (input, ground truth) image pairs through a closed-form solution, providing explicit supervision for training the VLM without iterative optimization.
Critical weaknesses
  1. Insufficient details and ablations for the training objectives. The paper introduces a prototype InfoNCE loss but does not clearly specify the query features, prototype construction, or positive/negative sample definitions. Moreover, although the paper compares different stage-wise supervision objectives, it lacks ablations on the Color Code MSE and prototype InfoNCE losses.

  2. Limited flexibility in spatial region selection. Although the framework emphasizes decoupling WHERE and WHAT for local retouching, most spatial masks are predefined based on color or luminance, and only the Subject mask is predicted by the VLM. This may limit the model's ability to handle complex instructions requiring different adjustments to multiple semantic objects.

Questions for authors
  1. Could the authors provide more details on the prototype InfoNCE loss and ablate the Color Code MSE and InfoNCE losses?

  2. How does the proposed framework handle complex local editing instructions that require different adjustments to multiple semantic objects, given that only the Subject mask is learned?

4 comments
AuthorsOpenReview ↗

Response to Reviewer qy4W (1/2)

Response to Reviewer qy4W (1/2)

Dear Reviewer qy4W,

Thank you for the review. We now define the prototype InfoNCE in full (query, prototype bank, positives, and negatives), ablate codeMSE and InfoNCE with six models trained under the same setting, diagnose how each loss affects the local stages, and discuss multiple semantic objects with new segmentation experiments.

Q1: Training objectives and ablation

With the same data and random seed, we trained six models that use different combinations of the stagewise rollout pixel loss (Pixel) and the two auxiliary losses during local-stage training; the model with all three is Full.

Both auxiliary losses use a rendered-effect distance. Different parameters can render the same color change, so we compare two actions W1,W2W_1, W_2 by the color change they render on the current image ZkZ_k within support MkM_k: dk2(W1,W2)=∥diag⁡(Mk)Φ(Zk)(W1−W2)T∥F2d_k^2(W_1, W_2) = \Vert \operatorname{diag}(M_k) \Phi(Z_k) (W_1 - W_2)^{\mathsf{T}} \Vert_F^2

  • codeMSE: the effect distance dk2(W^k,Wk∗)d_k^2(\widehat W_k, W_k^{\ast}) between the predicted action W^k\widehat W_k and the teacher action Wk∗W_k^{\ast}, a direct regression target for each step.
  • Prototype InfoNCE: the query is W^k\widehat W_k, the positive is the closed-form teacher action Wk∗W_k^{\ast}, and there are 64 negatives [1][2][3].
    • Prototype bank: we represent each training action by its effect on a fixed grid of 17317^3 RGB colors, group the actions into 1,024 clusters with MiniBatch K-means, take the real action nearest each cluster mean as its prototype, and collect 32 real actions per prototype under the same distance.
    • 32 direction negatives: drawn from the prototypes nearest to the teacher's own (excluded), so they have similar but different effects.
    • 32 amplitude negatives: the teacher's effect scaled by factors in [0.5,0.8]∪[1.2,1.5][0.5, 0.8] \cup [1.2, 1.5], so both the strength and the direction of an edit matter.
    • Logits: −dk2(W^k,⋅)/(Ekτ)-d_k^2(\widehat W_k, \cdot)/(E_k \tau) over the 65 candidates, with teacher edit energy Ek=dk2(Wk∗,Wid)E_k = d_k^2(W_k^{\ast}, W_{\mathrm{id}}) and τ=0.1\tau = 0.1.

The six models share the backbone, initialization, and 6,000-update budget. Results on ArtEdit-400, with SC, PQ, and O scored by GPT-6.1-Sol:

ObjectivePixelcodeMSEInfoNCEPSNR ↑SC ↑PQ ↑O ↑
Pixel-only✅❌❌20.3717.5349.7808.492
codeMSE-only❌✅❌19.8747.8469.6798.647
InfoNCE-only❌❌✅20.1587.7699.5048.545
Pixel + codeMSE✅✅❌20.3947.6209.7778.538
Pixel + InfoNCE✅❌✅20.4207.5969.7878.549
Full✅✅✅20.4027.8269.7668.684

Full has the highest O of the six (8.684) and the highest SC among models with the Pixel term (7.826, +0.29 over Pixel-only), with similar PQ (9.766 vs. 9.777–9.787).

The contrastive term acts mainly on intermediate stages, where the pixel loss only weakly constrains the local actions. On 32 global chains, all four Pixel models already agree with the teacher (direction cosine about 0.90); the differences are in the local stages. On 32 held-out local chains:

ComparisonΔ\Delta direction cosine ↑Δ∣ratio−1∣\Delta\lvert\text{ratio}-1\rvert (amplitude error) ↓
Full −- (Pixel + codeMSE)+0.221-0.515
(Pixel + InfoNCE) −- Pixel-only+0.295-0.556
(Pixel + codeMSE) −- Pixel-only+0.069-0.067

InfoNCE raises the direction cosine by 0.221–0.295 and cuts the amplitude error by 0.515–0.556, so each local step moves closer to the teacher action.

AuthorsOpenReview ↗

Response to Reviewer qy4W (2/2)

Q2: Multiple semantic objects

In our original design, we assumed that most photos center on one salient subject, so we kept images with a clear subject for the Subject data. Our Subject head finds the salient subject well (Dice 93.7% on validation, 92.9% on test). The data also include couples and two-person portraits with the union of all subjects as the mask, so when an instruction edits several subjects as one region, the head predicts their joint mask (Appendix D.4, Figure 17).

Thank you for raising this point. When an instruction asks for different edits on different objects, each named object must be segmented separately, which our original design did not consider. We trained an instruction-conditioned multi-target head with 75K prompts naming an object, a position, or a box. It switches targets with the prompt and outputs one mask per target in a single pass, but its Dice on compound prompts is only 45.36%:

EvaluationSamplesResult
All validation prompts (IoU / Dice)4,23272.86% / 79.32%
Single-target / compound prompts (Dice)4,23280.53% / 45.36%
Target switching in the same image12 images / 64 pairsprobability difference 0.113; strict switching 21/64
Two targets in one pass (IoU)12 two-person images49.19% / 44.67%

We see two likely reasons. First, data scale: work on the same Qwen3-VL-4B usually uses large datasets; Qwen3-VL-Seg [4] trains its mask decoder on about 3 million samples built from 2 million SA-1B images, while our head uses 75K. Second, an external segmentation prior: with similar or smaller data, such work often relies on a pretrained segmenter; GenSeg-R1 [5] uses about 7K samples but a frozen SAM 2, while our head learns from VLM features alone.

Even so, the framework can already adjust different regions differently through parametric masks: a luminance mask can darken a background in shadow and brighten a subject in highlight, and a hue mask can cool a blue sky and warm green plants. These correspond to the Luminance Range / Color Range masks in Lightroom and the Luma Range in Capture One [6][7]. Unlike semantic masks, they follow pixel values and are self-feathering [8], so they need no object boundary, can work across semantic regions, and avoid hard edges and artifacts.

Your point that separating where to edit from what to apply should cover multiple objects marks the most valuable extension of our framework. In future work, we will build trajectories with several semantic objects and study applying different edits to each.


References

[4] Qwen3-VL-Seg: Unlocking Open-World Referring Segmentation with Vision-Language Grounding

[5] GenSeg-R1: RL-Driven Vision-Language Grounding for Fine-Grained Referring Segmentation

[6] Adobe: Masking Tool in Lightroom Classic

[7] Capture One: Luma Range Overview

[8] Luminosity Masks (T. Kuyper)

AuthorsOpenReview ↗

Looking Forward to Your Feedback

Dear Reviewer qy4W,

Thank you again for the time and care you put into reviewing our paper. Your comments on the training objectives and on multiple semantic objects helped us improve the paper considerably.

In our response, we defined the prototype InfoNCE loss in full, ablated the codeMSE and InfoNCE losses with six models trained under the same setting, and added new segmentation experiments on multiple semantic objects. We would be truly grateful if you could let us know whether our response has addressed your concerns. If anything remains unclear or unconvincing, we would be glad to provide further clarification.

We understand how busy this period is, and we sincerely appreciate your time and thoughtful input.

Best regards, The Authors

Dear Reviewer qy4W,

We extend our sincere gratitude to you for your valuable time and insightful feedback. We value your constructive feedback and hope that our responses have appropriately addressed all the concerns. We really appreciate the valuable time to respond to our feedback based on your comments. Further, we are happy to address any remaining concerns.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.