acceptodds
Under review as a conference paper at ICLR 2027

Direct Latent Editing for VFM-Based Image Generators

Abstract

Modern text-to-image generators train a flow model in a compressed latent space defined by an encoder. Recent work replaces the standard VAE encoder with a frozen Vision Foundation Model (VFM) such as DINOv3 to exploit self-supervised semantic features. However, editing for these VFM-based generators has not kept pace with their generation quality. We apply four inference-time editors built for VAE-based generators to VFM-based generators, at their published or tuned settings, and find that they rarely reach the target attribute. We propose Direct Latent Editing (DLE), an inference-time editing method for fully VFM-based generators with a frozen encoder. DLE precomputes one editing direction per concept from flow-model samples. At edit time it adds this direction to the encoded latent and decodes; no flow-model call, inversion or weight update is needed. A single scalar gives continuous, signed control. On three VFM-based generators we trained (CelebA faces, MIMIC-CXR chest X-rays and ADNI-1 brain MRI), DLE reaches targets that the tested baselines do not ( vs. at most on CelebA; vs. at most AUROC on MIMIC-CXR), at lower per-edit flow-model and decoder time than FlowEdit at its official configuration.

Primary area:
generative models
Keywords:
Image editing, Vision foundation models, Flow matching

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nHUeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Direct Latent Editing (DLE) for image generators operating in the latent space of a frozen vision foundation model. For each concept, the method generates samples from positive and negative prompts using shared initial noise and averages their latent differences. Editing then consists of encoding an image, adding the precomputed direction, and decoding. No flow-model evaluation or inversion is required at edit time. The experiments cover CelebA, MIMIC-CXR, and ADNI-1 using separately trained generators. DLE substantially outperforms the adapted baselines on the reported target metrics within these models. The paper also studies alternative direction estimators, different encoders, prompt wording, and a matched VAE generator. Online editing is inexpensive, although each concept requires offline direction extraction.

Critical strengths

1.The method is simple and has a clear practical advantage when the same concept is used repeatedly. Once a direction has been computed, editing requires only an encoder pass, a latent addition, and a decoder pass. Appendix F provides useful timing details, including the precomputation cost and amortization point.

2.The CelebA improvement is not solely an artifact of measuring target accuracy. In Table 9, DLE achieves 65.4% successful edits that also pass the identity threshold, compared with 18.0% for FlowEdit in the authors’ VFM generator. The independent attribute classifier provides additional support for the observed changes.

3.The paper includes several useful controls and reports negative findings openly. Development-set tuning, independent evaluators, alternative encoders, prompt variations, and the unsuccessful transfer to SVG-T2I all help establish what the experiments do and do not show.

Critical weaknesses

Weaknesses

1. The methodological contribution is limited, and the closest alternatives are not compared convincingly.

Section 5 correctly acknowledges that latent translation is not new. However, the remaining claim about the “paired-endpoint estimator” is not particularly convincing. Equation (1) is exactly the difference between the positive and negative sample means:

d=1N∑i(zi+−zi−)=zˉ+−zˉ−.d=\frac{1}{N}\sum_i(z_i^+-z_i^-)=\bar z^+-\bar z^-.

Shared initial noise may reduce estimation variance, but the paper does not isolate that benefit through a comparison with independent-noise sampling. It also does not compare against a direction obtained from the mean encoded features of real positive and negative examples. That comparison would help establish whether the expensive flow-model sampling stage is necessary.

The comparison with SVM and PCA directions is also incomplete. Appendix B.2 specifies norm matching for the random control but does not clearly explain the scale convention for the SVM and PCA directions. Comparing them at the same numerical value of α does not necessarily compare equivalent edit strengths. The reported advantage over SVM should be tested with separately calibrated strengths and an identity-preserving success metric.

2. The results do not establish what makes DLE work beyond the authors’ own generators.

Appendix B.8 is particularly important. On the released SVG-T2I model, which also uses a frozen DINOv3 encoder, DLE reaches only 25.7% target accuracy, compared with 64.0% for FlowEdit. Joint target-and-identity success is low for both methods, at 9.1% and 6.9%. The authors acknowledge this limitation, but it substantially narrows the contribution.

The explanation in Section 3.4 does not resolve this discrepancy. Better linear separability and higher translation consistency support the possibility of additive editing; they do not explain why trajectory-based editors should fail. The VAE comparison changes the encoder, decoder, and trained flow model together, as the paper itself notes. It therefore cannot identify which component causes the observed behavior.

The successful datasets also have relatively standardized spatial layouts. Since DLE applies the same spatial tensor to every source image, robustness to changes in pose, position, and cropping would be informative. At present, the conditions under which the method should be expected to work remain unclear.

3. The baseline experiments do not adequately support the explanation for their failure.

The paper does tune several baselines, so the concern is not simply a lack of tuning. Rather, SDEdit and FlowEdit select the strongest temporal settings included in their respective search grids. FlowEdit is capped at nmax⁡=33n_{\max}=33 with T=50T=50, although its original formulation allows nmax⁡=Tn_{\max}=T and explicitly identifies longer editing paths with stronger edits. The SD3 operating point is not a methodological upper bound. A broader sweep is needed before attributing the near-identity outputs to VFM latent geometry. See FlowEdit, Section 5.1.

Source preservation also changes the interpretation of the VAE comparison. In Table 9, official FlowEdit achieves 86.0% target accuracy but only 15.4% joint target-and-identity success, below DLE’s 36.3%. The searched FlowEdit setting instead reaches 49.4% joint success. Thus, the claimed reversal between VFM and VAE generators depends substantially on the operating point and evaluation criterion. Curves showing target success against source preservation would be more informative than the selected accuracy comparisons.

4. The medical experiments mainly establish changes in classifier scores, rather than valid edits of the intended findings.

The headline CXR AUROC of 0.927 compares classifier scores for edits toward and away from a finding. It does not directly establish that the corresponding pathology has been introduced or removed correctly. This distinction matters because Table 19 reports a mean SSIM of 0.64 at α=3, and the qualitative examples show changes spread across substantial portions of the image.

The independent CXR classifiers are useful controls, but agreement between classifiers does not establish anatomical specificity. The cross-talk analysis is performed at α=1 on an earlier checkpoint, whereas the main results use α=3 on the final checkpoint. Appendix B.3 also states that structural preservation was not measured. The paper therefore lacks a direct assessment of selectivity at its main operating point.

The MRI evidence is weaker still. The evaluator achieves 42.5% three-class test accuracy, and its MCI discrimination is near chance. Including the MCI direction in the headline average makes that result difficult to interpret. The authors disclose these limitations and avoid clinical claims, but the reported scores remain insufficient evidence of reliable disease-related editing.

5. The CXR test protocol does not establish generalization to patients unseen during generator training.

Appendix A.1 states that the flow model and decoder were trained on a pool containing most of the evaluation sources. The patient-disjoint development/test split protects hyperparameter selection, but it does not make the test patients independent of generator training.

Comparing toward and away edits of the same source controls their shared starting point. It does not address whether the method works equally well on unfamiliar patients. Given the prominence of the CXR results, an evaluation on patients excluded from both flow-model and decoder training is needed. Without it, the scope of the claimed held-out performance is narrower than the main tables initially suggest.

Questions for authors

How much does shared-noise sampling improve the direction estimate compared with independent-noise sampling? How does DLE compare with mean-difference directions computed from encoded real images, and with SVM directions after separate strength calibration?

Do SDEdit and FlowEdit remain ineffective when their editing windows are extended beyond the current search boundaries? Please compare target success at matched source preservation.

What accounts for the failure on SVG-T2I? Which architectural or training conditions distinguish the successful generators from this model, and how sensitive is the fixed spatial direction to pose or spatial misalignment?

Can the medical experiments be repeated on patients excluded from generator training, with direct checks of anatomical preservation and a more reliable MRI evaluator? The MRI uncertainty estimates should also reflect variation across subjects, rather than only resampling the direction-extraction pool.

Please clarify the sign convention in the CelebA evaluation. Appendix A.6 says negative edits are evaluated against negative targets, whereas Appendix B.2 describes negative α as a wrong-direction control. An explicit definition of how source labels, target labels, and α determine each reported accuracy would resolve this ambiguity.

Reviewer_2ziLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes Direct Latent Editing (DLE), an efficient inference-time image editing method for flow-based image generators that use frozen Vision Foundation Models (VFMs) instead of conventional variational autoencoders (VAEs). Motivated by the limited effectiveness of existing diffusion and flow-based editing methods in VFM latent spaces, DLE precomputes concept-specific editing directions by averaging the latent differences between flow-generated samples conditioned on positive and negative prompts with shared initial noise. During inference, these directions are directly added to the encoded image latent with a scalar controlling editing strength, eliminating the need for flow-model sampling, inversion, or parameter updates. Experiments on three domains—CelebA facial attributes, MIMIC-CXR chest radiographs, and ADNI-1 brain MRI—demonstrate substantial improvements over SDEdit, SEGA, Concept Sliders, and FlowEdit, achieving 82.0% editing accuracy on CelebA and AUROCs of 0.927 and 0.878 on MIMIC-CXR and ADNI-1, respectively. DLE also achieves a 203× reduction in per-edit flow-model and decoder computation time compared with FlowEdit.

Critical strengths
  • Effective and computationally efficient editing in VFM-based generators. The paper addresses an important limitation of existing image editing methods when applied to generators using frozen Vision Foundation Model (VFM) encoders. DLE achieves substantial improvements over established baselines, including SDEdit, SEGA, Concept Sliders, and FlowEdit, across facial images, chest radiographs, and brain MRI. Notably, it achieves 82.0% editing accuracy on CelebA compared with at most 48.3% for the baselines, while reducing per-edit flow-model and decoder computation time by 203× relative to FlowEdit. These results demonstrate a compelling practical advantage in both editing effectiveness and computational efficiency.

  • Insightful investigation of the relationship between latent representations and editing mechanisms. Beyond empirical performance, the paper provides useful insights into why direct additive editing can be effective in VFM latent spaces. Through linear probing, translation-consistency analysis, and comparisons with matched VAE-based generators, the authors demonstrate that VFM representations exhibit stronger linear semantic separability and are more amenable to concept-specific latent translations. The ablation studies further show that the proposed paired mean-difference estimator outperforms PCA- and SVM-based alternatives. These findings provide empirical support for the proposed design and highlight how the choice of latent representation influences the effectiveness of different editing strategies.

Critical weaknesses
  • Limited generalizability and insufficient evidence of effectiveness on independently developed generators. The main experimental improvements are demonstrated on three VFM-based generators trained by the authors, all following a similar architecture with frozen DINO-family encoders and domain-specific decoders. However, Appendix B.8 reports substantially weaker results on the independently developed SVG-T2I generator, where DLE achieves only 25.7% editing accuracy compared with 64.0% for FlowEdit. This directly challenges the broader applicability of the proposed approach and suggests that its effectiveness may depend heavily on the particular latent representations, decoder architectures, or training configurations. Without stronger results on independently developed, large-scale VFM-based generators, it remains unclear whether DLE offers a generalizable editing solution rather than a method tailored to the authors' experimental setup.

  • Oversimplified global editing direction and insufficient control over concept disentanglement. The proposed method assumes that a single concept-specific translation can be universally applied to different source images:

    dfm[c]=1N∑i=1N(z0,pos(i)−z0,neg(i)),d_{\mathrm{fm}}[c] = \frac{1}{N} \sum_{i=1}^{N} \left( z_{0,\mathrm{pos}}^{(i)} - z_{0,\mathrm{neg}}^{(i)} \right),

    zedit=z+α dfm[c].z_{\mathrm{edit}} = z + \alpha \, d_{\mathrm{fm}}[c].

    However, the optimal editing direction may depend on the source image, its semantic context, and interactions between correlated attributes. Since dfm[c]d_{\mathrm{fm}}[c] is independent of the input zz, DLE cannot adapt its transformation to source-specific characteristics. Moreover, a fixed additive direction offers no explicit constraint to preserve unrelated attributes. This limitation is supported by the translation-consistency analysis in Appendix B.4, where a single translation explains only 35.1% of the paired-difference energy in VFM latent space on average, leaving substantial variation unexplained. Appendix B.3 further reports semantic cross-talk between clinical findings, such as cardiomegaly and edema. These observations suggest that the global linear approximation may be insufficient for complex, context-dependent editing tasks, potentially compromising semantic disentanglement and source fidelity.

  • Insufficient evaluation of editing fidelity and preservation of clinically meaningful structures. The quantitative evaluation relies primarily on automated classifiers, including CLIP, CheXzero, and ADNI-DN121, which measure whether edited images shift toward the intended semantic category but do not necessarily establish that the edits are visually faithful or anatomically plausible. This concern is particularly consequential for the medical imaging experiments, where changes in classifier confidence may arise from altered image statistics rather than realistic modifications of pathological structures. Although the authors provide qualitative results and SSIM measurements, they do not systematically evaluate anatomical preservation or clinical validity. For example, Appendix B.3 explicitly acknowledges that structural preservation was not measured, while Appendix A.6 reports that the ADNI-DN121 evaluator achieves only 42.5% three-class accuracy on its held-out test set. Consequently, the reported improvements in editing AUROC provide limited evidence that DLE generates clinically meaningful counterfactual images while preserving unrelated anatomical information.

Questions for authors
  • Generalizability to independently developed generators (Clarification). Appendix B.8 reports that DLE achieves only 25.7% editing accuracy on SVG-T2I, substantially below FlowEdit's 64.0%. What explains this performance reversal despite SVG-T2I also using a frozen DINOv3 encoder? Is the discrepancy primarily attributable to differences in the decoder, flow-model training, or learned latent distributions? Can the authors provide further analysis to establish the conditions under which DLE is effective?

  • Methodological novelty (Clarification). The proposed formulation closely resembles established latent arithmetic methods, where a concept direction is estimated through differences between latent representations. Beyond applying paired mean-difference estimation at the endpoint of a VFM-based flow generator, what is the fundamental methodological contribution that distinguishes DLE from existing linear latent editing approaches? Can the authors provide stronger evidence that the paired flow-endpoint estimator offers advantages over conventional mean-difference directions computed directly from encoded real images?

  • Validity of the global linear editing assumption (Clarification). DLE applies a source-independent editing direction, (z_{\mathrm{edit}}=z+\alpha d_{\mathrm{fm}}[c]). However, Appendix B.4 reports that the estimated direction explains only 35.1% of the paired-difference energy on average, suggesting considerable source-dependent variation. Can the authors investigate whether this unexplained variation affects editing consistency across different source images? In particular, how does the global direction compare with source-adaptive directions derived from locally similar samples or conditional latent transformations?

  • Editing fidelity and clinical validity (Clarification). The medical imaging experiments primarily evaluate editing effectiveness through classifier-based AUROC, while structural preservation and clinical plausibility are not systematically assessed. How can the authors establish that the observed score improvements correspond to realistic pathological changes rather than classifier-sensitive image perturbations? Could the authors provide quantitative anatomical-preservation measurements or expert assessments of the edited medical images to substantiate their claims of meaningful semantic editing?

Reviewer_HVj3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an image editing approach Direct Latent Editing (DLE), which works on generators with a frozen vision foundation model as the encoder. It builds on the idea that semantic concepts can be linearly interpolated in this latent space, and computes an editing direction from the difference between average latents generated with positive and negative prompts. This edit direction is then added to an input image’s encoded features before decoding, allowing control over editing strength without running the flow model for every edit. Experiments on faces, X-rays, and MRI show improvements over the evaluated editing baselines based on the paper’s trained generators.

Critical strengths
  1. Efficient image editing is a relevant problem, and the proposed approach allows a precomputed concept direction to be reused across images with low computational cost.
  2. The paper evaluates multiple domains and provides ablations on direction estimation and editing strength.
Critical weaknesses

It would be helpful for readers if the following points were clarified or improved:


  1. The main results rely on generators trained by the paper, but their generation quality is not established. On the released SVG-T2I model whose generation quality has been established, DLE gets lower scores compared with FlowEdit baseline. It would be helpful to explain this difference and provide stronger evidence on established VFM-based generators, maybe RAEv2 (Improved Baselines with Representation Autoencoders) or SVG-T2I with suitable editing tasks, to understand whether the benefits generalize beyond their own generator models.


  2. It would also be helpful to include some qualitative editing examples from SVG-T2I in the appendix. Maybe showing the source images alongside DLE and FlowEdit outputs could help readers understand the quantitative results and better compare editing quality and preservation of the original image.


  3. Minor typo, The Ethics and Reproducibility statement contains two unresolved “Appendix ??” references.


The approach offers efficient editing, but the applicability beyond the authors’ generators need stronger evidence.

Questions for authors

Please refer to Weakness section for questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.