acceptodds
Under review as a conference paper at ICLR 2027

LatentCompass: T2I Diffusion Steering via Orthogonal Attribute Spaces for Debiasing, Concept Erasure, and Red Teaming

Abstract

Text-to-image (T2I) diffusion models suffer from biased results stemming from entangled generative priors and a lack of accurate control over outputs. Current mitigation attempts rely on imprecise, adversarially-vulnerable prompt and text embedding interventions, or they require prohibitive and invasive fine-tuning. Further, text-based methods can only control descriptive attributes, i.e., what an image depicts, but not evaluative attributes, i.e., how it is perceived by an external judge. We propose LatentCompass, an exemplar-based approach that enables disentangled and controllable generation for both descriptive and evaluative concepts in a training-free manner. LatentCompass steers the generative trajectory by (a) constructing a nonlinear, low-dimensional, and orthogonal attribute space via a closed-form solution that explicitly isolates desired concepts, (b) defining a target- class intervention in this space, and (c) reflecting the corresponding intervention directly in the T2I latent space. Extensive evaluations demonstrate that LatentCompass on average (i) mitigates generative stereotypes in medical professions by 100%, (ii) reduces unsafe concept generation to 1.3%, (iii) enhances aesthetic score by 27%, (iv) boosts red-teaming success rates against Deepfake detectors by 40%, and (v) enables high-fidelity attribute editing without leakage.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI Safety, Responsible AI, Diffusion Models, Text-to-Image Generative Models, Concept Steering

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_K73tOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

With this paper, the authors propose another inference-time steering method called LatentCompass for text-to-image generation. It differentiates between two types of attributes: "descriptive" attributes as properties that can be easily expressed in language, and "evaluative" attributes as properties that can only be evaluated after the completed image generation by an external judge. They make the argument that prior work that focuses on text-based steering struggles with controlling such evaluative properties. Therefore, they take an exemplar-based approach, i.e. they take a set of images for guidance, not just pieces of a prompt, to define the subspace the the steering-based control at inference-time. The paper evaluates LatentCompass on the suppression of explicit nudity content (descriptive) on established benchmarks, on debiasing gender (descriptive) and skin color (descriptive), on fooling synthetic-image detectors (evaluative), and on improving image aesthetic scores (evaluative).

Critical strengths
  • Broad and diverse demonstration of the use of LatentCompass.
Critical weaknesses
  • Substantial chunk of conceptually extremely similar prior work is entirely neglected as baselines and in the related work section discussion. The "conventional" steering approaches. I want to name a few examples here: CASteer (Gaintseva et al., ICLR 2026), SafeRoPE (Yang et al., CVPR 2026), and GrOCE (Han et al., CVPR 2026). Also careful prompt adjustments ("A female doctor with brown skin" and not just "A doctor"), negative prompting (steering away from "male"), or compositional prompting (using additional positive guidance vectors) is not evaluated against, despite them being very clear and simpler baselines.
  • The framing of the paper is a bit convoluted: two types of attributes to control (descriptive and evaluative) of which only one of them (the evaluative attributes) really counts as a selling point for using an exemplar-based approach, because the comparison on descriptive attribute control (suppression of nudity or debiasing) is not extensive enough itself due to missing baselines, while positioning as a full inference-time control over "evaluative attributes" would require a broader set of baselines itself as well. Currently, there is no baseline for debiasing and no baseline for steering towards more aesthethics. Both of them individually are interesting, but showing competitiveness for a method for both would probably better be done in to focused separate papers for more clarity and a deeper evaluation.
Questions for authors

Clarifications

  • Why do you consider UCE and RECE as training-free? They clearly manipulate the weights of a model before inference happens, especially RECE even for multiple consecutive closed-form updates.
  • What is the run-time overhead of LatentCompass? It is said to be closed-form but the preparation/generation of the exemplars needs to be taken into account.

Future Work

  • Can you compare against more recent, applicable baseline beyond SAFREE? Without such a comparison, the current claim in the short related work section on related concept-erasure methods that steer away from toxic subspace do "struggle with safeguarding against subversive harmful prompts" is unsubstantiated.
  • The main results used throughout the main paper are on SD v1.4, which is considered fairly outdated regarding its architecture and training paradigm.
Reviewer_QYX8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes LatentCompass, a training-free, exemplar-based framework for controllable text-to-image generation. It constructs a nonlinear, orthogonal attribute space using multi-scale Random Fourier Features and closed-form supervised projections, enabling control over both descriptive and evaluative attributes. The framework steers diffusion latents through a log-likelihood ratio objective and demonstrates promising results across bias mitigation, attribute editing, concept suppression, aesthetic enhancement, and synthetic image detector red teaming.

Critical strengths
  1. The paper addresses a practically meaningful problem by extending controllable generation beyond descriptive attributes to evaluative attributes, such as aesthetic quality and detector realness, which are difficult to control through conventional text prompts.
  2. LatentCompass provides a unified and flexible framework that supports multiple downstream applications without modifying the pretrained diffusion model. Its exemplar-based formulation allows different control objectives to be incorporated within the same steering framework.
  3. The experimental results are promising across diverse tasks and models. In particular, LatentCompass achieves strong performance in bias mitigation, attribute editing, nudity suppression, aesthetic enhancement, and synthetic image detector red teaming, demonstrating the broad applicability of the proposed framework.
Critical weaknesses
  1. LatentCompass does not remove target concepts from the pretrained model, but instead suppresses their generation through inference-time steering. The paper should distinguish concept suppression from model-level erasure or unlearning.
  2. Appendix G (p 30) claims that geometric orthogonality guarantees non-target attribute preservation. However, the proofs in Appendix I only establish the optimality of the projection and alignment procedures, without proving semantic disentanglement.
  3. Although LatentCompass uses closed-form attribute-space construction, it requires substantial exemplar preparation and iterative gradient-based optimization during inference, potentially limiting its practicality and scalability.
  4. The paper lacks comparisons with recent baselines, including SPEED [1], ScaPre [2], DP [3], and OCE [4]. Moreover, the contributions of nonlinear mapping, orthogonality, preservation guidance, and early stopping are not sufficiently disentangled through ablation studies.
  5. Appendix F shows that attribute separability nearly saturates with approximately 2K exemplars, yet the main experiments use 40K generated images. The necessity of such a large exemplar set remains unclear, as downstream steering performance is not evaluated across different exemplar sizes.
  6. The Gaussian class-conditional assumption is not empirically validated. Since the attribute coordinates are obtained through supervised nonlinear projections, existing Gaussian approximation results for random projections do not directly justify this assumption.
  7. The paper lacks systematic failure case analysis, particularly for correlated or conflicting attributes, unsuccessful steering, and unintended suppression of benign content.

[1] SPEED: https://arxiv.org/abs/2503.07392 (ICLR 2026)
[2] ScaPre: https://arxiv.org/abs/2601.06162 (ICLR 2026)
[3] DP: https://arxiv.org/abs/2604.10032 (CVPR 2026)
[4] OCE: https://arxiv.org/abs/2605.28902 (ICML 2026)

Questions for authors
  1. Can the authors theoretically justify why geometric orthogonality guarantees non-target attribute preservation? Please compare orthogonal and non-orthogonal projections, with and without the preservation term in Eq. (7).
  2. Table 3 reports only 3.2 seconds of additional inference time, while Appendix B.4 reports approximately 4.9 PGD iterations per timestep over 50 denoising steps. Under comparable settings, this implies roughly 13 ms per PGD iteration, despite requiring VAE decoding, feature extraction, and backward computation. Does the reported runtime include all these operations? Please provide a detailed runtime breakdown.
  3. Can the authors evaluate simultaneous multi-concept suppression and report how steering success, attribute leakage, inference time, and optimization iterations scale with the number of target concepts?
  4. Why are 40K exemplars used when Figure 13 shows nearly saturated AUROC with approximately 2K examples? Please evaluate downstream steering performance using smaller exemplar sets and report the preprocessing costs and storage requirements of the timestep-specific attribute encoders.
  5. Can the authors compare LatentCompass with OCE, SPEED, ScaPre, and DP under consistent evaluation settings? Please also provide ablations isolating the effects of nonlinear mapping, multi-scale RFF, orthogonality, preservation guidance, and early stopping.
  6. Can the authors provide failure cases involving strongly correlated or conflicting attributes, unsuccessful PGD optimization, and unintended suppression of benign content?
  7. Can the authors empirically validate the Gaussian class-conditional assumption and compare it with alternative density models? Additionally, how many black-box detector queries are required to construct the attribute space for SID red teaming?
Reviewer_1qxZOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces LatentCompass, which is a training-free method for enabling concept wise control in text-to-image models. The proposed method operates completely on the latent space of the diffusion model, and motivated by a close-form composition crafted by expernal samples. Furthermore, with the proposed orthogonal composition of the latent space using Random Fourier Features and an orthogonal alignment, the authors specify that the concepts that are intended to be edited can be modified with minimal intervention to the remaining concepts in the model. This disentaglement property is shown over multiple tasks in the paper, which are bias/stereotype mitigation, face editing, concept erasure, and as a tool for increasing the success rate of classifiers used in red teaming. The paper provides sufficient mathematical foundations with a clear objective and provides qualitative and quantitative results showing the effectiveness of the method.

Critical strengths
  • The foundations of the orthogonal composition proposed by the paper is provided sufficiently where detailed derivations are also provided in the supplementary material (Appendix I).
  • The paper shows the effectiveness of the proposed methodology in diverse but related tasks. Specifically, LatentCompass demonstrates its effectivess in tasks such as bias mitigation, attribute editing, red teaming and aesthetics improvements.
  • Quantitatively, the method appears to be effective with significant gains especially in bias mitigation and concept removal.
Critical weaknesses
  • The main concern on the paper is the weakness of some evaliation setups. As also mentioned in the questions section, the evaluation of the editing performance is questionable as the only measurement provided is the edit confusion matrix, where qualitative metrics on such edits are also not available. Additionally, the number of samples (important for FID measurements presented in Table 2), prompts used and MLLMs used for the quantitative results are missing, which makes it difficult to assess the success of the method in certain aspects.
  • The writing of the methodology can be further improved. While the notations appear coherent, the flow of the paper appears to be weaker than the rest of the paper in sections 3.1 and 3.2, which is a known effect of AI aided writing. The use of such tools does not violate any policy and also acknowledged by the authors, but a revision would make the paper stronger.
  • The paper has strong assumptions on a practical control mechanism which is summarized with three items R1, R2, R3 between lines L058-L064. While these constrains lead to LatentCompass, especially defining R3 as ideal refutes the existence of many editing methods that enable controlling the models over their baseline capacity. Furthermore, I believe that a revision in the itroduction accordingly would benefit the paper.
  • In Figure 9 of the appendix, for the skin tone mitigation it appears that the skin tone of the generated content is steered on a fixed direction and possibly creating a reverse bias, which can also be observed in the other biases investogated in the figure. Can th authors clairify if these are selected samples or does the model shift in the opposite direction (and creating another bias), which would help understanding the limitations of LatentCompass.
Questions for authors
  • In the introduction (L062), the authors state that an ideal control mechanism should keep the pre-trained weights fixed. While this is a sensible objective, it is a bold claim to state this as a necessity, as a method can both try to keep the model capabilities while enabling advanced control. While it is a different line of work, this statement refutes all adapter based methods enabling additional controls. Authors are advised to revise the statement.
  • Can the authors provide supplementary details on the edit evaliation presented in section 4.2? Currently the stamement mentions "we edit four facial attributes, {age, skin tone, perceived gender, expression}, and measure the corresponding attribute leakage using an external MLLM." (L344-346). For transparency, the details of the evaliuation setup and the editing prompts, along with supplementary examples would help understanding the types of the edits that can be performed and what is the disentanglement ratio, as this is some aspect of editing that is commonly mistaken or not accurately measured by external metrics.
  • For the quantitative results presented in Table 2, can the authors provide more details on the evaluation setup? especially for FID, it is widely known that the measurements are dependent on the number of samples, which would help interpreting the significance of the results (for example the difference with UCE).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.