- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper discusses the role of sparse autoencoder (SAE) features in concept editing for diffusion models. It identifies a set of features, scaffold features, that activate in many diverse prompts without necessarily corresponding to similar semantic concepts. It shows that these features respond strongly to target-concept edits and that blocking their responses can disrupt image structure and semantic editing. Building on this, the paper proposes Scaffold Coordinated Feature Editing (SCFE), a training-free method that estimates prompt-specific scaffold responses and applies constrained updates to selected scaffold features alongside the target concept intervention. Experiments on SDXL-Turbo and FLUX.1 demonstrate improvements in editing and image preservation compared with concept-only SAE steering.
Critical strengths
-
The identification of highly recurrent SAE features that influence concept editing despite lacking consistent semantic similarity is interesting. The paper highlights the potential importance of features beyond those directly associated with a target concept.
-
The response-blocking experiments provide evidence that scaffold features influence editing outcomes. Matching ordinary features by the number of coordinates and response strength makes these comparisons more convincing than activation-correlation analysis alone.
Critical weaknesses
-
SCFE includes cross-prompt feature frequency analysis, finite-difference response estimation, and constrained activation updates. The underlying observation that SAE features selected by semantic similarity or activation statistics are not necessarily optimal steering variables has been explored in prior work [1]. Similarly, [2] establishes that direct SAE interventions can introduce out-of-distribution activations and artifacts. SCFE builds on these existing observations by selecting additional responsive features (scaffolds) and applying bounded updates with target-concept interventions. The resulting approach appears to be an incremental combination of established ideas rather than a substantially novel editing methodology.
-
The authors identify scaffold features using a 90% cross-prompt activation threshold but subsequently select features based on response magnitude and directional consistency (sign). It remains unclear whether the improvements arise specifically from cross-prompt recurrence or simply from selecting the most influential features with stable responses (same sign across prompts). Although responsive non-scaffold features are included as a control, the experiments do not fully isolate these two properties. The need and advantage of the proposed scaffold-selection process over more direct simple influence-based feature selection remain unclear.
-
The main experiments primarily compare SCFE against concept-only editing and internal controls such as random-feature editing, responsive non-scaffold editing, prompt mismatch, and spatial shift. However, comparisons with relevant baselines are missing, such as Conditioned Activation Transport, OrthoEraser, or other methods addressing coupled representations and unintended editing effects.
-
SCFE requires prompt-specific finite-difference response estimation, with two additional probes per concept coordinate and denoising timestep. The paper states that SCFE performs complete diffusion trajectories for these probes, needing substantially more model evaluations than concept-only editing. Although responses can be reused across editing strengths for the same prompt and initial noise, the computational overhead remains significant. It is unclear whether the improvements justify it, particularly when using efficient models such as SDXL-Turbo and FLUX.1.
-
The quantitative evaluation covers only two editing concepts per diffusion architecture, with additional object-specific concepts only evaluated qualitatively. It is difficult to infer generalization across editing tasks. LPIPS, DINOv2 distance, and CLIP-I primarily measure similarity to the original image rather than preservation of specific non-target content or structural integrity. The semantic evaluation relies on GPT-5.6-Sol without any human evaluations. Given that the main claim is about improving semantic editing while preserving image content, human evaluations and analysis using image preservation and metrics like ImageReward, VQA, etc. (commonly used in editing and composition works) are needed.
[1] Arad, D., Mueller, A., & Belinkov, Y. (2025, November). Saes are good for steering–if you select the right features. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (pp. 10252-10270).
[2] Cassano, E., Renzulli, R., Ahmed, R., Grangetto, M., & Alaniz, S. (2026, September). Look But Don’t Touch with Sparse Autoencoders for Unlearning in Diffusion Models. In European Conference on Computer Vision (pp. 586-603). Cham: Springer Nature Switzerland.
Questions for authors
- Does SCFE improve the semantic-editing versus image-preservation trade-off?
The authors compare at identical target-concept intervention strengths, but SCFE additionally modifies scaffold features, so it is not a fair comparison. Could the authors clarify whether the improvements persist when comparing methods at matched semantic-editing success rather than matched intervention coefficients? A preservation-versus-semantic-success Pareto comparison would help distinguish genuine improvements from changes in effective editing strength.
- How reliable is the local response approximation under strong intervention strengths?
SCFE estimates scaffold responses using small finite-difference perturbations and extrapolates these estimates to stronger concept edits, where nonlinear effects and changes in SAE activation support may become significant. Since the largest improvements occur under strong editing, how sensitive is the method to deviations from this local approximation?
- Is prompt-specific response estimation essential?
The authors demonstrate that using another prompt's response degrades performance, but this does not necessarily establish the need for separate response estimation for every prompt. Could a globally averaged response estimated during calibration achieve comparable results? This would be particularly relevant given the inference-time overhead.
8 comments
W3 Comparisons with relevant external methods, including Conditioned Activation Transport and OrthoEraser, are missing.
Thank you for raising this issue. We already discuss this comparison principle in Appendices I.1 and I.2 of the original manuscript. SCFE enhances an existing SAE editing interface. We evaluate its added benefit while retaining the base diffusion model, SAE, and target-concept intervention.
Our method survey examined several related approaches. CAT [1] learns conditioned activation transport for safety. OrthoEraser [2] projects erasure directions to protect coupled benign features. SAEdit [3] edits text-token embeddings, while SAEmnesia [4] reshapes concept-feature mappings through supervised SAE training. The original approaches differ in their task objectives, editing interfaces, or feature representations. Absolute editing performance also depends on the diffusion model, SAE dictionary, and target features. OrthoEraser [2] explicitly notes in Appendix C that its precision is limited by the underlying SAE's dictionary capacity and reconstruction fidelity. We evaluate the benefit of adding SCFE to a fixed base system. Comparing final scores across different base systems mixes these system differences with SCFE's gains, making it difficult to isolate the scaffold updates' contribution.
We implement Concept-only using feature interventions supported by the public SAEs [5], [6]. We compare it with its SCFE-enhanced version within each model-SAE configuration. Our cross-model evaluation follows the same paired design. This directly tests our contribution: whether scaffold responses improve an existing concept edit.
[1] Chrabąszcz, M., Szymczyk, A., Dubiński, J., Trzciński, T., Boenisch, F., & Dziedzic, A. (2026). Conditioned activation transport for T2I safety steering. arXiv preprint arXiv:2603.03163.
[2] Shi, C., Wu, W., Shen, F., Zhu, X., Hu, K., & Wang, Z. (2026). OrthoEraser: Coupled-neuron orthogonal projection for concept erasure. arXiv preprint arXiv:2603.11493.
[3] Kamenetsky, R., Dorfman, S., Garibi, D., Paiss, R., Patashnik, O., & Cohen-Or, D. (2026, September). SAEdit: Token-level control for continuous image editing via sparse autoencoder. In European Conference on Computer Vision (pp. 359-377). Cham: Springer Nature Switzerland.
[4] Cassano, E., Renzulli, R., Nurisso, M., Zaffaroni, M., Perotti, A., & Grangetto, M. (2026). SAEmnesia: Erasing concepts in diffusion models with supervised sparse autoencoders. In Forty-third International Conference on Machine Learning (ICML).
[5] Surkov, V., Wendler, C., Mari, A., Terekhov, M., Deschenaux, J., West, R., Gulcehre, C., & Bau, D. (2025). One-step is enough: Sparse autoencoders for text-to-image diffusion models. In Advances in Neural Information Processing Systems (Vol. 38, pp. 99956-100027). Curran Associates, Inc.
[6] Shabalin, S., Panda, A., Kharlapenko, D., Ali, A. R., Hao, Y., & Conmy, A. (2025). Interpreting large text-to-image diffusion models with dictionary learning. arXiv preprint arXiv:2505.24360.
W4/Q3 The cost of per-prompt response estimation and whether a global mean can replace it need clarification.
Thank you for raising this issue. The newly added Table F.1 in Appendix F.5 reports first-image latency for the current implementation on a single H20, with final-image batch size 1. For SDXL-Turbo, Concept-only takes 0.721 s, response estimation takes 1.056 s, and SCFE takes 1.787 s in total. For FLUX.1-schnell, the corresponding times are 0.947 s, 1.238 s, and 2.188 s. We also explain in Appendix F.5 that SCFE's execution can be optimized. Our current implementation shares the unedited trajectory prefix and stops probe branches at the scaffold-response readout. For the same-step paths used in the main experiments, we batch the reference trajectory and positive and negative probes. Response maps can also be reused across strengths when the prompt, initial noise, and concept are unchanged. For edits requiring stronger semantic changes and good image preservation, we consider the measured additional latency and editing improvements a meaningful trade-off.
We also test the globally averaged responses you suggested in the newly added Appendix I.7. On SDXL-Turbo, the global mean's online first-image latency is 0.732-0.733 s, compared with 1.787 s for per-prompt SCFE. These online times exclude one-time offline calibration and mean construction. The results show that global means retain some editing gains, while per-prompt SCFE achieves better semantic change and image preservation. We believe these additional gains make per-prompt response estimation the better default for SCFE. For latency-sensitive applications, your proposed global mean also has practical potential and merits further exploration as a lightweight alternative.
W5 Concept coverage is limited, and evidence for semantic editing and non-target preservation needs human assessment and evaluation using more metrics.
We agree that the original quantitative evaluation covered too few concepts. In the newly added Appendix J, we evaluate subject blue, knit texture, and container slenderness on SDXL-Turbo, and subject blue, eyeglass addition, and metallic appearance on FLUX.1-schnell.
Broader testing was previously limited by whether generated images contained a visible, editable target. Section J.1 adds automated screening: GPT-5.6-Sol first screens prompts, then checks target visibility and edit suitability in unedited images generated from those prompts. Selection uses neither edited outputs nor method scores or performance differences. Tables J.1 and J.2 show gains in both target semantics and image preservation across the added tasks. These results show that SCFE generalizes well across different concepts and edit types.
We also agree that GPT-based semantic evaluation requires human validation. We have added blinded human ratings for the original concepts in Appendix I.6. Raters independently judge target-directed change, with method labels, editing strengths, and GPT-5.6-Sol scores hidden. Table I.1 shows strong agreement between human and GPT rankings. Human ratings also support SCFE's semantic gains, including for threatening appearance. These results support our use of GPT-5.6-Sol for large-scale target-semantic evaluation.
We assess whether the suggested metrics directly measure target-change magnitude. ImageReward [1] predicts human preferences from an image and prompt. VQA-based TIFA [2] and VQAScore [3] evaluate text-image faithfulness through answer accuracy or probability. These scores depend on the input text, questions, and evaluator, rather than directly measuring change relative to the original image. For example, suppressing blue may reduce faithfulness to an original prompt describing a blue object. We could rewrite the target prompt for post-edit faithfulness scoring, but this introduces unnecessary wording choices and potential human bias. These metrics also do not directly compare non-target content before and after editing. We instead combine human-validated image-pair semantic scores, preservation metrics, and paired visualizations to assess target change and preservation of existing content.
[1] Xu, J., Liu, X., Wu, Y., Tong, Y., Li, Q., Ding, M., Tang, J., & Dong, Y. (2023). ImageReward: Learning and evaluating human preferences for text-to-image generation. In Advances in Neural Information Processing Systems (Vol. 36, pp. 15903-15935). Curran Associates, Inc.
[2] Hu, Y., Liu, B., Kasai, J., Wang, Y., Ostendorf, M., Krishna, R., & Smith, N. A. (2023, October). TIFA: Accurate and interpretable text-to-image faithfulness evaluation with question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 20406-20417). IEEE.
[3] Lin, Z., Pathak, D., Li, B., Li, J., Xia, X., Neubig, G., Zhang, P., & Ramanan, D. (2025). Evaluating text-to-visual generation with image-to-text generation. In Computer Vision - ECCV 2024 (pp. 366-384). Cham: Springer Nature Switzerland.
Q1 Equal intervention coefficients do not replace matched semantic success, and the advantage in the semantic-preservation trade-off needs validation.
We agree that SCFE changes additional coordinates, so equal target coefficients are not equivalent to equal semantic success. Table 1 controls the target injection to isolate the added update, but that control alone does not answer the trade-off question. The newly added Fig. 6 uses Pareto curves to compare achieved target-semantic scores against LPIPS across intervention strengths.
The comparison covers SDXL-Turbo and FLUX.1-schnell, including enhancement and suppression for the original concepts. Over most of the semantic range shared by the methods, SCFE lies below Concept-only: it attains lower image deviation at comparable target change. The newly added Pareto curves provide strong evidence that SCFE can achieve comparable semantic change while better preserving the original image content.
Thank you for reviewing our paper and for your questions and suggestions. We use WX, MWX, and QX to label Weaknesses, Minor Weaknesses and Questions respectively. X indicates the order within each category, and / connects related items addressed together. We have added experiments, clarified methodological details, and expanded the discussion in response to these comments, and have uploaded the revised manuscript PDF. Below, we address the comments one by one with reference to the revised manuscript. Unless otherwise stated, all section, appendix, figure, and table references are to the revised manuscript.
W1 SCFE's substantive differences from prior research and its independent contribution remain unclear.
Thank you for raising this issue. Overall, our contribution is the discovery and validation of a population of scaffold features that recur across prompts and have content-dependent effects. We validate their functional role in concept editing through response-blocking experiments with matched controls. We build SCFE on these findings, using concept-induced responses for the current prompt to guide bounded scaffold updates.
Arad et al. [1] distinguish features' concept-detection capabilities from their direct steering capabilities and select steering features based on their output effects. We study how a shared feature population participates in realizing target concepts. They explicitly state in their Limitations section that their method uses a single SAE feature for steering, while multi-feature interactions remain an open challenge. We retain the target-concept intervention, test scaffold responses with matched controls, and use these responses for joint editing. Our contribution concerns discovering this shared population, validating its function, and using it for editing.
The study by Cassano et al. [2] differs fundamentally from ours in both its starting point and objective. Regarding the starting point, Cassano et al. [2] start from distribution shifts and artifacts induced by direct SAE interventions. They propose PER, which uses detections of a specified concept to locate targets and replaces the corresponding patch embeddings in intermediate feature maps to erase that concept. We start from the discovery and functional validation of the distinct scaffold-feature population. We propose SCFE, a response-guided joint-update method, to improve semantic change and image preservation in concept editing. Regarding the objective, concept erasure is one type of concept editing. PER aims to erase specified concepts, while our SCFE addresses broader concept editing, covering target-concept enhancement and suppression.
Overall, our work turns an experimentally validated population-level finding into an editing method. We believe that the empirical discovery of this shared population and its functional relationships fundamentally distinguishes SCFE from a combination of existing techniques.
[1] Arad, D., Mueller, A., & Belinkov, Y. (2025, November). SAEs are good for steering - if you select the right features. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (pp. 10241-10259). Association for Computational Linguistics.
[2] Cassano, E., Renzulli, R., Ahmed, R., Grangetto, M., & Alaniz, S. (2026, September). Look but don't touch with sparse autoencoders for unlearning in diffusion models. In European Conference on Computer Vision (pp. 586-603). Cham: Springer Nature Switzerland.
W2 The role of high-reuse selection has not been sufficiently separated from response strength and sign consistency.
Thank you for raising this issue. In the newly added Appendix G.6, we first ablate the 90% cross-prompt prevalence threshold. Lowering the threshold admits more strong-response non-scaffold features into editing. Image preservation worsens, while semantic change shows no consistent improvement.
This trend leads directly to your question: can response magnitude and sign consistency replace scaffold preselection? We test this with Response-only in the latter half of Appendix G.6. Response-only selects stronger, directionally consistent responses, yet both mean semantic change and image preservation worsen.
These results directly support retaining scaffold preselection. We combine cross-prompt reuse with response-based selection to achieve a better semantic-change versus image-preservation trade-off with SCFE.