acceptodds
Under review as a conference paper at ICLR 2027

Don't Ignore Them: Scaffold Features Matter for Stable SAE Steering in Diffusion Models

Abstract

Sparse autoencoders (SAEs) enable concept editing in diffusion models by manipulating internal features, and existing methods typically select editing targets based on the target semantics. In contrast, we identify a class of features based on reuse across prompts, without relying on target semantics. Their effects vary with the generated content and are difficult to characterize with a single concept, yet they respond strongly to concept editing. Blocking their responses impairs editing results more than blocking the responses of matched ordinary features. These results show that these features are not high-frequency background unrelated to the target edit, but important internal variables that carry shared generative context. We therefore call them scaffold features. We further incorporate scaffold features into editing and propose Scaffold-Coordinated Feature Editing (SCFE), a method built on existing SAEs that requires no additional training. Using small concept interventions, SCFE estimates spatial responses for the current prompt and adds constrained scaffold-feature updates along the corresponding directions while manipulating the target concept. Experiments show that, under stronger editing, SCFE both increases target-semantic change and improves overall image preservation, and these gains generalize across diffusion-model architectures and their associated SAEs. These findings show that a feature's value for concept editing is not limited to the semantics it represents, as scaffold features selected without relying on target semantics can also improve targeted editing.

Primary area:
interpretability and explainable AI
Keywords:
diffusion models, sparse autoencoders, mechanistic interpretability, activation steering, concept editing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_21dhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper discusses the role of sparse autoencoder (SAE) features in concept editing for diffusion models. It identifies a set of features, scaffold features, that activate in many diverse prompts without necessarily corresponding to similar semantic concepts. It shows that these features respond strongly to target-concept edits and that blocking their responses can disrupt image structure and semantic editing. Building on this, the paper proposes Scaffold Coordinated Feature Editing (SCFE), a training-free method that estimates prompt-specific scaffold responses and applies constrained updates to selected scaffold features alongside the target concept intervention. Experiments on SDXL-Turbo and FLUX.1 demonstrate improvements in editing and image preservation compared with concept-only SAE steering.

Critical strengths
  1. The identification of highly recurrent SAE features that influence concept editing despite lacking consistent semantic similarity is interesting. The paper highlights the potential importance of features beyond those directly associated with a target concept.

  2. The response-blocking experiments provide evidence that scaffold features influence editing outcomes. Matching ordinary features by the number of coordinates and response strength makes these comparisons more convincing than activation-correlation analysis alone.

Critical weaknesses
  1. SCFE includes cross-prompt feature frequency analysis, finite-difference response estimation, and constrained activation updates. The underlying observation that SAE features selected by semantic similarity or activation statistics are not necessarily optimal steering variables has been explored in prior work [1]. Similarly, [2] establishes that direct SAE interventions can introduce out-of-distribution activations and artifacts. SCFE builds on these existing observations by selecting additional responsive features (scaffolds) and applying bounded updates with target-concept interventions. The resulting approach appears to be an incremental combination of established ideas rather than a substantially novel editing methodology.

  2. The authors identify scaffold features using a 90% cross-prompt activation threshold but subsequently select features based on response magnitude and directional consistency (sign). It remains unclear whether the improvements arise specifically from cross-prompt recurrence or simply from selecting the most influential features with stable responses (same sign across prompts). Although responsive non-scaffold features are included as a control, the experiments do not fully isolate these two properties. The need and advantage of the proposed scaffold-selection process over more direct simple influence-based feature selection remain unclear.

  3. The main experiments primarily compare SCFE against concept-only editing and internal controls such as random-feature editing, responsive non-scaffold editing, prompt mismatch, and spatial shift. However, comparisons with relevant baselines are missing, such as Conditioned Activation Transport, OrthoEraser, or other methods addressing coupled representations and unintended editing effects.

  4. SCFE requires prompt-specific finite-difference response estimation, with two additional probes per concept coordinate and denoising timestep. The paper states that SCFE performs complete diffusion trajectories for these probes, needing substantially more model evaluations than concept-only editing. Although responses can be reused across editing strengths for the same prompt and initial noise, the computational overhead remains significant. It is unclear whether the improvements justify it, particularly when using efficient models such as SDXL-Turbo and FLUX.1.

  5. The quantitative evaluation covers only two editing concepts per diffusion architecture, with additional object-specific concepts only evaluated qualitatively. It is difficult to infer generalization across editing tasks. LPIPS, DINOv2 distance, and CLIP-I primarily measure similarity to the original image rather than preservation of specific non-target content or structural integrity. The semantic evaluation relies on GPT-5.6-Sol without any human evaluations. Given that the main claim is about improving semantic editing while preserving image content, human evaluations and analysis using image preservation and metrics like ImageReward, VQA, etc. (commonly used in editing and composition works) are needed.

[1] Arad, D., Mueller, A., & Belinkov, Y. (2025, November). Saes are good for steering–if you select the right features. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (pp. 10252-10270).

[2] Cassano, E., Renzulli, R., Ahmed, R., Grangetto, M., & Alaniz, S. (2026, September). Look But Don’t Touch with Sparse Autoencoders for Unlearning in Diffusion Models. In European Conference on Computer Vision (pp. 586-603). Cham: Springer Nature Switzerland.

Questions for authors
  1. Does SCFE improve the semantic-editing versus image-preservation trade-off?

The authors compare at identical target-concept intervention strengths, but SCFE additionally modifies scaffold features, so it is not a fair comparison. Could the authors clarify whether the improvements persist when comparing methods at matched semantic-editing success rather than matched intervention coefficients? A preservation-versus-semantic-success Pareto comparison would help distinguish genuine improvements from changes in effective editing strength.

  1. How reliable is the local response approximation under strong intervention strengths?

SCFE estimates scaffold responses using small finite-difference perturbations and extrapolates these estimates to stronger concept edits, where nonlinear effects and changes in SAE activation support may become significant. Since the largest improvements occur under strong editing, how sensitive is the method to deviations from this local approximation?

  1. Is prompt-specific response estimation essential?

The authors demonstrate that using another prompt's response degrades performance, but this does not necessarily establish the need for separate response estimation for every prompt. Could a globally averaged response estimated during calibration achieve comparable results? This would be particularly relevant given the inference-time overhead.

8 comments

Thank you for reviewing our paper and for your questions and suggestions. We use WX, MWX, and QX to label Weaknesses, Minor Weaknesses and Questions respectively. X indicates the order within each category, and / connects related items addressed together. We have added experiments, clarified methodological details, and expanded the discussion in response to these comments, and have uploaded the revised manuscript PDF. Below, we address the comments one by one with reference to the revised manuscript. Unless otherwise stated, all section, appendix, figure, and table references are to the revised manuscript.

W1 SCFE's substantive differences from prior research and its independent contribution remain unclear.

Thank you for raising this issue. Overall, our contribution is the discovery and validation of a population of scaffold features that recur across prompts and have content-dependent effects. We validate their functional role in concept editing through response-blocking experiments with matched controls. We build SCFE on these findings, using concept-induced responses for the current prompt to guide bounded scaffold updates.

Arad et al. [1] distinguish features' concept-detection capabilities from their direct steering capabilities and select steering features based on their output effects. We study how a shared feature population participates in realizing target concepts. They explicitly state in their Limitations section that their method uses a single SAE feature for steering, while multi-feature interactions remain an open challenge. We retain the target-concept intervention, test scaffold responses with matched controls, and use these responses for joint editing. Our contribution concerns discovering this shared population, validating its function, and using it for editing.

The study by Cassano et al. [2] differs fundamentally from ours in both its starting point and objective. Regarding the starting point, Cassano et al. [2] start from distribution shifts and artifacts induced by direct SAE interventions. They propose PER, which uses detections of a specified concept to locate targets and replaces the corresponding patch embeddings in intermediate feature maps to erase that concept. We start from the discovery and functional validation of the distinct scaffold-feature population. We propose SCFE, a response-guided joint-update method, to improve semantic change and image preservation in concept editing. Regarding the objective, concept erasure is one type of concept editing. PER aims to erase specified concepts, while our SCFE addresses broader concept editing, covering target-concept enhancement and suppression.

Overall, our work turns an experimentally validated population-level finding into an editing method. We believe that the empirical discovery of this shared population and its functional relationships fundamentally distinguishes SCFE from a combination of existing techniques.

[1] Arad, D., Mueller, A., & Belinkov, Y. (2025, November). SAEs are good for steering - if you select the right features. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (pp. 10241-10259). Association for Computational Linguistics.

[2] Cassano, E., Renzulli, R., Ahmed, R., Grangetto, M., & Alaniz, S. (2026, September). Look but don't touch with sparse autoencoders for unlearning in diffusion models. In European Conference on Computer Vision (pp. 586-603). Cham: Springer Nature Switzerland.

W2 The role of high-reuse selection has not been sufficiently separated from response strength and sign consistency.

Thank you for raising this issue. In the newly added Appendix G.6, we first ablate the 90% cross-prompt prevalence threshold. Lowering the threshold admits more strong-response non-scaffold features into editing. Image preservation worsens, while semantic change shows no consistent improvement.

This trend leads directly to your question: can response magnitude and sign consistency replace scaffold preselection? We test this with Response-only in the latter half of Appendix G.6. Response-only selects stronger, directionally consistent responses, yet both mean semantic change and image preservation worsen.

These results directly support retaining scaffold preselection. We combine cross-prompt reuse with response-based selection to achieve a better semantic-change versus image-preservation trade-off with SCFE.

W3 Comparisons with relevant external methods, including Conditioned Activation Transport and OrthoEraser, are missing.

Thank you for raising this issue. We already discuss this comparison principle in Appendices I.1 and I.2 of the original manuscript. SCFE enhances an existing SAE editing interface. We evaluate its added benefit while retaining the base diffusion model, SAE, and target-concept intervention.

Our method survey examined several related approaches. CAT [1] learns conditioned activation transport for safety. OrthoEraser [2] projects erasure directions to protect coupled benign features. SAEdit [3] edits text-token embeddings, while SAEmnesia [4] reshapes concept-feature mappings through supervised SAE training. The original approaches differ in their task objectives, editing interfaces, or feature representations. Absolute editing performance also depends on the diffusion model, SAE dictionary, and target features. OrthoEraser [2] explicitly notes in Appendix C that its precision is limited by the underlying SAE's dictionary capacity and reconstruction fidelity. We evaluate the benefit of adding SCFE to a fixed base system. Comparing final scores across different base systems mixes these system differences with SCFE's gains, making it difficult to isolate the scaffold updates' contribution.

We implement Concept-only using feature interventions supported by the public SAEs [5], [6]. We compare it with its SCFE-enhanced version within each model-SAE configuration. Our cross-model evaluation follows the same paired design. This directly tests our contribution: whether scaffold responses improve an existing concept edit.

[1] Chrabąszcz, M., Szymczyk, A., Dubiński, J., Trzciński, T., Boenisch, F., & Dziedzic, A. (2026). Conditioned activation transport for T2I safety steering. arXiv preprint arXiv:2603.03163.

[2] Shi, C., Wu, W., Shen, F., Zhu, X., Hu, K., & Wang, Z. (2026). OrthoEraser: Coupled-neuron orthogonal projection for concept erasure. arXiv preprint arXiv:2603.11493.

[3] Kamenetsky, R., Dorfman, S., Garibi, D., Paiss, R., Patashnik, O., & Cohen-Or, D. (2026, September). SAEdit: Token-level control for continuous image editing via sparse autoencoder. In European Conference on Computer Vision (pp. 359-377). Cham: Springer Nature Switzerland.

[4] Cassano, E., Renzulli, R., Nurisso, M., Zaffaroni, M., Perotti, A., & Grangetto, M. (2026). SAEmnesia: Erasing concepts in diffusion models with supervised sparse autoencoders. In Forty-third International Conference on Machine Learning (ICML).

[5] Surkov, V., Wendler, C., Mari, A., Terekhov, M., Deschenaux, J., West, R., Gulcehre, C., & Bau, D. (2025). One-step is enough: Sparse autoencoders for text-to-image diffusion models. In Advances in Neural Information Processing Systems (Vol. 38, pp. 99956-100027). Curran Associates, Inc.

[6] Shabalin, S., Panda, A., Kharlapenko, D., Ali, A. R., Hao, Y., & Conmy, A. (2025). Interpreting large text-to-image diffusion models with dictionary learning. arXiv preprint arXiv:2505.24360.

W4/Q3 The cost of per-prompt response estimation and whether a global mean can replace it need clarification.

Thank you for raising this issue. The newly added Table F.1 in Appendix F.5 reports first-image latency for the current implementation on a single H20, with final-image batch size 1. For SDXL-Turbo, Concept-only takes 0.721 s, response estimation takes 1.056 s, and SCFE takes 1.787 s in total. For FLUX.1-schnell, the corresponding times are 0.947 s, 1.238 s, and 2.188 s. We also explain in Appendix F.5 that SCFE's execution can be optimized. Our current implementation shares the unedited trajectory prefix and stops probe branches at the scaffold-response readout. For the same-step paths used in the main experiments, we batch the reference trajectory and positive and negative probes. Response maps can also be reused across strengths when the prompt, initial noise, and concept are unchanged. For edits requiring stronger semantic changes and good image preservation, we consider the measured additional latency and editing improvements a meaningful trade-off.

We also test the globally averaged responses you suggested in the newly added Appendix I.7. On SDXL-Turbo, the global mean's online first-image latency is 0.732-0.733 s, compared with 1.787 s for per-prompt SCFE. These online times exclude one-time offline calibration and mean construction. The results show that global means retain some editing gains, while per-prompt SCFE achieves better semantic change and image preservation. We believe these additional gains make per-prompt response estimation the better default for SCFE. For latency-sensitive applications, your proposed global mean also has practical potential and merits further exploration as a lightweight alternative.

W5 Concept coverage is limited, and evidence for semantic editing and non-target preservation needs human assessment and evaluation using more metrics.

We agree that the original quantitative evaluation covered too few concepts. In the newly added Appendix J, we evaluate subject blue, knit texture, and container slenderness on SDXL-Turbo, and subject blue, eyeglass addition, and metallic appearance on FLUX.1-schnell.

Broader testing was previously limited by whether generated images contained a visible, editable target. Section J.1 adds automated screening: GPT-5.6-Sol first screens prompts, then checks target visibility and edit suitability in unedited images generated from those prompts. Selection uses neither edited outputs nor method scores or performance differences. Tables J.1 and J.2 show gains in both target semantics and image preservation across the added tasks. These results show that SCFE generalizes well across different concepts and edit types.

We also agree that GPT-based semantic evaluation requires human validation. We have added blinded human ratings for the original concepts in Appendix I.6. Raters independently judge target-directed change, with method labels, editing strengths, and GPT-5.6-Sol scores hidden. Table I.1 shows strong agreement between human and GPT rankings. Human ratings also support SCFE's semantic gains, including for threatening appearance. These results support our use of GPT-5.6-Sol for large-scale target-semantic evaluation.

We assess whether the suggested metrics directly measure target-change magnitude. ImageReward [1] predicts human preferences from an image and prompt. VQA-based TIFA [2] and VQAScore [3] evaluate text-image faithfulness through answer accuracy or probability. These scores depend on the input text, questions, and evaluator, rather than directly measuring change relative to the original image. For example, suppressing blue may reduce faithfulness to an original prompt describing a blue object. We could rewrite the target prompt for post-edit faithfulness scoring, but this introduces unnecessary wording choices and potential human bias. These metrics also do not directly compare non-target content before and after editing. We instead combine human-validated image-pair semantic scores, preservation metrics, and paired visualizations to assess target change and preservation of existing content.

[1] Xu, J., Liu, X., Wu, Y., Tong, Y., Li, Q., Ding, M., Tang, J., & Dong, Y. (2023). ImageReward: Learning and evaluating human preferences for text-to-image generation. In Advances in Neural Information Processing Systems (Vol. 36, pp. 15903-15935). Curran Associates, Inc.

[2] Hu, Y., Liu, B., Kasai, J., Wang, Y., Ostendorf, M., Krishna, R., & Smith, N. A. (2023, October). TIFA: Accurate and interpretable text-to-image faithfulness evaluation with question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 20406-20417). IEEE.

[3] Lin, Z., Pathak, D., Li, B., Li, J., Xia, X., Neubig, G., Zhang, P., & Ramanan, D. (2025). Evaluating text-to-visual generation with image-to-text generation. In Computer Vision - ECCV 2024 (pp. 366-384). Cham: Springer Nature Switzerland.

Q1 Equal intervention coefficients do not replace matched semantic success, and the advantage in the semantic-preservation trade-off needs validation.

We agree that SCFE changes additional coordinates, so equal target coefficients are not equivalent to equal semantic success. Table 1 controls the target injection to isolate the added update, but that control alone does not answer the trade-off question. The newly added Fig. 6 uses Pareto curves to compare achieved target-semantic scores against LPIPS across intervention strengths.

The comparison covers SDXL-Turbo and FLUX.1-schnell, including enhancement and suppression for the original concepts. Over most of the semantic range shared by the methods, SCFE lies below Concept-only: it attains lower image deviation at comparable target change. The newly added Pareto curves provide strong evidence that SCFE can achieve comparable semantic change while better preserving the original image content.

Reviewer_LGU8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how SAE features participate in diffusion-model steering. They identify scaffold features as SAE features that activate across at least 90% of diverse prompts, and shows that these features respond strongly when semantic concept features are edited. Through restoring scaffold responses to their unedited ones, the authors find larger changes in the final output than when restoring matched ordinary features, suggesting that scaffold responses causally contribute to the edit. Based on this observation, the paper proposes SSCFE, which uses small concept interventions to estimate prompt-specific downstream scaffold responses and jointly updates the target concept and selected scaffold features. Experiments on SDXL-Turbo and FLUX.1-schnell report improved target-semantic change and image preservation compared with editing the target concept feature alone.

Critical strengths
  1. The response-restoration experiment is stronger than simply showing that scaffold features correlate with concept edits.

  2. Random features, responsive non-scaffolds, prompt mismatch, and spatially shifted responses test different alternative explanations for SCFE's gains. The fact that these controls do not reproduce SCFE's performance supports the importance of both feature identity and prompt-specific spatial response.

Critical weaknesses
  1. The main SDXL quantitative evaluation uses only two concept features: cartoon/photorealistic rendering and threatening appearance. FLUX evaluation adds only two more: anime style and Christmas elements. Several additional local concepts, such as texture, collars, dials, and object shape, are shown only qualitatively. This matters because evaluating 10,000 prompts gives strong prompt diversity, but not concept diversity.

  2. SCFE and Concept-only are compared using the same target-feature intervention strength. However, SCFE additionally changes scaffold features and often produces a stronger semantic edit. The practical question is not necessarily which method is better at the same internal coefficient, but which method preserves the image better for the same achieved amount of semantic change. For example, Concept-only at a different strength might reach the same semantic score as SCFE while producing comparable or better preservation.

  3. For SDXL, Appendix G states that quantitative evaluation uses 10000 prompts and that the main experiments also use 10000 prompts for scaffold-response calibration and feature selection. It is not clear whether these are independent sets. This matters because cross-prompt response consistency and mean response are used to decide which scaffold features will be edited. If the same prompts are then used to evaluate SCFE, feature selection is partly optimized on the evaluation distribution itself. Interestingly, the FLUX experiment explicitly states that its 32 calibration prompts are separate from the 10000 evaluation prompts.

  4. Scaffold features are operationally defined as features active somewhere in at least 90% of prompts. The causal and responsive-non-scaffold controls provide useful evidence that the selected features are special, but they do not show that 90% prevalence identifies a distinct population or that functional importance is specifically associated with cross-prompt reuse. For example, does causal importance increase continuously with feature prevalence? Would features selected using 80%, 95%, or 99% reuse behave similarly? This is particularly relevant because prior SAE work has already shown that highly active latents can represent important functional variables rather than noise. The novelty here therefore depends partly on establishing what is specifically distinctive about the proposed scaffold population.

Minor Weaknesses:

  1. SCFE is computationally expensive. Response estimation is performed separately for each concept and timestep, requiring positive and negative probes. The current implementation runs a complete denoising trajectory for each probe.
  2. The semantic evaluation relies entirely on GPT-5.6-Sol without human validation. This is particularly relevant for subjective concepts such as “threatening appearance.”
Questions for authors
  1. Could the authors quantitatively evaluate SCFE on a broader set of concept types, particularly localized attributes such as color, material, object geometry, or object-specific details?

  2. Could the authors compare SCFE and Concept-only at matched target-semantic change, or report a semantic-change versus preservation Pareto curve? This would determine whether SCFE genuinely provides a better editing trade-off rather than simply outperforming Concept-only at the same internal feature coefficient.

  3. Are the 10000 SDXL calibration prompts used to select scaffold responses separate from the 10,000 evaluation prompts? If not, could the authors repeat the evaluation on a held-out prompt set, as is already done for FLUX?

  4. How sensitive are the causal and SCFE results to the 90% reuse threshold?

  5. What is the generation-time overhead of SCFE relative to Concept-only steering?

  6. Could the authors validate the GPT-based semantic score against human judgments on a subset of images, especially for threatening-appearance editing?

5 comments

Thank you for reviewing our paper and for your questions and suggestions. We use WX, MWX, and QX to label Weaknesses, Minor Weaknesses and Questions respectively. X indicates the order within each category, and / connects related items addressed together. We have added experiments, clarified methodological details, and expanded the discussion in response to these comments, and have uploaded the revised manuscript PDF. Below, we address the comments one by one with reference to the revised manuscript. Unless otherwise stated, all section, appendix, figure, and table references are to the revised manuscript.

W1/Q1 The quantitative evaluation covers few concepts and needs additional tests of local attributes and object details.

We agree that many prompts for a small concept set do not establish concept diversity. The newly added Appendix J provides quantitative tests of 5 concept types: subject blue, knit texture, container slenderness, eyeglass addition, and metallic appearance. SDXL-Turbo evaluates blue, knit texture, and slenderness; FLUX.1-schnell evaluates blue, eyeglass addition, and metallic appearance.

The main obstacle to broader quantitative testing was that images from random prompts might lack a visible, editable target. Section J.1 of this new appendix addresses this with two-stage automated screening. GPT-5.6-Sol first screens LAION-COCO prompts. It then checks target visibility and edit suitability in unedited images generated from the original prompts. Screening uses only prompts and unedited images, not edited outputs, method scores, or performance differences. We apply fixed criteria until each concept has 1,000 distinct prompts, then compare Concept-only and SCFE on the same subset.

Tables J.1 and J.2 show improvements in target-semantic scores and all 3 preservation metrics for every added task. These experiments extend the evidence to color, texture, material, geometry, and object-specific details. The results show that SCFE generalizes well across different concepts and edit types.

W2/Q2 Equal intervention coefficients do not establish a fair comparison, and preservation should be compared at matched semantic change.

We agree that an equal concept coefficient does not guarantee equal achieved semantic change. Holding the concept injection fixed isolates the effect of adding scaffold updates, but does not by itself establish a better semantic-preservation trade-off. We now compare image preservation using the Pareto curves in the newly added Fig. 6. In Fig. 6, the horizontal axis is the achieved target-semantic score and the vertical axis is LPIPS. Each point averages 10,000 paired records at one strength. Overall, SCFE has lower LPIPS at comparable semantic scores, including comparisons between different internal coefficients. The advantage is clearer for larger target changes. This comparison confirms that, at equal or similar semantic-change levels, SCFE can achieve comparable or better image preservation.

W3/Q3 The independence of calibration and evaluation prompts needs clarification, and editing gains require validation on held-out prompts.

The original SDXL-Turbo calibration and main evaluation used the same 10,000 prompts, rather than independent splits. Appendix H serves to test the generalization of scaffold features and SCFE to a new diffusion model and SAEs. For this reason, in its FLUX.1-schnell experiments we deliberately use 32 calibration prompts separate from the 10,000 evaluation prompts.

We acknowledge that the original manuscript lacked a clear explanation and controlled comparison of these settings, making the difference appear arbitrary. The newly added Appendix G.8 reports SDXL-Turbo evaluations on held-out sets of 32, 500, 2,000, and 5,000 prompts. All are disjoint from the original 10,000-prompt set. Table G.10 reports SCFE's results at each size alongside the original non-held-out evaluation. Results at different held-out sizes remain close to the original evaluation, showing that SCFE's absolute editing performance is not confined to calibration prompts. This consistency supports SCFE's stability across prompt partitions.

W4/Q4 The functional distinctiveness of features selected at 90% reuse and the threshold sensitivity of causal and editing results need validation.

The newly added Appendix G.6 presents threshold-sensitivity and preselection-removal experiments to examine the 90% threshold and the selected scaffolds' functional role. On two SDXL-Turbo concepts, we compare 80%, 90%, 95%, and 99% prevalence gates, plus Response-only without a gate. All settings retain the same response sign-consistency criterion and a cap of 2 selected features per step. Table G.2 reports candidate and selection changes; Table G.3 compares editing results for 90%, 80%, and Response-only.

The experimental results show that raising the threshold excludes usable features, leaving some steps without scaffold updates and potentially reducing a concept's editing to Concept-only. Lowering it admits larger-response features below the original scaffold prevalence criterion, which replace existing candidates. In the count-matched comparison of 80% and 90%, these replacements do not improve mean preservation or provide consistent semantic gains. The 90% threshold offers an experimentally supported balance between retaining usable features and excluding lower-prevalence responses.

This result naturally raises a more direct question: can strong, directionally consistent responses replace scaffold preselection? The latter part of Appendix G.6 tests this with Response-only. Removing the gate admits high-response non-scaffold features, but mean semantics and preservation are worse than SCFE-90 for both tested concepts. Larger responses and consistent directions alone do not guarantee better joint editing; high-reuse scaffold preselection adds editing value. These controls support a specific functional role for the selected scaffold population in SCFE, rather than interchangeability with arbitrary high-response features.

MW1/Q5 Response probes for each concept and timestep add computation, so the implementation needs clarification and actual inference overhead needs reporting.

We report the computational cost in the newly added Table F.1 in Appendix F.5. Table F.1 measures first-image latency on a single H20 with final-image batch size 1. For SDXL-Turbo, Concept-only takes 0.721 s, response estimation takes 1.056 s, and SCFE takes 1.787 s in total. For FLUX.1-schnell, the corresponding values are 0.947 s, 1.238 s, and 2.188 s. Although SCFE adds response probes, the implementation shares the unedited trajectory prefix and stops each probe after the required scaffold-response readout. The probes do not each run a complete image-generation trajectory. For the same-step paths used in the main experiments, the reference and positive and negative probes are batched together. This reduces redundant computation.

SCFE aims primarily to improve editing performance. We consider the measured overhead reasonable for the gains in target change and image preservation, especially for edits requiring stronger intervention. Response maps can also be reused across strengths for the same prompt, initial noise, and concept, avoiding repeated probing during strength adjustment.

MW2/Q6 The reliability of GPT-based semantic scores needs human validation, especially for subjective judgments of threatening appearance.

We fully agree that human validation is important. We have added a blinded human study for the 4 original concepts in Appendix I.6. We sample 300 prompts per concept, with 9 raters participating and 3 independent raters scoring each prompt group. Method, strength, and GPT-5.6-Sol scores are hidden during rating. Human raters judge target-directed change on the same 0-10 scale.

Table I.1 reports human agreement, agreement between human and GPT-5.6-Sol scores, and paired method differences. For threatening appearance, ordinal Krippendorff's alpha is 0.798 and the Spearman correlation between GPT-5.6-Sol and human scores is 0.865. Overall, these results show strong agreement between GPT-5.6-Sol and human rankings of target-semantic change. They indicate that GPT-5.6-Sol is reasonably reliable for large-scale target-semantic scoring of the evaluated concepts.

Dear Reviewer LGU8,

Thank you for your detailed review and constructive questions. In response, we have added experiments on additional concepts, semantic-preservation trade-off curves, and evaluations on held-out SDXL prompts. We have also included threshold-sensitivity analyses, measured latency, and a human validation study.

As the discussion period draws to a close, we would greatly appreciate any thoughts you may have on whether these additions address your concerns, or whether any points would benefit from further clarification.

Thank you again for your time and careful consideration.

Best regards, The authors

Reviewer_V3nWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies so called scaffold features in diffusion models, internal features that are activated across multiple prompts, thus their semantics are difficult to identify, however have strong impact on editing image content. Authors hypothesize that these features carry important shared generative context and internal variables. Building upon these findings, authors propose an editing method that can augment any SAE-based diffusion editing framework by estimating scaffold feature responses and correcting the concept-only edit accordingly. Experiments show promising results on editing improvements across various editing metrics.

Critical strengths
  • The paper adds an interesting observation, the existence of scaffold features, to ongoing research efforts on interpreting diffusion-based image generation.
  • The editing framework can be added on top of existing techniques without further training.
  • The experimental results show consistent improvements compared to the baseline technique it has been added to.
Critical weaknesses
  • A key shortcoming of the editing methodology component of the paper is that it only provides clear benefits in the higher editing strength regime. Depending on the desired edit, the proposed method may only constitute an overhead without much improvement in edit quality.
  • The definition of the scaffold set is somewhat ad hoc, and the sensitivity on the selected hyper-parameter is unclear.

Minor: typos (?) in lines 353 and 377. \tau_s is not defined in the paper.

Questions for authors

I couldn't find a discussion on how 90% co-occurrence was selected in Eq. (1). Do the findings change if we have a stricter requirement (say retain instances with 99% prompt coverage) or a more relaxed one (80%)?

Can authors give more intuition and a detailed discussion in the main paper on why the gap decreases with higher editing strength?

4 comments

Thank you for reviewing our paper and for your questions and suggestions. We use WX, MWX, and QX to label Weaknesses, Minor Weaknesses and Questions respectively. X indicates the order within each category, and / connects related items addressed together. We have added experiments, clarified methodological details, and expanded the discussion in response to these comments, and have uploaded the revised manuscript PDF. Below, we address the comments one by one with reference to the revised manuscript. Unless otherwise stated, all section, appendix, figure, and table references are to the revised manuscript.

W1/Q2 The strength-dependent gap between SCFE and Concept-only needs explanation, as do the costs and benefits at low editing strengths.

We thank you for this insightful question. Sec. 5.1 and Fig. 5, together with Appendix G.2 and Fig. G.1, show the overall trend. Within the tested range, SCFE's average advantage over the compared control methods, especially Concept-only, generally grows with editing strength. This raises two related issues: the source of this trend and the need for SCFE at low strengths.

For the first issue, we have added a discussion in Sec. 5.1. SCFE applies a bounded update along the scaffold response predicted for the current concept intervention. As the intervention approaches zero, the predicted response and extra update also approach zero. Both methods converge to the same generation process, leaving less room for improvement at low strength. Intuitively, stronger concept interventions are more likely to damage subject structure and existing content. Correct scaffold updates then have more room to help, offering an intuitive explanation for the larger average gains under stronger editing.

For the second issue, Appendix I.5 and Fig. I.1 discuss when stronger interventions are needed. At low strengths, average semantic and CLIP-I gains can be small, so we do not argue that SCFE is necessary for every weak edit. Some images already show a clear target change at low strength, whereas others require stronger intervention. SCFE's value is most apparent in the latter setting: it improves preservation while supporting a larger target change.

We also report first-image latency on a single H20 in the newly added Table F.1 in Appendix F.5. SDXL-Turbo takes 0.721 s with Concept-only and 1.787 s with SCFE; FLUX.1-schnell takes 0.947 s and 2.188 s. The response maps can be reused across strengths for the same prompt, initial noise, and concept, avoiding repeated probing while adjusting strength. For edits requiring stronger semantic changes and good content preservation, we consider this measured quality-time trade-off meaningful.

W2/MW1/Q1 The choice of the 90% threshold, the results' sensitivity to it, and the meaning of τ_s need clarification.

We thank you for pointing out this important issue. In the original manuscript, τ_s at lines 353 and 377 lacked an explicit main-text definition. We have added it immediately after Eq. 3 in Sec. 4. It is the cross-prompt response sign-consistency threshold, filtering out candidates whose response direction is insufficiently consistent across prompts. It is distinct from the 90% reuse threshold used to identify scaffolds. The newly added Appendices G.6 and G.7 examine the reuse threshold and the other hyperparameters, respectively.

Appendix G.6 compares reuse thresholds of 80%, 90%, 95%, and 99%, as well as removing the threshold entirely. Stricter thresholds reduce eligible scaffold candidates. For f4998, neither 95% nor 99% leaves any eligible candidates, so the additional update disappears and the method reduces to Concept-only. Very strict reuse requirements can therefore remove the features needed for joint editing. Relaxing the threshold to 80% admits larger-response features that fail the original 90% scaffold criterion and replace some selected features. Table G.3 shows slightly worse mean preservation for both tested concepts, without a consistent semantic improvement. Removing the gate entirely allows stronger, sign-consistent non-scaffold responses to enter, but Response-only performs worse than SCFE-90 in both mean semantics and preservation. These results support 90% as a reasonable balance between candidate availability and editing quality in the tested setting.

Appendix G.7, Tables G.4-G.9, ablates τ_s, the accompanying-feature count K, response gain lambda, amplitude bound kappa, and calibration and online probe magnitudes. These ablations demonstrate that the default configuration strikes a good balance between semantic change and image preservation under the tested strong-editing conditions, providing experimental support for the defaults used in the paper.

Dear Reviewer V3nW,

Thank you again for your thoughtful review and suggestions. In response to your concerns, we have added an analysis of the scaffold reuse threshold, including the 80% and 99% settings you suggested, and expanded our discussion of the benefits and costs at different editing strengths. We have also reported the measured inference overhead.

As the discussion period draws to a close, we would be grateful if you could take a moment to consider our response and let us know whether these additions address your concerns. We would be happy to clarify any remaining points.

Thank you for your time and consideration.

Best regards, The authors

Reviewer_V3nWOpenReview ↗

Response to authors

Thank you for your response and the additional details on the scaffold reuse threshold ablations. I maintain my score.

Thank you for your response and for the time you have devoted to reviewing our work. We sincerely appreciate your thoughtful feedback and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.