acceptodds
Desk rejected at ICLR 2027

Where Guidance Matters Most: Spatially Adaptive CFG for Diffusion Models

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Classifier-free guidance (CFG) improves prompt adherence by extrapolating beyond conditional predictions, but applies a uniform guidance scale to spatial regions that may respond differently to conditioning. We introduce pattern-based classifier-free guidance (PCFG), a training-free method that adaptively regulates further extrapolation by measuring how conditioning alters local spatial patterns. PCFG represents small patches by ordered prediction amplitudes and constructs neighborhood-level pattern distributions from their occurrence frequencies, capturing local spatial relationships beyond pointwise residual magnitudes. It then compares the corresponding conditional and unconditional distributions using JS divergence to quantify local pattern discrepancy. Finally, a bounded power mapping converts this discrepancy into spatially varying guidance scales, suppressing further extrapolation in regions with pronounced pattern changes while allowing stronger amplification where the two distributions remain similar. At each sampling step, PCFG updates the guidance map from the current conditional and unconditional predictions without additional denoiser evaluations. Experiments on SD3.5 Medium across COCO, DrawBench, and GenEval2 show that PCFG achieves the highest reported ImageReward and HPSv2.1 means in all nine benchmark–budget settings against standard CFG and seven competing guidance methods. It surpasses the metric-specific best-scanned scalar CFG in 41 of 45 comparisons; moreover, its 20-step results outperform the best-scanned 30-step scalar CFG on both ImageReward and HPSv2.1 across all three benchmarks. These results demonstrate that local pattern discrepancy provides an effective signal for spatially adaptive guidance.

Primary area:
generative models
Keywords:
Classifier-Free Guidance, Diffusion Models, Spatially Adaptive Guidance, Local Pattern Distributions, Text-to-Image Generation

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_vqcsOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Pattern-based Classifier-Free Guidance (PCFG), a training-free guidance method that assigns spatially varying CFG scales during diffusion sampling. It converts the conditional and unconditional prediction fields into quantized channel-norm maps, encodes ordered local patches, estimates neighborhood pattern histograms, and measures their Jensen-Shannon divergence. Regions with larger conditional–unconditional pattern discrepancies receive less additional extrapolation, whereas regions with similar patterns receive stronger guidance. Favorable empirical results are reported on SD3.5 Medium and SDXL across multiple datasets.

Critical strengths
  1. The proposed method is well motivated by the observation and analysis that identical pointwise residual energy can conceal different spatial arrangements. Using local pattern-distribution discrepancy as a feedback signal for spatial guidance is interesting and potentially useful.

  2. The proposed method is sufficiently concrete in its design and clearly presented. Extensive experiments demonstrate generally consistent gains across diverse settings and metrics.

Critical weaknesses
  1. The evaluation relies entirely on automated metrics. A human preference study would strengthen the performance claim but is absent. This is particularly important because the metrics exhibit non-negligible disagreement: PCFG consistently improves ImageReward and HPSv2.1, but does not consistently lead PickScore or Aesthetic Score.

  2. The stated implementation forms a tensor of shape [B, 393^9, H, W] = [B, 19683, H, W]. At a 128×128 latent in FP32, this is approximately 1.2 GiB per branch before other tensors. No latency, peak-memory, or FLOP results are reported. Consequently, “no additional denoiser evaluations” does not establish practical efficiency.

  3. The generalization evidence remains limited to the Stable Diffusion family, although the models cover velocity and noise predicition and involve different architectures. It is unclear whether the proposed method generalizes to other modern model families, such as FLUX.

Questions for authors
  1. Could the authors report wall-clock latency, throughput, FLOP, or peak-memory measurements and make a direct efficiency comparison against CFG and SAMG?

  2. Could the authors clarify whether PCFG is applicable only to models exposing explicit conditional and unconditional prediction branches? Does it work with FLUX models?

  3. Why is the S-CFG result absent from the experimental comparison despite being recognized a closely related spatially adaptive CFG method?

  4. Do all mean values reported are obtained with a single seed of 4416? If yes, authors are encouraged to report the mean over different seeds.

4 comments
AuthorsOpenReview ↗

Efficiency, Reliability, and More Evaluation

We thank the reviewer for the careful and constructive review, and for finding the method "well motivated", "clearly presented", and supported by "extensive experiments". Every question raised is concrete and testable. We answer each with new measurements, and in each case we include more baselines than were requested.

W1: Metric disagreement

We conducted a multi-seed study. Against CFG, PCFG improves all five metrics, including PickScore and Aesthetic, in every seed and with intervals excluding zero. In this SD3.5 study, PCFG significantly improves Aesthetic relative to SAMG and CFG-Rescale. PickScore intervals span zero, and no metric shows a statistically significant decrease relative to these baselines. Image-level paired win rates against CFG are 69% (HPS), 61% (IR), 61% (Aesthetic), 57% (Pick), and 55% (CLIP). In addition, PCFG improves distribution-level fidelity on COCO-5k: precision 0.726 → 0.740, coverage 0.729 → 0.753, FID 31.8 → 30.7 (Table G4).

W2 / Q1: Memory, latency, throughput, and comparison with CFG and SAMG

The estimate concerns the dense histogram used to define the statistic in the paper; our implementation never builds it. The reason is simple: a k×kk\times k neighborhood contains only (k−p+1)2=121(k-p+1)^{2} = 121 patches per branch, so at any location at most 242 of the 39=19,6833^{9} = 19,683 possible patterns actually occur. All other patterns have zero probability in both branches and contribute exactly zero to the JS divergence. We therefore count only the patterns that occur, which gives exactly the same SS with no approximation. This reduces the per-step cost from 8.7 GiB and 93 ms (dense) to 126 MiB and 2.7 ms; App. G of the revised paper gives the derivation.

SD3.5-Medium, A100, 102421024^{2}, 20 steps.

CFGSAMGPCFG
Latency (s / image)4.2024.2324.262
Overhead vs. CFG—+0.71%+1.43%
Throughput (images / s)0.2380.2360.235
Extra peak memory of the guidance step1 MiB3 MiB126 MiB
Whole-pipeline peak allocated memory17.60 GiB17.60 GiB17.60 GiB

Compared with standard CFG, PCFG's inference-time overhead at 20 steps is only +1.43%. General Response Table G1 covers all nine methods.

W3 / Q2: Beyond SD; FLUX; the branch requirement

PCFG requires exactly what CFG already computes: a conditional and an unconditional prediction on a spatial grid. It needs no attention maps, so it applies to any model sampled with CFG. We ran FLUX.2 Klein Base 4B (50 steps; 600 prompts) and deliberately kept the SD3.5 configuration.

CLIPIRHPSPickAest.
CFG (w=4w=4)28.5781.0630.2910.8705.466
PCFG28.7091.0730.2930.8685.469

PCFG runs on this model family without retuning. HPS improves significantly and PickScore decreases significantly; CLIP, IR, and Aesthetic have small mean differences whose 95% intervals include zero. In summary, PCFG is now evaluated on SD3.5 (MMDiT, velocity), SDXL and SD2.1 (UNet, noise), and FLUX.2, as well as four solvers (Table G3a, 39/40 significant gains).

Q3: S-CFG

S-CFG derives its regions from the cross-attention maps of UNet text-to-image models; the official code supports SD1.5. To provide a fair comparison, we ported the official implementation to SD2.1-base on a shared pipeline, preserving its attention processor and mask computation and keeping the setup as close as possible to the official pipeline. We also included CFG-Rescale and APG. The experiments use 600 prompts, DPM-Solver++, and two sampling budgets; full results are given in Table G3b of the General Response.

PCFG gives the best CLIP, IR, and HPS at 20 steps and the best scores on all five quality metrics at 50 steps. It is also faster than S-CFG at both budgets: 0.80 vs. 0.87 s/image at 20 steps and 1.85 vs. 1.99 s/image at 50 steps.

Q4: Multiple seeds

Yes, the original results used seed 4416. We now report four seeds, and the conclusions are unchanged. SD3.5-M, 20 steps, 600 prompts × seeds {4416,5525,6636,7749}\{4416, 5525, 6636, 7749\}, paired.

PCFG −CLIPIRHPSPickAest.
CFG+0.201 · 4/4+0.085 · 4/4+0.009 · 4/4+0.004 · 4/4+0.075 · 4/4
SAMG+0.239 · 4/4+0.058 · 4/4+0.005 · 4/4+0.000 · 3/4+0.040 · 4/4
CFG-Rescale+0.265 · 4/4+0.045 · 4/4+0.005 · 4/4+0.001 · 3/4+0.053 · 4/4

The "x/4x/4" entries give the number of seeds in which PCFG is better. Further details are given in Tables G2a–b.

We hope these experiments fully address the reviewer's questions, and we would be happy to provide further details.

Dear Reviewer vqcs,

We sincerely thank you for your constructive feedback, which has helped us strengthen the paper and clarify its contributions.

We hope that our detailed responses and additional results have adequately addressed your concerns. Should any questions or concerns remain, we would be more than happy to provide further clarification.

If you find that our responses have satisfactorily resolved your concerns, we would greatly appreciate your reconsideration of the score and stronger support for the paper.

Thank you again for your time and thoughtful evaluation.

Best regards,

The Authors

Thanks for the authors' response. It addresses most of my concerns, and I decided to raise my rating.

Thank you for your thoughtful feedback and for reconsidering our response. We are glad that our clarifications addressed your concerns, and sincerely appreciate your decision to raise the rating.

Reviewer_n9ccOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Pattern-based Classifier-Free Guidance (PCFG), a training-free method that adapts classifier-free guidance (CFG) using a spatially varying guidance map. Rather than relying on pointwise residual magnitudes, PCFG compares local distributions of quantized prediction-amplitude patterns in the conditional and unconditional branches using Jensen–Shannon divergence. The resulting discrepancy is mapped to spatially varying guidance scales. The method is evaluated on SD3.5 Medium and SDXL across COCO, DrawBench, and GenEval2, using five automatic metrics and comparisons against existing guidance methods. The authors report improvements over standard CFG and other baselines, including gains over a best-scanned scalar-CFG reference on SD3.5.

Critical strengths
  1. Evaluation across multiple models and benchmarks: I appreciate that the authors evaluate the method on two models, SD3.5 Medium and SDXL, and across three benchmarks. This provides evidence beyond a single model or dataset and strengthens the empirical evaluation.
  2. Training-free integration into sampling: PCFG does not require additional training or additional denoiser evaluations. The method can therefore be integrated into an existing sampling pipeline without retraining the underlying model.
Critical weaknesses
  1. Unclear motivation for the proposed local-pattern statistic: The paper motivates PCFG by arguing that pointwise residual magnitudes fail to capture local spatial relationships. However, the conditional and unconditional predictions are themselves produced by a trained generative model that processes spatially structured representations. Thus, the residual (\Delta=f^y-f^u) already reflects the model's processing of spatial information; it is not simply a signal devoid of local context. PCFG subsequently computes local statistics of the prediction amplitudes to construct the spatial guidance map, but the paper does not explain what meaningful information this additional representation provides for guidance allocation or why it should be necessary.
The method may therefore be viewed as feature engineering for the guidance scheduler: instead of directly using pointwise residual energy, it constructs a richer statistic from local prediction patterns. However, the paper provides limited insight into why this particular statistic is useful beyond its empirical performance.
  2. Weak qualitative results: The qualitative evaluation is limited, and the examples shown do not demonstrate a compelling visual improvement over the baselines. This makes it difficult to assess whether the reported quantitative gains translate into perceptually meaningful improvements and limits the evidence for the method's practical value.

Minor concerns:

  1. Incomplete related work and unclear positioning: The related-work discussion does not sufficiently engage with relevant prior work on guidance. In particular, it should also discuss Classifier-Free Guidance Is a Predictor-Corrector, Navigating with Annealing Guidance Scale in Diffusion Space, Learning to Guide Your Diffusion Model, and Towards a Golden Classifier-Free Guidance Path via Foresight Fixed Point Iterations.
  2. Limited evaluation of fidelity and distributional coverage: The evaluation relies on five automatic metrics, but does not fully characterize the trade-off between generation fidelity and distributional coverage. I would encourage the authors to report precision and recall, which provide complementary measures of fidelity and coverage in generative modeling.
Questions for authors

My main concern is that, in its current form, the work does not clearly elevate beyond feature engineering for a spatial guidance scheduler, making the conceptual contribution appear incremental. While the empirical results are promising, the paper provides limited insight into why the proposed local-pattern statistic is particularly useful or what it reveals about the guidance process. Can the authors provide a compelling toy example based on a trained generative model that better motivates the proposed representation and offers geometric insight into its effect on the guidance update? In particular, what meaningful information does this statistic provide beyond what is already encoded in the conditional and unconditional prediction fields and their residual? Such an example could help clarify the conceptual contribution and potentially change my assessment of the work.

4 comments

We thank the reviewer for the thoughtful review and for finding the method "well motivated" and "clearly presented" with "generally consistent gains". We especially appreciate Q1: it asks exactly the right question, and answering it on a trained model has made the paper's contribution considerably sharper. The revised paper now includes a new main-text subsection (Sec. 4.4, Figure 6).

W1 / Q1: What the statistic adds beyond the residual, shown on a trained model

We agree that Δ=fy−fu\Delta = f^{y} - f^{u} is spatially informed and that SS contains no information beyond the two predictions. The point is what a guidance rule keeps. Energy-based rules reduce Δ\Delta to one number per location, E(x)=mean⁡cΔ(x)2E(x) = \operatorname{mean}_{c} \Delta(x)^{2}, which depends on Δ\Delta alone. Adding the same Δ\Delta to different unconditional fields preserves residual energy but can change their local amplitude-pattern distributions differently. SS compares those distributions in both branches. PCFG uses larger SS as a heuristic for reducing further extrapolation and smaller SS for allowing more; SS does not certify semantic correctness or extrapolation safety. Figure 6 of the revised paper (new Sec. 4.4) shows that this matters in a trained model.

Step 1: a real example (Figure 6a–c). SD3.5 Medium, "A train on top of a surfboard", prediction 10 of 20. Among the 13×1313\times13 regions on the sampled fixed grid, we show the eligible pair with the largest difference in SS subject to at most 5% mean-energy difference.

Region ARegion B
Mean residual energy (×10−3\times10^{-3})4.244.36only 2.8% apart
Pattern discrepancy SS0.0960.3543.7× apart
PCFG weight WW15.259.25different treatment

Figure 6c illustrates different amplitude-pattern changes at similar regional mean energy; pointwise energy maps and SAMG weights need not match.

Step 2: observations across recorded states (Figure 6d). Across 24 states (8 prompts × early, middle, and late steps), a predictor using mean SS within 200 quantile bins of local energy explains 45% of its variance in sample on average, and 21% at late steps. This is not an upper bound over all energy functions. Among the 3,336 region pairs with at most 5% difference in mean energy, the median PCFG weight difference is 2.0.

Step 3: it changes the update (Figure 6e). Spatial weights never rotate Δ\Delta at a location; they move extrapolation between locations. PCFG's extra update dW=h(W−1)⊙Δd_W = h(W-1)\odot\Delta and a scalar update da=h(a−1)Δd_a = h(a-1)\Delta of exactly equal norm differ by 17.7° in the full latent space for this state, and by 19.4° on average over the 24 states (8.4°–31.9°). At the same extra-guidance norm, this demonstrates a different spatial allocation of extrapolation; it does not by itself establish a quality benefit. In the recorded trajectories, the mean weight rises from 9.9 to 13.7 and 14.6 from early to late steps; this is an observed trajectory pattern, not a guaranteed monotonic schedule.

Effect on quality. Against SAMG, which uses exactly the energy feedback above, PCFG is significantly better over 600 prompts × 4 seeds:

PCFG − SAMG (95% CI over 4 seeds)CLIPIRHPSAest.
Difference+0.239 [0.191, 0.286]+0.058 [0.041, 0.076]+0.005 [0.003, 0.007]+0.040 [0.017, 0.063]

In addition, maps that keep PCFG's weight values but move them (random shuffle, or low weights to high-energy locations) or flatten them to their mean lower IR from 1.41 to 1.25–1.31.

These observations show that similar regional mean energy can correspond to different pattern discrepancies and update directions. The four-seed comparison supports the complete PCFG rule against SAMG, without isolating the statistic's causal contribution.

W2: Qualitative results

The revised App. F adds CFG / Rescale / SAMG / PCFG comparison grids. Quantitatively, over all 2,400 prompt–seed pairs, PCFG's image-level win rates against CFG are 69% (HPS), 61% (IR), 61% (Aest.).

W3: Related work

Thank you; the revised Sec. 2 now discusses all four.

W4: Precision and recall

COCO-val, 5k real images vs. 5k generated images per method (full table: General Response, Table G4):

Precision ↑Recall ↑ (k=3k=3 / k=5k=5)Density ↑Coverage ↑FID ↓
CFG0.7260.383 / 0.4911.1470.72931.811
PCFG0.7400.386 / 0.4931.2410.75330.708

PCFG improves precision and coverage and lowers FID by 1.1 in Table G4. Recall point estimates also increase slightly at both neighborhood sizes. These results describe this COCO evaluation and do not establish the absence of diversity trade-offs.

We appreciate the reviewer's willingness to reassess and would be happy to provide any further clarification.

Dear Reviewer n9cc,

We sincerely thank you for your constructive feedback, which has helped us strengthen the paper and clarify its contributions.

We hope that our detailed responses and additional results have adequately addressed your concerns. Should any questions or concerns remain, we would be more than happy to provide further clarification.

If you find that our responses have satisfactorily resolved your concerns, we would greatly appreciate your reconsideration of the score and stronger support for the paper.

Thank you again for your time and thoughtful evaluation.

Best regards,

The Authors

I thank the authors for their detailed response. While I appreciate the effort for improving the paper, I still feel like it's not quite ready for acceptance. Two concerns remain:

  • I couldn't find a considerable difference or improvement in the added qualitative results compared to the baseline.
  • The added analysis in Sec. 4.4 was quite hard to read and understand. I think that the paper could benefit from a considerable rewriting and polishing of some sections to make them more clear and easy to follow.

Thank you for your time, continued engagement, and candid feedback throughout the discussion. Your suggestions have helped us improve the clarity of the paper. We have uploaded a revised PDF: changes made in this round are marked in green, and earlier revisions in blue.

Qualitative differences

We agree that Appendix F previously did not clearly identify which prompt requirements to compare. All methods in Figures 12–14 use SD3.5 Medium, the same initial noise for each prompt, and 20 sampling steps. The overall compositions are similar, and the differences become clearer when examining specific prompt requirements:

  • Four wooden backpacks (Figure 13, row 1): CFG, CFG-Rescale, and SAMG each produce five backpacks; PCFG produces the requested four.
  • A plane on a grassy country runway (Figure 14, row 1): In all three baselines, the plane is on a distant paved runway, leaving the foreground grass without a plane. PCFG places a clearly visible plane on the foreground grass, more directly matching the requested setting.
  • A church with stained glass windows depicting a hamburger and french fries (Figure 12, row 2): In the baseline images, the hamburger appears three-dimensional and projects in front of the windows. In PCFG, the food appears within an arched window and is visually more integrated into the stained-glass design.

Other examples show smaller or mixed differences, and complex compositional requirements remain challenging for all methods. These examples illustrate improvements in specific prompt requirements, with their extent varying across prompts.

We have also improved the layout of Figures 11–14.

Clarity of Sec. 4.4 and related sections

We agree that the previous presentation of Section 4.4 was difficult to follow. We have rewritten it around three points:

  1. In a real SD3.5 state, two regions differ by only 2.8% in mean residual energy, yet the larger pattern discrepancy S is approximately 3.7 times the smaller one. Their corresponding guidance weights are 15.25 and 9.25.
  2. This extends beyond one example. Across 24 recorded states, regions with similar mean residual energy still exhibit different pattern discrepancies. An energy predictor using 200 quantile bins explains, on average, 45% of the variance of S in sample, and 21% at late steps.
  3. PCFG’s extra update differs in direction from a scalar-guidance extra update of the same norm by 19.4° on average. Table 3 additionally compares the complete guidance rules of PCFG and the energy-based SAMG, providing evidence on their performance.

Specifically, we made the following changes:

  • A clearer caption for Figure 6: We rewrote the caption to explain the A/B comparison, branch layout, distribution and percentile band, and update arrows.
  • Supporting details in Appendix E.2 (p. 23): We collected the experimental setup, region-matching procedure, energy binning, and update-angle definitions there, allowing the main text to focus on the reasoning.
  • Polished Section 4.3 (pp. 8–9) and improved table presentation: We present the findings on seeds, solvers, backbones, and computational cost more directly, and have refined the layout of several tables to make the results easier to read and compare.

The central point is now clearer: regions with similar mean residual energy can have different pattern discrepancies, leading PCFG to assign different guidance weights.

Thank you again for your detailed suggestions, which have helped us improve the paper's clarity. We hope these clarifications and revisions address your remaining concerns. If you find them satisfactory, we would be grateful if you could reconsider your assessment of the paper.

Reviewer_QoFZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2(−1)
Soundness
2
Read review
Paper summary

The paper proposes PCFG, a region-specific CFG technique based on the cond-uncond discrepancy. Specifically, it measures the condtion and unconditional discrepancy under the JS-divergence formulation. If the discrepancy is larger, less guidance strength is to be added, alleviating the redundancy effect of global guidance scales. The authors verify it's effectiveness across various models and benchmark and shows promising results.

Critical strengths
  1. The method and presentation is reasonable and easy to follow. The global -> redundancy -> local / region-specific -> JS-divergence line is logical sound and the experimentation verify the effectiveness of the approach.

  2. Strong empirical performance across multiple benchmarks. PCFG consistently improves ImageReward and HPSv2.1 over eight competing guidance methods across three benchmarks (COCO, DrawBench, and GenEval2) and three sampling budgets (10, 20, and 30 steps).

Critical weaknesses
  1. The extra computational cost of calculating the divergence is not reported. The method is effective, while the extra computational cost is not reported, since the author comparing PCFG with other methods with the measurement of NFE, I think this new computational cost should also be taken into consideration, otherwise directly saying 20 NFE better than 30 NFE is not precise.

  2. L2 distance might not be the general protocol in image. Measuring the L2 distance and then do JS divergence might not be optimal choice for image, since L2 distance does not generally represent the differences of images, and I think l2 also ignores the semantic differences in images (low frequency features instead of high frequency local details)

Questions for authors
  1. What is the extra computational cost when using PCFG compared to CFG's vanilla NFE?
  2. Compared to SAMG [1], what's the notable discrepancy of PCFG, given two methods all share the same ideas of applying smaller guidance scale to regions where cond-uncond gap is larger.
  3. Have the author considered turning v-pred to x-pred and apply different distance metrics (e.g. perceptual: lpips.) for distance calculation? (Or some discussion on this topic is also acceptable.)
  4. Have the authors tried combination of PCFG with multi-step solvers like Uni-PC / Heun / DPM-Solver to further show it's few-step capacity? [1] Delta Score Matters! Spatial Adaptive Multi Guidance in Diffusion Models (2026).
4 comments
AuthorsOpenReview ↗

Measured Cost and Solver Compatibility

We thank the reviewer for finding the method "reasonable and easy to follow", the reasoning "logically sound", and the empirical performance "strong across multiple benchmarks". We address each point below.

W1 / Q1: Cost beyond NFE, and the 20-vs-30-step claim in wall-clock time

We agree that NFE alone is not a complete cost measure, and we now report measured cost. PCFG adds 2.7 ms per step on top of a 198 ms denoiser step: +1.43% end to end, 126 MiB of transient memory, unchanged peak memory, and zero additional denoiser calls (full comparison: General Response Table G1). The low-budget claim therefore holds in wall-clock time. We compare 20-step PCFG against the best of ten scanned CFG scales at 30 steps, chosen separately for each benchmark and metric:

SD3.5-MTimeCOCO IR / HPSDrawBench IR / HPSGenEval2 IR / HPS
Best-scanned CFG (NFE 30)6.16 s1.057 / 0.2900.936 / 0.2851.378 / 0.311
PCFG (NFE 20)4.26 s (−31%)1.100 / 0.2910.963 / 0.2881.457 / 0.314

PCFG is ahead on all six comparisons while taking 31% less time.

W2: Is "L2L_2 then JS" appropriate for images?

We respectfully clarify a possible misunderstanding: PCFG never computes an L2L_2 distance between images or between the two predictions. The channel norm ∥fb(:,x)∥2\lVert f^{b}(:, x)\rVert_{2} is used only to assign each latent location to one of c=3c=3 shared amplitude levels (low / mid / high). The comparison itself is the JS divergence between the distributions of ordered 3×33\times3 level patterns inside each neighborhood. The statistic is therefore invariant to amplitude changes that preserve the levels, and it is sensitive to local arrangement: edges, textures, and alternating or rearranged structure.

The ablation confirms that this arrangement is what carries the signal:

Pattern representationIRHPS
CFG (w=7w=7)0.6070.251
3×33\times3 ordered patterns, 3 levels (PCFG)1.0180.279
2×22\times2 patterns (less spatial arrangement)−0.6330.190
2 levels (coarser amplitude)0.7810.261

Q2: Notable difference from SAMG

Both methods attenuate guidance where the branches differ more. They differ in three substantive ways:

SAMGPCFG
What is measuredPointwise energy E(x)=mean⁡cΔ(x)2E(x) = \operatorname{mean}_{c} \Delta(x)^{2}, a function of Δ\Delta aloneJS divergence between local pattern distributions of both branches
CalibrationMin–max normalized within each image at each step: every step spans the full [ωmin⁡,ωmax⁡][\omega_{\min}, \omega_{\max}] regardless of how much the branches differAbsolute in [0,1][0, 1]: the overall strength adapts over the trajectory (mean WW 9.9 → 13.7 → 14.6, early → late)
Information content—On real states, a 200-bin energy predictor explains 45% of SS variance in sample; energy-matched regions receive weights differing by a median of 2.0
Cost+0.71%+1.43%
PCFG − SAMG, 600 prompts × 4 seeds—CLIP +0.239, IR +0.058, HPS +0.005, Aest. +0.040

In addition, PCFG outperforms SAMG in all 18 IR/HPS comparisons in Table 1.

Q3: xx-prediction and perceptual distances

Thank you for raising this; we are happy to discuss it.

PCFG does not compute a distance between images. As in W2, SS is the JS divergence between the distributions of local level patterns in the two predictions, computed at each latent location. Replacing it with an image distance changes what is measured, not just the metric.

LPIPS solves a different problem. It compares two RGB images by feeding both through a pretrained image network (e.g., VGG) and comparing their deep features. Using it in guidance would require, at each step, (i) converting both predictions into clean-image estimates, (ii) decoding both through the VAE to pixels, and (iii) running the feature network twice. Even then, two fundamental mismatches remain:

  • It measures perceptual similarity between finished images. The network is trained on natural photographs; early clean-image estimates are blurry and far from this distribution, precisely when guidance decisions matter most.
  • It gives one score per image. PCFG needs a weight at every latent location.

Thus LPIPS is not another distance for the same quantity, but a different image-space similarity signal, requiring two VAE decodes and two network passes per step. PCFG's latent-space readout takes only 2.7 ms.

On xx-prediction. PCFG uses each model's native predictions without conversion: velocity for SD3.5 and FLUX.2, noise for SDXL and SD2.1. Gains hold across all of them (Tables 4 and 2 of the revised paper), so it does not depend on one parameterization.

Q4: Multi-step solvers

We compared several solvers; full results are in Table G3a of the General Response. All 40 paired differences are positive, and 39 are statistically significant.

We hope these results address the reviewer's concerns and welcome further discussion.

Dear Reviewer QoFZ,

We sincerely thank you for your constructive feedback, which has helped us strengthen the paper and clarify its contributions.

We hope that our detailed responses and additional results have adequately addressed your concerns. Should any questions or concerns remain, we would be more than happy to provide further clarification.

If you find that our responses have satisfactorily resolved your concerns, we would greatly appreciate your reconsideration of the score and stronger support for the paper.

Thank you again for your time and thoughtful evaluation.

Best regards,

The Authors

Thank you for the detailed and thoughtful response, NFE issue is addressed, distincition is clarified, my main concern is still the limited distinction from previous approaches, i.e. the parameterization and formulation, while the idea is similar I think. Overall, given the thoughtful response, I raise my score.

Thank you very much for the thoughtful follow-up and for reconsidering our work. We greatly appreciate your time and constructive feedback.

Reviewer_6Lg2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Pattern-based Classifier-Free Guidance (PCFG)​, a training-free method that adapts CFG spatially using discrepancies between local pattern distributions of conditional and unconditional predictions. It captures spatial relationships beyond pointwise score differences and requires no additional denoiser evaluations. On SD3.5 Medium, one fixed configuration outperforms the metric-specific best-scanned scalar CFG in 41 of 45 comparisons across three benchmarks and three sampling budgets.

Critical strengths
  1. The paper clearly situates its method among related guidance approaches.
  2. PCFG is a simple inference-time modification and reports promising gains across multiple benchmarks and sampling budgets.
Critical weaknesses
  1. The motivation for using local patterns needs further analysis. It is unclear what information they add beyond the magnitude of the conditional–unconditional score difference.
  2. Evaluation is limited to SD3.5 Medium. Testing stronger models and other modalities would better establish the method’s generality.
  3. The reported gains appear modest; results across multiple random seeds with confidence intervals would help establish their reliability.
Questions for authors
  1. What additional information do local pattern distributions capture compared with pointwise score-difference magnitudes?
  2. Could PCFG generalize to video or other multimodal generation tasks?
  3. How costly is hyperparameter selection, and how sensitive is performance to interactions among the hyperparameters?
4 comments
AuthorsOpenReview ↗

Mechanism, Reliability, and Hyperparameter Sensitivity

We thank the reviewer for the constructive feedback and for recognizing PCFG as a "simple inference-time modification", clearly positioned among related methods, with "promising gains across multiple benchmarks and sampling budgets". Each suggestion led to new experiments, summarized below.

W1 / Q1: What local patterns capture beyond the residual magnitude

In short: how the two predictions are locally organized, which pointwise magnitudes discard by construction. The residual-energy rules considered here depend on Δ=fy−fu\Delta = f^{y} - f^{u}. PCFG's statistic depends on the local arrangement of both fields. The same residual added to different unconditional fields has identical residual energy but can lead to different local amplitude-pattern distributions. PCFG uses pattern discrepancy as a heuristic control signal: larger discrepancies receive less additional extrapolation, and smaller discrepancies receive more. This policy is evaluated empirically; the statistic does not certify semantic correctness or extrapolation safety.

We verified this substantial difference on 24 real SD3.5 states (8 prompts × early / middle / late steps):

Measured on real SD3.5 statesValue
In-sample variance of SS explained by a 200-quantile-bin predictor of local residual energy45% on average (only 21% at late steps)
Neighborhood pairs with ≤5%\le 5\% mean-energy difference3,336
Median PCFG weight difference within these energy-matched pairs2.0 (49% of pairs >2> 2)
Angle between PCFG's extra-guidance displacement and its norm-matched scalar counterpart19.4° on average
Mean weight at early → middle → late steps9.9 → 13.7 → 14.6 (automatic schedule)

The last row shows that the mean guidance strength also varies along these recorded trajectories. The values describe observed behavior and do not guarantee the same schedule for every prompt or model.

The complete PCFG rule also performs well empirically. Over 600 prompts × 4 seeds, it improves the following four metrics relative to the tested SAMG configuration. This comparison includes differences in signal, calibration, and guidance mapping:

PCFG − SAMGCLIPIRHPSAest.
4 seeds × 600 prompts+0.239+0.058+0.005+0.040

See our reply to n9cc and the new Sec. 4.4 for details.

W2 / Q2: Stronger models and other modalities

The submission already included SDXL (Table 2). We now add:

New evaluation (600 prompts each)Result
4 solvers × 2 budgets on SD3.5 (Euler, UniPC, Heun, DPM-Solver++)39 / 40 metric differences significant
SD2.1 with S-CFG, CFG-Rescale, APGBest CLIP / IR / HPS at 20 steps; best on all five quality metrics at 50 steps
FLUX.2 Klein Base 4B, untunedRuns unchanged; HPS significantly improves, PickScore significantly decreases; CLIP/IR intervals include zero

W3: Magnitude of the gains and multiple seeds

Reliability. Over 600 prompts × 4 seeds, with paired comparisons and 95% intervals over seeds:

PCFG −CLIPIRHPSPickAest.
CFG+0.201+0.085+0.009+0.004+0.075
SAMG+0.239+0.058+0.005+0.000+0.040
CFG-Rescale+0.265+0.045+0.005+0.001+0.053

Further details are in General Response Tables G2a–b.

Magnitude. We respectfully note two points:

  1. The reference is deliberately strong. The paper compares one fixed PCFG configuration against a hindsight envelope: the best of 10 CFG scales, selected separately for every metric, benchmark, and budget. PCFG exceeds this envelope in 41 of 45 cases.
  2. The measured overhead is small. At 20 steps on SD3.5 with the tested A100 configuration, PCFG takes 1.43% longer than the reported CFG reference and uses no extra denoiser calls (Table G1).

Q3: Cost of hyperparameter selection and sensitivity

Cost. The search comprised 48 runs on 100–200 prompts at 10 steps: about 8,000 images or 5 A100 GPU-hours, small compared with the evaluation itself.

Sensitivity. The guidance mapping is robust, while the representation matters, as expected if the signal carries the benefit. The rows summarize a joint grid and separate sensitivity tests, with the CFG reference shown per row; scores are ImageReward on SD3.5-M at 10 steps:

Factor variedRange testedIR rangeCFG (w=7w=7)
wmax⁡×γw_{\max} \times \gamma (full grid){7,10,13,15,18}×{1,2,3}\{7, 10, 13, 15, 18\} \times \{1, 2, 3\}0.76 – 0.94 (all 15 >CFG> \mathrm{CFG})0.57
Context size kk7 – 251.00 – 1.050.61
Floor wmin⁡w_{\min}1 – 40.98 – 1.020.61
Ceiling wmax⁡w_{\max}18 – 271.00 – 1.020.61
Seeds4 seeds0.95 – 1.040.61
Pattern size pp / levels ccp=2p=2 / c=2c=2−0.633 / 0.7810.61

We hope this new evidence addresses the reviewer's concerns and welcome further discussion.

Dear Reviewer 6Lg2,

We sincerely thank you for your constructive feedback, which has helped us strengthen the paper and clarify its contributions.

We hope that our detailed responses and additional results have adequately addressed your concerns. Should any questions or concerns remain, we would be more than happy to provide further clarification.

If you find that our responses have satisfactorily resolved your concerns, we would greatly appreciate your reconsideration of the score and stronger support for the paper.

Thank you again for your time and thoughtful evaluation.

Best regards,

The Authors

Reviewer_6Lg2OpenReview ↗

Sorry for the late reply

Thanks for your reply. Although I want to see the results in models like MOSS-Image/Hunyuan-Image 3.0/Z-Image etc open-source foundation model and in modalities like videos, the reply addressed my concern about the motivation. I will increase my score accordingly.

And just for discussion: does the local patterns come from the patchification operations in DiT? Since the velocities are actually predicted independently for every patch in the final velocity prediction layer. For pixel-space diffusion, many papers observed the implicit 'patch effect' in the final image (i.e, the image are combined by patches) induced by patchifcations, will PCFG make this effect more obvious?

Thank you for the positive reassessment and for this thoughtful question. We also appreciate your suggestions regarding newer foundation models and video generation. Regarding the relationship between local patterns and DiT patchification, we would like to clarify this with three points:

  1. The statistic is defined on prediction fields. PCFG compares local pattern distributions of the conditional and unconditional predictions at each sampling step. In SD3.5, these are velocity fields on the latent grid. Here, “local patterns” describes spatial arrangements of prediction amplitudes; we do not assume that these arrangements originate from DiT patchification.
  2. The construction also applies beyond DiT architectures. The same construction yields benefits on UNet-based SDXL and SD2.1, which do not use DiT-style patchification (Table 2 in the paper and Table G3b in the General Response). This supports that PCFG does not require this architectural feature.
  3. The statistical neighborhoods cross token boundaries. In SD3.5, PCFG operates on the spatial prediction fields after unpatchification. Both the small pattern windows and the larger statistical neighborhoods slide with stride one. Adjacent locations share most of their observations, and PCFG does not impose a token-aligned partition on its guidance map. The final DiT projection is indeed token-wise, but its input representations already incorporate cross-token context through attention.

Regarding possible amplification of patch artifacts, we did not observe systematic grid-like artifacts in the PCFG samples we inspected. PCFG introduces no independent tile generation or patchwise reconstruction: its windows compute guidance statistics, and the resulting weights rescale the original CFG residual at each location. Overlapping neighborhoods provide cross-boundary context. Our observation concerns the evaluated latent-space models; examining patch effects in pixel-space diffusion is an interesting direction for further study.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Additional Experiments and Clarifications (1/2)

General Response: New evidence addressing the shared concerns

We thank all reviewers for their constructive feedback. They found PCFG "well motivated by the observation and analysis that identical pointwise residual energy can conceal different spatial arrangements" (n9cc, vqcs), "logically sound" with "strong empirical performance across multiple benchmarks" (QoFZ), a "simple inference-time modification" with "promising gains across multiple benchmarks and sampling budgets" (6Lg2), and reported that "extensive experiments demonstrate generally consistent gains across diverse settings and metrics" (n9cc, vqcs).

We address four shared questions: (i) computational cost, (ii) statistical reliability across seeds, (iii) generality beyond SD3.5, and (iv) what local patterns add beyond residual magnitude. New controlled experiments generated about 40,000 images to address these questions, all using a single fixed PCFG configuration, (c,p,k,wmin⁡,wmax⁡,γ)=(3,3,13,3,18,2)(c, p, k, w_{\min}, w_{\max}, \gamma) = (3, 3, 13, 3, 18, 2).

Shared questionNew experimentHeadline result
(i) Cost9 methods, one pipeline, timing in one pipeline, percentages relative to the reported CFG reference+1.43% latency, no extra denoiser calls; 20-step PCFG outperforms 30-step CFG in 31% less time
(ii) Reliability600 prompts × 4 seeds × 4 methods (9,600 images)Better than CFG on all 5 metrics in 4/4 seeds; significantly better than SAMG / Rescale on 4 of 5 metrics
(iii) Generality4 solvers × 2 budgets (9,600 images); SD2.1 with S-CFG (6,000); FLUX.2 (1,200)39/40 solver comparisons significant; best CLIP / IR / HPS at 20 steps and all five metrics at 50 steps on SD2.1; runs on FLUX unchanged
(iv) Mechanism24 real SD3.5 states; 3,336 energy-matched neighborhood pairsA 200-bin energy predictor explains 45% of SS variance in sample; extra-guidance displacements differ by 17.7° at equal norm
Coverage (n9cc)COCO-5k distribution metricsPrecision ↑, coverage ↑, FID 31.8 → 30.7

G1. Measured cost on SD3.5

SD3.5-Medium, A100-80GB, 102421024^{2}, 20 steps, 10 prompts × 6 rounds.

Table G1. Cost of all nine guidance methods. Percentages use the displayed times relative to the listed CFG reference, not method-specific paired estimates; small time differences should not be used to rank methods.

Methods / imageOverhead vs. CFGGuidance (ms / step)Extra peak memory (guidance step)Branch predictions
CFG4.202—0.011 MiB20 × 2
CFG-Rescale4.257+1.31%0.062.5 MiBsame
CFG++4.312+2.62%0.011 MiBsame
CFG-Zero*4.287+2.02%0.051.5 MiBsame
APG4.216+0.33%0.1212.5 MiBsame
CFG-Ctrl4.251+1.17%0.032.5 MiBsame
VAGS4.219+0.40%0.115 MiBsame
SAMG4.232+0.71%0.083 MiBsame
PCFG4.262+1.43%2.73126 MiBsame
  • Measured overhead: at 20 steps, PCFG adds 0.060 s/image over CFG.
  • Peak memory is unchanged: the whole-pipeline peak is 17.60 GiB for every method.
  • The dense [B,39,H,W][B, 3^{9}, H, W] tensor is never formed, and nothing is approximated. Each neighborhood contains at most 242 of the 19,683 possible patterns; the others contribute exactly zero to JS, so counting only the observed patterns gives the same SS with 126 MiB of additional memory and 2.7 ms of computation per step instead of 8.7 GiB and 93 ms (reply to vqcs; App. G).
  • Hence the low-budget claim holds in wall-clock time. 20-step PCFG (4.26 s) exceeds the best-scanned 30-step CFG (6.16 s) on ImageReward and HPSv2.1 on all three benchmarks: the same quality in 31% less time (details in our reply to QoFZ).

G2. Reliability: consistent across seeds, significant against strong baselines

SD3.5-Medium, 20 steps, 600 prompts (200 each from COCO / DrawBench / GenEval2) × 4 seeds {4416,5525,6636,7749}\{4416, 5525, 6636, 7749\}.

Table G2a. Absolute scores (mean ± s.d. over the four seeds).

MethodCLIPIRHPSPickAest.
CFG (w=7w=7)29.363 ± 0.1081.103 ± 0.0600.289 ± 0.0040.867 ± 0.0015.406 ± 0.019
CFG-Rescale29.299 ± 0.1481.142 ± 0.0380.293 ± 0.0030.870 ± 0.0015.428 ± 0.019
SAMG29.326 ± 0.0631.129 ± 0.0350.293 ± 0.0030.871 ± 0.0025.441 ± 0.017
PCFG29.564 ± 0.0771.187 ± 0.0450.298 ± 0.0030.871 ± 0.0025.481 ± 0.020

Table G2b. Paired difference PCFG − baseline, with 95% t-intervals over seeds and the number of seeds with a positive difference.

PCFG −CLIPIRHPSPickAest.
CFG+0.201 [0.121, 0.282] · 4/4+0.085 [0.049, 0.121] · 4/4+0.009 [0.004, 0.014] · 4/4+0.004 [0.003, 0.005] · 4/4+0.075 [0.038, 0.113] · 4/4
CFG-Rescale+0.265 [0.136, 0.395] · 4/4+0.045 [0.023, 0.067] · 4/4+0.005 [0.003, 0.007] · 4/4+0.001 [−0.001, 0.002] · 3/4+0.053 [0.031, 0.076] · 4/4
SAMG+0.239 [0.191, 0.286] · 4/4+0.058 [0.041, 0.076] · 4/4+0.005 [0.003, 0.007] · 4/4+0.000 [−0.001, 0.002] · 3/4+0.040 [0.017, 0.063] · 4/4
AuthorsOpenReview ↗

Additional Experiments and Clarifications (2/2)

G2. Reliability (continued from Part 1)

  • 13 of 15 comparisons are significant; the remaining two (PickScore against Rescale / SAMG) have intervals spanning zero, so no significant difference was detected.
  • Per benchmark, 43 / 45 mean differences are positive, 29 / 45 intervals exclude zero, and none is significantly negative.
  • Image-level paired win rate against CFG: 69% (HPS), 61% (IR), 61% (Aesthetic), 57% (Pick), 55% (CLIP).

G3. Generality: solvers, budgets, architectures, model families

Table G3a. Four solvers at matched NFE on SD3.5-Medium, 600 prompts. NFE counts denoiser forwards with both branches in one batch: the main 10/20/30 steps cost NFE 10/20/30. Heun's first-order last step gives NFE 2N−12N-1 for NN steps, allowing only odd budgets. We use NFE 9/19: 5/10 steps for Heun and 9/19 for the others. The first metric column shows IR CFG → PCFG; the others show paired differences PCFG − CFG.

SolverNFEIR: CFG → PCFGΔ\Delta CLIPΔ\Delta IRΔ\Delta HPSΔ\Delta PickΔ\Delta Aest.
Euler90.680 → 1.002+0.381+0.322+0.022+0.013+0.108
UniPC (2nd order)90.791 → 1.055+0.277+0.264+0.018+0.012+0.115
Heun90.325 → 0.894+0.626+0.568+0.033+0.024+0.292
DPM-Solver++90.809 → 1.078+0.395+0.269+0.019+0.012+0.101
Euler191.071 → 1.179+0.315+0.108+0.008+0.003+0.057
UniPC (2nd order)191.082 → 1.182+0.355+0.100+0.007+0.004+0.058
Heun191.037 → 1.173+0.035+0.136+0.013+0.009+0.110
DPM-Solver++191.102 → 1.197+0.202+0.095+0.006+0.002+0.060

Table G3b. SD2.1-base (UNet, ϵ\epsilon-prediction, 5122512^{2}), DPM-Solver++, 600 prompts. Five methods share one pipeline; the official S-CFG code is ported.

StepsMethodCLIPIRHPSPickAest.s / img
20CFG26.7350.0990.2580.8335.5040.79
20S-CFG26.8360.0820.2600.8345.5100.87
20CFG-Rescale26.8220.1200.2600.8345.5100.77
20APG25.834−0.1580.2440.8215.4550.77
20PCFG27.1410.1350.2640.8325.4940.80
50CFG26.7330.0600.2610.8355.5121.78
50S-CFG26.8840.0900.2620.8365.5111.99
50CFG-Rescale26.8070.0880.2620.8365.5081.78
50APG26.326−0.0870.2460.8295.4751.77
50PCFG27.1170.1710.2660.8385.5151.85

Table G3c. FLUX.2 Klein Base 4B (50 steps, 600 prompts).

CLIPIRHPSPickAest.
CFG (w=4w=4)28.5781.0630.2910.8705.466
PCFG28.7091.0730.2930.8685.469
Δ\Delta [95% CI]+0.131 [−0.023, 0.283]+0.010 [−0.024, 0.047]+0.002 [0.001, 0.003]−0.003 [−0.004, −0.001]+0.003 [−0.015, 0.020]

Among the five SD2.1 methods, PCFG has the best CLIP, IR, and HPS at 20 steps and the best scores on all five quality metrics at 50 steps. It needs no attention maps and is faster than S-CFG (20 steps: 0.80 vs. 0.87 s/image; 50 steps: 1.85 vs. 1.99 s/image). On FLUX, it runs without retuning: HPS improves significantly, PickScore decreases significantly, and CLIP, IR, and Aesthetic intervals include zero, supporting compatibility with metric-dependent trade-offs. With the paper's SDXL results, PCFG covers velocity and noise prediction, MMDiT and UNet, and the SD and FLUX families.

G4. Distribution metrics (requested by n9cc)

Table G4. COCO-val, 5,000 real images vs. 5,000 generated images per method (SD3.5-M, 20 steps), Inception features. FID is independent of the P/R neighborhood size and is repeated unchanged.

kk-NNMethodPrecision ↑Recall ↑Density ↑Coverage ↑FID ↓
3CFG0.7260.3831.1470.72931.811
3PCFG0.7400.3861.2410.75330.708
5CFG0.8080.4911.1550.86031.811
5PCFG0.8230.4931.2470.87930.708

G5. Changes in the revised PDF

  • New Sec. 4.3 (main text): multi-seed reliability (Table 3), solvers at matched NFE and other backbones (Table 4; SDXL, SD2.1 with S-CFG, FLUX.2), and measured efficiency.
  • New Sec. 4.4 (main text): the trained-model analysis requested by n9cc (Figure 6).
  • New App. G: why the dense histogram is never formed, and the measured cost of nine methods (Table 11). NFE-only wording is replaced by measured latency.
  • New App. H: multi-seed results per benchmark (Tables 12–13).
  • New App. I: solvers, SD2.1 with S-CFG, and FLUX.2 in full (Tables 14–16).
  • App. E: search cost of the configuration.
  • App. F: samples for randomly drawn prompts (Figures 12–14).
  • Sec. 2: discussion of the four works suggested by n9cc.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.