- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CompFM, a training-free method for composing pretrained flow models by introducing a closed-form score correction based on Gaussian approximations of endpoint posteriors. Experiments on synthetic and image datasets demonstrate improved compositional generation compared to score addition and velocity combination baselines.
Critical strengths
The paper provides a theoretically grounded approach to sampling from a specified compositional distribution, supported by theoretical analysis and experimental validation. The proposed method also achieves promising results without additional training.
Critical weaknesses
- Limited justification of the target distribution. It remains unclear whether accurately sampling from (p_A^\alpha p_B^\beta) necessarily improves generation quality, particularly for CFG and high-dimensional image generation. The product-based distribution may not always be the optimal target for semantic composition or perceptual quality.
- Computational overhead and scalability. The method requires additional velocity-Jacobian-related computations, introducing substantial inference overhead. Its efficiency and scalability to higher-dimensional, high-resolution image generation remain insufficiently demonstrated.
Questions for authors
- Could the authors further evaluate CompFM on more challenging, higher-dimensional generation tasks, such as ImageNet-64 in pixel space or ImageNet-256 in latent space? Given the reliance on covariance approximations and additional Jacobian-related computations, such experiments would help assess the scalability, computational efficiency, and practical effectiveness of the proposed method.
- Practical benefits for CFG: The covariance-aware CFG formulation is theoretically interesting, but its evaluation is mainly limited to Gaussian settings. Could the authors clarify whether more accurate distribution composition translates into better generation quality or conditional alignment compared with standard CFG on realistic image generation tasks?
I would be willing to reconsider my rating if the authors can provide convincing evidence or clarification addressing these concerns, particularly regarding the practical benefits of the proposed composition target and the scalability of the method to more challenging generation tasks.
5 comments
Part2
We then performed the pixel-space ImageNet-64 and latent-space ImageNet-256 experiments, analyzed computational efficiency, and summarized our clarity revisions (Weakness 2 & Question 1).
1. Scalability: CompFM raises target purity across both pixel-space ImageNet-64 and latent-space ImageNet-256 (Question 1) We ran both benchmarks you proposed, keeping our scalar covariance estimator completely unchanged and pairing it with a 100-step predictor-corrector sampler.
Composition combines separately trained models with overlapping support, whereas public checkpoints are single conditional networks. We therefore fine-tuned public ADM checkpoints (296M parameters) for pixel-space ImageNet-64 ( dimensions) and public SiT-XL/2 checkpoints (675M parameters) for latent-space ImageNet-256 into three unconditional flows each on a 10-class ImageNet subset. Each flow covers 5 classes, and each pair shares either one class (church or golf ball) or three classes (tench, English springer, and cassette player). Target purity is the percentage of generated images assigned to the shared target by a 10-class ResNet-50 classifier (5 seeds, 256 samples per seed):
| Benchmark & model | Shared target | Velocity avg. | Summed scores | CompFM |
|---|---|---|---|---|
| ImageNet-64, pixel (ADM) | Church | |||
| Golf ball | ||||
| Three-class set | ||||
| ImageNet-256, latent (SiT-XL/2) | Church | |||
| Golf ball | ||||
| Three-class set |
CompFM raises target purity over both baselines on all six benchmarks, exceeding two standard deviations on five of the six. Together with our ImageNet-256 Bayesian restoration result in Part 1 ( dB for CompFM vs. dB for summed scores and dB for the flow trained on both observations, on the same 10 classes), this confirms both the scalability and practical effectiveness of our scalar approximation on class intersection (ImageNet-64 and ImageNet-256) and restoration (ImageNet-256).
2. Computational efficiency: no full Jacobians, and a flat runtime ratio from 9M to 675M parameters (Weakness 2) Our theoretical formulas write posterior covariances using velocity Jacobians, so it is natural to ask about their cost. In practice, CompFM never forms a Jacobian matrix (§4 now states this upfront). Each model's scalar variance uses a single Hutchinson Jacobian-vector product (JVP), and the score correction uses one scalar backward pass.
Because one JVP and one backward pass each cost a fixed small multiple of a forward pass, the runtime ratio does not grow with model size. Under our 100-step predictor-corrector sampler (re-estimating covariances at every step), CompFM takes about the time of summed scores on a 9M-parameter UNet (AFHQ-64), on the 296M-parameter ADM (ImageNet-64), and on the 675M-parameter SiT-XL/2 (ImageNet-256) (and end-to-end under our CIFAR-10 annealed sampler). Refreshing the covariance every 25 steps instead of every step cuts the AFHQ-64 runtime ratio to ( s vs. s on an idle GPU) without losing purity. The main cost is peak GPU memory (– summed scores at ImageNet scales: vs. GB on ADM, and vs. GB on SiT-XL/2) from saving activations for the backward pass, which can be lowered with smaller batches.
3. Manuscript clarity improvements (Presentation) To make the paper easier to read, we added a scope paragraph in §1 separating target choice from path correction, a step-by-step compute summary in §4 clarifying JVP estimation, and a notation table alongside explicit -factor fusion formulas in App. A.
Dear reviewer, as the discussion deadline approaches, we wanted to share quick pointers to where your questions are addressed and check if anything remains unclear:
- Practical applications where composition beats single conditionals (§1, §6.3, §6.5, Tables 3–4): Without paired training data, dual-sensor Bayesian restoration (, , and ImageNet-256 latents) beats single-sensor flows and both baselines at every scale—matching jointly trained flows at () and ()—and 30-pair attribute composition averages vs. for a single flow.
- Inference cost breakdown (§4, §6.4, App. B.1 Table 11): CompFM never forms full Jacobians; at 2 steps/slice on an H100, JVP divergence probes take – of time, the backward pass , and forward passes ( CIFAR end-to-end).
- Scaling with model size (§6.4, Table 11): Because JVPs and backward passes cost a fixed multiple of a forward pass, the 100-step PC runtime ratio stays flat from 9M to 675M parameters ( UNet, ADM, SiT-XL/2), and 25-step covariance refresh lowers AFHQ-64 to .
Please let us know if any additional details would be helpful—thank you again for your review!
Part1
Thank you for your thoughtful review and for stating clearly what evidence would change your assessment. We ran all of the higher-dimensional evaluations you suggested, including both pixel-space ImageNet-64 and latent-space ImageNet-256. Part 1 addresses the practical benefits of the composition target and covariance-aware CFG (Weakness 1 & Question 2). Part 2 covers ImageNet-64 and ImageNet-256 scalability, computational efficiency, and clarity revisions (Weakness 2 & Question 1).
1. Practical value of the target: when the product is the desired target, accurate path sampling improves image quality and reconstruction (Weakness 1) Selecting the composition operator (product, geometric mean, or negation in §3, or K-factor Bayesian conjunction in Rem. 3) is an upstream modeling choice, and the raw product is not intended as a universal semantic target. Once a target is chosen, however, summed scores and CompFM share that exact target distribution at t=1. Summed scores follow biased intermediate marginals that omit ∇logRt (Thm. 2), while CompFM estimates ∇logRt in closed form to correct the path without training. Comparing CompFM with summed scores under the identical sampler therefore isolates what accurate sampling buys.
Benefit A: Lower KID than summed scores in class intersection. On AFHQ-64 and pixel-space ImageNet-64 (ADM, Part 2), each flow models a subset of classes, so their product targets the shared classes. Using the identical 100-step predictor-corrector sampler, our path correction lowers KID to real target-class images (×10−3, lower is better, 5 seeds) across all six targets, each by more than two standard errors:
Benefit B: Bayesian restoration where Bayes' rule fixes the weighted product. When independent sensor measurements y1 and y2 observe an image x without paired training data, Bayes' rule fixes the posterior p(x∣y1,y2)∝p(x∣y1)p(x∣y2)/p(x), our (1,1,−1) member. At three scales, we separately trained a 50% pixel-inpainting flow, a noisy block super-resolution flow, and an unconditional prior, then composed them. Reconstruction PSNR on 256 test images (dB, 3–5 seeds):
Without ever seeing paired observations during training, CompFM beats both baselines at every scale and matches the flow trained on both observations at 64×64 and ImageNet-256.
2. Practical benefits for CFG: on ImageNet-256, covariance-aware CFG matches standard CFG at a smaller guidance scale, as our fusion rule predicts (Question 2) CFG is a case where users do not want faithful samples from pc1+γp∅−γ. Instead, CFG sharpens samples toward high-confidence class modes, as shown by Bradley & Nakkiran (2024) and Chidambaram et al. (2024). Our fusion rule (Eq. 3, Cor. 1) predicts what covariance-aware fusion does here: along class-informative directions where the unconditional posterior is broader than the conditional one, precision weighting pulls less strongly than standard CFG.
We tested this on 1,000-class ImageNet-256 with pretrained SiT-XL/2 (5,000 images per point). Full-latent scalar variances of the two branches are nearly equal (ratio 0.96–1.00), so scalar fusion is almost identical to CFG. Testing the variant that differs most from CFG (a separate covariance on the coarsest frequency band) gives the following generation quality (FID) and conditional alignment (ResNet-50 top-1 accuracy):
At each scale s, covariance-aware CFG lowers FID and raises recall while slightly lowering top-1 accuracy. This traces the same trade-off as standard CFG at a smaller scale (within noise at equal FID). We therefore do not claim an improvement over CFG, and the abstract now calls this a covariance-aware form of CFG rather than a correction.