- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes JIVE, a training-free method for improving sample diversity in flow-based generative models. The main idea is to perturb the sampling trajectory along directions that induce large changes in the predicted endpoint, identified using the leading right singular subspace of the endpoint Jacobian. The authors also provide a theoretical analysis connecting these perturbations to increased pairwise distances between generated samples, and introduce efficient approximations to avoid explicitly computing the Jacobian. Experiments on FLUX and SD3.5 show improvements in sample diversity, including few-step and one-step generation, with additional results on inversion-based generation.
Overall, I have a positive impression of this paper. The presentation is clear, the proposed approach seems reasonable, and the experimental results are encouraging. I therefore lean towards acceptance. This topic is somewhat outside my main research area, and I am not sufficiently familiar with the relevant literature to make a confident assessment of the paper's novelty relative to prior work. My recommendation should therefore be considered with relatively low confidence.
Critical strengths
-
The paper is well written and easy to follow. The motivation for using the endpoint Jacobian to guide perturbations is clearly explained, and the toy example helps illustrate why this approach can be useful compared with existing diversity guidance methods.
-
The experimental evaluation is fairly comprehensive. The authors consider different generative models and sampling settings, and report both diversity and quality metrics. I also appreciate the analysis of computational overhead and the inclusion of low-cost approximations, which make the proposed method more practical.
Critical weaknesses
The proposed method introduces additional computation during sampling, particularly when estimating the leading singular directions. Although the authors provide cheaper approximations and report their computational costs, this overhead may still be relevant for practical applications where inference speed is important.
I do not have other major concerns based on my reading.
Questions for authors
I have a few questions for the authors:
-
Perturbation timing. The experiments mainly apply JIVE at the initial sampling step (). Is there a particular reason for this choice? Would applying the perturbation at intermediate steps lead to different results?
-
Perturbation magnitude. The paper uses a heuristic to determine the perturbation norm. How sensitive is the method to this choice? Does the same heuristic work well across different generative models?
-
JVP vs. JᵀJ. The JᵀJ variant generally achieves better diversity-quality tradeoffs, while JVP is computationally cheaper. Could the authors provide more guidance on when each variant would be preferable in practice?
5 comments
[2/4] Rebuttal by Authors
Q1. Is there a particular reason for applying mainly at the initial sampling step rather than a later one? Would applying the perturbation at intermediate steps lead to different results?
Thank you for raising this question. We inject at the first step since we reasoned that injecting sooner would cause a greater overall change to the trajectory. This is supported by the table below where we compare injecting at the first step and at intermediate steps. Injecting at the first step raised diversity (Vendi-DINO) the most, but also had greater losses in terms of quality (HPSv2 and Pickscore.) Overall, injecting sooner seems to move JIVE further away from the baseline on the diversity-fidelity trade-off curve.
We test on a multi-step model (Flux.1 dev) below.
FLUX.1-dev (28 step, ) GenEval at 4 images per prompt. Values are mean se
JIVE (, norm 8), injected at timestep K
| Injection step | Vendi DINO | Vendi pixel | CLIP score | HPSv2 | PickScore | GenEval |
|---|---|---|---|---|---|---|
| Baseline | 2.054 ± 0.024 | 1.516 ± 0.013 | 32.49 ± 0.14 | 30.41 ± 0.12 | 23.40 ± 0.04 | 0.651 |
| K = 0 | 2.220 ± 0.025 | 1.600 ± 0.013 | 32.49 ± 0.14 | 29.54 ± 0.11 | 23.19 ± 0.04 | 0.639 |
| K = 3 | 2.138 ± 0.025 | 1.572 ± 0.014 | 32.47 ± 0.14 | 30.16 ± 0.11 | 23.29 ± 0.04 | 0.646 |
| K = 8 | 2.088 ± 0.024 | 1.523 ± 0.013 | 32.44 ± 0.14 | 30.34 ± 0.12 | 23.34 ± 0.04 | 0.643 |
| K = 14 | 2.074 ± 0.024 | 1.517 ± 0.013 | 32.45 ± 0.14 | 30.35 ± 0.12 | 23.35 ± 0.04 | 0.652 |
[3/4] Rebuttal by Authors
Q2: How sensitive is the method to the choice of perturbation norm? Does the same heuristic work well across different generative models?
We thank the reviewer for this question. The heuristic in Sec. 2.2 gives a model-specific upper bound on the norm, which keeps the perturbed latent near the typical norm of a Gaussian sample. Thus, our proposed heuristic provides a practical upper bound on the perturbation norm, rather than a recommended or optimal value.
The choice of heuristic norm depends only on the latent dimension (about 26 for FLUX.1-schnell and 38 for SD3.5-Turbo in Fig 5). Within this range, the norm acts as a control on the diversity–quality tradeoff. To assess sensitivity, we swept the norm for both variants on both models in Fig. 5: over {4, 6, 8, 12, 19} on FLUX.1-schnell and {8, 12, 16, 24} on SD3.5-Turbo, and JVP over {8,12,19,26,35,45} and {19,26,35,45}, spanning roughly 0.15–1.7× the respective bounds. Across these ranges, diversity and quality change smoothly and monotonically with the smaller norm than herustic norm. However, as norm approaches the herustic norm, quality would become more sensitive and thus degrade more quickly. The same behavior holds on both models when norms are expressed relative to each model’s bound, which indicates that the heuristic transfers. As noted in our limitations, the heuristic provides a practical upper bound rather than an optimal perturbation norm. In practice, we recommend choosing a smaller norm within this bound, as Fig. 5 shows that smaller perturbations generally achieve more favorable diversity–quality trade-offs.
[1/4] Rebuttal by Authors
We thank reviewer VB4b for the helpful feedback. We are glad that the reviewer found the paper easy to follow and the explanation of using endpoints Jacobians clear. We address the reviewer's questions below.
CW1: Computational overhead of JIVE.
Thanks reviewer for highting this. We acknowledge that JIVE incurs a computational overhead. However, unlike methods that require interventions throughout the sampling process, JIVE applies a single perturbation at the beginning of sampling, with no further interventions required during subsequent denoising steps. Hence, JIVE only needs to compute a perturbation once.
To evaluate the practical computational cost, we measure the wall-clock cost of generating images with JIVE (JVP) with one direction and one iteration and OSCAR on FLUX.1-schnell at 512×512 with 4 denoising steps. Both methods generate a batch of 16 images, and the per-image time is the batch time divided by 16, averaged over 5 runs:
Thus, while JIVE introduces overhead compared with unmodified sampling, its computational cost remains competitive with existing diversity-enhancement methods.