- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Joint Semantic Flow Matching (JsFlow), which jointly generates image latents and semantic tokens from pretrained CLIP, DINOv1, and DINOv2 encoders. Its central argument is that semantic variables can reduce ambiguity in image-velocity prediction when they provide information unavailable from the intermediate image state. JsFlow completes semantic transport before image transport, allowing the generated semantic state to guide later image refinement. An enhanced variant uses a detached semantic prediction branch to limit gradient interference.
The authors prove a conditional-variance reduction result and evaluate the method on ImageNet 256×256 across LightningDiT and SiT backbones. On LightningDiT-XL/1, JsFlow reports an unguided FID of 5.91 after 50K iterations and a guided FID of 1.18 after 4M iterations. Ablations examine encoder choice, transport timing, and gradient decoupling.
Critical strengths
-
Strong gains at matched training budgets. With LightningDiT-XL/1 at 50K iterations, JsFlow reduces unguided FID from REG’s 11.76 to 5.91. It reaches roughly REG’s 400K-iteration FID in about 100K iterations. This substantial early-training improvement is the strongest reason to consider acceptance.
-
Evidence beyond one model configuration. Improvements are reported across LightningDiT model sizes and on SiT, with ablations supporting the choices of semantic features, transport timing, and gradient decoupling. This makes the result more compelling than a gain confined to one backbone.
Critical weaknesses
-
The proposed mechanism is not directly validated. The conditional-variance result applies generally when a predictor receives additional information. The paper uses lower training MSE as a proxy for reduced irreducible velocity ambiguity, but MSE also reflects optimization and model error. This leaves the central explanation for JsFlow’s gains unconfirmed.
-
The gains are insufficiently attributed to individual design choices. Comparisons with REG change the number of semantic tokens, transport schedule, and architecture together. The detached variant also adds about 100M parameters. Without controlled, capacity-matched ablations, it is difficult to tell which contributions are necessary for the reported improvement.
Questions for authors
-
Clarification of existing evidence: Figure 6 uses image-velocity MSE to support the velocity-ambiguity explanation. What evidence separates a reduction in irreducible conditional variance from reductions in optimization or approximation error?
-
Additional evidence that could change my assessment: Are there comparisons that isolate multiple semantic tokens, early semantic completion, and gradient decoupling under the same training setup, with a capacity-matched control for JsFlow†? If so, which component accounts for the largest gain?
8 comments
Response to Reviewer tSG5 part2,2/2
Reply to W2 and Q2
[W2] The gains are insufficiently attributed to individual design choices. Comparisons with REG change the number of semantic tokens, transport schedule, and architecture together. The detached variant also adds about 100M parameters. Without controlled, capacity-matched ablations, it is difficult to tell which contributions are necessary for the reported improvement.
[Q2] Are there comparisons that isolate multiple semantic tokens, early semantic completion, and gradient decoupling under the same training setup, with a capacity-matched control for JsFlow†? If so, which component accounts for the largest gain?
To reduce the influence of architectural and parameter differences, all configurations use the same two-layer semantic-tower architecture as JsFlow†. The following ablations use LightningDiT-B/1(187M parameters), under the same training setup. Single-semantic configurations use DINOv2, while multi-semantic configurations combine CLIP, DINOv1, and DINOv2.
| Method | FID ↓ | IS ↑ | sFID ↓ |
|---|---|---|---|
| Single-semantic REG-style baseline | 15.40 | 85.31 | 8.37 |
| Single-semantic + temporal scheduling | 14.57 | 89.33 | 8.32 |
| Multi-semantic | 12.10 | 91.36 | 6.35 |
| Single-semantic + gradient decoupling | 14.53 | 83.60 | 7.48 |
| Multi-semantic + temporal scheduling | 9.68 | 108.35 | 6.26 |
| Multi-semantic + gradient decoupling | 11.70 | 98.37 | 7.35 |
| Single-semantic + temporal scheduling + gradient decoupling | 13.94 | 90.65 | 7.54 |
| All components | 9.64 | 111.20 | 6.41 |
Multiple semantic tokens provide the largest individual FID improvement, reducing FID from 15.40 to 12.10. Temporal scheduling provides a further substantial improvement in the multi-semantic setting, reducing FID to 9.68. Combining all components achieves the best FID and IS.
Gradient decoupling serves a different purpose: semantic information reduces image velocity uncertainty, while gradient decoupling prevents semantic-loss interference with image-backbone optimization. It preserves semantic conditioning without allowing the semantic loss to directly update the image backbone. Together, these results clarify the complementary roles of semantic composition, temporal scheduling, and gradient decoupling.
To further examine whether the multi-semantic gain depends on encoder composition, we compare individual encoders and their subsets using LightningDiT-B/1 + JsFlow† without temporal scheduling.
| Encoder | Semantic tokens | FID ↓ | IS ↑ | sFID ↓ |
|---|---|---|---|---|
| CLIP | 1 | 15.84 | 74.50 | 7.44 |
| DINOv1 | 1 | 12.25 | 92.77 | 7.55 |
| DINOv2 | 1 | 14.53 | 83.60 | 7.48 |
| CLIP + DINOv1 | 2 | 12.12 | 94.07 | 7.55 |
| DINOv1 + DINOv2 | 2 | 12.22 | 97.30 | 7.56 |
| CLIP + DINOv2 | 2 | 14.09 | 84.68 | 7.71 |
| CLIP + DINOv1 + DINOv2 | 3 | 11.70 | 98.37 | 7.35 |
The three-encoder combination achieves the best FID and IS. Removing any encoder worsens both metrics, confirming the value of complementary semantic information. However, the gains depend on the encoder combination and the information each encoder contributes.
Clarification on VarField’s variance estimates
Thank you for the detailed response and the additional experiments. The shared-architecture ablations in Table 18 substantially address my concern about attributing the gains to individual components.
My remaining question concerns the interpretation of VarField. As acknowledged in Appendix D.2.2, Eq. (107), its estimate contains both the irreducible conditional variance and the frozen predictor’s squared error. The synthetic validation is helpful, but for the ImageNet results in Table 7, is there evidence that the predictor-error contribution is sufficiently small or comparable across encoders? Otherwise, differences in the estimates could still reflect differences in prediction accuracy rather than irreducible velocity ambiguity.
These additions improve my assessment of the paper. I would appreciate clarification on this remaining point before finalizing my rating.
Response to Reviewer tSG5 part1,1/2
We sincerely thank the reviewer for the careful assessment and constructive suggestions.
Reply to W1 and Q1
We agree that training loss alone is an imperfect proxy for irreducible velocity ambiguity. However, directly estimating velocity variance in high dimensions is challenging [1,2], making training loss a practical indirect proxy. We assess its validity through theoretical and experimental evidence.
Using VarField to estimate velocity variance
DINOv1 achieves the lowest estimated variance and the best FID, followed by DINOv2. These measurements supplement the loss curves and support the connection between reduced prediction uncertainty and improved generation.
Assessing the Magnitude of Velocity Variance via Convergence Speed
- Theorem 3 relates velocity variance to SGD convergence toward a stationary point, connecting variance analysis with optimization efficiency.
\leq \frac{2(\mathcal{L}(\theta_0)-\mathcal{L}^{\star})}{\eta T} + \frac{L_{\mathcal L}\eta}{B}(\sigma_{\mathrm{state}}^2+G^2\overline{\mathcal{V}}_Z).$$ This bound provides a theoretical connection between velocity variance and optimization efficiency. To examine this relationship experimentally, we measure the actual training steps required for image-branch MSE to reach ≤ 0.76 and compare them with the estimated variance across encoders. | Method | Actual Steps | VarField Average Variance ↓ | JsFlow† FID ↓ | |---|---:|---:|---:| | REPA | 120K | 0.7377 | 20.21 | | **DINOv1-B** | **50K** | **0.7155** | **4.31** | | DINOv2-B | 80K | 0.7282 | 5.02 | | CLIP-B | 90K | 0.7354 | 7.38 | | MAE-L | 100K | 0.7354 | 19.40 | | MoCo-v3-B | 110K | 0.7370 | 18.22 | The results across the six methods broadly show that **lower FID accompanies fewer training steps and lower estimated variance. Under comparable optimization conditions, convergence speed can serve as an approximate indicator of velocity variance.** Further details and figures are provided in the revised manuscript. [1] https://www.stochastik.math.uni-goettingen.de/preprints/B67.1%20(2005)%20019-041%20Munk.pdf [2] https://statistics.berkeley.edu/sites/default/files/tech-reports/696.pdf