acceptodds
Under review as a conference paper at ICLR 2027

PACT: Preserving Accurate Continuous Trajectories via Flow-Map Distillation

Abstract

Large-scale Flow Matching models have set a new standard for image and video generation, but their reliance on many sequential inference steps makes them slow to deploy. While current distillation approaches like Distribution Matching Distillation (DMD) offer a way to accelerate this process, they often fail to preserve the teacher model's original data distribution, suffer from reduced visual diversity, and require highly complex training setups. To address this, we introduce PACT (Preserving Accurate Continuous Trajectories), a simple, stable, and compute-efficient framework that extends shortcut-path learning to pre-trained large-scale generators. By building an offline bank of the teacher's generation trajectories, we train a lightweight student model, using only Low-Rank Adaptation (LoRA) and new time-conditioning inputs, to predict the correct flow maps between any two visited states. This approach allows us to reuse offline data efficiently and avoids the need for auxiliary score models. We demonstrate PACT's scalability by distilling Stable Diffusion 3.5 Medium (2.5B parameters) and the LTX-2.3 video model (22B parameters) into highly efficient 4-step and 8-step generators. Our video student generates five-second audiovisual clips faster than real time, on a single GPU. Our distilled models achieve competitive visual quality while preserving the rich distributional diversity and exact seed-level behaviors of their teachers. Furthermore, PACT exhibits robust zero-shot generalization, successfully generating videos at resolutions and aspect ratios completely unseen during training.

Primary area:
generative models
Keywords:
Video Generation, Flow Matching, Distillation, Flow Map

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_XX6nOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PACT, an offline flow-map distillation method for accelerating pre-trained image and video diffusion backbones. PACT retains and reuses teacher rollout states to replace differential flow-map supervision with explicit transition regression.The student is initialized from the pre-trained backbone frozen with LoRA adapters and destination-time and guidance conditioning trained with the regression loss. This removes teacher evaluations and derivative-based supervision from student training.

Experiments cover four-step text-to-image generation with SD3.5 Medium and eight-step text-to-video and image-to-video generation with LTX-2.3. Compared with DP-DMD and official LTX distilled checkpoints, PACT generally retains more visual diversity and improves matched-seed teacher correspondence in image and text-to-video generation, with tradeoffs in quality metrics.

Critical strengths
  • Simple and coherent training procedure. Cached teacher transitions provide explicit regression targets, allowing student training without online teacher evaluation, auxiliary score networks, adversarial objectives, or Jacobian-vector products.

  • Robust Pipeline across tasks and model sizes The experiments cover both a 2.5B image model and a 22B audiovisual model, showing that the training recipe is compatible with different architectures and generation tasks.

  • Concrete inference-speed measurements and transfer results. The paper reports detailed latency breakdowns, including encoding and decoding costs, and demonstrates that the video student retains much of the teacher’s performance at spatial resolutions and aspect ratios absent from its training bank.

Critical weaknesses
  • Incomplete accounting of training efficiency. PACT simplifies student training by generating and storing teacher trajectories beforehand. However, constructing the trajectory bank requires substantial teacher computation, particularly for the 22B audiovisual model, while storing intermediate states adds storage and data-loading costs. These costs are amortized through reuse, whereas online distillation incurs teacher computation during training. Without reporting bank-generation cost, storage requirements, and total training GPU-hours under matched quality targets, the overall efficiency advantage remains unclear.

  • Limited methodological novelty. The method adapts existing offline flow-map regression to large pretrained models through trajectory caching, LoRA, and destination-time and guidance conditioning. This provides a useful practical training recipe, but the paper offers limited new theoretical insight into why this approach succeeds at scale. It also lacks a controlled comparison with JVP-based flow-map distillation using the same backbone and adaptation architecture, leaving the relative benefits of cached-transition regression unclear.

  • Weaker compositional accuracy and preference scores on T2I tasks. At four steps, PACT achieves better COCO FID than DP-DMD (20.60 versus 25.45), but lower GenEval accuracy (0.622 versus 0.649), COCO ImageReward (0.753 versus 0.988), and Pick-a-Pic ImageReward (0.921 versus 1.128).

Questions for authors
  • Have the authors compared cached-transition regression with a derivative-based flow-map distillation objective under the same SD3.5 setup? The existing teacher, cached source states, and student architecture could be reused to isolate the supervision objective. Comparing generation quality, teacher correspondence, and training cost would clarify the benefit of the proposed regression scheme over existing flow-map distillation.

  • How were the interval-sampling probabilities (0.3,0.3,0.3,0.1)(0.3, 0.3, 0.3, 0.1) chosen? Does this sampling policy improve results compared with uniformly sampling valid state pairs from the same trajectory bank?

4 comments

We thank the reviewer for their time and for recognizing the simplicity of our method and the breadth of our experiments. We address their concerns below.

1. Training efficiency and derivative-based comparison.

We appreciate the request for detailed compute and storage accounting. We report the following:

  • SD3.5: 247,260 trajectories, 5.10 TB of bank data, 158.54 GPU-hours for bank generation, and 326.85 GPU-hours for training (485.39 total).
  • LTX: 50,000 trajectories, ~2.49 TB of bank data, 1,364.48 GPU-hours for bank generation, and 3,246.93 GPU-hours for training (4,611.41 total).

We have not optimized bank size or training duration, and data-loading overhead was not separately measured, but it is highly cluster-relevant and was not a bottleneck for us. These details are now included in the appendix.

For the derivative-based comparison, we implemented a MeanFlow-style JVP objective under the same SD3.5 PACT setup, changing only the supervision objective. Despite stabilization attempts (detached targets, FP32 JVP computation, and FP32 trainable parameters/optimizer states), training diverged in all experiments. Moreover, JVP required ~14× longer per update (3.67 s vs. 0.267 s, same 16 GPUs and global batch size 32). This comparison excludes PACT's one-time bank-generation cost, reported above. While divergence prevents a quality-matched comparison, these results highlight the practical stability and training-speed advantages of PACT in this setup. We have added these findings to the paper.

2. Methodological novelty.

We agree that flow maps are not new. Our contribution is a simple, scalable offline distillation recipe demonstrated on large pretrained image and video models. Compared with the widely used DMD family, PACT avoids complex training components while better preserving teacher correspondence and visual diversity, with competitive generation quality. We believe the new JVP comparison further supports its practical value. Generalization to unseen resolutions and flexibility in inference steps and sigma schedules are additional useful properties.

3. Compositional accuracy and preference scores.

We acknowledge the lower four-step scores, as already discussed in Sec. 4.1 (lines 360–363), alongside the improvements in teacher correspondence and diversity. However, the gaps narrow considerably with just two additional steps:

  • GenEval: 0.622 → 0.652 (DP-DMD: 0.649).
  • COCO ImageReward: 0.753 → 0.882 (DP-DMD: 0.988).
  • Pick-a-Pic ImageReward: 0.921 → 1.030 (DP-DMD: 1.128).

While this requires six rather than four inference steps, it shows that quality improves substantially with a modest increase in step count, if a usecase requires higher level of quality.

4. Interval-sampling probabilities.

The probabilities (0.3, 0.3, 0.3, 0.1) were chosen to balance jump lengths and emphasize longer transitions needed for few-step inference. Following the reviewer's suggestion, we trained an SD3.5 student with uniform sampling under identical settings. At four inference steps, both models achieve nearly identical results: GenEval 0.622 vs. 0.620, COCO FID 20.60 vs. 20.66, and COCO ImageReward 0.753 vs. 0.749 (PACT vs. uniform). Diversity and teacher correspondence are also nearly unchanged. Predicted-velocity MSE on unseen trajectories is similarly close across all jump buckets, with only a small advantage for our original policy in the third bucket (0.025 vs. 0.027). These results suggest that PACT is robust to the sampling distribution, with no substantial benefit from our weighted policy in this setting. We have added the ablation to the appendix.

Thank you for the detailed response and additional experiments. The compute accounting and uniform-sampling ablation largely address my questions, and the JVP comparison provides useful practical evidence within the tested setup. The six-step results also help clarify the trade-off between generation quality and inference cost. I have no further major concerns regarding these points.

My assessment of the methodological novelty remains unchanged, and I will maintain my score.

Thank you very much for taking the time to review our rebuttal and additional experiments, and for following up during the discussion period. We are glad to hear that the compute accounting, uniform-sampling ablation, JVP comparison, and six-step results addressed your empirical questions and clarified the quality–inference cost trade-off.

We really appreciate your thoughtful review and constructive feedback throughout the review process.

Best regards, The Authors

Thank the authors for the response, my concerns on performance side are resolved. However, my overall concern on the novelty remains, I will therefore keep the score

Reviewer_gGCROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a simple and effective methods to reduce the inference steps of diffusion models by collection teacher trajectories and save them for offline student learning. Extensive experiments on various baseline models demonstrate the effectiveness of the proposed method.

Critical strengths
  1. the paper is well written and well motivated
  2. the proposed method is simple and effective
  3. the authors try to justify that the generation quality preserves while inference time has significant reduced.
Critical weaknesses
  1. it's better to measure the results across more baseline methods since the proposed method could be a general technique for many flow matching models.
  2. the results are not well-presented, the whole main paper mainly uses figure to show the results which makes qualitatively comparison hard.
  3. it lacks the comparison with other popular distillation methods other than DMD
Questions for authors

please see weakness, can the author also show some results with even fewer steps such as 4?

3 comments

We thank the reviewer for acknowledging the clarity of our paper and the simplicity and effectiveness of our method. We address each concern below.

1. Generalization to additional baselines

We agree that our method is general and can be applied to different flow-matching models, which is why we initially evaluated it on two large-scale image and video generation baselines. To further address the reviewer's request, we have also distilled a newer video generation model, MiniMax H3, with 33B parameters.

The VBench and VBench-I2V results below show that PACT closely matches the strong teacher across all reported metrics. Compared with the DMD-based Turbo model, PACT achieves better semantic alignment and conditioning fidelity (by 0.71 percentage points on average), while having only marginally lower quality scores (by 0.29 percentage points on average).

BenchmarkMetric (%)Teacher (40 steps)PACT (4 steps)Turbo (4 steps)
VBench (T2V)Total85.03584.82284.851
VBench (T2V)Quality85.99885.42385.713
VBench (T2V)Semantic81.18182.42181.405
VBench-I2VTotal89.54789.39189.328
VBench-I2VQuality81.37481.06981.355
VBench-I2VConditioning97.71997.71497.301

These results have been added to the paper.

2. Numerical presentation of results

We use bar charts in the main paper to highlight relative performance across different methods and metrics. However, we agree that numerical tables are useful for more precise comparisons.

We have already provided detailed numerical results in the appendix: image generation results in Appendix A.3.3 (Tables 1–10), video generation results in Appendix A.3.4 (Tables 11–24), latency measurements in Appendices A.4–A.5 (Tables 25–27), and published external baseline results in Appendix A.6 (Tables 28–30).

3. Comparison with other distillation methods

DMD-based distillation frameworks are widely used in both research and industry, particularly for large-scale image and video generation models, which is why we primarily compare PACT with these strong baselines.

Nevertheless, to address the reviewer's concern, we have also implemented, trained, and evaluated PCM, a consistency-based distillation method with adversarial loss, on Stable Diffusion 3.5. We summarize the key results below, with the complete numerical results provided in the appendix.

BenchmarkMetricTeacher (28 steps)PACT (4 steps)DP-DMD (4 steps)PCM (4 steps)
GenEvalOverall ↑0.6730.6220.6490.586
GenEvalImageReward ↑0.9210.6660.9680.447
COCOFID10K ↓20.5220.6025.4520.24
COCOCLIPScore ↑26.7126.3826.5425.93
COCOPickScore ↑22.4121.9322.3121.97
Pick-a-PicCLIPScore ↑29.1928.5428.4227.68
Pick-a-PicDINO diversity ↑0.2500.2510.2120.273
GenEval teacher similarityCLIP global ↑1.0000.8130.7990.750
COCOFID-teacher ↓0.008.588.8412.21

As shown above, PACT generally outperforms PCM in prompt adherence and teacher similarity, while PCM achieves higher diversity and slightly better results on some metrics, such as COCO FID and PickScore. These results further support the effectiveness of PACT compared with a different class of distillation methods.

4. Results with fewer inference steps (4 steps)

Our Stable Diffusion 3.5 results already use 4-step generation, and the additional MiniMax H3 results reported above also use 4 steps.

For LTX 2.3, reducing the number of steps to 4 degrades generation quality, which we suspect may be related to the high compression ratio of its VAE latent space. The official distilled LTX models also uses 8 steps. Both our model and the official distilled model already achieve faster-than-real-time generation, producing a 5-second video in approximately 4.4–4.5 seconds.

Therefore, we chose to retain 8 steps for LTX 2.3 to preserve generation quality while still achieving faster-than-real-time inference.

Thank you again for your time and constructive feedback on our submission.

As the author-reviewer discussion period finishes today, we wanted to gently follow up to ensure our rebuttal and additional results adequately addressed your concerns, particularly regarding teacher trajectory preservation and composition consistency, where the new MiniMax H3 results that show our method could be applied to even larger flow matching models, and the PCM baseline comparison where PACT is substantially better in prompt adherence and teacher similarity.

We would greatly appreciate any further thoughts or remaining questions you might have before the discussion window closes, and we would be happy to provide any quick clarifications.

Best regards, The Authors

Reviewer_gGCROpenReview ↗

thanks for the response

thank the authors for the response, my concerns are solved.

Reviewer_bAR4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PACT, an offline flow-map distillation framework that trains a LoRA-adapted student to predict transitions between states along teacher-generated trajectories. It accelerates image and video generation while aiming to preserve the teacher's sample diversity and seed-level correspondence. Experiments on SD3.5-M and LTX-2.3 demonstrate substantial inference speedups with competitive generation quality.

Critical strengths

1. Simple and scalable distillation framework: PACT uses cached teacher trajectories and direct regression on state transitions, avoiding auxiliary score networks, online teacher evaluations, and complex adversarial objectives. This makes the training procedure conceptually simple and reusable.

2. Strong practical efficiency: The method successfully distills large pretrained image and video models using LoRA adaptation. In particular, achieving approximately 12× end-to-end acceleration for a 22B-parameter audiovisual model is a meaningful practical contribution.

S3. Encouraging preservation of teacher behavior. The results demonstrate improved seed-level teacher correspondence and diversity preservation compared with several distilled baselines. The ability to vary inference steps and generalize to unseen video resolutions further strengthens the practical value of the method.

Critical weaknesses

1. Lack of guarantees for trajectory and distribution preservation: The method minimizes transition errors on cached teacher trajectories, but this does not guarantee accurate recursive sampling. Student errors can accumulate and shift states outside the training distribution. Moreover, the objective does not enforce flow-map composition consistency, *F_{t\rightarrow r}=F_{s\rightarrow r}\circ F_{t\rightarrow s}$. Thus, the central claim of preserving the teacher's trajectories and output distribution remains insufficiently justified.

2. Limited methodological novelty over existing flow-map distillation: The core objective directly regresses average velocities between cached teacher states, closely resembling existing offline Flow Map Matching approaches. The main additions—LoRA adaptation, destination-time conditioning, and guidance conditioning—appear largely implementation-oriented. The paper does not clearly establish a fundamentally new learning principle or demonstrate its advantage over closely related flow-map methods.

3. Diversity metrics do not establish distributional fidelity: Matching pairwise CLIP/DINO feature diversity does not imply preservation of the teacher's generation distribution. PACT even exceeds teacher diversity on some metrics, while underperforming DP-DMD in four-step GenEval and ImageReward. Therefore, the evidence supports improved feature diversity and teacher correspondence, but not the stronger claim of faithful distribution preservation.

Questions for authors

See Wekanesses.

2 comments

We thank the reviewer for recognizing the simplicity and practicality of our method and for the constructive feedback. We address the three concerns below.

1. Trajectory and distribution preservation.

Our preservation claim assumes perfect student learning, which implies accurate recursive sampling and composition consistency along teacher trajectories. We agree that finite training provides no such guarantees, and student errors can accumulate. We have clarified this distinction in the paper.

To empirically assess consistency, we evaluated 256 teacher trajectories (2,816 time triples). The composition discrepancy, normalized by teacher displacement, averages 9.41%, 5.77%, and 4.10% for 2-, 4-, and 8-partition schedules, respectively. On the full noise-to-data interval, PACT's two-hop endpoint error is 29.13%, compared with 48.21% for two-step teacher Euler integration. These results support approximate composition consistency and trajectory reconstruction, despite the absence of formal guarantees. Our same-seed teacher similarity results (Figs. 4g–h, 6c) provide further evidence.

2. Methodological novelty.

We agree that flow maps and offline regression are not new, as discussed in our related work. Our contribution is a simple, stable, and scalable distillation framework for large pretrained image and video models.

PACT directly regresses cached teacher transitions, requiring no online teacher evaluations, Jacobian–vector products, or auxiliary score models during training. The trajectory bank enables efficient reuse of teacher computation, while LoRA and lightweight conditioning support flexible inference schedules and guidance strengths. We demonstrate this on models up to 33B parameters, achieving ~12× acceleration while retaining strong teacher correspondence and diversity.

In response to reviewer XX6n, we also tested a MeanFlow-style JVP-based target, but training consistently diverged in our setting despite multiple attempts, highlighting the practical challenges of stable large-scale distillation.

3. Evidence for distribution preservation.

We agree that diversity metrics alone cannot establish distribution preservation. Beyond same-seed CLIP/DINO similarity (Figs. 4g–h, 6c), we report COCO FID, where PACT closely matches the teacher (20.60 vs. 20.52), compared with DP-DMD (25.45).

To directly assess teacher–student distribution matching, we additionally compute Teacher-FID (T-FID), using teacher-generated images as the reference:

Model (4 steps)T-FID (COCO-10K) ↓
PACT8.58
DP-DMD8.84
PCM12.21

PACT achieves the lowest T-FID, providing further evidence of closer teacher distribution matching in Inception feature space, although the improvement over DP-DMD is modest.

We acknowledge DP-DMD's stronger four-step GenEval and ImageReward results. However, PACT emphasizes a complementary trade-off favoring diversity and teacher correspondence rather than superiority across all quality metrics.

We have added the new experiments and clarified our preservation claims in the revised paper.

Thank you again for your time and constructive feedback on our submission.

As the author-reviewer discussion period finishes today, we wanted to gently follow up to ensure our rebuttal and additional results adequately addressed your concerns, particularly regarding teacher trajectory preservation and composition consistency, where the new evaluations show we are more accurate than teacher euler integration and the composition consistency is fairly high, and also regarding the new Teacher-FID evaluation showing we better match teacher distribution.

We would greatly appreciate any further thoughts or remaining questions you might have before the discussion window closes, and we would be happy to provide any quick clarifications.

Best regards, The Authors

Comments on the paper

1 comment
Area_Chair_QqWbOpenReview ↗

Please read the response from authors and discuss

Dear reviewer fellows,

The author rebuttal has been posted. The rebuttal period is a critical part of our review process, allowing authors to clarify misunderstandings and address specific concerns.

Please take a moment to read the author response, engage in the discussion by replying to the authors' responses, and update your review if the rebuttal changes your assessment.

Your active participation is vital to ensuring a fair and thorough evaluation of the submissions. A constructive discussion helps us make the most informed decisions.

Thank you for your time and valuable contributions to the review process.

Best regards,

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.