- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposed a new flow-matching model steering method. With only 20 images from the student and teacher, it can guide the student model to generate images it would not be able to generate using the delta between the student's images and the teacher's images in the student's latent space.
Critical strengths
- The proposed method is effective, and the solution is interesting, especially when combined with a negative guidance method like NAG.
- Including a diversity metric is a great choice. One concern I had for methods for strong-to-poor distillation/steering/improvement is that they tend to generate homogeneous output. The paper correctly addressed this concern.
- The judge is correctly validated against humans.
Critical weaknesses
- (minor) The headline "no call to the teacher during inference" is slightly misleading. It is correct that during denoising of the student, we do not need to call the teacher. However, we still need to collect the teacher's output per prompt. This means calling the teacher every time we use a new prompt. This also raised a question about the use cases of this method: if I need to generate a large batch of images X, it might be worth collecting the teacher's output. Otherwise, given that capability transfer is not universal, the use case is limited.
- (minor) Table 1 bolds every DFS row, which reads as marking the best results, but LoRA outperforms DFS alone in every column. If bold is meant to highlight the proposed method rather than the best result, the caption should say so.
Questions for authors
Could the authors explain the use case of this method better?
3 comments
Not a weakness and the author does not need to add anything to this but just some other thing I thought about (I posted it earlier but forget to select author as reader): Seems like the teacher images do not need to be from a bigger model, so my concern about the cost and use case is somewhat resolved: real images or hand crafted/rendered images can be used as "teacher". If this is true, the paper would be very interesting.
Yes, your reading is right. The teacher images do not have to come from another model. DFS uses only the images, so they could also be rendered, made by hand or edited, as long as they show the capability. In principle, DFS can then pass that capability to the student in the same way.
Response to Reviewer wxdG: teacher calls, use cases, and Table 1
W1. "No call to the teacher during inference".
Thank you. The phrase means that the teacher is never called while the student samples, and the camera-ready version will say it in these words. A direction needs one set of teacher images, collected once. The paper uses 20, and ten pass as often on a 30-prompt subset (46.7% against 45.0%, Table 11).
A new prompt does not always need a new set. New experiments for this rebuttal show that a direction can be reused on prompts that keep the same structure (Appendix G). Applied unchanged to 73 new prompts, 27 directions raise the pass rate from 17.6% to 45.0%, with no teacher image of the new prompts. A prompt that needs a different structure needs its own set.
Q1 and W1. The use case.
DFS is for people who can reach a stronger, closed image model but cannot or do not want to route every image through it. They buy one small set of images, 10 to 20, that show a structure their open model cannot draw. After that the open model draws this structure itself, as often as needed and for new content, with no further call. Some examples of who this serves:
A marketing team with one layout and many documents. A restaurant chain or a retailer has a menu or flyer layout that works, made with a closed model. Every week it needs the same layout with new dishes, prices and branches. With DFS the team keeps the layout as one stored direction and produces each new version on its own machines.
A designer with a template that is easier to show than to describe. Examples are a poster with coloured bands around any word, or one subject shown in four seasons in a 2×2 grid. The open model receives the full description and still gets it wrong. A few example images carry the template to new words and subjects (Figure 14), and trying variations costs no call. The examples can come from a closed model. In principle a designer could also make or correct them by hand, which we have not tested.
A company whose content must stay in-house. A pharmaceutical firm, a bank or a team preparing a product launch cannot send unreleased names or internal documents to an outside service. It can fit the direction on a harmless prompt with the same structure and then generate the confidential version locally.
A team that depends on a closed model it does not control. Closed models are updated, repriced, rate-limited and retired. A studio or publisher that built a workflow on one can store directions for the structures it relies on, and keep producing them on an open model when the service changes.
A group that needs images in bulk. A textbook publisher needs hundreds of diagrams in one format. A research lab builds a synthetic dataset of documents with a fixed layout. Both need volume and variety, and the open model supplies both.
These cases rest on two results of the paper: a direction keeps working on new subjects and reworded prompts (Appendix G), and the images keep the student's seed diversity (Table 12). Our experiments use open teachers as stand-ins for the closed model. A direction carries one structure that sits at the same place in every example, such as columns, bands, grid cells or panels. It does not carry textures or drawing styles. Not every image succeeds, so a user generates a few and keeps the good ones. The camera-ready version will state these use cases in the introduction.
W2. Bold rows in Table 1.
Thank you for catching this. Bold in Table 1 marks our method and not the best value. In the camera-ready version, bold marks the best value in each column, and our rows are marked by shading with a note in the caption. The numbers do not change. The per-prompt LoRA passes more images than DFS alone, as Section 5 states. LoRA trains an adapter for every prompt, and DFS needs no training. With NAG, DFS is level with that LoRA (60.7% against 58.1% on fresh seeds, Table 23).