- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper is based on a new few-shot learning paradigm, FMA, and proposes a decoder-based inference method to solve FMA's inference hyperparameter-dependent problem. With the same training velocity, the decoder-based method can achieve stable improvement across datasets and shot settings. Finally, this manuscript also analyzes the limitations of this method, i.e., the decoder recovers information that weaker upstream adapters leave unused.
Critical strengths
- This paper aims at a key challenge in flow-based few-shot learning methods, that is, the inference/transport between the text and visual modalities. The proposed method is clean and useful, showing stable improvement.
- The experimental results are rigorous, especially by providing higher reproducibility results for FMA.
- The whole analysis is systematic and reuses the FMA's trained velocity field to ablate the decoder-based inference contribution. The FMA proposes a new flow-based few-shot learning method; the training is reasonable and uses a heuristic inference setting. However, in this manuscript, the inference becomes a closed-form stable transport, making this paper and FMA contribute to a new paradigm together.
- The limitation and contribution claim are rigorous and reasonable.
- The ablation and theoretical justification are sufficient to support the main claim.
Critical weaknesses
Lacks discussion and citation of the related (following FMA) flow-matching-based adaptation works.
Questions for authors
- Can this inference method be extended to other interpolation fields, such as in spherical or hyperbolic space?
- Can this method be general enough for other transport tasks, such as retrieval or even visual generation tasks?
- Why can this kind of flow-based adaptation outperform one-step adaptation methods like CoOp or CLIP-LoRA?
4 comments
I still have one question: why this kind of flow matching based adaptation can further improve CLIP-based adaptation even with the same training data.
Thank you for the follow-up. Methods such as CoOp or CLIP-LoRA reduce the gap between image features and class text embeddings, but they do not close it: on the same training data some misalignment between the two modalities remains.
The flow head is trained on exactly this remaining gap, moving each image feature toward its class text embedding, so it adds to the upstream method rather than repeating it.
This is the motivation of the original FMA, and our analysis is consistent with it: the class scores after integration are the upstream method's own logits plus a learned correction toward the class embeddings (Corollary 4). Accordingly the gain is largest when more of the gap is left (CLIP-Adapter 62.46 to 70.51, Table 8) and smallest when the upstream method has already aligned the modalities well (CoOp, and CLIP-LoRA at 16 shots).
OK, I decided to maintain my score.
Thank you for the positive review.
Related flow-matching work. Section 2 now discusses the follow-ups to FMA: hyperbolic and radial-angular flows for few-shot adaptation (Li et al., Chen et al.), and FlowCIR, FlowComposer and GeneFlow for other tasks. They change how or where the field is trained; our analysis is about how a trained field should be read, so it applies to them as well.
Q1 (spherical or hyperbolic). Partly, and the analysis says which part. Our closed form rests on one step: given the class anchor and the current state, the source feature is determined, so it drops out of the regression target. This still holds on the sphere, because the state lies on the great circle through the image feature and its anchor, and in hyperbolic space, where it lies on the corresponding geodesic. So the optimal field is still a posterior-weighted mixture of one velocity per anchor, and the network still only has to estimate class probabilities.
What changes is the velocity toward each anchor. In the linear case it is linear in the anchor, which is what makes integration equal to "zero-shot logits plus posterior times the Gram matrix". On the sphere or in hyperbolic space it follows the logarithmic map, which is not linear in the anchor, so this decomposition, and with it the exact form of our decoder, holds only to first order in the angle between the state and the anchors. CLIP features occupy a narrow cone, so we expect the approximation to be good, but we have not tested it. This is relevant in practice because two follow-ups to FMA already train in these geometries with fixed class anchors (Li et al., Chen et al.), so their fields should have the same posterior structure. Extending the decoder to them is the natural next step, and we mention it in the conclusion.
Q2 (retrieval or generation). Retrieval against a fixed gallery fits our setting, with gallery items as anchors. Generation does not, because the targets are not a finite set.
Q3 (vs. CoOp or CLIP-LoRA). FMA is stacked on these methods rather than replacing them. It adds a learned class posterior on top of their zero-shot logits, which is why the gain shrinks when the upstream method is already strong (CoOp, CLIP-LoRA at 16 shots) and is largest at 1 shot (Table 3).