- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper tackles the task of image analogies (calls it only in-context learning though for some reason), and proposes to train a Flux-based encoder and another decoder to "compress" the image analogy exemplar-pair into a "code" vector that represents the analogy, and then to condition the decoder on that vector to apply the analogy to a new query input image. This approach also allows for averaging the code vectors of multiple exemplars to refine the needed analogy.
Critical strengths
I like the idea of compressing the analogy to a vector which can be reapplied later and refined this way via averaging the code vectors. Usually in analogy editing it's just one exemplar and this approach allows for an intuitive solution to that.
I appreciate the author's efforts in analyzing the transformation space and its 'semantics', although I have some worries about it (in the weaknesses section)
Critical weaknesses
-
The paper writing is problematic in my opinion. The phrasing is a bit vague, cumbersome, and imprecise, making the reading experience very tiering. Also, many instances of not well-defined words suddenly being used to represent something that the reader should guess what the authors meant. For example, L314 – "Simliar transfer" to what? L367 – withhold what "experience"? I feel the paper needs a massive editing effort to make it presentable. Minor but related, why are there both experiments, discussion, and conclusion? The discussion is mostly repetitive of ideas present in the experiments section.
- One important example is the VAP-data. The text in Sec 4 mentioned that only VAP-data frames are used, making the reader think the targets are also from VAP-Data. But the appendix reveals that actually QWEN was used to edit the images, and it seems only the source images are from VAP. Will the source frames and the generated dataset be made public? Otherwise it's quite hard to reproduce the results later, since nothing about code release, checkpoint release, or data release was mentioned in the reproducibility statement.
-
The paper doesn't present the task clearly. It's an image analogy task (Hertzman et al. 2001), not just in-context learning (which is a new, very LLM-centric viewpoint). The paper doesn't acknowledge seminal works in this field. Related, in the related work on visual learning from examples – Image analogies are presented as in-context learning, which it is not (always), for example – Hertzman et al. 2001 which is patch-based. Also Diffusion Image Analogies from the paragraph is not an in-context learning approach, as it uses inversion. Image analogies should be discussed explicitly, and make the distinction to in-context learning based approaches. Nguyen et al (2023) is only mentioned as an afterthought even though it's a conceptual predecessor.
-
The experiments are lacking.
- Most importantly, the authors don't test generalization to held-out transformations (L744) at all. In the appendix when they train and test on Relation252K, the original authors provided specifically 10 unsees transformations just to check this. However, the authors ignore this and only test on analogy types seen during training. While it is true the comparison to RelationAdapter is harder since they don't give the train-test-split, it was nonetheless possible to compare all approaches on the unseen analogies. It is possible that the models just learned to act as a classifier to distinguish between the different types of analogies, instead of actually compression the analogy data (Since K=2 query tokens, each 3072 in dimensions, is quite a large space to overfit for analogy-type classification).
- Also very important - the main paper has no comparisons to baselines at all. The comparison in App F, as I mentioned above, is built on purpose for some reason to be not comparable with prev works. This is not a new task, and even if it was it is expected to provide some baseline comparisons.
- Next, the dataset is confusing to me, specifically the edit-groups. Why is the Naruto in fig 2 considered a styling edit but the sculpture is a subject edit? Why is the man in the city a scene edit but when the man is at sea on a dolphin it's not a scene edit? Fig.8 is also weird on this regard, why is eg giant figurine viewpoint and geometry? This this is relevant to the clustering experiment, which uses colors to distrinugish the different classes, but seems like the grouping is a bit random.
- The paper reiles heavily on paired metrics (MSE, SSIM, etc). "Qwen3.5-9B judge (QwenTeam,2026) sees the support pair, query, and prediction, but not the paired query target." (the wording isn't clear, because the word "target" wasn't used as the GT image until now, but rather interchangeability with prediction (eg L162)). I don't agree it makes sense to compare to the GT in pixel-wise metrics, since analogies can be applied in many ways, for multiple realizations.
- The user study is problematic.
- Related to the unclear writing, I could not understand what really the human study results are from Sec 6.4. What is "per-edit application"? The study should be expanded upon. I couldn't even understand from the appendix what's the meaning there.
- Next, the user study from my understanding is on a different encoder-decoder models, which was continue to be trained for 480x480 images. No automated metrics computed on those images.
- The appendix (only) mentions that the images in the user study included quite excessive filtering (L1050 incompatible edits, inapplicable or unclear demonstrations, duplicates, and invalid assets). Also some ratings include additional exclusions (L1058). This doesn't allow seeing if the method actually edits successfully in general, and instead introduces survivorship bias: WHEN the method happens to edit in a correct way, then it will also produce a good-looking result.
- L1061 seems to say the results were filtered after receiving the answers from 2-7 humans (why not a balanced questioning?). So not only this is an extremely small set, but it was filtered even further? If this is the case the user study is barely representative.
-
Table 13 shows the variance of the VLMs scores is 1.8. It's not reported in other VLM metrics, but if it's similar for all the other VLM results, then it seems the VLM-based metrics do no mean much, sadly, as all fall within the same STDs, making the claims on different Ks statistically indistinguishable.
Questions for authors
My most important questions are basically the weaknesses. Most importantly:
- the transfer of the results to held-out novel transformations, unseen during training. How would the 1-NN accuracy in code space behave in this space of transformations?
- Why are there no comparison to baselines?
Minor questions:
- Which GPU was used for training? How long did training 45K steps take
- Just out of interest - can the longer codes maybe be used for personalization? Since you say they represent the exemplar pair "too much" for analogies.
Thanks for the review. Please make an effort to comment on the authors' rebuttal.