- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors proposed RefRoute, an efficient framework for multi-reference image generation that reduces inference latency and peak memory usage as the number of visual references increases, while achieving competitive Weighted-Ref-VIEScore results. Specifically, the RefRoute is composed of the compact residual conditioning, condition routing and attention routing. Compact residual conditioning helps reduce the visual tokens input to DiT while maintaining high-frequency details of references. Condition routing uses separate reference and spatial LoRA adapters for condition-specific adaptation while attention routing helps explicit reference-region correspondence. Moreover, the authors devised the RefRoute-Data and ManyRef100 for training and evaluation for the multi-reference image generation task.
Critical strengths
- The innovation of the paper are concise and straightforward, and the experimental results claimed in the paper are promising and seem to be sound. Extracting tokens from downsampled reference images and compensating for the lost high-frequency details with a pre-trained residual prediction module provides a compression strategy that explicitly considers reconstruction quality. Meanwhile, the attention routing reduces unnecessary computation costs.
- The figures in the paper are easy to understand and clear. Figure 5 convictively presents the effectiveness of dynamic top-1 routing compared to strict spatial masking from qualitative comparison perspective.
Critical weaknesses
- The writing of experiments is kind of disordered. Line 369 and 370 claim that the authors examined both inference cost and generation quality as the number of references increases. However, in the main paper, I can not find the examination of generation quality as the number of references increases. If I misunderstand, I hope the author will correct me during the rebuttal.
- The experiments in the main paper are not that sufficient. The authors only compared the proposed method with general multimodal generation models that not specifically devised for multi-reference generation in Table 1 and 2 without comparing task-specific methods like StructGen or MultiRef mentioned in RelatedWorks. In the ablation study, Table 4 only confirms the effectiveness of stage 1 pretraining. However, I would like to see the impact of attention routing trained in stage 2 as well, such as its impact on the evaluation metrics, computation cost or inference latency.
Questions for authors
- Which part of the main paper is the mentioned examination of generation quality as the number of references increases?
- The reason of not comparing with the methods like StructGen or MultiRef mentioned in RelatedWorks.
- How to choose the top-k ranked reference. What is the choosing principle.
- Is corresponding target-region of reference not accessible during inference?
- Will the access of the top-k reference for the outside mask region increases the probability of generating the unnatural associations of references with the background, such as the man standing on the blue box with one foot in Figure 5?