- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes an evaluation pipeline that accelerates diagnostic evaluation for text-to-image generation. It organizes evaluation conditions into a graph structure and utilizes the estimated success probabilities of parent nodes as priors for child nodes. It then performs early stopping when sufficient evidence is gathered. Compared with full-budget evaluation, the method accelerates the evaluation process while maintaining high agreement with the full-budget evaluation results.
Critical strengths
The idea of structuring evaluation conditions into a graph and utilizing parent information to provide priors for more fine-grained conditions intuitively makes sense. I see the paper as a well-executed study of this evaluation acceleration pipeline, which may have practical benefits in making T2I evaluation more efficient. The paper also presents the idea clearly.
Critical weaknesses
- The main novelty of the method is the hierarchical testing, but in the ablation study, Flat Sequential achieves a very similar speedup and reasonably close Jaccard agreement compared to the hierarchical version. It remains unclear how critical the hierarchical design is in practice. Besides, the sample gate seems to be a relatively simple design choice. Overall, I am not fully convinced that the methodological novelty and contributions are strong enough.
- The method involves several hyperparameters, such as the parent weight, prior strength, and the sample gate. Although the authors provide some hyperparameter studies in the appendix, the main hyperparameter search is conducted with one model on one dataset. It is still unclear how sensitive the method is to these hyperparameters or whether the same settings generalize across different evaluation scenarios.
Questions for authors
Please refer to the critical weaknesses section.