- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces MoGE-Bench, a benchmark for evaluating on-device image generation and editing models. It aims to develop fine-grained evaluation rubrics for priority target categories identified through mobile application needs. For deployability, the paper considers inference latency, peak memory usage, and model size, and combines quality and deployability into a joint assessment using predefined tiers. The proposed framework enables researchers to identify capability strengths and weaknesses of compact models and assess their suitability for mobile applications.
Critical strengths
-
Human-centered benchmark design. The use of human input to identify priority target categories for mobile applications is valuable for on-device model development. Given the resource constraints of mobile devices, understanding which capabilities matter most can help researchers prioritize model development and allocate limited resources to the most important tasks.
-
Fine-grained capability evaluation. The proposed Rubric Cube provides a structured evaluation framework for assessing model performance across different operations, target categories, and quality criteria. This enables a more detailed analysis of the strengths and weaknesses of compact models than a single overall score, potentially helping guide future model improvements.
Critical weaknesses
-
Limited deployment evaluation for image editing. The deployment experiments currently focus on text-to-image generation, leaving the deployment characteristics of image editing tasks largely unexplored. The computational costs of editing may differ substantially from those of text-to-image generation due to additional conditioning inputs and task-specific inference pipelines. Moreover, different models may employ different conditioning mechanisms, leading to substantially different latency and memory scaling behaviors. Consequently, the current deployment results may not adequately reflect the practical feasibility of deploying these models for the editing scenarios targeted by the benchmark.
-
Coarse-grained deployability tiers provide limited practical guidance. The threshold-based deployability assessment may be too coarse to support practical model selection. In real-world applications, deployment requirements are often highly specific to the target device and application, with different constraints on latency, memory, and model size. Models assigned to the same tier may still exhibit substantially different resource requirements, while models in different tiers may both be suitable for a particular application. Therefore, although the Quality-Deployability Map provides an intuitive visualization of the proposed framework, its coarse categorization offers limited actionable guidance for researchers and engineers selecting models under application-specific constraints.
-
Ambiguity in model size accounting The model size metric may not provide a fair comparison across models if the parameters of the text encoder are included without distinguishing their contribution from that of the generative model. Text encoders can differ substantially in architecture and parameter count, potentially causing the reported model sizes to reflect differences in text encoding components rather than the efficiency of the generation models themselves.
Questions for authors
Is it informative to quantify the latency for different models while their inference step are different?