acceptodds
Under review as a conference paper at ICLR 2027

MoGE-Bench: A Benchmark for On-device Diffusion Models

Abstract

Developing on-device image models requires considering practical user needs, task types and device resource constraints. However, existing benchmarks provide limited guidance on identifying capability gaps between on-device models and server-side models. We therefore introduce MoGE-Bench, a benchmark grounded in mobile user needs, comprising 64 subtasks and 1150 samples. It defines 16 operations, such as removal and replacement, and eight visual target categories covering content such as portraits and text. These form two orthogonal axes, with applicable combinations yielding explicit task coverage. To better guide model improvement, we propose MoGE-Eval, which introduces task-specific criteria covering instruction following, visual quality and detail preservation. It yields overall quality scores and capability profiles along the operation, target and criteria axes. We evaluate and deploy on-device models, and compare their output quality with server-side models. Our results show that this decoupled Operation × Target design provides finer-grained guidance for improving compact models. For example, it highlights the need to prioritize specific capability gaps (e.g., text-replacement) over broad application-level goals (e.g., improving text editing). We further construct a Quality-Deployability capability map using the deployment results. The map shows that current on-device models can achieve efficient generation or editing, but this deployment efficiency is not consistently accompanied by broad editing capability. Further progress requires improving editing quality and expanding task coverage within resource budgets, or exploring task-aware on-device–cloud routing. We will release the benchmark annotations, inference code and evaluation toolkit.

Primary area:
datasets and benchmarks
Keywords:
diffusion models, image generation, image editing, on-device AI, benchmarking, model evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Ao12OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces MoGE-Bench, a benchmark for evaluating on-device image generation and editing models. It aims to develop fine-grained evaluation rubrics for priority target categories identified through mobile application needs. For deployability, the paper considers inference latency, peak memory usage, and model size, and combines quality and deployability into a joint assessment using predefined tiers. The proposed framework enables researchers to identify capability strengths and weaknesses of compact models and assess their suitability for mobile applications.

Critical strengths
  1. Human-centered benchmark design. The use of human input to identify priority target categories for mobile applications is valuable for on-device model development. Given the resource constraints of mobile devices, understanding which capabilities matter most can help researchers prioritize model development and allocate limited resources to the most important tasks.

  2. Fine-grained capability evaluation. The proposed Rubric Cube provides a structured evaluation framework for assessing model performance across different operations, target categories, and quality criteria. This enables a more detailed analysis of the strengths and weaknesses of compact models than a single overall score, potentially helping guide future model improvements.

Critical weaknesses
  1. Limited deployment evaluation for image editing. The deployment experiments currently focus on text-to-image generation, leaving the deployment characteristics of image editing tasks largely unexplored. The computational costs of editing may differ substantially from those of text-to-image generation due to additional conditioning inputs and task-specific inference pipelines. Moreover, different models may employ different conditioning mechanisms, leading to substantially different latency and memory scaling behaviors. Consequently, the current deployment results may not adequately reflect the practical feasibility of deploying these models for the editing scenarios targeted by the benchmark.

  2. Coarse-grained deployability tiers provide limited practical guidance. The threshold-based deployability assessment may be too coarse to support practical model selection. In real-world applications, deployment requirements are often highly specific to the target device and application, with different constraints on latency, memory, and model size. Models assigned to the same tier may still exhibit substantially different resource requirements, while models in different tiers may both be suitable for a particular application. Therefore, although the Quality-Deployability Map provides an intuitive visualization of the proposed framework, its coarse categorization offers limited actionable guidance for researchers and engineers selecting models under application-specific constraints.

  3. Ambiguity in model size accounting The model size metric may not provide a fair comparison across models if the parameters of the text encoder are included without distinguishing their contribution from that of the generative model. Text encoders can differ substantially in architecture and parameter count, potentially causing the reported model sizes to reflect differences in text encoding components rather than the efficiency of the generation models themselves.

Questions for authors

Is it informative to quantify the latency for different models while their inference step are different?

Reviewer_5Ph6OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper propose MoGE-Bench, it evaluates 19 image generation and editing models across 64 subtasks and 1,150 samples. It assesses output quality and measures the deployment costs of selected models on mobile devices. The results show that some lightweight models run efficiently on phones but still struggle with diverse editing tasks. The benchmark helps identify capability gaps and guides the development of on-device image models.

Critical strengths
  • The benchmark covers 64 tasks spanning both image generation and editing, and its breakdown of evaluation scores helps identify specific capability gaps. Measuring performance on actual mobile devices is also more informative than reporting parameter counts alone, giving the benchmark practical value.

  • The automated evaluation is supported by blind human assessments and comparisons across different judges. The two human raters achieved a correlation of 0.904, while the two automated judges showed correlations of 0.779 and 0.852 with the human reference scores (Appendix D.2). The paper also includes a sensitivity analysis of the scoring weights.

Critical weaknesses
  • The paper evaluates reference quality using GPU-based inference and measures deployment costs separately on mobile devices. Mobile tests cover only one text-to-image prompt, leaving the benchmark’s predominantly editing-oriented tasks untested on-device. Moreover, the paper does not repeat the full quality evaluation for each mobile precision configuration or establish quality retention under quantization (Table 3, Appendix E.1 and E.3). Thus, the Quality–Deployability map does not directly establish the editing quality and resource costs achievable under actual mobile deployment settings.
  • Some subtasks contain only 4–5 samples, yet sample-level uncertainty intervals are not reported. Although the paper provides judge-agreement and sensitivity analyses, six of the 19 models drop one quality tier when switching from GPT-4o to Gemini 3.1 Pro.
  • Confirmed generation and judge-execution errors are excluded from aggregation, but per-model failure counts and retained sample sizes are not reported, this make the statistical uncertainty and robustness of the tier assignments difficult to assess.
Questions for authors
  • Could the author evaluate editing quality, end-to-end latency, and peak memory for at least one editing-capable model under its actual mobile deployment configuration, including the deployed precision and sampling settings?
  • How many generation and judge-execution failures were excluded for each model, and how many valid samples remained? Could the author report sample-level bootstrap confidence intervals and assess whether the quality tiers remain stable under resampling?
Reviewer_s5wTOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a benchmark for mobile-first diffusion models. The goal is to have a benchmark that takes device capability into account as well as matrix of tasks which was determined based on asking 1k humans about their preferred on-device image workflows. The benchmark therefore aims to reflect what "users" might care about for on-device image pipelines. The authors create this benchmark and then evaluate a host of models.

Score raised 2->3 post rebuttal

Critical strengths

The authors present a significant effort in creating a novel benchmark for mobile-first diffusion models. I appreciate that the benchmark was motivated by end-user's preferences about what they might care about. The benchmark is comprehensive and potentially provides actionably diagnostics for model developers who plan to release a mobile-first image model. I also appreciate the engineering effort in getting several models benchmark on-device. This benchmark might have long-term value and could be iterated-upon as end-user preferences and model capabilities evolve.

Critical weaknesses

The quality scores are not computed with actual on-device configurations. The authors evaluate models on A100 without fully taking potentially quality-affecting deployment configurations such as quantization into account. I would expect evaluation on actually on-device generated images or a faithful (as much as possible) replication of the on-device configuration on the A100 deployment.

The authors put diffusion models front-and-center in their write-up, however include Janus-pro, which is an auto-regressive model. I believe this benchmark should be agnostic to this choice and the paper might clean up its formulation. More importantly though, on-device deployment configurations use T2I as a stand-in for editing tasks. Depending on model architectures, that stand-in might be more or less suitable.

Novelty. There's a lot of benchmarks out there, as the authors acknowledge. The authors claim that their benchmark and taxonomy provides more actionable insights, but this distinction is not mobile-specific. What is mobile-specific is the connection between quality and hardware measurements and constraints. The usefulness of a new benchmark arguably lies in whether or not it leads to new insights, in this case: different model-choices for a specific hardware configuration and constraints. How does MoGE-Bench lead to different (better) choices compared to other benchmarks given benchmark-independent hardware-measurements and constraints.

While the authors show 1150 samples across 64 subtasks, each subtask gets relatively few samples. Some combinations' numbers are based on small number of data-points.

Questions for authors

Based on the weaknesses, I would suggest / ask from the authors: Show alignment between A100 deployment and on-device deployment: If too expensive for the full set, then at least on a representative subset.

Show on-device editing workflows and show that T2I is a valid stand-in for the different model-architectures.

Show how MoGE-Bench leads to better model-choices compared to other benchmarks to clarify useful novelty.

Can the authors provide confidence intervals, bootstrap uncertainties or other measures of statistical significance for their quality metrics?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.