- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes MobileDev-Bench which is a benchmark of 407 issue-resolution tasks from 19 repositories (including apps in Android Native, Flutter, and React Native). Tasks were collected similar to SWE-bench (from real github issues and developer fixes). The results show low fix rates, particularly for larger fixes and those requiring changes to artifacts (e.g. manifest).
Critical strengths
-
Issues in mobile apps is an important and difficult problem that is less represented in existing benchmarks.
-
The build validation and human review are useful aspects for benchmarks.
Critical weaknesses
-
The main concern is understanding how much the build process affects evaluation. The authors verify the base and developer-fixed builds, which is good. However, Android dependencies can change, and compiling (Gradle builds) can take a considerable amount time. It would help to report build consistency across reruns, build/test times, and timeout rates.
-
I also think the choice of harness needs stronger validation. The improvement from using mini-SWE-agent already shows that agent harness/scaffolding is important. Testing other harnesses (particularly newer ones) such as OpenHands, or adding Gradle-specific tools under the same budget be useful in determining the actual difficulty of the tasks included. The related work should also discuss a very closely related work AndroidBuildBench and GradleFixer [1], which study build repair and show improvements from domain-specific tools, particularly since the authors do note that "Build/compile failure is the dominant failure mode for every model."
[1] Automating Android Build Repair: Bridging the Reasoning-Execution Gap in LLM Agents with Domain-Specific Tools. EACL 2026.
Questions for authors
-
How reproducible are builds across reruns, and what are the build/test times and timeout rates?
-
How do results change with another harness or Gradle-specific tools under the same budget?
Response to Reviewer F6Dg
We appreciate Reviewer F6Dg's feedback on build reproducibility and harness choice.
Q1. Build reproducibility, times, and timeouts
All 407 gold patches pass offline in the released images, which contain the repository-specific toolchains and cached dependencies. Across the four model runs, we completed evaluation of all 1,409 submitted patches using these images.
The table accounts for all 1,628 task–model pairs (407 tasks × 4 models). Evaluation wall times are in minutes.
Times include patch application, build/test execution, and cleanup, excluding image download. The logs do not separate compilation from test execution.
During generation, 180 of the 1,626 runs with logs (11.1%) hit at least one 15-minute command timeout.
We will add these timing and timeout measurements to the experimental setup.
Q2. Other harnesses and Gradle-specific tools
Thank you for pointing us to AndroidBuildBench and GradleFixer. Their objective is to repair failing Android builds. In MobileDev-Bench, we prepare and remediate build environments during benchmark construction (Section 3.3 and Appendix A.5). We retain only tasks whose base and developer-fixed configurations build and execute tests successfully (Section 3.4), excluding candidates with unresolved build failures or nondeterministic test outcomes (Appendix A.4). The agent's task is to implement the issue's requested change in this prepared environment and pass the required tests. We will add the distinction between this paper and our work to Section 2.
We have not run an OpenHands or Gradle-tool comparison under a matched budget. We prioritized making the required interfaces explicit in N2P task inputs, as detailed in our response to Reviewer YTKL (Q1). Our audit found missing or incompatible interfaces in 173 of the 232 GPT-5.2 runs in which no tests executed.
With mini-SWE-agent, GPT-5.2 resolved 88/260 N2P-only tasks (33.8%) in the interface-specification setting, compared with 9/260 (3.5%) in the original issue-only setting. We will release the specifications as task inputs and report this setting separately.