acceptodds
Under review as a conference paper at ICLR 2027

MobileDev-Bench: A Benchmark for Issue Resolution in Mobile Application Development

Abstract

Large language models (LLMs) achieve strong performance on repository-level issue resolution, yet existing benchmarks focus on library-style repositories and rarely capture mobile applications, where fixes must build under framework-specific toolchains and often coordinate changes across source, resource, and build files. This paper introduces MobileDev-Bench, a benchmark of 407 verified issue-resolution tasks from 19 production Android Native (Java/Kotlin), React Native (TypeScript), and Flutter (Dart) applications, each paired with developer-written tests and a containerized build environment. Fixes modify 12.9 files and 334.6 lines on average, with 41% spanning multiple artifact types. MobileDev-Bench distinguishes tests that fail before the fix from those that become compilable only after the fix introduces required symbols. Under the single-pass Agentless pipeline, four LLMs (Claude Sonnet 4.5, GPT-5.2, Gemini 2.5 Flash, and Qwen3-Coder) resolve only 3.19%–4.18% of tasks with automated retrieval and at most 5.65% with oracle retrieval. With the interactive mini-SWE-agent, the best-performing model, GPT-5.2, resolves 12.53% of tasks, compared with 69.0% for the same model and agent on the SWE-bench Verified leaderboard. We release the benchmark tasks, evaluation harness, and containerized environments.

Primary area:
datasets and benchmarks
Keywords:
Language models, Natural language Processing, Software engineering, Mobile App development

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_F6DgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MobileDev-Bench which is a benchmark of 407 issue-resolution tasks from 19 repositories (including apps in Android Native, Flutter, and React Native). Tasks were collected similar to SWE-bench (from real github issues and developer fixes). The results show low fix rates, particularly for larger fixes and those requiring changes to artifacts (e.g. manifest).

Critical strengths
  • Issues in mobile apps is an important and difficult problem that is less represented in existing benchmarks.

  • The build validation and human review are useful aspects for benchmarks.

Critical weaknesses
  • The main concern is understanding how much the build process affects evaluation. The authors verify the base and developer-fixed builds, which is good. However, Android dependencies can change, and compiling (Gradle builds) can take a considerable amount time. It would help to report build consistency across reruns, build/test times, and timeout rates.

  • I also think the choice of harness needs stronger validation. The improvement from using mini-SWE-agent already shows that agent harness/scaffolding is important. Testing other harnesses (particularly newer ones) such as OpenHands, or adding Gradle-specific tools under the same budget be useful in determining the actual difficulty of the tasks included. The related work should also discuss a very closely related work AndroidBuildBench and GradleFixer [1], which study build repair and show improvements from domain-specific tools, particularly since the authors do note that "Build/compile failure is the dominant failure mode for every model."

[1] Automating Android Build Repair: Bridging the Reasoning-Execution Gap in LLM Agents with Domain-Specific Tools. EACL 2026.

Questions for authors
  1. How reproducible are builds across reruns, and what are the build/test times and timeout rates?

  2. How do results change with another harness or Gradle-specific tools under the same budget?

1 comment
AuthorsOpenReview ↗

Response to Reviewer F6Dg

We appreciate Reviewer F6Dg's feedback on build reproducibility and harness choice.

Q1. Build reproducibility, times, and timeouts

All 407 gold patches pass offline in the released images, which contain the repository-specific toolchains and cached dependencies. Across the four model runs, we completed evaluation of all 1,409 submitted patches using these images.

The table accounts for all 1,628 task–model pairs (407 tasks × 4 models). Evaluation wall times are in minutes.

Framework and outcomeCountMedian90th percentileMaximum
Android native, tests ran3233.511.525.0
Android native, build failed before tests7313.47.213.6
Android native, patch rejected510.20.41.8
Flutter, all outcomes2460.30.92.7
React Native, all outcomes570.30.71.1
Evaluated, timing unavailable1———
No patch to evaluate219———
Total1,628———

Times include patch application, build/test execution, and cleanup, excluding image download. The logs do not separate compilation from test execution.

During generation, 180 of the 1,626 runs with logs (11.1%) hit at least one 15-minute command timeout.

We will add these timing and timeout measurements to the experimental setup.

Q2. Other harnesses and Gradle-specific tools

Thank you for pointing us to AndroidBuildBench and GradleFixer. Their objective is to repair failing Android builds. In MobileDev-Bench, we prepare and remediate build environments during benchmark construction (Section 3.3 and Appendix A.5). We retain only tasks whose base and developer-fixed configurations build and execute tests successfully (Section 3.4), excluding candidates with unresolved build failures or nondeterministic test outcomes (Appendix A.4). The agent's task is to implement the issue's requested change in this prepared environment and pass the required tests. We will add the distinction between this paper and our work to Section 2.

We have not run an OpenHands or Gradle-tool comparison under a matched budget. We prioritized making the required interfaces explicit in N2P task inputs, as detailed in our response to Reviewer YTKL (Q1). Our audit found missing or incompatible interfaces in 173 of the 232 GPT-5.2 runs in which no tests executed.

With mini-SWE-agent, GPT-5.2 resolved 88/260 N2P-only tasks (33.8%) in the interface-specification setting, compared with 9/260 (3.5%) in the original issue-only setting. We will release the specifications as task inputs and report this setting separately.

Reviewer_CUGpOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper gives the MobileDev-Bench that introduces 407 manually verified issue-resolution tasks from 19 repositories spanning Android Native, React Native, and Flutter. It provides containerized evaluation environments and a compilation-aware validation protocol incorporating Non-executable-to-Pass transitions. Four models are evaluated with Agentless and mini-SWE-agent, with a best reported resolution rate of 12.53%.

Critical strengths

Useful cross-repository and cross-framework coverage. The benchmark extends executable issue-resolution evaluation to diverse mobile repositories, including fixes spanning source code, resources, and build files. This is a meaningful resource contribution.

Critical weaknesses

The evaluation does not sufficiently represent modern coding agents. Recent model releases report substantial improvements in long-horizon planning, tool use, and autonomous debugging, making the coverage of strong agentic models particularly important for this benchmark. MobileDev-Bench emphasizes multi-file changes, cross-artifact coordination, and framework-specific build constraints. In my view, these characteristics call for an agent-centered evaluation of repository exploration, planning, and iterative repair using execution feedback, rather than primarily a localization-and-patch-generation analysis. Although the paper includes mini-SWE-agent, its interactive evaluation is limited to four models within this single scaffold. It's own finding that interactive execution improves resolution rates by two to four times for its two strongest models further highlights the importance of agentic evaluation. But the currently selected models are quite old as agentic models. So the benchmark can't fully show the true value of agentic models.

Questions for authors

1 The provided benchmark dataset link isn't anonymous.

2The difficulty label uses the estimated resolution time. Do authors have consistent assumptions regarding the familiarity with the developers' repositories and the scope of work included in the estimates? Have these estimates been calibrated using the time taken by a small number of actual fixes?

3 What directly checks the target behavior in tasks with only collateral N2P and no F2P?

4 Can mobile-specific difficulty be separated from ordinary large-patch difficulty? The benchmark already contains 58 platform-agnostic tasks. Could these support a comparison controlling, as far as possible, for repository, patch size, and file count? The zero-success categories are interesting, but do not by themselves establish missing platform knowledge as the cause.

1 comment
AuthorsOpenReview ↗

Response to Reviewer CUGp

We appreciate Reviewer CUGp's feedback, we address the concerns raised as follows:

W1. Coverage of modern agents

mini-SWE-agent lets each model inspect repository files, execute commands, and revise its patch using execution feedback (Section 5 and Appendix E.6). We will replace Gemini 2.5 Flash with Gemini 3.8 Flash and rerun mini-SWE-agent on all 407 tasks.

We also reran GPT-5.2 on the 260 N2P-only tasks with the required interfaces in the prompt. It resolved 88/260 tasks (33.8%), compared with 9/260 (3.5%) in the original issue-only setting. The other three model runs in this setting are in progress. We will release the specifications as task inputs and add the results and failure analysis to the main text, as detailed in our response to Reviewer YTKL (Q1).

Q1. Dataset anonymity

The dataset card linked in the paper contains no names, affiliations, or citations identifying the submission authors. It is hosted under the MobileDev-Bench organization.

Q2. Difficulty labels

Both annotators estimated the time for a developer already familiar with the repository to investigate the issue, implement the fix, set up the build, and test the change. We did not calibrate these estimates against timed fixes. We will document the estimation procedure in Appendix B.

Q3. Tasks with only collateral N2P tests

Collateral tests that already passed at the base commit check that the fix restores compilation without regressions. Our audit of the required assertions across all 260 N2P-only tasks found:

  • 174 tasks with at least one required test that directly checks a behavior requested by the issue.
  • 32 maintenance tasks validated through build and regression checks.
  • 54 feature or bug-fix tasks with regression tests but no required assertion that directly checks the requested change.

We will add these validation labels to the dataset and report results for each group separately, as detailed in our response to Reviewer kn81 (Q1).

Q4. Mobile-specific difficulty and patch size

We compared the 58 platform-agnostic tasks with the other 349, counting a task as resolved if at least one of the four models resolved it using mini-SWE-agent.

Task groupTasksResolved
Platform-agnostic5813 (22.4%)
Other34947 (13.5%)

After controlling for repository, patch size, and file count, the odds ratio for platform-agnostic versus other tasks is 0.56 (95% interval 0.23–1.41). We used conditional logistic regression stratified by repository, covering 381 tasks in the 13 repositories with both resolved and unresolved tasks.

We will add this comparison to the appendix and present missing platform knowledge as a hypothesis.

Reviewer_kn81OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MobileDev-Bench, a benchmark of 407 verified issue-resolution tasks from 19 mobile application repositories covering Android Native, Flutter, and React Native. The construction process includes environment setup, execution-based validation, and manual review. The benchmark includes both tests that fail before a fix and tests that cannot run until the required code is added. Four models are evaluated using Agentless and mini-SWE-agent. Resolution rates generally decrease for harder tasks and larger changes. The best tested model-agent combination resolves 12.53% of the tasks.

Critical strengths
  1. I find the focus on mobile application development useful. These tasks often require coordinated changes across source code, resources, and build files. A benchmark covering this work can help evaluate whether coding agents can handle practical development tasks in existing projects.

  2. The construction process includes both execution checks and manual review, which are useful quality-control steps. Including N2P tests also makes sense: a test for a new feature may not compile until the required interface is implemented. This allows the benchmark to cover tasks that would be excluded by requiring F2P tests alone.

Critical weaknesses
  1. I would like clearer evidence about the validation of the 260 tasks that have N2P tests but no F2P tests. I understand why N2P is needed, and the paper already distinguishes primary and collateral N2P tests. However, unchanged collateral tests that already passed at the base commit mainly show that existing behavior has not regressed once compilation is restored. They do not necessarily check the behavior requested by the issue.

Could the authors provide a task-level breakdown showing which N2P-only tasks have tests that directly check the requested behavior, and whether any rely only on previously passing regression tests? This would help me assess how reliably passing the tests indicates that the task has been completed.

  1. The localization results are difficult to interpret separately from patch generation. Appendix E.4.2 measures overlap between the files edited by the model and those edited by the developer. A model may find a relevant file but fail to produce an edit for it. A valid alternative fix may also modify a different set of files.

The Oracle comparison changes both the context and the generation budget, as the paper acknowledges. I therefore think the conclusions about the relative importance of localization and generation should be more cautious. Measuring retrieval before patch generation, or comparing settings with the same generation budget, would make this analysis more convincing.

Questions for authors
  1. Among the 260 N2P-only tasks, how many have tests that directly check the behavior requested by the issue? Are there any tasks whose validation relies only on restoring compilation and passing unchanged regression tests that already passed at the base commit? A task-level breakdown with a few representative examples would help clarify this.

  2. What evidence supports the conclusion that patch generation is a larger bottleneck than localization? The reported localization metrics use the files edited in the final patch, and the Automated and Oracle settings also differ in context and generation budget. Do you have retrieval-stage measurements or a comparison with a matched generation budget? If not, could the conclusions be qualified to reflect these limitations?

1 comment
AuthorsOpenReview ↗

Response to Reviewer kn81

We appreciate Reviewer kn81's feedback on test validation and localization.

Q1. Validation of the 260 N2P-only tasks

We audited the required tests for all 260 N2P-only tasks:

  • 174 tasks have at least one required test that directly checks a behavior requested by the issue.
  • 32 maintenance tasks, including refactoring, migrations, build repairs, and test changes, are validated through build and regression checks.
  • 54 feature or bug-fix tasks have regression tests but no required assertion that directly checks the requested change.

Yes, some tasks rely on restoring compilation and passing regression tests with unchanged assertions. These tests already passed at the base commit.

Examples:

  • AntennaPod #7581: four new tests check conversion of plain-text links to HTML links.
  • Element #2588: an existing onboarding test checks that the screen sequence excludes the Welcome screen.
  • openHAB #3021: the issue requests splitting a preferences file. Its three required tests retain their original assertions and passed at the base commit.

We will add these labels to the dataset and report results for each group separately.

Q2. Localization versus generation

Appendix E.4.2 measures overlap between the final model patch and the developer patch. We will call this metric patch-file overlap. It excludes files inspected but not edited, and a correct alternative fix can edit a different set of files.

Even with the developer-modified files, the best Oracle resolution rate is 5.65%. As described in Appendix E.3, Oracle uses full files and one greedy sample, while Automated uses localized snippets and multiple samples with test-based selection. We do not report a retrieval-stage measurement or a comparison with matched generation budgets. We will revise Appendix F.4.1 and Appendix G to report the Oracle result without ranking generation against localization or treating patch-file recall as a ceiling on resolution.

Reviewer_YTKLOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Most SWE-bench-style benchmarks are built from library repositories and do not cover mobile apps, where patches must build through a framework toolchain and often modify resources, manifests and build scripts. The authors mine PRs from 19 open-source Android Native, Flutter and React Native apps, build Docker environments for them, and keep 407 tasks after execution-based filtering and review by two annotators. Because test patches often reference symbols introduced by the fix, the tests cannot compile before the fix is applied. The authors therefore add a Non-executable-to-Pass (N2P) category, and 63.9% of tasks are validated by N2P tests alone. The best configuration, mini-SWE-agent with GPT-5.2, resolves 12.53% of tasks, compared with 69.0% for the same setup on SWE-bench Verified. Agentless resolves 3-4%.

Critical strengths
  • Mobile app development is not covered by existing issue-resolution benchmarks, and building reproducible container environments for Gradle, Flutter and React Native projects requires substantial effort. The authors filtered about 74K PRs down to 407 tasks and release the images. The tasks are also considerably larger than in existing benchmarks (median 5 files and 93 lines).
  • The leakage controls are careful (reflog expiry, registry-only network access), and the N2P category addresses a real limitation of the standard F2P criterion for compiled test targets.
  • The numbers are mostly consistent across tables, and the paper is open about several of its own confounds.
Critical weaknesses
  • My main concern is that the tests may largely measure whether the model reproduces the developer's identifiers. 348 of 407 tasks have N2P tests and 260 are validated only by them. By definition these tests reference symbols that exist only in the gold fix, while the model sees only the issue text and no interface specification, so a correct fix with different names will not compile. The results are consistent with this: 65.5-75.4% of unresolved submissions are build failures where no tests ran (Table 25), and N2P tasks are resolved at 12.6% vs. 32.2% for F2P-only tasks. As a result, it is unclear how much of the gap to SWE-bench Verified is due to mobile development itself.
  • The mobile-specific explanations (missing platform knowledge, cross-artifact coordination) are not separated from task difficulty and patch size. For example, the seven categories with zero resolutions (48 tasks) may simply consist of hard tasks.
  • All results come from a single run with no confidence intervals. Gemini loses 158 tasks to tool-format errors, and Claude's mean cost ($2.01) is right at the $2 cap.
  • The compilation-aware validation is listed as a contribution (L93-95), but Multi-SWE-bench already records none-to-pass (N2P) tests and keeps tasks without any F2P test (e.g., facebook__zstd-3942 has 39 N2P and no F2P tests). The paper cites it as requiring an F2P test (L52-53), which does not seem accurate, and should position its procedure relative to it.
  • The dataset is unbalanced: four repositories contribute 74.4% of the tasks, and only 16 tasks come from React Native.
  • The main text is dense with percentages, and the most informative diagnostic (Table 25) is only in the appendix. A few numbers do not match (the 10.8% Medium-tier rate vs. Table 7; $2.0016 vs. $2.0132 in Table 20).
Questions for authors
  1. Among the build failures in Table 25, how many are caused by test files referencing symbols the model did not define, as opposed to errors in the model's own changes? Could you rerun mini-SWE-agent with the signatures of the new test-referenced symbols included in the prompt, as SWE-bench Pro does? This would most directly address my main concern.
  2. App. B says that tasks whose tests impose undue implementation constraints were excluded (Fig. 4). How was this criterion applied to the 260 tasks validated only by N2P tests, which need the symbols defined in the gold fix? What was the agreement on the accept/exclude decisions?
  3. What is the difficulty distribution of the 48 tasks in the zero-resolution categories?
  4. The model patch is applied after the test patch (E.1). Can it modify test files or the Gradle test configuration? Do the gold patches pass offline in the released images?
  5. Claude's mean cost of $2.01 per instance (Table 20) implies that nearly every run reached the $2 cap, but Table 17 lists only 38 runs that exceeded their limits. How is the budget enforced, and which figure is correct?
1 comment
AuthorsOpenReview ↗

Response to Reviewer YTKL

We thank the reviewer for their insightful feedback and suggesting the interface-specification experiment.

Q1. Missing interfaces and the rerun

We reran GPT-5.2 on all 260 N2P-only tasks with the required interfaces in the prompt. The specifications give the names, signatures, and dependency or interface requirements imposed by the tests, without implementation bodies. We will release them as task inputs and report this setting separately from the original issue-only setting.

N2P-only tasksIssue onlyWith specification
Resolved9/260 (3.5%)88/260 (33.8%)
95% Wilson interval1.8–6.4%28.4–39.8%

The 88 resolved tasks include all nine that GPT-5.2 solved with the issue-only prompt and 79 additional tasks. We will also report results for the other three models with the same interface specifications as the experiments are still in progress.

We inspected the logs and patches for all 232 GPT-5.2 runs in which no tests executed (Table 25's 228 build failures and 4 apply failures):

  • 159: Tests or test fixtures required interfaces missing from or incompatible with the model's code.
  • 13: Errors introduced by the model's edits.
  • 14: Both interface mismatches and errors introduced by the model's edits.
  • 26: Patch application failures.
  • 19: Missing modules or dependencies introduced by the gold fix.
  • 1: Incomplete log.

Q2. Annotation criterion and agreement

The rubric in Section 3 asks whether the tests correspond to the issue, and scores of 2 or 3 excluded 99 of 541 candidates. Before discussion, the annotators agreed on retain or exclude for test coverage on 411 of 536 candidates (76.7%, Cohen's κ=0.08\kappa=0.08). Among the 260 retained N2P-only tasks, both annotators initially retained 224. The other 36 required discussion.

Q3. Difficulty of the 48 zero-resolution tasks

The breakdown is:

  • 6 Easy tasks (12.5%).
  • 16 Medium tasks (33.3%).
  • 26 Hard tasks (54.2%).

Across the benchmark, 31.9% of tasks are Hard. The 48 tasks have a median patch size of 11.5 files and 279 lines, versus 5 files and 93 lines overall.

We will retain missing platform knowledge as a hypothesis and add these difficulty and size comparisons to the discussion.

Q4. Test edits and offline gold-patch validation

The prompt instructs the model not to modify test files. Gradle configuration can be changed as part of the fix. We inspected all 51 passing GPT-5.2 baseline patches and all 88 passing interface-specification patches. None changes test assertions or disables tests. A submission fails if any test listed for the task fails, is skipped, or is missing from the results.

Yes, all 407 gold patches pass offline in the released images. Section 3.4 reports 100% gold-patch success by construction. We will make the offline condition explicit.

Q5. Cost cap

mini-SWE-agent checks the budget before each model call, so the final call can exceed the cap. Claude's mean was USD 2.0132. 379 runs hit the cap, and only 38 of these ended without a patch. Table 17 counts only those 38. We will clarify its caption and correct the USD 2.0016 in the appendix text.

W3. Single-run results and Gemini

Each setting uses one generation run per task. We will add intervals for the headline rates. GPT-5.2's 12.53% has a 95% Wilson interval of 9.7–16.1%. These intervals do not measure run-to-run variability. Gemini 2.5 Flash's 158 tool-format failures prevent interpreting its overall rate as a clean measure of model capability. We will replace Gemini 2.5 Flash with Gemini 3.8 Flash and rerun the evaluation.

W4. Multi-SWE-bench

Appendix E.1 already credits Multi-SWE-bench for the evaluation protocol. Our analysis distinguishes primary and collateral N2P tests in mobile repositories. We will revise the introduction and contribution statement to acknowledge Multi-SWE-bench's existing support for N2P-only tasks.

W5–W6. Imbalance and presentation

Giving each of the 19 repositories equal weight preserves the mini-SWE-agent model ranking, with GPT-5.2 at 18.4%. We will report these averages alongside the overall rates and state next to the React Native results that they cover 16 tasks from four repositories.

We will add the failure analysis from the full experiments in the interface-specification setting to the main text and trim repeated percentages. We will correct the Medium-tier rate to 12.9% (18/139), counting tasks resolved by at least one of the four models using mini-SWE-agent, consistent with Table 7.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Contributions and Responses to Reviewer Concerns

Dear Area Chair,

We thank the reviewers for their feedback on MobileDev-Bench. Below, we present a summary of our contributions along with our responses to the reviewers' concerns.

Contribution

MobileDev-Bench provides 407 manually verified issue-resolution tasks from 19 Android Native, Flutter, and React Native repositories, with Docker environments and test-based evaluation.

Reviewers' assessment

  • Relevance and coverage. “an important and difficult problem” (F6Dg), “not covered by existing issue-resolution benchmarks” (YTKL), and “Useful cross-repository and cross-framework coverage” (CUGp).

  • Practical tasks and resource value. “considerably larger than in existing benchmarks” (YTKL), “coordinated changes across source code, resources, and build files” and “practical development tasks” (kn81), and “a meaningful resource contribution” (CUGp).

  • Construction and validation. YTKL noted the filtering of 74K PRs, released images, and effort behind reproducible environments. Reviewers praised “useful quality-control steps” (kn81) and “build validation and human review” (F6Dg).

  • N2P and leakage controls. “addresses a real limitation” and “The leakage controls are careful” (YTKL). N2P covers tasks “excluded by requiring F2P tests alone” (kn81).

  • Reporting and transparency. “mostly consistent across tables” and “open about several of its own confounds” (YTKL).

Responses to the main concerns

  1. Interfaces (YTKL Q1, F6Dg Q2). We found interface mismatches in 173/232 GPT-5.2 runs with no tests executed. With required interfaces and no implementation bodies in the prompt, GPT-5.2 with mini-SWE-agent resolved 88/260 N2P-only tasks (33.8%), versus 9/260 (3.5%) in the issue-only setting.

  2. Validation (kn81 Q1, CUGp Q3). Our audit classified all 260 N2P-only tasks. 174 have a required test directly checking an issue-requested behavior. 32 maintenance tasks use build and regression checks. 54 feature or bug-fix tasks have regression tests but no required assertion checking the requested change.

  3. Builds (YTKL Q4, F6Dg Q1–Q2). All 407 gold patches pass offline in the released images. We evaluated all 1,409 submitted model patches and report evaluation-time distributions and command-timeout counts.

  4. Difficulty and imbalance (YTKL Q3, W5, CUGp Q4). Controlling for repository, patch size, and file count gives an odds ratio of 0.56 for platform-agnostic versus other tasks (95% interval 0.23–1.41). Equal repository weighting preserves the mini-SWE-agent model ranking, with GPT-5.2 at 18.4%.

Final Version Plans

  • Report the other three models' interface-specification runs, currently in progress. Replace Gemini 2.5 Flash with Gemini 3.8 Flash and run mini-SWE-agent on all 407 tasks.
  • Release interface specifications as task inputs and validation labels in the dataset. Report the specification setting separately and results for each validation group.
  • Add the full specification results and failure analysis to the main text. Add headline confidence intervals, repository averages alongside overall rates, and the 16-task, four-repository coverage beside React Native results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.