acceptodds
Under review as a conference paper at ICLR 2027

Slice2Fix: Discovering Error Slices for Targeted GUI Agent Repair

Abstract

Graphical-user-interface (GUI) agents are increasingly used to operate mobile applications, yet they still fail whole groups of tasks that share a common property. Existing ways of learning from execution feedback stay at the level of single tasks or written reports: they repair the tasks that failed, keep the tasks the agent already solves, or summarise the failures into weakness reports, and none of them turns the property behind the failures into new tasks without relying on a stronger model to solve them. To address these limitations, we propose Slice2Fix, which repairs a GUI agent by its systematic failures rather than by its failed tasks. Its core idea is to define a systematic failure as an error slice, a group of tasks described by a combination of controllable task features, so that the object that names a failure also specifies the tasks that repair it, and a reference executor written for each task family solves them, closing the loop from diagnosis to repair. Across three strong open-source end-to-end GUI agents at the 8B scale, slice-directed repair raises success on the repaired slices by 20 to 29 percentage points. With the same amount of data, untargeted tasks from the same generator, applications and operations gain 13.1 points there on GUI-Owl-1.5-8B, against 20.0 for slice-directed repair. The repaired agents also improve on AndroidWorld, including templates that appear in no training data. We also release FeatBench, a feature-labelled benchmark of 974 mobile tasks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
GUI Agent, error slice discovery, failure pattern mining

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BporOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that GUI agents fail in groups: tasks sharing some property, such as near-duplicate entries in a list or an option hidden in a long-press menu, tend to fail together. So repair should target the property, not the individual failed tasks. The authors build a task generator whose tasks are written from hand-designed feature labels. They run three 8B agents on a pool of about 1,000 generated tasks and greedily search for label combinations with unusually high failure rates. Each candidate is then checked against control tasks with the feature turned off. For the slices that survive, they generate new tasks with the same labels, record solutions with scripted per-family executors, and fine-tune the agents with LoRA plus a rehearsal set. Success on the target slices goes up by 20–29 points, while gains on AndroidWorld are small (1–5 points) and not significant individually. The paper also releases FeatBench, a 974-task labelled benchmark.

Critical strengths

The main idea is neat: because a slice is defined by the generator’s own settings, the thing that describes a failure also tells you how to make training data for it, and how to build a feature-off control to check that the feature is really the cause. This removes the need for a stronger teacher, which matters here since the 30B models fail these tasks about as often as the 8B ones. The control check is not just decoration either: it threw out two high-failure slices whose difficulty actually came from the app. The experiments within this setting are also reasonably careful. The untargeted-data ablation matches the generator, apps and data budget (+13.1 vs +20.0), and the split-half analysis keeps slice selection separate from evaluation.

Critical weaknesses

Most of the gain is in-distribution. The features, repair tasks, executors and FeatBench all come from the same generator, so large gains on target slices are expected. The real test is AndroidWorld and AndroidLab, and there the gains are 1–5 points and none is significant. MAI-UI does not move on AndroidLab, and UI-Venus is worse than both baselines there. The paper also never compares against simpler ways of choosing what to oversample, such as picking features by hand or having an LLM summarize failures. So it is unclear whether the mining step matters, or whether the gain just comes from putting more data where the test set is.

Narrow scope, and the headline number is inflated. Every task family needs hand-designed features, a generator and a scripted executor, and the method can only find failures that the feature list already anticipates. The claim that 12 slices cover 59.6% of failures includes the two slices the paper itself rejects. Without them, the figure is about 50%.

The executor does much of the work. It scrolls through and reads every candidate before choosing, which is exactly the behavior that fixes the near-duplicate failures. The fix comes from engineering, not from the diagnosis.

Questions for authors

Table 6 counts the two rejected slices toward the 59.6% coverage. Could the authors also report coverage using only the confirmed slices?

The body-text slice was not rejected but received no repair data. Are its tasks included in the 275 target tasks?

In Appendix C, how is a slice handled when its gap falls between X− and X+?

Are results of the untargeted-data ablation available for MAI-UI and UI-Venus, and how much do the results vary across training seeds?

Have the authors compared slice mining with simpler ways of choosing target features, such as manual selection from failure cases or LLM-summarized failures, under the same generator and data budget?

I find the core idea interesting. If the authors can address these points, especially Q1, Q4 and Q5, I would be happy to reconsider my score.

5 comments
AuthorsOpenReview ↗

Response to Reviewer Bpor (Part 1/2)

We thank the reviewer for the clear summary and concrete questions. Below we describe the additional comparisons and clarify the coverage and evaluation rules. Changes are marked in blue in the revised paper.


W1: Value of slice mining and external transfer

To distinguish the effect of slice selection from simply adding data, our untargeted control uses the same generator, applications, operations and data budget as slice-directed repair (Table 3 of the paper). On GUI-Owl, untargeted data gains +13.1 points, while data for the mined slices gains +20.0. The additional gain therefore comes from selecting the repair tasks. We extended this comparison to the other two agents, three training seeds on GUI-Owl and LLM-selected targets; mining gives larger gains in each comparison (Q4 and Q5).

To test transfer to a new application, we built deletion tasks in Contacts, which appears in neither the discovery pool, the repair data nor FeatBench. We included tasks with same-name contacts and controls without same-name distractors. The controls test whether the agents can perform the deletion when name ambiguity is absent. We evaluated the Slice2Fix models from Table 1 of the paper without retraining.

Table 1. Success (%) on the new Contacts tasks.

AgentControl tasks, baseSame-name tasks, baseSame-name tasks, + Slice2Fix
GUI-Owl-1.5-8B94.427.150.8 (+23.7)
MAI-UI-8B100.027.147.5 (+20.3)
UI-Venus-1.5-8B100.011.927.1 (+15.3)

The contrast between the control and same-name tasks reproduces the failure pattern in Contacts. On the same-name tasks, the existing repairs gain +15.3 to +23.7 points across the three agents, showing that the benefit transfers to this new application.

AndroidWorld and AndroidLab were constructed independently of the repaired slices. To assess whether the gains on these benchmarks persist across training runs, we trained GUI-Owl under two additional seeds and evaluated each of the three models three times.

Table 2. Success (%) of GUI-Owl with Slice2Fix under three training seeds on the external benchmarks.

GUI-Owl-1.5-8BAndroidWorldAndroidLab
Base64.728.3
+ Slice2Fix, training seed 1 (the run in the paper)69.3 (+4.6)33.3 (+5.1)
+ Slice2Fix, training seed 269.3 (+4.6)37.9 (+9.7)
+ Slice2Fix, training seed 367.2 (+2.6)36.7 (+8.5)

Every training seed improves over the base agent on both benchmarks. We have added the Contacts and seed results to Appendix E.


W2: Feature reuse and coverage

The features describe properties such as how a target is identified and how similar its distractors are. These properties recur across applications and operations, so the feature set is shared across task families. The 19 features in Table 4 of the paper describe all 20 families in our ten applications, and we used the same features to label the 116 AndroidWorld tasks.

A new family largely reuses this vocabulary, the generator code implementing it and common executor steps. A coding agent writes the remaining family-specific code, which is validated automatically (Appendix F). If a failure involves a property outside the vocabulary, we add a feature as another generator option and rerun mining on the trajectories already collected. Q1 reports the coverage of the confirmed slices.


W3: Task selection and the shared executor

To separate task selection from the executor's behavior, all three controls use the same reference executors as Slice2Fix. Each executor scrolls through and reads every candidate in the same way; the training tasks differ. On GUI-Owl's target tasks, instance-level repair gains +12.6 points (Table 1 of the paper), untargeted data gains +13.1 (Table 3 of the paper), and LLM-selected targets gain +14.5 (Table 4 below). Data for the mined slices gains +20.0.

The executor supplies correct actions, but the choice of tasks determines which failures those actions train the agent to handle. For example, only 40 of the 261 untargeted deletion tasks contain near-duplicates. Reading every candidate in all 261 tasks does not concentrate the demonstrations on near-duplicate failures. Mining selects such conditions explicitly, which accounts for the additional benefit beyond using the same executor.

AuthorsOpenReview ↗

Response to Reviewer Bpor (Part 2/2)

Q1: Coverage of confirmed slices

Thank you for pointing this out. Counting only confirmed slices gives 647 of the 1,294 failures in the discovery pool, which equals to 50.0%. We now report this coverage in the main text and the caption of Table 6.


Q2: The body-text slice and target tasks

FeatBench contains no task from the body-text slice, so none is included in the 275 target tasks. For repair, a slice must contain at least five discovery tasks; with fewer examples, the failures may be specific to those tasks rather than the feature. The body-text slice contains four tasks and therefore received no repair data. We have added this criterion to Section 3.4.


Q3: Slices between the confirmation thresholds

A gap between X− and X+ leaves the slice neither confirmed nor rejected. We then distinguish its use for training from its use in evaluation:

  • Repair data. Among slices with at least five discovery tasks, we exclude only those rejected by the check (Section 3.3). An intermediate slice still has a failure rate of at least 75% in the discovery pool, and the check has not ruled out its feature as the cause. It therefore receives repair data.
  • Target evaluation. Only confirmed slices contribute target tasks. Tasks from an intermediate slice are excluded, so the reported target-task gains come from confirmed slices only.

We have clarified these rules in Appendix C.


Q4: Untargeted controls and training seeds

We extended the comparison with untargeted data to MAI-UI and UI-Venus to test whether the advantage of selecting slices also holds for these agents. To assess training variability, we trained GUI-Owl with each type of data under two additional seeds, giving three seeds in total.

Table 3. Success on the target tasks (%) with untargeted and targeted data. GUI-Owl reports the mean ± standard deviation over three training seeds.

AgentBase+ untargeted tasks+ Slice2Fix tasks
GUI-Owl-1.5-8B (three training seeds)43.655.3 ± 1.5 (+11.6)66.8 ± 2.9 (+23.2)
MAI-UI-8B31.640.7 (+9.1)55.7 (+24.0)
UI-Venus-1.5-8B5.123.6 (+18.5)33.8 (+28.7)

Targeted data gives larger gains on both additional agents. For GUI-Owl, each Slice2Fix run (+20.0 to +25.8) exceeds each untargeted run (+10.2 to +13.1), so the advantage is consistent across the tested seeds. We have added these results to Appendix E.


Q5: LLM-based target selection

Thank you for suggesting this comparison. Selecting targets manually requires reviewing many failure trajectories to identify recurring task properties. Our discovery pool alone contains 1,294 failed trajectories, so manual selection is difficult to scale. We therefore implemented an LLM baseline to automate the proposed failure-summary approach.

In a single prompt, GPT-5.4 received all failed discovery tasks and their feature labels, and was asked to summarize the agents' weaknesses and select feature combinations to repair. We generated repair tasks for those combinations with the same generator and executors, and fine-tuned GUI-Owl with the same data budget and training recipe as Slice2Fix.

Table 4. Success on the target tasks (%) of GUI-Owl with targets chosen in different ways. The rows other than the LLM row are from Tables 1 and 3 of the paper.

Targets chosen bySuccess (%)
Base agent, no added tasks43.6
No targets (untargeted tasks)56.7 (+13.1)
LLM58.2 (+14.5)
Mining (Slice2Fix)63.6 (+20.0)

Under the same budget, LLM-selected targets reach 58.2%, compared with 63.6% for mining. The LLM's targets emphasize application-level operations. Three of its descriptions are:

  • "scrolled multi-item deletion and disambiguation in Pro Expense"
  • "attribute-based screen reading and filtered querying in Tasks"
  • "scrolled event deletion using name/date/time disambiguation in Simple Calendar"

The last operation has a failure rate of 16.2% in the discovery pool, below the pool-wide 42.2%. Mining uses both successful and failed tasks to rank feature combinations by excess failure. This identifies high-failure properties within an operation, such as other entries sharing the target's name, and directs repair data to those conditions. We have added this comparison to Appendix E.


We hope these answers address the reviewer's concerns, and we would be grateful if the reviewer would reconsider the score.

Reviewer_BporOpenReview ↗

Response to Authors

Thanks for the response and the extra experiments. The coverage fix, the clarified slice rules, and the untargeted controls with seeds answer Q1–Q4, and the Contacts experiment is a nice addition.

That said, my main concern about the method hasn't really changed. The Contacts tasks still come from the same generator and features, so they don't tell us much about transfer outside it. On AndroidWorld and AndroidLab the gains are still small, the seed runs only cover GUI-Owl, and the MAI-UI and UI-Venus results from before aren't discussed. The LLM baseline is a good start, but with one seed on one agent and a gap close to the seed variation, I'm not yet convinced that mining is what makes the difference. The LLM also only sees failed tasks, which isn't quite a fair comparison.

More broadly, every task family still needs hand-designed features, a generator, and a scripted executor, and the method can only find failures the feature list already covers. To me this reads more as a solid engineering pipeline than a strong method contribution.

So I'll keep my score, but I appreciate the effort that went into the rebuttal.

Reviewer_bL1qOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies improving of GUI agents by consolidating recurring failure patterns rather than individual failed tasks. SLICE2FIX is introduced to consolidate tasks where agents fail often, and generates targeted repair tasks. 'Reference executors' are used to create ground truth trajectories for fine-tuning.

Critical strengths
  • Focusing on systematic failures rather than individual failed tasks is intuitive.

  • The paper evaluates multiple open-weight GUI agents.

  • The method shows large improvement on FeatBench (though not on AndroidWorld and AndroidLab)

Critical weaknesses
  • I am unsure why the authors chose to newly define the term “systematic failures,” since we already have a widely used term: "failure modes". Agent benchmarks already frequently identify recurring failure modes, so it would help to clarify what this terminology/formulation adds.

  • The motivation of the method is that multimodal trajectories do not provide useful per-sample features, but the authors largely sidesteps this by manually defining the task features a priori. It is unclear how well this extends to other systematic failures.

  • Following this, the approach seems expensive to scale, since each task family requires feature definitions, a generator, success checks, and a reference executor.

Questions for authors
  1. What does the definition of “systematic failures” add beyond existing "failure mode" definitions?

  2. How would the method discover systematic failures if it cannot be categorized by one of the predefined task features?

  3. How scalable is the approach given the engineering required for each task family?

  4. Why is the improvement on Androidworld and Androidlab much lower than Featbench?

4 comments
AuthorsOpenReview ↗

Response to Reviewer bL1q (Part 1/2)

We thank the reviewer for the careful reading and constructive questions. Below we explain the distinction from failure modes, the role of predefined features, and the implementation and transfer results. Changes are marked in blue in the revised paper.


W1 / Q1: Systematic failures and failure modes

Thank you for asking us to clarify what this definition adds. Consider the deletion example in Figure 3b. The agent deletes the wrong entry when the target is specified by name and amount and other entries share its name. On the same template, the agent succeeds when no other entry shares the name.

A failure-mode description identifies the observed error: "the agent deletes the wrong entry". Failure-mode taxonomies classify what went wrong in an executed trajectory [1]. Our systematic failure instead identifies the task-side cause: "the target is given by name and amount, and other entries share its name". We represent this as an error slice—a combination of task properties under which the agent fails far more often than elsewhere.

The task-side description tells us what to test and what to train on. We can generate control tasks without the shared-name property to check its effect, and new tasks with that property to obtain repair data. A label such as "wrong deletion" alone does not specify those tasks. The two descriptions are therefore complementary, with the error slice providing the link from diagnosis to repair. We have clarified this distinction in Section 2.


W2 / Q2: Predefined features and coverage

The features provide a shared vocabulary for two steps: grouping failures with the same task properties and specifying which properties the generator should reproduce. Embedding clusters or free-form LLM descriptions do not supply these generator settings. Without a shared vocabulary, the same failure can receive different descriptions and cannot be grouped consistently. Predefined features are therefore a necessary part of the method.

Coverage can be extended by addressing two cases:

  • A label is too coarse. Failed and successful tasks share a label, so we refine it to separate them. For example, distractor similarity has six levels (Appendix A), allowing mining to distinguish entries with the same name from entries that only resemble the target.
  • A feature is missing. If no existing feature describes the property causing the failure, we add that feature to the vocabulary.

Both changes only relabel the existing tasks. We rerun mining on the same stored trajectories; the agents do not need to be run again. We have added these two ways of extending the features to Appendix F.

The method does not require exhaustive coverage to improve the agent. The confirmed slices cover 50.0% of the 1,294 failures in the discovery pool (Table 6 of the paper). Repairing them gives all three agents the highest success rate on the full FeatBench, above both baselines, with gains of +13.4, +6.8 and +11.7 points over the base models (Table 1 of the paper). All three also improve on AndroidWorld.

[1] M. Cemri et al. Why Do Multi-Agent LLM Systems Fail? NeurIPS, 2025.

AuthorsOpenReview ↗

Response to Reviewer bL1q (Part 2/2)

W3 / Q3: Engineering cost and scalability

A new task family reuses the mining, confirmation, recording and fine-tuning pipeline. The implementation work is in the task template, application data fields and operation steps. Existing features, such as target count and distractor similarity, also carry over. We add a feature only when the family introduces a new property, implemented as an additional generator option; code for existing properties, such as generating distractors at a given similarity, is reused.

The executor can reuse common operations as well. For deletion, these include scrolling a list to its end, reading every candidate and selecting the target by its full description. The deletion families in Broccoli, Pro Expense and Simple Calendar share this code, with changes for the application-specific steps. Success checks compare the final application data with the expected result; AndroidWorld already provides functions for reading this data.

After we select the application and operation, a coding agent writes the family-specific code. We validate every generated task with three automated checks: the success check must fail on the untouched application, pass after the executor completes the task, and fail after a near-miss action on a distractor. We built all 20 families in ten applications this way. Appendix F now describes the reused components and the work needed for a new family.


Q4: In-distribution and external gains

The repair data train the agent on particular task properties. FeatBench's target tasks have exactly those properties, so they directly measure improvement on the repaired slices. AndroidWorld and AndroidLab were constructed independently and contain a broader range of tasks; they test whether that repair also helps beyond its targets.

This distinction also appears within FeatBench. On its non-target tasks, the gains are +10.8, +0.1 and +4.9 points for GUI-Owl, MAI-UI and UI-Venus, respectively. These are of the same order as the gains on AndroidWorld (+4.6, +1.1 and +3.1) and AndroidLab (+5.1, 0.0 and +5.8), reported in the same table. The larger gains on the target tasks thus reflect the conditions the repair data were designed to address.

On AndroidWorld, GUI-Owl-1.5-8B with Slice2Fix reaches 69.3. MobileForge, which adapts the same base model using tasks generated in the AndroidWorld applications, reaches 67.2 (ForgeOwl-8B in Table 1 of the paper). GUI-Owl-1.5-32B, a model four times larger, also reaches 67.2. Slice2Fix therefore remains ahead of both on this external benchmark. We have added the explanation of the gain differences to Section 4.3.


We hope these answers address the reviewer's concerns, and we would be grateful if the reviewer would reconsider the score.

I thank the authors for their response. To be honest, I'm still not entirely sure what the difference is between systematic failures and failure modes. I suggest the authors think of a more intuitive example. Currently they just seem like two sides of the same coin.

My main concern is still with the predefined features. I understand that once a feature is added, the authors can rerun the analysis without running the agents again. But how do they know which feature to add in the first place? I think it would be useful to at least show an example where looking at the failures led them to add a feature they had not thought of beforehand.

Thank you for following up.


1. Failure modes and systematic failures

A failure mode describes what went wrong without explaining why. A systematic failure explains why the error recurs by identifying the task condition that makes it more likely.

Take a deletion task in the expense app Pro Expense. One of the three expenses to delete is a “Train Pass” for 192.55. The list also contains another “Train Pass” for 202.67. If the agent deletes the 202.67 entry, the failure mode is “deleting the wrong entry.”

This task requires the agent to keep track of three expenses to delete. It also requires the agent to distinguish two “Train Pass” entries by amount. If all we know is that the agent deleted the wrong entry, should we give it more practice keeping track of multiple targets or distinguishing entries with the same name?

Mining found that deletion tasks failed more often when other entries shared or nearly shared the target’s name (Table 6). To test the effect of shared names, we generated two sets of fresh tasks in Pro Expense, Broccoli and Simple Calendar. Both sets required deleting three targets. Across the three agents, 147 of 207 attempts failed when other entries had the same names as the targets, compared with 6 of 207 when those entries had unrelated names.

“Deleting the wrong entry” is the failure mode. The systematic failure is that the agents fail far more often when a name alone cannot identify the target and a second attribute must be checked. This tells us where to spend a limited training budget. Even if every task requires deleting three expenses, the agent can succeed by matching names alone whenever each expense has a unique name. To give it practice checking the amount, the training list needs expenses with the same name but different amounts.


2. Reliance on predefined features

Predefined features are what make each slice interpretable. A slice is a short combination of named features, such as "other entries share the target's name" in point 1, so it reads as a description of the tasks the agent fails and states what repair data to generate. Clusters of trajectory embeddings come with no such description, and free-form LLM descriptions differ from one failure to the next and cannot be grouped. The feature set is not closed and can be made as broad as needed.

To extend it, we take new features from the failures that no slice covers. An LLM proposes candidate features from them, and we use mining and tests on fresh tasks to determine which ones identify systematic failures. After mining, the LLM reads a sample of these failed trajectories and suggests properties they share. We keep a candidate if it can be expressed as a rule that a program can apply to every task in the pool. We then label the tasks with that rule and rerun mining.

The icon-only attribute (Table 4) is a feature we had not thought of beforehand. One uncovered failure came from an OpenTracks task in which GUI-Owl failed to list the running activities. The final screenshot showed that the app marks each activity's type only with an icon. Our existing features captured that the task required filtering a list, but not how the relevant attribute was displayed. The LLM therefore proposed a new feature: "the attribute used to filter the list is shown only as an icon."

After relabeling, this slice ranked second in Table 6 by number of failed trajectories. It covered 36 tasks, with an 82.4% failure rate compared with 42.2% over the whole pool. We confirmed it on fresh tasks and used it to generate repair data. Verbatim text, insertion at an exact point and distractor similarity in Table 4 also came from reading agent trajectories. We have added the OpenTracks example to Appendix F.


We hope our response has addressed your concerns. If you have any further questions, we would be happy to discuss them.

Reviewer_8wXSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Slice2Fix mines conjunctions of controllable task features that concentrate GUI-agent failures using excess-failure ranking, support and failure-rate gates, and harvest-and-peel selection. Feature-on/off confirmation identifies repair targets, then feature-controlled generation and parameterized reference executors produce demonstrations without requiring a stronger teacher to solve each task. Evaluation covers three 8B GUI agents, a 974-task feature-labeled benchmark, and AndroidWorld and AndroidLab. The selected repair slices improve by 20.0, 24.0, and 28.7 percentage points, respectively, and a matched untargeted-data control on GUI-Owl supports the value of slice-directed selection.

Critical strengths

S1 A coherent diagnosis-to-repair interface. The same feature description both identifies a failure group and specifies new training instances. This closes a practical gap between descriptive weakness reports and a data-collection procedure that can act on those reports. S2 Meaningful controlled evidence. The GUI-Owl ablation compares 448 targeted trajectories with budget-matched untargeted data from the same generator and operations. Targeted success gains 20.0 points versus 13.1, with a reported McNemar p-value of 0.005. Rehearsal-only and repair-only analyses also expose the contribution of data composition rather than attributing all gains to mining. S3 Attention to selection and exposure. Appendix E cross-fits confirmation and repair measurement within FEATBENCH task groups, including 200 random splits, and finds the target selection stable. AndroidWorld results are also split into 77 rehearsal templates and 39 unseen templates. These are important qualifications and should not be overlooked when judging the evaluation.

Critical weaknesses

W1 The complete selection loop is not yet tested prospectively. FEATBENCH baseline outcomes determine which discovered slices receive repair. The appendix cross-fitting is a substantive safeguard and yields gains of similar magnitude; I do not regard the headline results as unmitigated leakage. However, the splitting is within the same template groups after the feature system and discovery process have been developed. A fully separate prospective confirmation and final-test cohort, ideally involving new templates or applications, would better establish robustness of the complete selection procedure. W2 Training variance and the key data-selection comparison are limited. Each training setting uses one LoRA run. Three evaluation runs on external benchmarks do not quantify variation from fine-tuning seeds or independently generated repair sets. Moreover, the clean budget-matched targeted-versus-untargeted experiment is reported only for GUI-Owl. Extending it to the other two agents is important because improvements over instance repair are not uniform: UI-Venus instance repair slightly exceeds slice repair on targeted FEATBENCH tasks and substantially exceeds it on AndroidLab. W3 Out-of-distribution benefits are modest. The large gains are on feature-defined in-domain targets. AndroidWorld gains of 4.6, 1.1, and 3.1 points are individually nonsignificant, and much of that benchmark overlaps the rehearsal template set. The separate unseen-template analysis is welcome, but it is small. External transfer should therefore be described as encouraging rather than established, and the paper should avoid implying uniform superiority across agents and benchmarks. W4 The engineering assumption limits scalability. The generator, feature ontology, and reference executors require task-family-specific implementation. Appendix F explains that the procedures exploit fixed family structure and accessibility-tree labels; new pages or interactions can require executor changes. This is a reasonable design, but the manual and coding-agent effort per family, executor maintenance cost, and rate of unsupported tasks are not quantified. The claim of avoiding a stronger teacher should be distinguished from avoiding domain-specific supervision or privileged tooling.

Questions for authors

Q1 Can the same budget-matched targeted-versus-untargeted comparison be run for MAI-UI and UI-Venus with several fine-tuning seeds and independently generated repair batches? Q2 How much human and coding-agent effort is required to add one task family, feature, or application, and what fraction of candidate tasks cannot be solved by the reference executor? Q3 How much of the benefit survives a prospectively frozen feature ontology and target-selection rule evaluated on entirely new templates or applications? Q4 Can you audit the recorded demonstrations to quantify steps whose actions are justified by visible screen evidence, and explain how accessibility-tree information is prevented from creating an unavailable-information advantage in training?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 8wXS (Part 1/2)

We thank the reviewer for the careful review and constructive suggestions. We have added four experiments and clarified the engineering requirements and demonstration checks. Results are reported in Tables 1–4 below, and changes to the paper are marked in blue.


W1 / Q3: Evaluation on a new application

Thank you for suggesting an evaluation on a new application. To test whether the same-name failure pattern and its repair transfer to an application outside our development and training data, we evaluated deletion tasks in Contacts. This application appears in neither the discovery pool, the repair data nor FeatBench.

We created 59 tasks with same-name contacts and 36 control tasks without same-name distractors. The controls test whether the agents can perform the deletion when name ambiguity is absent. The features, target slices and models were fixed before these tasks were created. We evaluated the base agents and the repaired models from Table 1 of the paper without further training, so the Contacts results could not influence feature or slice selection.

Table 1. Success (%) on the new Contacts tasks.

AgentControl tasks, baseSame-name tasks, baseSame-name tasks, + Slice2Fix
GUI-Owl-1.5-8B94.427.150.8 (+23.7)
MAI-UI-8B100.027.147.5 (+20.3)
UI-Venus-1.5-8B100.011.927.1 (+15.3)

The base agents succeed on 94% to 100% of the controls, but only 12% to 27% of the tasks with same-name contacts. Thus, they can perform the basic operation in Contacts, while the same-name failure pattern persists. The repaired models improve by 15.3 to 23.7 points on the same-name tasks, showing that the existing repair transfers to the new application. We have added this experiment to Appendix E.


W2 / Q1: Data-selection controls and training variability

To test whether selecting slices helps beyond simply adding training data, we compared slice-directed repair with the same amount of untargeted data from the same generator. We extended this comparison from GUI-Owl to MAI-UI and UI-Venus. We also examined two sources of variation separately: the training seed and the generated repair tasks.

Table 2. Success on the target tasks (%) with untargeted and targeted data. The base and Slice2Fix values of MAI-UI and UI-Venus are from Table 1 of the paper.

AgentBase+ untargeted tasks+ Slice2Fix tasks
GUI-Owl-1.5-8B (three training seeds)43.655.3 ± 1.5 (+11.6)66.8 ± 2.9 (+23.2)
MAI-UI-8B31.640.7 (+9.1)55.7 (+24.0)
UI-Venus-1.5-8B5.123.6 (+18.5)33.8 (+28.7)

Table 2 shows larger gains from slice-directed data on all three agents. To assess variation across training runs, we repeated both GUI-Owl settings with two additional seeds; its row reports the mean and standard deviation over all three. The individual Slice2Fix runs gain 20.0 to 25.8 points, compared with 10.2 to 13.1 for untargeted data. Thus, every Slice2Fix run exceeds every untargeted run.

To examine variation in the repair data, we generated another batch for the same slices using different generation seeds, with no task shared with the original batch. We then retrained GUI-Owl with the original training seed, so that this comparison changes the repair data while keeping the training seed fixed.

Table 3. Success on the target tasks (%) of GUI-Owl trained on two batches of repair tasks.

Repair tasksSuccess (%)
None (base agent)43.6
Original batch (the run in the paper)63.6 (+20.0)
Different batch63.3 (+19.6)

The new batch gives a gain of 19.6 points, close to the 20.0-point gain from the original batch (Table 3).

Instance-level repair, which you also discuss, uses demonstrations of the discovery tasks that an agent failed. UI-Venus fails 92% of the tasks in the repaired slices (Table 6 of the paper), so this baseline already includes demonstrations for almost all of those tasks. This helps explain why it is close to slice-directed repair on the target tasks, where it is ahead by 0.8 points. Instance repair also performs better on AndroidLab; Slice2Fix performs better on the other FeatBench tasks and AndroidWorld (Table 1 of the paper). We have added the new comparisons to Appendix E.

AuthorsOpenReview ↗

Response to Reviewer 8wXS (Part 2/2)

W3: External benchmark gains

The FeatBench target tasks contain the properties addressed by the repair data. AndroidWorld and AndroidLab cover a broader set of tasks, so their overall gains need not match the gains on the repaired slices. To assess whether the external gains persist across training runs, we evaluated each of the three GUI-Owl Slice2Fix models three times on both benchmarks.

Table 4. Success (%) of the three Slice2Fix seeds of GUI-Owl on the external benchmarks.

GUI-Owl-1.5-8BAndroidWorldAndroidLab
Base64.728.3
+ Slice2Fix, training seed 1 (the run in the paper)69.333.3
+ Slice2Fix, training seed 269.337.9
+ Slice2Fix, training seed 367.236.7

All three training seeds improve over the base model: by 2.6 to 4.6 points on AndroidWorld and 5.1 to 9.7 on AndroidLab. No AndroidLab task is used in fine-tuning, so the improvement also extends to tasks outside the training data. We have revised the third contribution to report modest gains on AndroidWorld for all three agents and on AndroidLab for two, and added the seed results to Appendix E.


W4 / Q2: Engineering effort and domain supervision

For each task family, we implement a task template and a reference executor. For example, supporting deletion in an application requires code to create the instruction and initial entries, and a script to carry out the deletion. These components are then reused for the tasks in that family. The mining, confirmation, recording and fine-tuning pipeline is shared across families. We used a coding agent to write the templates and executors for all 20 families.

Operations can also share code across applications: the deletion families in Broccoli, Pro Expense and Simple Calendar reuse their operation code. The application-specific work includes setting the initial state and checking the final state, which are also required for programmatic benchmark evaluation. In the new Contacts experiment, we reused the existing deletion features and added about 550 lines for the instruction template, initial contacts, success check and self-test. The coding agent wrote the code; our work was selecting the application and operation. Contacts was used only for evaluation, so this example did not require a reference executor.

Adding a feature usually means adding an option to the generator while reusing the existing executor steps. If the feature introduces a page the executor does not yet visit, we extend the script to handle it. The applications run at fixed versions, so the executors need no maintenance between runs.

Of the 1,400 candidate repair tasks, the executors solved 1,392, and we excluded the remaining 8 (0.6%). The demonstrations are therefore supplied by these task-family scripts without requiring a stronger model. The scripts do provide domain supervision: they encode the operation and use accessibility-tree text to identify elements. The agent itself does not receive this tree, as explained below. We have added these implementation details, the exclusion rate and the distinction between a stronger teacher and domain supervision to Appendix F.


Q4: On-screen evidence in demonstrations

The executor reads the accessibility tree to locate elements and their text, but the agent is trained only on the instruction, its previous actions and the screenshot. These are the same inputs it receives at test time. To ensure that a recorded action can be justified from those inputs, we check each step against the visible screen evidence and retain a trajectory only if every step passes this check (Appendix F).

For 7 candidate tasks, the entries needed for the answer could not fit on one screen, so no trajectory could meet this requirement. These tasks are among the 8 exclusions reported above. The 448 retained GUI-Owl repair trajectories contain 8,722 steps, and every step passed the evidence check.


We hope these answers address the reviewer's concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.