acceptodds
Under review as a conference paper at ICLR 2027

MobileSkillBench: Automatically Constructing Memory-Dependent Mobile-Agent Tasks from Reusable Skills

Abstract

Personal mobile assistance requires agents to combine reusable operations with user information that determines which actions are appropriate. Evaluating this ability requires tasks in which personal records influence the workflow and its expected state. Existing work provides interactive environments, personalized tasks, and structured task synthesis, but constructing the dependencies between operations and user records remains a separate design problem. We present MobileSkillBench, a framework for constructing mobile-agent tasks from raw skill documents and de-identified user memories. A Skill–Memory Graph represents operation dependencies and the records that supply inputs, initialize application data, or constrain valid results. Joint Skill-Memory Search (JSMS) builds candidate subgraphs containing skill workflows and records from one user, then balances candidate quality against repeated use of the same graph resources. Auditable Task Compilation translates each selected subgraph into an instruction, initial state, expected state, and evaluator, with source evidence linking these components. Controlled recompilation checks whether removing a record changes task requirements or leaves essential information unresolved. The resulting MobileSkillBench comprises 281 tasks that evaluate LLM agents’ ability to use user memory to complete complex, multi-step tasks. Across eight current LLM agents, the best model achieves only 28.8% strict success and 67.9% Completion, with strict success falling to 10.9% on multi-app tasks. These results reveal substantial room for improvement in personalized mobile assistance.

Primary area:
datasets and benchmarks
Keywords:
LLM-based Mobile Agents, Benchmark Construction, Skill Composition, User Memory, Automated Task Synthesis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nSMnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a pipeline for generating mobile-assistant tasks by jointly composing reusable skills and user records. A Skill–Memory Graph links operations to the records needed for their inputs and constraints, then selected subgraphs are compiled into executable task specifications. The resulting 281-task benchmark is evaluated with eight LLMs through JSON tools, with the best model achieving 28.8% success.

Critical strengths
  1. Jointly selecting skills and user records is a useful approach to task construction. The graph explicitly connects personal information to operation inputs and valid outcomes, making the role of each record easier to inspect.
  2. The construction ablation addresses the main design choice. Table 3 reports higher task validity for joint selection than skill-only construction (86.7% vs. 61.7%). This is encouraging evidence that selecting memories during workflow construction helps.
Critical weaknesses
  1. I could not determine how the final benchmark was validated. Table 3 reports 86.7% validity on an ablation set, but the paper does not explain how invalid tasks were handled when producing the final 281 tasks. Appendix G provides schematic tool interactions rather than recorded executions. Given that 185 tasks fail for all eight models, passing reference executions would be particularly useful. Without this evidence, it is difficult to tell how much of the low success rate comes from task difficulty versus problems in the generated tasks or evaluators.
  2. The memory-dependence check needs stronger validation. Section 4.3 recompiles the initial state and success checks after removing a record. This shows that the record affects task generation, but does not establish that the agent needs to use it in the final task. Table 6 also includes 32 tasks with no selected memories. Comparing different tasks by memory count in Figure 4 does not resolve this concern, since their workflows and execution difficulty also differ.
Questions for authors
  1. How many of the final 281 tasks have a reference execution that passes the active evaluator? Were the 185 tasks failed by all models checked separately?
  2. What were the aggregate results of the record-removal audit? Was memory dependence also checked on the final compiled tasks without regenerating their evaluators?
  3. What are the evaluation denominators and numbers of runs in Table 3? For example, 35.0% SR cannot be obtained from a single binary evaluation of 52 tasks. Does SR use a different subset or average multiple runs?
Reviewer_UgomOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MobileSkillBench, a framework for constructing personalized agent tasks from reusable skills and user records. It introduces a Skill–Memory Graph, a joint search procedure for selecting workflows and supporting records, and a compilation stage that produces task instructions, initial states, and evaluation criteria. The resulting benchmark contains 281 tasks evaluated through JSON tool interactions. Experiments with eight LLM backbones report a best strict success rate of 28.8%.

Critical strengths
  1. The paper addresses a relevant problem: evaluating whether agents can carry user-specific information and constraints through multi-step workflows.
  2. Explicitly connecting records to operation inputs, initial states, and outcome constraints is a useful task-construction design. The separation between strict success and partial completion also provides useful descriptive information.
Critical weaknesses
  1. The executable environment and final evaluator are insufficiently documented and validated. Appendix G explicitly describes the initial state, tool schema, and interaction examples as schematic, while Listing 5 comes from an earlier evaluation-plan format rather than the final runtime policy. The manuscript reports 1,676 fine-grained evaluation conditions and 474 active final-state scoring points, but does not establish a sufficiently concrete mapping between them. Consequently, readers cannot verify whether the reported scores reliably capture all requested outcomes. A complete measured episode, the exact active checks, and evaluator agreement with independent human judgments would substantially strengthen the benchmark.

  2. The evidence for memory dependence is incomplete. Table 6 reports that 32 of the 281 tasks contain zero memory records, which needs reconciliation with the benchmark’s central framing. More generally, removing a record and recompiling both the initial state and success checks establishes sensitivity of the construction procedure; it does not independently establish that the original task requires the intended memory reasoning. The experiments lack controls that separate memory access and integration from ordinary tool execution, such as oracle provision of relevant records and counterfactual changes to record values. The performance differences across memory-count groups are also potentially confounded by workflow length and task composition.

  3. The ablations do not adequately isolate the proposed method’s contributions. Table 3 compares different generated task sets containing 52, 37, 41, and 38 tasks. Differences in downstream success can therefore reflect differences in task difficulty and composition. The reported validity advantage is potentially useful, but the paper does not specify the human-review sample sizes, annotation procedure, or inter-annotator agreement. The full method’s 86.7% validity also leaves an important question about filtering and validation of the final benchmark. Separate ablations of joint search, diversity selection, and record-removal checking would make the methodological contribution more convincing.

  4. The scope of the evaluation and benchmark comparisons needs clearer qualification. Section 4.3 evaluates agents through JSON tools rather than a graphical phone interface. API-based evaluation is a reasonable choice, as demonstrated by AppWorld, but the resulting evidence concerns structured tool execution in mobile-themed scenarios. Table 1 compares app coverage and task structure against mobile GUI benchmarks without clearly distinguishing these interaction settings; the paper’s own coverage is described as 67 primary-app labels. The implementation and fidelity of these app environments need to be specified before interpreting these counts as comparable coverage.

Questions for authors
  1. Can you provide a complete recorded evaluation episode, including actual runtime tool definitions, state transitions, active checks, and the calculation of SR and Completion? How were evaluator errors assessed?
  2. Why do 32 tasks have no memory records, and what are the results on the memory-dependent subset alone? What fraction of selected records was removed by the necessity check?
  3. For Table 3, what are the generation, human-review, and agent-evaluation denominators? Were invalid tasks filtered before agent evaluation, and how was the final 281-task set validated?
  4. Can you report controlled results with oracle access to relevant memories and with counterfactually modified records, while accounting for workflow length?
Reviewer_bsRiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents MobileSkillBench, a framework for automatically constructing mobile-agent tasks from reusable skill documents and user records. It builds a Skill–Memory Graph, jointly selects compatible workflows and supporting records, and compiles them into instructions, initial states, target conditions, and evaluators. The resulting benchmark contains 281 tasks evaluated through JSON tools, with the best of eight evaluated LLMs achieving only 28.8% strict success.

Critical strengths
  • This paper explores how to construct tasks that integrate personal information with multi-step execution processes, addressing a core requirement for personalized intelligent assistants: the ability to perform operations tailored to a user’s specific context and constraints.

  • The Skill–Memory Graph distinguishes between operational dependencies and various auxiliary records, enhances the transparency and auditability of how user information influences task execution.

  • The experiments evaluate not only complete success but also provide granular analysis across dimensions such as application scenarios, skill types, and the number of memory entries. These detailed perspectives offer deeper insights into the challenges and bottlenecks agents encounter when executing the constructed workflows.

Critical weaknesses
  • Local dependency compatibility does not guarantee the executability of an entire workflow. The process of constructing and searching the graph relies primarily on pairwise skill dependencies, ordering constraints, and memory footprint considerations. However, these conditions do not necessarily ensure that all prerequisites remain satisfied throughout the entire execution: prior operations might disrupt the state required for subsequent steps, or dependencies that are individually compatible could introduce conflicting requirements when combined. Although the paper acknowledges that execution validation is necessary, it does not sufficiently explain how such global inconsistencies are detected and resolved before tasks enter the benchmark.
  • Task validity and evaluator correctness lack sufficient independent validation. Without stronger validation, it is difficult to determine whether the low success rates reflect agent limitations or defects in the generated tasks and evaluators. I would particularly like to see a human validity audit of the final task set, the execution success rate of reference solutions, and the evaluator’s acceptance and rejection rates on independently verified correct and incorrect trajectories. These checks would provide more direct evidence that the benchmark tasks are solvable and their outcomes are scored reliably.
  • Experiments fail to adequately distinguish between "memory-related difficulty" and "execution-related difficulty." Performance differences observed across groups with varying numbers of memory items (Fig. 4c–d) and between single-application and multi-application tasks (Table 2) may also reflect variations in workflow length, tool complexity, and the number of required checks. Consequently, these comparisons cannot isolate the specific roles that "memory retrieval" or "information integration" play in causing task failure. Designing matched task variants would yield more robust evidence. For example, by providing explicit information or implementing "oracle-memory" or "oracle-plan" conditions.
Questions for authors
  • Since the record-removal procedure recompiles both the initial state and success checks, how do the authors distinguish memory dependence from variation introduced by recompilation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.