- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a pipeline for generating mobile-assistant tasks by jointly composing reusable skills and user records. A Skill–Memory Graph links operations to the records needed for their inputs and constraints, then selected subgraphs are compiled into executable task specifications. The resulting 281-task benchmark is evaluated with eight LLMs through JSON tools, with the best model achieving 28.8% success.
Critical strengths
- Jointly selecting skills and user records is a useful approach to task construction. The graph explicitly connects personal information to operation inputs and valid outcomes, making the role of each record easier to inspect.
- The construction ablation addresses the main design choice. Table 3 reports higher task validity for joint selection than skill-only construction (86.7% vs. 61.7%). This is encouraging evidence that selecting memories during workflow construction helps.
Critical weaknesses
- I could not determine how the final benchmark was validated. Table 3 reports 86.7% validity on an ablation set, but the paper does not explain how invalid tasks were handled when producing the final 281 tasks. Appendix G provides schematic tool interactions rather than recorded executions. Given that 185 tasks fail for all eight models, passing reference executions would be particularly useful. Without this evidence, it is difficult to tell how much of the low success rate comes from task difficulty versus problems in the generated tasks or evaluators.
- The memory-dependence check needs stronger validation. Section 4.3 recompiles the initial state and success checks after removing a record. This shows that the record affects task generation, but does not establish that the agent needs to use it in the final task. Table 6 also includes 32 tasks with no selected memories. Comparing different tasks by memory count in Figure 4 does not resolve this concern, since their workflows and execution difficulty also differ.
Questions for authors
- How many of the final 281 tasks have a reference execution that passes the active evaluator? Were the 185 tasks failed by all models checked separately?
- What were the aggregate results of the record-removal audit? Was memory dependence also checked on the final compiled tasks without regenerating their evaluators?
- What are the evaluation denominators and numbers of runs in Table 3? For example, 35.0% SR cannot be obtained from a single binary evaluation of 52 tasks. Does SR use a different subset or average multiple runs?