acceptodds
Under review as a conference paper at ICLR 2027

Benchmarking General Mobile Assistants in Challenging Real-World Scenarios

Abstract

Graphical user interfaces provide a challenging setting for evaluating agents that must perceive, reason, and act over interaction trajectories. Existing mobile benchmarks have established reproducible Android evaluation environments, but there remains room to broaden application coverage and capture more demanding user workflows. We present GMA, a benchmark for evaluating general mobile assistants in challenging real-world scenarios across seven custom applications adapted from open-source projects and 300 tasks spanning four progressive difficulty tiers, from atomic interactions to realistic long-horizon workflows. Evaluation of nine frontier models under a shared agent setup reveals a sharp decline in task completion as execution demands increase, with agents increasingly making partial progress without completing the full workflow. We further find that preserving interaction history substantially improves performance, especially on demanding tasks, while retaining increasingly more raw context eventually yields diminishing returns. Motivated by these results, we use GMA as a controlled testbed for mobile harness design, covering context management, within-task memory, planning, and verification. Our experiments show that harness effectiveness is highly conditional. The effect of a given design varies with task demands, alternative implementations within the same module can yield substantially different outcomes, composition can diminish or even reverse marginal gains, and the same design can affect foundation models differently. Overall, GMA provides a reproducible testbed for mobile agent evaluation and a controlled setting for understanding how foundation models and their surrounding harness jointly shape reliable execution in complex mobile workflows.

Primary area:
datasets and benchmarks
Keywords:
Mobile Agent, Agent Evaluation, Agent Harness

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_AuowOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces GMA to test agents' ability to complete complex mobile tasks. The authors adapt seven open-source applications, self-host their backend services, and build 300 tasks across four tiers: Atomic, Compositional, Cross-Application, and Realistic. The system uses programmatic checks to determine whether the full task and each subtask have been completed.

The authors also compare different implementations and combinations of context management, within-task memory, planning, and verification modules. The results show that the same module can have different effects on different tasks. Combining more modules does not always bring further gains.

Critical strengths
  1. The application environment and task verification are fairly well developed.The seven applications and self-hosted backend services broaden the coverage of mobile evaluation. The initialization tools restore both the phone and backend states. The tasks have been checked through manual execution, and the system can record which subtasks have been completed. This makes it easier to compare agent performance and see which requirements remain unmet.

  2. The comparisons between modules are useful.The authors compare specific implementations under a shared setup, which helps make the differences between modules clearer. There are experiments on history retention, memory formats, planning, and verification methods. The appendix also includes results for module combinations and task trajectories. These materials help researchers judge which tasks a given module may be useful f

Critical weaknesses
  1. The reasons for differences in module performance are not fully explained.Tables 6–7 show results for different implementations and combinations, but do not fully explain why structured memory performs better overall or why performance drops with some module combinations. The appendix includes failure examples, but their link to differences in module performance is still unclear. The authors could use the existing trajectories to show which errors become less or more common when different memory methods or module combinations are used on the same task.

  2. Task tiers and execution length need to be distinguished more clearly.The four tiers involve several factors, including the applications used, task length, the number of requirements, and how spread out the information is. Appendix E.1 also groups tasks by the actual number of steps taken by the baseline, including both successful and failed runs. However, the opening of Appendix E calls this “the interaction length required for successful execution.” These two descriptions do not fully match. The actual step count depends on both the task requirements and how the agent carries out the task. These results therefore do not directly show that the differences come from the task's long-horizon dependencies. The wording should be made consistent, and the authors should explain what conclusions each analysis supports.

Questions for authors
  1. Based on the existing trajectories, what different errors occur with different memory methods and module combinations? Which changes are supported by examples, and which causes are still unclear?

  2. How does ''the interaction length required for successful execution'' at the start of Appendix E relate to the actual step counts in E.1, which include both successful and failed runs? Can existing records of manual execution help distinguish the actions required by the task from extra actions caused by agent errors?

2 comments
AuthorsOpenReview ↗

Rebuttal Part 1

We sincerely thank you for the careful reading, constructive feedback, and thoughtful suggestions. We appreciate the opportunity to clarify our contributions, address the concerns, and provide additional analyses.

W1 & Q1 Trajectory-Level Analyses of Harness Effects

We agree that detailed trajectory analysis helps explain how different harness modules affect agent behavior. Our case study examines actions, module states, and evaluation outcomes, with representative examples below.

  1. M2 keeps unresolved requirements explicit during recovery. The task requires transferring a food recommendation into an order and then sending a message. When search results remain blank, M2 keeps the unresolved search goal and downstream ordering and messaging requirements explicit. The actor retries the search and switches to the Products tab while the memory preserves the remaining workflow. The run succeeds in 44 steps, whereas the baseline repeatedly waits and reaches 150 steps with a score of 0.33. This comparison illustrates explicit task state maintenance alongside recovery from an unproductive action pattern.

  2. M2 can record procedural completion without validating the target. The task requires adding "Lego House" to a specified playlist. The actor selects an acoustic version, but M2 records the menu and playlist selection actions as complete and empties its remaining task list. The actor then declares success despite failing final verification. In contrast, the baseline selects the required song and succeeds. This case exposes a gap between tracking completed actions and establishing that they satisfy the intended requirement. Memory preserves the agent's mistaken completion state rather than correcting it.

  3. E3 preserves requirements that remain usable after execution errors. The task requires creating two same name contacts with distinct attributes. E3's summaries explicitly associate each contact with its required phone number, labels, and email. After the actor enters information into the wrong field and accidentally clears the second contact's phone number, these requirements remain available in context. The actor identifies the missing number, re-enters the correct value, and resumes the required configuration. E3 succeeds in 44 steps, while the baseline reaches 150 steps with a score of 0.50. The useful behavior is recovery with the correct requirements intact, rather than error-free execution.

  4. V1 can redirect both memory and action through an incorrect judgment. The task requires configuring the initially earliest inactive, non-repeating alarm. In an E3 + M2 + V1 run, the actor correctly selects and enables the 7:15 alarm. V1 then judges this target incorrect because it is now active and identifies the still inactive 9:30 alarm as the target. M2 replaces its remaining requirements for 7:15 with requirements for 9:30, and the actor follows this change. The run ends with a score of 0.33. Here, verification incorrectly re-evaluates an initial state selection condition after the agent changes that state, and the resulting error propagates through memory into subsequent actions.

Together, these cases show how harness modules shape execution by maintaining requirements, recording progress, and supplying corrective feedback. These mechanisms support recovery when their representations remain accurate, but can also preserve mistaken completion judgments or propagate incorrect feedback into subsequent actions. The trajectories therefore help explain why adding harness support does not necessarily improve performance.

W2 & Q2 Clarification Regarding Task Tiers and Execution Length

The four difficulty tiers are defined by task design, independently of agent execution. In contrast, Appendix E.1 groups tasks by the actual number of interaction steps taken by Qwen3.7-Plus under the baseline harness, including both successful and failed episodes. These analyses examine performance across task structures and observed baseline execution lengths, respectively. The latter reflects both task demands and agent behavior, rather than isolating inherent long-horizon dependencies.

The phrase "the interaction length required for successful execution" at the beginning of Appendix E is a wording error. The analysis uses observed baseline step counts, including both successful and failed episodes, as explicitly defined in E.1. We will correct the phrase to "the observed baseline execution length" for consistency. We again thank you for pointing this out.

Thank you again for your helpful feedback. We would appreciate any further thoughts on whether our rebuttal has addressed your concerns. If any questions remain, we would be happy to clarify them. Thank you for your time and consideration.

Reviewer_b5gZOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GMA, a benchmark for evaluating mobile agents on complex tasks, featuring seven new applications and 300 tasks organized into four difficulty levels: Atomic, Compositional, Cross-Application, and Realistic. The authors evaluate nine models under a unified setup and further investigate the effects of harness modules, including context management, memory, planning, and verification. The experiments show that task success decreases as task complexity increases, while retaining interaction history improves performance with diminishing returns. The effectiveness of harness designs also varies with task difficulty, implementation choices, module combinations, and underlying foundation models.

Critical strengths
  1. The paper provides a reproducible mobile evaluation environment with additional application scenarios, including e-commerce, travel, and food delivery. The four difficulty levels and programmatic verification support the evaluation of both complete task success and partial progress in complex workflows.

  2. The paper conducts extensive experiments, comparing different foundation models and examining various harness modules and their combinations. The results reveal how harness performance varies across implementations, task difficulties, and module configurations.

Critical weaknesses
  1. The paper aims to expand app coverage and introduce longer cross-app tasks, but these directions have already been explored by multiple previous benchamrks, such as AndroidWorld and MobileWorld. A direct comparison with existing benchmarks in terms of app coverage, the proportion of cross-app tasks, and task length distributions would help clarify the additional value of GMA.

  2. The evaluation still relies mainly on task success rate and the proportion of completed subtasks. It lacks specific metrics for issues such as information consistency across apps, forgotten constraints, and cross-step dependencies, limiting the benchmark's ability to identify failures in long-horizon workflows.

  3. The harness experiments lack stability and computational cost analysis. Some performance differences are less than one percentage point, yet no repeated runs or confidence intervals are reported. Moreover, the additional model calls and token costs introduced by different harnesses are not controlled, making it difficult to separate the effects of harness design from additional computation.

  4. The harness experiments show different results across task difficulty levels but provide limited analysis of why these differences occur. Task difficulty involves multiple factors, including execution length, the number of apps, and constraint complexity. The experiments do not isolate their effects or analyze specific failure modes to explain what each harness module actually improves.

Questions for authors

see weaknesses

5 comments
AuthorsOpenReview ↗

Rebuttal Part 1.1

We sincerely thank you for the careful reading, constructive feedback, and thoughtful suggestions. We appreciate the opportunity to clarify our contributions, address the concerns, and provide additional analyses.

W1 Comparison Against Existing Mobile Benchmarks

GMA extends existing mobile benchmarks through seven additional applications and a progressive task design. Its 300 tasks span four tiers, Atomic, Compositional, Cross-Application, and Realistic, enabling evaluation from individual operations to complex workflows within a shared environment. We summarize the differences in functional coverage and task organization below.

DimensionAndroidWorldMobileWorldGMA
Ecosystem Coverage
Productivity and personal organizationNotes, calendars, task lists, expense trackingCalendars, email, documentsCalendars, email, contacts, alarms, files
Communication and collaborationSMSSMS, email, team messagingSMS, email, team messaging, group chat and polls
Social and lifestyle content—Social posting and interactionSocial posting, lifestyle discovery, comments and collections
Media creation and managementMusic, video, images, audio recording, drawingPhotos and image managementPhotos, music libraries and playlists
Shopping—Product browsing and shopping workflowsProduct comparison, variants, checkout and order management
Local discovery and food delivery——Business discovery, reviews, restaurant browsing and food ordering
Travel booking——Flight, hotel and attraction booking
Task Structure and Difficulty
Number of tasks116201300
Task organizationNo explicit progressive difficulty tiersNo explicit progressive difficulty tiersFour explicitly defined difficulty tiers
Maximum interaction steps5050150
Mean executed steps14.327.855.8

The comparison highlights GMA's expanded coverage of everyday service workflows, particularly local discovery, food delivery, and travel booking. Its four-tier organization supports comparisons across increasing task demands within the same environment. GMA also exhibits longer execution trajectories in the reported evaluations, highlighting the sustained execution demands agents encounter on our benchmark.

W2 Additional Diagnostic Metrics

Success rate and average score serve as GMA's primary benchmark metrics, measuring complete task success and partial progress. We retain these metrics for evaluating overall model performance and agree that additional trajectory-level diagnostics can help explain information use and failure patterns. We therefore introduce:

  1. Information-Use Outcomes (D1): We report two complementary measures. D1 (a) is the mention-then-write pass rate, measuring the proportion of mention-then-write value instances whose corresponding criteria pass final verification. D1 (b) is the value-expression rate among failures, measuring the proportion of failed criteria whose required values appeared in the agent's textual output or writing actions.
  2. Transfer Distance (D2): For mention-then-write value instances, we count the interaction steps between mention and subsequent writing. D2 (a) reports the mean distance for instances whose corresponding criteria ultimately pass, while D2 (b) reports the mean distance for those whose criteria fail.

We apply these diagnostics to the main evaluation and harness ablation runs. Selected results are shown below.

RunSuccessD1 (a)D1 (b)D2 (a)D2 (b)
Main Evaluation
Qwen3.8-Max67.33%86.1%57.5%7.514.6
GPT-5.555.67%70.3%50.0%9.113.7
Claude Opus 4.758.67%75.9%57.1%20.020.2
Harness Ablation
Baseline47.33%66.7%53.6%5.79.2
E3E_{3}52.33%74.3%63.5%6.415.0
M2M_{2}48.67%67.8%62.8%6.49.3

These results reveal execution patterns beyond aggregate success. D1 (a) generally aligns with overall task success, with stronger agents more often satisfying the corresponding requirements after a mention-then-write sequence. D1 (b) further shows that even when requirements remain unmet, agents have often already expressed the required values, suggesting that failures can occur after the relevant information appears in their outputs.

D2 shows another consistent pattern across the selected runs, with shorter average transfer distances for instances whose criteria pass than for those whose criteria fail. This suggests that more timely information use is associated with successful execution, although longer intervals may also reflect difficulties encountered along the way. Together, these diagnostics connect observable information-use patterns with execution outcomes on GMA.

AuthorsOpenReview ↗

Rebuttal Part 1.2

W3 Evaluation Stability

We conduct repeated evaluations for five representative configurations from Figure 3. Given computational constraints, we perform two additional runs per configuration, yielding three runs including the original evaluation. Run 1 corresponds to the original paper results.

ConfigurationMean Success ±\pm SDMean Average ±\pm SDRun 1 SuccessRun 1 AverageRun 2 SuccessRun 2 AverageRun 3 SuccessRun 3 Average
E1E_{1}50.89 ±\pm 1.0265.32 ±\pm 1.0352.0066.4850.6764.5150.0064.96
E3E_{3}52.55 ±\pm 0.3967.88 ±\pm 0.5652.3367.2753.0068.0052.3368.38
M2M_{2}50.00 ±\pm 1.2065.66 ±\pm 0.9648.6765.1451.0066.7750.3365.07
V1V_{1}49.44 ±\pm 0.5163.92 ±\pm 0.3449.3363.6449.0064.3050.0063.81
V2V_{2}51.22 ±\pm 0.8465.50 ±\pm 0.9751.3365.5650.3364.5052.0066.44

Across the three runs, all five configurations show generally modest variation, with sample standard deviations of 0.39 to 1.20 percentage points for Success and 0.34 to 1.03 points for Average. E3 consistently outperforms E1, and V2 consistently outperforms V1, on both metrics in every run. These results support the stability of the relative performance differences between these implementations.

Given computational and time constraints, we cannot repeat every configuration and instead use these selected settings as representative cases to better understand evaluation variability. These variability estimates provide a descriptive reference for interpreting the effects reported in Figures 3 and 4. Many reported differences are larger in magnitude than the variation observed in these repeats. Smaller differences remain less certain, but our conclusions do not depend on every marginal gain or exact ranking being preserved. The consistent implementation comparisons, together with the broader experimental results, continue to support the finding that harness effectiveness depends on task demands, implementation choices, and module interactions.

W3 Cost Analyses

We agree that harness improvements should be considered alongside their computational costs. We report mean token consumption per task across all 300 tasks on harness ablation runs. The totals include input and output tokens from both the actor and all harness modules. Representative configurations are shown below.

ConfigurationSuccessAverageTokens per task
Baseline47.33%62.56562k
V1V_{1}49.33%63.64848k
V2V_{2}51.33%65.56659k
E3E_{3} + V2V_{2}53.67%68.02798k
E3E_{3} + P1P_{1} + V2V_{2}50.33%66.37806k
E3E_{3} + M2M_{2}52.67%69.04944k
E1E_{1} + M2M_{2}51.00%67.671354k

These results show that higher token consumption does not consistently yield better performance. V2 outperforms V1 on both metrics with 22% fewer tokens. Likewise, E3 + V2 and E3 + M2 achieve the highest Success and Average, respectively, while consuming substantially fewer tokens than E1 + M2. Adding P1 to E3 + V2 increases tokens by only 1% but reduces Success by 3.34 percentage points, illustrating different composition outcomes at similar observed consumption.

Together, these comparisons show that the value of additional computation depends on how the harness uses it. The strongest configurations balance performance and token consumption through effective implementations and module combinations, rather than simply using more tokens.

AuthorsOpenReview ↗

Rebuttal Part 1.3

W4 Trajectory-Level Analyses of Harness Effects

We agree that trajectory analysis helps explain both how harness modules affect agent behavior and why their benefits vary with task demands. Our case study compares actions, module states, and evaluation outcomes on the same tasks under different configurations. The following examples illustrate how these mechanisms interact with the requirements of different task tiers.

  1. M2 maintains outstanding goals in a Cross-Application workflow. The task requires transferring a food recommendation into an order and then sending a message. When search results remain blank, M2 keeps the unresolved search goal and downstream ordering and messaging requirements explicit. The actor retries the search and switches to the Products tab while the memory preserves the remaining workflow. The run succeeds in 44 steps, whereas the baseline repeatedly waits and reaches 150 steps with a score of 0.33. This comparison illustrates explicit task state maintenance alongside recovery from an unproductive action pattern.

  2. M2 does not resolve incorrect target selection in an Atomic task. The task requires adding "Lego House" to a specified playlist. The actor selects an acoustic version, but M2 records the menu and playlist selection actions as complete and empties its remaining task list. The actor then declares success despite failing final verification. In contrast, the baseline selects the required song and succeeds. This case exposes a gap between tracking completed actions and establishing that they satisfy the intended requirement. Memory preserves the agent's mistaken completion state rather than correcting it.

  3. E3 maintains distributed requirements throughout a Realistic workflow. The task requires identifying an urgent apartment repair, submitting matching evidence, arranging an access window, setting reminders, and notifying the resident. E3's early summary consolidates the issue, repair instructions, and outstanding goals from multiple sources. Later summaries retain the selected access window and completed actions while explicitly identifying the remaining alarm settings and notification. The actor completes these requirements in 99 steps with a score of 1.00, compared with 0.20 after 150 steps for the baseline. This illustrates how persistent task state supports workflows in which later actions depend on information and decisions from earlier stages.

  4. E3 supports complex coordination but does not guarantee complete execution. Another Realistic task requires reconciling approved launch documents, communicating an authorized correction, and scheduling a review around calendar conflicts. E3's summaries retain the approved version, correction, risk information, and the derived meeting and reminder times. The run achieves a score of 0.78 in 50 steps, compared with 0.11 after 150 steps for the baseline. However, the summary's remaining alarm instructions emphasize time and label, and the actor declares completion after setting these while the alarm criterion still fails. The case illustrates substantial progress across interdependent requirements alongside an incomplete final state check.

Together, these cases suggest that harness benefits depend on how their support matches the task's execution demands. Persistent state is useful when information and intermediate decisions must remain available across stages, while procedural tracking alone does not ensure correct target selection or complete outcomes. These mechanisms help interpret the tier-level differences, although the cases do not independently isolate the effects of execution length, application count, and constraint complexity.

Reviewer_ayywOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces GMA, a benchmark with seven custom applications and 300 tasks organized into four difficulty tiers, from atomic interactions to realistic cross-application workflows. Programmatic subtask verification measures complete success and partial progress. The authors evaluate nine foundation models under a shared agent setup and conduct controlled comparisons of context management, within-task memory, planning, and verification. The results show substantial degradation on demanding workflows, diminishing returns from raw history retention, and implementation- and composition-dependent effects of harness mechanisms.

Critical strengths
  • GMA combines demanding workflows with fine-grained verification, enabling analysis of partial progress and premature completion. This provides more diagnostic information than aggregate success rates alone.
  • The strongest contribution is the controlled study of harness design. Comparing concrete implementations and combinations reveals that memory, planning, and verification do not necessarily provide additive improvements. This offers useful empirical insight beyond expanding application coverage.
Critical weaknesses
  • Several reported composition effects are small relative to the benchmark size. Changes of 0.33–1.34 percentage points correspond to only a few tasks in a 300-task suite. Without repeated runs or uncertainty estimates, some particular sign reversals and diminishing-return effects may be less robust than the broader finding that harness configuration matters.
  • Most harness configurations are studied with Qwen3.7-Plus, while the additional foundation-model comparison focuses on memory. Consequently, the cross-model evidence supports a narrower conclusion than the full four-module analysis. Additional inference calls and token consumption also need to be considered when interpreting improvements.
Questions for authors
  • Which composition effects remain consistent across repeated runs, and which fall within evaluation variability?
  • What additional calls, tokens, and latency does each configuration incur? Do the conclusions persist under comparable inference budgets?
  • Beyond memory, which findings about planning, verification, or context management have been reproduced with another foundation model?
4 comments
AuthorsOpenReview ↗

Rebuttal Part 1.1

We sincerely thank you for the careful reading, constructive feedback, and thoughtful suggestions. We appreciate the opportunity to clarify our contributions, address the concerns, and provide additional analyses.

W1 & Q1 Evaluation Stability

We conduct repeated evaluations for five representative configurations from Figure 3. Given computational constraints, we perform two additional runs per configuration, yielding three runs including the original evaluation. Run 1 corresponds to the original paper results.

ConfigurationMean Success ±\pm SDMean Average ±\pm SDRun 1 SuccessRun 1 AverageRun 2 SuccessRun 2 AverageRun 3 SuccessRun 3 Average
E1E_{1}50.89 ±\pm 1.0265.32 ±\pm 1.0352.0066.4850.6764.5150.0064.96
E3E_{3}52.55 ±\pm 0.3967.88 ±\pm 0.5652.3367.2753.0068.0052.3368.38
M2M_{2}50.00 ±\pm 1.2065.66 ±\pm 0.9648.6765.1451.0066.7750.3365.07
V1V_{1}49.44 ±\pm 0.5163.92 ±\pm 0.3449.3363.6449.0064.3050.0063.81
V2V_{2}51.22 ±\pm 0.8465.50 ±\pm 0.9751.3365.5650.3364.5052.0066.44

Across the three runs, all five configurations show generally modest variation, with sample standard deviations of 0.39 to 1.20 percentage points for Success and 0.34 to 1.03 points for Average. E3 consistently outperforms E1, and V2 consistently outperforms V1, on both metrics in every run. These results support the stability of the relative performance differences between these implementations.

Given computational and time constraints, we cannot repeat every configuration and instead use these selected settings as representative cases to better understand evaluation variability. These variability estimates provide a descriptive reference for interpreting the effects reported in Figures 3 and 4. Many reported differences are larger in magnitude than the variation observed in these repeats. Smaller differences remain less certain, but our conclusions do not depend on every marginal gain or exact ranking being preserved. The consistent implementation comparisons, together with the broader experimental results, continue to support the finding that harness effectiveness depends on task demands, implementation choices, and module interactions.

W2 & Q2 Cost Analyses

We agree that harness improvements should be considered alongside their computational costs. We report mean token consumption per task across all 300 tasks on harness ablation runs. The totals include input and output tokens from both the actor and all harness modules. Representative configurations are shown below.

ConfigurationSuccessAverageTokens per task
Baseline47.33%62.56562k
V1V_{1}49.33%63.64848k
V2V_{2}51.33%65.56659k
E3E_{3} + V2V_{2}53.67%68.02798k
E3E_{3} + P1P_{1} + V2V_{2}50.33%66.37806k
E3E_{3} + M2M_{2}52.67%69.04944k
E1E_{1} + M2M_{2}51.00%67.671354k

These results show that higher token consumption does not consistently yield better performance. V2 outperforms V1 on both metrics with 22% fewer tokens. Likewise, E3 + V2 and E3 + M2 achieve the highest Success and Average, respectively, while consuming substantially fewer tokens than E1 + M2. Adding P1 to E3 + V2 increases tokens by only 1% but reduces Success by 3.34 percentage points, illustrating different composition outcomes at similar observed consumption.

Together, these comparisons show that the value of additional computation depends on how the harness uses it. The strongest configurations balance performance and token consumption through effective implementations and module combinations, rather than simply using more tokens.

AuthorsOpenReview ↗

Rebuttal Part 1.2

W2 & Q3 Cross-Model Validation of Harness Findings

Our experiments serve distinct purposes. The main multi-model evaluation measures model performance on GMA. The controlled harness study fixes Qwen3.7-Plus to examine the effects of task demands, implementation choices, and module combinations. The cross-model memory study additionally examines how these effects vary with the underlying foundation model.

Our goal is not to establish a harness ranking from Qwen3.7-Plus that applies to all models, but to understand how harness design can shape agent performance. Section 5.2 and 5.3 show that effects can vary across task demands, differ between implementations of the same module, and change under composition. These findings have implications beyond Qwen3.7-Plus. They identify task demands, implementation choices, and module interactions as key dimensions for harness evaluation, providing a framework for studying harness effectiveness across model settings rather than prescribing a universal configuration.

The memory experiments examine how the effects of the same harness designs vary with the underlying foundation model. We agree that extending this comparison to planning, verification, and context management would help distinguish patterns shared across models from model-specific effects. These additional experiments are ongoing but still incomplete due to limited computational resources. We will report the results and their implications once the experiments are complete.

Thank you for the repeated evaluations and the additional cost analysis. These results reduce my concerns.

The consistent E3/E1 and V2/V1 comparisons across three runs strengthen the evidence that implementation choices matter. My remaining concern is narrower: the repeats do not directly establish the stability of the composition effects in Figure 4, particularly the smaller marginal changes. I appreciate your acknowledgment of this uncertainty and suggest explicitly distinguishing these exploratory effects from the more consistently supported implementation comparisons in the revised paper.

The token analysis is helpful, especially because it includes the actor and all harness modules. V2 outperforming V1 with fewer tokens, and the similar consumption of E3 + V2 and E3 + P1 + V2, show that performance is not simply increasing with token usage. While these are not matched-budget experiments, they provide useful evidence about the performance–cost trade-offs. Please incorporate these results and the associated qualifications into the revision.

Thank you for clarifying the intended scope. I agree that the controlled study can provide useful insights without establishing a universal harness ranking.

The distinction between identifying useful dimensions for harness evaluation and demonstrating that specific empirical effects generalize across models is important. The current cross-model evidence supports the memory analysis; findings concerning planning, verification, and context management should remain explicitly scoped to Qwen3.7-Plus unless additional evidence becomes available. This clarification would adequately address my concern about the scope of the claims.

Considering both parts of the response, the additional evidence and clarifications strengthen the paper, and I maintain my weak-accept recommendation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.