acceptodds
Under review as a conference paper at ICLR 2027

OS-Wizard: Scaling Computer-Using Agents to Specialized Software via Knowledge-Guided Exploration

Abstract

Computer-using agents are increasingly capable of performing complex tasks in the digital world. However, high-quality interaction trajectories remain scarce for specialized desktop applications, where limited application knowledge, cumbersome GUI workflows, and incomplete exploration hinder automated collection. We introduce OS-Wizard, an automated trajectory collection framework. It combines tutorial-guided exploration with a hybrid GUI–Code action space, providing application-specific knowledge and simplifying GUI workflows through direct execution of application-native commands. The framework also checkpoints and revisits pivotal states during tree exploration to explore alternative workflows, complementing linear rollouts for broader coverage. We collect 10.9K trajectories comprising 54.9K state–action samples. Fine-tuning open-source base models on this corpus yields a best-performing model that improves performance on a benchmark of specialized desktop workflows by 58.2 percentage points to 64.1%, surpassing reported human performance of 59.8% and establishing a new state of the art. The fine-tuned model also improves performance on general desktop workflows using GUI actions only, suggesting that the collected supervision benefits general desktop tasks as well. Comparative experiments and ablations demonstrate the efficiency of trajectory collection and the importance of code actions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Computer-use Agent, Data Synthesis, Vision-Language Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_dRLdOpenReview ↗
Rating
2(+1)
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

OS-Wizard proposed an automated pipeline to collect trajectories for CUA in specialized desktop apps. It has three main contributions. First, it designed a GUI-code action space where application-native commands are dealt as a single actions. Second, it uses tutorial guidance to help agents do the action sampling and synthesizing instructions. Third, it proposes a two-stage tree search for exploration. It also collected data for training. And this method shows great performance compared with existing work.

Critical strengths
  1. Large-scale dataset is useful for the CUA research community.
  2. The proposed infra is well-documented.
Critical weaknesses
  1. The attribution of gains is not clear.
  2. Some claims like "new SOTA" and "surpass human" are not established since some headline comparisons are not like-for-like.
Questions for authors
  1. Could you elaborate more on the "OOD exemplar instructions"? Like the source.
  2. What is the total cost for collecting the dataset? Such as API and VM hours.
  3. Better ablation study is needed. For example, you need to do that for the tutorials and pivot scoring.
1 comment
AuthorsOpenReview ↗

Response to Reviewer dRLd

Thank you for reviewing our work and sharing your suggestions!

W1: Attribution of performance improvements

Thank you for your question. In the introduction (L83–103), we describe how OS-Wizard collects trajectories for specialized software: tutorials and domain-specific code provide specialized knowledge and operations (L263–282), pivotal-state scoring identifies valuable states and expands their continuations (L196–210), and tree exploration is combined with linear rollouts to broaden trajectory coverage (L214–258). We demonstrate the improvements contributed by these components through the corresponding experiments. We would be happy to discuss any questions you may have about these details.

W2: Clarification of “new SOTA” and “surpassing human performance”

Our statement “surpassing reported human performance of 59.8% and establishing a new state of the art” refers to the factual comparison that OS-Wizard-Qwen3.5-27B achieves higher accuracy on ScienceBoard than the accuracies reported in the two papers for the current best model, SCALECUA-Qwen3.5-9B, and for human performance on ScienceBoard. We do not make claims such as “our method is superior to SCALECUA.”

This factual comparison is based on the accuracies of OS-Wizard, SCALECUA, and the ScienceBoard human evaluation being obtained using the same evaluation scripts. Both OS-Wizard and the ScienceBoard human evaluation use the official ScienceBoard evaluation scripts. For SCALECUA, the public materials we reviewed do not report modifications to the evaluation scripts.

The claim we make in the paper is “outperforming alternative collection strategies under a matched training-data budget.” This conclusion follows from our comparison of OS-Wizard with two other data collection frameworks of the same kind, OS-Genesis and ANCHOR, under matched training-sample budgets. We would be happy to discuss any further questions you may have about the details.

Q1: Details of the OOD exemplars

The OOD exemplars consist of a small set of instructions synthesized using GPT and manually checked for decontamination to ensure that their content does not overlap with the dataset. We aim for GPT to generate task instructions that incorporate domain knowledge, for example:

  • Open EMDB map 1080 and set its volume contour level to 0.9 in ChimeraX.

Rather than GUI operation recipes such as "click a button and then adjust a parameter," for example:

  • Apply a white background in ChimeraX by clicking the White background toolbar button.

Q2: Total cost of data collection

We computed the total monetary and time costs of OS-Wizard's tree exploration and linear rollouts separately. We also computed the costs of OS-Genesis and ANCHOR. The results are shown below:

MethodMean inputMean outputMean timeTotal timeTotal cost
TREE13,531.23225.4476.14 s1131.7 hUSD 2131.87
LINEAR12,895.09278.0047.77 s301.1 hUSD 926.93
OS-Genesis28,815.78516.8583.21 s243.0 hUSD 943.93
ANCHOR26,160.40651.6055.22 s176.3 hUSD 966.98

Overall, our method has lower mean token consumption than both baselines and lower mean time consumption than OS-Genesis.

Thank you for pointing out these omissions in our writing. We will add these details in a new appendix subsection.

Q3: Ablation experiments

Thank you for pointing out this issue. We collected data from five scientific applications under four conditions: (1) the standard configuration, (2) without tutorials, (3) without the pivot mechanism, and (4) with random pivot selection. We proportionally sampled 5K state–action examples for each condition for comparison. Evaluation after training yielded the following results:

ModelAstroBiochemGISAlgDocOverall w/o ATP
w/o Tutorial6.117.22.925.812.512.6
w/o Pivots15.244.826.522.612.525.2
w/ Random Pivots18.241.417.635.512.525.9
5K w/o Lean21.244.823.541.918.830.8

These results show that, with all other conditions unchanged, modifying only one component of OS-Wizard reduces model performance in each case, indicating that our performance improvements can be attributed to these factors.

Reviewer_xWNHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces OS-Wizard, an automated trajectory collection framework for adapting computer-using agents to specialized desktop applications. The framework combines three main components: (1) tutorial-guided exploration to provide application-specific knowledge, (2) a two-stage exploration strategy that identifies pivotal states and combines tree-based exploration with continuous linear rollouts, and (3) a hybrid GUI-Code action space that augments standard GUI interactions with application-specific command or file-editing primitives. Using OS-Wizard, the authors have collected 10,919 trajectories comprising 54,938 state-action training samples across 13 applications, including six specialized applications. Full-parameter supervised fine-tuning substantially improves OS-Wizard-Qwen3-VL-8B and OS-Wizard-Qwen3.5-27B on ScienceBoard. In particular, OS-Wizard-Qwen3.5-27B improves from 5.9% to 64.1% overall success. The paper further conducts a series of analyses to evaluate the effectiveness of the proposed data collection framework and its individual components. Under a matched 10K-sample budget, OS-Wizard-Qwen3.5-27B-10K also outperforms Task-driven and Branch-driven collection baselines (36.5% vs. 13.0% and 18.9%). The paper further reports improvements on OSWorld and provides ablations on code supervision, tree/linear collection strategies, data scaling, and state-recovery efficiency. Overall, I find the empirical improvements substantial and the system carefully engineered. However, I have several concerns regarding the novelty of the central methodological contribution relative to prior work and to the ScienceBoard environment itself, the comparability of the headline comparisons with the previous best result, with human performance, and with the re-implemented baselines, confounds in the component ablations that limit the attribution of the gains, the lack of direct evaluation of the proposed pivotal-state selection mechanism, and the insufficient documentation of how the collected training data are separated from the evaluation tasks.

Critical strengths

S1. Strong improvements on specialized applications The empirical gains on ScienceBoard are substantial. OS-Wizard-Qwen3.5-27B improves from 5.9% to 64.1%, while OS-Wizard-Qwen3-VL-8B improves from 6.5% to 46.6%. The matched-data comparison is also encouraging: with the same 10K training-sample budget, OS-Wizard-Qwen3.5-27B-10K obtains 36.5%, compared with 13.0% for Task-driven collection and 18.9% for Branch-driven collection. At the level of the full pipeline, this suggests that OS-Wizard produces more useful supervision per training sample than the compared collection pipelines as implemented, although the implementation of these baselines needs to be documented more fully (see M2). S2. A complete and practical pipeline for automated trajectory collection The paper addresses several practical bottlenecks in collecting trajectories from specialized software, including limited application-specific knowledge, inefficient branching, long replay paths, non-deterministic GUI recovery, and brittle GUI-only workflows. The combination of tutorial retrieval, pivotal checkpoints, tree/linear exploration, execution verification, and VM-based state recovery constitutes a relatively complete data-generation system rather than an isolated algorithmic component. S3. Comprehensive ablations and scaling analyses The paper reports a broad set of ablations and scaling analyses under matched sample budgets. Under a matched 20K-sample budget, mixed GUI-Code supervision achieves 55.0%, compared with 26.6% without code supervision. Tree-only and linear-only data achieve 41.4% and 30.7%, respectively, while their combination reaches 55.0%. The data-scaling results further suggest that performance improves as more OS-Wizard supervision is added, and the model-scaling results show that larger backbones achieve higher success at every non-zero data budget. These analyses are informative, although the ablated training sets also differ in application composition, which limits how directly the gaps can be attributed to individual components (see M3). S4. Good implementation transparency The appendix provides unusually detailed descriptions of the model-facing action space, application-specific CODE primitives, VM checkpoint implementation, dataset statistics, training setup, and generation/verification prompts. These details substantially improve reproducibility and make it possible to understand what the agent actually observes and executes.

Critical weaknesses

M1. The methodological novelty and distinction relative to prior work needs to be articulated more clearly OS-Wizard integrates tutorial-guided exploration, pivotal-state-based tree exploration, and a hybrid GUI-Code action space into an effective trajectory collection framework. However, the individual ingredients of OS-Wizard have substantial conceptual overlap with prior work on computer-use agents and trajectory synthesis. In particular, hybrid GUI-Code interactions have been explored by systems such as CoAct/ActionEngine, UltraCUA/UFO²/Axis, and ToolCUA/EE-MCP. Tutorial- or document-guided trajectory generation has also appeared in prior work, such as AgentTrek/TongUI/Video2GUI, while exploration-based methods such as OS-Genesis and ANCHOR have already explored task-driven and branch-driven trajectory collection. In particular, ANCHOR’s use of meaningful branch states appears conceptually close to OS-Wizard’s pivotal-state-based branching strategy. More importantly, the hybrid action space is not specific to OS-Wizard at the level of the environment. The original ScienceBoard benchmark already allows agents to use application-specific CLI or scripting mechanisms in addition to GUI actions, and 131 of its 169 tasks are categorized as CLI-only (33) or GUI+CLI (98) tasks. The paper nevertheless describes the hybrid action space as “the key design of OS-Wizard” (Section 3.3). As far as I can tell from Appendix A, the authors’ own contribution at this level consists of a set of adapters, such as displaying GRASS GIS commands and their output in the console, adding a calc endpoint to the KAlgebra server, and providing file-editing primitives for Lean 4 and TeXstudio. The paper should state explicitly which parts of the action space are inherited from ScienceBoard and which are new. Similarly, the tree exploration is shallower than the framing suggests. With f(d) = max{10 - 3d, 1}, branching only occurs within the first three steps, after which every branch becomes a single instruction-following rollout, and the resulting trajectories have a mean length of 5.03 steps and a median of 4. In effect, the procedure explores a short action prefix and then synthesizes an instruction from the reached state, which is close in spirit to the reverse task synthesis of OS-Genesis and the branch-point expansion of ANCHOR. A clearer statement of what the tree structure adds beyond these methods would strengthen the novelty claim. The current results convincingly demonstrate that the overall OS-Wizard pipeline performs better than the implemented baselines. However, this comparison does not fully isolate the methodological contribution of the proposed knowledge-guided/pivotal-state exploration strategy, since OS-Wizard simultaneously differs in tutorial guidance, branch selection, and the hybrid GUI-Code action space. Consequently, it remains unclear which component constitutes the primary methodological advance and to what extent the substantial downstream gains can be specifically attributed to the proposed exploration strategy. I therefore encourage the authors to clarify more precisely: • What is the primary methodological contribution of OS-Wizard compared with prior work, particularly given that tutorial-guided exploration, branch/state-based exploration, and hybrid GUI-Code interaction have all been explored in related studies? • Which parts of the hybrid action space and of the tree exploration are new relative to ScienceBoard’s native interfaces and to the branching and task-synthesis procedures of OS-Genesis and ANCHOR? M2. Several headline comparisons are not made under matched or documented conditions Several of the paper’s main claims rest on comparisons whose conditions differ in ways that are not stated in the paper. (1) Comparison with the previous best result. The paper claims a new state of the art by comparing 64.1% with the 54.0% reported for SCALECUA-Qwen3.5-9B. However, the two numbers do not appear to be measured on the same task set or through the same interface. The corresponding paper (Lv et al., 2026) evaluates ScienceBoard with mouse/keyboard actions plus a few auxiliary actions (e.g., answer and write_file), rather than with application-native command interfaces. Moreover, its per-domain results appear to be consistent only with 163 rather than 169 tasks: with four rollouts per task, 19.0% on ATP corresponds to 16/84 (i.e., 21 tasks), 86.7% on Doc corresponds to 52/60 (i.e., 15 tasks), and the six domains together give 352/652 = 54.0%, whereas neither the ATP nor the Doc value can be obtained with 26 ATP and 16 Doc tasks. If this reading is correct, the 10.1-point margin compares different task sets and different action interfaces, and the state-of-the-art claim should be qualified accordingly, ideally by also reporting OS-Wizard on the same task subset. (2) Comparison with human performance. The paper reports 64.1%, compared with the human score of 59.8% attributed to the original ScienceBoard paper, and the abstract states that OS-Wizard surpasses reported human performance. Two points need clarification. First, the original paper reports an overall human score of 60.27%, which is the unweighted mean of the six domains, whereas 59.8% corresponds to a task-weighted recomputation (101/169); recomputing for consistency is reasonable, but Table 1 marks the number as reported in the original paper. Second, and more importantly, the agent is evaluated through the authors’ modified interface (Appendix A), and it is unclear whether the human participants had access to command-based tools comparable to this interface, or to a comparable step budget. The current experiments also do not isolate how much of the margin is attributable specifically to tutorial-guided exploration. Therefore, the result should not be interpreted as evidence that OS-Wizard surpasses human performance under comparable conditions, nor that knowledge-guided exploration itself enables it to do so. (3) Re-implemented baselines. The Task-driven and Branch-driven baselines are described only as being implemented “following OS-Genesis and ANCHOR.” The paper does not state whether these baselines had access to the CODE actions, whether they used retrieved tutorials, which models served as instruction proposers, action samplers, and verifiers, which backbone was fine-tuned, or whether their 10K samples cover the same applications in the same proportions as the OS-Wizard-10K subset. Since both the action space and the application mix appear to affect downstream performance substantially (see M3), these details determine whether the 36.5% vs. 13.0%/18.9% comparison reflects the exploration strategy itself. In addition, the “Human Data” baseline (9.5%) is not described anywhere in the paper; its source, size, and collection protocol should be reported. Finally, the baseline and ablation results are reported as single numbers, whereas the OS-Wizard rows report standard deviations over three evaluation runs; these standard deviations reflect evaluation variance only and do not capture variance across training runs. (4) Comparison with the data-collection model. The paper notes that OS-Wizard-Qwen3-VL-8B surpasses GPT-5.6-Sol (46.6% vs. 42.6%), “the strongest model used for data collection.” During collection, however, GPT-5.6-Sol operates with retrieved tutorials, reward-model filtering, and repeated sampling, whereas it is evaluated as a plain agent without these aids. A baseline that provides the same retrieved tutorials at inference time, both to GPT-5.6-Sol and to the base Qwen models, would help separate the value of collecting and training on trajectories from the value of simply giving the agent access to application knowledge. M3. The component ablations confound the ablated factor with application composition The ablations in Figure 4(b) are central to the paper’s attribution of the gains, but in both of them the ablated factor changes together with the application mix of the training data. (1) Code supervision. The w/o-code variant is built by randomly sampling 20K samples from trajectories that contain no code calls at all. According to Appendix C, Celestia (7,401 samples, GUI-only) and the seven general-purpose applications (17,442 samples, for which Table 3 defines no CODE primitives) already provide 24,843 code-free samples, whereas all 7,546 Lean samples consist of file-editing CODE actions and are therefore excluded. Unless the sampling was stratified by application, the w/o-code training set is likely dominated by general-purpose and Celestia data, contains no theorem-proving data at all, and contains comparatively little data for ChimeraX, GRASS GIS, KAlgebra, and TeXstudio. The 55.0% vs. 26.6% gap may therefore reflect missing in-domain data as much as the absence of code supervision. It is also worth noting that SCALECUA-Qwen3.5-9B reaches 54.0% on ScienceBoard with a GUI-centric action space (albeit on a slightly different task set, see M2), which suggests that GUI-centric supervision is not inherently limited to the 26.6% level. (2) Tree exploration vs. linear rollouts. Lean data are collected exclusively through linear rollouts (Section 4.1), so the “w/o linear rollouts” variant contains no theorem-proving data, and the two variants may also differ in their application mix more generally. The 41.4% vs. 30.7% comparison therefore does not cleanly isolate the collection strategy either. I encourage the authors to (i) report the per-application composition of every ablated training set, (ii) report per-domain results for the ablations, and (iii) ideally repeat the code ablation with the application mix held fixed, for example by re-collecting GUI-only trajectories for the same applications, or by sampling code-free and code-containing trajectories within each application in matched proportions. M4. The contribution of tutorial knowledge is not directly established The hybrid GUI-Code action space is useful and can make many tasks easier by reducing the number of interaction steps and avoiding complex GUI operations. Although CODE actions may not provide additional functionality beyond what can be achieved through the GUI, they provide a higher-level and more direct way to complete many operations. The ablation reports a very large gap for code supervision (55.0% with mixed GUI-Code supervision vs. 26.6% without code supervision), although, as discussed in M3, this gap is partly confounded by application composition. Given that “Knowledge-Guided Exploration” is the central framing of the paper, it is important to clarify how much of the performance improvement can be attributed specifically to tutorial-guided exploration, beyond the substantial benefit provided by CODE abstraction and supervision. The current no-document experiment shows that tutorial guidance improves data-collection efficiency: without documentation, 75.71% of branches reach the maximum depth (D=16) without completion on KAlgebra and GRASS GIS. However, the paper does not report the corresponding rate with tutorials, so even the size of the efficiency gain cannot be assessed. Moreover, this result does not directly show whether tutorial-guided trajectories provide higher-quality downstream supervision under the same amount of usable training data. A matched-data ablation comparing models trained on data collected with and without tutorial guidance, while keeping the GUI-Code action space unchanged, would provide more direct evidence of the contribution of tutorial knowledge. Such a comparison would also clarify to what extent the performance gains arise specifically from knowledge-guided exploration rather than from the richer hybrid GUI-Code action space. M5. The proposed pivotal-state scoring mechanism is not directly validated A central methodological component is the pivotal-state score S(v) = λd Sd(v) + λu Su(v) + λp Sp(v), followed by selecting the top 20% of candidate states. However, the paper does not appear to provide a direct ablation demonstrating that this scoring rule selects better branch points than simpler alternatives. (1) Lack of direct validation. The existing tree-vs.-linear experiment shows Tree-only = 41.4% and Linear-only = 30.7%. This establishes that tree-generated supervision is useful, but it does not establish that the proposed pivotal-state scoring is responsible for that benefit. (2) Heuristic design choices. Several design choices also appear heuristic, including the combination of depth, transition magnitude, and empirical action success; the λ weights; selecting the top 20%; and the branching schedule 10→7→4→1. (3) The scoring rule appears to be largely inactive in the actual procedure. According to Section 3.2, every first-level state forms a singleton candidate set and is retained as a pivot, so the score is only used to choose 2 pivots within each group of ten second-level siblings. Within such a group, all candidates have the same depth, so the depth prior Sd is constant and its min-max normalization is degenerate; moreover, Sp is used only “when such statistics are available,” which is unlikely for freshly generated states. In practice, pivot selection therefore seems to reduce to ranking siblings by the transition magnitude Su alone. The expected count of 187.6 branches per worker in Section 5.3 further implies that being a pivot mainly changes how many valid children are collected from each second-level state (4 for a pivot vs. about 2.15 in expectation for a non-pivot at the 53.8% validation rate). The authors should clarify whether this reading is correct and, if so, describe the mechanism accordingly. (4) Limited contribution to recovery efficiency. The recovery analysis shows that pivotal checkpoints reduce the average replay overhead only from 15.7 to 14.2, whereas continuous single-branch execution further reduces it to 2.7. This suggests that most of the reported improvement in recovery efficiency comes from continuous execution rather than pivotal checkpointing itself. (5) More direct comparisons are needed. The authors could compare the proposed pivotal-state selection with simpler alternatives, such as random selection or individual scoring components, under the same experimental conditions. A comparison with an ANCHOR-style branch-selection method would further clarify the advantage over prior work.

M6. The separation between the collected training data and the evaluation tasks is insufficiently documented (1) OOD exemplar instructions. When the branching factor reaches f(d)=1, the method uses retrieved tutorials together with 5-10 OOD exemplar instructions to synthesize the task instruction for the subsequent rollout. However, I could not find a clear description of where these exemplar instructions come from, what ‘OOD’ means operationally, whether they originate from ScienceBoard or another benchmark/dataset, how they are selected, and how overlap with ScienceBoard evaluation instructions is prevented. The instruction-proposal prompt in Appendix F.1 only mentions an optional field containing “reference instructions,” without stating their provenance. This is important because the exemplars directly influence the distribution of automatically synthesized training tasks. Without knowing their provenance, it is difficult to rule out benchmark-specific instruction leakage or substantial semantic overlap between training and evaluation tasks. I am not claiming that leakage occurs, but the current description does not provide enough information to exclude it. (2) Initial states and task resources. The paper does not describe the root states from which exploration starts, nor the files, datasets, and application configurations loaded in the collection environments (e.g., the molecular structures opened in ChimeraX, or the GRASS GIS locations and mapsets). If these coincide with the initial states or resources of ScienceBoard evaluation tasks, the synthesized training tasks may be much closer to the test tasks than the phrase “scaling to specialized software” suggests. Relatedly, the six specialized applications used for collection are exactly the six ScienceBoard applications, so the paper does not test whether the framework or the collected supervision transfers to specialized software outside the benchmark. (3) Lean problems. The 1,608 Lean problems are taken from the Formal Math Autoformalization Dataset. Given that the ATP domain improves from 0% to 21.8%, the authors should check and report whether any of these theorems, or close variants of them, coincide with the theorem-proving tasks in ScienceBoard. (4) General-purpose applications and OSWorld. Similarly, it would help to state whether the 17,442 general-purpose samples were collected in OSWorld’s own VM image and task resources, and whether any synthesized instructions overlap with OSWorld tasks, since this affects the interpretation of the OSWorld improvement (see also Minor Comment 2). The authors should report the exact source and construction procedure for these exemplars, define the relevant in-/out-of-distribution boundary, and ideally report both exact and semantic overlap checks between synthesized training instructions, exemplar instructions, and ScienceBoard evaluation tasks. Analogous checks for the initial states, the Lean problems, and the OSWorld tasks would make the evaluation considerably more convincing.

Questions for authors

Q1. Did the Task-driven and Branch-driven baselines have access to the CODE actions and to retrieved tutorials? Which models were used to collect and verify their data, which backbone was fine-tuned, and what is their application composition? Q2. What are the source, size, and collection protocol of the “Human Data” baseline? Q3. Was SCALECUA-Qwen3.5-9B evaluated on all 169 ScienceBoard tasks? If not, how does OS-Wizard perform on the same task subset? Q4. In the original ScienceBoard human study, did participants have access to command-line interfaces comparable to the CODE actions, and under what step or time budget? Q5. Where do the 5-10 OOD exemplar instructions and the “reference instructions” in Appendix F.1 come from? Which root states, files, and datasets were used during collection, and were exact and semantic overlap checks against ScienceBoard and OSWorld tasks performed?

4 comments
AuthorsOpenReview ↗

Response to Reviewer xWNH (1/4)

Thank you for your thoughtful review and detailed suggestions!

M1: Methodological novelty and distinctions from prior work

(1) ScienceBoard counts an application's native CLI input field as CLI interaction, but this is not any of the code primitives we provide in OS-Wizard. Moreover, OS-Wizard does more than simply add adapters to ScienceBoard: models generally lack knowledge of the DSL syntax associated with these interfaces. We also need tutorials to help models learn the domain-specific syntax of these applications and improve their ability to operate specialized software.

(2) Regarding branch-based exploration, ANCHOR's main idea is to identify pivotal points in offline seed trajectories and branch from the main trajectory, whereas our method directly grows a complete tree from the root in a dynamic environment. Although both methods share the idea of branching from pivotal states, we believe they differ substantially in other respects.

(3) Regarding the actual tree depth, after its initial branching exploration, OS-Wizard generates an instruction based on the current VM state and the supplied documents, and then switches to linear exploration. In contrast, OS-Genesis synthesizes instructions retrospectively from trajectories after exploration has finished. This is a clear distinction between the two methods. We also believe that mean trajectory length cannot be directly equated with exploration capability, because code can directly perform complex operations and reduce the number of interactions needed to complete a task. For example, many ScienceBoard tasks can be completed with a single code submission, illustrating that the number of GUI interaction steps is not a sufficient measure of functional coverage.

We discuss the tutorial and pivotal-state ablations further in M4 and M5 below.

AuthorsOpenReview ↗

Response to Reviewer xWNH (2/4)

M2 & Q1–Q4: Comparisons with the baselines

Q1: We made the following efforts to ensure a fair comparison between our method and the baseline methods.

We reimplemented OS-Genesis and ANCHOR within the OS-Wizard framework following the methods described in their respective papers. Since neither method uses code actions or tutorials, we did not provide these resources, to remain consistent with the original papers.

The collection programs produce trajectories in the same format as the OS-Wizard framework, allowing them to be converted into the same Qwen-style trajectories. During collection, GPT-5.6-Sol serves as the instruction proposer and action sampler. GPT-5.6-Sol also serves as the reward model and as ANCHOR's seed verifier and branch selector.

We use the same model for training (Qwen3.5-27B), the same training hyperparameters (see Appendix F), and the same scientific-application proportions as OS-Wizard, based on the state–action sample proportions in the full dataset. We also use the same prompt template during evaluation (see Appendix H). We do not provide the optional APIs or their descriptions during evaluation, because code actions were not included in the training action space.

Thank you for pointing out these omissions in our writing. We will add these details in a new appendix subsection.

Q2: For the human data, we worked with several collaborators, including qualified master's and doctoral supervisors, to collect data for six scientific applications on Ubuntu.

The collaborators have relevant domain knowledge and can design instructions using the supplied documents during collection. They then use our custom desktop collection scripts to record system states and their actions. Over one month, we collected 1,004 trajectories containing 3,590 state–action samples. We converted these into Qwen-style trajectories and trained with the same hyperparameters as OS-Wizard. The collaborators did not directly use code during collection, and the model was not provided with code interfaces during evaluation.

Because human data collection for specialized software is costly, we could not match the scale of automatic collection. We therefore sought the best performance achievable under practical constraints while making the training and evaluation conditions as fair as possible.

Q3: Thank you for raising this question. We reviewed the SCALECUA paper and public repository, neither of which mentions evaluation on only 163 tasks.

Q4: Our hybrid GUI–Code interface does not introduce task functionality that is unavailable to humans. Rather, it wraps operations already accessible through the specialized applications' native command interfaces, consoles, or editors as code actions that agents can call directly. We therefore believe that, in the ScienceBoard human evaluation environment, which supports both GUI and CLI interaction, humans can access command interfaces comparable to the code actions by entering commands through the GUI.

The original ScienceBoard paper does not specify the step or time budget for the human study. We acknowledge that the reported human results and our agent evaluation were not obtained under fully identical experimental protocols. Accordingly, our statement in the paper is limited to our highest achieved accuracy exceeding the human performance reported in the ScienceBoard paper.

Regarding the discrepancy in the human score calculation, thank you for checking this carefully. We confirm that the overall score reported in the original paper contains a calculation error and will add a note explaining this discrepancy in the revision.

Regarding the issue raised in M2(4), thank you for your suggestion. Under our current evaluation setup, no model is provided with documents at test time.

AuthorsOpenReview ↗

Response to Reviewer xWNH (3/4)

M3: Ablation experiments

(1) Thank you for pointing out this issue. We agree that sampling from existing trajectories without code calls can exclude Lean training data and shift the training-data distribution. We collected data without code actions from the five scientific applications other than Lean and proportionally sampled a total of 5K state–action examples. We compared these with 5K examples collected under the standard configuration, also excluding Lean. Training and evaluating Qwen3.5-27B yielded the following results:

ModelAstroBiochemGISAlgDocOverall w/o ATP
w/o Code21.234.514.76.512.518.2
5K w/o Lean21.244.823.541.918.830.8

These results show that, with all other conditions unchanged, removing only code actions from the action space during collection and evaluation reduces model performance. This supports the effectiveness of our code mechanism in improving performance.

(2) We reexamined the experimental data for w/o linear and w/o tree and recomputed the overall accuracy excluding the ATP domain:

ModelAstroBiochemGISAlgATPDocOverall w/o ATP
w/o linear33.362.144.161.3043.849.0
w/o tree42.451.717.638.77.718.835.0
OS-Wizard-20K45.579.364.774.23.856.364.3

Although w/o linear contains no Lean training data, accuracy on Lean tasks is generally low, so the recomputed overall accuracy does not change our original conclusion. Moreover, the per-domain results above show that both ablations reduce performance to varying degrees in every non-Lean domain relative to the mixed configuration.

M4: Contribution of tutorials

We computed statistics for the existing tree-exploration runs on scientific applications and found that collection with tutorials achieves a success rate of approximately 81.2%.

To better demonstrate the role of tutorial-guided generation in data generation, we collected data without tutorials from the five scientific applications other than Lean and proportionally sampled a total of 5K state–action examples. We compared these with 5K examples collected under the standard configuration, also excluding Lean. Training and evaluating Qwen3.5-27B yielded the following results:

ModelAstroBiochemGISAlgDocOverall w/o ATP
w/o Tutorial6.117.22.925.812.512.6
5K w/o Lean21.244.823.541.918.830.8

These results show that, with all other conditions unchanged, removing only tutorials during collection reduces the performance of the trained model. This supports the effectiveness of our tutorial mechanism in improving performance.

AuthorsOpenReview ↗

Response to Reviewer xWNH (4/4)

M5: Contribution of pivotal-state scoring

Thank you for pointing out this issue. For the five scientific applications other than Lean, we collected data with no pivotal-state selection and with random selection of the same proportion of pivotal states. We proportionally sampled a total of 5K state–action examples for each condition and compared them with 5K examples collected under the standard configuration, also excluding Lean. Training and evaluating Qwen3.5-27B yielded the following results:

ModelAstroBiochemGISAlgDocOverall w/o ATP
w/o Pivots15.244.826.522.612.525.2
w/ Random Pivots18.241.417.635.512.525.9
5K w/o Lean21.244.823.541.918.830.8

These results show that, with all other conditions unchanged, disabling pivotal-state selection or selecting pivotal states randomly during collection reduces the performance of the trained model. This demonstrates the advantage of our method over simple pivotal-state selection approaches and its effectiveness in improving performance.

We do not intend to claim that our pivotal-state scoring is the best available state-selection strategy. We can only establish that our method identifies potentially valuable branches to some extent. In practice, the individual scoring terms may not each contribute independently as envisioned theoretically, and the settings of the scoring parameters have not undergone detailed empirical analysis. Our aim is for the scoring heuristic as a whole to contribute positively to automatic data synthesis for computer-using agents.

Regarding recovery efficiency, pivot selection focuses on improving the efficiency of exploring valuable branches, while continuous execution focuses on reducing repeated recovery and replay overhead. We agree with your observation about the important contribution of continuous execution to reducing recovery overhead. Pivot selection also serves to identify valuable branches, and we will distinguish the roles of these two mechanisms more clearly.

M6 & Q5: Separation between training and evaluation

(1) The OOD exemplars consist of a small set of instructions synthesized using GPT and manually checked for decontamination to ensure that their content does not overlap with the dataset. We aim for GPT to generate task instructions that incorporate domain knowledge, for example:

  • Open EMDB map 1080 and set its volume contour level to 0.9 in ChimeraX.

Rather than GUI operation recipes such as "click a button and then adjust a parameter," for example:

  • Apply a white background in ChimeraX by clicking the White background toolbar button.

(2) The initial VM state is obtained by booting the VM and launching the corresponding application, without performing any other additional actions. We also do not preload extra small data files, such as PDB files for ChimeraX, because our environment allows the model to download them from the internet itself.

Regarding transfer to specialized software outside ScienceBoard, we believe that domain knowledge and application-specific operational knowledge can differ substantially across specialized applications, posing additional challenges for transfer. Its effectiveness therefore requires dedicated evaluation. We have not evaluated specialized software outside ScienceBoard in this paper and leave this question for further investigation in future work.

(3) & (4) The Lean training data come from other work, while the OSWorld training instructions are synthesized by GPT using OS-Wizard's own virtual environment as context. Both undergo decontamination. We compute embedding cosine similarities between the training data and test instructions; the 95th-percentile similarities are 0.578 and 0.506, with no evidence of semantic overlap. Thank you for pointing out these omissions in our writing. We will add the corresponding details to the appendix.

Reviewer_ybtkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents OSwizard, a framework for collection training traces of computer use tasks on specialized desktop applications. It uses tutorials to mitigate knowledge gaps and revisiting pivotal states to explore alternative workflows. It also proposes hybrid GUI–Code tools that reduce the number of interactions needed to control specialized applications. This code tool proves to be useful in evaluation. The collected data contain ~10K traces. Fine-tuning Qwen-based models on these data produces clear improvements on ScienceBoard and also improves performance on OSWorld.

Critical strengths
  • Interesting and practical problem. Specialized scientific applications are less studied than web and general desktop environments, and their domain-specific operations are new for computer-use agents.

  • Evaluation shows strong performance. Fine-tuning on OS-Wizard data substantially improves Qwen models on ScienceBoard and also improves OSworld.

  • The framework design and hybrid GUI–Code tool design are practical and useful. This a sensible way to improve the efficiency and coverage of trajectory collection.

Critical weaknesses
  • The gain from hybrid GUI-code supervision lacks a clean ablation.

The ablation shows the difference between GUI-code training and no code training: 55.0% versus 26.6% under a 20K-sample budget. The authors state that "randomly samples trajectories containing no code calls throughout, yielding 20K training samples". This sampling can cause bias, because this essentially filters out many tasks that should use code rather than converts the same tasks and outcomes into GUI-only traces. The two conditions may therefore differ in task difficulty, application distribution, functional coverage, trajectory length, and collection success.

  • The important comparison with related prior work baselines are not complete.

Authors mention that "Under a matched 10K-sample training budge, ...". More details on how baseline methods are implemented and tested would help audience understand the comparison. More importantly, this 10K budget is not necessarily the fair comparison. Data quality under the same number is not the only metric for dataset collection framework. We also need to compare (1) collection efficiency and (2) scaling performance. For example, under the same amount of time and compute, are the baseline methods able to produce more training data due to potential high efficiency? And when the training dataset size grows to 20K, 50K or more, how does the scaling look like? Figure 5 is helpful to show the scaling performance, but it is unclear how the baseline methods would perform.

Questions for authors
  • For the 20K-sample code ablation, how do the mixed and no-code subsets compare in application distribution, task category, trajectory length, and task difficulty?

  • Are all baseline models evaluated with exactly the same code tools, api descriptions, prompt examples ...? For dagger results imported from prior work, which aspects of the tool set and evaluation protocol differ?

  • Can the authors show (1) collection efficiency and (2) scaling performance in comparison to baseline methods?

5 comments
AuthorsOpenReview ↗

Response to Reviewer ybtk

Thank you for your constructive review and suggestions!

W1 & Q1: Ablation of code actions

Thank you for pointing out this issue. We agree that sampling from existing trajectories without code calls can indeed shift the training-data distribution, so we conducted new collection and evaluation experiments. Since the Lean data consist entirely of code actions, we collected data without code actions from the five scientific applications other than Lean and proportionally sampled a total of 5K state–action examples. We compared these with 5K examples collected under the standard configuration, also excluding Lean. Training and evaluating Qwen3.5-27B yielded the following results:

ModelAstroBiochemGISAlgDocOverall w/o ATP
w/o Code21.234.514.76.512.518.2
5K w/o Lean21.244.823.541.918.830.8

These results show that, with all other conditions unchanged, removing only the code primitives from the action space during collection and evaluation reduces model performance. This supports the effectiveness of our code mechanism in improving performance.

W2 & Q2: Comparison with baseline methods

We made the following efforts to ensure a fair comparison between our method and the baseline methods.

We reimplemented OS-Genesis and ANCHOR within the OS-Wizard framework following the methods described in their respective papers. Since neither method uses code actions or tutorials, we did not provide these resources, to remain consistent with the original papers.

The collection programs produce trajectories in the same format as the OS-Wizard framework, allowing them to be converted into the same Qwen-style trajectories. We use the same model for training (Qwen3.5-27B), the same training hyperparameters (see Appendix F), and the same scientific-application proportions as OS-Wizard, based on the state–action sample proportions in the full dataset. We also use the same prompt template during evaluation (see Appendix H). We do not provide the optional APIs or their descriptions during evaluation, because code actions were not included in the training action space.

Regarding the two results marked with daggers, ScaleCUA uses the full set of official ScienceBoard evaluation scripts. According to the SCALECUA paper, the authors added a file-writing tool to the original ScienceBoard and used their own system prompts during RL training and evaluation. The paper does not describe any modifications to the evaluation protocol.

Thank you for pointing out these omissions in our writing. We will add these details in a new appendix subsection.

Q3: Data collection efficiency and scaling performance

We compared the cost and time efficiency of OS-Wizard's tree exploration and linear rollouts with those of OS-Genesis and ANCHOR:

MethodInput tokens per stepOutput tokens per stepTime per stepMean cost per step
tree13,531.23225.4476.14 sUSD 0.066
linear12,895.09278.0047.77 sUSD 0.041
OS-Genesis28,815.78516.8583.21 sUSD 0.090
ANCHOR26,160.40651.6055.22 sUSD 0.084

Overall, our method is faster than OS-Genesis but slower than ANCHOR, while being substantially less expensive than both baseline methods. In balancing these considerations, we place greater emphasis on synthesizing higher-quality data from dynamic environments than on matching ANCHOR's faster collection speed.

Regarding scaling performance, time and cost constraints make it infeasible in the short term to collect baseline datasets comparable in size to OS-Wizard's and conduct the corresponding training and evaluation. Based on the existing results, increasing the data budget from 0 to 10K improves OS-Wizard by 31.4%, compared with 7.1% and 13% for OS-Genesis and ANCHOR. This indicates the higher quality of our data and the advantages of our framework.

Thank you for the response. The new evaluations on ablation of code actions and collection efficiency, substantially address my concerns.

My concern on data collection scaling and comparison is not fully resolved. The cost table above shows per step cost, but cost for each sample or whole dataset is unclear here. Since linear collection and ANCHOR likely have lower cost or shorter time, does OS-Wizard perform better under matched collection cost? Even a smaller scale comparison would be helpful.

I understand time and cost constraints for such comparison experiments. And current results do show higher data quality under matched number of data samples. Thanks for the response. I am keeping my positive score.

Thank you for the rebuttal. My concerns have been addressed, and I confirm my original score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their careful reviews and constructive feedback. Reviewer ybtk recognized the practical importance of studying specialized scientific software, the substantial performance improvements, and the utility of the hybrid GUI–Code interface. Reviewer xWNH acknowledged the completeness of the framework, the quality of supervision under matched data budgets, and the detailed implementation documentation. Reviewer dRLd recognized the dataset’s value to the CUA research community and the thorough documentation of the infrastructure.

We have addressed the reviewers’ main concerns through additional experiments and point-by-point clarifications:

Component contributions and attribution of performance gains. In response to concerns from all three reviewers, we collected new data across five scientific applications and conducted controlled comparisons using consistent application proportions and matched training-sample budgets. The full configuration achieves a success rate of 30.8%, which drops to 18.2%, 12.6%, and 25.2% when code actions, tutorials, and pivots are removed, respectively. These results provide direct evidence for the effectiveness of the hybrid action space, tutorial guidance, and pivotal-state selection.

Baseline comparisons and interpretation of results. In response to Reviewers ybtk and xWNH, we provided further details on our implementations of OS-Genesis and ANCHOR, including the collection models, training model, training hyperparameters, application proportions, and evaluation prompt template. Under the same 10K training-sample budget, OS-Wizard achieves a higher success rate than both baselines, providing direct evidence for the effectiveness of our collection framework. Comparisons with other models and reported human performance serve to contextualize OS-Wizard’s evaluation results on ScienceBoard.

Collection cost and efficiency. In response to Reviewers ybtk and dRLd, we provided statistics on token consumption, collection time, and monetary cost. Both tree exploration and linear collection have lower average per-step costs than OS-Genesis and ANCHOR, and lower average per-step collection times than OS-Genesis.

Data sources and separation between training and evaluation. In response to Reviewers xWNH and dRLd, we clarified that the OOD exemplar instructions were synthesized using GPT and manually verified. We also explained how the collection environments were initialized and the source of the Lean data.

OS-Wizard aims to address the scarcity of high-quality interaction data for specialized desktop software through knowledge-guided automated exploration. It provides the computer-using agent community with a scalable trajectory collection framework and training data, supporting the extension of agents to specialized applications. We appreciate the reviewers’ acknowledgment of our additional work and hope these experiments and clarifications help all reviewers and the Area Chair fully assess OS-Wizard’s methodological and data contributions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.