M1. The methodological novelty and distinction relative to prior work needs to be articulated more clearly
OS-Wizard integrates tutorial-guided exploration, pivotal-state-based tree exploration, and a hybrid GUI-Code action space into an effective trajectory collection framework. However, the individual ingredients of OS-Wizard have substantial conceptual overlap with prior work on computer-use agents and trajectory synthesis. In particular, hybrid GUI-Code interactions have been explored by systems such as CoAct/ActionEngine, UltraCUA/UFO²/Axis, and ToolCUA/EE-MCP. Tutorial- or document-guided trajectory generation has also appeared in prior work, such as AgentTrek/TongUI/Video2GUI, while exploration-based methods such as OS-Genesis and ANCHOR have already explored task-driven and branch-driven trajectory collection. In particular, ANCHOR’s use of meaningful branch states appears conceptually close to OS-Wizard’s pivotal-state-based branching strategy.
More importantly, the hybrid action space is not specific to OS-Wizard at the level of the environment. The original ScienceBoard benchmark already allows agents to use application-specific CLI or scripting mechanisms in addition to GUI actions, and 131 of its 169 tasks are categorized as CLI-only (33) or GUI+CLI (98) tasks. The paper nevertheless describes the hybrid action space as “the key design of OS-Wizard” (Section 3.3). As far as I can tell from Appendix A, the authors’ own contribution at this level consists of a set of adapters, such as displaying GRASS GIS commands and their output in the console, adding a calc endpoint to the KAlgebra server, and providing file-editing primitives for Lean 4 and TeXstudio. The paper should state explicitly which parts of the action space are inherited from ScienceBoard and which are new.
Similarly, the tree exploration is shallower than the framing suggests. With f(d) = max{10 - 3d, 1}, branching only occurs within the first three steps, after which every branch becomes a single instruction-following rollout, and the resulting trajectories have a mean length of 5.03 steps and a median of 4. In effect, the procedure explores a short action prefix and then synthesizes an instruction from the reached state, which is close in spirit to the reverse task synthesis of OS-Genesis and the branch-point expansion of ANCHOR. A clearer statement of what the tree structure adds beyond these methods would strengthen the novelty claim.
The current results convincingly demonstrate that the overall OS-Wizard pipeline performs better than the implemented baselines. However, this comparison does not fully isolate the methodological contribution of the proposed knowledge-guided/pivotal-state exploration strategy, since OS-Wizard simultaneously differs in tutorial guidance, branch selection, and the hybrid GUI-Code action space. Consequently, it remains unclear which component constitutes the primary methodological advance and to what extent the substantial downstream gains can be specifically attributed to the proposed exploration strategy.
I therefore encourage the authors to clarify more precisely:
• What is the primary methodological contribution of OS-Wizard compared with prior work, particularly given that tutorial-guided exploration, branch/state-based exploration, and hybrid GUI-Code interaction have all been explored in related studies?
• Which parts of the hybrid action space and of the tree exploration are new relative to ScienceBoard’s native interfaces and to the branching and task-synthesis procedures of OS-Genesis and ANCHOR?
M2. Several headline comparisons are not made under matched or documented conditions
Several of the paper’s main claims rest on comparisons whose conditions differ in ways that are not stated in the paper.
(1) Comparison with the previous best result. The paper claims a new state of the art by comparing 64.1% with the 54.0% reported for SCALECUA-Qwen3.5-9B. However, the two numbers do not appear to be measured on the same task set or through the same interface. The corresponding paper (Lv et al., 2026) evaluates ScienceBoard with mouse/keyboard actions plus a few auxiliary actions (e.g., answer and write_file), rather than with application-native command interfaces. Moreover, its per-domain results appear to be consistent only with 163 rather than 169 tasks: with four rollouts per task, 19.0% on ATP corresponds to 16/84 (i.e., 21 tasks), 86.7% on Doc corresponds to 52/60 (i.e., 15 tasks), and the six domains together give 352/652 = 54.0%, whereas neither the ATP nor the Doc value can be obtained with 26 ATP and 16 Doc tasks. If this reading is correct, the 10.1-point margin compares different task sets and different action interfaces, and the state-of-the-art claim should be qualified accordingly, ideally by also reporting OS-Wizard on the same task subset.
(2) Comparison with human performance. The paper reports 64.1%, compared with the human score of 59.8% attributed to the original ScienceBoard paper, and the abstract states that OS-Wizard surpasses reported human performance. Two points need clarification. First, the original paper reports an overall human score of 60.27%, which is the unweighted mean of the six domains, whereas 59.8% corresponds to a task-weighted recomputation (101/169); recomputing for consistency is reasonable, but Table 1 marks the number as reported in the original paper. Second, and more importantly, the agent is evaluated through the authors’ modified interface (Appendix A), and it is unclear whether the human participants had access to command-based tools comparable to this interface, or to a comparable step budget. The current experiments also do not isolate how much of the margin is attributable specifically to tutorial-guided exploration. Therefore, the result should not be interpreted as evidence that OS-Wizard surpasses human performance under comparable conditions, nor that knowledge-guided exploration itself enables it to do so.
(3) Re-implemented baselines. The Task-driven and Branch-driven baselines are described only as being implemented “following OS-Genesis and ANCHOR.” The paper does not state whether these baselines had access to the CODE actions, whether they used retrieved tutorials, which models served as instruction proposers, action samplers, and verifiers, which backbone was fine-tuned, or whether their 10K samples cover the same applications in the same proportions as the OS-Wizard-10K subset. Since both the action space and the application mix appear to affect downstream performance substantially (see M3), these details determine whether the 36.5% vs. 13.0%/18.9% comparison reflects the exploration strategy itself. In addition, the “Human Data” baseline (9.5%) is not described anywhere in the paper; its source, size, and collection protocol should be reported. Finally, the baseline and ablation results are reported as single numbers, whereas the OS-Wizard rows report standard deviations over three evaluation runs; these standard deviations reflect evaluation variance only and do not capture variance across training runs.
(4) Comparison with the data-collection model. The paper notes that OS-Wizard-Qwen3-VL-8B surpasses GPT-5.6-Sol (46.6% vs. 42.6%), “the strongest model used for data collection.” During collection, however, GPT-5.6-Sol operates with retrieved tutorials, reward-model filtering, and repeated sampling, whereas it is evaluated as a plain agent without these aids. A baseline that provides the same retrieved tutorials at inference time, both to GPT-5.6-Sol and to the base Qwen models, would help separate the value of collecting and training on trajectories from the value of simply giving the agent access to application knowledge.
M3. The component ablations confound the ablated factor with application composition
The ablations in Figure 4(b) are central to the paper’s attribution of the gains, but in both of them the ablated factor changes together with the application mix of the training data.
(1) Code supervision. The w/o-code variant is built by randomly sampling 20K samples from trajectories that contain no code calls at all. According to Appendix C, Celestia (7,401 samples, GUI-only) and the seven general-purpose applications (17,442 samples, for which Table 3 defines no CODE primitives) already provide 24,843 code-free samples, whereas all 7,546 Lean samples consist of file-editing CODE actions and are therefore excluded. Unless the sampling was stratified by application, the w/o-code training set is likely dominated by general-purpose and Celestia data, contains no theorem-proving data at all, and contains comparatively little data for ChimeraX, GRASS GIS, KAlgebra, and TeXstudio. The 55.0% vs. 26.6% gap may therefore reflect missing in-domain data as much as the absence of code supervision. It is also worth noting that SCALECUA-Qwen3.5-9B reaches 54.0% on ScienceBoard with a GUI-centric action space (albeit on a slightly different task set, see M2), which suggests that GUI-centric supervision is not inherently limited to the 26.6% level.
(2) Tree exploration vs. linear rollouts. Lean data are collected exclusively through linear rollouts (Section 4.1), so the “w/o linear rollouts” variant contains no theorem-proving data, and the two variants may also differ in their application mix more generally. The 41.4% vs. 30.7% comparison therefore does not cleanly isolate the collection strategy either.
I encourage the authors to (i) report the per-application composition of every ablated training set, (ii) report per-domain results for the ablations, and (iii) ideally repeat the code ablation with the application mix held fixed, for example by re-collecting GUI-only trajectories for the same applications, or by sampling code-free and code-containing trajectories within each application in matched proportions.
M4. The contribution of tutorial knowledge is not directly established
The hybrid GUI-Code action space is useful and can make many tasks easier by reducing the number of interaction steps and avoiding complex GUI operations. Although CODE actions may not provide additional functionality beyond what can be achieved through the GUI, they provide a higher-level and more direct way to complete many operations. The ablation reports a very large gap for code supervision (55.0% with mixed GUI-Code supervision vs. 26.6% without code supervision), although, as discussed in M3, this gap is partly confounded by application composition.
Given that “Knowledge-Guided Exploration” is the central framing of the paper, it is important to clarify how much of the performance improvement can be attributed specifically to tutorial-guided exploration, beyond the substantial benefit provided by CODE abstraction and supervision. The current no-document experiment shows that tutorial guidance improves data-collection efficiency: without documentation, 75.71% of branches reach the maximum depth (D=16) without completion on KAlgebra and GRASS GIS. However, the paper does not report the corresponding rate with tutorials, so even the size of the efficiency gain cannot be assessed. Moreover, this result does not directly show whether tutorial-guided trajectories provide higher-quality downstream supervision under the same amount of usable training data. A matched-data ablation comparing models trained on data collected with and without tutorial guidance, while keeping the GUI-Code action space unchanged, would provide more direct evidence of the contribution of tutorial knowledge. Such a comparison would also clarify to what extent the performance gains arise specifically from knowledge-guided exploration rather than from the richer hybrid GUI-Code action space.
M5. The proposed pivotal-state scoring mechanism is not directly validated
A central methodological component is the pivotal-state score S(v) = λd Sd(v) + λu Su(v) + λp Sp(v), followed by selecting the top 20% of candidate states. However, the paper does not appear to provide a direct ablation demonstrating that this scoring rule selects better branch points than simpler alternatives.
(1) Lack of direct validation. The existing tree-vs.-linear experiment shows Tree-only = 41.4% and Linear-only = 30.7%. This establishes that tree-generated supervision is useful, but it does not establish that the proposed pivotal-state scoring is responsible for that benefit.
(2) Heuristic design choices. Several design choices also appear heuristic, including the combination of depth, transition magnitude, and empirical action success; the λ weights; selecting the top 20%; and the branching schedule 10→7→4→1.
(3) The scoring rule appears to be largely inactive in the actual procedure. According to Section 3.2, every first-level state forms a singleton candidate set and is retained as a pivot, so the score is only used to choose 2 pivots within each group of ten second-level siblings. Within such a group, all candidates have the same depth, so the depth prior Sd is constant and its min-max normalization is degenerate; moreover, Sp is used only “when such statistics are available,” which is unlikely for freshly generated states. In practice, pivot selection therefore seems to reduce to ranking siblings by the transition magnitude Su alone. The expected count of 187.6 branches per worker in Section 5.3 further implies that being a pivot mainly changes how many valid children are collected from each second-level state (4 for a pivot vs. about 2.15 in expectation for a non-pivot at the 53.8% validation rate). The authors should clarify whether this reading is correct and, if so, describe the mechanism accordingly.
(4) Limited contribution to recovery efficiency. The recovery analysis shows that pivotal checkpoints reduce the average replay overhead only from 15.7 to 14.2, whereas continuous single-branch execution further reduces it to 2.7. This suggests that most of the reported improvement in recovery efficiency comes from continuous execution rather than pivotal checkpointing itself.
(5) More direct comparisons are needed. The authors could compare the proposed pivotal-state selection with simpler alternatives, such as random selection or individual scoring components, under the same experimental conditions. A comparison with an ANCHOR-style branch-selection method would further clarify the advantage over prior work.
M6. The separation between the collected training data and the evaluation tasks is insufficiently documented
(1) OOD exemplar instructions. When the branching factor reaches f(d)=1, the method uses retrieved tutorials together with 5-10 OOD exemplar instructions to synthesize the task instruction for the subsequent rollout. However, I could not find a clear description of where these exemplar instructions come from, what ‘OOD’ means operationally, whether they originate from ScienceBoard or another benchmark/dataset, how they are selected, and how overlap with ScienceBoard evaluation instructions is prevented. The instruction-proposal prompt in Appendix F.1 only mentions an optional field containing “reference instructions,” without stating their provenance.
This is important because the exemplars directly influence the distribution of automatically synthesized training tasks. Without knowing their provenance, it is difficult to rule out benchmark-specific instruction leakage or substantial semantic overlap between training and evaluation tasks. I am not claiming that leakage occurs, but the current description does not provide enough information to exclude it.
(2) Initial states and task resources. The paper does not describe the root states from which exploration starts, nor the files, datasets, and application configurations loaded in the collection environments (e.g., the molecular structures opened in ChimeraX, or the GRASS GIS locations and mapsets). If these coincide with the initial states or resources of ScienceBoard evaluation tasks, the synthesized training tasks may be much closer to the test tasks than the phrase “scaling to specialized software” suggests. Relatedly, the six specialized applications used for collection are exactly the six ScienceBoard applications, so the paper does not test whether the framework or the collected supervision transfers to specialized software outside the benchmark.
(3) Lean problems. The 1,608 Lean problems are taken from the Formal Math Autoformalization Dataset. Given that the ATP domain improves from 0% to 21.8%, the authors should check and report whether any of these theorems, or close variants of them, coincide with the theorem-proving tasks in ScienceBoard.
(4) General-purpose applications and OSWorld. Similarly, it would help to state whether the 17,442 general-purpose samples were collected in OSWorld’s own VM image and task resources, and whether any synthesized instructions overlap with OSWorld tasks, since this affects the interpretation of the OSWorld improvement (see also Minor Comment 2).
The authors should report the exact source and construction procedure for these exemplars, define the relevant in-/out-of-distribution boundary, and ideally report both exact and semantic overlap checks between synthesized training instructions, exemplar instructions, and ScienceBoard evaluation tasks. Analogous checks for the initial states, the Lean problems, and the OSWorld tasks would make the evaluation considerably more convincing.
Response to Reviewer dRLd
Thank you for reviewing our work and sharing your suggestions!
Thank you for your question. In the introduction (L83–103), we describe how OS-Wizard collects trajectories for specialized software: tutorials and domain-specific code provide specialized knowledge and operations (L263–282), pivotal-state scoring identifies valuable states and expands their continuations (L196–210), and tree exploration is combined with linear rollouts to broaden trajectory coverage (L214–258). We demonstrate the improvements contributed by these components through the corresponding experiments. We would be happy to discuss any questions you may have about these details.
Our statement “surpassing reported human performance of 59.8% and establishing a new state of the art” refers to the factual comparison that OS-Wizard-Qwen3.5-27B achieves higher accuracy on ScienceBoard than the accuracies reported in the two papers for the current best model, SCALECUA-Qwen3.5-9B, and for human performance on ScienceBoard. We do not make claims such as “our method is superior to SCALECUA.”
This factual comparison is based on the accuracies of OS-Wizard, SCALECUA, and the ScienceBoard human evaluation being obtained using the same evaluation scripts. Both OS-Wizard and the ScienceBoard human evaluation use the official ScienceBoard evaluation scripts. For SCALECUA, the public materials we reviewed do not report modifications to the evaluation scripts.
The claim we make in the paper is “outperforming alternative collection strategies under a matched training-data budget.” This conclusion follows from our comparison of OS-Wizard with two other data collection frameworks of the same kind, OS-Genesis and ANCHOR, under matched training-sample budgets. We would be happy to discuss any further questions you may have about the details.
The OOD exemplars consist of a small set of instructions synthesized using GPT and manually checked for decontamination to ensure that their content does not overlap with the dataset. We aim for GPT to generate task instructions that incorporate domain knowledge, for example:
Rather than GUI operation recipes such as "click a button and then adjust a parameter," for example:
We computed the total monetary and time costs of OS-Wizard's tree exploration and linear rollouts separately. We also computed the costs of OS-Genesis and ANCHOR. The results are shown below:
Overall, our method has lower mean token consumption than both baselines and lower mean time consumption than OS-Genesis.
Thank you for pointing out these omissions in our writing. We will add these details in a new appendix subsection.
Thank you for pointing out this issue. We collected data from five scientific applications under four conditions: (1) the standard configuration, (2) without tutorials, (3) without the pivot mechanism, and (4) with random pivot selection. We proportionally sampled 5K state–action examples for each condition for comparison. Evaluation after training yielded the following results:
These results show that, with all other conditions unchanged, modifying only one component of OS-Wizard reduces model performance in each case, indicating that our performance improvements can be attributed to these factors.