acceptodds
Under review as a conference paper at ICLR 2027

ScienceCUA: Resource-to-Skill Construction and Execution-Guided Skill Evolution for Scientific Computer Use Agents

Abstract

Computer-use agents for scientific software require application-specific expertise. Much of this expertise is already documented in human-authored software manuals and web pages, making them readily available resources for improving agents’ operational capabilities. However, turning these resources into executable, reusable skills remains challenging due to mismatches between documented and deployed software versions and differences in scope between documentation coverage and task requirements. To address these challenges, we introduce ScienceCUA, a two-stage framework combining resource-to-skill construction with execution-guided skill evolution. First, ScienceCUA identifies tasks supported by the software from its documentation, infers operational trajectories, and generalizes them into reusable Skills. Command catalogs, syntax constraints, and shortcuts are organized separately as References for on-demand lookup. Then, ScienceCUA uses execution feedback to drive skill self-evolution by contrasting successful and failed trajectories for the same task, identifying deficiencies, revising existing skills or adding new ones, and updating the skill library only after repeated validation in real environments. To enable reliable evaluation without benchmark noise or task leakage, we introduce two improved benchmarks. ScienceBoard-Verified provides an audited and corrected basis for measuring task performance, while ScienceBoard-VE separates skill evolution from final testing through disjoint evolution and test tasks and further supports native-agent evaluation. Experiments show that the initial Skills and References improve average task success rate by up to 7.23 percentage points over the base model, while skill self-evolution increases the total gain over the baseline to as much as 17.12 percentage points. ScienceCUA turns software documentation into an evolving set of executable skills and queryable references, offering a practical way for general-purpose agents to use scientific software reliably and at scale.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
computer use agents, scientific software, resource-to-skill construction, execution-guided skill evolution

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_KVrrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ScienceCUA, a framework that converts scientific software documentation into reusable Skills and References. Skills describe operational procedures, while References provide commands, syntax, and other lookup information. The framework then uses execution feedback to revise or add knowledge, with candidate updates validated in fresh sessions.

The authors also correct issues in ScienceBoard and introduce separate tasks for skill evolution. Experiments across six scientific applications show that the initial knowledge library improves task success, and execution-guided evolution provides further gains.

Critical strengths

1. A practical approach to organizing software knowledge.

The paper addresses a practical problem: software documentation can be long, incomplete for a task, or inconsistent with the installed version. Separating operational procedures from command and syntax references is a reasonable design. Table 2 shows that combining Skills and References performs better on average than using either alone in both evaluated settings.

2. Performance improvements across multiple settings.

The experiments show benefits across multiple models and applications. In Table 3, evolution improves average success from 44.70% to 54.59% for Grok-4.6 and from 57.16% to 65.81% for GPT-5.5. The documented benchmark corrections are also useful, since ambiguous instructions and overly restrictive evaluators can make agent results difficult to interpret.

Critical weaknesses

1. Limited evidence for transfer to different workflows.

The scope of the demonstrated transfer is somewhat limited. Section 4.2 and Appendix J explain that evolution tasks retain the same operation types as test tasks while changing objects, parameters, or requirements. The manual deduplication is useful, but this setup mainly tests transfer within familiar operation families. The plotting and map-styling examples also follow this pattern. It remains unclear how much improvement extends to tasks requiring different workflows or combinations of operations.

2. Incomplete cost comparison for self-evolution.

The cost and budget comparison for self-evolution needs more detail. Appendices E and F report execution time and initial library construction time, while Appendix K gives upper bounds for evolution executions. The actual evolution cost is not reported. Appendix Q also states that Trace2Skill uses the same first-execution trajectories, whereas ScienceCUA obtains additional feedback through re-exploration and validation. This leaves uncertainty about how much of the advantage comes from the update procedure versus additional environment interaction.

Questions for authors

1. Scope of skill transfer.

Could the authors summarize the similarity between evolution and test tasks at the operation or workflow level? Reporting gains separately for closely related tasks and tasks requiring different combinations of operations would help clarify the scope of transfer.

2. Actual evolution cost and budget matching.

Could the authors report the actual environment execution counts, token usage, and wall-clock costs for ScienceCUA and Trace2Skill? How were interaction budgets matched given their different exploration and validation procedures? This would help interpret the performance advantage in Table 3.

3. Stability of the reported improvements.

How many independent runs were used for the main results? Could the authors provide variability estimates for the comparison between the initial library and Full ScienceCUA? Please also clarify whether the same constructed libraries were reused across runs.

6 comments
AuthorsOpenReview ↗

Response to Weakness 1 and Question 1

Weakness 1

We are very grateful to the reviewer for raising the question of whether the evolution tasks and the test tasks are too similar. We have added Appendix R to the revised manuscript to examine two questions using KAlgebra as an example. First, can a Skill or Reference revised during evolution transfer across task templates? Second, can the revised operational guidance be reused in different workflows?

Appendix R.1 distinguishes between task categories and reusable atomic functions. For example, different categories of tasks, including calculation, 2D plotting, and 3D plotting, can all use the atomic function of script import, while the operations required after import differ. This distinction helps us identify which specific operational capability the evolution tasks and the test tasks share.

To test whether transfer depends on using the same task template, in Appendix R.2, we conducted single-task evolution separately using the task of plotting a torus (E-01) and the task of plotting two intersecting planes (E-02). Each task started from the same initial knowledge base, and both exposed an incorrect restriction on input forms in the initial 3D Skill during execution. The two evolution runs respectively validated the two approaches of plotting with a function of three variables and directly entering an implicit equation containing z (Table 17 and Figure 10). When using the knowledge base constructed with Resource-to-Skill, the model received incorrect guidance from the 3D plotting Skill extracted from the old version of the manual on all three tasks involving a hyperboloid of one sheet (A-03), a sphere (C-14), and an oblique cylinder (C-17), and failed all three tasks (Appendix M). After replacing it separately with the 3D Skill independently revised through E-01 and the one independently revised through E-02, all six transfer evaluations passed (Appendix R.2, Table 18, and Figure 11). The two evolution tasks and the three test tasks share the atomic function of implicit 3D plotting, but do not use the same task template, and their differences are not limited to replacing parameters or objects.

Appendix R.3 further examines the reuse of the same 3D plotting Skill in different workflows. Since the existing KAlgebra test tasks lack cases that combine 3D plotting with multiple other software operations, we added two tasks for this investigation. W-01 links script import, calculation and assignment, and double-cone plotting in sequence (Figure 12). W-02 links parameter import, sphere plotting, and Lines style configuration in sequence (Figure 13). Using the fully evolved knowledge base, the agent completed both tasks, showing how the guidance for implicit 3D plotting can be reused at the corresponding stages in different combinations of operations.

Answer 1

We are very grateful to the reviewer for suggesting that transfer gains be reported in groups based on the similarity of operation combinations. Since it is indeed difficult to use the same set of criteria to quantify the degree of operational overlap and task template similarity between the evolution and test sets across different software applications, and the time available for the rebuttal is limited, this supplementary analysis focuses on a case study of KAlgebra. The above cases show that a Skill or Reference revised and validated for a particular atomic function can transfer to different tasks requiring the same function. In composite workflows, the agent can also reuse the same piece of knowledge at the stage where that function is needed and combine that function with other operations according to the task requirements.

AuthorsOpenReview ↗

Response to Weakness 2 and Question 2 (Part 1/2)

Weakness 2 (Part 1/2)

We are very grateful to the reviewer for suggesting that we report actual costs and provide a more thorough comparison of the execution budgets of the two methods. First, the single-trajectory setting in the original manuscript followed the experimental configuration of the original Trace2Skill paper. Section 3.1 explicitly states that one trajectory is generated for each task during the trajectory generation stage, after which the success and error analysis modules separately propose patches, followed by hierarchical integration. We reproduced Trace2Skill in the original manuscript following the settings of the original Trace2Skill paper, so each evolution task was executed once during trajectory collection, and the resulting trajectories were then used for analysis and knowledge updates.

To make a fair comparison, we expanded Trace2Skill’s trajectory collection to three independent executions per task. All successful and failed trajectories were passed to the corresponding success or failure modules for processing, after which knowledge was integrated following the original Combined process, retaining the default merge batch size of Bmerge=32B_{\mathrm{merge}} = 32.

Under the model-based setting with Grok-4.6 (xhigh), the actual costs of the two methods are as follows. Each cell lists ScienceCUA / Trace2Skill with three executions per task, in that order.

Env. exec. = number of valid environment executions; Model tasks = number of model tasks involving non-environment logic; M = millions; Time (h) = cumulative effective runtime in hours.

SoftwareEnv. exec.Model tasksTokens (M)Time (h)
KAlgebra45 / 6031 / 6317.668 / 33.0778.861 / 13.880
Celestia51 / 6034 / 6336.440 / 43.12515.239 / 18.975
GRASS GIS58 / 6056 / 6373.713 / 62.60614.132 / 15.915
ChimeraX37 / 6026 / 6328.908 / 45.1728.314 / 13.836
Lean 458 / 6049 / 6336.954 / 82.94713.554 / 17.084
TeXstudio23 / 2417 / 2515.701 / 17.6609.505 / 9.057
Total272 / 324213 / 340209.384 / 284.58769.605 / 88.747

ScienceCUA’s environment executions include exploration, candidate validation, submission validation after success, and regression runs after integration. Its count of model tasks involving non-environment logic includes error analysis, knowledge writing, the addition or modification of Skills and References, and knowledge integration.

The environment executions of Trace2Skill (the three-execution version) include three independent runs of each evolution task. Its count of model tasks involving non-environment logic includes analysis of successful and failed trajectories, generation of the corresponding knowledge updates, and final knowledge integration.

Token usage covers all model calls throughout the complete evolution process, and elapsed time is the sum of the end-to-end durations of all tasks. The final Verified evaluation is accounted for separately, and costs caused by infrastructure, API, and network failures have been excluded.

After completing one round of knowledge base evolution, we independently evaluated the same frozen Trace2Skill knowledge base twice on Verified. Full ScienceCUA uses the results from Table 3 in the original manuscript. The results are as follows.

SoftwareFull ScienceCUATrace2Skill (three executions per task)
KAlgebra80.65%(67.74 ± 4.56)%
Celestia63.64%(54.55 ± 4.29)%
GRASS GIS47.06%(36.76 ± 2.08)%
ChimeraX85.71%(83.93 ± 2.53)%
Lean 419.23%(17.31 ± 2.72)%
TeXstudio31.25%(31.25 ± 8.84)%
Average54.59%(48.59 ± 1.90)%

In this supplementary comparison, ScienceCUA’s average success rate was 6.00 percentage points higher, while its total number of environment executions was reduced by 16.05%, its number of model tasks involving non-environment logic by 37.35%, its token consumption by 26.43%, and its cumulative elapsed time by 21.57%. After the number of independent executions was increased, ScienceCUA still achieved a higher success rate and a lower total evolution cost.

AuthorsOpenReview ↗

Response to Weakness 2 and Question 2 (Part 2/2)

Weakness 2 (Part 2/2)

We further calculated the distribution of outcomes across the three independent executions of each evolution task for Trace2Skill.

SoftwareNumber of evolution tasksSuccess in all three executionsFailure in all three executionsMixed success and failure outcomes
KAlgebra201334
Celestia20965
GRASS GIS20686
ChimeraX201532
Lean 4205105
TeXstudio8242
Total108503424
Percentage of all tasks100%46.30%31.48%22.22%

Of the 108 tasks, 84 had consistent success or failure outcomes across the three executions, accounting for 77.78%. This indicates that, when the number of independent retries is increased under a fixed initial knowledge base, most tasks still produce the same success or failure outcomes. For tasks that continue to be affected by incorrect operational knowledge, independent retries may repeatedly encounter the same obstacles.

In contrast, ScienceCUA’s exploration guided by failure history can provide subsequent attempts with previous action paths, environment observations, and failure outcomes, helping the agent identify and correct version-mismatched knowledge. ScienceCUA achieved better results with a smaller budget, indicating that the gains from ScienceCUA’s self-evolution method do not come solely from advantages in interaction.

Finally, we will incorporate all supplementary experimental results and cost statistics from the rebuttal period into the revised manuscript.

Answer 2

We have already reported the actual environment execution counts, non-environment model calls, token consumption, and actual elapsed time for ScienceCUA and Trace2Skill under Weakness 2. Please refer to Weakness 2 for the specific data.

Reviewer_D6DCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ScienceCUA, a two-stage framework that constructs and evolves an external knowledge library. In the first stage, the system converts software documentation into procedural Skills and separate References containing commands, syntax, and shared lookup information. In the second stage, it uses execution feedback on dedicated evolution tasks to revise or add library entries, validates candidate updates in fresh sessions, and consolidates accepted changes. To evaluate this pipeline reliably, the authors repair ScienceBoard to establish 168 verified test tasks and construct 108 separate evolution tasks. Using this benchmark, they evaluate the initial library across four model backbones and two native-agent settings, and then test the fully evolved library under two models with the library frozen and all evolution history removed at test time.

Critical strengths
  1. Section 4 and Appendix I substantially improve benchmark validity by revising 37 flawed tasks, removing one broken task that required an unavailable external account, and fixing shared environment bugs such as malformed answer-submission protocols and biased evaluators that rejected valid console commands. These repairs ensure that measured failures reflect agent limitations instead of benchmark defects. Building on this cleaned test set of 168 tasks, Appendix J introduces 108 disjoint evolution tasks verified through manual deduplication and domain-expert execution checks, establishing a sound basis for evaluating skill evolution without test leakage.

  2. The experiments validate both stages of the pipeline across multiple executors. Table 1 shows that the initial library improves pass rates across four model backbones and two native-agent settings while avoiding the heavy runtime overhead of direct documentation injection shown in Appendix E. Table 2 further confirms that procedural Skills and lookup References provide complementary gains. On top of this initial library, Table 3 shows that full evolution raises average pass rates from 44.70% to 54.59% on Grok and from 57.16% to 65.81% on GPT, whereas removing either knowledge addition or failure history degrades performance. Together, these results demonstrate the practical value of combining documentation-based initialization with execution-guided evolution.

  3. The validation protocol in Section 3.2 and Appendix K cleanly separates exploratory trajectory history from reusable library knowledge. Because every candidate update is tested in a reset environment and a fresh session stripped of earlier trajectories and diagnoses, validation success cannot rely on prior conversation history and must come directly from the written knowledge entry. The KAlgebra and GRASS GIS case studies in Appendices M and N reinforce this design by tracing specific evolved entries directly to successful executions on unseen test tasks that previously failed.

Critical weaknesses
  1. Tables 1 through 3 report only single point estimates without confidence intervals or the number of independent evaluation runs, making small performance differences difficult to trust. Because the 168 test tasks are split across six applications, individual subsets are small, so a single TeXstudio task shifts that application score by 6.25 percentage points and a single Lean 4 task shifts its score by 3.85 percentage points. This sensitivity directly affects the knowledge-addition ablation in Table 3, where the overall gap is only 3.09 percentage points on Grok and 2.76 percentage points on GPT, with four of the six applications on Grok showing zero difference. Without independent reruns and per-task paired outcomes, it remains unclear whether these narrow component-level gains exceed run-to-run execution noise.

  2. The comparison against Trace2Skill in Table 3 and Appendix Q is confounded by an unequal experience and execution budget. While Appendix Q restricts Trace2Skill to the first-execution trajectories from the 108 evolution tasks, ScienceCUA gathers substantial additional experience on those tasks through repeated history-guided explorations, candidate validations, live documentation browsing, and consolidation replays. Moreover, although Appendix K.6 provides a loose theoretical upper bound on executions and Appendices E and F report test and construction runtimes, the paper never reports the realized evolution-stage executions, model calls, tokens, or wall-clock time. As a result, the experiment cannot separate how much of ScienceCUA's advantage over Trace2Skill comes from its update algorithm versus simply collecting far more environment rollouts.

Questions for authors
  1. How many independent final evaluation runs produced Tables 1 through 3, and how much do paired task outcomes vary across repeats? Please provide repeat counts, per-task paired gains and losses, and confidence intervals for the main comparisons and evolution ablations, particularly for the small gains in the knowledge-addition ablation.

  2. What are the realized evolution-stage environment executions, model calls, token counts, and wall-clock runtimes for both ScienceCUA and Trace2Skill? To separate the value of the update procedure from the volume of collected experience, how does Trace2Skill perform when given a matched execution budget or access to all additional evolution trajectories gathered by ScienceCUA?

8 comments
AuthorsOpenReview ↗

Response to Weakness 1 and Question 1

Weakness 1

We are very grateful to the reviewer for their comments on experimental stability and statistical reporting. To further evaluate the stability of the main experimental results, during the rebuttal period we increased the number of independent final evaluations for Vanilla, RAG, VisualSkill, SkillFoundry, and ScienceCUA under Grok-4.6 (xhigh) in Table 1 to three per method. Each evaluation covered all 168 tasks in ScienceBoard-Verified, and the results are as follows.

MethodKAlgebraCelestiaGRASS GISChimeraXLean 4TeXstudio
Vanilla(56.99 ± 4.93)%(35.35 ± 1.75)%(13.73 ± 4.49)%(82.14 ± 0.00)%(11.54 ± 3.85)%(20.83 ± 3.61)%
RAG(68.82 ± 3.72)%(35.35 ± 4.63)%(19.61 ± 4.49)%(82.14 ± 3.57)%(10.26 ± 2.22)%(27.08 ± 7.22)%
VisualSkill(62.37 ± 1.86)%(38.38 ± 3.50)%(20.59 ± 2.94)%(78.57 ± 3.57)%(12.82 ± 2.22)%(29.17 ± 3.61)%
SkillFoundry(54.84 ± 8.53)%(38.38 ± 6.31)%(17.65 ± 2.94)%(84.52 ± 2.06)%(11.54 ± 0.00)%(33.33 ± 3.61)%
ScienceCUA (Stage 1)(72.04 ± 1.86)%(45.45 ± 3.03)%(25.49 ± 1.70)%(85.71 ± 0.00)%(12.82 ± 2.22)%(27.08 ± 3.61)%
MethodRun 1Run 2Run 3Mean ± sample standard deviation
Vanilla37.47%36.35%36.48%(36.76 ± 0.61)%
RAG41.50%37.87%42.26%(40.54 ± 2.35)%
VisualSkill40.42%42.72%37.81%(40.32 ± 2.46)%
SkillFoundry37.93%39.01%43.19%(40.04 ± 2.78)%
ScienceCUA (Stage 1)44.70%45.91%43.70%(44.77 ± 1.10)%

In addition, to evaluate the benefits of the second stage of our ScienceCUA, namely knowledge self-evolution, we increased the number of independent final evaluations for the initial knowledge base and Full ScienceCUA to three each under the Grok-4.6 (xhigh) setting. The results are as follows:

MethodRun 1Run 2Run 3Mean ± sample standard deviation
ScienceCUA (Stage 1)44.70%45.91%43.70%(44.77 ± 1.10)%
Full ScienceCUA54.59%52.55%55.66%(54.27 ± 1.58)%

For reference, the original ScienceBoard paper conducted three independent evaluations under the GPT-4o, a11ytree + screenshot setting, with identical success rates for the algebra tasks across all three runs. The average success rate for the biochemistry tasks was approximately 32.18%, with an error margin of ±3.25 percentage points, showing relatively small fluctuations between runs.

In addition, the reviewer pointed out that the scores for some software applications in the knowledge addition ablation under Grok were identical to those of Full. ScienceCUA classifies failure causes into existing knowledge requiring correction, insufficient knowledge coverage, and grounding errors, and triggers knowledge addition only when insufficient knowledge coverage is diagnosed. For KAlgebra, Celestia, and ChimeraX, this branch was never triggered throughout the evolution process. The knowledge extracted from the materials for these three software applications already covered the scope of knowledge required by the test tasks. Therefore, removing the knowledge addition mechanism did not affect the final evolution results for these three software applications. We conducted independent final evaluations for Full and the ablation configuration separately, and the two configurations achieved the same success rates on these four software applications. Tables 1‒3 in our original manuscript all report results from single runs.

Considering the execution time and API costs of real software tasks, we have not yet extended repeated evaluations to all models and ablation settings, and we earnestly ask for the reviewer’s understanding. Finally, we will incorporate all supplementary experiments from the rebuttal period into the revised manuscript.

Answer 1

The results in Tables 1‒3 are from a single independent evaluation. To assess the stability of the main experimental results, we have added experiments under Weakness1.

AuthorsOpenReview ↗

Response to Weakness 2 and Question 2 (Part 1/2)

Weakness 2 (Part 1/2)

We are very grateful to the reviewer for pointing out the ambiguity in the budget description. The phrase “the same run configuration and budget” in Section 5.3 refers to the three ScienceCUA settings―Full ScienceCUA, −Knowledge addition, and −Failure history―which use the same self-evolution hyperparameters. The wording in the paper could lead readers to believe that this also includes Trace2Skill; we apologize for this and will revise this sentence to clarify its meaning.

Our reproduction of Trace2Skill in the original manuscript followed a trajectory analysis and patch integration pipeline, using the trajectory from the first execution of each evolution task. To follow the method proposed in the original Trace2Skill paper, we also used only the first trajectory when reproducing Trace2Skill in the original manuscript.

To further ensure a fair comparison with similar budgets, we added Trace2Skill experiments in which each task was executed independently three times, matching ScienceCUA’s maximum number of exploration attempts. The resulting successful and failed trajectories were processed by their corresponding analysis modules, respectively, after which knowledge was integrated according to the original Combined configuration, retaining the default merge batch size of Bmerge=32B_{\mathrm{merge}} = 32.

Under the model-based evaluation setting with Grok-4.6 (xhigh), the final test success rates for each software application are as follows.

SoftwareFull ScienceCUATrace2Skill (three executions per task)
KAlgebra80.65%(67.74 ± 4.56)%
Celestia63.64%(54.55 ± 4.29)%
GRASS GIS47.06%(36.76 ± 2.08)%
ChimeraX85.71%(83.93 ± 2.53)%
Lean 419.23%(17.31 ± 2.72)%
TeXstudio31.25%(31.25 ± 8.84)%
Average54.59%(48.59 ± 1.90)%

Full ScienceCUA uses the results from Table 3 in the original manuscript. For Trace2Skill with three executions per task, after one round of knowledge base evolution, the same frozen knowledge base is independently evaluated twice on Verified, and the mean and sample standard deviation are reported.

The full evolution costs of the two methods are as follows. Each cell lists ScienceCUA / Trace2Skill with three executions per task, in that order.

Env. exec. = number of valid environment executions; Model tasks = number of model tasks involving non-environment logic; M = millions; Time (h) = cumulative effective runtime in hours.

SoftwareEnv. exec.Model tasksTokens (M)Time (h)
KAlgebra45 / 6031 / 6317.668 / 33.0778.861 / 13.880
Celestia51 / 6034 / 6336.440 / 43.12515.239 / 18.975
GRASS GIS58 / 6056 / 6373.713 / 62.60614.132 / 15.915
ChimeraX37 / 6026 / 6328.908 / 45.1728.314 / 13.836
Lean 458 / 6049 / 6336.954 / 82.94713.554 / 17.084
TeXstudio23 / 2417 / 2515.701 / 17.6609.505 / 9.057
Total272 / 324213 / 340209.384 / 284.58769.605 / 88.747

ScienceCUA’s environment execution count includes exploration, validation of candidate updates, submission validation after success, and regression runs after knowledge integration. Model tasks involving non-environment logic include error analysis, the addition and revision of Skills and References, and knowledge integration.

Trace2Skill’s environment execution count comes from three independent executions of each evolution task using the same initial knowledge base. Model tasks involving non-environment logic include separate analyses of successful and failed trajectories and integration of the knowledge updates obtained from these analyses.

Token usage covers all model calls throughout the complete evolution process, and cumulative elapsed time is the sum of the end-to-end durations of all tasks. The final Verified evaluation is not included in these statistics, and costs caused by infrastructure, API, and network failures have been excluded.

In this supplementary comparison, ScienceCUA’s average success rate was 6.00 percentage points higher, while its total number of environment executions was reduced by 16.05%, its number of model tasks involving non-environment logic by 37.35%, its token consumption by 26.43%, and its cumulative elapsed time by 21.57%. After the number of independent executions was increased, ScienceCUA still achieved a higher success rate and a lower total evolution cost.

AuthorsOpenReview ↗

Response to Weakness 2 and Question 2 (Part 2/2)

Weakness 2 (Part 2/2)

We further calculated the distribution of outcomes across the three independent executions of each evolution task for Trace2Skill.

SoftwareNumber of evolution tasksSuccess in all three executionsFailure in all three executionsMixed success and failure outcomes
KAlgebra201334
Celestia20965
GRASS GIS20686
ChimeraX201532
Lean 4205105
TeXstudio8242
Total108503424
Percentage of all tasks100%46.30%31.48%22.22%

Of the 108 tasks, 84 had consistent success or failure outcomes across the three executions, accounting for 77.78%. This indicates that, when the number of independent retries is increased under a fixed initial knowledge base, most tasks still produce the same success or failure outcomes. For tasks that continue to be affected by incorrect operational knowledge, independent retries may repeatedly encounter the same obstacles.

In contrast, ScienceCUA’s exploration guided by failure history can provide subsequent attempts with previous action paths, environment observations, and failure outcomes, helping the agent identify and correct version-mismatched knowledge.

Finally, we will incorporate all supplementary experimental results and cost statistics from the rebuttal period into the revised manuscript.

Answer 2

Statistics on the actual environment execution counts, numbers of model tasks involving non-environment logic, token consumption, and runtime for ScienceCUA and Trace2Skill (three executions per task) during the evolution stage have been added under Weakness 2. In addition, the reviewer suggested conducting experiments in which Trace2Skill is allowed to use all additional evolution trajectories collected by ScienceCUA. However, providing these additional trajectories to Trace2Skill would couple ScienceCUA’s exploration guided by failure history and its validation process after knowledge updates with Trace2Skill’s trajectory analysis and patch integration, so this hybrid setting would not be appropriate as a direct comparison between the two methods when run independently.

Reviewer_RbN7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces ScienceCUA, a two-stage framework for scientific computer-use agents that first distills raw software documentation into task-level procedural Skills and on-demand lookup References. To resolve version and coverage mismatches between static manuals and deployed environments, ScienceCUA further evolves this library through failure-history-guided exploration and environment-validated updates on a set of evolution tasks. Evaluated on the newly audited ScienceBoard-Verified and expanded ScienceBoard-VE benchmarks across six scientific applications, ScienceCUA improves average task success rates by up to 7.23% after initial construction and up to 17.12% after self-evolution.

Critical strengths
  1. Well-motivated problem formulation for scientific CUAs: Operating specialized scientific software requires precise workflows and command syntax, yet human-authored manuals are full of narrative noise and frequently suffer from version or coverage mismatches with deployed environments (Figure 1). The paper provides a clear and compelling motivation for why static document injection falls short and why documentation-derived knowledge must be iteratively refined through real execution feedback.

  2. Carefully audited and repaired ScienceBoard-Verified benchmark: The authors conduct a meticulous audit of all 169 original ScienceBoard tasks, systematically repairing 37 flawed tasks involving ambiguous instructions, mathematically incorrect reference answers, GUI-biased evaluators, and broken runtime environments (Appendix I). Extending the benchmark harness to support native agent loops (Codex and Grok Build) further increases its practical value for the CUA community.

  3. Effective decoupling of procedural Skills and declarative References: Separating task-level workflows ("know-how") from on-demand command and syntax catalogs ("know-what") keeps task guidance concise without losing fine-grained parameter constraints. Ablation studies (Table 2) and native agent call logs (Appendix G) clearly demonstrate that both resource types are actively invoked and play complementary roles across the six scientific applications.

Critical weaknesses
  1. Implicit operational overlap between the evolution set EE and test set VV: After manually auditing and modifying the 168 test tasks and their evaluators in VV (e.g., updating GRASS GIS evaluators B-03–B-10 to accept d.vect console commands and fixing KAlgebra sphere task C-14), the same authors manually constructed the 108 evolution tasks in EE to "use the same types of software operations as the test tasks" (Appendix J). Consequently, EE acts as a near-clone of the test task templates, and the headline self-evolution gains (such as learning d.vect styling in B-18 or implicit sphere plotting in C-19) align closely with the exact operations and evaluator modifications introduced in VV.

  2. Unfair interaction budget in the comparison with Trace2Skill: In Section 5.3, the paper claims that all evolution variants share the "same runtime configurations and budgets," yet Appendix Q reveals that Trace2Skill is restricted solely to the static first-execution trajectories (1 episode per task) without any re-exploration or live documentation access. In contrast, ScienceCUA is granted up to 2Mexplore+Msubmit=92M_{\text{explore}} + M_{\text{submit}} = 9 environment executions per task (plus up to 4 consolidation replay cycles and live documentation browsing), making it impossible to separate algorithmic superiority from the massive advantage in interaction budget.

  3. Higher execution time overhead compared to retrieval and skill baselines: While the main text focuses on how ScienceCUA reduces wall-clock time relative to full-document Direct Injection, Appendix E (Figure 4) reveals that ScienceCUA (Ours) is actually slower than Vanilla, RAG, VisualSkill, and SkillFoundry on three out of the four evaluated models (e.g., 28h55m vs. 24h08m for RAG on Qwen3.8-Max, and 19h16m vs. 15h26m for RAG on Grok 4.6). Furthermore, the main text omits the substantial additional wall-clock and token overhead required to run the multi-episode self-evolution and regression loops across the 108 evolution tasks.

Questions for authors

See the weakness.

7 comments
AuthorsOpenReview ↗

Response to Weakness 1

Weakness 1

We are very grateful to the reviewer for raising the issue of implicit operational overlap between the evolution tasks and the test tasks. We have added Appendix R to the revised manuscript to examine two questions using KAlgebra as an example. First, can a Skill or Reference revised during evolution transfer across task templates? Second, can the revised operational guidance be reused in different workflows?

Appendix R.1 distinguishes between task categories and reusable atomic functions. For example, different categories of tasks, including calculation, 2D plotting, and 3D plotting, can all use the atomic function of script import, while the operations required after import differ. This distinction helps us identify which specific operational capability the evolution tasks and the test tasks share.

As shown in Appendix M, the evolution task C-19 and the test task C-14 mentioned by the reviewer do indeed both involve plotting a sphere. However, the similarity between these two tasks alone cannot establish whether the revised Skill can only transfer to test tasks that follow the same template, such as tasks that only replace parameters or objects. To examine this, in Appendix R.2, we conducted single-task evolution separately using the task of plotting a torus (E-01) and the task of plotting two intersecting planes (E-02). Each task started from the same initial knowledge base, and both exposed an incorrect restriction on input forms in the initial 3D Skill during execution. The two evolution runs respectively validated the two approaches of plotting with a function of three variables and directly entering an implicit equation containing z (Table 17 and Figure 10). When using the knowledge base constructed with Resource-to-Skill, the model received incorrect guidance from the 3D plotting Skill extracted from the old version of the manual on all three tasks involving a hyperboloid of one sheet (A-03), a sphere (C-14), and an oblique cylinder (C-17), and failed all three tasks (Appendix M). After replacing it separately with the 3D Skill independently revised through E-01 and the one independently revised through E-02, all six transfer evaluations passed (Appendix R.2, Table 18, and Figure 11). The two evolution tasks and the three test tasks share the atomic function of implicit 3D plotting, but do not use the same task template, and their differences are not limited to replacing parameters or objects.

Appendix R.3 further examines the reuse of the same 3D plotting Skill in different workflows. Since the existing KAlgebra test tasks lack cases that combine 3D plotting with multiple other software operations, we added two tasks for this investigation. W-01 links script import, calculation and assignment, and double-cone plotting in sequence (Figure 12). W-02 links parameter import, sphere plotting, and Lines style configuration in sequence (Figure 13). Using the fully evolved knowledge base, the agent completed both tasks, showing how the guidance for implicit 3D plotting can be reused at the corresponding stages in different combinations of operations.

We thank the reviewer again for raising the issue of possible operational overlap between the evolution tasks and the test tasks! Since it is indeed difficult to use the same set of criteria to quantify the degree of operational overlap and task template similarity between the evolution and test sets across different software applications, and the time available for the rebuttal is limited, this supplementary analysis focuses on a case study of KAlgebra. The above cases show that a Skill or Reference revised and validated for a particular atomic function can transfer to different tasks requiring the same function. In composite workflows, the agent can also reuse the same piece of knowledge at the stage where that function is needed and combine that function with other operations according to the task requirements.

AuthorsOpenReview ↗

Response to Weakness 2 (Part 1/2)

Weakness 2 (Part 1/2)

We are very grateful to the reviewer for pointing out the ambiguity in the budget description. The phrase “the same run configuration and budget” in Section 5.3 means that the three ablation experiments, Full ScienceCUA, −Knowledge addition, and −Failure history, used the same self-evolution hyperparameters. The wording in the paper could lead readers to believe that this also includes Trace2Skill; we apologize for this and will revise this sentence to clarify its meaning.

Our reproduction of Trace2Skill in the original manuscript followed a trajectory analysis and patch integration pipeline, using the trajectory from the first execution of each evolution task. To follow the method proposed in the original Trace2Skill paper, we also used only the first trajectory when reproducing Trace2Skill in the original manuscript.

To further ensure a fair comparison with similar budgets, we added Trace2Skill experiments in which each task was executed independently three times, thereby matching ScienceCUA’s maximum number of exploration attempts. The resulting successful and failed trajectories were processed by their corresponding analysis modules, respectively, after which knowledge was integrated according to the original Combined configuration, retaining the default merge batch size of Bmerge=32B_{\mathrm{merge}} = 32.

Under the model-based evaluation setting with Grok-4.6 (xhigh), the final test success rates for each software application are as follows.

SoftwareFull ScienceCUATrace2Skill (three executions per task)
KAlgebra80.65%(67.74 ± 4.56)%
Celestia63.64%(54.55 ± 4.29)%
GRASS GIS47.06%(36.76 ± 2.08)%
ChimeraX85.71%(83.93 ± 2.53)%
Lean 419.23%(17.31 ± 2.72)%
TeXstudio31.25%(31.25 ± 8.84)%
Average54.59%(48.59 ± 1.90)%

Full ScienceCUA uses the results from Table 3 in the original manuscript. For Trace2Skill with three executions per task, after one round of knowledge base evolution, the same frozen knowledge base is independently evaluated twice on Verified, and the mean and sample standard deviation are reported.

The full evolution costs of the two methods are as follows. Each cell lists ScienceCUA / Trace2Skill with three executions per task, in that order.

Env. exec. = number of valid environment executions; Model tasks = number of model tasks involving non-environment logic; M = millions; Time (h) = cumulative effective runtime in hours.

SoftwareEnv. exec.Model tasksTokens (M)Time (h)
KAlgebra45 / 6031 / 6317.668 / 33.0778.861 / 13.880
Celestia51 / 6034 / 6336.440 / 43.12515.239 / 18.975
GRASS GIS58 / 6056 / 6373.713 / 62.60614.132 / 15.915
ChimeraX37 / 6026 / 6328.908 / 45.1728.314 / 13.836
Lean 458 / 6049 / 6336.954 / 82.94713.554 / 17.084
TeXstudio23 / 2417 / 2515.701 / 17.6609.505 / 9.057
Total272 / 324213 / 340209.384 / 284.58769.605 / 88.747

ScienceCUA’s environment execution count includes exploration, validation of candidate updates, submission validation after success, and regression runs after knowledge integration. Model tasks involving non-environment logic include error analysis, the addition and revision of Skills and References, and knowledge integration.

Trace2Skill’s environment execution count comes from three independent executions of each evolution task using the same initial knowledge base. Model tasks involving non-environment logic include separate analyses of successful and failed trajectories and integration of the knowledge updates obtained from these analyses.

Token usage covers all model calls throughout the complete evolution process, and cumulative elapsed time is the sum of the end-to-end durations of all tasks. The final Verified evaluation is not included in these statistics, and costs caused by infrastructure, API, and network failures have been excluded.

In this supplementary comparison, ScienceCUA’s average success rate was 6.00 percentage points higher, while its total number of environment executions was reduced by 16.05%, its number of model tasks involving non-environment logic by 37.35%, its token consumption by 26.43%, and its cumulative elapsed time by 21.57%. After the number of independent executions was increased, ScienceCUA still achieved a higher success rate and a lower total evolution cost.

AuthorsOpenReview ↗

Response to Weakness 2 (Part 2/2)

Weakness 2 (Part 2/2)

We further calculated the distribution of outcomes across the three independent executions of each evolution task for Trace2Skill.

SoftwareNumber of evolution tasksSuccess in all three executionsFailure in all three executionsMixed success and failure outcomes
KAlgebra201334
Celestia20965
GRASS GIS20686
ChimeraX201532
Lean 4205105
TeXstudio8242
Total108503424
Percentage of all tasks100%46.30%31.48%22.22%

Of the 108 tasks, 84 had consistent success or failure outcomes across the three executions, accounting for 77.78%. This indicates that, when the number of independent retries is increased under a fixed initial knowledge base, most tasks still produce the same success or failure outcomes. For tasks that continue to be affected by incorrect operational knowledge, independent retries may repeatedly encounter the same obstacles.

In contrast, ScienceCUA’s exploration guided by failure history can provide subsequent attempts with previous action paths, environment observations, and failure outcomes, helping the agent identify and correct version-mismatched knowledge. ScienceCUA achieved better results with a smaller budget, indicating that the gains from ScienceCUA’s self-evolution method do not come solely from advantages in interaction.

Finally, we will incorporate all supplementary experimental results and cost statistics from the rebuttal period into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.