- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ScienceCUA, a framework that converts scientific software documentation into reusable Skills and References. Skills describe operational procedures, while References provide commands, syntax, and other lookup information. The framework then uses execution feedback to revise or add knowledge, with candidate updates validated in fresh sessions.
The authors also correct issues in ScienceBoard and introduce separate tasks for skill evolution. Experiments across six scientific applications show that the initial knowledge library improves task success, and execution-guided evolution provides further gains.
Critical strengths
1. A practical approach to organizing software knowledge.
The paper addresses a practical problem: software documentation can be long, incomplete for a task, or inconsistent with the installed version. Separating operational procedures from command and syntax references is a reasonable design. Table 2 shows that combining Skills and References performs better on average than using either alone in both evaluated settings.
2. Performance improvements across multiple settings.
The experiments show benefits across multiple models and applications. In Table 3, evolution improves average success from 44.70% to 54.59% for Grok-4.6 and from 57.16% to 65.81% for GPT-5.5. The documented benchmark corrections are also useful, since ambiguous instructions and overly restrictive evaluators can make agent results difficult to interpret.
Critical weaknesses
1. Limited evidence for transfer to different workflows.
The scope of the demonstrated transfer is somewhat limited. Section 4.2 and Appendix J explain that evolution tasks retain the same operation types as test tasks while changing objects, parameters, or requirements. The manual deduplication is useful, but this setup mainly tests transfer within familiar operation families. The plotting and map-styling examples also follow this pattern. It remains unclear how much improvement extends to tasks requiring different workflows or combinations of operations.
2. Incomplete cost comparison for self-evolution.
The cost and budget comparison for self-evolution needs more detail. Appendices E and F report execution time and initial library construction time, while Appendix K gives upper bounds for evolution executions. The actual evolution cost is not reported. Appendix Q also states that Trace2Skill uses the same first-execution trajectories, whereas ScienceCUA obtains additional feedback through re-exploration and validation. This leaves uncertainty about how much of the advantage comes from the update procedure versus additional environment interaction.
Questions for authors
1. Scope of skill transfer.
Could the authors summarize the similarity between evolution and test tasks at the operation or workflow level? Reporting gains separately for closely related tasks and tasks requiring different combinations of operations would help clarify the scope of transfer.
2. Actual evolution cost and budget matching.
Could the authors report the actual environment execution counts, token usage, and wall-clock costs for ScienceCUA and Trace2Skill? How were interaction budgets matched given their different exploration and validation procedures? This would help interpret the performance advantage in Table 3.
3. Stability of the reported improvements.
How many independent runs were used for the main results? Could the authors provide variability estimates for the comparison between the initial library and Full ScienceCUA? Please also clarify whether the same constructed libraries were reused across runs.
6 comments
Response to Weakness 2 and Question 2 (Part 1/2)
Weakness 2 (Part 1/2)
We are very grateful to the reviewer for suggesting that we report actual costs and provide a more thorough comparison of the execution budgets of the two methods. First, the single-trajectory setting in the original manuscript followed the experimental configuration of the original Trace2Skill paper. Section 3.1 explicitly states that one trajectory is generated for each task during the trajectory generation stage, after which the success and error analysis modules separately propose patches, followed by hierarchical integration. We reproduced Trace2Skill in the original manuscript following the settings of the original Trace2Skill paper, so each evolution task was executed once during trajectory collection, and the resulting trajectories were then used for analysis and knowledge updates.
To make a fair comparison, we expanded Trace2Skill’s trajectory collection to three independent executions per task. All successful and failed trajectories were passed to the corresponding success or failure modules for processing, after which knowledge was integrated following the original Combined process, retaining the default merge batch size of .
Under the model-based setting with Grok-4.6 (xhigh), the actual costs of the two methods are as follows. Each cell lists ScienceCUA / Trace2Skill with three executions per task, in that order.
Env. exec. = number of valid environment executions; Model tasks = number of model tasks involving non-environment logic; M = millions; Time (h) = cumulative effective runtime in hours.
| Software | Env. exec. | Model tasks | Tokens (M) | Time (h) |
|---|---|---|---|---|
| KAlgebra | 45 / 60 | 31 / 63 | 17.668 / 33.077 | 8.861 / 13.880 |
| Celestia | 51 / 60 | 34 / 63 | 36.440 / 43.125 | 15.239 / 18.975 |
| GRASS GIS | 58 / 60 | 56 / 63 | 73.713 / 62.606 | 14.132 / 15.915 |
| ChimeraX | 37 / 60 | 26 / 63 | 28.908 / 45.172 | 8.314 / 13.836 |
| Lean 4 | 58 / 60 | 49 / 63 | 36.954 / 82.947 | 13.554 / 17.084 |
| TeXstudio | 23 / 24 | 17 / 25 | 15.701 / 17.660 | 9.505 / 9.057 |
| Total | 272 / 324 | 213 / 340 | 209.384 / 284.587 | 69.605 / 88.747 |
ScienceCUA’s environment executions include exploration, candidate validation, submission validation after success, and regression runs after integration. Its count of model tasks involving non-environment logic includes error analysis, knowledge writing, the addition or modification of Skills and References, and knowledge integration.
The environment executions of Trace2Skill (the three-execution version) include three independent runs of each evolution task. Its count of model tasks involving non-environment logic includes analysis of successful and failed trajectories, generation of the corresponding knowledge updates, and final knowledge integration.
Token usage covers all model calls throughout the complete evolution process, and elapsed time is the sum of the end-to-end durations of all tasks. The final Verified evaluation is accounted for separately, and costs caused by infrastructure, API, and network failures have been excluded.
After completing one round of knowledge base evolution, we independently evaluated the same frozen Trace2Skill knowledge base twice on Verified. Full ScienceCUA uses the results from Table 3 in the original manuscript. The results are as follows.
| Software | Full ScienceCUA | Trace2Skill (three executions per task) |
|---|---|---|
| KAlgebra | 80.65% | (67.74 ± 4.56)% |
| Celestia | 63.64% | (54.55 ± 4.29)% |
| GRASS GIS | 47.06% | (36.76 ± 2.08)% |
| ChimeraX | 85.71% | (83.93 ± 2.53)% |
| Lean 4 | 19.23% | (17.31 ± 2.72)% |
| TeXstudio | 31.25% | (31.25 ± 8.84)% |
| Average | 54.59% | (48.59 ± 1.90)% |
In this supplementary comparison, ScienceCUA’s average success rate was 6.00 percentage points higher, while its total number of environment executions was reduced by 16.05%, its number of model tasks involving non-environment logic by 37.35%, its token consumption by 26.43%, and its cumulative elapsed time by 21.57%. After the number of independent executions was increased, ScienceCUA still achieved a higher success rate and a lower total evolution cost.
Response to Weakness 2 and Question 2 (Part 2/2)
Weakness 2 (Part 2/2)
We further calculated the distribution of outcomes across the three independent executions of each evolution task for Trace2Skill.
| Software | Number of evolution tasks | Success in all three executions | Failure in all three executions | Mixed success and failure outcomes |
|---|---|---|---|---|
| KAlgebra | 20 | 13 | 3 | 4 |
| Celestia | 20 | 9 | 6 | 5 |
| GRASS GIS | 20 | 6 | 8 | 6 |
| ChimeraX | 20 | 15 | 3 | 2 |
| Lean 4 | 20 | 5 | 10 | 5 |
| TeXstudio | 8 | 2 | 4 | 2 |
| Total | 108 | 50 | 34 | 24 |
| Percentage of all tasks | 100% | 46.30% | 31.48% | 22.22% |
Of the 108 tasks, 84 had consistent success or failure outcomes across the three executions, accounting for 77.78%. This indicates that, when the number of independent retries is increased under a fixed initial knowledge base, most tasks still produce the same success or failure outcomes. For tasks that continue to be affected by incorrect operational knowledge, independent retries may repeatedly encounter the same obstacles.
In contrast, ScienceCUA’s exploration guided by failure history can provide subsequent attempts with previous action paths, environment observations, and failure outcomes, helping the agent identify and correct version-mismatched knowledge. ScienceCUA achieved better results with a smaller budget, indicating that the gains from ScienceCUA’s self-evolution method do not come solely from advantages in interaction.
Finally, we will incorporate all supplementary experimental results and cost statistics from the rebuttal period into the revised manuscript.
Answer 2
We have already reported the actual environment execution counts, non-environment model calls, token consumption, and actual elapsed time for ScienceCUA and Trace2Skill under Weakness 2. Please refer to Weakness 2 for the specific data.
Response to Weakness 1 and Question 1
Weakness 1
We are very grateful to the reviewer for raising the question of whether the evolution tasks and the test tasks are too similar. We have added Appendix R to the revised manuscript to examine two questions using KAlgebra as an example. First, can a Skill or Reference revised during evolution transfer across task templates? Second, can the revised operational guidance be reused in different workflows?
Appendix R.1 distinguishes between task categories and reusable atomic functions. For example, different categories of tasks, including calculation, 2D plotting, and 3D plotting, can all use the atomic function of script import, while the operations required after import differ. This distinction helps us identify which specific operational capability the evolution tasks and the test tasks share.
To test whether transfer depends on using the same task template, in Appendix R.2, we conducted single-task evolution separately using the task of plotting a torus (E-01) and the task of plotting two intersecting planes (E-02). Each task started from the same initial knowledge base, and both exposed an incorrect restriction on input forms in the initial 3D Skill during execution. The two evolution runs respectively validated the two approaches of plotting with a function of three variables and directly entering an implicit equation containing
z(Table 17 and Figure 10). When using the knowledge base constructed with Resource-to-Skill, the model received incorrect guidance from the 3D plotting Skill extracted from the old version of the manual on all three tasks involving a hyperboloid of one sheet (A-03), a sphere (C-14), and an oblique cylinder (C-17), and failed all three tasks (Appendix M). After replacing it separately with the 3D Skill independently revised through E-01 and the one independently revised through E-02, all six transfer evaluations passed (Appendix R.2, Table 18, and Figure 11). The two evolution tasks and the three test tasks share the atomic function of implicit 3D plotting, but do not use the same task template, and their differences are not limited to replacing parameters or objects.Appendix R.3 further examines the reuse of the same 3D plotting Skill in different workflows. Since the existing KAlgebra test tasks lack cases that combine 3D plotting with multiple other software operations, we added two tasks for this investigation. W-01 links script import, calculation and assignment, and double-cone plotting in sequence (Figure 12). W-02 links parameter import, sphere plotting, and Lines style configuration in sequence (Figure 13). Using the fully evolved knowledge base, the agent completed both tasks, showing how the guidance for implicit 3D plotting can be reused at the corresponding stages in different combinations of operations.
Answer 1
We are very grateful to the reviewer for suggesting that transfer gains be reported in groups based on the similarity of operation combinations. Since it is indeed difficult to use the same set of criteria to quantify the degree of operational overlap and task template similarity between the evolution and test sets across different software applications, and the time available for the rebuttal is limited, this supplementary analysis focuses on a case study of KAlgebra. The above cases show that a Skill or Reference revised and validated for a particular atomic function can transfer to different tasks requiring the same function. In composite workflows, the agent can also reuse the same piece of knowledge at the stage where that function is needed and combine that function with other operations according to the task requirements.