- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes BioMaster, a framework that uses scientific execution experience for both model post-training and external experience reuse. Validated trajectories provide supervised training data, while structured Research Units (RUs) preserve methods, applicability conditions, dependencies, and verification evidence for retrieval. The framework also uses failure diagnosis to guide additional task construction. Experiments compare complete systems under a fixed backbone and different models under a fixed agent harness. A component study on 203 tasks suggests that post-training increases the benefit of using a fixed RU library.
Critical strengths
-
The interaction between post-training and experience reuse is interesting. With the harness and RU library fixed, RU access changes performance by -1.31 percentage points for the base model and +4.27 points for the post-trained model. This comparison provides useful evidence that the value of external experience depends on the capabilities of the model using it.
-
The RU representation addresses practical requirements of scientific workflows. Retaining input conditions, execution dependencies, validation evidence, and known failures makes the stored experience more useful for adaptation and verification. The system-level and model-level evaluations also help distinguish improvements from the execution framework and those from model training.
Critical weaknesses
-
The evidence for failure-guided updating is unclear. In Table 21, random expansion outperforms failure-diagnosis expansion both without RUs (59.44 vs. 57.31) and with RUs (63.71 vs. 61.74). Moreover, the random-expansion scores exactly match the final v2 scores in Figure 4, although v2 is described as a failure-guided update. This makes it difficult to determine which checkpoint supports the main conclusions and whether failure-guided task construction provides an advantage over ordinary data expansion.
-
The separation between training, RU construction, failure diagnosis, and final evaluation is insufficiently specified. Appendix A.3 explains how the 203 evaluation tasks are sampled, but does not clearly establish their exclusion from training trajectories and RU sources. It is also unclear whether the tasks used for failure diagnosis are separate from the final test set. These details are necessary to assess whether the reported gains demonstrate transfer to unseen tasks.
Questions for authors
-
Which checkpoint is reported as BioMaster-A1-v2 in Figure 4: the failure-diagnosis version or the random-expansion version? How should the results in Table 21 be reconciled with the failure-guided update described in the main text?
-
Were all 203 evaluation tasks and their execution records excluded from both SFT data and RU construction? Please clarify how related tasks from the same paper or workflow were handled, and whether failure diagnosis used a separate development set.
-
Please clarify the training sequence. Section 3.3 and Table 15 report 6,264 Stage-2 trajectories, while Table 18 reports 4,264. Section C.4 states that v2 starts from v1, but Table 19 lists the Stage-1 checkpoint as its initialization. Which configuration was actually used?
4 comments
How are the 203 tasks separated from training and RU construction, and used for failure diagnosis?
Q2. How are the 203 tasks separated from training and RU construction, and used for failure diagnosis?
We clarify both the use of evaluation content and the workflow-level feedback passed to task construction. Performance on the 203 tasks identifies workflow clusters requiring improvement, but their prompts, answers, and execution traces are not supplied to new-task construction, SFT, or RU construction.
1. How failure feedback enters task construction
There are two steps. First, v1 executes the 203 component tasks, and its performance identifies workflow categories needing additional training experience. The task-construction module then receives only the selected cluster IDs to specify the categories for new tasks and experience collection.
The IDs indicate which workflows should receive more experience. New tasks must still be constructed, executed, and validated before their records become SFT or RU-extraction material.
| Stage | Material used | Relationship to the 203 component tasks |
|---|---|---|
| SFT | Validated scientific execution trajectories from training tasks | Excludes these 203 tasks, their answers, and execution records |
| RU construction | Methods, applicability conditions, and validation experience extracted from source execution records | No RUs are extracted from these tasks' answers or execution traces |
| Failure diagnosis and task expansion | v1's performance on the 203 tasks and the workflow cluster IDs selected accordingly | New-task construction receives only selected cluster IDs, not original task prompts, reference answers, or execution traces |
| Component evaluation | The same 203 tasks and their evaluation materials | Compares model and RU configurations within the workflow-feedback setting |
The 203 tasks therefore inform the direction of expansion, while the material used to update parameters and construct RUs comes from training-task executions. We will explicitly describe this dual role of workflow feedback and component evaluation, together with the information passed between stages.
2. Workflow coverage and task provenance
Appendix A.3's “one task per selected cluster” sampling controls component-evaluation coverage and weighting; it does not hold out entire workflow clusters. Our partition permits training and evaluation tasks to share general workflow types.
A scientific task can combine cell annotation, differential expression, and pathway enrichment, and the same method can address different biological questions. For example, comparing monocytes from infected versus healthy donors and comparing CD8+ T cells before versus after treatment can both use differential expression. Each still requires analysis and validation using its own data, target population, and comparison design. RUs provide methodological guidance; task-specific results depend on the task's inputs and execution.
For tasks linked through the same paper, data entity, or derivation, we will examine input instances, scientific objectives, required outputs, and associated execution records. The supplement will list all 203 tasks' IDs and prompts, source papers or original source IDs, input-data and sample identifiers, parent-task and derivation relationships, RU sources, and workflow memberships, specifying what related tasks share.
We will also present representative evaluation tasks alongside the closest training tasks and RU source records to distinguish shared methods, common sources, and reuse of specific problems or solutions. Different task IDs or wording do not establish independence, and shared workflows do not by themselves establish task duplication.
3. Gains on other public task sets
Beyond the 203-task component comparison, Figure 4(a) reports v1 and final-v2 results on three public benchmarks:
| Public benchmark | BioMaster-A1-v1 | BioMaster-A1-v2-r | Improvement (pp) |
|---|---|---|---|
| STELLA Simple Set | 83.53 | 85.19 | +1.66 |
| Biomni-Eval1 | 47.11 | 53.66 | +6.55 |
| BioMystery | 49.63 | 50.37 | +0.74 |
| Equal-weight average | 60.09 | 63.07 | +2.98 |
Final v2 improves over v1 on all three benchmarks, raising the mean by 2.98 pp. The observed gains therefore extend beyond the 203 tasks used for workflow selection. Component results characterize model and RU performance within the feedback setting; the public benchmarks provide additional evidence of the training update's benefits on other scientific task sets.
We will align Appendix A.3, Appendix C.4, and the configuration tables so that task provenance, workflow feedback, new-experience construction, and evaluation can be traced consistently.
We hope these clarifications and corrected results address your concerns and would appreciate your reconsideration of the assessment and score. We welcome further discussion and are happy to revise the manuscript in response to any remaining questions or suggestions.
Follow-up on Failure-Guided Expansion Across Three Public Benchmarks
To evaluate generalization on tasks separate from the 203-task feedback set, we added three direct comparisons between failure-guided and random expansion, one on each public benchmark.
We used the fixed-Harness model-level protocol from Figure 3. Both variants started from the same Stage-1 checkpoint and added 2,000 trajectories, collected with the same teacher, to the same initial training set.
| Benchmark | Random expansion (v2-d) | Failure-guided expansion (v2-r) | v2-r minus v2-d |
|---|---|---|---|
| STELLA Simple Set | 84.51 | 85.19 | +0.68 pp |
| Biomni-Eval1 | 51.96 | 53.66 | +1.69 pp |
| BioMystery v11 | 47.41 | 50.37 | +2.96 pp |
Failure-guided expansion scored higher on all three benchmarks, with the largest difference on BioMystery. The Biomni-Eval1 difference is calculated before rounding.
Prior work has used model failures to guide data generation [1, 2], compared failure-driven and random sampling at a matched data budget [3], and evaluated failure-guided training on a separate benchmark [4]. In BioMaster, the 203 component tasks inform which workflow clusters to expand. The task generator receives only the selected cluster IDs. The original task prompts, reference answers, and execution traces are excluded from new task construction, supervised fine-tuning, and RU extraction (Appendix A.3). This lets us use workflow-level feedback while keeping the original evaluation materials out of the training pipeline.
We will include these comparisons and clarify the feedback protocol in the revised manuscript. We would appreciate hearing whether the new results address your concerns, and we would be happy to discuss any remaining questions.
References
- He et al. (2023). Targeted Data Generation: Finding and Fixing Model Weaknesses. ACL.
- Stan et al. (2026). Learning from Reasoning Failures via Synthetic Data Generation. AAAI.
- Hao et al. (2026). HardGen: Generating Hard Samples for Tool-use Agents. EMNLP.
- Kang et al. (2026). TRACE: Capability-Targeted Agentic Training. ICML 2026 AIWILD Workshop.
Looking forward to further discussion
Dear Reviewer,
We are looking forward to your feedback and would greatly appreciate any further questions or concerns you may have.
If there are additional issues that require clarification or further experiments, an earlier response would help us better address them within the remaining rebuttal period.
Thank you again for your time and consideration.
Corrected v2 Attribution and Training Configuration
Thank you for your careful review. Your comments have helped us clarify the checkpoint mapping, data flow, and training procedure. We address Q1 and Q3 below, followed by Q2 in our second reply.
Q1. Is the v2 model in Figure 4 the random-expansion or failure-guided variant?
Figure 4 reports BioMaster-A1-v2-r, the failure-diagnosis-guided variant. The suffix r denotes reflection-guided expansion; d denotes direct random expansion without failure diagnosis. Some appendix descriptions and plot labels reversed this mapping, causing the failure-guided results to be interpreted as random-expansion results. The correct mapping is:
Failure-guided expansion exceeds random expansion by 2.13 pp without RU and 1.97 pp with RU. Figure 4's scores of 59.44 and 63.71 both belong to v2-r, consistent with the final model described in the main text. The correction concerns strategy labels and model attribution; the numerical scores are unchanged.
Both variants start with the same 4,264 initial scientific trajectories, independently add 2,000 trajectories, and use full-parameter SFT on 6,264 trajectories from the same Stage-1 checkpoint. They are separate training branches. v1 provides diagnostic feedback, not their initial parameters.
The additional data differ in task selection. Random expansion does not target v1's failures. Failure-guided expansion uses v1's performance on the 203 component tasks to select workflow clusters requiring further training. Only the selected cluster IDs are passed to task construction; the original evaluation prompts, answers, and execution traces are not supplied. Newly collected trajectories are validated before training.
The corrected results therefore show higher mean component scores for failure-guided expansion in both RU conditions, at matched trajectory count, initialization, and parameter-update method, within this workflow-feedback setting.
We will correct Figure 4, original Table 21, Appendix C.4, and the associated plots, explicitly stating BioMaster-A1-v2 = BioMaster-A1-v2-r, so that model names, training datasets, and scores correspond consistently.
Q3. How should the 4,264/6,264 trajectory counts and v2 initialization be reconciled?
4,264 is the initial scientific dataset used for v1; 6,264 is the expanded dataset used for each v2 variant. All three models initialize from the same Stage-1 checkpoint.
1. Dataset versions and counting convention
Counts refer to filtered execution trajectories actually included in scientific-task SFT:
Each v2 branch retains all 4,264 initial trajectories and combines them with its own 2,000 additional trajectories for full-parameter SFT. The branches use different expansion data; their additions are not cumulative. Stage 1's 17,332 Harness-adaptation execution examples are counted separately. The two scientific-data counts therefore use the same convention, with the difference corresponding to 2,000 additional trajectories.
2. Feedback sequence and parameter initialization
v1's workflow-level feedback guides task construction for the diagnosis-guided branch. Once the expanded dataset is ready, v2 restarts scientific-task training from the shared Stage-1 checkpoint.
Thus, v1 precedes v2 in the data-construction process, while their parameter training starts from the same checkpoint. The original wording “continued training from v1” is inaccurate. “Stage 3” denotes failure feedback, data expansion, and retraining, rather than a third consecutive optimization stage starting from v1.
The v1-to-v2 comparison captures the overall effect of the expanded dataset. Q1's comparison of v2-r and v2-d, both using 6,264 trajectories and the same initialization, further compares the two expansion strategies.
3. Manuscript corrections
We will use 4,264 consistently for v1 in Section 3.3, Table 15, and related descriptions, and reserve 6,264 for the expanded v2 datasets. Table 18's count of 4,264 and Table 19's Stage-1 initialization match the actual configuration; contradictory descriptions of continued training in Appendix C.4 will be corrected.