- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
BioCausalX turns 43 published executable biological models into budgeted discovery tasks with biological descriptions and 168 source-supported intervention axes. Agents receive descriptions and simulated observations while equations, exact action bindings, and reference graphs remain hidden. The recovery target is the original model’s directed dynamical dependencies among declared states, excluding self-edges. The paper evaluates classical and LLM inference, context changes, experimental acquisition, and GRPO training on held-out systems. This is a model-grounded environment; its reference graphs describe the implemented models rather than complete biological networks.
Critical strengths
- The environment combines heterogeneous published mechanisms with a precise model-level target. Linking declared biological states, executable operations, and direct-dependency scoring provides a useful basis for reproducible evaluation. The explicit limits of the graph target and operation mappings help distinguish recovery of the implemented dynamics from broader biological knowledge.
Critical weaknesses
-
The empirical consequence of the added grounding layer is not established relative to prior environments. SciGym already supports discovery using published biological models and perturbations, and BioCausalX builds on the CausalDynamics/causalx codebase. The combination of source-supported descriptions, action mappings, and a common target is concrete, but I need a comparison showing what the added layers change in task construction or agent behavior. An example explains the workflow; it does not establish that comparative contribution.
-
The results do not sufficiently separate structural learning from prior knowledge and submission validity. Biological-context recovery can include prior or source-specific knowledge, and observation-versus-intervention comparisons do not replace a context-only, zero-trajectory control. In the GRPO study, Directed F1 rises from 0.235 to 0.448 with context while validity rises from 50.0% to 97.7%; without context, F1 rises from 0.092 to 0.212 while validity rises from 34.1% to 100%. Invalid-as-zero scoring is a meaningful end-to-end measure, but these changes do not isolate improvement in graph recovery among valid submissions. Both separations are needed before interpreting the results as learning mechanisms from newly acquired evidence.
Questions for authors
- Can you provide a same-source comparison or ablation that isolates the source-supported operation and mapping layers, and quantify changes in executable action coverage, task behavior, or agent decisions? A complete, inspectable operation record would also clarify mapping correctness : )
- How does recovery compare with a biological-context-only, zero-trajectory baseline? Can you identify a setting where the acquired trajectories distinguish mechanisms that the supplied prior information alone cannot resolve?
- I encourage the authors to deepen the existing SciGym [1] comparison with one shared-source example, explaining how the allowed operations, hidden execution mappings, and recovery targets differ. This would help readers see what task behavior the added grounding layer enables beyond an existing biological dry lab.
- Please report Directed F1 conditional on valid submissions, with per-system coverage at the initial and final GRPO checkpoints, or use a common formatting control. How much of the gain remains after separating submission validity from structural accuracy?
[1] Duan, H., Lu, S., Harrigan, C.F., Desai, N., Lu, J., Koziarski, M., Cotta, L. and Maddison, C., 2026. Measuring Scientific Capabilities of Language Models with a Systems Biology Dry Lab. Advances in Neural Information Processing Systems, 38.
4 comments
Response 2/2
Q2 How does recovery compare with a biological-context-only, zero-trajectory baseline? Can you identify a setting where the acquired trajectories distinguish mechanisms that the supplied prior information alone cannot resolve?
Response:
Context-only versus agentic evidence use. The zero-trajectory control uses the same biological context and operation catalog on 43 systems × 3 trials, with every trajectory request rejected:
| Model | Context-only | Agentic interactive |
|---|---|---|
| Terra | 0.758 | 0.765 |
| Claude | 0.642 | 0.546 |
| Gemini | 0.528 | 0.551 |
Context-only recovery is already strong, while the benefit of interaction depends on the model and context. A biological-network review notes that current interaction maps remain incomplete and noisy, and that interactions vary across tissues and conditions [2]. We therefore use Anonymous as a controlled stress test of missing or misaligned prior information. In this condition, biological identities and mechanisms are removed but trajectories remain; Terra improves from 0.160 observational to 0.355 interactive Directed F1 (+0.195), with a +0.117 gain on PC12. The interactive gain shows that trajectories provide additional structural information when the supplied context is incomplete. Agentic RL further raises Qwen's interactive Directed F1 to 0.579 at step 200, above its context-only zero-trajectory baseline of 0.464 (Appendix D.1.1, the context-only versus agentic interactive recovery table).
Q3 I encourage the authors to deepen the existing SciGym [1] comparison with one shared-source example, explaining how the allowed operations, hidden execution mappings, and recovery targets differ. This would help readers see what task behavior the added grounding layer enables beyond an existing biological dry lab.
Response:
SciGym provides a partial SBML model and prompt-level biological framing, then lets the agent write code and request initial-concentration perturbations before submitting a reaction/kinetic model scored by NTS, RMS, and STE. BioCausalX provides source-grounded node-level context and reviewed operations, supports sequential intervention--observation interaction with hidden executable bindings, and evaluates a complete directed graph over all declared states. This state-level causal target directly measures which biological states influence one another.
Q4 Please report Directed F1 conditional on valid submissions, with per-system coverage at the initial and final GRPO checkpoints, or use a common formatting control. How much of the gain remains after separating submission validity from structural accuracy?
Response:
The complete 0–200 context-policy training curve rises from 0.448 Directed F1 at step 100 to 0.579 at step 200 (+0.131), while validity changes from 97.7% to 100.0%. The gain after validity nears saturation shows structural improvement beyond submission format.
[1] Duan, H., Lu, S., Harrigan, C.F., Desai, N., Lu, J., Koziarski, M., Cotta, L. and Maddison, C., 2025. Measuring Scientific Capabilities of Language Models with a Systems Biology Dry Lab. Advances in Neural Information Processing Systems, 38.
[2] Barabási, A.-L., Gulbahce, N. and Loscalzo, J., 2011. Network medicine: a network-based approach to human disease. Nature Reviews Genetics, 12, 56–68.
[3] Marbach, D., Costello, J.C., Küffner, R., Vega, N.M., Prill, R.J., Camacho, D.M., Allison, K.R., DREAM5 Consortium, Kellis, M., Collins, J.J. and Stolovitzky, G., 2012. Wisdom of crowds for robust gene network inference. Nature Methods, 9, 796–804.
[4] Chevalley, M., Roohani, Y.H., Mehrjou, A., Leskovec, J. and Schwab, P., 2025. A Large-Scale Benchmark for Network Inference from Single-Cell Perturbation Data. Communications Biology, 8, 412.
In light of these responses, we hope we have addressed your concerns, and hope you will consider raising your score. If there are any additional notable points of concern that we have not yet addressed, please do not hesitate to share them, and we will promptly attend to those points.
Kind reminder
Thank you again for your thoughtful comments. With less than three days remaining in the discussion period, we would greatly appreciate any further feedback or concerns you may have. If our clarifications have addressed your concerns, we would sincerely appreciate your consideration of raising your score.
With the discussion deadline approaching, we wanted to briefly follow up. If any concerns remain or if there is anything we can clarify, we would be grateful for the chance to respond while the discussion is still open. If our clarifications resolve your remaining concerns, we hope you might consider raising your overall score.
Thank you again for your time and feedback.
Response 1/2
We thank the reviewer for their careful and constructive feedback. Below, we address each concern and question in turn, using W to denote a critical weakness and Q to denote the corresponding question raised by the reviewer.
Response:
BioCausalX differs from SciGym in both interface and target: it provides source-grounded node-level context and reviewed operations, supports sequential intervention–observation interaction with hidden executable bindings, and evaluates the complete directed dependency graph. Directed F1 and nSHD therefore measure structural causal recovery directly, consistent with graph-recovery targets in biological network-inference benchmarks [3,4], whereas SciGym’s NTS, RMS, and STE evaluate reaction/kinetic-model reconstruction [1]. The same interface also supports GRPO: the context-trained Qwen policy reaches 0.448 Directed F1 versus 0.212 without context on held-out systems, and 0.579 at step 200. This shows that the full grounded task interface is trainable; the GRPO comparison jointly varies biological descriptions and action naming, while the reviewed operation mappings are audited separately in Appendix D.1.1.
Response: The complete 0–200 training curve gives the structural check requested:
The 0–100 gain is mainly a validity gain: F1 conditional on validity changes by only 0.001. Across the complete 0–200 curve, strict Directed F1 then rises from 0.448 at step 100 to 0.579 at step 200 (+0.131), while validity changes by only 2.3 percentage points. The complete curve in Appendix C, Figure 10, shows validity at 93.2% by step 50 and 97.7% by step 70, with Directed F1 continuing to improve through step 200. This supports an improvement in causal discovery beyond formatting.