- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes the new task of utilising LLM harnesses for simulator design; it interestingly reformulates this type of program synthesis to use distributional metrics rather than binary verification. It additionally enhances the prompt of the agent with a discrepancy report which holds actual statistical discrepancy information for the LLM to use in its refinement process, overcoming the issue of certain prompt engineering approaches which use scalar feedback scores, and providing noticeable improvements. However, this is a 1D signal simulation harness and not necessarily shown to be an all-round scientific simulation harness.
Critical strengths
Interesting Formulation: This paper presents an interesting formulation of the simulator synthesis problem, contrary to most program synthesis works that use binary validation of producing working code this work uses distributional metrics to evaluate simulator performance.
Improved selection feedback: Using the discrepancy report with multiple items of feedback on output quality works more efficiently than giving a scalar feedback score to the LLM for improvement, and ablations confirm the empirical value of this.
Effective ablation studies: The token controlled ablation in appendix G and Audits in section 5 demonstrate that the gains from the proposed harness are from real improvements in the simulator not hyperparameter tuning by the LLM.
Validated downstream utility: Experiments effectively demonstrate downstream improvements when augmenting training sets with output simulator samples, demonstrating the utility of their improvement to providing better training representations. This is tempered via the accurate claim that this alone does not validate clinical validity.
Critical weaknesses
Overclaiming in the title: Despite the demonstrable value of this work in 1D signal simulation calling it a harness for scientific simulation without any tests of simulating data with greater than 2 variates or differential equation dynamics is too broad of a claim.
Generalisation issues: All tested scenarios are well known diseases with mathematical simulation methods, prior to seeing the empirical data the agent is prompted to begin planning a simulation approach, this seems to indicate that if this harness is used on a novel task where there is not a large body of research on simulation methods the performance may not generalise.
Questions for authors
Is this method capable of simulating more complex dynamical systems such as ODEs and PDEs? Can the authors discuss how the method would work without this prior planning step or on a novel simulation task?
3 comments
Q1. Can SimAuthor simulate ODEs and PDEs?
In principle, yes. ODE and PDE solvers can be implemented as executable programs and are therefore within the framework's search space. The harness also allows the model to explore and refine different formulations over successive revisions. While our current experiments have not demonstrated explicit ODE or PDE solvers, they do contain simpler dynamical components.
To characterize what the authored simulators contain, we inspected all 18 final programs for dynamical components (new, Appendix L):
| Component | Examples in the final simulators |
|---|---|
| Closed-form dynamics | RR-dependent ventricular filling in AF, : shorter intervals produce weaker pulses |
| Temporally correlated stochastic processes | Random walk in AS; autoregressive noise in COPD |
| Recursive filters and resonators | Signal filtering in audio and PPG; murmur resonators in VSD |
| Explicit numerical ODE integration | None |
The simulators thus capture temporal dependencies, but in simple forms. This reflects the current evaluators, which measure distributional similarity rather than physiological mechanism and therefore do not explicitly encourage explicit dynamical models.
Authoring such models would need feedback that rewards mechanism. For example, the evaluator could check that the simulator responds correctly when a physiological parameter changes, or require consistency between coupled signals. Another route is to give the search established ODE models as building blocks.
Thanks again for the feedback, which has helped us clarify the scope and limitations of the work. We hope the additional experiments and clarifications address your main concerns.
Response to rebuttal
Thank you for your extensive and detailed response, it showcases the effectiveness of the model without the blueprint and clarifies the scope. I will maintain my Weak Accept score
We thank the reviewer for the constructive feedback. We address the three concerns that the reviewer mentiones: the scope of SimAuthor, dependence on prior knowledge of the system, and whether complex dynamics are produced under the harness. We have made necessary changes in the updated manuscript and marked them in blue.
W1. Scope of the claim.
We agree that our empirical evaluation is limited to one- and two-channel biomedical signals. Our title reflects the broader problem of scientific simulator authoring, which motivates our general framework, while biomedical signals provide a concrete setting for evaluating it. We acknowledge that the generality beyond this biomedical setting remains to be established. We have clarified this distinction in the revision:
W2 / Q2. Dependence on planning and generalization to novel tasks.
How important is the planning step (the blueprint)? (New: Appendix I)
We added a blueprint ablation to test whether refinement requires an explicit scientific plan. We replace the blueprint with a three-sentence prompt naming only the condition and modality, while keeping the root simulator, feedback, and search procedure unchanged. Importantly, the root simulators are generated independently of the blueprint.
Final score Smax@100. Full: median [range] over three seeds; − Blueprint: one seed.
Without the blueprint, all six tasks still improve substantially over their roots. The full blueprint helps on COPD, AF, and LQT, while its effect is less consistent elsewhere. This suggests that explicit planning can guide refinement, but substantial progress is possible without it. Given the single-seed ablation, we do not draw stronger conclusions about its overall contribution.
Can SimAuthor generalize to novel tasks? (New: Appendix M)
We distinguish three levels of task novelty, based on the available scientific knowledge:
To explore the latter two levels, we added three experiments (one seed each, with the blueprint withheld during refinement).
Level 2: On both ECG conditions, search improves the score and refines condition-related signal characteristics. This suggests that SimAuthor can translate descriptive scientific knowledge into executable simulators even without an established simulator to build upon. But clinical fidelity remains to be validated.
Level 3: For COVID-19 cough, the search score improves, but the final simulator scores similarly against COVID-negative recordings (.641). The evaluator also does not clearly distinguish real positive from negative coughs. Thus, the improvement appears to capture generic cough characteristics rather than a disease-specific signature.
These experiments illustrate both the potential and the limits of generalization: SimAuthor can author simulators beyond established simulation tasks, but when condition-specific signal characteristics are unclear, distributional improvement alone does not establish condition-specific simulation. We discuss these findings in the revised Discussion and Appendix M.