acceptodds
Under review as a conference paper at ICLR 2027

SimAuthor: Harnessing Foundation Models for Persistent Scientific Simulator Authoring

Abstract

Foundation models can generate scientific code, but authoring a scientific simulator (an executable program encoding hypotheses about how mechanisms generate observable signals) requires iterative refinement. Scientific adequacy rarely admits a unique implementation or exact test, so simulators must instead be judged against limited real observations. We study this setting as **scientific simulator authoring under weak empirical feedback**, where distributional comparisons between simulated and real signals guide revision, and the target is the simulator itself rather than only its generated samples. We introduce **SimAuthor**, a persistent authoring harness that retains and revises executable simulators, separates scalar search scores from structured discrepancy feedback, and accumulates reusable implementation mechanisms. We evaluate SimAuthor on six biomedical tasks spanning cardiac and respiratory audio, photoplethysmography (PPG), and electrocardiography (ECG). Under a fixed 100-attempt budget, SimAuthor outperforms PUCT score search on all six tasks, generally outperforms textual-strategy optimization, and achieves the highest endpoint score on five of six. The authored simulators also improve on unseen recordings, transfer to independent pretrained representations, and yield substantial out-of-distribution gains in downstream ECG classification. Finally, 111 of 138 audited revisions alter program structure and account for 86.1% of the signed score improvement. These results suggest that persistent revision can progressively convert foundation-model knowledge into better executable scientific simulators from limited empirical evidence.

Primary area:
foundation or frontier models, including LLMs
Keywords:
foundation models, program search and optimization, synthetic data for health, ECG, biomedical signal processing, AI for science, agentic systems

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_KSHLOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the new task of utilising LLM harnesses for simulator design; it interestingly reformulates this type of program synthesis to use distributional metrics rather than binary verification. It additionally enhances the prompt of the agent with a discrepancy report which holds actual statistical discrepancy information for the LLM to use in its refinement process, overcoming the issue of certain prompt engineering approaches which use scalar feedback scores, and providing noticeable improvements. However, this is a 1D signal simulation harness and not necessarily shown to be an all-round scientific simulation harness.

Critical strengths

Interesting Formulation: This paper presents an interesting formulation of the simulator synthesis problem, contrary to most program synthesis works that use binary validation of producing working code this work uses distributional metrics to evaluate simulator performance.

Improved selection feedback: Using the discrepancy report with multiple items of feedback on output quality works more efficiently than giving a scalar feedback score to the LLM for improvement, and ablations confirm the empirical value of this.

Effective ablation studies: The token controlled ablation in appendix G and Audits in section 5 demonstrate that the gains from the proposed harness are from real improvements in the simulator not hyperparameter tuning by the LLM.

Validated downstream utility: Experiments effectively demonstrate downstream improvements when augmenting training sets with output simulator samples, demonstrating the utility of their improvement to providing better training representations. This is tempered via the accurate claim that this alone does not validate clinical validity.

Critical weaknesses

Overclaiming in the title: Despite the demonstrable value of this work in 1D signal simulation calling it a harness for scientific simulation without any tests of simulating data with greater than 2 variates or differential equation dynamics is too broad of a claim.

Generalisation issues: All tested scenarios are well known diseases with mathematical simulation methods, prior to seeing the empirical data the agent is prompted to begin planning a simulation approach, this seems to indicate that if this harness is used on a novel task where there is not a large body of research on simulation methods the performance may not generalise.

Questions for authors

Is this method capable of simulating more complex dynamical systems such as ODEs and PDEs? Can the authors discuss how the method would work without this prior planning step or on a novel simulation task?

3 comments

We thank the reviewer for the constructive feedback. We address the three concerns that the reviewer mentiones: the scope of SimAuthor, dependence on prior knowledge of the system, and whether complex dynamics are produced under the harness. We have made necessary changes in the updated manuscript and marked them in blue.

W1. Scope of the claim.

We agree that our empirical evaluation is limited to one- and two-channel biomedical signals. Our title reflects the broader problem of scientific simulator authoring, which motivates our general framework, while biomedical signals provide a concrete setting for evaluating it. We acknowledge that the generality beyond this biomedical setting remains to be established. We have clarified this distinction in the revision:

  • Abstract: We now explicitly state that our evaluation focuses on biomedical signal simulation.
  • Title: We would be happy to add a scope qualifier to the title if the area chair permits.

W2 / Q2. Dependence on planning and generalization to novel tasks.

How important is the planning step (the blueprint)? (New: Appendix I)

We added a blueprint ablation to test whether refinement requires an explicit scientific plan. We replace the blueprint with a three-sentence prompt naming only the condition and modality, while keeping the root simulator, feedback, and search procedure unchanged. Importantly, the root simulators are generated independently of the blueprint.

Final score Smax⁡@100S_{\max}@100. Full: median [range] over three seeds; − Blueprint: one seed.

VSDASCOPDAFLQTWPW
Root.258.139.165.216.286.364
Full.587 [.448, .611].557 [.429, .559].672 [.521, .689].738 [.738, .748].568 [.541, .594].581 [.543, .653]
− Blueprint.533.625.487.677.506.552

Without the blueprint, all six tasks still improve substantially over their roots. The full blueprint helps on COPD, AF, and LQT, while its effect is less consistent elsewhere. This suggests that explicit planning can guide refinement, but substantial progress is possible without it. Given the single-seed ablation, we do not draw stronger conclusions about its overall contribution.

Can SimAuthor generalize to novel tasks? (New: Appendix M)

We distinguish three levels of task novelty, based on the available scientific knowledge:

  • Level 1 — Well-studied conditions: Conditions with established signal characteristics and substantial prior knowledge to guide simulation, as in our six main tasks.
  • Level 2 — Less-explored conditions: Conditions whose signals are clinically described but lack established simulation methods.
  • Level 3 — Conditions with uncertain signal signatures: Conditions for which distinctive signal characteristics are not well established.

To explore the latter two levels, we added three experiments (one seed each, with the blueprint withheld during refinement).

ConditionLevelScore (root → final)Observation
Digitalis effect (ECG)2.280 → .399Refines scooped ST depression
Left ventricular aneurysm (ECG)2.208 → .509Refines persistent ST elevation
COVID-19 (cough audio)3.234 → .666Primarily captures generic cough features

Level 2: On both ECG conditions, search improves the score and refines condition-related signal characteristics. This suggests that SimAuthor can translate descriptive scientific knowledge into executable simulators even without an established simulator to build upon. But clinical fidelity remains to be validated.

Level 3: For COVID-19 cough, the search score improves, but the final simulator scores similarly against COVID-negative recordings (.641). The evaluator also does not clearly distinguish real positive from negative coughs. Thus, the improvement appears to capture generic cough characteristics rather than a disease-specific signature.

These experiments illustrate both the potential and the limits of generalization: SimAuthor can author simulators beyond established simulation tasks, but when condition-specific signal characteristics are unclear, distributional improvement alone does not establish condition-specific simulation. We discuss these findings in the revised Discussion and Appendix M.

Q1. Can SimAuthor simulate ODEs and PDEs?

In principle, yes. ODE and PDE solvers can be implemented as executable programs and are therefore within the framework's search space. The harness also allows the model to explore and refine different formulations over successive revisions. While our current experiments have not demonstrated explicit ODE or PDE solvers, they do contain simpler dynamical components.

To characterize what the authored simulators contain, we inspected all 18 final programs for dynamical components (new, Appendix L):

ComponentExamples in the final simulators
Closed-form dynamicsRR-dependent ventricular filling in AF, 1−e−RR/τ1-e^{-RR/\tau}: shorter intervals produce weaker pulses
Temporally correlated stochastic processesRandom walk in AS; autoregressive noise in COPD
Recursive filters and resonatorsSignal filtering in audio and PPG; murmur resonators in VSD
Explicit numerical ODE integrationNone

The simulators thus capture temporal dependencies, but in simple forms. This reflects the current evaluators, which measure distributional similarity rather than physiological mechanism and therefore do not explicitly encourage explicit dynamical models.

Authoring such models would need feedback that rewards mechanism. For example, the evaluator could check that the simulator responds correctly when a physiological parameter changes, or require consistency between coupled signals. Another route is to give the search established ODE models as building blocks.

Thanks again for the feedback, which has helped us clarify the scope and limitations of the work. We hope the additional experiments and clarifications address your main concerns.

Reviewer_KSHLOpenReview ↗

Response to rebuttal

Thank you for your extensive and detailed response, it showcases the effectiveness of the model without the blueprint and clarifies the scope. I will maintain my Weak Accept score

Reviewer_Rr6fOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper bring a interesting question, whether LLM can author a scientific simulator. By designing a harness with evaluator, search score, and discrepancy report, The LLM based scientific simulator authoring outperform Independent Sampling, PUCT Score Search, Textual Strategy Optimization on Task of audio, PPG, ECG generation in Signal quality. For downstream task, the LLM simulated signal also outperform the baseline, indicating the generalization of the harness. In the analysis, it shows the most improvement comes from structural improvement rather than parametric tuning by the LLM.

Critical strengths

This paper investigates an interesting question: whether LLMs can author scientific simulators. It presents extensive experiments across multiple biomedical signal modalities, with a thorough evaluation design that addresses the risk of overfitting to the authoring objective through held-out recordings, independent representations, and downstream tasks. The results show that the generated signals better match real recordings and can improve downstream ECG classification. The authors also analyze how LLMs revise simulator code and introduce structural changes, providing valuable insight into the iterative authoring process.

Critical weaknesses

The current framework primarily focuses on explicit, parameterized signal generators. An interesting extension would be to investigate whether LLM-guided feedback can iteratively improve the training of signal foundation models, such as PaPaGei (Pillai et al., 2025). This would broaden the framework’s applicability, but the current scope does not diminish the value of the paper’s contributions.

Questions for authors

1.What LLM have the most capability of authoring signal other than Gemini?

2 comments

We thank the reviewer for recognizing the relevance of the topic and for the positive assessment of the paper's soundness. We address the question and weakness below.

Q1. Which LLMs other than Gemini are capable of authoring simulators?

Among the models we tested, DeepSeek-V4 Pro (open-weight) performs comparably to Gemini 3.1 Pro (Table 6). It performs better on LQT and slightly worse on VSD. This suggests that persistent simulator authoring is not specific to one model family. During the rebuttal, we also evaluated a smaller model, DeepSeek-V4 Flash (Appendix K).

Final scores (Smax⁡@100S_{\max}@100) across three seeds, in ascending order (median in bold):

BackboneVSDLQT
Gemini 3.1 Pro.448 / .587 / .611.541 / .568 / .594
DeepSeek-V4 Pro.543 / .551 / .579.575 / .598 / .614
DeepSeek-V4 Flash.476 / .522 / .566.391 / .423 / .447

On LQT, all three Flash runs score below those of both larger models, while on VSD its median falls within their range. Flash also produces more invalid programs, reaching 34 out of 100 attempts in one run. These failures mainly involve output-format violations or scripts that fail to execute, reducing the number of attempts that can be evaluated. Nevertheless, all Flash runs improve over their root simulators.

These results suggest that backbone capability affects both authoring quality and the reliability of generated programs. We have not tested other frontier model families and do not claim a broader model ranking.

W1. Extension to training signal foundation models.

We agree that this is an interesting direction. We see two possible connections to our framework:

  • Simulators as controllable training data generators. The authored simulators are parameterized and can generate labelled signals under controllable conditions. Such data could be used to pretrain or fine-tune signal models, potentially targeting specific weaknesses identified through evaluation.
  • Authoring the training pipeline. A similar framework could be applied to iteratively improving the training code of a foundation model. For example, validation performance could serve as the search score, while discrepancy reports identify model weaknesses and guide revisions. Persistent candidates and the mechanism library could help retain and reuse effective training changes, as they do for simulator mechanisms in our setting.

We thank the reviewer for this suggestion and see both as promising directions for future work.

Thank you for the feedback, and I would like to maintain the positive score.

Reviewer_R9Q1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript proposed a harness, SimAuthor, for a simulator to generate "time series". The harness works as a loop with scientific blueprint, root simulator, evaluator, search orchestrator, refiner and mechanism library. On multiple tasks across audio, PPG and EEG, the proposed approach outperforms previous baselines such as independent sampling, PUCT Score Search and Text-Opt.

Critical strengths
  • The manuscript test on multiple datasets and models.
  • Overall I like the idea and I found it is interesting, however the presentation is quite sloppy.
Critical weaknesses

Major:

  • The proposed simulator has not evaluated with ECGSYN (https://physionet.org/content/ecgsyn/) or AF-PPG simulator (https://physionet.org/content/ecg-ppg-simulator-arrhythmia/1.3.1/). These models are available in PhysioNet. Another suggestions is to compare with generative models (e.g., Diffusets (Lai et al., 2025)).
  • The paper compares against only some conventional augmentation. There is no guarantee that they are useful. Please consider recent findings from Fattahi, et al., 2026, and perform more comprehensive experiments.
  • When you use frozen encoder there is a chance that it is already trained to ignore nuisance variation such as noise, I think it would be important to train scratch or control for that.
  • It seems the main evaluation is done through the downstream classification, however, specifically for physiological time series it is important to measure realism via established clinical features (PR interval or QRS), expert review, discriminator test (whether the model can we discriminate between synthetic and real), fidelity, diversity, distributionally or visually.

Lai, Yongfan, et al. "Diffusets: 12-lead ecg generation conditioned on clinical text reports and patient-specific information." Patterns 6.10 (2025).

Fattahi, Davood, et al. "A comprehensive inference-time augmentation framework in physiological signals: application to PPG-based AF detection." Physiological Measurement 47.7 (2026): 075004.

Minor:

  • The writing is very AI in places, and doest not contribute to clarity.
  • There is no confidence intervals or spread reported for the performance metrics
  • There is no clarity how fold were used. Did the authors select hyperparameters and checkpoints on the validation set, and report on holdout set?
Questions for authors

See weaknesses.

4 comments
AuthorsOpenReview ↗

Reply to weaknesses

Thanks for the constructive suggestions. We have strengthened the evaluation in response to the concerns of the reviewer and updated the manuscript. All changes are marked in blue. See below for a summary.

W1 & W4. Expert simulators and physiological realism

We agree that expert simulators should be considered when evaluating authored simulators, especially since they are often designed by human experts with physiological and clinical knowledge. To enhance our evaluation, we compare SimAuthor, expert simulators, and real recordings in terms of clinical and signal characteristics, to shed more light on this aspect.

We use two well-known simulators suggested by the reviewer: ECGSYN for ECG and the PhysioNet ECG–PPG simulator for AF. However, ECGSYN does not directly model the delta wave characteristic of WPW, so we only compare it with SimAuthor on LQT.

Below is a summary of the results, with full results and measurement details in the updated Section 5.3 and Appendix J.

We compare both primary condition-related features, such as QTc for LQT and RMSSD for AF, and additional features such as QRS duration, T/R ratio, and sample entropy. The latter are not directly optimized during authoring but are relevant to ECG morphology and pulse-interval irregularity.

LQT — Median [IQR]

SignalsQTc (ms)QRS (ms)T/R
Real, search500 [480, 527]118 [110, 132]0.12 [0.05, 0.20]
Real, held-out503 [471, 540]120 [112, 140]0.09 [−0.04, 0.20]
Root572 [523, 619]136 [136, 136]0.20 [0.15, 0.25]
SimAuthor521 [473, 565]116 [110, 122]0.23 [0.14, 0.33]
ECGSYN550 [509, 589]130 [126, 136]0.40 [0.39, 0.40]

AF — Median [IQR]

SignalsRMSSD (ms)Sample entropy
Real, search209 [172, 258]1.79 [1.52, 2.30]
Real, held-out478 [298, 809]1.49 [1.33, 1.79]
Root257 [205, 324]1.99 [1.69, 2.35]
SimAuthor295 [244, 355]1.87 [1.56, 2.23]
Expert simulator303 [261, 353]1.95 [1.67, 2.30]

Overall, SimAuthor produces signal characteristics comparable to the expert simulators, and for some LQT features, even closer to real recordings. For AF, SimAuthor and the expert simulator show similar RMSSD and sample entropy values, although the held-out wrist PPG recordings differ substantially in RMSSD.

This comparison still has limitations, since we use automated measurement tools rather than expert annotations. But we hope this additional evaluation strengthens the paper from a signal realism and expert knowledge perspective.

W2 & W3. Stronger augmentation and training from scratch

We adopted both of the reviewer's suggestions to strengthen the downstream evaluation. We implemented the framework of Fattahi et al. (2026) using the authors' package as a stronger augmentation baseline. To test whether the benefit depends on ECGFounder representations, we also trained an eight-block 1D ResNet directly on raw ECG signals, across 20 random seeds. We also repeated the frozen-probe experiments with 10 independently sampled augmentation runs to improve the reliability of the results. This led to small changes (less than 0.008 AUPRC) from the initial submission, without changing the overall findings.

Below is a summary of the results. The full updated experiments can be found in Section 4.2 and Appendix H.

The table reports external-test AUPRC on Chapman–Shaoxing (95% bootstrap CI). All augmentation methods add 300 positive ECGs. Frozen-probe results are means over 10 augmentation runs; scratch results are means over 20 seeds.

TrainingLQT, frozenWPW, frozenLQT, scratchWPW, scratch
Real only.633 [.59, .67].691 [.59, .79].592 [.55, .63].700 [.68, .72]
+ Conventional.650 [.61, .69].731 [.64, .82].686 [.65, .72].790 [.77, .80]
+ Fattahi tuned.720 [.68, .76].724 [.63, .81].715 [.69, .74].783 [.76, .81]
+ SimAuthor.750 [.71, .79].803 [.72, .88].747 [.71, .78].841 [.83, .86]

This enhanced evaluation shows that SimAuthor's advantage in downstream classification, especially on OOD data, still holds against stronger augmentation baselines. With frozen ECGFounder representations, its improvements over all alternatives are statistically significant on both tasks. Training from scratch also preserves the ranking: SimAuthor significantly outperforms all alternatives on WPW, and conventional augmentation on LQT, although its difference from tuned augmentation on LQT is not significant.

We hope these additional experiments address the reviewer's concerns.

AuthorsOpenReview ↗

Final remarks

Minor weaknesses

We also went through the manuscript to improve the writing and clarity, especially in the experimental sections. We now report confidence intervals and variability across runs, and have clarified how the data splits, hyperparameter tuning, and checkpoint selection were handled (in main text or in appendix).

Final remarks

Overall, we agree that clinical features, physiological realism, and downstream utility are all important aspects of evaluating authored simulators. At the same time, we want to emphasize that SimAuthor's central contribution is the automatic construction and iterative refinement of executable scientific simulators from limited real recordings and empirical feedback. Downstream classification is one way to evaluate the usefulness of the generated signals, while distributional agreement with real recordings, including held-out data and independent representations, is also important for assessing the effectiveness of the authoring framework.

We hope to further explore clinical fidelity and expert-based evaluation in future work. For now, we hope the additional evidence and clarifications address the reviewer's concerns and would appreciate reconsideration of the assessment.

Reviewer_R9Q1OpenReview ↗

Rebuttal response

The W2 & W3. Stronger augmentation and training from scratch looks very encouraging. I think "W1 & W4. Expert simulators and physiological realism" is partly addressed, I think generative evaluation is harder. I like W2 & W3 result, so I decided to increase to Weak accept.

Thank you very much for taking the time to review our updated materials. We sincerely appreciate your continued engagement!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Updated manuscript

We thank all reviewers for their constructive feedback. We updated the manuscript with necessary changes that we think can strengthen the work, based on reviewers' input; all changes are marked in blue. The main additions are:

1. Expert simulators and clinical features (in response to reviewer R9Q1; Section 5.3, Appendix J). We compare SimAuthor with ECGSYN (LQT) and the PhysioNet ECG–PPG simulator (AF) on established clinical features including those that the authoring evaluators do not measure explicitly.

2. Stronger downstream evaluation (in response to reviewer R9Q1; Section 4.2, Appendix H).

  • We add a tuned augmentation baseline (Fattahi et al., 2026) that is optimized for the downstream task.
  • We add a 1D ResNet trained from scratch to eliminate the effect that trained encoder has.
  • We report medians over independent draws with paired bootstrap intervals.

3. Dependence on prior knowledge and novel tasks (in response to reviewer KSHL; Appendices I, M).

  • Removing the blueprint during refinement retains most of the improvement.
  • On two ECG conditions without any dedicated simulation mechanisms to our knowledge, SimAuthor still turns descriptive knowledge about the conditions into simulation mechanisms and improves the search score.
  • A COVID-19 cough task, where the signal manifestation is not established, shows the limit of the approach.

4. Other backbones (in response to reviewer Rr6f; Section 5.4, Appendix K). DeepSeek-V4 Pro is comparable to Gemini 3.1 Pro. The smaller DeepSeek-V4 Flash authors less effectively on LQT and produces more invalid code, but still improves on every root.

5. Scope and on authoring more complex mechanisms (in response to reviewer KSHL; Abstract, Section 7, Appendix L). We scope our claims to biomedical signals. We also analyse the dynamical components of the authored simulators. They encode physiological relations in simple forms but no explicit ODE even though authoring complex systems is possible; we discuss evaluators that would reward mechanism as future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.