- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
In order to make simulation-based inference more efficient, the paper proposes a pipeline for finding parameter combinations that more clearly reflect what actually changes the data. To do so, they first use simulated parameter–output pairs to estimate which parameter changes make a noticeable difference to the outputs, measured by Fisher matrices. They then use this estimate to train a neural network which maps the original parameters to new coordinates that bring the Fisher matrix closer to the identity. Finally, they use symbolic regression to search for simple equations mapping the original parameters to the new coordinates, balancing how well each equation matches the network with how complex it is. Importantly, this does not require new simulations calls. They validate this approach using both synthetic and real-world scientific applications. In the synthetic examples, they find the pipeline generally recovers the expected parameter structure. For the scientific applications, they find it recovers recognizable parameter combinations as well as hidden useful alternatives.
Critical strengths
One strength of the paper is the interpretability of the symbolic regression step. In contrast to other neural representations, the fact that the learned coordinates are converted into interpretable formulas gives this method the ability of both understanding the model and using the coordinates in later inference.
Another strong contribution from the paper is the finding that in the scientific simulator examples the method can both recover recognizable relationships as well as find new hidden relationships. This is consequential because it gives evidence that the method can produce actual scientifically meaningful results outside of the simpler synthetic tests.
Critical weaknesses
One weakness in the paper is that the demonstrated scale of the method is limited. They show dimension reduction up to 32 parameters however that was a controlled example with only a small number of relevant parameters and may not demonstrate that the full symbolic pipeline actually scales to large parameter spaces. The authors acknowledge this however it still is significant that the method may not be able to scale.
Additionally, some of the efficiency gains are ambiguous with regards to how simulations are re-used. The NPE comparisons demonstrate improved sample efficiency however the paper does not consistently report the total simulation cost. This makes the end-to-end efficiency benefit harder to assess and is significant because efficiency gains are a main contribution to the proposed method.
Questions for authors
Could the authors clarify how exactly simulations are compared? Specifically, for SIR it is not clear to me whether simulations were re-used or whether the simulations used pre-inference are factored in to the comparison. A per-experiment count of unique simulator calls would make the efficiency comparisons easier to interpret.
4 comments
Dear Reviewer MNz7, as the discussion closes shortly, we would be grateful to know whether the per-experiment simulation counts and the input-screening experiment address your concerns, or whether anything remains unclear that we can answer in the time left.
Response
Hello, thank you for the detailed response to my review.
The ambiguity of simulator usage has been addressed which fully removes that weakness from the paper. As far as scalability, the new experiment does add some evidence for a better ability to handle larger sizes however it does not run the whole pipeline so concerns over end-to-end scaling still remain. The changes do improve the paper however not substantially and so I am keeping my recommendation of weak acceptance.
Thank you for your reply.
We have run the full pipeline (screening, neural coordinate learning, symbolic regression, downstream posterior inference) for three of the d=100 trials, each with 2,000 simulations. All three recover the two hidden Rosenbrock coordinates symbolically. Downstream NPE posterior log density in the discovered coordinates is 2.92, in line with the d=32 result (2.86) and close to that of the true coordinates (2.95). As anticipated with increased dimensionality, log density degrades when performing NPE in the 100 raw coordinates (2.21), which is worse than what raw scored at d=32 (2.61). This means our discovery scaling plot (Fig. 2A) can be extended to d=100 to demonstrate the scalability of the method. We will include this end-to-end d=100 experiment in the camera-ready version.
We thank Reviewer MNz7 for a thoughtful review and for recognising the value of interpretable symbolic coordinates and scientifically meaningful parameter combinations. We also appreciate the requests for clearer simulation accounting and a sharper distinction between our demonstrated scaling results and broader scalability.
1. Simulator-call accounting.
We agree that the relevant cost is the number of unique simulations used by the complete learning procedure. Discovery-training and discovery-validation simulations must both be counted, and simulations reused for downstream inference must be counted only once.
Our accounting distinguishes the following discovery datasets:
We will make these counts and the reuse of simulations explicit in the experiment descriptions and figure captions. Independent evaluation simulations will be listed separately; any simulations used for training or model selection will be included in the learning cost.
The total simulation cost includes discovery and NPE training, with shared simulations counted only once. Points below the discovery budget therefore compare NPE training budgets rather than total simulator calls, and we will label them accordingly.
For Rosenbrock, restricting the existing curves to NPE training budgets of at least 500 gives approximately 4-fold and 9-fold differences in the training budget required to reach matched validation log-probability, at eta-training budgets of 500 and 1,000 respectively. We will distinguish these curve-based comparisons from the fully accounted simulator-call totals. The 14-fold endpoint occurs at an NPE training budget of 100, below the discovery budget, and will not be described as an end-to-end saving.
2. Additional SIR check at a smaller discovery budget.
The original SIR discovery experiment used 1,000 simulations including validation. We separately tested discovery using 500 simulations in total, split into 450 training and 50 validation, with observation noise 0.05, across three random seeds.
A symbolic R0-type coordinate was recovered in all three runs with different random seeds, with gradient cosine approximately 0.99 against the R0 gradient. This demonstrates recovery at the smaller total budget.
3. Scaling the method.
Prompted by your comment, we tested input screening with the two-parameter Rosenbrock structure embedded among d=100 input parameters. Using 2,000 simulations per trial (1,700 for training and 300 held out for scoring), we ranked the original inputs using their sensitivity under the one-step map of Eq. (6): for each input, the score is the square root of the corresponding diagonal entry of the held-out average of JᵀJ; equivalently, the root-mean-square norm of the corresponding Jacobian column over the 300 held-out parameter points. Scores are compared against ambient dimensions held at the prior.
In each of ten trials, each with a different randomly chosen pair of active inputs, an independent set of simulations and a different network initialisation, the two active inputs ranked first and second among all 100. Successive reductions from 100 to 25, then 7, then 5 inputs retained both active inputs in every trial, bringing the number of retained inputs within the range tested in Fig. 2A.
This extends the evidence for identifying relevant inputs in a controlled sparse problem. It also provides a practical route to reducing the number of variables presented to the subsequent coordinate-learning and symbolic-regression stages. We have not yet completed those downstream stages for this new experiment, so it is not an end-to-end demonstration of the full pipeline at 100 parameters.
The existing hidden-banana experiment demonstrates coordinate discovery and downstream inference up to 32 ambient parameters (Fig. 2A; Appendix F.1). We will distinguish these results explicitly: successful input screening at 100 parameters, and the previously demonstrated discovery and inference pipeline at 32. Symbolic search complexity still depends on the retained input variables and expression complexity, as well as the number of latent coordinates.