- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents ALPS. It uses an LLM to search for program structures and a gradient-based method to fit their parameters in DeGAS. It keeps the best candidate programs ans the LLM then improves these programs based on their code, data statistics, and optimization results. It also compare the ALPS with four other methods on nine benchmarks. They use held-out negative log-likelihood and Wasserstein-type distances to measure performance. They also test the effects of the LLM, parameter fitting, program changes, and feedback. Finally, they use real UCI heart disease data to test ALPS on real-world data. They also use natural language hypotheses to guide the search for program structures.
Critical strengths
-
ALPS combined LLM-guided structural search with gradient-based parameter fitting, it addressed two interdependent challenges in probabilistic program discovery.
-
ALPS produces clear generative programs and it evaluates their ability to reproduce observed distributions beyond likelihood fit.
Critical weaknesses
-
ALPS leaves the comparability of likelihood scores across discrete and continuous representations unclear, weakening its interpretation of its model-discovery gains.
-
ALPS does not separate the benefit of its search strategy from the greater capability of its LLM backend relative to the baselines.
Questions for authors
1.How does ALPS make sure that the likelihood scores can fully reflect model quality, instead of how discrete variables are written? 2. Does ALPS still perform better than the existing methods when they use the same LLM and the same budget? 3. Do the real-data results show that the ALPS can find reliable programs when all runs, including failed ones, are counted?
3 comments
Dear Reviewer 5rqJ,
Thank you again for your review. As the discussion period ends on November 18, we would like to ask whether our response addresses your concerns, in particular the comparability of likelihoods across representations, the comparison with the same 7B model (App. E) and the real-data experiment repeated with 10 runs, counting failed ones (Sec. 5). We would be happy to clarify any remaining points.
Best regards, The Authors
Follow up on discrete variables
Dear Reviewer 5rqJ,
Following up on your first question, we re-evaluated all methods with a corrected likelihood that scores discrete variables as probability masses, so that scores no longer depend on how a discrete variable is written. ALPS remains best on 6 of 9 benchmarks, and on all 9 when re-run with the corrected likelihood as its objective; the comparison is on equal terms with all baselines except RS-Grad, which could not yet be re-run due to time constraints. Details are in the general response (additional results) and in the new App. F.
We thank you for raising this point, which will improve the evaluation in the final version.
Response to Reviewer 5rqJ
We agree that likelihoods computed with respect to different reference measures are not directly comparable: DeGAS scores a discrete value as a narrow Gaussian density, whereas a language with native discrete distributions scores it as a probability mass. For this reason, our evaluation only compares likelihoods between programs written in DeGAS and scored by the same DeGAS likelihood. This holds for all baselines, including RS-MCMC, whose programs are translated back into DeGAS for evaluation (App. C.2). To check that ALPS's advantage does not depend on how discrete variables are written, we note that on the three purely continuous benchmarks (If, MoG, MultiBranch), where no discrete variables are present, ALPS still has the best NLL, significantly against all four baselines (p < 0.05, Table 6).
To complement the likelihood, we also report W₂², the Wasserstein-type distance between the Gaussian-mixture representations of each synthesized program and the ground-truth program. When comparing against a method outside DeGAS, as with the original RefineStat (in PyMC) in our real-world case study, we do not compare likelihoods and instead report the empirical Wasserstein distance W₁ between the distribution of num in generated samples and in the real data, which does not depend on how a variable is represented.
To separate the contribution of the search from that of the LLM, we re-ran ALPS with Qwen2.5-Coder-7B-Instruct, the model used by the RefineStat-based baselines, with and without grammar-constrained decoding. Without it, ALPS converges on five of nine benchmarks and achieves the best NLL and W₂² on all five; its weakness at this scale is producing valid programs (29/45 valid runs). With grammar-constrained decoding through llguidance, ALPS converges on all nine benchmarks (44/45 valid runs) and, against the baselines with the same model, achieves the best NLL on seven of nine and the best W₂² on six. Results are reported in the general response to all reviewers (above) and in the new App. E.
Regarding budget, the methods spend compute in different units, so they cannot be equalized on a single one; each method runs until its own stopping criterion (App. D.7). With gpt-oss-120b, an ALPS run makes on average 8.1 LLM calls (39.6 generated programs) and takes 16.3 minutes; with the 7B model, it makes 9.3 calls (≈46 programs) and takes 5.6 minutes. RS-Grad generates on average 9.7 programs in 4.3 minutes, and RS-MCMC 53.7 programs in 237 minutes. RS-MCMC thus sees a comparable number of programs at about 40× the wall-clock cost of ALPS with the same 7B model, and still performs worse on most benchmarks. RS-Grad stops early because its acceptance criterion is met, not because its budget is exhausted: it allows up to 100 refinement rounds, and some runs generate up to 102 programs.
We repeated the heart-disease experiment with 10 runs per method, using the same prompt as for the benchmarks (no prior knowledge injected), and counted every run, including the failed ones (revised Sec. 5). ALPS produced a valid program in 8/10 runs, RefineStat in 4/10. Over the successful runs, the Wasserstein-1 distance between the real and the predicted distribution of num had a median of 0.197 (IQR 0.178–0.495) for ALPS and 0.469 (IQR 0.351–0.479) for RefineStat. One of RefineStat's successful runs (W₁ = 0.000) uses num as a predictor of itself; excluding it gives 0.469 (IQR 0.469–0.490). Counting failed runs as the worst score, the ALPS median is 0.343, while the median RefineStat run is a failure. The ALPS failures occur when the LLM writes programs over placeholder variable names instead of the dataset's columns. RefineStat's failures are runs that never produce a model PyMC can sample from, or that hang during NUTS sampling.