- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper addresses post-training LLMs for inductive reasoning, i.e., inferring uncertain latent quantities and predictions from sparse natural-language observations, a setting where labeled data are scarce and targets are inherently distributional. It proposes Program-based Posterior Training (PPT), which extends the Model Synthesis Architecture (MSA): an LLM generates natural-language scenarios with queries in sports, healthcare, and general domains, translates each scenario into a Pyro probabilistic program, and posterior inference (MCMC or rejection sampling) yields a posterior over each query's answer on a 0–100 scale. The discretized posterior serves as a soft target in a cross-entropy loss over integer answer tokens, used to LoRA-fine-tune Llama3-8B-Instruct and Qwen2-7B-Instruct, with additional Qwen2.5-14B/72B runs. The dataset contains roughly 15k scenarios and 60k queries, split by inferential motifs. Evaluation covers mean absolute error against program posteriors on held-out queries, motifs, and domains; comparisons with Gemini-3.1-Pro and a model fine-tuned on Gemini answers; correlation with previously published human judgments; and accuracy, NLL, and ECE on OpenEstimate, PosteriorDB, Bayesian Teaching, and five multiple-choice benchmarks, alongside temperature scaling, chain-of-thought with self-consistency, mean-target, and forward-sampling variants.
Critical strengths
-
A practical source of distributional supervision.
PPT turns probabilistic program synthesis into a training-data pipeline for tasks where uncertainty matters and human labels are costly. The approach is conceptually simple and potentially reusable. -
Consistent empirical improvements.
Synthetic sports MAE decreases from approximately 10.8 to 5.0–5.7 (Fig. 2), and human alignment improves from to (Fig. 3). Raw NLL and ECE improve across the reported evaluations for both primary backbones, with benefits generally remaining after temperature scaling (Table 2).
Critical weaknesses
-
The benefit of posterior-specific structure remains unclear.
Although PPT generally complements temperature scaling, the experiments do not isolate posterior-specific supervision from broader soft-label effects. The mean-target comparison changes both uncertainty and distributional shape, while forward sampling also changes the supervision procedure and potentially data volume. Forward sampling achieves lower raw ECE in five of seven columns of Table 4. The results therefore support improved uncertainty estimates more directly than the stronger claim of “more deeply internalized uncertainty.” -
Generalization and label-quality claims require qualification.
The displayed synthetic domains largely share one broad inferential structure, limiting what motif and domain holdouts establish. PosteriorDB provides relevant external evidence, but its evaluation construction needs clarification. Furthermore, agreement with program posteriors does not independently establish that program-derived labels better represent the intended scenarios than Gemini-derived labels. The MSA, ForestDB, and human-alignment checks are useful, but the PDF does not report a representative semantic audit of the generated corpus.
Questions for authors
Data generation and experimental setup
-
Data construction and splits.
Which models generated the scenarios and programs? Please report filtering rates, semantic validation procedures, exact answer-scale mappings, and the splits used for tuning and early stopping. -
Evaluation protocol.
Please specify external-task prompts, probability normalization and NLL definitions, the temperature-fitting split, and seed counts and uncertainty statistics.
Attribution of the gains
-
Posterior structure.
Under matched training conditions, how does PPT compare with simplified soft targets that preserve posterior means while controlling entropy or variance? Distribution-sensitive metrics would complement MAE and ECE. -
Label source.
What target format does Gemini-Answer-FT use? A comparison with matched label formats, including evaluation on the human judgments in Fig. 3, would better isolate label-source quality.
Additional Suggestions
-
Generalization.
Report the PosteriorDB model families, their natural-language conversion, and per-family results to clarify structural transfer. -
Backbone coverage.
The Qwen2.5-14B/72B results provide useful additional coverage; a more recent backbone would be an optional extension. -
Scalability.
Report data-generation cost, inference compute, and usable-example yield.
3 comments
Continued Response
Implementational Detail Questions
Q: Data generation and experimental setup. Data construction and splits. Which models generated the scenarios and programs? Please report filtering rates, semantic validation procedures, exact answer-scale mappings, and the splits used for tuning and early stopping.
Evaluation protocol. Please specify external-task prompts, probability normalization and NLL definitions, the temperature-fitting split, and seed counts and uncertainty statistics.
A: Data. Gemini-3.1-Pro writes the scenarios (temperature 0.5) and the Pyro programs (temperature 0, MSA's three prompting steps). No LLM labels any answer: labels come from exact rejection sampling in each program (1,000 accepted draws). Of 19,296 scenarios, 85.4% yield a working program; most losses are sampler timeouts where 1000 accepted draws are not reached. We drop a further 1.8% of examples with unreliable posteriors (>5% of draws outside 0-100, or 0/1 outcomes instead of probabilities). At the pipeline level, our programs agree with ForestDB reference programs (posterior-mean MAE 0.017) and with human judgments (r = .71). This validation is summarized at the end of section 3.1, and we will add the other details in Appendix A.2.
Answer scale. Every answer is an integer 0-100: probabilities x 100. The distributional target is the histogram of the rounded posterior draws over the 101 integer tokens. This is defined in main text section 3.2, and how the prompt communicates this to LLM when answering the question is shown in appendix section A.2.
Splits and selection. We split train-validation-test by motif: in every domain, two of the 16 motif combinations are held out and divided 50/50 into validation and test (56,481 / 4,180 / 4,195 examples). All runs use one setting (LoRA r = 8, lr 5e-5, 500 epochs), and we keep the checkpoint with the lowest loss on a fixed 773-example validation subset. There is no early stopping, and the test split is never used for selection. Motifs are discussed in Section 3.3, and we will further clarify train/val/test in Section 4.
Evaluation. Appendix A.2 lists most of our prompts, but here we add how the answer is scored: All tasks are zero-shot, scored from one teacher-forced pass. Multiple-choice prompts give the question and lettered choices, and the assistant turn is prefilled with "The answer is: <". Section 3.2 includes relevant definitions, and to further add to that, our evaluation NLL = -log p(gold), with p a softmax over the letters A-Z for multiple choice questions, or over integers from 0 to 101 for numerical questions (always converted to 0 to 100 scale and instructed to output integers). Temperature scaling fits one temperature per model and benchmark on up to 1,000 items from the benchmark's training split. Each fine-tuned model is trained with 3 seeds, and tables report the mean +/- standard error (SD / sqrt 3). These details will be added to Appendix A.2.
Q: Scalability. Report data-generation cost, inference compute, and usable-example yield.
A: Data-generation cost: about 12k input + 29k output tokens per program. Inference compute: exact rejection sampling on 1 CPU core per scenario, median 12 minutes; about 8,200 CPU-hours in total, no GPU. Fine-tuning: one 40 GB A 100 GPU, 14-24h per run. At test time the model answers in one forward pass. Usable-example yield: 85.4% of scenarios give a working program, and how-to-filter is answered above.
Thank you for the additional experiments and clarifications. While they strengthen the empirical results, concerns regarding the contribution of posterior-specific supervision and the semantic validity of generated programs remain. The generalization claims also require further qualification. Therefore, I maintain my original rating (2: Weak Reject).
Response to Reviewer Ca9f
We thank the reviewer for helpful feedback, which will improve the paper.
General Questions
Q: The benefit of posterior-specific structure is unclear.
A: We view forward sampling and posterior sampling as two versions of our pipeline, and we view the pipeline and the data created with labels from probabilistic programs as the source of improvement. The two versions arise naturally from the idea of scenario + program generation. We will bring extra care and adjust any claims that suggest that distributional labels are the performance driver or a key contribution. Forward sampling’s advantage lies in having more data: it can be thought of as training on individual noisy samples behind posterior sampling, whereas posterior sampling trains on that one posterior distribution estimated from many samples. Table 4 shows that they achieve similar performance, reflecting that the noisier data from forward sampling is partly offset by its large scale.
The key result is that training language models with data generated from probabilistic programs results in significant improvements in their probabilistic reasoning ability and calibration, with different data generating methods (forward sampling and posterior sampling) being effective in different settings.
Q: Generalization and label-quality.
A: We agree that the synthetic domains share one broad family of inference: latent traits inferred from noisy outcomes written in a broadly similar template. Therefore, for empirical evaluations we test trained models in a progression of similar data to external data: held-out motifs from our synthetic data, and completely external datasets like OpenEstimate, PosteriorDB and the multiple-choice benchmarks.
Program vs. Gemini labels. While the paper extends beyond agreement with program posteriors with human judgement, it is only comparing base and PPT model with human judgements. For the below table we ran this comparison on more models, including the Gemini-distilled one. At matched format (one-hot on a single integer, rather than on a distribution), the model trained on program means agrees better with the human judgments of Fig. 3 than the model trained on Gemini's answers.
Semantic audit. Every example in the corpus passes automated checks: at least 1000 accepted rejection-sampling draws, and a check that each query returns the quantity its answer convention asks for; otherwise the example is rejected. An LLM judge is not a reliable substitute at scale, because strong LLMs answer these queries poorly (Fig. 2). Downstream evaluation is itself a check on label quality: semantically wrong labels would not improve agreement with human judgments and external benchmarks. We additionally ran a controlled ablation with the same training but with labels and scenarios shuffled. It loses those gains by having only a 0.2 correlation with humans, much lower than the base model’s 0.509 shown above.
PosteriorDB construction. 72 scalar queries on six posteriors (eight schools, surgical, seeds, dugongs, kilpisjarvi,mesquite): 45 queries on posterior means and 27 queries on posterior probabilities of an event (e.g., P(theta_A > theta_C)). The prompt states the Bayesian model in prose (likelihood and priors), lists the observed data, and asks one query on a stated 0-100 grid, in the same answer format. Gold answers come from PosteriorDB's own published programs.
Q: Attribution of the gain. Posterior structure. Under matched training conditions, how does PPT compare with simplified soft targets that preserve posterior means while controlling entropy or variance? Distribution-sensitive metrics would complement MAE and ECE.
A: Our matched control is PPT (Mean) (Table 2). It uses the same questions, recipe, hyperparameters, seeds and selection rule as PPT (Distribution), but its target is a point mass at the rounded posterior mean. It matches PPT on point accuracy (held-out-scenario MAE 5.1 vs. 5.6; OpenEstimate 21.8 vs. 22.9) but loses on every distribution-sensitive metric.