- Rating
- 3(+2)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes AR-BED, a method for amortizing sequential Bayesian experimental design into a reasoning LLM. AR-BED uses a fixed finite contrastive prior and an external semantic likelihood oracle to score natural-language questions by expected information gain, then trains a reasoning-plus-question policy with GRPO and optionally finite-horizon information returns. Experiments on Twenty Questions compare AR-BED with direct policy training, inference-time BED-LLM, and ASIG, and study contrastive-set coverage, expected versus realized information rewards, and multi-step objectives. The paper also evaluates a separately trained medical AR-BED policy on the MediQ clinical-questioning benchmark.
Critical strengths
- The core methodological formulation is sensible and clearly motivated. In particular, separating the acting LLM from the generative model used to evaluate it addresses a real instability in policy-induced information objectives.
- The experimental design contains several informative controls. Figure 1 is particularly useful because it distinguishes test-time reasoning from BED training and contrasts AR-BED with a direct non-reasoning policy. The result that merely enabling reasoning reduces the BASE model's solve rate in several domains is an important negative control; it makes the AR-BED gains more meaningful than a comparison against a non-reasoning base alone.
- The prior-coverage ablation is valuable. Figure 2 shows that restricting the contrastive space to mammals harms performance on non-mammals and, notably, also underperforms the broader animal-trained policy on held-out mammals. This supports the paper's claim that the content of the finite hypothesis space matters, not merely its cardinality.
Critical weaknesses
- The paper’s writing and formatting contain numerous shortcomings, such as multiple instances of missing citations and a lack of appropriate figures in the main text to aid comprehension.
- The paper’s strongest causal claim, that information-gain training has been amortized “into reasoning,” is not isolated by the current ablations.
- AR-BED differs from ASIG both in using an external GPT-5 likelihood model and in optimizing an explicit reasoning-plus-question policy. The paper concludes that the benefit comes from both “amortizing the information-gain objective into the model’s reasoning process” and using an external model, but Figure 3 cannot identify the contribution of either factor separately. Similarly, BED-LLM is an inference-time candidate-selection method, whereas AR-BED is a trained question generator. This is a meaningful systems comparison, but it is not an attribution experiment.
- The clinical evaluation depends on a single LLM Patient and a single LLM decider, with no active-questioning baselines beyond the base policy.
Questions for authors
- Please define explicitly in the main paper and provide the exact posterior/predictive equations used in Equation 7. How are these weights renormalized after hypothesis pruning, and what objective is optimized after pruning?
- For Table 2, what hardware, batching policy, cache behavior, number of oracle calls, and token/API costs are included? Are the reported times end-to-end wall-clock times for identical numbers of games and training trajectories?
- For the MediQ result, can the authors clarify prominently that the evaluated medical model is separately trained on synthetic medical TQ data rather than directly transferred from the animal-trained policy? Can they add medical active-questioning baselines, repeated patient/decider samples, and qualitative clinical dialogue examples?
- Please correct any errors in writing and formatting, such as missing citations.
4 comments
Part 2/2
4. Updated MediQ Transfer Experiment
As the reviewer correctly points out, we initially trained a separate policy on Twenty Questions using a medical domain and evaluated its transfer to MediQ, a different task. We intended this MediQ experiment to test cross-task transfer within a similar domain, as opposed to the Twenty Questions evaluations that assess cross-domain transfer within the same task.
Zero-shot Transfer. To assess transfer across both tasks and domains, we now evaluate the TQ Animals-trained policy directly on the full 269 samples of our MediQ clinical diagnosis task, without further training.
Active Question-Asking Baseline. Following the reviewer's suggestion, we add BED-LLM applied to the base model as an inference-time active-questioning baseline.
Repeated Patient/Decider Samples. We now report the mean and standard error of each method across three independent repeats of the evaluation.
New Results. The results are summarized below.
| Model | Accuracy (%) ↑ | Correct-answer likelihood (%) ↑ | Brier score ↓ |
|---|---|---|---|
| Base | 88.7 ± 0.4 | 75.2 ± 0.1 | 0.191 ± 0.004 |
| BED-LLM | 89.1 ± 0.2 | 75.2 ± 0.2 | 0.187 ± 0.004 |
| AR-BED | 90.7 ± 0.6 | 76.7 ± 0.3 | 0.168 ± 0.005 |
AR-BED outperforms both BASE and BED-LLM across all three metrics, achieving 90.7% accuracy compared to 88.7% and 89.1%, respectively.
Action: We have updated Table 3 and the corresponding Section 5.4 to discuss the zero-shot transfer results.
5. Clarification of Eq. 7 & Hypothesis Pruning (Q1)
We apologize for the ambiguity in Section 4.2 and Equation 7. We have revised Section 4.2 and fixed the cross-referencing to the extended details (Appendix D in the updated manuscript) to improve the clarity of our presentation.
Clarification on Hypothesis Pruning. To reduce computational cost, hypotheses with where are excluded from subsequent likelihood updates and reward calculations.
The posterior is approximated by retaining only the active hypotheses:
The posterior predictive distribution is then recomputed using these weights:
The EIG reward (Eq. 7) is computed using the renormalized posterior and posterior predictive distributions.
Action. We have revised Section 4.2 and Equation 7 to express the EIG reward directly in terms of the posterior and posterior predictive distributions. Section 4.2 now explicitly defines both distributions and states the hypothesis-pruning procedure and its effect on the reward calculation.
6. Table 2: Experimental Setup and Cost (Q2)
We thank the reviewer for pointing this out. Table 2 originally reported the average wall-clock time per evaluation game and training step from our experimental runs. Training configurations were consistent across methods but evaluation settings (e.g., batch size) were set to maximize the available computational resources for each method.
We agree with the reviewer that a more controlled comparison that reports cache behavior, oracle calls, and token/API costs would provide a better understanding of the computational cost of each method.
New Results.
Inference Time Comparison. Both methods were benchmarked on an RTX 4070 Ti SUPER (16 GB) over 5 sequential games.
| Metric | BED-LLM | AR-BED |
|---|---|---|
| Wall time/game, mean ± SD (min) | 36.84 ± 15.18 | 5.70 ± 3.08 |
| Generated tokens/game, mean ± SD (k) | 179.20 ± 62.43 | 4.03 ± 2.28 |
| Hypotheses generated | 15 | 0 |
| Questions generated | 15 | 1 |
Training Time Comparison. Both methods were benchmarked on an NVIDIA RTX 6000 Ada GPU (48 GB) over 3 training steps with identical training settings: a batch size of 128 (microbatch size of 8 with 16 gradient-accumulation steps per update).
| Metric | AR-BED | ASIG |
|---|---|---|
| Step time, mean ± SD (s) | 216.99 ± 18.16 | 733.80 ± 120.02 |
| Total output tokens/step (local + Azure), mean ± SD (k) | 68.28 ± 9.64 | 62.10 ± 3.11 |
| Estimated API cost/step, mean ± SD (USD) | $0.36 ± $0.07 | $0.00 |
We note that AR-BED uses an external API for likelihood estimation and caches previously computed values, whereas ASIG performs likelihood estimation locally using its continuously updated training policy. Despite generating more tokens, AR-BED has a lower training time, suggesting that the speedup is primarily due to faster API-based inference, with caching potentially providing additional savings.
Action. We have replaced Table 2 and updated the corresponding experimental details in the revised manuscript.
Thank you for your time and effort. We hope that the improved manuscript and new ablations address your concerns. We are happy to engage with any more comments you may have.
Thanks for the additional response and results. I think my concerns are mostly resolved. I have raised the score.
Dear Reviewer d8VS,
Thank you for taking the time to engage with our work. Your feedback has helped us significantly improve both our experimental validation and the manuscript. We are glad to have addressed your concerns and appreciate your decision to increase your score.
The Authors of Submission #51155
Part 1/2
We thank the reviewer for their constructive feedback and for acknowledging our clearly motivated method formulation, experimental design controls, and ablation study. We have taken several steps to improve the quality of our presentation in the updated manuscript (all changes are marked in blue), and address each of the reviewers' points directly:
We specify the changes we make as Action and any new experiments as New Results.
1. Presentation (W1 & Q4)
Action: We have improved the clarity of the manuscript's writing. In particular we have:
2. Isolating the Contributions of Reasoning (W2)
We agree with the reviewer that an additional ablation is required to isolate the effect of reasoning. Our initial set of ablations (see below) evaluated AR-BED against the variant AR-BED-Direct, which ablates reasoning at training time. This demonstrated that training with reasoning improved test-time performance. However, it does not establish whether these gains arise from generating reasoning traces at inference time or a stronger question-selection policy.
New results. To isolate the contribution of reasoning at inference time, we now evaluate our trained AR-BED model with reasoning disabled during inference and show the result in Figure 1 of the updated manuscript. We show that disabling reasoning at inference time leads to a significant drop in average EIG per turn and solve rate across all four domains (e.g., -15% solve rate for both Animals and Plants). This demonstrates that the performance gains following training are largely attributable to the model's ability to reason towards information-gain.
Action: We incorporated the new ablation into Fig. 1 and the corresponding discussion in Sec. 5.1.
3. Reasoning vs. Generative Model Ablations (W3)
Thank you for raising this important point. We agree that our initial set of experiments did not clearly isolate each factor in the comparison between AR-BED and ASIG. We have added two experiments to isolate these contributions in Section 5.2.
AR-BED vs ASIG. As the reviewer correctly notes, AR-BED differs from ASIG in two ways:
The table below gives an overview of our initial and new experiments:
To isolate the contribution of the different generative models and reasoning, we train two ablations:
New Results. (Fig. 3) AR-BED-Student-Likelihood has minor performance differences compared to AR-BED across all four domains. ASIG-Reasoning performs substantially worse than AR-BED on most domains. These results suggest that the gains of AR-BED compared to ASIG are attributable to estimating the information gain with a fixed contrastive set instead of using the training policy to sample hypotheses at each step, with a stronger likelihood model providing only marginal additional benefits.
Moreover, ASIG does not exhibit the same gains from reasoning at inference time as AR-BED. This suggests that the stronger information-gain signal from the fixed contrastive set is more easily amortized into the reasoning policy than the weaker information-gain signal that uses sampled hypotheses.
Action. We present the results in Fig. 3 and discussion in Sec. 5.2 of the updated manuscript.