acceptodds
Under review as a conference paper at ICLR 2027

Amortizing Sequential Experimental Design in Reasoning LLMs

Abstract

Despite their strong reasoning capabilities and extensive pretraining knowledge, large language models (LLMs) struggle to adaptively gather information in experimental design tasks. We study how to amortize sequential Bayesian experimental design (BED) into a reasoning LLM, using information gain as the training signal. Our approach decouples the experimental generative model from the policy, providing a fixed training distribution and avoiding reliance on the LLM's own potentially biased beliefs about the experiment. This construction also allows us to marginalize over hypotheses and outcomes to train on expected rewards with less variance, and to extend the objective across future interactions for non-myopic training. On Twenty Questions, the resulting policies improve information acquisition and task performance, outperform inference- and training-time BED baselines at substantially lower computational cost, and transfer to unseen domains and to Medi-Q, a real-world clinical dialogue dataset. These results suggest that information gathering can be learned as a transferable reasoning skill in LLMs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Bayesian Experimental Design, Large Language Models, Amortization

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_d8VSOpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes AR-BED, a method for amortizing sequential Bayesian experimental design into a reasoning LLM. AR-BED uses a fixed finite contrastive prior and an external semantic likelihood oracle to score natural-language questions by expected information gain, then trains a reasoning-plus-question policy with GRPO and optionally finite-horizon information returns. Experiments on Twenty Questions compare AR-BED with direct policy training, inference-time BED-LLM, and ASIG, and study contrastive-set coverage, expected versus realized information rewards, and multi-step objectives. The paper also evaluates a separately trained medical AR-BED policy on the MediQ clinical-questioning benchmark.

Critical strengths
  1. The core methodological formulation is sensible and clearly motivated. In particular, separating the acting LLM from the generative model used to evaluate it addresses a real instability in policy-induced information objectives.
  2. The experimental design contains several informative controls. Figure 1 is particularly useful because it distinguishes test-time reasoning from BED training and contrasts AR-BED with a direct non-reasoning policy. The result that merely enabling reasoning reduces the BASE model's solve rate in several domains is an important negative control; it makes the AR-BED gains more meaningful than a comparison against a non-reasoning base alone.
  3. The prior-coverage ablation is valuable. Figure 2 shows that restricting the contrastive space to mammals harms performance on non-mammals and, notably, also underperforms the broader animal-trained policy on held-out mammals. This supports the paper's claim that the content of the finite hypothesis space matters, not merely its cardinality.
Critical weaknesses
  1. The paper’s writing and formatting contain numerous shortcomings, such as multiple instances of missing citations and a lack of appropriate figures in the main text to aid comprehension.
  2. The paper’s strongest causal claim, that information-gain training has been amortized “into reasoning,” is not isolated by the current ablations.
  3. AR-BED differs from ASIG both in using an external GPT-5 likelihood model and in optimizing an explicit reasoning-plus-question policy. The paper concludes that the benefit comes from both “amortizing the information-gain objective into the model’s reasoning process” and using an external model, but Figure 3 cannot identify the contribution of either factor separately. Similarly, BED-LLM is an inference-time candidate-selection method, whereas AR-BED is a trained question generator. This is a meaningful systems comparison, but it is not an attribution experiment.
  4. The clinical evaluation depends on a single LLM Patient and a single LLM decider, with no active-questioning baselines beyond the base policy.
Questions for authors
  1. Please define wt−1,ℓw_{t-1,\ell} explicitly in the main paper and provide the exact posterior/predictive equations used in Equation 7. How are these weights renormalized after hypothesis pruning, and what objective is optimized after pruning?
  2. For Table 2, what hardware, batching policy, cache behavior, number of oracle calls, and token/API costs are included? Are the reported times end-to-end wall-clock times for identical numbers of games and training trajectories?
  3. For the MediQ result, can the authors clarify prominently that the evaluated medical model is separately trained on synthetic medical TQ data rather than directly transferred from the animal-trained policy? Can they add medical active-questioning baselines, repeated patient/decider samples, and qualitative clinical dialogue examples?
  4. Please correct any errors in writing and formatting, such as missing citations.
4 comments

We thank the reviewer for their constructive feedback and for acknowledging our clearly motivated method formulation, experimental design controls, and ablation study. We have taken several steps to improve the quality of our presentation in the updated manuscript (all changes are marked in blue), and address each of the reviewers' points directly:

  1. Presentation [W1 & Q4]
  2. Isolating the Contributions of Reasoning [W2]
  3. Reasoning vs. Generative Model Ablations [W3]
  4. Updated MediQ Transfer Experiment [W4 & Q3]
  5. Clarification of Eq. 7 and Hypothesis Pruning [Q1]
  6. Table 2: Experimental Setup and Cost [Q2]

We specify the changes we make as Action and any new experiments as New Results.

1. Presentation (W1 & Q4)

Action: We have improved the clarity of the manuscript's writing. In particular we have:

  • Corrected cross-reference errors.
  • Clarified the details of the Method and Experiments (Sec. 4.2 and App. D.3, Sec. 5).
  • Revised the Introduction and Background (Sec. 2) to improve readability.

2. Isolating the Contributions of Reasoning (W2)

We agree with the reviewer that an additional ablation is required to isolate the effect of reasoning. Our initial set of ablations (see below) evaluated AR-BED against the variant AR-BED-Direct, which ablates reasoning at training time. This demonstrated that training with reasoning improved test-time performance. However, it does not establish whether these gains arise from generating reasoning traces at inference time or a stronger question-selection policy.

ModelPolicyReasoning at Inference
BASE–No
BASE + Reasoning–Yes
AR-BED-Directπ(x∣h)\pi(x\mid h)No
AR-BED-Direct + Reasoningπ(x∣h)\pi(x\mid h)Yes
AR-BED + No Reasoning (NEW, see below)π(z,x∣h)\pi(z,x \mid h)No
AR-BED (Ours)π(z,x∣h)\pi(z,x \mid h)Yes

New results. To isolate the contribution of reasoning at inference time, we now evaluate our trained AR-BED model with reasoning disabled during inference and show the result in Figure 1 of the updated manuscript. We show that disabling reasoning at inference time leads to a significant drop in average EIG per turn and solve rate across all four domains (e.g., -15% solve rate for both Animals and Plants). This demonstrates that the performance gains following training are largely attributable to the model's ability to reason towards information-gain.

Action: We incorporated the new ablation into Fig. 1 and the corresponding discussion in Sec. 5.1.

3. Reasoning vs. Generative Model Ablations (W3)

Thank you for raising this important point. We agree that our initial set of experiments did not clearly isolate each factor in the comparison between AR-BED and ASIG. We have added two experiments to isolate these contributions in Section 5.2.

AR-BED vs ASIG. As the reviewer correctly notes, AR-BED differs from ASIG in two ways:

  1. Policy: AR-BED uses reasoning instead of directly sampling questions.
  2. Generative-model: AR-BED specifies a generative model separate from the training policy, defined by:
    • Prior: A fixed contrastive hypothesis set instead of hypotheses sampled at each step.
    • Likelihood Evaluator: An external model (GPT-5) instead of the training policy (Gemma-E4B-it).

The table below gives an overview of our initial and new experiments:

Policy \ Gen. ModelExternal: Fixed Prior, GPT-5 LikelihoodExternal: Fixed Prior, Gemma E-4B-it LikelihoodTraining Policy: Sampled hypotheses, Likelihood
Reasoning π(z,x∣h)\pi(z,x \mid h)AR-BEDAR-BED-Student-Likelihood (NEW)ASIG-Reasoning (NEW)
Direct π(x∣h)\pi(x \mid h)AR-BED-DirectASIG

To isolate the contribution of the different generative models and reasoning, we train two ablations:

  1. AR-BED-Student-Likelihood uses the base model Gemma-E4B-it instead of GPT-5 to evaluate likelihoods during training. This isolates the effect of the likelihood evaluator.
  2. ASIG-Reasoning uses reasoning at training time. This isolates the effect of reasoning under different generative-model specifications.

New Results. (Fig. 3) AR-BED-Student-Likelihood has minor performance differences compared to AR-BED across all four domains. ASIG-Reasoning performs substantially worse than AR-BED on most domains. These results suggest that the gains of AR-BED compared to ASIG are attributable to estimating the information gain with a fixed contrastive set instead of using the training policy to sample hypotheses at each step, with a stronger likelihood model providing only marginal additional benefits.

Moreover, ASIG does not exhibit the same gains from reasoning at inference time as AR-BED. This suggests that the stronger information-gain signal from the fixed contrastive set is more easily amortized into the reasoning policy than the weaker information-gain signal that uses sampled hypotheses.

Action. We present the results in Fig. 3 and discussion in Sec. 5.2 of the updated manuscript.

4. Updated MediQ Transfer Experiment

As the reviewer correctly points out, we initially trained a separate policy on Twenty Questions using a medical domain and evaluated its transfer to MediQ, a different task. We intended this MediQ experiment to test cross-task transfer within a similar domain, as opposed to the Twenty Questions evaluations that assess cross-domain transfer within the same task.

Zero-shot Transfer. To assess transfer across both tasks and domains, we now evaluate the TQ Animals-trained policy directly on the full 269 samples of our MediQ clinical diagnosis task, without further training.

Active Question-Asking Baseline. Following the reviewer's suggestion, we add BED-LLM applied to the base model as an inference-time active-questioning baseline.

Repeated Patient/Decider Samples. We now report the mean and standard error of each method across three independent repeats of the evaluation.

New Results. The results are summarized below.

ModelAccuracy (%) ↑Correct-answer likelihood (%) ↑Brier score ↓
Base88.7 ± 0.475.2 ± 0.10.191 ± 0.004
BED-LLM89.1 ± 0.275.2 ± 0.20.187 ± 0.004
AR-BED90.7 ± 0.676.7 ± 0.30.168 ± 0.005

AR-BED outperforms both BASE and BED-LLM across all three metrics, achieving 90.7% accuracy compared to 88.7% and 89.1%, respectively.

Action: We have updated Table 3 and the corresponding Section 5.4 to discuss the zero-shot transfer results.

5. Clarification of Eq. 7 & Hypothesis Pruning (Q1)

We apologize for the ambiguity in Section 4.2 and Equation 7. We have revised Section 4.2 and fixed the cross-referencing to the extended details (Appendix D in the updated manuscript) to improve the clarity of our presentation.

Clarification on Hypothesis Pruning. To reduce computational cost, hypotheses with C≤τC\leq\tau where τ=0.01\tau=0.01 are excluded from subsequent likelihood updates and reward calculations.

The posterior is approximated by retaining only the active hypotheses:

p~C(θℓ∣ht−1)=Ct−1,ℓ1{Ct−1,ℓ>τ}∑jCt−1,j1{Ct−1,j>τ}\tilde p_C(\theta_\ell\mid h_{t-1}) = \frac{C_{t-1,\ell}\mathbf{1}\{C_{t-1,\ell}>\tau\}}{\sum_j C_{t-1,j}\mathbf{1}\{C_{t-1,j}>\tau\}}

The posterior predictive distribution is then recomputed using these weights:

p~C(y∣ht−1,xt)=∑ℓ=0Lp~C(θℓ∣ht−1)poracle(y∣θℓ,xt)\tilde p_C(y\mid h_{t-1},x_t) = \sum_{\ell=0}^{L}\tilde p_C(\theta_\ell\mid h_{t-1})p_{\mathrm{oracle}}(y\mid\theta_\ell,x_t)

The EIG reward (Eq. 7) is computed using the renormalized posterior and posterior predictive distributions.

Action. We have revised Section 4.2 and Equation 7 to express the EIG reward directly in terms of the posterior and posterior predictive distributions. Section 4.2 now explicitly defines both distributions and states the hypothesis-pruning procedure and its effect on the reward calculation.

6. Table 2: Experimental Setup and Cost (Q2)

We thank the reviewer for pointing this out. Table 2 originally reported the average wall-clock time per evaluation game and training step from our experimental runs. Training configurations were consistent across methods but evaluation settings (e.g., batch size) were set to maximize the available computational resources for each method.

We agree with the reviewer that a more controlled comparison that reports cache behavior, oracle calls, and token/API costs would provide a better understanding of the computational cost of each method.

New Results.

Inference Time Comparison. Both methods were benchmarked on an RTX 4070 Ti SUPER (16 GB) over 5 sequential games.

MetricBED-LLMAR-BED
Wall time/game, mean ± SD (min)36.84 ± 15.185.70 ± 3.08
Generated tokens/game, mean ± SD (k)179.20 ± 62.434.03 ± 2.28
Hypotheses generated150
Questions generated151

Training Time Comparison. Both methods were benchmarked on an NVIDIA RTX 6000 Ada GPU (48 GB) over 3 training steps with identical training settings: a batch size of 128 (microbatch size of 8 with 16 gradient-accumulation steps per update).

MetricAR-BEDASIG
Step time, mean ± SD (s)216.99 ± 18.16733.80 ± 120.02
Total output tokens/step (local + Azure), mean ± SD (k)68.28 ± 9.6462.10 ± 3.11
Estimated API cost/step, mean ± SD (USD)$0.36 ± $0.07$0.00

We note that AR-BED uses an external API for likelihood estimation and caches previously computed values, whereas ASIG performs likelihood estimation locally using its continuously updated training policy. Despite generating more tokens, AR-BED has a lower training time, suggesting that the speedup is primarily due to faster API-based inference, with caching potentially providing additional savings.

Action. We have replaced Table 2 and updated the corresponding experimental details in the revised manuscript.


Thank you for your time and effort. We hope that the improved manuscript and new ablations address your concerns. We are happy to engage with any more comments you may have.

Thanks for the additional response and results. I think my concerns are mostly resolved. I have raised the score.

Dear Reviewer d8VS,

Thank you for taking the time to engage with our work. Your feedback has helped us significantly improve both our experimental validation and the manuscript. We are glad to have addressed your concerns and appreciate your decision to increase your score.

The Authors of Submission #51155

Reviewer_a6spOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to improve information gathering in reasoning LLMs by amortizing sequential Bayesian experimental design (BED) into the model's policy. The authors propose AR-BED, which uses a fixed contrastive hypothesis set and an external LLM to estimate expected information gain. What's more, they further introduce an expected information-gain reward that marginalizes over possible hypotheses and outcomes, and extend it to multi-step optimization using GRPO. Experiments on Twenty Questions show improvements in information acquisition and solve rate across several domains.

Critical strengths
  • Overall, I think this paper presents a well-motivated approach to improving information gathering in reasoning LLMs. Decoupling the generative model from the policy is indeed promising to avoid relying on potentially inaccurate predictions from the policy itself.

  • In addition, I find the expected information-gain reward theoretically well motivated.

  • AR-BED generally outperforms existing BED baselines across multiple domains while reducing inference cost.

Critical weaknesses
  • The comparison with existing BED methods does not fully isolate the proposed contributions. If I understand correctly, AR-BED uses GPT-5 as an external likelihood evaluator, whereas the baselines rely on predictions from the smaller policy model. It is therefore unclear how much of the improvement comes from decoupling the generative model versus using a stronger model to provide the training signal.

  • In addition, it seems that the MediQ experiments use a separately trained medical policy rather than directly transferring the Animals-trained policy. While the results demonstrate useful generalization to held-out diagnoses and specialties, they do not establish zero-shot transfer across tasks as the paper may suggest as it's contribution.

  • Finally, the improvements from multi-step training on Twenty Questions are relatively small. In addition, the reported results are based on three evaluation repeats of the same trained model, rather than multiple independently trained models. More training seeds and statistical analysis would help show whether these improvements are consistent.

Questions for authors

Please refer to my summary of weaknesses.

3 comments

We thank the reviewer for acknowledging the motivation behind our approach, the promise of decoupling the generative model from the policy, the theoretical motivation for our EIG reward, and the empirical improvements over existing BED baselines.

We also address of the reviewer's concerns in four parts:

  1. Isolating the Contribution of the Decoupled Generative Model [W1]
  2. Cross-Task Generalization [W2]
  3. Limitations of Multi-step Training in Twenty Questions [W3, part 1]
  4. Training Seeds [W3, part 2]

We specify the changes to the manuscript as Action and any new experiments as New Results. Please see the updated manuscript with changes marked in blue.

1. Isolating the Contribution of the Decoupled Generative Model [W1]

We thank the reviewer for this comment. One advantage of decoupling the generative model from the training policy is that it allows us to define it ourselves, potentially providing (1) a more meaningful hypothesis space to disambiguate and (2) better-calibrated likelihood estimates compared to using the training policy. However, we agree that it is important to disentangle whether the improvements arise from this decoupling or from using a stronger model as the likelihood evaluator.

To answer the reviewer's question, we introduce two new AR-BED ablations: (1) AR-BED-Student-Likelihood that uses the fixed contrastive hypothesis set and replaces the GPT-5 likelihood evaluator with the base Gemma4-E4B model used as the training policy, and (2) ASIG-Reasoning which trains ASIG with reasoning at training time.

New Results. (Fig. 3) AR-BED-Student-Likelihood leads to minor performance differences compared to AR-BED across all four domains, while ASIG-Reasoning leads to a larger drop in EIG and solve rate across domains. These results suggest that the gains of AR-BED are attributable to decoupling the generative model--using a fixed contrastive set to estimate the information-gain rather than sampling hypothesis with the training policy--, with a stronger likelihood model providing only marginal additional benefits. One possible explanation is that estimating p(y∣θ,x)p(y\mid\theta,x) is relatively tractable even for smaller models, consistent with the observations of Choudhury et al, whereas sampling diverse and accurate possible hypotheses given a multi-step history p(θ∣h)p(\theta|h) is a more challenging task.

Action. We present these new results in Figure 3 and the corresponding discussion in Section 5.2 of the updated manuscript.

2. Clarification of Cross-Task Generalization [W2]

We thank the reviewer for raising this point. The reviewer is correct that we trained a separate policy on Twenty Questions using medical dermatology terms and evaluated its transfer to MediQ, a different task, with both in-distribution (dermatology) and out-of-distribution medical questions. We initially designed this MediQ experiment to test cross-task transfer within a similar domain, as opposed to the Twenty Questions evaluations to assess cross-domain transfer within the same task.

Zero-shot Generalization Performance. We agree with the reviewer that evaluating the Animals TQ policy directly on MediQ would assess zero-shot trasnsfer across both tasks and domains. To investigate this, we now evaluate the Animals-trained TQ policy on MediQ. We report the mean and standard error of each method across three independent repeats of the evaluation.

New Results. The results are summarized below.

ModelAccuracy (%) ↑Correct-answer likelihood (%) ↑Brier score ↓
Base88.7 ± 0.475.2 ± 0.10.191 ± 0.004
BED-LLM89.1 ± 0.275.2 ± 0.20.187 ± 0.004
AR-BED90.7 ± 0.676.7 ± 0.30.168 ± 0.005

AR-BED outperforms both BASE and BED-LLM across all three metrics, achieving 90.7% accuracy compared to 88.7% and 89.1%, respectively. These results suggest that the reasoning abilities learned on a synthetic Twenty Question transfers positively to other tasks and domains without further training.

Action: We have updated Table 3 and the corresponding Section 5.4 to discuss the zero-shot transfer results.

3. Multi-step Training [W3, part 1]

We agree with the reviewer that the gains from multi-step training on Twenty Questions are relatively small. We would like to clarify that multi-step training is not the primary contribution of our work, but we investigate it because our reward formulation allows information-gain rewards to be accumulated across multiple steps.

Multi-step Training on Twenty Questions. We compare policies trained with different rollout horizons (K=1,2,3K=1,2,3), evaluating their information acquisition over one-, two-, and three-question horizons. We find that the benefits of non-myopic training in Twenty Questions are modest and do not consistently increase with the training horizon (Appendix B.5.2).

Multi-step Training on Non-Myopic Environments. To investigate when multi-step training is beneficial, we additionally evaluate on a natural-language adaptation of Cube-NM, where an initial experiment enables more informative subsequent experiments. Here, multi-step training substantially improves information acquisition over the one-step policy (Appendix B.5.3-4).

Action. We clarify the motivation and findings of our multi-step experiments in Section 5.3.3, and update the limitations section in Section 6.

4. Additional Training Seeds [W3, part 2]

The reviewer is correct that our results use three evaluation repeats of the trained models. This is due to the computational cost of training separate models for each ablation and benchmark, which takes up to 30 hours per model on our hardware. We agree that additional training seeds are important for assessing the consistency of our results. For the rebuttal period, we prioritized running additional training seeds for our main method, AR-BED.

New Results. We train AR-BED with two additional seeds (three repeats in total), and compare the training reward curves and the solve rate across the four TQ domains. The three repeats perform similarly at both training and evaluation: the solve rates at 20 turns across the three repeats has a standard deviation of 3.21 percentage points on average across the eight domains-reasoning settings. The results demonstrate that our findings are consistent across independently trained models.

Action. We added these additional results in Appendix B.3 to demonstrate the consistency across training seeds.


Thank you for your time and effort. We hope that the additional ablations address your concerns. We are more than happy to engage with any further comments you may have.

Dear Reviewer a6sp,

Thank you once again for your careful evaluation of our submission. We hope that our response, new results, and updated manuscript have addressed your concerns. Should any questions remain, we would be happy to use the remaining discussion period to provide further clarification. We look forward to your response.

Best regards,

The Authors of Submission #51155

Reviewer_YphaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on the adaptive information-acquisition capabilities of LLMs in multi-turn interactive tasks. In scenarios such as medical diagnosis or experimental design, where information is incomplete, a model must not only reason based on existing data but also proactively determine what information to seek next and how to refine its subsequent information-gathering strategy. Although current LLMs possess strong reasoning abilities and a broad base of pre-trained knowledge, they often struggle to continuously update their internal assessments or effectively adjust their information-seeking strategies throughout a multi-turn interaction. This work approaches the problem by "amortizing" Bayesian experimental design into the LLM, transforming information acquisition from an explicit search-and-optimization process (conducted during inference) into a strategy that can be learned during the training phase.

Critical strengths

1.The work focuses on the issue of adaptive information acquisition during multi-turn interactions with LLMs, a research problem of practical significance.

2.The approach decouples the generative model from the policy and constructs training rewards based on expected information gain; the design logic is clear.

3.The method’s effectiveness is validated through cross-domain and cross-task experiments, demonstrating a certain degree of transferability.

Critical weaknesses
  1. Although this paper decouples the generative model from the hypothesis-based strategy, the external LLM’s estimates of answer probabilities under various hypotheses may still be biased. Furthermore, a fixed, finite set of hypotheses may not adequately cover the full hypothesis space present in real-world tasks; consequently, the reliability of the training signal remains dependent on both the external model and the definition of the hypothesis space. Experiments show that when the hypothesis space is restricted to "Mammals," the model’s performance on the "Animals" task drops significantly, indicating that the method is highly sensitive to the coverage of the hypothesis space.

  2. The paper decomposes the model’s output into a reasoning process and a final question, optimizing both jointly. However, the information-gain reward is calculated based solely on the final question, with no independent quality metric for the reasoning process itself. As a result, it is unclear whether performance improvements stem from the model acquiring more generalizable information-analysis capabilities during the reasoning phase, or simply from learning a more effective strategy for generating questions. There is also insufficient evidence to quantify the actual contribution of the reasoning process to the observed performance gains.

  3. The authors argue that single-step information gain is "myopic," and thus introduce a multi-step reward to account for how a current question facilitates future information acquisition. However, they do not sufficiently demonstrate the prevalence of this issue in general natural-language interaction tasks. Moreover, the actual training employs a short rollout horizon and relies on the external evaluator’s most probable response to drive subsequent interactions—rather than sampling from the full distribution of stochastic outcomes—which deviates from the theoretical ideal of complete sequential optimization. While controlled experiments with Cube-NM suggest that multi-step optimization can foster non-myopic strategies, those experiments did not involve a reasoning process, relied on a single random seed, and showed only limited performance gains in natural-language tasks.

  4. The study focuses primarily on the "Twenty Questions" task. While experiments involving categories such as "Animals," "Plants," "Celebrities," and "Countries" demonstrate cross-domain transfer, these tasks all fundamentally rely on narrowing down a candidate hypothesis space through a sequence of questions—sharing a common mechanism for information acquisition. Therefore, these results are insufficient to prove that the method can generalize to tasks that employ fundamentally different mechanisms for gathering information.

  5. Although the MediQ experiment extends to clinical dialogue scenarios, the authors actually trained a separate policy specifically for medical tasks rather than directly transferring the policy learned from "Twenty Questions" to these medical tasks; consequently, the evidence for cross-task transfer capability remains limited.

Questions for authors

see weaknesses

5 comments

We thank the reviewer for recognizing the practical significance of our research problem, the clarity of our methodological approach, and the empirical evidence of generalization. We found their feedback constructive and address each of their concerns below:

  1. Generative Model Reliance [W1]
  2. Isolating the Contribution of Reasoning [W2]
  3. Non-Myopic Information Acquisition [W3]
  4. Clarification on our Cross-Task Transfer [W4 & W5]

We specify the changes to the manuscript as Action and any new experiments as New Results. Please see the updated manuscript with changes marked in blue.

1. Generative Model Reliance [W1]

We thank the reviewer for raising this important point. Our method, like any BED-based approach, depends on how the generative model of the experiment is constructed (Choudhury et al 2025). We thus agree that the reliability of the training signal is dependent on both the external likelihood evaluator and the definition of the hypothesis space.

We study the effect of generative model in three experiments:

  1. Hypothesis-space coverage (Section 5.3.1). We show that a misspecification between the task definition (Animals) and the hypothesis space (Mammals) worsens performance.
  2. Hypothesis-space cardinality (Appendix B.2). We demonstrate that increasing the number of hypotheses in the contrastive set improves performance.
  3. Likelihood Evaluator (NEW). We now investigate the effect of using different likelihood models (GPT-5 vs. Gemma4-E4B) to generate the training signal. Our new results (Figure 3, Section 5.2) suggests that using a weaker model like Gemma4-E4B does not significantly affect performance. Instead, the gains over ASIG primarily come from stimating the information-gain using a fixed contrastive set rather than sampling hypotheses with the training policy.

Future Work. We agree that addressing generative model misspecification is an important ongoing direction for BED-based methods. For instance, Step-DAD (Hedman et al., 2025) addresses this using test-time adaptation of a pre-trained policy. We view this research direction as beyond the primary objective of our work: investigating if amortizing BED into a reasoning LLM can develop generalizable experimental design reasoning capabilities.

Action. We have revised our limitations section to stress the importance of the generative model more broadly, including the likelihood-model.

2. Isolating the Contribution of Reasoning [W2]

We thank the reviewer for raising this point. We agree that it is important to distinguish whether our method's performance arises from training the reasoning process or simply learning better question-sampling behavior.

The table below summarizes our inital and new ablations designed to isolate these contributions.

ModelPolicyReasoning at Inference
BASE–No
BASE + Reasoning–Yes
AR-BED-Directπ(x∣h)\pi(x \mid h)No
AR-BED-Direct + Reasoningπ(x∣h)\pi(x \mid h)Yes
AR-BED + No Reasoning (NEW, see below)π(z,x∣h)\pi(z,x \mid h)No
AR-BED (Ours)π(z,x∣h)\pi(z,x \mid h)Yes

Training with reasoning improve performance. Our existing ablations compare AR-BED with AR-BED-Direct, isolating the effect of training a reasoning policy rather than directly training question selection. We find that reasoning-based training improves information acquisition and generalization compared to direct question-sampling training.

New results. To isolate whether the benefits of reasoning-based training arise from generating a reasoning trace at inference time or from the learned question-sampling behavior, we introduce a new ablation evaluating our trained AR-BED model with reasoning disabled. We show that disabling reasoning at inference time leads to a significant drop in average EIG per turn and solve rate across all four domains (e.g., -15% solve rate for both Animals and Plants). This suggests that the performance gains following training are largely attributable to the model's ability to reason towards information-gain.

Action: We incorporated the new ablation into Figure 1 and the corresponding discussion in Section 5.1.

3. Non-Myopic Information Acquisition [W3]

We thank the reviewer for allowing us to clarify this point. We would like to clarify that addressing the myopic nature of one-step EIG is not the primary motivation or contribution of our method. Rather, following DAD (see Sec. 2), our objective is to amortize experimental design into a policy while avoiding repeated posterior inference and EIG maximization at each step. A result of this formulation is that it allows information-gain rewards to be accumulated across multiple steps, and so we investigate the potential benefits of non-myopic training.

Prevalence of Non-Myopic Environments in Natural-Language. We find that the benefits of non-myopic training in TQ are modest and do not consistently increase with the training horizon. We include Cube-NM (Appendix B.5.3–B.5.4) to demonstrate the existence of a natural-language environment in which myopic information acquisition is suboptimal. In this environment, an initial context test provides zero immediate information about the target disease, but reveals which subsequent tests will be informative. We formally show that a one-step information-gain objective cannot incentivize this action, whereas a two-step objective can. Empirically, this is shown with a 75.0% solve rate for the two-step policy versus 30.2% for the one-step policy after five tests.

Limitations. We acknowledge that short training horizons and the use of most-probable oracle responses are practical approximations to full sequential optimization. We leave a more thorough investigation of non-myopic environemnts and training to future work, which we will clarify more explicitly in our limitations section.

Action. We clarify the motivation, findings, and limitations of our non-myopic experiments in Section 5.3.3 and clarify this as a future work in our limitations section.

4. Clarification on our Cross-Task Transfer [W4 & W5]

We thank the reviewer for raising these concerns.

Cross-Domain Transfer.

As the reviewer correctly notes, the TQ experiments examine if a reasoning LLM trained with our information-gain reward transfers across different domains, where the underlying information-gathering strategy is the same. We agree that these results do not demonstrate generalization to tasks that require different information-gathering strategies.

Cross-Task Transfer.

We would first like to clarify that in the MediQ experiment, we initially trained a separate policy on Twenty Questions using a medical domain and evaluated its transfer to MediQ, a different task. We intended this MediQ experiment to test cross-task transfer within a similar domain, as opposed to the Twenty Questions evaluations that assess cross-domain transfer within the same task.

Nevertheless, we agree with the reviewer that evaluating the Animals-trained Twenty Questions policy directly on MediQ would provide a stronger test of cross-task generalization. We now compare this to the base model and BED-LLM and report the mean and standard error of each method across three independent repeats of the evaluation.

New Results. The results are summarized below.

ModelAccuracy (%) ↑Correct-answer likelihood (%) ↑Brier score ↓
Base88.7 ± 0.475.2 ± 0.10.191 ± 0.004
BED-LLM89.1 ± 0.275.2 ± 0.20.187 ± 0.004
AR-BED90.7 ± 0.676.7 ± 0.30.168 ± 0.005

AR-BED outperforms both BASE and BED-LLM across all three metrics, achieving 90.7% accuracy compared to 88.7% and 89.1%, respectively. These results suggest that the reasoning abilities learned on a synthetic Twenty Question transfers positively to other tasks and domains without further training.

Action: We have updated Table 3 and the corresponding Section 5.4 to discuss the cross-task generalization results.


Thank you for your time and effort. We hope that the modified manuscript and additional experiments address your concerns. We are more than happy to engage with any further comments you may have.

Dear Reviewer Ypha,

Thank you once again for your careful evaluation of our submission. We hope our clarifications, new results, and updated manuscript have addressed your concerns. Should any questions remain, we would be happy to use the remaining discussion period to provide further clarification. We look forward to your response.

Best regards,

The Authors of Submission #51155

Reviewer_BiTzOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper fine-tunes an LLM to ask better questions in the sequential Bayesian experimental design framework. They use a teacher model to estimate the likelihoods and use that information to obtain the rewards used for the training via GPRO. The fine-tuned model asks better questions and improves the solve rate compared to the base-model and other approaches and generalizes to unseen experiments.

Critical strengths

The paper is well motivated and offers good insight of the literature.

Decoupling the policy form the reward model is a good idea and Sections G5 and G7 provide theorotical analysis.

Experimental evalutions are comprehensive and demonstrate the benfits of the approach, including the generalizabilty to new domains.

Critical weaknesses

1- The main issue of the paper is that EIG as the evaultion metric uses the same GPT-5 that is used to train the model. In particular, it is not clear to me that the imporvements are actually coming from the method, and not just by generally improving the model through fine-tuning. In other words, if the benefit is coming from the sepeartion, it does not need to use a bigger model like GPT-5 but two copies should work well enough.

2- Only one seed of traning is used.

Questions for authors

1- Why you did not compare with unmodified ASIG?

2- Report all training costs and hardware used.

3- Report your experiments across multiples traning seeds.

Please confirm that all editorial issues will be fixed, such as broken references and typos.

2 comments

We thank the reviewer for their positive assessment of our work. We address each of the reviewer's remaining concerns and questions below.

  1. Assessing the Impact of our Training Method on Model Performance [W1]
  2. Additional Training Seeds [W2 & Q3]
  3. Comparison with Unmodified ASIG [Q1]
  4. Training Costs and Hardware [Q2]
  5. Confirmation of fixed editorial issues

We specify the changes to the manuscript as Action and any new experiments as New Results. Please see the updated manuscript with changes marked in blue.

1. Assessing the Impact of our Training Method on Model Performance [W1]

We thank the reviewer for this important point. We agree that it is important to disentangle whether the improvements arise from decoupling the generative model from the training policy or simply from using a stronger likelihood evaluator.

Isolating the Contribution of the Likelihood Evaluator. To investigate this, we introduce to new ablations: (1) AR-BED-Student-Likelihood, which uses the same contrastive hypothesis set but replaces GPT-5 with a fixed copy of the base Gemma-E4B-it model, as suggested by the reviewer, and (2) ASIG-Reasoning which trains ASIG with reasoning at training time.

Are the benefits from our training method or general fine-tuning? The ablations in the table below isolate the contributions of (i) reasoning during training, (ii) decoupling the generative model from the policy, and (iii) the strength of the likelihood evaluator.

MethodReasoning During TrainingGenerative ModelLikelihood Evaluator
AR-BEDYesDecoupledGPT-5
AR-BED-Student-Likelihood (New)YesDecoupledGemma E4B
AR-BED-DirectNoDecoupledGPT-5
ASIG + Reasoning (New)YesPolicy-basedPolicy LLM
ASIGNoPolicy-basedPolicy LLM

New Results. We find that AR-BED-Student-Likelihood reaches slightly lower average EIG and comparable solve rate to AR-BED across all four domains, while ASIG-Reasoning leads to a larger drop in EIG and solve rate across domains.

These results suggest that the gains of AR-BED compared to ASIG are attributable to estimating the information gain with a fixed contrastive set instead of using the training policy to sample hypotheses at each step. As the reviewer correctly points out, using GPT-5 as a stronger likelihood model only provides marginal additional benefits.

Action. We present the new ablation results in Figure 3 and update the corresponding discussion in Section 5.2.

2. Training Seeds [W2 & Q3]

As the reviewer correctly states, our initial results only use one seed of training for each model. This is due to the computational cost of training separate models for each ablation and benchmark (up to 30 hours for a single model on our hardware). We agree that additional training seeds are important for assessing the consistency of our results. For the rebuttal period, we prioritized running additional training seeds for our main method, AR-BED.

New Results. We train AR-BED with two additional seeds (three repeats in total), and compare the training reward curves and the solve rate across the four TQ domains. The three repeats perform similarly at both training and evaluation: the solve rates at 20 turns across the three repeats has a standard deviation of 3.21 percentage points on average across the eight domains-reasoning settings. The results demonstrate that our findings are consistent across independently trained models.

Action. We added these additional results in Appendix B.3 to demonstrate the consistency across training seeds.

3. Why did we not compare to unmodified ASIG? [Q1]

We thank the reviewer for raising this point. The original ASIG method by Hartmann et al. uses a reward that combines outcome-based performance (i.e., whether the model solves the Twenty Questions game) and EIG (i.e., how effectively it eliminates hypotheses). As Hartmann et al. note in their conclusion, this combined reward makes it difficult to disentangle the effects of EIG and outcome-based rewards.

In our work, we wanted to assess the effect of the EIG reward alone, focusing on (i) the importance of reasoning for learning generalizable experimental design strategies and (ii) the construction of the generative model and its impact on performance. For this reason, we isolate the EIG reward and exclude the outcome-based reward as stated in Section 5.2.

4. Training Cost [Q2]

AR-BED was trained for 250 optimizer updates on a single NVIDIA RTX 6000 Ada GPU (48 GB), requiring 16 GPU-hours and $77.45 in LLM API costs.

Action. We now report the training cost and hardware in Appendix B.1.

5. Confirmation of fixed editorial issues

Action. We have fixed all editorial issues including the broken cross-references and typos.


Thank you for your time and effort. We hope that the modified manuscript and additional ablations address your concerns. We are more than happy to engage with any further comments you may have.

Dear Reviewer BiTz,

Thank you once again for your careful evaluation of our submission. We hope our clarifications, new results, and updated manuscript have addressed your concerns. Should any questions remain, we would be happy to use the remaining discussion period to provide further clarification. We look forward to your response.

Best regards,

The Authors of Submission #51155

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary for the Area Chair

Dear Area Chair,

Thank you for your time and effort in overseeing the review of our paper. We would also like to thank the reviewers once again for their constructive feedback which has greatly strengthened our submission. As the author–reviewer discussion period comes to a close, we would like to briefly summarize how we have addressed the reviewers' concerns.

As a result, both reviewers who responded acknowledged the improvements to our submission and increased their scores: Reviewer d8VS from 1 to 3, and Reviewer Ypha from 2 to 3. Although we did not receive responses from Reviewers a6sp and BiTz, we hope that our new results, clarifications, and manuscript revisions have addressed their concerns.

The table below summarizes the major changes and where they can be found in our rebuttal responses and revised manuscript (with all updates shown in blue).

ConcernReviewerChangeRelevant Section
Isolating the Contribution of Reasoning at Inference Timed8vs W2; ypha W2We added an ablation showing that disabling reasoning at inference time degrades AR-BED's performance across all domains, demonstrating the importance of reasoning to its information-gathering performance.Rebuttal: d8vs (2.), ypha (2.). Manuscript: Sec. 5.1
Isolating Reasoning vs. Generative-Model Constructiond8vs W3; BiTz W1We added two ablations to disentangle reasoning and generative-model construction: (1) replacing the likelihood evaluator with the training model and (2) enabling reasoning in the baseline. Results suggest that AR-BED's gains primarily stem from estimating information gain over a fixed contrastive hypothesis set rather than policy-generated hypotheses, with stronger likelihood evaluators providing marginal benefits.Rebuttal: d8vs (3.), BiTz (1.). Manuscript: Sec. 5.2
Limited Non-Myopic Gainsa6sp W3; ypha W3We clarified the motivation for the non-myopic reward and the interpretation of our experimental findings.Rebuttal: a6sp (3.), ypha (1.). Manuscript: Sec. 5.3.3
Zero-Shot Cross-Task Generalizationd8vs W4 & Q3; ypha W4 & W5; a6sp W2We added an experiment evaluating zero-shot transfer from an Animals-trained Twenty Questions policy to MediQ, demonstrating improvements over both the base model and the active-questioning baseline BED-LLM.Rebuttal: d8vs (4.), ypha (4.), a6sp (2.). Manuscript: Sec. 5.4

For further details, we refer the Area Chair to our full responses.

Thank you once again for your time and consideration.

Best regards,

The Authors of Submission #51155

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.