acceptodds
Under review as a conference paper at ICLR 2027

BALAR : A Bayesian Agentic Loop for Active Reasoning

Abstract

Large language models increasingly operate in interactive settings where solving a task requires multiple rounds of information exchange with a user. However, most current systems treat dialogue reactively and lack a principled mechanism to reason about what information is missing. We propose BALAR (Bayesian Agentic Loop for Active Reasoning), a task-agnostic outer-loop algorithm that requires no fine-tuning and enables multi-turn interaction between an LLM agent and a user. BALAR maintains a structured belief over latent states, selects clarifying questions by maximizing expected mutual information, and dynamically expands its state representation when the current one proves insufficient. We evaluate BALAR on three diverse benchmarks: AR-Bench-DC (detective cases), AR-Bench-SP (thinking puzzles), and iCraft-MD (clinical diagnosis). BALAR outperforms all baselines across the three benchmarks, with 14.6% higher accuracy on AR-Bench-DC, 38.5% on AR-Bench-SP, and 30.5% on iCraft-MD. We further study whether BALAR can serve as a teacher for a questioning policy through supervised fine-tuning (SFT), direct preference optimization (DPO), and dense-reward reinforcement learning (RL). Across 18 iCraft-MD replications, distilling BALAR into a Llama-8B yields relative gains in frozen-Qwen final-answer accuracy of 8.1% with SFT, 10.9% with DPO, and 12.1% with RL over the untuned policy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Active Reasoning, Bayesian Inference, Test Time Compute

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_tFfFOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

BALAR is a training-free framework that maintains Bayesian beliefs over LLM-generated latent states, selects clarifying questions by mutual information, and dynamically expands its representation. Experiments cover detective reasoning, situation puzzles, and clinical diagnosis, with additional SFT, DPO, and RL experiments transferring questioning behavior to a smaller model.

Critical strengths
  1. The framework coherently separates LLM-based representation construction from deterministic belief updating and question selection, while allowing the representation to evolve during interaction.

  2. Question-selection ablations and additional analyses of prior sensitivity, calibration, and state tracking provide useful evidence beyond aggregate benchmark performance.

  3. Distillation is evaluated using both a frozen model and the student for final-answer generation, helping separate information-gathering improvements from the student’s answering ability.

Critical weaknesses
  1. Headline results use full-dataset configuration selection. Although BALAR-tuned adds held-out evaluation, it is unclear whether configurations are selected independently within each split or after aggregating validation results across splits, leaving potential selection bias unresolved.

  2. Disabling EXPAND freezes both dimensions and questions, conflating representation refinement with question-bank renewal. Matching user-question budgets also does not establish computational efficiency. For instance, Qwen-32B uses approximately 306 calls per iCraft-MD dialogue, mostly for expansion.

  3. Distillation gains over the untuned policy do not isolate BALAR-specific supervision from ordinary teacher demonstrations or reduced repetition. The base policy’s mean replication-level median is only 2.3 distinct questions over 25 turns, motivating matched teacher-data and no-repeat controls.

Questions for authors
  1. Is configuration selection performed separately within each validation–test split? Please provide paired uncertainty estimates accounting for repeated evaluations of the same cases.

  2. How does BALAR compare with fixed dimensions but dynamically refreshed questions? Please also report token usage and measured latency under comparable resources.

  3. Do distillation gains persist against equal-sized non-Bayesian teacher demonstrations and an untuned policy with repetition prevention?

  4. What observation model justifies Equation (8), and does Appendix E’s guarantee cover the implemented soft updates?

  5. Do AR-SP rankings persist under a stricter correctness rubric or an independent judge?

5 comments

We thank the reviewer for the careful reading and precise questions that will substantially improve the paper.

We are pleased that the reviewer recognized how BALAR coherently separates LLM-based representation building from deterministic belief updating, valued our ablations and additional analyses (prior sensitivity, calibration, state tracking), and found the frozen-model readout in the distillation experiments informative.

The reviewer asks us to clarify configuration selection and the observation model, to add controls for EXPAND, distillation and AR-SP scoring, to report paired uncertainty estimates, and to compare computational cost. We address each of these points below.

1. Configuration selection. Yes, selection is performed separately within each split. For each of the 500 random 50/50 val/test splits, the configuration is selected on that split's validation half only and scored on its test half. The 500 test accuracies are then averaged. No validation results are aggregated across splits. We have clarified the procedure in Section 4 and Figure 2.

2. EXPAND control. We ran the suggested variant in the Table 1c setting, where EXPAND keeps the dimensions fixed and only generates new questions.

EXPAND variantAccuracy
New dimensions + new questions (BALAR, ours)67.9 ± 3.9
New questions only, fixed dimensions59.3 ± 4.2
No EXPAND58.6 ± 4.2

Refreshing questions recovers only 0.7 of the 9.3 points that EXPAND adds. New questions over a fixed state can only resolve the existing dimensions, so they cannot capture a factor that the initial representation misses. We added this variant to Table 1c and Section 4.5.

3. Computational cost. Our main runs logged BALAR’s LLM calls (Appendix I, Table 2) but not the baselines’ calls, token usage or latency, so we compare asymptotic call counts. For TT turns with one user, the LLM calls and sequential depth of each method are:

MethodLLM callsSequential depth
Zero-Shot, Few-Shot, ProCoTO(T)O(T)O(T)O(T)
MediQ Expert (nn self-consistency samples)O(nT)O(nT)O(T)O(T)
ToT (beam search, breadth bb, kk candidates, depth dd)O(Tdbk)O(Tdbk)O(Td)O(Td)
UoT (exhaustive, mm candidates, depth DD)O(T(2m)D)O(T(2m)^D)O(TD)O(TD)
BALARO(T+pmax⁡∣Qmax⁡∣)O(T + p_{\max}\lvert Q_{\max}\rvert)O(T+E)O(T+E)

pmax⁡p_{\max} and ∣Qmax⁡∣\lvert Q_{\max}\rvert are the final numbers of dimensions and questions, and EE is the number of EXPAND steps (0 to 8, Appendix R). Many of BALAR's calls run concurrently, so its sequential depth stays linear in TT. We added this analysis to Appendix I (Table 3).

4. Paired uncertainty estimates. Within each of the 500 splits, we compute the paired difference between BALAR-tuned and the strongest baseline on the same test half. The bootstrap reruns the whole select-then-test procedure (1,000 resamples).

SettingStrongest baselineBALAR-tuned (%)Paired differenceBootstrap 95% CI
AR-DC, Llama-8BToT56.4+2.4[−15.6, +14.5]
AR-DC, Qwen-32BFew-Shot38.9−2.2[−11.9, +11.5]
AR-SP, Llama-8BToT18.4−12.9[−23.2, −1.2]
AR-SP, Qwen-32BToT45.8+6.9[−3.4, +17.2]
iCraft-MD, Llama-8BMediQ Expert51.4+4.1[−5.2, +15.1]
iCraft-MD, Qwen-32BMediQ Expert68.7+11.9[+3.1, +20.1]

BALAR-tuned is ahead on average in four of six settings. The intervals are wide because the benchmarks are small: each test half has only 50 to 70 cases. Two intervals still exclude zero: in BALAR’s favor on iCraft-MD with Qwen-32B, and against it on AR-SP with Llama-8B, where the smaller model struggles with BALAR’s structured prompting. We added these results in Appendix L.1.

5. Distillation controls. We ran both suggested controls on the same 18 replications as Figure 3: SFT on an equal number of demonstrations from a non-Bayesian teacher (Qwen-32B Zero-Shot), and the untuned policy with repeated questions blocked.

PolicyFrozen-Qwen readout accuracyStudent readout accuracy
Base67.552.9
Base, repeats blocked68.353.7
SFT on Zero-Shot teacher69.354.7
SFT on BALAR teacher (ours)72.963.5

With repeats blocked, the base policy asks distinct questions yet gains only 0.8 points. BALAR supervision also beats an equal number of non-Bayesian demonstrations. We added both controls to Section 4.2 and Appendix U.

6. Observation model. Thank you for this question. Eq. (8) is exact Bayes under the model θ→Y→r\theta \to Y \to r, where the free-form answer rr depends on the state only through the discrete answer YY. Then p(r∣θ)=∑yp(r∣y)K(y∣θ)p(r\mid\theta)=\sum_y p(r\mid y)K(y\mid\theta), and ω^y\hat\omega_y plays the role of p(r∣y)p(r\mid y) up to a constant. Appendix E's guarantee is stated for the discrete answers that BALAR's MI selection uses. Under this model the soft answers remain conditionally independent given θ\theta.

7. AR-SP scores. AR-SP is scored by non-strict semantic equivalence with a Qwen-32B judge (Appendix K). We re-scored all the saved AR-SP final answers of the Qwen-32B agent with the strict rubric (entailment in both directions) and with GPT-5-mini as an independent judge.

MethodQwen judge, non-strictQwen judge, strictGPT-5-mini judge, non-strict
BALAR (ours)541542
Few-Shot37329
ToT39931
UoT12113
ProCoT36324

BALAR ranks first under all three scorings. We added these results to Appendix K.

We hope these changes address your concerns, and we are happy to clarify anything further.

Thank you for the detailed response and the additional experiments. Most of my concerns have been resolved. Please explicitly state in the revised manuscript that the guarantee in Appendix E applies only to discrete answers.

I have two remaining points.

Q1. The asymptotic analysis is useful, but the requested comparison of token usage and measured latency remains missing. Sequential depth describes the dependency structure of the calls and does not establish wall-clock latency under finite resources and concurrency limits. Several baselines in the table also have linear sequential depth in the number of interaction turns.

Could you provide measurements on a representative subset using the same backbone and comparable hardware and concurrency limits, reporting input/output tokens and separating initialization from online interaction latency? If these measurements cannot be added, please qualify the practical efficiency and low-latency claims to reflect what the call-count and parallelism analyses establish.

Q2. Thank you for providing paired differences and confidence intervals, including the setting where BALAR performs worse. Could you clarify the bootstrap resampling unit and how repeated evaluations of the same case across overlapping splits are handled? In particular, do the reported intervals quantify uncertainty in the mean paired difference across the 500 splits? A short description or pseudocode showing the resampling and configuration-selection steps would help establish how the procedure preserves pairing and accounts for these dependencies.

The new intervals support a positive difference for Qwen-32B on iCraft-MD and a negative difference for Llama-8B on AR-SP, while the other four comparisons remain inconclusive. The statement that BALAR-tuned has a higher mean in four of six settings is accurate, but broader claims of consistent superiority should reflect this uncertainty and variation across settings. Please ensure that the abstract and main conclusions clearly distinguish the full-dataset-best results from the held-out results and align the strength of the claims with the latter evidence.

Reviewer_ZrQqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

BALAR is a training-free framework for resolving ambiguous requests through multi-turn interaction. An LLM proposes discrete latent dimensions, priors, candidate questions, and likelihood tables; a deterministic layer tracks the joint belief, selects questions by mutual information, and updates the posterior from user responses. The representation can be expanded with new dimensions and questions when it proves insufficient. Experiments cover detective cases, situation puzzles, and clinical diagnosis, and the questioning behavior is distilled into Llama-3.1-8B with SFT, DPO, and RL. The results are encouraging, but their interpretation depends on clarifying the interaction protocol and accounting for computational cost.

Critical strengths
  1. BALAR tackles a practical problem: an interactive LLM needs to work out what information is missing before it can answer usefully. The interesting part is the ability to revise the state representation. Rather than selecting questions only within a fixed question bank, the method can introduce a dimension that was missing from its initial formulation.
  2. The component-level results are useful. In Table 1c, MI selection reaches 67.9% accuracy on iCraft-MD against 52.1% for random selection and 58.6% without expansion. I also appreciate the frozen Qwen answerer in the distillation experiments, which separates improvements in questioning from improvements in the student's final answers.
Critical weaknesses
  1. The stated interaction protocol does not match the supplied code. Section 3.1 says users respond in free form without seeing the discrete answer set, and Section 3.4 uses this to motivate soft response mapping. Yet in the supplementary balar_colm/agent.py, the user simulator is given BALAR's candidate choices and must return one of them by ID. Appendix V includes both choice-based and free-form simulator prompts, but it is unclear which produced the reported results. If the choice-based path was used, responses are restricted to BALAR's own answer vocabulary, removing much of the ambiguity the soft update is meant to handle. The baselines would then need to be run under the same protocol before the comparison is meaningful.
  2. Matching the maximum number of user questions does not establish that the methods use comparable computational budgets. Table 2 reports 270–360 LLM calls per dialogue for Qwen-32B, with expansion often the dominant term, and there is no baseline cost or matched-budget comparison to weigh this against; the low-latency claim is also not backed by measured latencies. The expansion ablation is additionally confounded: turning expansion off removes both new dimensions and new questions, so the value of changing the representation is not separated from simply asking more questions.
  3. The selection procedure for BALAR-tuned is ambiguous. If the configuration was chosen once using validation errors averaged over all 500 splits, validation cases were reused during selection, and the four-of-six advantage under held-out selection looks weaker than the headline gains.
  4. The stopping criterion based on the belief state is not validated where it matters. Appendix O calibrates the final answer distribution, not the likelihoods used to rank questions, and the answer selected by the belief state is much less accurate than the final LLM call (62.9% vs 73.6% for Qwen-32B on iCraft-MD). Without the error rate on cases stopped by the confidence criterion, it is unclear whether early stopping is actually reliable.
Questions for authors
  1. Were Figure 2 and Table 1 generated with free-form responses or with selection from BALAR's candidate choices? If the latter, what happens when the simulator receives only the question and its private context, under the same protocol for all methods?
  2. How does BALAR compare against the strongest baselines at matched token or compute budgets? Can generating additional questions without adding dimensions recover most of the gain attributed to EXPAND?
  3. For BALAR-tuned, was the configuration selected per split or once over all 500 splits? Please compare all methods under the same held-out selection rule with paired uncertainty estimates.
  4. Does the final answerer still improve when it receives the dialogue history but no MAP-state summary? This would show whether the structured belief adds value beyond the collected conversation.
4 comments

We thank the reviewer for the thoughtful and detailed feedback that we believe will greatly strengthen our revision.

We are happy that the reviewer considers the problem practical, highlighted BALAR's ability to revise its state representation during the dialogue, and found the component-level results and the frozen-Qwen answerer useful.

Below, we answer the reviewer's questions on the user-simulator protocol, configuration selection, BALAR's cost relative to the baselines, the separation of new dimensions from new questions in EXPAND, the paired uncertainty estimates, and the validity of the stopping criterion.

1. Simulator protocol. Thank you for checking the code closely; we should have documented this more clearly. All reported results on AR-DC and iCraft-MD use the free-form simulator. It receives only the question and the user's private context and answers in natural language, and the soft map then turns this answer into a distribution over BALAR's choices. The choice-based path in our code and in Appendix V is used only for AR-SP, where the benchmark restricts the narrator to Yes/No/Unknown for every method. The baselines follow the same protocol: their simulators answer in natural language on AR-DC and iCraft-MD, and with Yes/No/Unknown on AR-SP. All methods therefore share the same protocol on each dataset. We stated this explicitly in Appendix H.

2. Computational cost. We agree that matching the question budget does not establish a matched compute budget. With Qwen-32B, BALAR uses 270 to 362 calls per dialogue, with expansion often dominant (Appendix I, Table 2). Our main runs logged BALAR’s LLM calls (Appendix I, Table 2) but not the baselines’ calls, token usage or latency, so we compare asymptotic call counts. For TT turns with one user, the LLM calls and sequential depth of each method are:

MethodLLM callsSequential depth
Zero-Shot, Few-Shot, ProCoTO(T)O(T)O(T)O(T)
MediQ Expert (nn self-consistency samples)O(nT)O(nT)O(T)O(T)
ToT (beam search, breadth bb, kk candidates, depth dd)O(Tdbk)O(Tdbk)O(Td)O(Td)
UoT (exhaustive, mm candidates, depth DD)O(T(2m)D)O(T(2m)^D)O(TD)O(TD)
BALARO(T+pmax⁡∣Qmax⁡∣)O(T + p_{\max}\lvert Q_{\max}\rvert)O(T+E)O(T+E)

pmax⁡p_{\max} and ∣Qmax⁡∣\lvert Q_{\max}\rvert are the final numbers of dimensions and questions, and EE is the number of EXPAND steps (0 to 8, Appendix R). Many of BALAR's calls run concurrently, so its sequential depth stays linear in TT. We added this analysis to Appendix I (Table 3).

3. EXPAND control. We ran the suggested variant in the Table 1c setting, where EXPAND keeps the dimensions fixed and only generates new questions.

EXPAND variantAccuracy
New dimensions + new questions (BALAR, ours)67.9 ± 3.9
New questions only, fixed dimensions59.3 ± 4.2
No EXPAND58.6 ± 4.2

Refreshing questions recovers only 0.7 of the 9.3 points that EXPAND adds. New questions over a fixed state can only resolve the existing dimensions, so they cannot capture a factor that the initial representation misses. We added this variant to Table 1c and Section 4.5.

4. Configuration selection. Yes, selection is performed separately within each split. For each of the 500 random 50/50 val/test splits, the configuration is selected on that split's validation half only and scored on its test half. The 500 test accuracies are then averaged. No validation results are aggregated across splits. We have clarified the procedure in Section 4 and Figure 2.

5. Paired uncertainty estimates. Within each of the 500 splits, we compute the paired difference between BALAR-tuned and the strongest baseline on the same test half. The bootstrap reruns the whole select-then-test procedure (1,000 resamples).

SettingStrongest baselineBALAR-tuned (%)Paired differenceBootstrap 95% CI
AR-DC, Llama-8BToT56.4+2.4[−15.6, +14.5]
AR-DC, Qwen-32BFew-Shot38.9−2.2[−11.9, +11.5]
AR-SP, Llama-8BToT18.4−12.9[−23.2, −1.2]
AR-SP, Qwen-32BToT45.8+6.9[−3.4, +17.2]
iCraft-MD, Llama-8BMediQ Expert51.4+4.1[−5.2, +15.1]
iCraft-MD, Qwen-32BMediQ Expert68.7+11.9[+3.1, +20.1]

BALAR-tuned is ahead on average in four of six settings. The intervals are wide because the benchmarks are small: each test half has only 50 to 70 cases. Two intervals still exclude zero: in BALAR’s favor on iCraft-MD with Qwen-32B, and against it on AR-SP with Llama-8B, where the smaller model struggles with BALAR’s structured prompting. We added these results in Appendix L.1.

6. Stopping criterion and the structured belief This is a fair concern. Appendix O calibrates the predictive answer belief (Eq. 13). The emitted answer comes from a final LLM call that reads the MAP state and the dialogue history. This call returns a single answer rather than a distribution, and it is unclear how to calibrate it. In the Table 1c setting, the 24 dialogues that stopped before the 10-question budget reach 66.7% accuracy, close to the 68.1% of the 116 that used the full budget. We added this result to Appendix O. The belief decides when to stop; the answer is emitted by the final LLM call, which is more accurate than the belief’s argmax (73.6% vs 62.9%, Appendix O). To measure what the structured belief adds, we ran this final call in the Table 1c setting, with the conversation history but without the MAP state.

VariantAccuracy (%)
MAP state + conversation history (BALAR, ours)67.9 ± 3.9
Conversation history only62.1 ± 4.1

The MAP state improves accuracy by 5.8 points. We added this result to Table 1c and Section 4.5.

We hope these changes address your concerns, and we are happy to clarify anything further.

Thanks for the clarifications and new experiments. Although an empirical cost comparison with the baselines is still missing, the complexity analysis helps address my concern from a different angle. The results suggest that claims of consistent gains across datasets should be made more cautiously. Overall, the response makes me more positive about the paper, and I am now leaning toward weak acceptance.

We sincerely thank the reviewer for the thoughtful follow-up. We are glad the clarifications and new experiments were helpful, and we have taken both remaining points into account in the revision.

Empirical cost comparison. We measured token usage and latency on 10 randomly drawn iCraft-MD instances, as running the full dataset was not feasible within the discussion period. All methods used the same backbone (Qwen2.5-32B-Instruct), the same server (one A100-80GB with vLLM), the same concurrency limit (64 requests) and a budget of 10 questions.

Mean over 10 casesZero-ShotMediQ ExpertBALAR
Questions asked10105.8
Agent input tokens8.7k41.2k82.4k
Agent output tokens0.5k3.5k31.5k
Initialization (s)012211
Online, per question (s)3.612.45.4
Total per dialogue (s)38142299

BALAR's total time per dialogue is about twice that of MediQ Expert and eight times that of Zero-Shot, and it uses about twice the input tokens and nine times the output tokens of MediQ Expert. Most of this cost comes from initialization, which runs before the first question, and from EXPAND, which occurred in 2 of the 10 dialogues and added about 270 s each. Excluding EXPAND, BALAR's time per question (5.4 s) lies between the two baselines. Because the sample is small, we treat these numbers as indicative. We added these measurements to Appendix I and revised the efficiency claims accordingly (Sections 1 and 5, Appendices C and I): BALAR makes more LLM calls than prompting baselines (84 to 362 per dialogue), most of them independent, and compute cost is listed as a limitation.

More cautious claims. We agree and have aligned the claims with the held-out evidence. The full-dataset results, which follow the benchmarks' protocol, now appear only in Section 4.1, labeled as such, and the contributions and the Conclusion state the held-out result in the same terms as the abstract, which now reads: "Under held-out configuration selection, BALAR yields a clear gain on iCraft-MD with Qwen2.5-32B-Instruct and trails on AR-Bench-SP with Llama-3.1-8B-Instruct. The other four comparisons are statistically inconclusive at the size of these benchmarks, although three of them favor BALAR."

Thank you again for your engagement throughout the discussion, which has made the paper's claims more precise and its evaluation more complete.

Reviewer_tDFQOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper looks at interactive dialogue settings where LLMs must ask questions in order to seek information, such as clinical diagnosis or twenty-questions-style environments. The authors introduce BALAR, an inference-time procedure that (1) prompts an LLM to construct a structured Bayesian belief model over a feature space, (2) pre-samples questions and answer-conditional likelihoods, and (3) performs online belief updates. Experiments on 3 representative active learning benchmarks show that BALAR generally results in improvements over a reasonable number of standard baselines (few-shot, ToT, UoT, CoT) and is distillable into the model via SFT/DPO/RL.

Critical strengths
  • BALAR can handle tasks with both discrete and open-ended label sets, whereas many related approaches only handle the discrete case.
  • The choice of benchmarks (detective cases and thinking puzzles from AR-Bench as well as medical diagnosis from iCraft-MD) is reasonably diverse and canonical.
  • The results show that BALAR generally results in improvements over a reasonable number of standard baselines (few-shot, ToT, UoT, CoT) and is distillable into the model via SFT/DPO/RL. The distillation experiments are particularly important for making the case that this approach is applicable to real-world LLM training.
  • The framework is a nice generalization of existing frameworks that combine LLMs with Bayesian information-seeking; however, the framework is less novel than the authors’ framing would suggest, and the paper misses citations to a number of key related works (see below).
  • The paper is generally well-written, readable, and free of obvious AI slop.
Critical weaknesses
  • The intro argues that existing approaches (tree-of-thought, uncertainty-of-thought, CollabLLM, etc.) are computationally costly — however, the proposed method (BALAR) also introduces a number of inference steps (e.g., estimating full likelihood tables) that are also costly. The authors do include a detailed cost analysis in Appendix I, but this lacks a formal comparison to other baselines.
  • The first phase of BALAR is an initialization phase whose naming references wake/sleep algorithms (Hinton, 1995) recently re-popularized for the LLM audience (e.g., the authors cite the recent “sleep-time compute” work from Lin et al., 2025). As someone who previously reviewed that Letta paper and is highly familiar with the relevant literature, I feel that the naming is a bit of a stretch: a central aspect of wake/sleep methods is that they are iterated cyclically, so that computation from the inference (wake) phase is amortized in the next sleep phase. Both here, and in Lin et al., 2025, sleep only occurs once upfront. It would actually be really interesting to consider a version of BALAR where the Bayesian model is re-derived from scratch depending on the conversation; however, in the current implementation “sleep-time” is really just a buzzy word for preprocessing / initialization.
  • The first step of the initialization (3.1) involves an LLM proposing “disambiguating dimensions” for use in a Bayesian model. This approach is very similar to the Bayesian Preference Elicitation model of Handa et al., 2024 (https://arxiv.org/pdf/2403.05534). That would should be included in the references and cited appropriately.
  • BALAR assumes a finite label set L\mathcal{L} for prior elicitation, likelihood construction, and inference. In the real world, the label set may not be known in advance, or it may not be discrete at all. This should be acknowledged in the limitations and it would be useful to see discussion of algorithmic generalizations that overcome this limitation.
  • Various aspects of BALAR are algorithmically greedy/myopic: e.g., (1) picking the argmax question at every turn foregoes the opportunity to plan out a sequence of jointly information-maximizing questions; (2) adding new dimensions (EXPAND) only when the algorithm is stuck sacrifices possible inform from having had these dimensions from the start, and may also lead to the addition of redundant dimensions. It would be nice to see both theoretical and empirical characterization of the complexity-vs-performance tradeoffs of the greedy strategies here.
  • The evaluations use relatively old/weak LLMs (Llama-3.1-8B-Instruct and Qwen2.5-32B-Instruct). It’s unclear if the performance advantage for BALAR would hold for newer open models like the Qwen3, GPT-OSS, or Gemma 4.
  • The authors missed citations to a surprising number of key related works on LLMs + Bayesian active learning. Some of these (particularly Handa et. al., 2024; which I have no relation to) are substantially similar to the proposed method.
Questions for authors
  1. “In Figure 2, results for BALAR are for its best configuration of pα,β,p,|Q|q, matching the procedure done in the compared-to published benchmark results.”
    • Were these hyperparameters chosen on a held-out validation split, or are you literally reporting the best results based on performance on the test set?
  2. The teacher-student setup in 4.2 was not immediately clear from reading the main paper - looking at Fig. 3a, it’s confusing that “Frozen Qwen” is being trained with SFT, DPO, RL. Can you bring some of the methods details from the appendix back into 4.2 and make Fig. 3 more self-explanatory?
  3. Can you please review the above additional references and update your related work accordingly? In particular, can you provide some statements on how your approach differs from Bayesian Preference Elicitation?
3 comments

We thank the reviewer for the rigorous, detailed critique that will make the revised paper clearer and more complete.

We were happy that they found the paper well-written and readable, noted that BALAR handles both discrete and open-ended label sets, considered our benchmarks diverse and canonical, and viewed the distillation experiments as important for applying BALAR to real-world LLM training.

The main questions concern BALAR's cost relative to the baselines, its position within prior work on Bayesian active learning, its limitations, and the details of configuration selection and the distillation setup. We address each of these points below.

1. Computational cost. We acknowledge that BALAR requires many LLM calls (Appendix I). For TT turns with one user, the LLM calls and sequential depth of each method are:

MethodLLM callsSequential depth
Zero-Shot, Few-Shot, ProCoTO(T)O(T)O(T)O(T)
MediQ Expert (nn self-consistency samples)O(nT)O(nT)O(T)O(T)
ToT (beam search, breadth bb, kk candidates, depth dd)O(Tdbk)O(Tdbk)O(Td)O(Td)
UoT (exhaustive, mm candidates, depth DD)O(T(2m)D)O(T(2m)^D)O(TD)O(TD)
BALARO(T+pmax⁡∣Qmax⁡∣)O(T + p_{\max}\lvert Q_{\max}\rvert)O(T+E)O(T+E)

pmax⁡p_{\max} and ∣Qmax⁡∣\lvert Q_{\max}\rvert are the final numbers of dimensions and questions, and EE is the number of EXPAND steps (0 to 8, Appendix R). Many of BALAR's calls run concurrently, so its sequential depth stays linear in TT. We have added this analysis to Appendix I (Table 3).

2. "Sleep-time" naming. We agree the analogy to wake–sleep is only partial: BALAR’s construction runs once before interaction rather than alternating with inference. EXPAND re-runs parts of it mid-dialogue (Section 3.5), so the two phases partly alternate. We now say this in Sections 3 and 3.5, and added to Section 5 (Future work) periodically rebuilding the whole model during the dialogue, so that the two phases fully alternate.

3. Related work. We have added citations and a discussion of all six papers.

We agree that BALAR builds on this line of work: using an LLM to construct a Bayesian model and selecting queries by expected information gain (EIG) are shared with OPEN and BED-LLM. BALAR’s distinct contributions are an instance-specific factored state with LLM-elicited likelihood tables over free-form questions, EXPAND, which grows this state during the dialogue, and its use as a teacher for distilled questioning policies. We have added all six papers to Appendix A and positioned the closest in the Introduction.

OPEN (Handa et al.) uses an LLM to propose the features of a Bayesian model, sets the prior from LLM judgments, and selects queries by EIG. However, OPEN infers continuous preference weights over domain-specific features from pairwise comparisons under a Bradley–Terry model. BALAR tracks a joint state over instance-specific dimensions, asks LLM-generated free-form questions rather than pairwise comparisons, adds dimensions through EXPAND, and can query several users. BED-LLM (Choudhury et al.) updates a uniformly weighted set of LLM-generated hypotheses, whereas BALAR factorizes the state and reuses its likelihood tables. Grand et al. sample game states from known rules, while BALAR's latent space is built by the LLM. Li et al. keep no explicit belief, and Collins et al. perform inference from supplied clinical information rather than actively eliciting missing evidence. Our DPO pairs follow Mazzaccara et al., with MI computed over BALAR's latent state instead of a known candidate set.

4. Finite label set. We should have made this clearer, and have reworded Section 3.1 to describe L={likely,neutral,unlikely}\mathcal{L}=\{\texttt{likely},\texttt{neutral},\texttt{unlikely}\} as an elicitation scale fixed across tasks (Appendix J), not the task’s label set. The task answers need not be finite (AR-SP answers are free-form paragraphs). One limitation of BALAR is that each latent dimension takes finitely many values, which we now state in Section 5 (Limitations) along with generalizations to continuous dimensions.

5. Greedy question selection and expansion. We agree that both steps are greedy. Under an assumption of conditional independence of answers given the state, greedy MI selection recovers at least (1−1/e)(1-1/e) of the information gain of the best fixed set of kk questions (Appendix E). kk expansions with an η\eta-approximate proposal mechanism recover at least (1−e−η)(1-e^{-\eta}) of the best kk-dimension refinement (Appendix F). A redundant dimension has zero value, and the EXPAND prompt asks for one not already in the state. We discuss non-myopic planning as an extension in Section 5 (Future work).

6. Newer models. We agree this is a limitation; re-running all methods with newer models was not feasible during the rebuttal period. Within the Qwen2.5 family, BALAR does improve with model size: iCraft-MD accuracy rises from 50.7% (Qwen2.5-7B) to 62.1% (14B) and 73.6% (32B) (Table 1b).

7. Configuration selection in Figure 2. The reviewer is right that Figure 2’s BALAR bars use the best configuration on the full data; this follows the protocol of AR-Bench and MediQ, which report best configurations on the full data with no validation set. To remove this bias, Figure 2 also reports BALAR-tuned: in each of 500 random 50/50 val/test splits, the configuration is selected on the validation half and scored on the test half. The 500 test accuracies are averaged. BALAR-tuned stays ahead of the strongest baseline in four of six settings, even though the baselines keep their full-dataset selection advantage. Applying the same held-out rule to the baselines would, in expectation, lower their scores, since a configuration chosen on the full data is optimistically biased. We have clarified the procedure in Section 4 and in the caption of Figure 2.

8. Teacher–student setup. We have moved these details into Section 4.2 and Figure 3. Only the student policy (Llama-3.1-8B, LoRA) is trained, with BALAR run on Qwen2.5-32B as teacher. SFT and DPO learn from BALAR's states, questions and MI scores. DPO and RL start from the SFT model, and RL uses a frozen-Qwen reward (Appendix U). "Frozen Qwen" in Figure 3a is a readout: an untrained Qwen reads the student's 25-question dialogue and gives the final answer, isolating the effect of the questions. "Student" uses the trained Llama instead.

We hope these changes address your concerns, and we are happy to clarify anything further.

Thank you to the authors for the detailed rebuttal. The majority of the weaknesses I raised above were primarily informational, and the authors' comments do a nice job addressing each of these. I appreciate the inclusion of the additional references. Regarding new models, while this would be nice to have, I appreciate that it may not be feasible during rebuttal and would not count this as a major obstacle to acceptance.

I appreciate the clear communication and notable absence of obvious AI slop in the authors' submission and rebuttal. Provided the final manuscript reflects the changes discussed above, I am happy to recommend acceptance.

Comments on the paper

1 comment

We thank the reviewers for their careful reviews and their engagement during the discussion. To help assess the revised submission, we briefly summarize the main points addressed. We are also grateful for the many other helpful questions, which we discuss in the individual responses. Revisions are marked in blue in the PDF.

1. Configuration selection and paired uncertainty estimates (tDFQ, tFfF, ZrQq). For each of the 500 random 50/50 val/test splits, the configuration of BALAR-tuned is selected on that split's validation half only and scored on its test half (Section 4 and Figure 2). Within each split, we compute the paired difference between BALAR-tuned and the strongest baseline on the same test half. The bootstrap reruns the whole select-then-test procedure with the case (benchmark instance) as the resampling unit (Appendix L.1). As tFfF and ZrQq requested, the abstract, contributions and Conclusion now state the held-out result, and the abstract reads: "Under held-out configuration selection, BALAR yields a clear gain on iCraft-MD with Qwen2.5-32B-Instruct and trails on AR-Bench-SP with Llama-3.1-8B-Instruct. The other four comparisons are statistically inconclusive at the size of these benchmarks, although three of them favor BALAR."

2. Computational cost (tDFQ, tFfF, ZrQq). We added the asymptotic LLM calls and sequential depth of every method (Appendix I, Table 3). We also measured token usage and latency on 10 randomly drawn iCraft-MD instances, with the same backbone, server, concurrency limit and question budget for all methods (Appendix I). BALAR's total time per dialogue is about twice that of MediQ Expert. Most of this cost comes from initialization, which runs before the first question, and from EXPAND. We revised the efficiency claims accordingly, and compute cost is listed as a limitation.

3. New controls (tFfF, ZrQq). Refreshing questions while keeping the dimensions fixed recovers only 0.7 of the 9.3 points that EXPAND adds. Including the MAP state in the final LLM call, alongside the conversation history, improves accuracy by 5.8 points (Table 1c and Section 4.5). In distillation, BALAR supervision beats an equal number of non-Bayesian demonstrations (Section 4.2 and Appendix U). On AR-SP, BALAR ranks first under all three scorings, including the strict rubric and GPT-5-mini as an independent judge (Appendix K).

4. Clarifications. We added all six suggested papers to Appendix A and positioned the closest in the Introduction (tDFQ). We now state explicitly that all methods share the same simulator protocol on each dataset (ZrQq, Appendix H). We also state that the guarantee in Appendix E applies only to the discrete answers used for question selection (tFfF, Section 3.3).

We thank the reviewers again for their thoughtful feedback throughout the discussion, which has substantially strengthened the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.