- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
BALAR is a training-free framework that maintains Bayesian beliefs over LLM-generated latent states, selects clarifying questions by mutual information, and dynamically expands its representation. Experiments cover detective reasoning, situation puzzles, and clinical diagnosis, with additional SFT, DPO, and RL experiments transferring questioning behavior to a smaller model.
Critical strengths
-
The framework coherently separates LLM-based representation construction from deterministic belief updating and question selection, while allowing the representation to evolve during interaction.
-
Question-selection ablations and additional analyses of prior sensitivity, calibration, and state tracking provide useful evidence beyond aggregate benchmark performance.
-
Distillation is evaluated using both a frozen model and the student for final-answer generation, helping separate information-gathering improvements from the student’s answering ability.
Critical weaknesses
-
Headline results use full-dataset configuration selection. Although BALAR-tuned adds held-out evaluation, it is unclear whether configurations are selected independently within each split or after aggregating validation results across splits, leaving potential selection bias unresolved.
-
Disabling EXPAND freezes both dimensions and questions, conflating representation refinement with question-bank renewal. Matching user-question budgets also does not establish computational efficiency. For instance, Qwen-32B uses approximately 306 calls per iCraft-MD dialogue, mostly for expansion.
-
Distillation gains over the untuned policy do not isolate BALAR-specific supervision from ordinary teacher demonstrations or reduced repetition. The base policy’s mean replication-level median is only 2.3 distinct questions over 25 turns, motivating matched teacher-data and no-repeat controls.
Questions for authors
-
Is configuration selection performed separately within each validation–test split? Please provide paired uncertainty estimates accounting for repeated evaluations of the same cases.
-
How does BALAR compare with fixed dimensions but dynamically refreshed questions? Please also report token usage and measured latency under comparable resources.
-
Do distillation gains persist against equal-sized non-Bayesian teacher demonstrations and an untuned policy with repetition prevention?
-
What observation model justifies Equation (8), and does Appendix E’s guarantee cover the implemented soft updates?
-
Do AR-SP rankings persist under a stricter correctness rubric or an independent judge?
5 comments
4. Paired uncertainty estimates. Within each of the 500 splits, we compute the paired difference between BALAR-tuned and the strongest baseline on the same test half. The bootstrap reruns the whole select-then-test procedure (1,000 resamples).
| Setting | Strongest baseline | BALAR-tuned (%) | Paired difference | Bootstrap 95% CI |
|---|---|---|---|---|
| AR-DC, Llama-8B | ToT | 56.4 | +2.4 | [−15.6, +14.5] |
| AR-DC, Qwen-32B | Few-Shot | 38.9 | −2.2 | [−11.9, +11.5] |
| AR-SP, Llama-8B | ToT | 18.4 | −12.9 | [−23.2, −1.2] |
| AR-SP, Qwen-32B | ToT | 45.8 | +6.9 | [−3.4, +17.2] |
| iCraft-MD, Llama-8B | MediQ Expert | 51.4 | +4.1 | [−5.2, +15.1] |
| iCraft-MD, Qwen-32B | MediQ Expert | 68.7 | +11.9 | [+3.1, +20.1] |
BALAR-tuned is ahead on average in four of six settings. The intervals are wide because the benchmarks are small: each test half has only 50 to 70 cases. Two intervals still exclude zero: in BALAR’s favor on iCraft-MD with Qwen-32B, and against it on AR-SP with Llama-8B, where the smaller model struggles with BALAR’s structured prompting. We added these results in Appendix L.1.
5. Distillation controls. We ran both suggested controls on the same 18 replications as Figure 3: SFT on an equal number of demonstrations from a non-Bayesian teacher (Qwen-32B Zero-Shot), and the untuned policy with repeated questions blocked.
| Policy | Frozen-Qwen readout accuracy | Student readout accuracy |
|---|---|---|
| Base | 67.5 | 52.9 |
| Base, repeats blocked | 68.3 | 53.7 |
| SFT on Zero-Shot teacher | 69.3 | 54.7 |
| SFT on BALAR teacher (ours) | 72.9 | 63.5 |
With repeats blocked, the base policy asks distinct questions yet gains only 0.8 points. BALAR supervision also beats an equal number of non-Bayesian demonstrations. We added both controls to Section 4.2 and Appendix U.
6. Observation model. Thank you for this question. Eq. (8) is exact Bayes under the model , where the free-form answer depends on the state only through the discrete answer . Then , and plays the role of up to a constant. Appendix E's guarantee is stated for the discrete answers that BALAR's MI selection uses. Under this model the soft answers remain conditionally independent given .
7. AR-SP scores. AR-SP is scored by non-strict semantic equivalence with a Qwen-32B judge (Appendix K). We re-scored all the saved AR-SP final answers of the Qwen-32B agent with the strict rubric (entailment in both directions) and with GPT-5-mini as an independent judge.
| Method | Qwen judge, non-strict | Qwen judge, strict | GPT-5-mini judge, non-strict |
|---|---|---|---|
| BALAR (ours) | 54 | 15 | 42 |
| Few-Shot | 37 | 3 | 29 |
| ToT | 39 | 9 | 31 |
| UoT | 12 | 1 | 13 |
| ProCoT | 36 | 3 | 24 |
BALAR ranks first under all three scorings. We added these results to Appendix K.
We hope these changes address your concerns, and we are happy to clarify anything further.
Thank you for the detailed response and the additional experiments. Most of my concerns have been resolved. Please explicitly state in the revised manuscript that the guarantee in Appendix E applies only to discrete answers.
I have two remaining points.
Q1. The asymptotic analysis is useful, but the requested comparison of token usage and measured latency remains missing. Sequential depth describes the dependency structure of the calls and does not establish wall-clock latency under finite resources and concurrency limits. Several baselines in the table also have linear sequential depth in the number of interaction turns.
Could you provide measurements on a representative subset using the same backbone and comparable hardware and concurrency limits, reporting input/output tokens and separating initialization from online interaction latency? If these measurements cannot be added, please qualify the practical efficiency and low-latency claims to reflect what the call-count and parallelism analyses establish.
Q2. Thank you for providing paired differences and confidence intervals, including the setting where BALAR performs worse. Could you clarify the bootstrap resampling unit and how repeated evaluations of the same case across overlapping splits are handled? In particular, do the reported intervals quantify uncertainty in the mean paired difference across the 500 splits? A short description or pseudocode showing the resampling and configuration-selection steps would help establish how the procedure preserves pairing and accounts for these dependencies.
The new intervals support a positive difference for Qwen-32B on iCraft-MD and a negative difference for Llama-8B on AR-SP, while the other four comparisons remain inconclusive. The statement that BALAR-tuned has a higher mean in four of six settings is accurate, but broader claims of consistent superiority should reflect this uncertainty and variation across settings. Please ensure that the abstract and main conclusions clearly distinguish the full-dataset-best results from the held-out results and align the strength of the claims with the latter evidence.
We thank the reviewer for the careful reading and precise questions that will substantially improve the paper.
We are pleased that the reviewer recognized how BALAR coherently separates LLM-based representation building from deterministic belief updating, valued our ablations and additional analyses (prior sensitivity, calibration, state tracking), and found the frozen-model readout in the distillation experiments informative.
The reviewer asks us to clarify configuration selection and the observation model, to add controls for EXPAND, distillation and AR-SP scoring, to report paired uncertainty estimates, and to compare computational cost. We address each of these points below.
1. Configuration selection. Yes, selection is performed separately within each split. For each of the 500 random 50/50 val/test splits, the configuration is selected on that split's validation half only and scored on its test half. The 500 test accuracies are then averaged. No validation results are aggregated across splits. We have clarified the procedure in Section 4 and Figure 2.
2. EXPAND control. We ran the suggested variant in the Table 1c setting, where EXPAND keeps the dimensions fixed and only generates new questions.
Refreshing questions recovers only 0.7 of the 9.3 points that EXPAND adds. New questions over a fixed state can only resolve the existing dimensions, so they cannot capture a factor that the initial representation misses. We added this variant to Table 1c and Section 4.5.
3. Computational cost. Our main runs logged BALAR’s LLM calls (Appendix I, Table 2) but not the baselines’ calls, token usage or latency, so we compare asymptotic call counts. For T turns with one user, the LLM calls and sequential depth of each method are:
pmax and ∣Qmax∣ are the final numbers of dimensions and questions, and E is the number of EXPAND steps (0 to 8, Appendix R). Many of BALAR's calls run concurrently, so its sequential depth stays linear in T. We added this analysis to Appendix I (Table 3).