acceptodds
Under review as a conference paper at ICLR 2027

Learning What to Ask and What to Believe: Distilling a Variational Bayesian Agent into Language Models

Abstract

Interactive tasks such as eliciting a user's preferences or diagnosing a hidden mechanism require an agent to track its uncertainty over a latent state, ask informative questions and act on the answers. Language models have no explicit mechanism for this, whereas Bayesian information seeking provides one through posterior inference and expected information gain (EIG), and we ask whether it can be distilled into an 8B language model. Our teacher, a lightweight Bayesian agent with a sequential variational belief and EIG query selection, matches or approaches exact and particle references on three task families. We generate synthetic interactions with the teacher on episodes disjoint from the benchmark and fine-tune Llama-3.1-8B-Instruct to reproduce its queries and its belief state as a structured scratchpad. On three Multi-Turn Puzzles tasks the student raises Circuit Decoding from 0.00 to 0.93 success, lowers the normalized rank of the recommended film from 0.50 to 0.10 and cuts word guessing from 32 to 3.4 attempts, exceeding the strongest reported language models on the two comparable tasks. Exposing the belief state closes much of the gap where the decision is an implicit computation rather than a read-out of the interaction; in matched teacher contrasts, EIG supervision consistently outperforms random querying, whereas greater inference fidelity does not uniformly improve the distilled student; a 70B language-model teacher does not match the Bayesian teacher in downstream student performance, and the learned skill does not transfer to an unseen task family. Variational Bayesian information seeking can thus be amortized into a language model by transferring the belief state that drives its questions and decisions.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Large Language Models, Bayesian Information Seeking, Variational Inference, Knowledge Distillation, Interactive Agents

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aLbeOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper looks at interactive dialogue settings where LLMs must ask questions in order to seek information. The authors instantiate a Bayesian teacher model with explicit belief tracking and EIG question-selection and use it to distill a student model (Llama-3.1-8B). Experiments on 3 toy puzzle tasks suggest distillation closes most of the gap between student and teacher.

Critical strengths
  • The general question-asking / information-seeking setting is an interesting research problem.
  • As someone very familiar with this research area, the literature review appears reasonably complete.
  • The ablations on teacher components are interesting.
Critical weaknesses
  • The method is not methodologically novel relative to existing LLM + information-seeking work. All aspects of the Bayesian teacher (belief tracking, EIG question-selection) have been studied in prior work, while student distillation is also common both in information-seeking and more general LLM settings. The contributions are not clear, which is compounded by writing issues described below.
  • The empirical evaluation is restricted to 3 toy tasks from a single benchmark and one LLM (Llama-3.1-8B-Instruct). Furthermore, the Multi-turn Puzzles benchmark (Badola et al., 2025) does not appear to have been accepted to any venue besides arxiv and to my knowledge has not been widely adopted. Given the relatively thin methodological contribution, I would expect a broader set of empirical contributions on more standard info-seeking benchmarks for comparability to the cited methods.
  • Relatedly, while the related work demonstrates clear knowledge of other methods in the space, none of these are included as baselines.
  • Overall, the paper is unnecessarily hard to follow; the authors appear to rely heavily on AI-generated prose and figures, which sacrifices human comprehensibility.
    • Prose is overly dense and difficult to follow. Many passages bear hallmarks of AI-generated writing; e.g., here is a single sentence from the abstract:

“Exposing the belief state closes much of the gap where the decision is an implicit computation rather than a read-out of the interaction; in matched teacher contrasts, EIG supervision consistently outperforms random querying, whereas greater inference fidelity does not uniformly improve the distilled student; a 70B language-model teacher does not match the Bayesian teacher in downstream student performance, and the learned skill does not transfer to an unseen task family.” - Basic methodological details are not clearly described; e.g., “The teacher keeps a belief qtq_t in a fixed variational family Q\mathbb{Q} and updates it after every observation” (L114) — how is qtq_t represented? how is the variational family chosen? - The paper contains only two figures, both of which are not very illustrative: - Fig 1: A tikz-style schematic. Many aspects of the figure appear AI-generated; e.g., flavor text. Overall, the figure does not clearly convey the pipeline. - Fig 2: Results bar graph. Figure lacks error bars and in the left-hand side the differences between bar heights is not apparent.

Questions for authors
  1. How did you choose the variational family Q\mathbb{Q} -- it appears to be chosen specifically for each domain?
  2. What do you believe are the main contributions of this method relative to existing works?
  3. Are there any other figures you could include to illustrate details of the method or experimental details (e.g., scaling, ablations)? The current set of figures is very minimal, which contributes to the overly wordy nature of the manuscript.
  4. Would it be possible for you to rewrite the paper without using generative AI? This is a sincere question; I believe it would significantly improve comprehensibility.
6 comments

We thank the reviewer for the detailed review. It raises four weaknesses and four questions, and we answer them as follows.

  • W1 / Q2 (the method is not novel and the contributions are unclear): we state the three contributions.
  • W2 (three toy tasks from one little-adopted benchmark, and one LLM): new results on two further backbones and a second seed, and the reason for the choice of benchmark.
  • W3 (no baselines from related work): we map our controlled arms to the families of related methods.
  • W4b / Q1 (how the variational family is chosen): the rule behind the choice.
  • W4a / Q4 (prose and use of generative AI) and W4c / Q3 (figures): we will address in the revised PDF.

W1 / Q2. Novelty and main contributions

We agree that variational belief updating, EIG query selection and distillation are each established, and we claim none of them as new. The contribution is what their combination lets us measure. We make three claims.

1. What has to be distilled, and when. Cloning the teacher's actions transfers its query policy but not the computation that turns the interaction into a decision. When the decision can be read off the history (Circuit Decoding), action-only distillation is enough for a strong student: 0.920 vs. 0.927 Success@18 with the scratchpad (Table 3). When it must be computed from a belief (Movie Recommendation, Word Guess), action-only students gather the information and fail to use it: 0.085 vs. 0.345 Success@10 and 19.2 vs. 3.4 attempts. Figure 2a locates the failure: the teacher's questions leave the action-only movie student at 0.085, while a Bayes-optimal read-out of its own transcript lifts it to 0.305. The pattern holds on two further backbones and a second seed (W2).

2. Which part of the teacher matters. The teacher's components can be swapped with everything else held fixed (Table 4). Replacing random querying by EIG changes the student by +0.42 / +0.17 / +0.21 (circuit / movie / word). Replacing variational by exact or particle inference changes it by −0.28 / +0.06 / 0.00. A new experiment shows that most of the −0.28 comes from random tie-breaking in the exact teacher and not from inference fidelity: with deterministic ties the difference is −0.05 [−0.09, +0.00] (our reply to Reviewer ve4v). A 70B language-model teacher yields students with almost no success on Circuit Decoding (−0.92). These attributions require a teacher whose components are explicit.

3. A verifiable teacher as the source of supervision. Training-based methods (Andukuri et al. 2024, Zhang et al. 2025, Chen et al. 2025, Mazzaccara et al. 2024) take their targets from the model itself or from heuristics. Inference-time methods (BED-LLM, UoT) keep the Bayesian loop at test time. Our targets come from an agent validated against exact and particle references (Table 6) and are amortized into the weights.

Closest prior work. Qiu et al. (2025) fine-tune a language model on a Bayesian assistant's recommendations. Their appendix shows the same read-out gap from the other side: a model trained on the assistant's preferences supports 79% accuracy under an external read-out but recommends at 55%. We differ in three ways: our student also learns which questions to ask, the belief target is a posterior summary with per-candidate scores rather than a point estimate, and we identify when the belief has to be written.

W2. Three toy tasks from one benchmark and one LLM (models)

We ran the paper's recipe on Qwen3-8B and Llama-3.2-3B-Instruct and trained a second seed of Llama-3.1-8B-Instruct, for the ELBO+EIG teacher in both formats (A = action-only, B = belief scratchpad). Only the backbone and its chat template changed. We used test subsets fixed before any run: every third Circuit Decoding episode (100), the first 200 Movie Recommendation episodes and the first 100 Word Guess episodes. All comparisons are paired on the same rows.

Task, metricBackboneBaseABB − A [95% CI]
Circuit, Success@18Llama-3.1-8B (paper)0.000.920.92+0.00 [−0.07, +0.07]
Qwen3-8B0.000.910.84−0.07 [−0.16, +0.02]
Llama-3.2-3B0.000.810.91+0.10 [+0.01, +0.19]
Movie, Success@10Llama-3.1-8B (paper)0.0400.0600.295+0.235 [+0.165, +0.305]
Qwen3-8B0.0200.0850.270+0.185 [+0.110, +0.255]
Llama-3.2-3B0.0650.0450.155+0.110 [+0.055, +0.170]
Word, mean attemptsLlama-3.1-8B (paper)32.919.03.1−15.9 [−19.1, −12.9]
Qwen3-8B32.515.73.3−12.5 [−15.8, −9.2]
Llama-3.2-3B32.714.44.1−10.3 [−13.4, −7.3]

The paper's findings hold on both new backbones. Distillation improves every backbone over its base model. Actions alone are enough on Circuit Decoding for the 8B models, and the scratchpad helps the weaker 3B student. On the other two tasks the scratchpad is what turns the gathered information into the decision.

Training seeds (Llama-3.1-8B, same data, same rows).

Task, metricSeed 0: A / BSeed 1: A / BB − A, seed 0B − A, seed 1
Circuit, Success@180.92 / 0.920.94 / 0.96+0.00 [−0.07, +0.07]+0.02 [−0.04, +0.08]
Movie, Success@100.060 / 0.2950.065 / 0.290+0.235 [+0.165, +0.305]+0.225 [+0.160, +0.290]
Word, mean attempts19.0 / 3.120.8 / 18.2−15.9 [−19.1, −12.9]−2.5 [−7.6, +2.8]
Word, attempts, re-decoded19.0 / 3.120.8 / 8.3−15.9 [−19.1, −12.9]−12.5 [−16.9, −7.7]
Word, Success@30.11 / 0.960.13 / 0.55+0.85 [+0.77, +0.92]+0.42 [+0.30, +0.54]
Word, Success@3, re-decoded0.11 / 0.960.13 / 0.77+0.85 [+0.77, +0.92]+0.64 [+0.52, +0.75]

In the re-decoded rows the scratchpad student's invalid attempts are re-decoded under a trie of the vocabulary. The action-only student is left unchanged.

Circuit Decoding and Movie Recommendation agree to within 0.04 in every cell. On Word Guess the seed-1 scratchpad student fails on one of the three vocabularies in these rows. Its first guess there is not a vocabulary word, a state absent from training, and 38 of the 40 episodes run to the budget. Re-decoding recovers 27 of them and brings the format contrast to −12.5 attempts, within the range of the new backbones. The other 11 fail by repeating one legal guess, the dead end described in the paper. The unconstrained rows remain the reported result, since the paper's protocol does not repair outputs (Section 4).

W2. Three toy tasks from one benchmark and one LLM (tasks)

Choice of benchmark. We chose the benchmark for one property, not for its adoption. Our claims need a teacher that can be checked against a Bayesian reference, which requires a task whose latent state, query space and observation model are fully specified and whose respondent is a deterministic program. The three tasks are the benchmark's tasks with this property. The references let us validate the teacher (Table 6), score every question by its reference EIG (Appendix E.3), apply an oracle read-out (Figure 2a) and attribute student performance to single teacher components (Table 4). The other two tasks lack the property: in Twenty Questions the model under test is the answerer, and Word Chaining has no hidden state.

Standard benchmarks. In the 20-Questions and clarification settings of the cited methods the respondent is typically a language model or a human, so the likelihood is not specified and a Bayesian teacher could not be validated. We therefore have no results there, and we accept that this limits direct comparison. The comparison in the paper is to the benchmark's published frontier-model numbers and is indicative only (Table 2, Appendix B.5).

"Toy". The tasks are small because smallness is what makes exact references possible, but they are not easy. The strongest reported model scores 39.8 on Circuit Decoding and 84.3 on Movie Recommendation, where 50 is a random pick, and the 8B base model scores 0.7 and 50.4. The benchmark is also used by later work: Zou et al. (ICLR 2026) train on simplified variants of its tasks, and on their circuit variant Gemini 2.5 Pro reaches 92% against 40% on the released task (Appendix H). The three tasks span continuous, structured-binary and finite-symbolic latent states, and neither the policy nor the information-gain criterion transfers between them (Appendix G), so they are not three instances of one puzzle.

W3. Baselines from related work

We do not include re-implementations of the cited systems, and we cannot add them within the discussion period. Our controlled arms represent the families of methods in the related work, with episodes, student, recipe and evaluator held fixed. The paper does not label them this way, and it should.

Family in the related workCorresponding armResult
Prompted language models without trainingBase model and the benchmark's reported modelsStrongest reported model 39.8 and 84.3, student 97.4 and 90.0 on Circuit Decoding and Movie Recommendation (Table 2, indicative)
Supervision generated by a language model70B LLM teacher with action-only students−0.92 Success@18 and −0.03 Success@10 against the Bayesian teacher's action-only students (Table 4)
Imitating a Bayesian model's outputs without its belief (Qiu et al., 2025)Action-only students of the Bayesian teacher0.085 vs. 0.345 Success@10 and 19.2 vs. 3.4 attempts once the belief is written (Table 3)
Other query rules under the same beliefELBO+Random and ELBO+Uncertainty teachersEIG is better by +0.42 / +0.17 / +0.21 over random and by +0.15 over uncertainty sampling (Table 4)
Bayesian loop at inference time (BED-LLM, UoT)The Bayesian teachers themselvesTeacher 1.00 / 0.41 / 2.29, student 0.93 / 0.35 / 3.37 (Table 2)

These arms are analogues under our recipe, not the published systems, and outcome-based filtering as in STaR-GATE is not covered. For the inference-time family the teacher row is the idealized case: it runs the posterior update and EIG at every turn with the exact task model, and the student reaches most of that level with one forward pass per turn.

We chose arms over systems because the cited systems differ in backbone, prompts, training data and task at once, so a system-level table would not show which design choice matters. The closest reward-based methods cannot be set against our numbers: Hartmann et al. (2026) and Deng et al. (2026) are concurrent preprints, and Zou et al. (2026) use simplified variants of these tasks (Appendix H).

Reviewer_ve4vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework for distilling the information-seeking policy and belief states of a variational Bayesian agent into LMs. The proposed approach uses ELBO for belief updates, EIG for query selection, and structured belief scratchpads to supervise Llama-3.1-8B-Instruct, achieving substantial improvements across three interactive reasoning tasks

Critical strengths

The proposed method achieves improvements on Circuit Decoding, Movie Recommendation, and Word Guess. Beyond reporting final task performance, the paper provides comparisons between action-only and belief-scratchpad supervision, ablations of teacher components, and detailed failure analyses

This paper leverages a verifiable Bayesian agent to generate high-quality supervision rather than relying on LLM-generated reasoning trajectories

Critical weaknesses

The experimental design does not sufficiently isolate the contribution of EIG supervision. The EIG, Random, and Uncertainty teachers differ substantially in their action distributions and predictability, while all students are trained using SFT and evaluated with greedy decoding. For example, in Movie Recommendation, the Random teacher repeats only 2.2% of its queries, whereas the corresponding student repeats nearly 80%, suggesting that the performance gap may partly reflect the difficulty of imitating stochastic policies rather than the intrinsic advantage of EIG-based query selection. Furthermore, belief-scratchpad supervision introduces substantially more supervised tokens and additional intermediate computations, but the paper lacks sufficient controls to isolate the contribution of explicit belief supervision

The use of test-set feedback during method development raises concerns about evaluation independence. The authors acknowledge that several design choices, including components of the Movie Recommendation teacher and the Word Guess scratchpad format, were informed by test-set evaluations. In particular, the Word Guess scratchpad was revised and evaluated on the first 100 test episodes, which were subsequently included in the final reported results. Although the training data are disjoint from the benchmark, using test-set feedback during method development may introduce selection bias. Additionally, the poor performance of the base LLM on some tasks is partly attributable to output-format violations, making it difficult to attribute the full improvement to enhanced reasoning capabilities

Questions for authors

How can authors disentangle the quality of the EIG policy from the learnability of the teacher's behavior? It would be useful to evaluate an Exact Bayes teacher with deterministic tie-breaking or non-EIG baselines with controlled action entropy to determine whether the observed improvements primarily result from information-gain optimization or from more predictable supervision trajectories

4 comments

We thank the reviewer for the careful review. It raises two weaknesses and one question, and we answer them in order.

  • W1 (the design does not isolate the contribution of EIG supervision or of belief supervision): we separate the query rule from imitation per task, and we report a new token-matched control for the belief.
  • W2 (test-set feedback during development, and base-model format errors): results on episodes that played no part in development, and a format-only control.
  • Q1 (query quality vs. learnability of the teacher): the experiment the reviewer proposes, an Exact Bayes teacher with deterministic tie-breaking.

W1. Isolating EIG supervision and belief supervision

EIG supervision. The reviewer is right that a student-level gap mixes how informative the teacher's queries are and how well the student can imitate them. For each task we ask whether the Random student matches its teacher, and compare the EIG − Random gap between teachers (no imitation involved) with the gap between students.

Task, metricDoes the Random student match its teacher?Gap between teachersGap between students
Circuit, Success@18Yes: EIG-optimal rate 0.30 vs. 0.31, no repeats+0.30+0.42
Word Guess, Success@3Yes: 0.5% repeats, Success@3 0.73 vs. 0.71+0.27+0.21
Movie, Success@10No: teacher repeats 2.2% of questions, student 80%+0.04+0.17

On Circuit Decoding and Word Guess the Random student behaves like its teacher, and the gap between students is comparable to the gap between teachers (Table 6, Appendices F.1 and F.3). The student gap there is not an artifact of the Random student failing to imitate its teacher. How much the predictability of the EIG teacher contributes is a separate question, answered in Q1.

On Movie Recommendation the reviewer's reading is correct. The student gap is four times the teacher gap, and the excess comes from the Random student failing to imitate its teacher. The same holds for uncertainty sampling, which falls behind EIG after distillation because its student repeats 59% of its questions (Section 5.1). On this task the student gap mostly measures which policy survives distillation under greedy decoding. At the teacher level EIG is the better rule against random querying on all three tasks.

Belief supervision. We ran the control the reviewer asks for on Circuit Decoding, where the scratchpad is the belief alone. Every belief line in the Exact Bayes+EIG scratchpad training set was replaced by a constant line of identical token count (24 × "0.50"), with everything else unchanged. The resulting student C was evaluated on the full 300 test episodes and paired with the paper's action-only (A) and belief-scratchpad (B) students.

StudentSuccess@18 [95% CI]EIG-optimal queriesFailures after / before collapse
A: action-only0.643 [0.588, 0.695]0.89689 / 18
C: uninformative belief line0.717 [0.663, 0.765]0.86464 / 21
B: belief scratchpad0.840 [0.794, 0.877]0.95940 / 8
Paired contrastDifference [95% CI]DiscordantMcNemar p
C − A: format and tokens without content+0.073 [+0.010, +0.137]58 / 360.030
B − C: belief content+0.123 [+0.060, +0.187]67 / 300.0002
B − A: whole scratchpad+0.197 [+0.133, +0.257]82 / 236e-9

The reviewer is partly right: part of the gain is structural and part comes from the belief itself. The intervals overlap, so we read this as both parts contributing and not as a precise split. Query quality improves only with the content. The read-out gain is shared: failures after the version space has collapsed fall from 89 to 64 with the format alone and to 40 with the real belief.

This is one task, one teacher and one seed, and this teacher's action-only student is weak because of its random tie-breaking (Q1). The control is not run on Movie Recommendation and Word Guess, where the scratchpad also carries scores and check lines and adds far more supervised tokens. There our evidence is the read-out decomposition of Figure 2a.

W2. Test-set feedback and the base model's format errors

Test-set feedback. The concern is fair, and it is why Section 4 discloses these choices. Three facts bound their effect.

  1. The Word Guess format was a choice between two designs, made once. On the first 100 test episodes the first design failed (31.9 attempts, 22% solved) and the second worked (3.09 attempts, 98% solved). The remaining 300 test episodes were then run without further change (Appendix D.5). The student solves 293 of them (97.7%), in line with 98 of the first 100 and 391 of the 400 held-out synthetic episodes (Appendix F.3).
  2. Held-out synthetic episodes reproduce the main results. They were never used in development and are disjoint from training, validation and the benchmark. On Word Guess the scratchpad student needs 3.37 attempts with Success@3 0.917 on both the 400 held-out episodes and the complete test split, and the scratchpad saves 14.3 and 15.8 attempts (Appendix F.3). On Movie Recommendation the scratchpad gain is +0.215 [+0.140, +0.290] on 200 held-out episodes, checked against a rule fixed before the benchmark run, and +0.260 on the test split (Appendix E.5). This check also covers the decision to expose beliefs, which followed a test-split failure analysis.
  3. The movie temperature and range transform are shared by the three ELBO teachers. They can raise a teacher's absolute score, but they do not favor one arm of our paired contrasts. They were tuned with a Bradley–Terry respondent and not under the deterministic protocol of the reported numbers (Appendix C.2). The teacher scores 0.434 on the synthetic training episodes and 0.410 on the test split.

Base model and format. The reviewer is right that part of the base model's score is format. On Movie Recommendation all of its questions are invalid, so its 0.032 is a no-information baseline. Two observations separate format from content.

First, the students of the 70B LLM teacher are a format-only control. That teacher is constrained to legal actions but is uninformed, and its students produce valid queries on every turn. They reach 0.000 Success@18 on Circuit Decoding and 0.052 Success@10 on Movie Recommendation, against 0.000 and 0.032 for the base model (Appendices D.6, F.1 and F.2). Fixing the format alone gives almost nothing.

Second, where the base model's output does parse, it is still wrong. On Circuit Decoding 203 of its 300 final guesses parse and none is correct (Appendix E.6).

We do not claim enhanced reasoning in general. The student learns a task-specific procedure (Appendix G).

W2. Test-set feedback and the base model's format errors

Test-set feedback. The concern is fair, and it is why Section 4 discloses these choices. Three facts bound their effect.

  1. The Word Guess format was a choice between two designs, made once. On the first 100 test episodes the first design failed (31.9 attempts, 22% solved) and the second worked (3.09 attempts, 98% solved). The remaining 300 test episodes were then run without further change (Appendix D.5). The student solves 293 of them (97.7%), in line with 98 of the first 100 and 391 of the 400 held-out synthetic episodes (Appendix F.3).
  2. Held-out synthetic episodes reproduce the main results. They were never used in development and are disjoint from training, validation and the benchmark. On Word Guess the scratchpad student needs 3.37 attempts with Success@3 0.917 on both the 400 held-out episodes and the complete test split, and the scratchpad saves 14.3 and 15.8 attempts (Appendix F.3). On Movie Recommendation the scratchpad gain is +0.215 [+0.140, +0.290] on 200 held-out episodes, checked against a rule fixed before the benchmark run, and +0.260 on the test split (Appendix E.5). This check also covers the decision to expose beliefs, which followed a test-split failure analysis.
  3. The movie temperature and range transform are shared by the three ELBO teachers. They can raise a teacher's absolute score, but they do not favor one arm of our paired contrasts. They were tuned with a Bradley–Terry respondent and not under the deterministic protocol of the reported numbers (Appendix C.2). The teacher scores 0.434 on the synthetic training episodes and 0.410 on the test split.

Base model and format. The reviewer is right that part of the base model's score is format. On Movie Recommendation all of its questions are invalid, so its 0.032 is a no-information baseline. Two observations separate format from content.

First, the students of the 70B LLM teacher are a format-only control. That teacher is constrained to legal actions but is uninformed, and its students produce valid queries on every turn. They reach 0.000 Success@18 on Circuit Decoding and 0.052 Success@10 on Movie Recommendation, against 0.000 and 0.032 for the base model (Appendices D.6, F.1 and F.2). Fixing the format alone gives almost nothing.

Second, where the base model's output does parse, it is still wrong. On Circuit Decoding 203 of its 300 final guesses parse and none is correct (Appendix E.6).

We do not claim enhanced reasoning in general. The student learns a task-specific procedure (Appendix G).

Thank you to the authors for their response and clarifications. However, these revisions are not sufficient to warrant a higher score, so I am maintaining my original rating.

Reviewer_kLjaOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes to distill the information seeking skills of a Bayesian teacher into a lightweight language model, so that it can guide uncertainty tracking and question/action selection in agentic tasks in an amortized manner. Experiments on three tasks, i.e., circuit decoding, movie recommendation, and work guess, demonstrate the effectiveness of this distilled mechanism and present the individual effects from different components of the Bayesian teacher.

Critical strengths

1.The paper reformulates interactive agent tasks as Bayesian inference problems with latent states, queries, beliefs, and decisions, thus providing a principled way to study information-seeking behavior in human-AI interactions.

2.The ablation studies provide clear and meaningful insights into how different components of the Bayesian teacher contribute to the observed performance improvements, separating the effects from query, inference, and teacher type. The paper also analyzes the cross-task bottlenecks of distillation, providing hints on when and why the proposed method may succeed or fail.

3.Substantial empirical improvements over SOTA models are observed on three tasks.

Critical weaknesses

1.The task diversity, or external validity, could be further improved. Although the latent state structures of the three tasks are different, they are all synthetic puzzles with a well-defined hypothesis space and known interaction rules, which may limit their generalization to more realistic, open-ended settings. Moreover, as discussed in the paper, the cross-task transfer is already limited among these three similar tasks, which makes generalization a main concern for this study.

  1. If the transfer is limited, then whether the improvement is robust enough within a given task should be adequately examined. Currently there is only one backbone model, Llama-3.1-8B-Instruct being evaluated throughout the paper, so evaluating 1-2 more model sizes may help strengthen the study.

3.The benefits of belief supervision might be confounded with hints from task-specific latent state structures. Specifically, the belief scratchpads contain not only teacher beliefs but also structural problem-solving processes. The distilled model may have learned these problem-solving patterns rather than the task-agnostic uncertainty patterns. Additional ablations would help disentangle these effects.

Questions for authors

1.I understand the limitations in applicable task range can be partly attributed to Bayesian teacher’s underlying assumptions, but could author provide some insights on how this study could be expanded to open-ended task settings (i.e., coding tasks)?

2.Could you clarify which adaptations in your design are necessary to integrate Bayesian inference with agentic tasks?

4 comments

We thank the reviewer for the careful reading. The review raises three weaknesses and two questions, and we answer them in order.

  • W1 (synthetic tasks with a well-defined hypothesis space, and limited cross-task transfer): we explain why the tasks were chosen this way and agree on the limit.
  • W2 (robustness within a task, with only one backbone evaluated): new results on two further backbones and a second training seed.
  • W3 (belief supervision may be confounded with task-specific structure): a new token-matched control that separates the two.
  • Q1 (extension to open-ended settings such as coding) and Q2 (which design adaptations are necessary): answered in the last part.

W1. Task diversity and external validity

We agree with this description. The tasks are synthetic with a fully specified hypothesis space, and this is a design choice. It is what lets us validate the teacher against a Bayesian reference (Table 6), score every question by its reference information gain (Appendix E.3) and apply a Bayes-optimal read-out to a student's transcript (Figure 2a). The three tasks still differ in latent state, observation and read-out, and the leave-one-task-out result shows that the student learns a task-specific procedure (Appendix G). Its robustness within a task is therefore what has to be shown, which W2 addresses. An open-ended task cannot be built and validated within the discussion period, so we analyze what we expect to carry over in Q1.

W2. One backbone and robustness within a task

We ran the paper's recipe on Qwen3-8B and Llama-3.2-3B-Instruct and trained a second seed of Llama-3.1-8B-Instruct, for the ELBO+EIG teacher in both formats (A = action-only, B = belief scratchpad). Only the backbone and its chat template changed. We used test subsets fixed before any run: every third Circuit Decoding episode (100), the first 200 Movie Recommendation episodes and the first 100 Word Guess episodes. All comparisons are paired on the same rows.

Task, metricBackboneBaseABB − A [95% CI]
Circuit, Success@18Llama-3.1-8B (paper)0.000.920.92+0.00 [−0.07, +0.07]
Qwen3-8B0.000.910.84−0.07 [−0.16, +0.02]
Llama-3.2-3B0.000.810.91+0.10 [+0.01, +0.19]
Movie, Success@10Llama-3.1-8B (paper)0.0400.0600.295+0.235 [+0.165, +0.305]
Qwen3-8B0.0200.0850.270+0.185 [+0.110, +0.255]
Llama-3.2-3B0.0650.0450.155+0.110 [+0.055, +0.170]
Word, mean attemptsLlama-3.1-8B (paper)32.919.03.1−15.9 [−19.1, −12.9]
Qwen3-8B32.515.73.3−12.5 [−15.8, −9.2]
Llama-3.2-3B32.714.44.1−10.3 [−13.4, −7.3]

The paper's findings hold on both new backbones. Distillation improves every backbone over its base model. Actions alone are enough on Circuit Decoding for the 8B models, and the scratchpad helps the weaker 3B student. On the other two tasks the scratchpad is what turns the gathered information into the decision.

Training seeds (Llama-3.1-8B, same data, same rows).

Task, metricSeed 0: A / BSeed 1: A / BB − A, seed 0B − A, seed 1
Circuit, Success@180.92 / 0.920.94 / 0.96+0.00 [−0.07, +0.07]+0.02 [−0.04, +0.08]
Movie, Success@100.060 / 0.2950.065 / 0.290+0.235 [+0.165, +0.305]+0.225 [+0.160, +0.290]
Word, mean attempts19.0 / 3.120.8 / 18.2−15.9 [−19.1, −12.9]−2.5 [−7.6, +2.8]
Word, attempts, re-decoded19.0 / 3.120.8 / 8.3−15.9 [−19.1, −12.9]−12.5 [−16.9, −7.7]
Word, Success@30.11 / 0.960.13 / 0.55+0.85 [+0.77, +0.92]+0.42 [+0.30, +0.54]
Word, Success@3, re-decoded0.11 / 0.960.13 / 0.77+0.85 [+0.77, +0.92]+0.64 [+0.52, +0.75]

In the re-decoded rows the scratchpad student's invalid attempts are re-decoded under a trie of the vocabulary. The action-only student is left unchanged.

Circuit Decoding and Movie Recommendation agree to within 0.04 in every cell. On Word Guess the seed-1 scratchpad student fails on one of the three vocabularies in these rows. Its first guess there is not a vocabulary word, a state absent from training, and 38 of the 40 episodes run to the budget. Re-decoding recovers 27 of them and brings the format contrast to −12.5 attempts, within the range of the new backbones. The other 11 fail by repeating one legal guess, the dead end described in the paper. The unconstrained rows remain the reported result, since the paper's protocol does not repair outputs (Section 4).

W3. Belief supervision vs. task-specific structure

We agree with this description. The scratchpad is the teacher's belief plus the computation that turns it into an action, so it is task-specific by construction. We do not claim a task-agnostic representation of uncertainty (Appendix G). Our claim is narrower: where the decision is a computation on the belief, the student benefits from explicit supervision of that belief and computation (Section 5.3).

On Word Guess the procedure is needed, as the reviewer suggests: a scratchpad that listed the surviving candidates as whole words without check lines failed (31.9 attempts on 100 test episodes), and the letter-spelled format with check lines fixed it (3.09, Appendix D.5). On Movie Recommendation a Bayes-optimal read-out of the action-only student's own transcript raises Success@10 from 0.085 to 0.305 (Figure 2a), so what that student lacks is the read-out.

New control. We ran the ablation the reviewer asks for on Circuit Decoding, where the scratchpad is the belief alone. Every belief line in the Exact Bayes+EIG scratchpad training set was replaced by the constant line 24 × "0.50", with identical token counts and everything else unchanged. The resulting student C was evaluated on the full 300 test episodes and paired with the paper's action-only (A) and scratchpad (B) students.

StudentSuccess@18 [95% CI]EIG-optimal queriesFailures after / before collapse
A: action-only0.643 [0.588, 0.695]0.89689 / 18
C: uninformative belief line0.717 [0.663, 0.765]0.86464 / 21
B: belief scratchpad0.840 [0.794, 0.877]0.95940 / 8
Paired contrastDifference [95% CI]DiscordantMcNemar p
C − A: format without content+0.073 [+0.010, +0.137]58 / 360.030
B − C: belief content+0.123 [+0.060, +0.187]67 / 300.0002
B − A: whole scratchpad+0.197 [+0.133, +0.257]82 / 236e-9

The reviewer is partly right. The format alone gives +0.073 and the belief content a further +0.123. The intervals overlap, so we read this as both parts contributing and not as a precise split. Query quality improves only with the content. The read-out gain is shared: failures after the version space has collapsed fall from 89 to 64 with the format alone and to 40 with the real belief.

Limits. This is one task, one teacher and one seed, and a constant line matches tokens but not learning difficulty. The control is not run on Movie Recommendation and Word Guess, where the scratchpad also carries scores and check lines. The teacher used here breaks ties at random, which makes its action-only student weak: with deterministic ties that student reaches 0.873 without a scratchpad (our reply to Reviewer ve4v).

Q1. Extension to open-ended settings such as coding

The teacher needs a hypothesis space for the latent state, an observation model for each query, and synthetic episodes with known ground truth. Coding tasks supply the last two readily, because program execution is a deterministic oracle. In debugging the latent state is the fault, a query is a test input and the observation is the program's output, which is structurally Circuit Decoding.

The obstacle is the hypothesis space. If the candidate faults can be enumerated, our categorical family applies as is. If not, a language model has to propose them, and the teacher is then only as good as the proposals and has no exact reference. The scratchpad also relies on a small, fixed belief parameterization (Section 2), which a belief over an open set of programs does not have. By the leave-one-task-out result we expect no transfer between such tasks, so each would need its own teacher traces.

Q2. Which design adaptations are necessary?

There are five.

  1. A belief with a compact, fixed-size summary, because the student can only be supervised on a belief it can write (Appendix D.4).
  2. Supervision of the belief and the read-out, not only the action. It is necessary where the decision is a computation on the belief (+0.260 Success@10 on Movie Recommendation, 19.2 → 3.4 attempts on Word Guess). A strong student does not need it on Circuit Decoding (+0.007, Table 3).
  3. A format the language model can read. A whole-word candidate list failed on Word Guess and a letter-spelled format worked (Appendix D.5).
  4. Offline synthetic episodes with known latent states, so that the teacher's traces can be checked against a reference (Appendix D.1).
  5. Predictable teacher behaviour. With the lowest-index query among equally informative ones instead of a random one, the Exact Bayes+EIG circuit student rises from 0.643 to 0.873 Success@18 at unchanged teacher quality (new experiment, in our reply to Reviewer ve4v).

We would be grateful to know whether the within-task robustness results and the clarification of the belief-supervision claim address the reviewer's main concerns.

Thanks for the response! The additional experiments improve the assessment of the robustness of the findings. I would suggest authors to consider a broader range of model sizes in their next version. I would maintain the score unchanged, as it reflects my excitement about the work and the limitations acknowledged by the authors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.