- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper looks at interactive dialogue settings where LLMs must ask questions in order to seek information or resolve ambiguity. The authors argue that standard approaches based on expected information gain alone “may not capture whether a question is premature, redundant, or contextually misaligned.” Accordingly, they propose a dual-objective framework, PAUSE, that introduces a learned “appropriateness” score. On two multi-turn benchmarks (NewsInterview and ClariQ), they report PAUSE leads to modest gains in coverage and selectivity.
Critical strengths
- The information-seeking setting is both general and relevant to many applications beyond the ones considered here; e.g., similar approaches have recently been applied to clinical diagnosis and scientific discovery.
- The experiments include a nice set of ablations (different hyperparameters and thresholds) as well as prompt-only baselines and different model backbones.
- The method is clearly described. The writing is relatively readable and free of obvious AI slop, though too much space is allocated to certain figures (e.g., Fig. 2) and tables.
Critical weaknesses
My main concern here is that the method is solving a problem that I am not convinced actually exists outside the authors’ specific setting. The central premise is that PAUSE addresses contextual prematureness; in the authors’ words:
“A question can have high prospective information value while presupposing an event or preference not yet established.” (L165)
The problem here is that LLMs are already extremely adept at exactly this kind of pragmatic inference. This is a direct consequence of the standard next-token prediction objective, which encourages models to learn dialogue structure in which utterances follow naturally without presupposing events or preferences.
The kind of contextual inappropriateness that PAUSE addresses arises primarily in artificial benchmark settings where there is a fixed candidate set in which some of the answers contain presuppositions or are mutually-redundant.
When there is a learned generator , as is the case here, the presence of contextually-inappropriate responses suggests that something in the setup has caused the LLM to diverge from its base policy: either the generator prompt leaks privileged information about the setting, or the training procedure itself causes divergence. Either way, simply sampling candidate responses from a frozen LLM, and then reranking them by EIG, avoids these pitfalls and is already a well-established approach (e.g., 1-3 below).
In sum, I’m unconvinced by the need for a multi-objective optimization framework when simple sample-and-rerank methods already work quite well. In order to motivate this work, the authors would need to better establish that the problems they claim to solve occur in natural dialogue (not just benchmark settings) and as part of standard EIG-based methods (not just as a byproduct of utility optimization).
References
- Mazzaccara et al., 2024. Learning to Ask Informative Questions: Enhancing LLMs with Preference Optimization and Expected Information Gain. https://arxiv.org/pdf/2406.17453
- Choudhury et al., 2026. BED-LLM: Intelligent Information Gathering with LLMs and Bayesian Experimental Design. https://arxiv.org/abs/2508.21184
- Grand et al., 2026. Shoot First, Ask Questions Later? Building Rational Agents that Explore and Act Like People. https://arxiv.org/pdf/2510.20886
Questions for authors
- How robust is your method to different hyperparameter configurations? If you were to deploy this in production, how would you choose from the many reported combinations of coefficients and thresholds (Table 1)?
- In addition to presupposition, another problem you point to with value-of-information methods is redundancy. However, under a calibrated belief model, redundant questions should yield low information. Can you give some intuition for why this is not the case in your setting?
3 comments
I'm not sure if it's just me but I found this rebuttal a bit hard to parse. As a concrete example:
Removing T-based filtering from generator training retains a gate gain of +0.0795 [0.0467, 0.1131].
What is T-based filtering? What is a gate gain? Is the quoted value a positive or negative result?
After reading through the comment several times, I'm still not much clearer on most of the questions I asked. I think LLM writing may be making it harder than necessary for me to see how these comments address my questions or the questions of the other reviewers.
In the absence of a clear rebuttal, I'm going to retain my original score.
Thank you for your reply. We will explain the quoted result first, then answer your original questions directly.
What the quoted experiment means. T is the score from a separate model that predicts whether a question fits the current conversation. During training, we normally use it to exclude questions the model judges unsuitable.
We then tested that same generator on 348 simulated interviews, with eight questions per interview. Coverage is the average percentage of reference facts recovered; higher is better.
For the second rule, if no question passes the fit check, we choose the one with the highest fit score. By “gate gain” we meant this difference: 0.0795, or +7.95 percentage points, with a 95% confidence interval of +4.67 to +11.31 points. This is a positive result. It shows that checking fit when selecting questions helps even without fit-based filtering during training.
Why redundant questions can receive high scores. Under a calibrated belief model, redundant questions should have low EIG. Our original predictor scores a question without explicitly representing what the dialogue has already established. It can therefore overvalue information already known. A simple repetition filter reaches 55.60% coverage versus PAUSE’s 56.06%.
Whether the setting creates the problem. At each NewsInterview turn, four new questions are generated from the introduction and dialogue history, rather than taken from a fixed question bank. Neither the generator nor the selector sees the hidden reference facts.
How the threshold is chosen. PAUSE uses one fit-score cutoff, fixed at 0.47 before the main NewsInterview test. Across six tested cutoffs from 0.40 to 0.60, all coverage differences were positive and four were statistically significant. For deployment, we would choose this cutoff on development data. The weighted-score coefficients in the table belong to comparison methods, not to PAUSE’s selection rule.
Response to Reviewer grz3
Thank you for the connections to Bayesian information gathering and the questions about candidate generation, training, and redundancy. We added controls that distinguish the effects of candidate construction, utility estimation, and contextual selection.
Are the candidates or training procedures creating the problem?
NewsInterview samples four new questions from the current dialogue state at every turn. The generator and selector receive introductory context and dialogue history; the reference transcript and information-item list are reserved for the respondent and evaluator. ClariQ is a separate, single-turn candidate-selection task. The SPoRC position probe uses recorded human dialogue with controlled position perturbations to evaluate the scorer's placement sensitivity. We will clarify these distinct experimental roles and input boundaries.
Removing T-based filtering from generator training retains a gate gain of +0.0795 [0.0467, 0.1131]. The benefit therefore persists without using T to construct the training examples. This control retains utility-oriented SFT/DPO and specifically tests the role of the T filter.
A fully unadapted generator with a standard chat template and learned-utility reranking reaches 0.5055, compared with 0.5606 for the trained-and-gated system, an observed difference of +0.0551 from the reported means. This comparison changes both generation and selection. Within the frozen standard-chat variant, the gate adds +0.0038, which is not statistically significant. Thus, the measured value of the gate depends on the generator and prompt; our evidence supports the tested selection policies rather than a universal need for gating.
Why can utility scoring favor redundant questions?
Under a calibrated belief model, a question that adds no evidence about the latent variable should have low conditional information gain. The distinction in our setting is that V predicts an outcome-derived novelty target. The original scorer takes the question without an explicit representation of the information already established in the dialogue. This proxy can therefore assign a high score to a question whose answer is generally informative but already known in the current conversation.
The cited methods make this distinction important. BED-LLM constructs a probabilistic model and updates beliefs to estimate EIG; Grand et al. combine Bayesian information gathering with action selection. We will position PAUSE as contextual selection with learned utility proxies and distinguish it from these Bayesian formulations. Our frozen-generator control retains the learned scorer and therefore does not constitute a reproduction of posterior-based EIG or establish superiority over it.
Do stronger contextual baselines absorb the benefit?
The gate still improves the tested history-aware utility scorer by +0.0349 [0.0015, 0.0683] (paired 95% bootstrap CI). A complementary deduplication control improves the original utility-only baseline by +0.0517 [0.0166, 0.0869], reaching 0.5560 versus PAUSE's 0.5606; the remaining +0.0047 difference is not statistically significant. These controls establish a strong lexical deduplication baseline and a gate benefit in the history-aware configuration.
Soft score combinations provide another useful comparison. The rules 0.25T + 0.75V and 0.50T + 0.50V reach 0.5529 and 0.5478, respectively, above utility-only selection at 0.5042. Their differences from hard filtering are not established as statistically significant. We will present hard filtering as an interpretable eligibility constraint and soft weighting as a competitive alternative. The supported contribution is contextual selection with learned utility; timing-specific attribution remains a separate mechanism question.
How should the threshold and candidate budget be chosen?
The gain is positive across all six tested thresholds between 0.40 and 0.60, with four intervals excluding zero. The canonical NewsInterview threshold, τ = 0.47, was fixed before the main evaluation and remains unchanged; the sweep is a sensitivity analysis rather than a procedure for selecting a new test-optimal threshold.
At K = 2, 4, and 8, the gate effects range from approximately +0.030 to +0.056, with intervals excluding zero at K = 4 and 8. This supports the rule at multiple candidate budgets, although a production budget also requires latency measurements.
For deployment, we recommend fixing the scorer configuration and threshold on development data using an explicit operating target, then evaluating the frozen policy on test data. The weighted-score coefficients belong to comparison rules, not to PAUSE's hard-filter policy. We will make this distinction and the sensitivity results explicit in the experimental description.