- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how LLM judgments about the value of prospective information shape sequential elicitation. It constructs a controlled simulation in which models score candidate statements, the highest-scoring statement is selected, and simulated responses update the interaction history. Experiments compare topic breadth and depth across models and examine the effects of candidate availability, history, response labels, and instructions about redundancy.
Critical strengths
The controlled setup separates candidate evaluation from question generation and respondent behavior, making the effects of model choice easier to study. The comparison across models and the sensitivity analyses provide a useful characterization of how this component can influence an elicitation trajectory.
Critical weaknesses
The interpretation of the breadth and depth differences depends substantially on the scoring interface. The authors’ own label ablation shows that changing the affirmative response from “Yes” to “Valuable” can considerably alter trajectories and the relative ordering of models. This is a useful finding, but it leaves unclear which model-level conclusions remain stable across reasonable elicitor implementations. Positive candidate-level rank correlations do not fully resolve this because selection depends on the highest-ranked candidate and affects subsequent opportunities.
The broader implications for elicitation also remain limited. The experiments use one fixed argument collection, a prescribed candidate-generation rule, and confirmations sampled independently of content and previous answers. These choices help isolate scoring behavior, but topic coverage and consecutive topic runs do not establish how effectively an agent learns about a respondent. The paper acknowledges its descriptive scope. My reservation is whether the current findings provide sufficiently general insight beyond this particular setting.
Questions for authors
Which conclusions about differences between models remain stable across the alternative response labels, especially where the relative ordering changes?
Is there evidence that these conclusions persist under another elicitor implementation? A focused validation would help validate the findings.