acceptodds
Withdrawn from ICLR 2027

On Open-Ended Information Seeking for Information Elicitation Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Information elicitation is an open-ended information-seeking problem in which an interaction can unfold in many potentially valuable directions, requiring an elicitor to continually determine which information to pursue as new information emerges. In agentic elicitation, these decisions may be delegated to a foundation model, yet how model choice shapes the resulting information-seeking behavior remains understudied. We study how judgments about information value vary across LLMs and how these differences shape sequential information seeking. We first examine these judgments across 11 LLMs spanning multiple model families and parameter scales, using a shared set of information and elicitation objectives. We then develop a controlled elicitation simulation in which different models encounter the same information space and use the same selection rule, isolating these judgments from question generation and respondent behavior. Using this setting, we characterize the breadth-depth behavior that emerges from model-specific information-seeking preferences over the course of elicitation. We further examine how interaction history changes the evaluation and subsequent selection of prospective information. We test the robustness and boundaries of these findings through sensitivity analyses and ablations over the opportunities available to the elicitor, the response labels used to operationalize information-seeking preferences, the presence of interaction history, and whether redundancy is explicitly relevant to the assessment. The project code, data, and trajectory files are available at https://github.com/anonymousauthor2352/open-elicitation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
information elicitation, information seeking, conversational agents, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_p9UmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how LLM judgments about the value of prospective information shape sequential elicitation. It constructs a controlled simulation in which models score candidate statements, the highest-scoring statement is selected, and simulated responses update the interaction history. Experiments compare topic breadth and depth across models and examine the effects of candidate availability, history, response labels, and instructions about redundancy.

Critical strengths

The controlled setup separates candidate evaluation from question generation and respondent behavior, making the effects of model choice easier to study. The comparison across models and the sensitivity analyses provide a useful characterization of how this component can influence an elicitation trajectory.

Critical weaknesses

The interpretation of the breadth and depth differences depends substantially on the scoring interface. The authors’ own label ablation shows that changing the affirmative response from “Yes” to “Valuable” can considerably alter trajectories and the relative ordering of models. This is a useful finding, but it leaves unclear which model-level conclusions remain stable across reasonable elicitor implementations. Positive candidate-level rank correlations do not fully resolve this because selection depends on the highest-ranked candidate and affects subsequent opportunities.

The broader implications for elicitation also remain limited. The experiments use one fixed argument collection, a prescribed candidate-generation rule, and confirmations sampled independently of content and previous answers. These choices help isolate scoring behavior, but topic coverage and consecutive topic runs do not establish how effectively an agent learns about a respondent. The paper acknowledges its descriptive scope. My reservation is whether the current findings provide sufficiently general insight beyond this particular setting.

Questions for authors

Which conclusions about differences between models remain stable across the alternative response labels, especially where the relative ordering changes?

Is there evidence that these conclusions persist under another elicitor implementation? A focused validation would help validate the findings.

Reviewer_LAg1OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how different large language models evaluate candidate information and how these preferences influence sequential information-acquisition behavior. The authors use a probability score as a proxy for model-specific information-seeking preferences, and compare the topic coverage, topic-switching, and redundant-information selection behavior of 11 models under a fixed candidate-idea pool and controlled feedback mechanism.

Critical strengths
  • The paper addresses a clearly motivated question: when information-seeking tasks are delegated to LLMs, the model's judgments about "what is worth asking" may affect the information ultimately collected. Separating candidate-information evaluation from question generation and respondent behavior also helps analyze model differences under controlled conditions.
Critical weaknesses
  • The applicability of the experimental conclusions to real-world open-ended tasks is unclear.​ The current simulation in Section 4 uses random feedback and predefined candidate-generation rules, which do not fully capture the behavioral patterns of human or actual respondents. Although this design helps isolate model differences, their impact on the generalizability of the findings is not established. A case study based on authentic interviews dataset, or prevailing simulated interviews datasets that addresses consistency of simulated respondent, would help validate the reliability and practical significance of these findings.
Questions for authors
  • Please address the weakness.
  • What concrete insight or guidance for selecting or designing elicitation agents follows from the cross-model comparison? In particular, how should practitioners interpret the breadth/depth differences observed in Section 5.2 in realistic open-ended information seeking tasks?
Reviewer_jhpkOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how different LLMs judge which information is worth pursuing in open-ended elicitation. It scores candidate information via P(Yes) across 11 open-weight models, then runs a simulation where models pick from candidate hypotheses while a coin-flip respondent confirms or not. It reports that models disagree substantially, that this produces different breadth–depth trajectories, and that larger models discount redundant information when instructed to.

Critical strengths

The distinction between open-ended elicitation and target-driven information seeking is well motivated, and reading P(Yes) from a forward pass is cleaner than parsing generated scores.

The ablations are honest: Table 7 and Appendix C report results that weaken the paper’s own claims.

Critical weaknesses
  • Breadth–depth and redundancy discounting are the same measurement, and both come from the prompt. Each candidate is scored under its own topic with only that topic’s history, so on-topic candidates carry history and off-topic ones don’t. Switching therefore reduces to how much history lowers a score—which is exactly what §5.3 measures separately.

Table 7 then shows the discount nearly vanishes without the “repeated information is less valuable” instruction. So the “model-specific preferences” that drive trajectories are largely differences in how strongly models follow one sentence you wrote, not emergent information-seeking behavior.

  • The simulation removes what makes elicitation open-ended. The motivation is unanticipated discovery, but the respondent is content-independent noise, candidates come from a fixed menu, and no questions are generated. Confirmation carries zero information, so the only history effect left is redundancy. What remains is a score-driven sampler, not elicitation.

More importantly I think:

  • No standard for what good elicitation is. With no gold reference, human judgment, or downstream task, the finding is that models disagree (ρ = .2–.6). That is expected and has no consequence: the reader cannot tell which breadth–depth profile is preferable or whether the differences matter. WA measures argument quality for a speech, not value of asking. So what are the gains from this paper?
  • The P(Yes) construct is fragile by your own tests. Adding an Uncertain option shifts scores by up to .36 (Appendix B); switching to Valuable gives ρ = .46 for Gemma and changes Ministral-8B’s trajectories drastically (Appendix C). 142 arguments, two per topic, is also a thin base for 10k evaluations.
Questions for authors

Please see above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.