- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies a ranking and selection problem. Each of the candidates will output an interval, the paper aim to identify those candidates with the largest expected median of output intervals. If expected medians are tied, pick the one with the smallest expected interval length.
Critical strengths
I think that, for the problem formulated in this paper, the authors provide a solution along with some theoretical guarantees. The distributional assumptions are also relatively weak, and there is a fairly clear correspondence between the algorithmic mechanisms and the proofs.
Critical weaknesses
1.I think the motivation behind the problem formulation is really weak. When does an LLM actually output an interval? Personally, I have not seen this in practice.
-
The ranking criterion also seems somewhat strange. Have these intervals been calibrated? On what basis should we compare their midpoints first and then their lengths? What is the rationale behind this criterion? Suppose an interval is extremely wide, but its midpoint is only slightly higher than that of another, much narrower interval. Would we really want to select the wider one? I do not quite understand this choice, and I do not think it is well justified in the paper.
-
I feel that this paper has very little to do with LLMs. The LLM framing feels like a gimmick and is really a distraction.
Questions for authors
Could the authors further justify their ranking criterion? In particular, what is the rationale for comparing the midpoints first and then the interval lengths?
4 comments
2. Why compare midpoint first and radius second?
We agree that two distinct arguments are needed: why width carries useful information, and why a decision maker would give midpoint priority. Neither follows automatically from the other. Also, our primary quantity is the expected midpoint, not the median of a response distribution.
The intervals are LLM responses, not calibrated uncertainty intervals. In our motivating example, they are reported price ranges, not 95% confidence or prediction intervals. Our reason for retaining width comes from research on valuation ranges. Wang et al. (Journal of Marketing Research, 2007) used willingness-to-pay (WTP) intervals to represent uncertainty about consumers' preferences and product knowledge. Maier et al. (Marketing Letters, 2015) found that consumers who were more certain reported narrower WTP ranges. These studies suggest that both the price level and the precision of a reported valuation can be useful. When LLMs are used to simulate consumer responses, their price ranges may similarly express uncertainty about valuations. A single price can also be treated as an interval with zero width, expressing certainty to some extent. Thus we retain interval width in our selection problem. (To make this motivation more clear, we also revise the introduction in our main paper.)
The priority reflects a screening preference, not a rule for all interval data. A decision maker may first seek the highest expected valuation and then prefer narrower responses for which responses are more precise, among alternatives with the same expected valuation. Selecting by midpoint alone ignores response width, while selecting by width alone may favor an alternative with a low valuation. Lexicographic selection follows this order directly, without choosing a weight to trade off valuation against width.
This lexicographic ordering also appears in earlier work. The interval-ordering rule reviewed by Karmakar and Bhunia (Reliable Computing, 2012) favors larger centers and, when centers are equal, smaller radii. Kohli and Jedidi (Marketing Science, 2007) studied lexicographic preference models using data on personal-computer choices. Skalse et al. (IJCAI, 2022) studied objectives with different priorities in reinforcement learning. These studies support the use of explicit priorities in decision problems. Other preferences may call for different selection rules, but adapting our algorithms to those rules would require further analysis.
Your example motivates allowing an explicit midpoint tolerance. Thank you for this example. We have extended our algorithms to allow a decision maker to specify a midpoint tolerance. The extended algorithms and their theoretical guarantees are presented in the new Appendix G. For an externally specified tolerance , the selection target is
G_\tau=\{k:\mu_k^m\ge\mu_*^m-\tau\},\ k_\tau^*=\arg\min_{k\in G_\tau}\mu_k^r.$$ Here, $\mu_k^m$ and $\mu_k^r$ are the expected midpoint and expected radius. A narrower alternative whose expected midpoint is slightly lower can now be selected, provided the loss is at most $\tau$. Setting $\tau=0$ recovers the original target in the main paper. The extension changes only the candidate-set rules. F-IRIS uses an empirical midpoint allowance $\tau+\epsilon$, and A-IRIS retains candidates satisfying $U_k+\tau\ge\max_iL_i$, where $(L_i,U_i)$ are midpoint confidence bounds. The interlaced sampling and modal recommendation remain unchanged. Setting $\tau=0$ recovers the original algorithms in the main paper. Appendix G establishes linear-budget sample optimality and prescribed-accuracy consistency under positive separation between its boundary and excluded alternatives. This extension addresses the trade-off raised in your example, that is, the decision maker can specify how much expected valuation they are willing to give up for a smaller expected radius.3. What is the connection to LLMs?
LLMs provide an application of our methods. Our study considers how individuals and businesses can use repeated LLM responses to choose among candidate products or designs under a limited query budget. With stochastic decoding, the same query can produce different answers, so repeated queries help assess the model's response to each candidate. These answers may also contain ranges rather than single values. Our neutral-prompt experiment shows that such ranges occur even when the prompt does not explicitly request them. This motivates studying selection with stochastic interval-valued responses.
Why LLMs produce ranges, and when they are more likely to do so, remain open questions. One possible explanation is that their training data contain numerical ranges, which the models learn to generate in similar contexts. This is a possible explanation rather than a mechanism established by our study. Prompt wording also affects the form of the response, as our experiments show. Understanding how prompt design affects interval responses and subsequent selection decisions is a direction for future work.
Thank you for your response. However, my main concerns remain. The motivation for the problem and the proposed selection objective is still unclear to me. In particular, it remains unclear what these intervals actually mean. A shorter interval does not necessarily indicate a more reliable valuation, and LLMs can be very confidently wrong.
In my view, this is a fundamental concern about the paper. I do not think this concern can be resolved through a simple modification to the selection rule. While I appreciate the authors’ efforts to address my comments, I remain unconvinced and therefore maintain my original score of Weak Reject.
Thank you for your reviews and comments. The comments identify three questions that our motivation should answer more clearly: whether interval responses occur in practice, why midpoint and radius should be ordered this way, and what role LLMs play in the contribution. We address them in the order of the comments, combining Critical Weakness 2 and the corresponding Question for Authors.
1. Do LLMs actually return intervals?
We hope to answer this question through both prior literature and empirical evidence. We find the work Using LLMs for Market Research by Brand, Israeli, and Ngwe has reported that asking GPT for maximum willingness to pay can produce an explanatory answer or a price range, whereas explicitly requesting a single price leads to a point response. This observation suggests that the response format depends on how the question is posed: when no particular format is specified, a model may express its valuation as a range.
In our original laptop-screening experiment, 53.68%-63.00% of valid responses are nondegenerate intervals (please see the table below). Each configuration contributes 4,000 valid responses under a prompt that permits both formats and favors ranges when multiple prices seem plausible. We have added these results to Appendix D.1.
Table 1. Range occurrence under the original prompt.
Note that the original prompt required either a price or a range output and expressed a preference for a range when several prices were plausible. To check whether the restriction was necessary for intervals to appear, we conducted an additional experiment with a neutral question without any output format restriction:
This additional experiment contains the attributes of the candidate laptop in our main paper. We neither request an interval nor require a scalar answer, thus LLMs output in a text format with numbers. Using
Llama-3.2-3B-Instruct, we collected 200 responses per configuration across four design spaces, for 943,200 responses in total. The results are shown in Table 2.Table 2. Range occurrence under the neutral prompt.
Even under a neutral prompt that does not request an interval, 80.72%-82.89% of responses contain at least one explicit price range across the four settings. Thus, price ranges occur frequently in this task without explicit instructions to provide them. For a concrete example, one complete response in the K=360 experiment was:
Thus, an explicit request for a range is not necessary for an LLM to produce a WTP interval in this setting. The original prompt avoids text responses and thus simplifies the numeric analysis.
In conclusion, through both literature and experiments, we find the response format depends on the prompt and in our WTP problem the interval output is possible, which motivates our study.