- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes VISTA, a value-guided interactive framework designed to address skill routing under joint task-profile ambiguity. The framework models routing under uncertainty as a sequential "clarify-or-route" decision process, utilizing Expected Value of Perfect Information (EVPI) as a global stopping threshold and Expected Value of Sample Information (EVSI) to select the clarification question with the highest expected routing benefit. Furthermore, the authors construct a counterfactual benchmark dataset using real-world skills and user requests to validate VISTA's effectiveness in balancing routing accuracy and interaction cost across different LLM bases.
Critical strengths
- Precise Problem Identification: Through sampling analysis of real-world data from the SkillFeed platform, the authors astutely point out that "joint task-profile ambiguity" is a primary bottleneck in practical skill routing scenarios. This observation is highly meaningful and addresses the practical disconnect of existing one-shot routing methods, which implicitly assume that the input information is always sufficient.
- Principled Theoretical Formulation: The framework introduces EVPI and EVSI to quantify the benefits of questioning versus communication costs, providing a solid and interpretable decision-theoretic foundation for adaptive clarification. This approach of basing information acquisition on "downstream expected routing utility" effectively prevents meaningless and excessive user interactions.
Critical weaknesses
- Methodological Clarity and Reproducibility Issues: The paper severely lacks implementation details regarding the semantic estimator. For example, what are the specific LLM prompt templates used to generate the belief scores? How exactly does the system constrain the LLM to stably output a "finite answer space" and its corresponding prior conditional probabilities? The absence of these critical engineering details makes the framework extremely difficult to reproduce independently.
- Simulator Bias and Evaluation Fairness: The experiments rely entirely on an "offline evaluator-side simulator". This simulator makes deterministic selections from system-generated closed options based on predefined latent states. This experimental setup, which strictly toggles ambiguity on and off, fails to reflect the complexity of authentic open-ended dialogue and introduces an unrealistic advantage for the interactive framework when compared against traditional baseline methods.
- Lack of Real-world Usability Validation: Given the rigid simulator setup mentioned above, the usability of restricting user interactions strictly to model-generated multiple-choice options remains questionable in real-world scenarios. The paper lacks a real human evaluation or user study to demonstrate the actual user experience and practical acceptability of this multi-turn, mechanical clarification mechanism.
Questions for authors
- Implementation Details of the Semantic Estimator: Could you provide the exact LLM prompts and system instructions used to generate the belief scores and construct the "finite answer spaces"? Specifically, how do you mathematically or programmatically constrain the LLM to stably output valid conditional probabilities prior to observing the actual user response?
- Mitigating Simulator Bias and Baseline Fairness: The offline evaluator-side simulator operates on a rigid multiple-choice format by deterministically unlocking predefined latent states. How does this setup ensure a fair comparison against baselines like "Adaptive LLM Prompting"? Were the baselines heavily penalized by this rigid evaluation protocol designed around VISTA's specific assumptions?
- Real-world Usability & Human Evaluation: The current evaluation sidesteps open-ended natural language interactions by relying on the offline simulator. How does the system handle answer mapping and out-of-support recovery when a real user provides a free-form response? To address the concerns regarding practical usability, could you provide a small-scale human evaluation or user study during the rebuttal phase?
6 comments
Response to Reviewer BCbj (2/3)
Reviewer question Q2
Mitigating Simulator Bias and Baseline Fairness: The offline evaluator-side simulator operates on a rigid multiple-choice format by deterministically unlocking predefined latent states. How does this setup ensure a fair comparison against baselines like "Adaptive LLM Prompting"? Were the baselines heavily penalized by this rigid evaluation protocol designed around VISTA's specific assumptions?
Related weakness W2
Simulator Bias and Evaluation Fairness: The experiments rely entirely on an "offline evaluator-side simulator". This simulator makes deterministic selections from system-generated closed options based on predefined latent states. This experimental setup, which strictly toggles ambiguity on and off, fails to reflect the complexity of authentic open-ended dialogue and introduces an unrealistic advantage for the interactive framework when compared against traditional baseline methods.
Response to Q2 and W2
Thanks. We clarify the scope of the offline comparison and report a complementary human-interaction study with free-text supplementation.
Part 1: Controlled comparison and baseline fairness
As detailed in Appendices D.1–D.4, all interactive methods receive the same initial query, visible profile memory, candidate skills and ordering, and maximum clarification budget, under the same base-LLM configuration. Within this interface, Adaptive LLM Prompting retains its prompt-based decisions about whether to clarify or route and whether to ask about the task or profile.
The evaluator-side simulator responds after each method selects its question, choosing among that question’s options according to the hidden task–profile state. Neither the hidden state nor the reference routing labels are directly available to the methods. These controls provide comparable information access and interaction budgets for evaluating policies within the structured interface. A shared interface can still constrain methods differently, which motivates examining performance when the restriction on user responses is relaxed.
Part 2: Human-interaction evaluation with free-text supplementation
To examine performance with a more flexible response interface, we evaluated 30 cases (15 clarification-needed and 15 immediate-route) with 60 sessions across VISTA and Adaptive LLM Prompting. Each case was evaluated once per method. Both methods shared the base LLM, candidates, initial observations, and interaction budgets. Participants answered from case briefs by selecting an accurate option or freely supplying natural-language supplements. This response flexibility was available to both methods, allowing participants to provide information beyond the proposed options.
Table 1. Option coverage, supplementation, and routing accuracy.
| Metric | VISTA (Ours) | Adaptive LLM |
|---|---|---|
| Cases | 30 | 30 |
| Actual clarification exchanges | 18 | 22 |
| Accurate substantive-option coverage | 15/18 (83.3%) | 16/22 (72.7%) |
| Exchanges requiring free-text supplementation | 3/18 (16.7%) | 6/22 (27.3%) |
| Average clarification count per case | 0.600 | 0.733 |
| Final preferred-route accuracy (PRA) | 18/30 (60.0%) | 17/30 (56.7%) |
Across these cases, VISTA used 18.2% fewer clarification exchanges (18 versus 22), with observed PRA of 60.0% versus 56.7% and 10.6 percentage points higher substantive-option coverage. These results show that VISTA’s lower clarification count was also observed when participants could answer beyond the proposed options.
Part 3: Rationale for closed answer options and extension to free text
Closed answer options serve computational and usability purposes. Finite answer spaces make information-value calculations tractable, as in Dong et al.’s framework using multiple-choice or yes/no questions [1]. SAGE also uses structured uncertainty to guide clarification [2]. Concise options can reduce typing effort: in Zamani et al.’s user studies, participants found relevant clarification questions with candidate answers convenient and time-saving [3].
These considerations motivate the structured interface, while the supplementary study examines performance with more flexible user responses. Our response to Q3 details answer mapping and hypothesis revision for extending VISTA to free-text responses.
References
[1] Dong, Y. R., et al. (2026). Value of Information: A Framework for Human–Agent Communication. ACL, pp. 42879–42896.
[2] Suri, M., et al. (2026). Structured Uncertainty Guided Clarification for LLM Agents. Findings of ACL, pp. 40811–40838.
[3] Zamani, H., et al. (2020). Generating Clarifying Questions for Information Retrieval. The Web Conference, pp. 418–428.
Response to Reviewer BCbj (3/3)
Reviewer question Q3
Real-world Usability & Human Evaluation: The current evaluation sidesteps open-ended natural language interactions by relying on the offline simulator. How does the system handle answer mapping and out-of-support recovery when a real user provides a free-form response? To address the concerns regarding practical usability, could you provide a small-scale human evaluation or user study during the rebuttal phase?
Related weakness W3
Lack of Real-world Usability Validation: Given the rigid simulator setup mentioned above, the usability of restricting user interactions strictly to model-generated multiple-choice options remains questionable in real-world scenarios. The paper lacks a real human evaluation or user study to demonstrate the actual user experience and practical acceptability of this multi-turn, mechanical clarification mechanism.
Response to Q3 and W3
Thanks. We implemented a lightweight extension for free-form answers and conducted a small user study to assess practical usability.
Part 1: Lightweight extension to free-form responses
VISTA can be readily extended to free-form responses by adding a lightweight LLM semantic adapter. The adapter extracts task requirements and profile constraints from the user’s response and maps them to VISTA’s task–profile representation. Ambiguous interpretations can be confirmed with the user.
Information outside the current support can be incorporated by reapplying VISTA’s existing adaptive support-expansion mechanism to the newly extracted information. VISTA then updates the affected beliefs, utility estimates, and answer likelihoods and continues its existing clarify-or-route procedure. The additional component is therefore a semantic interface for free text, while support adaptation and subsequent decision-making reuse VISTA’s existing mechanisms.
Part 2: User study on practical usability
We recruited 20 users to try VISTA on their own tasks. Afterward, they completed a short questionnaire on a five-point scale (1 = strongly disagree; 5 = strongly agree) and could provide additional comments.
Table 2. Questionnaire results for 20 users.
| Item | Statement | Mean score |
|---|---|---|
| Clarity | The clarification questions were easy to understand. | 4.65/5 |
| Ease of expression | I could easily express my needs through options or free text. | 4.70/5 |
| Understanding | The system correctly understood my answers and corrections. | 4.75/5 |
| Interaction burden | The clarification process required little effort. | 4.55/5 |
| Overall satisfaction | I would be willing to use this interaction approach again. | 4.60/5 |
We also asked, “What worked well, and what was difficult to express?” Example comments included:
- User A: “The system helped identify the skill that better matched my needs, reducing the time I spent comparing options.”
- User B: “The options helped me answer quickly. When none matched, I could explain what I wanted in my own words.”
- User C: “I liked being able to add details, but I would prefer the system to confirm its understanding when my answer contains several conditions.”
These responses indicate that participants found VISTA’s combination of options and free text clear, convenient, and requiring little effort. The feedback also identifies confirmation of interpretations involving multiple conditions as a useful refinement.
Response to Reviewer BCbj (1/3)
Reviewer question Q1
Related weakness W1
Response to Q1 and W1
Thanks. The complete implementation code and prompts have been provided in our supplementary material (see code/vista/prompts.py). We will also include these prompts in the revised manuscript. Specifically:
Part 1: Exact LLM prompts and system instructions
Shared system instruction
Belief estimation
Input fields:
query,profile_memory,hypotheses,history.Finite answer-space construction
Input fields:
query,profile_memory,candidate_cards,hypotheses,utility_view.Conditional answer likelihoods
Input fields:
question,required_hypothesis_ids,hypotheses,history.Part 2: Programmatic constraints on probability validity
All likelihoods are estimated before observing the user’s answer, using only the retained hypotheses, question, and prior interaction history. The validator requires exactly one row per hypothesis covering every legal answer ID, with finite, nonnegative values and positive row sums. Accepted rows are normalized to sum to one; invalid outputs trigger bounded retries. This separates LLM estimation from numerical validation, ensuring valid probability distributions for subsequent Bayesian updates. Eq. (6) in section 4.2 specifies the predictive answer probabilities and corresponding Bayesian posterior updates.