acceptodds
Under review as a conference paper at ICLR 2027

What to Ask, When to Route: Value-Guided Skill Routing under Joint Task–Profile Uncertainty

20 views

Abstract

Skill routing for LLM-based agents is typically performed from a single observed request, implicitly assuming that the available information is sufficient to identify the appropriate skill. In practice, the intended task, relevant user profile conditions, or both may be underspecified, making one-shot routing unreliable. We formulate personalized skill routing under joint task–profile uncertainty as a sequential clarify-or-route problem and propose VISTA, a training-free framework for resolving such decision-relevant uncertainty. VISTA identifies candidate-distinguishing task and profile factors, constructs a compact candidate-conditioned joint belief over plausible routing states, and evaluates task clarification, profile clarification, and immediate routing under a shared downstream utility. Based on this belief, value-of-information criteria determine whether clarification is worthwhile and which question offers the greatest expected routing benefit. We further construct a controlled counterfactual benchmark seeded by real user queries and profiles, comprising 213 cases and 2,160 simulated interactive episodes across task-only, profile-only, and joint ambiguity. The benchmark also includes explicit clarification-stopping controls and independent human validation. Experiments on two base LLMs show that VISTA improves cost-aware routing performance by 13.8% on average over interactive baselines while reducing clarification turns by 40.1%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Skill Routing, Clarification, Value of Information, Task–Profile Uncertainty

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_BCbjOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes VISTA, a value-guided interactive framework designed to address skill routing under joint task-profile ambiguity. The framework models routing under uncertainty as a sequential "clarify-or-route" decision process, utilizing Expected Value of Perfect Information (EVPI) as a global stopping threshold and Expected Value of Sample Information (EVSI) to select the clarification question with the highest expected routing benefit. Furthermore, the authors construct a counterfactual benchmark dataset using real-world skills and user requests to validate VISTA's effectiveness in balancing routing accuracy and interaction cost across different LLM bases.

Critical strengths
  • Precise Problem Identification: Through sampling analysis of real-world data from the SkillFeed platform, the authors astutely point out that "joint task-profile ambiguity" is a primary bottleneck in practical skill routing scenarios. This observation is highly meaningful and addresses the practical disconnect of existing one-shot routing methods, which implicitly assume that the input information is always sufficient.
  • Principled Theoretical Formulation: The framework introduces EVPI and EVSI to quantify the benefits of questioning versus communication costs, providing a solid and interpretable decision-theoretic foundation for adaptive clarification. This approach of basing information acquisition on "downstream expected routing utility" effectively prevents meaningless and excessive user interactions.
Critical weaknesses
  • Methodological Clarity and Reproducibility Issues: The paper severely lacks implementation details regarding the semantic estimator. For example, what are the specific LLM prompt templates used to generate the belief scores? How exactly does the system constrain the LLM to stably output a "finite answer space" and its corresponding prior conditional probabilities? The absence of these critical engineering details makes the framework extremely difficult to reproduce independently.
  • Simulator Bias and Evaluation Fairness: The experiments rely entirely on an "offline evaluator-side simulator". This simulator makes deterministic selections from system-generated closed options based on predefined latent states. This experimental setup, which strictly toggles ambiguity on and off, fails to reflect the complexity of authentic open-ended dialogue and introduces an unrealistic advantage for the interactive framework when compared against traditional baseline methods.
  • Lack of Real-world Usability Validation: Given the rigid simulator setup mentioned above, the usability of restricting user interactions strictly to model-generated multiple-choice options remains questionable in real-world scenarios. The paper lacks a real human evaluation or user study to demonstrate the actual user experience and practical acceptability of this multi-turn, mechanical clarification mechanism.
Questions for authors
  1. Implementation Details of the Semantic Estimator: Could you provide the exact LLM prompts and system instructions used to generate the belief scores and construct the "finite answer spaces"? Specifically, how do you mathematically or programmatically constrain the LLM to stably output valid conditional probabilities prior to observing the actual user response?
  2. Mitigating Simulator Bias and Baseline Fairness: The offline evaluator-side simulator operates on a rigid multiple-choice format by deterministically unlocking predefined latent states. How does this setup ensure a fair comparison against baselines like "Adaptive LLM Prompting"? Were the baselines heavily penalized by this rigid evaluation protocol designed around VISTA's specific assumptions?
  3. Real-world Usability & Human Evaluation: The current evaluation sidesteps open-ended natural language interactions by relying on the offline simulator. How does the system handle answer mapping and out-of-support recovery when a real user provides a free-form response? To address the concerns regarding practical usability, could you provide a small-scale human evaluation or user study during the rebuttal phase?
6 comments
AuthorsOpenReview ↗

Response to Reviewer BCbj (1/3)

Reviewer question Q1

Implementation Details of the Semantic Estimator: Could you provide the exact LLM prompts and system instructions used to generate the belief scores and construct the "finite answer spaces"? Specifically, how do you mathematically or programmatically constrain the LLM to stably output valid conditional probabilities prior to observing the actual user response?

Related weakness W1

Methodological Clarity and Reproducibility Issues: The paper severely lacks implementation details regarding the semantic estimator. For example, what are the specific LLM prompt templates used to generate the belief scores? How exactly does the system constrain the LLM to stably output a "finite answer space" and its corresponding prior conditional probabilities? The absence of these critical engineering details makes the framework extremely difficult to reproduce independently.

Response to Q1 and W1

Thanks. The complete implementation code and prompts have been provided in our supplementary material (see code/vista/prompts.py). We will also include these prompts in the revised manuscript. Specifically:

Part 1: Exact LLM prompts and system instructions

Shared system instruction

You are a component inside VISTA. Return exactly one valid JSON object, with no Markdown and no commentary. Follow the requested fields exactly.

Belief estimation

Estimate one probability vector over the listed hypotheses and a residual other_probability. Use the observation and history only. Output {probabilities:{hypothesis_id:number},other_probability:number}; numbers must be nonnegative and sum to one including other_probability.
INPUT=<INPUT_JSON>

Input fields: query, profile_memory, hypotheses, history.

Finite answer-space construction

Generate up to six concise decision-relevant clarification questions. Each question must be about Task or Profile, have 2-3 mutually exclusive answer options plus an optional unsure option, and distinguish at least two hypotheses. Output {questions:[{question_id,side,text,answer_options:[{answer_id,text}],target_hypothesis_ids}]}. Do not mention hidden labels or evaluator fields.
INPUT=<INPUT_JSON>

Input fields: query, profile_memory, candidate_cards, hypotheses, utility_view.

Conditional answer likelihoods

For this question, estimate one answer-probability distribution for every required hypothesis. Output exactly one row for every ID in required_hypothesis_ids and no other rows: {rows:[{hypothesis_id,probabilities:{answer_id:number}}]}. Each row must cover every answer option, including unsure when present, and sum to one. The question's target contrast is not an output filter; include hypotheses that are not target states.
INPUT=<INPUT_JSON>

Input fields: question, required_hypothesis_ids, hypotheses, history.

Part 2: Programmatic constraints on probability validity

All likelihoods are estimated before observing the user’s answer, using only the retained hypotheses, question, and prior interaction history. The validator requires exactly one row per hypothesis covering every legal answer ID, with finite, nonnegative values and positive row sums. Accepted rows are normalized to sum to one; invalid outputs trigger bounded retries. This separates LLM estimation from numerical validation, ensuring valid probability distributions for subsequent Bayesian updates. Eq. (6) in section 4.2 specifies the predictive answer probabilities and corresponding Bayesian posterior updates.

AuthorsOpenReview ↗

Response to Reviewer BCbj (2/3)

Reviewer question Q2

Mitigating Simulator Bias and Baseline Fairness: The offline evaluator-side simulator operates on a rigid multiple-choice format by deterministically unlocking predefined latent states. How does this setup ensure a fair comparison against baselines like "Adaptive LLM Prompting"? Were the baselines heavily penalized by this rigid evaluation protocol designed around VISTA's specific assumptions?

Related weakness W2

Simulator Bias and Evaluation Fairness: The experiments rely entirely on an "offline evaluator-side simulator". This simulator makes deterministic selections from system-generated closed options based on predefined latent states. This experimental setup, which strictly toggles ambiguity on and off, fails to reflect the complexity of authentic open-ended dialogue and introduces an unrealistic advantage for the interactive framework when compared against traditional baseline methods.

Response to Q2 and W2

Thanks. We clarify the scope of the offline comparison and report a complementary human-interaction study with free-text supplementation.

Part 1: Controlled comparison and baseline fairness

As detailed in Appendices D.1–D.4, all interactive methods receive the same initial query, visible profile memory, candidate skills and ordering, and maximum clarification budget, under the same base-LLM configuration. Within this interface, Adaptive LLM Prompting retains its prompt-based decisions about whether to clarify or route and whether to ask about the task or profile.

The evaluator-side simulator responds after each method selects its question, choosing among that question’s options according to the hidden task–profile state. Neither the hidden state nor the reference routing labels are directly available to the methods. These controls provide comparable information access and interaction budgets for evaluating policies within the structured interface. A shared interface can still constrain methods differently, which motivates examining performance when the restriction on user responses is relaxed.

Part 2: Human-interaction evaluation with free-text supplementation

To examine performance with a more flexible response interface, we evaluated 30 cases (15 clarification-needed and 15 immediate-route) with 60 sessions across VISTA and Adaptive LLM Prompting. Each case was evaluated once per method. Both methods shared the base LLM, candidates, initial observations, and interaction budgets. Participants answered from case briefs by selecting an accurate option or freely supplying natural-language supplements. This response flexibility was available to both methods, allowing participants to provide information beyond the proposed options.

Table 1. Option coverage, supplementation, and routing accuracy.

MetricVISTA (Ours)Adaptive LLM
Cases3030
Actual clarification exchanges1822
Accurate substantive-option coverage15/18 (83.3%)16/22 (72.7%)
Exchanges requiring free-text supplementation3/18 (16.7%)6/22 (27.3%)
Average clarification count per case0.6000.733
Final preferred-route accuracy (PRA)18/30 (60.0%)17/30 (56.7%)

Across these cases, VISTA used 18.2% fewer clarification exchanges (18 versus 22), with observed PRA of 60.0% versus 56.7% and 10.6 percentage points higher substantive-option coverage. These results show that VISTA’s lower clarification count was also observed when participants could answer beyond the proposed options.

Part 3: Rationale for closed answer options and extension to free text

Closed answer options serve computational and usability purposes. Finite answer spaces make information-value calculations tractable, as in Dong et al.’s framework using multiple-choice or yes/no questions [1]. SAGE also uses structured uncertainty to guide clarification [2]. Concise options can reduce typing effort: in Zamani et al.’s user studies, participants found relevant clarification questions with candidate answers convenient and time-saving [3].

These considerations motivate the structured interface, while the supplementary study examines performance with more flexible user responses. Our response to Q3 details answer mapping and hypothesis revision for extending VISTA to free-text responses.

References

[1] Dong, Y. R., et al. (2026). Value of Information: A Framework for Human–Agent Communication. ACL, pp. 42879–42896.

[2] Suri, M., et al. (2026). Structured Uncertainty Guided Clarification for LLM Agents. Findings of ACL, pp. 40811–40838.

[3] Zamani, H., et al. (2020). Generating Clarifying Questions for Information Retrieval. The Web Conference, pp. 418–428.

AuthorsOpenReview ↗

Response to Reviewer BCbj (3/3)

Reviewer question Q3

Real-world Usability & Human Evaluation: The current evaluation sidesteps open-ended natural language interactions by relying on the offline simulator. How does the system handle answer mapping and out-of-support recovery when a real user provides a free-form response? To address the concerns regarding practical usability, could you provide a small-scale human evaluation or user study during the rebuttal phase?

Related weakness W3

Lack of Real-world Usability Validation: Given the rigid simulator setup mentioned above, the usability of restricting user interactions strictly to model-generated multiple-choice options remains questionable in real-world scenarios. The paper lacks a real human evaluation or user study to demonstrate the actual user experience and practical acceptability of this multi-turn, mechanical clarification mechanism.

Response to Q3 and W3

Thanks. We implemented a lightweight extension for free-form answers and conducted a small user study to assess practical usability.

Part 1: Lightweight extension to free-form responses

VISTA can be readily extended to free-form responses by adding a lightweight LLM semantic adapter. The adapter extracts task requirements and profile constraints from the user’s response and maps them to VISTA’s task–profile representation. Ambiguous interpretations can be confirmed with the user.

Information outside the current support can be incorporated by reapplying VISTA’s existing adaptive support-expansion mechanism to the newly extracted information. VISTA then updates the affected beliefs, utility estimates, and answer likelihoods and continues its existing clarify-or-route procedure. The additional component is therefore a semantic interface for free text, while support adaptation and subsequent decision-making reuse VISTA’s existing mechanisms.

Part 2: User study on practical usability

We recruited 20 users to try VISTA on their own tasks. Afterward, they completed a short questionnaire on a five-point scale (1 = strongly disagree; 5 = strongly agree) and could provide additional comments.

Table 2. Questionnaire results for 20 users.

ItemStatementMean score
ClarityThe clarification questions were easy to understand.4.65/5
Ease of expressionI could easily express my needs through options or free text.4.70/5
UnderstandingThe system correctly understood my answers and corrections.4.75/5
Interaction burdenThe clarification process required little effort.4.55/5
Overall satisfactionI would be willing to use this interaction approach again.4.60/5

We also asked, “What worked well, and what was difficult to express?” Example comments included:

  • User A: “The system helped identify the skill that better matched my needs, reducing the time I spent comparing options.”
  • User B: “The options helped me answer quickly. When none matched, I could explain what I wanted in my own words.”
  • User C: “I liked being able to add details, but I would prefer the system to confirm its understanding when my answer contains several conditions.”

These responses indicate that participants found VISTA’s combination of options and free text clear, convenient, and requiring little effort. The feedback also identifies confirmation of interpretations involving multiple conditions as a useful refinement.

Reviewer_hQwjOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper frames personalized skill routing under task and profile ambiguity as a clarify-or-route problem and proposes VISTA, a training-free method that builds a compact joint task–profile belief and uses EVPI/EVSI to decide whether to ask, what to ask, and when to stop.

Critical strengths

The core idea that to clarify only when the answer could change the chosen skill, not whenever the belief is uncertain is the right principle. the VISTA vs VISTA-ER comparison isolates it cleanly.

Critical weaknesses

The most obvious problem is about the evaluation.

  • The evaluation is circular. Gold labels come from deepseek-v4-flash scoring task–skill and profile–skill compatibility separately, combined with α=0.5 via Eq. 1. VISTA’s router estimates the same two quantities with the same model (in one of the two conditions) and combines them with the same equation and the same α.

The benchmark is then filtered to keep only cases where this additive utility is stable. So VISTA’s utility estimate is structurally a copy of the label-generating function, and baselines like Generic VoI and SAGE that don’t use the R_T/R_P decomposition are disadvantaged by construction.

I can’t tell whether the 13.8% gain reflects better clarification or just matching the labeler. The VISTA vs VISTA-ER comparison survives this (same representation, different criterion); the cross-method comparisons don’t. A fix: label with a different model family and a non-additive utility, or at least show results hold when the router’s α and judge model differ from the benchmark’s.

  • The benchmark is too small for the claims, and this dataset cannot prove the utility of method. 2,160 episodes come from only 213 cases (145 in test), and ~30 joint cases contribute 80% of test episodes since each expands to 16 correlated variants.
Questions for authors

Please see above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer hQwj (1/2)

Related weakness W1

The most obvious problem is about the evaluation.

The evaluation is circular. Gold labels come from deepseek-v4-flash scoring task–skill and profile–skill compatibility separately, combined with α=0.5 via Eq. 1. VISTA’s router estimates the same two quantities with the same model (in one of the two conditions) and combines them with the same equation and the same α. The benchmark is then filtered to keep only cases where this additive utility is stable. So VISTA’s utility estimate is structurally a copy of the label-generating function, and baselines like Generic VoI and SAGE that don’t use the R_T/R_P decomposition are disadvantaged by construction.

I can’t tell whether the 13.8% gain reflects better clarification or just matching the labeler. The VISTA vs VISTA-ER comparison survives this (same representation, different criterion); the cross-method comparisons don’t. A fix: label with a different model family and a non-additive utility, or at least show results hold when the router’s α and judge model differ from the benchmark’s.

Response to W1

Thanks. We clarify the rationale behind our evaluation design and the additional checks needed to address evaluator dependence.

Part 1: Original evaluation design and the role of the controlled comparison

  1. A shared objective with separately estimated utilities. The benchmark derives (U†U^\dagger) from separate task–skill and profile–skill judgments. VISTA estimates (U^\widehat U) from runtime-visible information, without access to benchmark scores or labels. The shared objective provides a consistent measure of routing outcomes, while the evaluator and router produce their estimates separately.
  2. Filtering for route stability. For joint cases, the original benchmark requires the preferred route to remain unchanged across α∈{0.4,0.5,0.6}\alpha\in\{0.4,0.5,0.6\}. This reduces dependence on a particular weighting of task and profile compatibility within that interval.
  3. A controlled test of the clarification criterion. VISTA and VISTA-ER share the joint representation and question space, allowing their comparison to assess value-guided versus entropy-based clarification. The additional evaluations below test whether cross-method gains also persist when the evaluation references and utility construction change.

Part 2: Independent Evaluation and Utility-Mismatch Tests

We conducted three complementary checks:

  1. Human-reference rescoring. We rescored frozen routing predictions against independently adjudicated human labels on the 150-case audit cohort, comprising 44 development cases and 106 test cases. Table 1 reports results for the full cohort.
  2. A different model family with holistic judgments. We evaluated 300 new cases that were not subjected to the original (α\alpha)-stability filter. A Qwen-3.8-flash judge, from a different family than the original DeepSeek evaluator, directly assigns holistic utilities in [0,1][0,1] and preferred/acceptable skills.
  3. Router-weight mismatch. We varied VISTA’s internal α∈{0.25,0.50,0.75}\alpha\in\{0.25,0.50,0.75\} while keeping the holistic judge’s utilities and labels fixed. This separates the router’s weighting choice from the evaluation reference.

Table 1. Evaluation against independent human annotations (150 audited cases).

MethodNR↑FAA (%)↑PRA (%)↑Avg. Q↓
Direct Router0.3687-39.600.000
Adaptive LLM0.347447.8742.800.701
SAGE0.380451.5349.600.941
Generic VoI0.389352.3350.130.848
VISTA-ER0.406953.4054.531.035
VISTA0.413056.0753.930.557

Table 2. Evaluation under the independent Qwen-3.8-flash holistic judge on 300 new cases.

MethodNR↑FAA (%)↑PRA (%)↑Avg. Q↓
Direct Router0.4246-43.050.000
Adaptive LLM0.415048.2046.140.680
SAGE0.460551.2452.510.833
Generic VoI0.469550.6553.650.830
VISTA-ER0.497851.8758.090.991
VISTA0.510558.5157.270.563

Table 3. Sensitivity of VISTA's internal router α\alpha evaluated against the fixed holistic judge (300 new cases).

VISTA router αNR↑FAA (%)↑PRA (%)↑Avg. Q↓
0.250.496659.2255.450.543
0.500.510558.5157.270.563
0.750.498657.8655.960.586

Compared with Generic VoI, VISTA achieves 6.1% and 8.7% higher NR, with 34.3% and 32.2% fewer clarification exchanges, under the human and holistic-judge references, respectively. Compared with VISTA-ER, VISTA achieves higher NR with 43.2%–46.2% fewer exchanges. Across the tested router weights, NR decreases by less than 2.8% from (α=0.50\alpha=0.50) and remains above Generic VoI.

These results show that VISTA’s efficiency advantage persists under alternative evaluation references and router utility weights, providing evidence beyond the original label-generating model and additive utility construction.

AuthorsOpenReview ↗

Response to Reviewer hQwj (2/2)

Related weakness W2

The benchmark is too small for the claims, and this dataset cannot prove the utility of method. 2,160 episodes come from only 213 cases (145 in test), and ~30 joint cases contribute 80% of test episodes since each expands to 16 correlated variants.

Response to W2

Thanks. The original benchmark already provides systematic coverage of skill routing, comprising 213 cases from 51 source families, spanning 86 distinct task names and 381 candidate skills. These cases cover task-only, profile-only, and joint ambiguity and include 75 stopping controls, supporting evaluation of both useful clarification and appropriate stopping.

To strengthen this evidence further, we added and evaluated 300 cases from new source groups disjoint from the original families, expanding the test set from 145 to 445 cases. The additional results show that VISTA’s routing-efficiency advantage persists across all three ambiguity tracks and the expanded test set.

Part 1: Coverage of the Original Benchmark

  1. Application coverage.

Table 4. Application coverage of the original benchmark.

Application areaSource familiesDistinct task namesCases
Software development and cybersecurity182444
Scientific research and engineering5713
Biomedical and clinical analysis91230
Content creation and marketing81543
Business operations and product design61448
Education and learning41229
Legal documentation126
Total5186213
  1. Complementary routing scenarios. The benchmark includes 46 task-only, 58 profile-only, and 109 joint cases, with 75 stopping controls, testing both useful clarification and appropriate stopping.

  2. Family-separated evaluation. Source families remain intact across the 68-case development and 145-case test splits. The test set contains 73 joint cases, representing 80.2% of episodes but 50.3% of cases.

Part 2: Further Validation on 300 Additional Cases

Table 5. Summary of benchmark expansion.

Benchmark cohortCasesEpisodes
Original benchmark2132,160
Original test split1451,456
Newly added test cases3002,424
Combined test split4453,880
Full benchmark release5134,584

Table 6. Results on the 300 added cases.

TrackCasesVISTA NRGeneric VoI NRΔNRVISTA PRA (%)Generic VoI PRA (%)
task-only980.52010.4732+0.046959.2955.05
profile-only1000.56310.5263+0.036861.9558.65
joint1020.44950.4103+0.039350.7647.41

Table 7. Aggregate results across 445 test cases

MethodNR ↑FAA (%) ↑PRA (%) ↑Avg. Q ↓
Direct Router0.4158--42.860.000
Adaptive LLM0.405247.3846.370.686
SAGE0.440750.8653.070.844
Generic VoI0.448354.1652.910.832
VISTA-ER0.470257.2756.850.988
VISTA0.484060.8755.840.569

These results demonstrate that VISTA’s routing-efficiency advantage can extend across ambiguity types and a broader collection of source cases.

Dear Reviewer hQwj,

I hope this message finds you well. As the discussion period is nearing its end (with less than 3 days remaining), I wanted to ensure we have addressed all your concerns satisfactorily.

We have submitted our rebuttal and would greatly appreciate your feedback. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper.

Reviewer_RRiiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VISTA addresses skill routing under joint task and user-profile uncertainty. It constructs a compact joint belief and uses information value to decide whether to clarify or route. Evaluation includes a counterfactual benchmark, two base models, ablations, and human auditing.

Critical strengths

The framework coherently connects clarification decisions with downstream routing value.

The evaluation includes repeated runs, stopping controls, and a transparent human audit.

Critical weaknesses

Both human auditors disagree with 22 of 150 benchmark preferred-route judgments, but robustness of the method ranking to independent labeling remains untested.

Comparisons change representation, utility design, and question construction together, limiting attribution. Coverage and calibration of the retained belief are also insufficiently validated.

Value-guided clarification and stopping are established principles, so VISTA’s methodological novelty rests mainly on candidate-conditioned joint task–profile modeling. However, the holistic baseline also differs in utility decomposition, feasibility constraints, and question construction (Appendix D.4), leaving the contribution of joint modeling insufficiently isolated from the adaptation of existing VOI methods.

Questions for authors

Does VISTA retain its advantage under independently adjudicated reference labels?

With utilities and question generation fixed, how much improvement comes from joint modeling?

How often does support truncation omit relevant states, and how does this affect stopping decisions?

What measurable benefit comes from preserving task–profile dependence when utilities, question generation, and hypothesis budgets are controlled?

7 comments
AuthorsOpenReview ↗

Response to Reviewer RRii (1/3)

Reviewer question Q1

Does VISTA retain its advantage under independently adjudicated reference labels?

Related weakness W1

Both human auditors disagree with 22 of 150 benchmark preferred-route judgments, but robustness of the method ranking to independent labeling remains untested.

Response to Q1 and W1

Thanks. To verify that VISTA’s performance does not rely on benchmark-specific construction rules or the primary evaluator, we conducted two complementary evaluations:

  1. Human-Reference Rescoring: We rescore frozen routing predictions against independently adjudicated human labels on the 150-sample audit cohort.
  2. Cross-Family Holistic Evaluation: We evaluated all methods on 300 additional cases without stability filtering, using an independent cross-family judge (Qwen-3.8-flash) that directly assigns holistic utilities in [0,1][0, 1] and reference routes without task–profile decomposition or α\alpha-weighting.

Table 1. Routing performance under human-adjudicated reference labels (150 audit samples).

MethodNR ↑FAA (%) ↑PRA (%) ↑Avg. Q ↓
Direct Router0.3687--39.600.000
Adaptive LLM0.347447.8742.800.701
SAGE0.380451.5349.600.941
Generic VoI0.389352.3350.130.848
VISTA-ER0.406953.4054.531.035
VISTA0.413056.0753.930.557

Table 2. Cross-family holistic evaluation on 300 additional cases (Qwen-3.8-flash judge).

MethodNR ↑FAA (%) ↑PRA (%) ↑Avg. Q ↓
Direct Router0.4246--43.050.000
Adaptive LLM0.415048.2046.140.680
SAGE0.460551.2452.510.833
Generic VoI0.469550.6553.650.830
VISTA-ER0.497851.8758.090.991
VISTA0.510558.5157.270.563

VISTA achieves higher routing accuracy than Generic VoI with 34.3% fewer questions in Table 1. In Table 2, it delivers 8.7% higher NR with 32.2% fewer questions than Generic VoI, and 2.5% higher NR with 43.2% fewer questions than VISTA-ER, demonstrating its routing efficiency across evaluation references.


Reviewer question Q2

With utilities and question generation fixed, how much improvement comes from joint modeling?

Related weakness W2

Comparisons change representation, utility design, and question construction together, limiting attribution. Coverage and calibration of the retained belief are also insufficiently validated.

Response to Q2 and W2

Thanks. We distinguish the original representation ablation from an additional comparison that explicitly fixes utilities and question generation.

Part 1: Original Representation Ablation

The submitted Table 2 compares VISTA with Independent T/P, which approximates the joint belief as b(t,p)=bT(t)bP(p)b(t,p)=b_T(t)b_P(p). Both use the same DeepSeek-v4-Flash backbone, test episodes, candidate sets, VoI criterion, clarification budget, and communication cost.

Table 3. Original-paper ablation on the full test split.

Belief representationNR ↑FAA (%) ↑PRA (%) ↑Avg. Q ↓
Independent T/P0.460958.2350.210.5696
Joint T–P (VISTA)0.481362.6852.620.5372

VISTA achieves 4.4% higher NR, 2.41 percentage points higher PRA, and 5.7% fewer questions. This supports the implemented joint representation.

Part 2: Controlled Dependence Ablation

To address the requested controls, we use the same 300 cases and evaluation reference as Q1. Both variants share states, utilities, feasibility constraints, cached questions, answer likelihoods, and budgets. Joint T–P preserves dependence. Independent T/P uses the same initial marginals and factorizes the posterior after each update.

Table 4. Controlled dependence ablation results.

Belief representationNR ↑FAA (%) ↑PRA (%) ↑Avg. Q ↓
Independent T/P0.498156.9055.870.618
Joint T–P (VISTA)0.510558.5157.270.563

Preserving task–profile dependence improves NR by 2.5% and PRA by 1.40 percentage points, while reducing clarification questions by 8.9%. With state support, utilities, and question pools fixed, this result supports the contribution of dependence modeling to routing quality and clarification efficiency.

AuthorsOpenReview ↗

Response to Reviewer RRii (2/3)

Reviewer question Q3

How often does support truncation omit relevant states, and how does this affect stopping decisions?

Related weakness W2

Comparisons change representation, utility design, and question construction together, limiting attribution. Coverage and calibration of the retained belief are also insufficiently validated.

Response to Q3 and W2

Thanks. We evaluate support coverage and its effect on stopping through three checks.

  1. Coverage audit. Select 300 initial views, one per case, before the first ask-or-route decision. Independent assessors construct reference states consistent with visible information and match retained hypotheses against them. An omission is decision-relevant if it changes skill preference or feasibility.

  2. Stopping assessment. Compare variants with and without adaptive LL. A premature stop occurs when the policy routes despite an available question having positive net EVSI under the expanded reference model.

  3. State restoration. Restore missing reference states and recompute stopping decisions, keeping existing-state utilities, answer likelihoods, and questions fixed. Newly added states receive the estimates needed for this calculation.

Table 5. Coverage and stopping behavior with and without adaptive LL.

DiagnosticWithout adaptive LLWith adaptive LL
Any reference-state omission66/300 (22.0%)42/300 (14.0%)
Decision-relevant omission45/300 (15.0%)24/300 (8.0%)
Premature stops among relevant-omission views18/45 (40.0%)9/24 (37.5%)
Relevant omission and premature stopping, among all views18/300 (6.0%)9/300 (3.0%)

Adaptive LL reduces decision-relevant omissions from 15% to 8% and their co-occurrence with premature stopping from 6% to 3% of all views. After restoring the missing reference states and recomputing the stopping decisions, VISTA switched from routing to clarification in 7 of the 9 cases identified as premature stops under the expanded reference model. This suggests that support omissions contributed to premature stopping in these cases.

AuthorsOpenReview ↗

Response to Reviewer RRii (3/3)

Reviewer question Q4

What measurable benefit comes from preserving task–profile dependence when utilities, question generation, and hypothesis budgets are controlled?

Related weakness W3

Value-guided clarification and stopping are established principles, so VISTA’s methodological novelty rests mainly on candidate-conditioned joint task–profile modeling. However, the holistic baseline also differs in utility decomposition, feasibility constraints, and question construction (Appendix D.4), leaving the contribution of joint modeling insufficiently isolated from the adaptation of existing VOI methods.

Response to Q4 and W3

Thanks. We clarify the task characteristics, our contribution, and the controlled evidence below.

Part 1: Technical contribution beyond Bayesian and VoI foundations

VISTA’s technical contribution is a candidate-conditioned belief-construction mechanism that determines which task–profile states to represent, preserves their routing-relevant dependencies, and adaptively allocates hypothesis capacity. It consists of three technical components:

  1. Candidate-conditioned hypothesis construction. VISTA extracts the contrast set D(S)\mathcal{D}(\mathcal{S}) from candidate SkillRoutingCards and uses these distinguishing task and profile dimensions to construct routing-relevant hypotheses. The candidate skills therefore shape the uncertainty representation itself, focusing it on distinctions that can change the selected skill. Removing candidate-distinguishing dimensions reduces NR from 0.4813 to 0.4415 in the submitted Table 2.
  2. Task-conditional joint support. VISTA first constructs task hypotheses and then instantiates the applicable profile hypotheses P0(t)\mathcal{P}_0(t) for each task tt. This structure avoids irrelevant task–profile combinations while preserving dependencies through which a task answer can update profile beliefs and change the value of subsequent clarification. Together with joint-state feasibility constraints, it affects both posterior inference and candidate utility. Replacing this structure with independent task/profile modeling reduces NR to 0.4609.
  3. Adaptive support expansion using residual-mass estimates. During initialization, VISTA estimates the probability mass (ρ0\rho_0) assigned to states outside the explicit support. When (ρ0>ϵcov\rho_0>\epsilon_{\mathrm{cov}}), it adds hypotheses along candidate-distinguishing dimensions not yet covered, subject to the support limit (Lmax⁡L_{\max}). This adapts hypothesis capacity to the request’s unresolved ambiguity while bounding computation. Using fixed hypotheses reduces NR to 0.4524.

These mechanisms provide a common state representation for evaluating task clarification, profile clarification, and immediate routing under the same downstream skill utility. Bayesian updating, EVPI, and EVSI supply the decision-theoretic foundations; VISTA contributes the routing-specific construction of the belief model on which these rules operate.

Part 2: Controlled Evidence for Task–Profile Dependence

Controlled results. The holistic baseline compares multiple design choices. To isolate dependence, Q2’s Table 4 fixes the actual states and state counts, utilities, feasibility conditions, cached questions, answer likelihoods, and interaction budgets. Retaining dependence yields 2.5% higher NR, 1.40-percentage-point higher PRA, and 8.9% fewer questions, quantifying its contribution without additional hypothesis coverage.

Reviewer_BYD4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method for deciding what to ask users and when to select a skill under unclear task requirements and user constraints. It represents possible task–user profile combinations and uses value-of-information calculations to ask questions only when the expected improvement in skill selection outweighs the interaction cost. The authors also construct a controlled benchmark with more than 2,000 interaction episodes. The authors present experimental results with two LLMs that showed improved cost-adjusted routing performance and fewer questions.

Critical strengths

The proposed method distinguishes uncertainty that affects skill selection from uncertainty that can remain unresolved. It evaluates task questions, profile questions, and immediate selection under a shared utility. This provides a consistent basis for avoiding questions that are unlikely to improve the routing decision.

Comparisons with Generic VoI and VISTA-ER together with ablations and factorial analysis help separate the benefits of state representation and question-selection criteria. The strongest result is substantially fewer questions with comparable routing quality. These comparisons help explain where the gains come from rather than showing only an overall improvement.

The benchmark includes cases where missing information changes which skill best meets the user’s requirements and constraint and cases where the same skill remains appropriate despite uncertainty. This tests whether an agent asks when clarification is needed and stops when further questions would not help. It can therefore reveal agents that select the right skill but incur more costs with unnecessary questions. The human audit also reports disagreements, making limitations in the reference labels visible.

Critical weaknesses

The methodological novelty needs clearer positioning. Most of the formal framework follows established Bayesian decision-making and value-of-information principles. The paper should distinguish these foundations from its specific contributions in candidate-conditioned task–profile modeling. Moreover, the calculations depend on LLM-generated hypotheses, probabilities, and utility estimates. Their accuracy and the method’s sensitivity to estimation errors are not directly established.

Human involvement and validation coverage are documented without sufficient details. The authors report manual review and revision but do not specify how many cases were inspected or changed, what changes were made, or how much human effort was required. The clarification audit evaluates target and answer-state summaries rather than generated questions and actual user exchanges. Human judgments also do not correct the benchmark labels, leaving unresolved disagreements.

The benchmark’s contribution is primarily diagnostic, with limited evidence of broader validity. Its controlled design is useful for studying routing, but clarification needs depend on the fixed candidate set and how skills are divided. A question may be necessary when several specialized skills are available but irrelevant to routing when one general-purpose skill covers them all. It may still be essential for producing a useful result. Thus, the benchmark measures candidate-specific routing needs rather than clarification needs across the full task.

The evaluation mainly considers missing profile information, rather than profile information that is outdated or does not apply to the current task. Reliance on such information may produce an incorrect but confident belief, leading the method to skip useful clarification. The paper should test whether VISTA can recognize and resolve task-specific exceptions to stored user preferences.

Practical advantages over capable conversational agents remain unclear. Adaptive LLM Prompting provides a relevant baseline, but it does not establish superiority over a well-configured open or commercial agent framework. VISTA uses more inference tokens than this baseline, while the primary reward excludes inference cost. The experiments also omit execution-stage clarification and final task quality. Fewer routing questions may therefore shift interaction to a later stage rather than reduce total user effort.

Questions for authors
  1. Which technical elements are especially novel beyond established Bayesian and value-of-information methods? How accurate are the estimated beliefs, answer likelihoods, and utilities, and how sensitive are the decisions to errors in these estimates?

  2. How many people participated in benchmark construction and revision, how many cases were manually changed, and what criteria guided those changes? Would validation of actual generated dialogues support the current clarification labels?

  3. How do performance and clarification labels change when equivalent capabilities are packaged as several specialized skills versus one general-purpose skill?

  4. Do the gains persist against a strong conversational agent when total inference cost, all clarification turns, and final task quality are measured together?

6 comments
AuthorsOpenReview ↗

Response to Reviewer BYD4 (1/3)

Reviewer question Q1

Which technical elements are especially novel beyond established Bayesian and value-of-information methods? How accurate are the estimated beliefs, answer likelihoods, and utilities, and how sensitive are the decisions to errors in these estimates?

Related weakness W1

The methodological novelty...

Related weakness W4

The evaluation mainly...

Response to Q1 and W1 and W4

Thanks.

Part 1: Technical contribution beyond Bayesian and VoI foundations

VISTA’s technical contribution is a candidate-conditioned belief-construction mechanism that determines which task–profile states to represent, preserves their routing-relevant dependencies, and adaptively allocates hypothesis capacity. It consists of three technical components:

  1. Candidate-conditioned hypothesis construction. VISTA extracts the contrast set D\mathcal{D}(S\mathcal{S}) from candidate SkillRoutingCards and uses these distinguishing task and profile dimensions to construct routing-relevant hypotheses. The candidate skills therefore shape the uncertainty representation itself, focusing it on distinctions that can change the selected skill. Removing candidate-distinguishing dimensions reduces NR from 0.4813 to 0.4415 in the submitted Table 2.
  2. Task-conditional joint support. VISTA first constructs task hypotheses and then instantiates the applicable profile hypotheses P0(t)\mathcal{P}_0(t) for each task tt. This structure avoids irrelevant task–profile combinations while preserving dependencies through which a task answer can update profile beliefs and change the value of subsequent clarification. Together with joint-state feasibility constraints, it affects both posterior inference and candidate utility. Replacing this structure with independent task/profile modeling reduces NR to 0.4609.
  3. Adaptive support expansion using residual-mass estimates. During initialization, VISTA estimates the probability mass (ρ0\rho_0) assigned to states outside the explicit support. When (ρ0>ϵcov\rho_0>\epsilon_{\mathrm{cov}}), it adds hypotheses along candidate-distinguishing dimensions not yet covered, subject to the support limit (Lmax⁡L_{\max}). This adapts hypothesis capacity to the request’s unresolved ambiguity while bounding computation. Using fixed hypotheses reduces NR to 0.4524.

These mechanisms provide a common state representation for evaluating task clarification, profile clarification, and immediate routing under the same downstream skill utility. Bayesian updating, EVPI, and EVSI supply the decision-theoretic foundations; VISTA contributes the routing-specific construction of the belief model on which these rules operate.

Part 2: Estimation Accuracy and Routing Performance

  1. LLM-based probability and confidence estimation is an established approach [1,2], also relevant to VoI-based agent communication [3]. This supports our estimator interface.

  2. We compared VISTA and Generic VoI under matched backbones on 300 added cases. Diagnostics use common state events and question–answer pairs; evaluation references and interaction budgets are held fixed.

Table 1. Estimator comparison results.

EstimatorMethodBrier↓Answer NLL↓Utility-ranking agreement↑NR↑PRA (%)↑Avg. Q
deepseek-flashGeneric VoI0.2070.50971.2%0.469553.650.830
deepseek-flashVISTA0.1840.46274.8%0.510557.270.563
gemini-3.8-flashGeneric VoI0.1830.46875.1%0.491855.960.812
gemini-3.8-flashVISTA0.1610.41778.2%0.531859.630.578
gpt-6.1-solGeneric VoI0.1580.42178.9%0.514758.720.784
gpt-6.1-solVISTA0.1390.37181.6%0.550662.140.551

Lower estimation errors are associated with higher PRA and NR, and VISTA outperforms Generic VoI under each matched backbone. These results also suggest that more accurate semantic estimators can further improve VISTA’s routing performance without changing its decision policy.

Part 3: Response to outdated profiles

VISTA models profile conditions jointly with the task, allowing the applicable profile hypotheses to vary across tasks. When a user explicitly states a correction or exception, the semantic adapter described in response of Q3 for Reviewer BCbj can extract the new constraint and reuse VISTA’s support adaptation and belief updating to incorporate it into the routing decision. Our current benchmark focuses on incomplete information. Systematic evaluation of stale profiles and task-specific exceptions is a direction for further work.

References

[1] Lin, S., Hilton, J., and Evans, O. (2022). Teaching Models to Express Their Uncertainty in Words.

[2] Tian, K., et al. (2023). Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback. EMNLP, pp. 5433–5442.

[3] Dong, Y. R., et al. (2026). Value of Information: A Framework for Human–Agent Communication. ACL, pp. 42879–42896.

AuthorsOpenReview ↗

Response to Reviewer BYD4 (2/3)

Reviewer question Q2

How many people participated in benchmark construction and revision, how many cases were manually changed, and what criteria guided those changes? Would validation of actual generated dialogues support the current clarification labels?

Related weakness W2

Human involvement and validation coverage are documented without sufficient details...

Response to Q2 and W2

Thanks. We clarify benchmark auditing and dialogue-level validation separately.

Part 1: Benchmark Construction

150-Case Audit Breakdown: Candidate counterfactuals failing structural/semantic checks were filtered out before label generation (Appendix C.2.3). Two independent reviewers audited (examples in the paper Table 5) 150 stratified cases without post-hoc label modification: both agreed with the reference route in 76.7% (115/150) of cases, disagreed with each other in 8.7% (13/150), and jointly rejected only 14.7% (22/150). Human involvement was limited to assessing dataset quality.

Part 2: Label Validation and User Interaction Records

1. LLM-generated labels with human validation. Using LLMs for scalable data generation or annotation, complemented by human evaluation, has established precedents [1,2]. This motivates our workflow.

2. Validation through user interaction records. We examined generated questions and verbatim user responses across 30 cases, comprising 15 clarification-needed and 15 immediate-route cases. Records include option selections, free-text supplements, and corrections. Question usability is assessed on the 18 clarification exchanges, while clarification necessity is assessed across all 30 cases.

Table 2. Dialogue assessment based on user interaction records.

CriterionCases or exchanges meeting criterionRate (%)
Clear question17/1894.4
Answerable question15/1883.3
Adequate answer-option coverage15/1883.3
Agreement with the original clarification-necessity label28/3093.3

The 93.3% agreement between dialogue-based judgments and the original clarification-necessity labels provides support for the current labels.

References

[1] Wang, Y., et al. (2023). Self-Instruct: Aligning Language Models with Self-Generated Instructions. ACL.

[2] Gilardi, F., Alizadeh, M., and Kubli, M. (2023). ChatGPT outperforms crowd workers for text-annotation tasks. PNAS, 120(30).


Reviewer question Q3

How do performance and clarification labels change when equivalent capabilities are packaged as several specialized skills versus one general-purpose skill?

Related weakness W3

The benchmark’s contribution is primarily diagnostic...

Response to Q3 and W3

Thank you. VISTA asks when resolving task–profile uncertainty offers an expected improvement in routing utility that exceeds the question cost. This decision criterion applies to both specialized and general-purpose skills.

1. How clarification labels change. With specialized candidates, missing information can flip the optimal skill, warranting an ask action. If those candidates are bundled into a single general-purpose skill (or if one skill dominates across all plausible states), the external route becomes invariant. By Eq. 5, EVPI(bi)=0\mathrm{EVPI}(b_i)=0, and VISTA routes immediately. While the missing information may still be needed during downstream execution, it is no longer routing-relevant, and the reference action appropriately shifts to route.

2. How performance should be interpreted. Bundling alters the routing decision space and shifts unresolved ambiguity from pre-routing to post-routing execution. Even when terminal selection becomes trivial (e.g., a single omnibus skill), stopping metrics such as Unnecessary-Ask Rate (UAR) and First-Action Accuracy (FAA) still strictly distinguish policies that route directly from those that ask redundant pre-routing questions.

3. Why candidate dependence is appropriate for this benchmark. The value of routing clarification inherently depends on the available alternatives. Retaining an ask label when bundling eliminates routing distinctions would falsely penalize a router for avoiding useless pre-routing questions. Our benchmark explicitly captures this via route-invariant controls, where VISTA reduces Control UAR to 16.60% versus 43.40% for entropy-based VISTA-ER (Table 13).

The benchmark thus directly evaluates the stated clarify-or-route problem. A question can remain essential for execution while having no value for skill selection; this distinction is consistent with the routing labels. We will make the relationship between skill packaging, clarification labels, and performance interpretation explicit in Section 3.

AuthorsOpenReview ↗

Response to Reviewer BYD4 (3/3)

Reviewer question Q4

Do the gains persist against a strong conversational agent when total inference cost, all clarification turns, and final task quality are measured together?

Related weakness W5

Practical advantages over capable conversational agents remain unclear. Adaptive LLM Prompting provides a relevant baseline, but it does not establish superiority over a well-configured open or commercial agent framework. VISTA uses more inference tokens than this baseline, while the primary reward excludes inference cost. The experiments also omit execution-stage clarification and final task quality. Fewer routing questions may therefore shift interaction to a later stage rather than reduce total user effort.

Response to Q4 and W5

Thanks. We compare VISTA, Adaptive LLM Prompting, and Claude code-only across five tasks: C1: svg-graphics, C2: python-pptx, C3: wp-user-documentation, C4: audit-docs, and C5: pdf-translate.

The comparison uses the same underlying Opus5.5 model and Claude Code execution environment, accounting for routing and execution tokens, all clarification turns, and final artifact quality. A task that fails due to incorrect skill retrieval terminates and receives a quality score of 0. Final artifacts are evaluated blindly by gpt-6.1-sol on a 0–100 scale.

Table 3. Per-case outcomes and token consumption. Token counts are in thousands (k).

CaseMethodRouting tokens↓Execution tokens↓Total tokens↓OutcomeFinal quality↑
C1VISTA21.3738.7460.11Succeeded94
C1Adaptive LLM Prompting18.2440.1858.42Succeeded92
C1Claude code-only19.5239.4658.98Succeeded95
C2VISTA24.8691.26116.12Succeeded92
C2Adaptive LLM Prompting19.6389.73109.36Succeeded86
C2Claude code-only20.3792.08112.45Succeeded92
C3VISTA23.4983.59107.08Succeeded93
C3Adaptive LLM Prompting18.9185.24104.15Failed0
C3Claude code-only20.8182.67103.48Failed0
C4VISTA25.18146.82172.00Succeeded91
C4Adaptive LLM Prompting19.38145.61164.99Succeeded83
C4Claude code-only21.29147.35168.64Succeeded88
C5VISTA28.73185.37214.10Failed0
C5Adaptive LLM Prompting20.46183.92204.38Failed0
C5Claude code-only22.64184.63207.27Failed0

Table 4. Mean token consumption across all five tasks, including failures. Unit: k tokens.

MethodRouting tokens↓Execution tokens↓Total tokens↓
VISTA24.73109.16133.88
Adaptive LLM Prompting19.32108.94128.26
Claude code-only20.93109.24130.16

Table 5. Final quality and mean clarification counts across all five tasks.

MethodSuccessful tasks↑Final quality /100↑Routing questions↓Execution questions↓Total user interactions↓
VISTA4/574.00.60.61.2
Adaptive LLM Prompting3/552.21.20.82.0
Claude code-only3/555.00.80.81.6

VISTA uses 2.9%–4.4% more total tokens, achieves 19.0–21.8 points higher final quality, and requires 25%–40% fewer user interactions. This demonstrates a trade-off in which modest additional routing computation accompanies better task outcomes and lower interaction burden.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.