- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces CLEAT, a framework for learning value-guided clarification in interactive LLM agents. The authors argue that the usefulness of a clarification should be measured by its contribution to task outcomes rather than information gain alone. They further observe that clarification value depends on the agent's current policy, as the agent's ability to use additional information may change during training.
To address these issues, CLEAT defines Residual Query Value (RQV) as the expected utility difference between asking a targeted clarification question and allowing the task policy to continue on its own. The framework estimates RQV through paired QUERY/PASS rollouts and uses the resulting outcomes to jointly train a clarification-value controller and the task policy. The controller guides subsequent rollout collection, while updated task policies provide new value estimates, forming an online co-training process.
Experiments with Qwen3-4B cover four benchmarks and ten evaluation settings, including personalized decision-making, interactive tool use, and underspecified requests. CLEAT achieves the highest mean performance across all ten settings. Ablation studies further examine the contributions of value-guided sampling, controller updates, and inference-time intervention.
Critical strengths
-
The paper addresses a meaningful problem in interactive agents by considering clarification value relative to the agent's actual ability to use the information. The distinction between information gain and task utility is well established, but extending this perspective to a policy-relative setting is useful. In particular, comparing targeted clarification against the agent's native continuation provides a reasonable way to estimate the incremental benefit of asking and motivates adapting value estimates as the task policy evolves.
-
The paper presents a coherent framework that connects clarification-value estimation with policy learning. The paired-rollout construction provides supervision for both the controller and the task policy, while value-guided sampling and coverage sampling balance prioritization and exploration. The sampling correction is also supported by a theoretical analysis under clearly stated assumptions. Empirically, the method achieves encouraging results across diverse benchmarks, and the ablation studies provide useful evidence for the contributions of several key components.
Critical weaknesses
-
The motivation for value-guided co-training and the source of its gains could be established more clearly. The fact that RQV changes as the task policy evolves motivates refreshing the value predictor, but does not by itself explain why clarifications with higher current decision value should provide more useful training experiences for improving the policy. A question that the current agent handles poorly could have low RQV yet still be valuable for learning. The coverage component partly addresses this concern, and the existing ablations demonstrate practical benefits from guidance and dynamic controller updates. However, the relationship between RQV-based prioritization and policy improvement remains insufficiently explained. In addition, policy training jointly updates high-level QUERY/PASS selection and downstream task execution, making it difficult to distinguish improvements in clarification decisions from improvements in the agent's ability to use clarification feedback. The current results support the effectiveness of the overall framework, but provide less insight into how these learning components contribute to each other.
-
The sequential operation of the controller and the overall presentation of the method could be clearer. The paper defines clarification-eligible histories abstractly, but does not sufficiently explain how these decision points are identified during actual agent execution or how intermediate histories, such as those following tool observations, enter the training process. At inference, QUERY allows subsequent controller decisions, whereas PASS permanently terminates controller intervention. The implications of this asymmetric stopping rule are not discussed in much detail. Moreover, RQV is estimated from a single forced clarification followed by native policy continuation, while inference may involve multiple successive controller-guided clarifications. This may be a reasonable approximation, but its relationship to the sequential interaction process is not sufficiently explained. More generally, the paper requires considerable effort to follow. The motivations for policy learning, dynamic value estimation, and the additional sampling and gradient-correction mechanisms are presented in a somewhat fragmented manner, with important operational details spread across the appendices. This makes the overall method harder to understand than necessary.
Questions for authors
-
How reliably does the learned controller estimate the sign and relative ordering of RQV on unseen interaction histories? In particular, how should we understand the reliability of these predictions as the task policy evolves, especially near the zero threshold used to determine whether clarification is worthwhile?
-
How are clarification-eligible histories identified and scheduled during agent execution, particularly after intermediate actions or tool observations? What motivates treating PASS as a permanent termination of controller intervention rather than allowing the controller to reconsider clarification at a later state?
-
Could the authors clarify how training budgets are controlled across the learning-based baselines, particularly in terms of sampled continuations and total environment interactions? The paper describes matched continuation budgets for GRPO-related comparisons, but the degree of budget matching across the other training-based methods is less clear. Since CLEAT explicitly focuses on guiding experience collection, this distinction is relevant to interpreting the performance gains.
-
Could you provide further reasons into what drives such a substantial performance boost? Given that this method delivers a remarkably significant improvement over the baseline, I would be interested to learn more about the underlying reasons.
4 comments
Response to Reviewer 2PZ1 (Part 2/2)
For W2 and Q2: Decision points and sequential controller operation
A history is clarification-eligible when another user exchange is permitted and the candidate set is nonempty. The controller evaluates each such history until its first . After a it evaluates the next eligible history, and after the task policy handles the rest of the interaction, including lookups and its own questions. Training branches from a fixed pool of histories, mostly initial requests with a small number of nonterminal states after the first or second interaction round. Each yields fresh paired and continuations under the current policy, without further controller intervention. States after tool observations enter training within these continuations, where all agent actions are trained on the task return.
Deployment applies this single-intervention comparison repeatedly. Let be the expected utility under repeated controller decisions. With a fixed policy and exact RQV, backward induction over the finite budget gives , so
where and are continuation values from the given history, is the utility accrued during the exchange itself, net of its interaction cost, and includes the answer and the updated budget. The single-step label is therefore a lower bound on the value of asking, while passing yields exactly because PASS is final. Making PASS final thus keeps the rule conservative. The controller asks only when a single targeted question is worthwhile on its own.
The rule can miss gains that need several coordinated queries, and it leaves needs arising after PASS to the task policy. A re-entrant variant handles the latter by consulting the controller at every later eligible turn, including after tool observations, without retraining.
| Method | Airline | Retail | Telecom |
|---|---|---|---|
| GRPO | 24.33 ± 3.23 | 27.50 ± 4.33 | 24.17 ± 2.89 |
| CLEAT (controller stops after PASS) | 35.00 ± 2.00 | 32.50 ± 1.29 | 24.94 ± 1.77 |
| CLEAT (re-entrant controller) | 34.33 ± 2.31 | 32.83 ± 1.76 | 26.11 ± 2.04 |
The re-entrant variant performs comparably on Airline and Retail and reaches a higher mean on Telecom, where what to ask often depends on earlier diagnostic results. We will bring these operational details together in Section 3.5 and align Appendices A.2, B, and E with this description.
For Q3: Training budgets and environment interactions
CLEAT's schedule follows Appendix B, with a controller bootstrap, 120 co-training rounds of 16 histories each, and a final controller refresh, using five continuations per history. All continuations are generated afresh under the current policy and run until the episode ends. All training baselines use the same task pool, backbone, user simulator, and turn limits as CLEAT. Each RL baseline keeps the sampling procedure of its original method, and we aligned the total number of training rollouts and policy updates with CLEAT's schedule. IG-Reward, for instance, keeps the Qwen3-4B task agent and trains a separate Qwen3-1.7B clarifier with DAPO on individual clarification decisions.
Because CLEAT focuses on guiding experience collection, the comparisons that isolate its contribution are controlled exactly. The w/o guidance variant keeps the same per-history budget, paired comparisons, and inference rule and differs only in the value tilt, and it lowers scores by 1.97 to 8.33 points (Table 3). Menu-GRPO also uses the same per-history budget. We will document the schedule in Appendix B and each baseline's sampling unit and budget in Appendix C.2.
We thank the reviewer again for the careful reading and thoughtful feedback. We will clarify the rationale for value-guided collection, the controller’s sequential operation, and the training-budget comparisons. We will also improve the organization and wording to make the learning process and experimental setup clearer and easier to follow.
Thank you for your response. I will maintain my current score. Good luck to you.
We sincerely thank you for taking the time to read our rebuttal and for your valuable feedback, which has helped us improve the paper. We will incorporate the additional experimental results and clarifications into the revised manuscript. If any concerns remain, we would be happy to address them during the discussion period.
Response to Reviewer 2PZ1 (Part 1/2)
We thank the reviewer for recognizing the policy-relative formulation, the coherent learning framework, and the encouraging empirical results. We appreciate the questions about experience collection, controller reliability, and sequential interaction.
For W1 and Q4: Rationale for value-guided collection and sources of improvement
RQV measures the current benefit of a clarification, not its learning potential. Value guidance leaves the expected policy gradient unchanged (Proposition 1) and only decides where a fixed rollout budget is spent. Spending it on high predicted RQV lets the policy practice on the questions the controller will ask at inference and surfaces valuable targets that the policy rarely selects on its own. Questions the agent currently handles poorly remain covered through the unguided PASS and coverage draws, and renewed supervision raises their priority once the policy learns to use their answers. By contrast, IG guidance favors informative questions regardless of how well the policy uses the answers, and it underperforms RQV guidance in all six settings (Figure 3).
Paired collection and value guidance both contribute, in proportions that vary across settings. Menu-GRPO samples initial actions independently, so a group often lacks either the QUERY or the PASS branch and carries no signal on whether asking helps, whereas CLEAT evaluates both branches from the same history and user goal. The results below separate the two steps at a matched per-history budget.
On Travel and IN3, paired collection with the controller contributes 11.34, 8.92, and 6.02 points, and value guidance adds a further 7.55, 5.00, and 3.09. On AskOverconfidence, value guidance is the main driver, adding 3.99 points in accuracy and 3.79 in coverage. With the same trained policy, controller decisions add 0.44 to 6.14 points over the policy's own menu choices (Table 3). On Travel, the trained policy alone already exceeds the strongest baseline by 9.01 points, and on Airline and Retail the controller's decisions add 6.14 and 4.00 points on top. Policy learning and controller decisions are thus complementary, and we will make these connections explicit in the revised paper.
For Q1: Controller reliability as the task policy evolves
Beyond the task-level evidence on unseen test histories (Table 3 above, and transfer to Intention and Telepathy without training on those environments), we measured sign and ordering accuracy directly. On 100 held-out histories each from IN3 and AskMind, we recorded the controller's predicted RQV for every candidate and estimated the empirical RQV from independent paired continuations under the final policy.
Ask-or-pass accuracy checks the predicted sign against the empirical RQV of the best candidate, and best-target agreement checks whether the top-ranked candidate has the highest empirical RQV. Random flips a fair coin for the former and picks a uniformly random target for the latter. Entries are means over three seeds; best-target agreement uses only histories with at least two candidates.
The controller identifies both the sign and the best target more accurately than the trained policy's own choices and well above random. Near the zero threshold, mistakes cost little by construction, because the utility lost by a wrong ask-or-pass decision equals the magnitude of the true RQV. The threshold is also not a tuned hyperparameter. It follows from the definition of RQV, since a positive value means that asking is expected to outperform proceeding.
As the policy evolves, reliability is maintained by renewing the labels in every round and by refreshing the controller under the final policy, so its predictions refer to the policy that is deployed. Freezing the controller after bootstrap lowers scores by 1.27 to 6.70 points (Table 3), even though it is still refreshed under the final policy before evaluation. This shows that guidance fitted to an earlier policy is less effective during training and that co-training corrects this.