acceptodds
Under review as a conference paper at ICLR 2027

CLEAT: Learning Value-Guided Clarification for Interactive Agents

Abstract

LLM agents can execute increasingly complex tasks, but effective assistance also requires aligning their actions with goals and constraints that users may leave unstated. We study this problem as interactive intent alignment through clarification. Yet clarification is useful only when the agent can turn the answer into better decisions. As training changes how the agent handles incomplete information and uses additional answers, the same clarification can become more or less useful. We introduce CLEAT (**CL**arification through valu**E**-guided **A**gent co-**T**raining), a framework that co-trains a clarification-value controller and the task policy from shared interaction outcomes. We formulate residual query value to measure the task utility gained by asking rather than proceeding under the current policy, and estimate it through paired rollouts from the same interaction history. These paired returns train the controller to predict clarification value and the task policy to select useful questions and act on the answers. The controller guides subsequent rollout collection, while outcomes from the updated agent provide new value labels that allow clarification guidance to adapt as the agent learns. With Qwen3-4B, CLEAT achieves the highest mean score in all ten evaluation settings across four benchmarks. Gains over the strongest baseline reach and percentage points on UserGym Travel and IN3 underspecification judgment accuracy, respectively. Ablations show that co-training improves the task policy even when the controller is removed at inference, while adapting clarification guidance to the evolving policy yields further gains.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Clarification, Human-Agent Interaction, Reinforcement Learning, User-Centric AI

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_2PZ1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CLEAT, a framework for learning value-guided clarification in interactive LLM agents. The authors argue that the usefulness of a clarification should be measured by its contribution to task outcomes rather than information gain alone. They further observe that clarification value depends on the agent's current policy, as the agent's ability to use additional information may change during training.

To address these issues, CLEAT defines Residual Query Value (RQV) as the expected utility difference between asking a targeted clarification question and allowing the task policy to continue on its own. The framework estimates RQV through paired QUERY/PASS rollouts and uses the resulting outcomes to jointly train a clarification-value controller and the task policy. The controller guides subsequent rollout collection, while updated task policies provide new value estimates, forming an online co-training process.

Experiments with Qwen3-4B cover four benchmarks and ten evaluation settings, including personalized decision-making, interactive tool use, and underspecified requests. CLEAT achieves the highest mean performance across all ten settings. Ablation studies further examine the contributions of value-guided sampling, controller updates, and inference-time intervention.

Critical strengths
  1. The paper addresses a meaningful problem in interactive agents by considering clarification value relative to the agent's actual ability to use the information. The distinction between information gain and task utility is well established, but extending this perspective to a policy-relative setting is useful. In particular, comparing targeted clarification against the agent's native continuation provides a reasonable way to estimate the incremental benefit of asking and motivates adapting value estimates as the task policy evolves.

  2. The paper presents a coherent framework that connects clarification-value estimation with policy learning. The paired-rollout construction provides supervision for both the controller and the task policy, while value-guided sampling and coverage sampling balance prioritization and exploration. The sampling correction is also supported by a theoretical analysis under clearly stated assumptions. Empirically, the method achieves encouraging results across diverse benchmarks, and the ablation studies provide useful evidence for the contributions of several key components.

Critical weaknesses
  1. The motivation for value-guided co-training and the source of its gains could be established more clearly. The fact that RQV changes as the task policy evolves motivates refreshing the value predictor, but does not by itself explain why clarifications with higher current decision value should provide more useful training experiences for improving the policy. A question that the current agent handles poorly could have low RQV yet still be valuable for learning. The coverage component partly addresses this concern, and the existing ablations demonstrate practical benefits from guidance and dynamic controller updates. However, the relationship between RQV-based prioritization and policy improvement remains insufficiently explained. In addition, policy training jointly updates high-level QUERY/PASS selection and downstream task execution, making it difficult to distinguish improvements in clarification decisions from improvements in the agent's ability to use clarification feedback. The current results support the effectiveness of the overall framework, but provide less insight into how these learning components contribute to each other.

  2. The sequential operation of the controller and the overall presentation of the method could be clearer. The paper defines clarification-eligible histories abstractly, but does not sufficiently explain how these decision points are identified during actual agent execution or how intermediate histories, such as those following tool observations, enter the training process. At inference, QUERY allows subsequent controller decisions, whereas PASS permanently terminates controller intervention. The implications of this asymmetric stopping rule are not discussed in much detail. Moreover, RQV is estimated from a single forced clarification followed by native policy continuation, while inference may involve multiple successive controller-guided clarifications. This may be a reasonable approximation, but its relationship to the sequential interaction process is not sufficiently explained. More generally, the paper requires considerable effort to follow. The motivations for policy learning, dynamic value estimation, and the additional sampling and gradient-correction mechanisms are presented in a somewhat fragmented manner, with important operational details spread across the appendices. This makes the overall method harder to understand than necessary.

Questions for authors
  1. How reliably does the learned controller estimate the sign and relative ordering of RQV on unseen interaction histories? In particular, how should we understand the reliability of these predictions as the task policy evolves, especially near the zero threshold used to determine whether clarification is worthwhile?

  2. How are clarification-eligible histories identified and scheduled during agent execution, particularly after intermediate actions or tool observations? What motivates treating PASS as a permanent termination of controller intervention rather than allowing the controller to reconsider clarification at a later state?

  3. Could the authors clarify how training budgets are controlled across the learning-based baselines, particularly in terms of sampled continuations and total environment interactions? The paper describes matched continuation budgets for GRPO-related comparisons, but the degree of budget matching across the other training-based methods is less clear. Since CLEAT explicitly focuses on guiding experience collection, this distinction is relevant to interpreting the performance gains.

  4. Could you provide further reasons into what drives such a substantial performance boost? Given that this method delivers a remarkably significant improvement over the baseline, I would be interested to learn more about the underlying reasons.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 2PZ1 (Part 1/2)

We thank the reviewer for recognizing the policy-relative formulation, the coherent learning framework, and the encouraging empirical results. We appreciate the questions about experience collection, controller reliability, and sequential interaction.


For W1 and Q4: Rationale for value-guided collection and sources of improvement

RQV measures the current benefit of a clarification, not its learning potential. Value guidance leaves the expected policy gradient unchanged (Proposition 1) and only decides where a fixed rollout budget is spent. Spending it on high predicted RQV lets the policy practice on the questions the controller will ask at inference and surfaces valuable targets that the policy rarely selects on its own. Questions the agent currently handles poorly remain covered through the unguided PASSPASS and coverage draws, and renewed supervision raises their priority once the policy learns to use their answers. By contrast, IG guidance favors informative questions regardless of how well the policy uses the answers, and it underperforms RQV guidance in all six settings (Figure 3).

Paired collection and value guidance both contribute, in proportions that vary across settings. Menu-GRPO samples initial actions independently, so a group often lacks either the QUERYQUERY or the PASSPASS branch and carries no signal on whether asking helps, whereas CLEAT evaluates both branches from the same history and user goal. The results below separate the two steps at a matched per-history budget.

MethodTravel Pass-1IN3 Judg.AccIN3 RecoverAskOv. AccAskOv. Cov
GRPO42.2666.0553.6858.6743.50
Menu-GRPO (adds the menu)44.8370.0358.4759.1744.58
CLEAT w/o guidance (adds paired collection and the controller)56.1778.9564.4957.2847.70
CLEAT (adds value guidance)63.7283.9567.5861.2751.49

On Travel and IN3, paired collection with the controller contributes 11.34, 8.92, and 6.02 points, and value guidance adds a further 7.55, 5.00, and 3.09. On AskOverconfidence, value guidance is the main driver, adding 3.99 points in accuracy and 3.79 in coverage. With the same trained policy, controller decisions add 0.44 to 6.14 points over the policy's own menu choices (Table 3). On Travel, the trained policy alone already exceeds the strongest baseline by 9.01 points, and on Airline and Retail the controller's decisions add 6.14 and 4.00 points on top. Policy learning and controller decisions are thus complementary, and we will make these connections explicit in the revised paper.


For Q1: Controller reliability as the task policy evolves

Beyond the task-level evidence on unseen test histories (Table 3 above, and transfer to Intention and Telepathy without training on those environments), we measured sign and ordering accuracy directly. On 100 held-out histories each from IN3 and AskMind, we recorded the controller's predicted RQV for every candidate and estimated the empirical RQV from independent paired continuations under the final policy.

Decision makerIN3: Ask-or-pass accuracy (%)IN3: Best-target agreement (%)AskMind: Ask-or-pass accuracy (%)AskMind: Best-target agreement (%)
Random50.025.650.037.0
Trained policy's own choice69.746.865.352.1
Controller81.360.276.763.5

Ask-or-pass accuracy checks the predicted sign against the empirical RQV of the best candidate, and best-target agreement checks whether the top-ranked candidate has the highest empirical RQV. Random flips a fair coin for the former and picks a uniformly random target for the latter. Entries are means over three seeds; best-target agreement uses only histories with at least two candidates.

The controller identifies both the sign and the best target more accurately than the trained policy's own choices and well above random. Near the zero threshold, mistakes cost little by construction, because the utility lost by a wrong ask-or-pass decision equals the magnitude of the true RQV. The threshold is also not a tuned hyperparameter. It follows from the definition of RQV, since a positive value means that asking is expected to outperform proceeding.

As the policy evolves, reliability is maintained by renewing the labels in every round and by refreshing the controller under the final policy, so its predictions refer to the policy that is deployed. Freezing the controller after bootstrap lowers scores by 1.27 to 6.70 points (Table 3), even though it is still refreshed under the final policy before evaluation. This shows that guidance fitted to an earlier policy is less effective during training and that co-training corrects this.

AuthorsOpenReview ↗

Response to Reviewer 2PZ1 (Part 2/2)

For W2 and Q2: Decision points and sequential controller operation

A history is clarification-eligible when another user exchange is permitted and the candidate set is nonempty. The controller evaluates each such history until its first PASSPASS. After a QUERYQUERY it evaluates the next eligible history, and after PASSPASS the task policy handles the rest of the interaction, including lookups and its own questions. Training branches from a fixed pool of histories, mostly initial requests with a small number of nonterminal states after the first or second interaction round. Each yields fresh paired QUERYQUERY and PASSPASS continuations under the current policy, without further controller intervention. States after tool observations enter training within these continuations, where all agent actions are trained on the task return.

Deployment applies this single-intervention comparison repeatedly. Let W(h)W(h) be the expected utility under repeated controller decisions. With a fixed policy and exact RQV, backward induction over the finite budget gives W(h)≥Vπ(h)W(h)\ge V^{\pi}(h), so

Q~(h,z)=E[rq+W(h′)] ≥ E[rq+Vπ(h′)]=Qπ(h,z),\widetilde Q(h,z)=\mathbb{E}[r_q+W(h')]\ \ge\ \mathbb{E}[r_q+V^\pi(h')]=Q^\pi(h,z),

where VπV^\pi and WW are continuation values from the given history, rqr_q is the utility accrued during the exchange itself, net of its interaction cost, and h′h' includes the answer and the updated budget. The single-step label is therefore a lower bound on the value of asking, while passing yields exactly Vπ(h)V^{\pi}(h) because PASS is final. Making PASS final thus keeps the rule conservative. The controller asks only when a single targeted question is worthwhile on its own.

The rule can miss gains that need several coordinated queries, and it leaves needs arising after PASS to the task policy. A re-entrant variant handles the latter by consulting the controller at every later eligible turn, including after tool observations, without retraining.

MethodAirlineRetailTelecom
GRPO24.33 ± 3.2327.50 ± 4.3324.17 ± 2.89
CLEAT (controller stops after PASS)35.00 ± 2.0032.50 ± 1.2924.94 ± 1.77
CLEAT (re-entrant controller)34.33 ± 2.3132.83 ± 1.7626.11 ± 2.04

The re-entrant variant performs comparably on Airline and Retail and reaches a higher mean on Telecom, where what to ask often depends on earlier diagnostic results. We will bring these operational details together in Section 3.5 and align Appendices A.2, B, and E with this description.


For Q3: Training budgets and environment interactions

CLEAT's schedule follows Appendix B, with a controller bootstrap, 120 co-training rounds of 16 histories each, and a final controller refresh, using five continuations per history. All continuations are generated afresh under the current policy and run until the episode ends. All training baselines use the same task pool, backbone, user simulator, and turn limits as CLEAT. Each RL baseline keeps the sampling procedure of its original method, and we aligned the total number of training rollouts and policy updates with CLEAT's schedule. IG-Reward, for instance, keeps the Qwen3-4B task agent and trains a separate Qwen3-1.7B clarifier with DAPO on individual clarification decisions.

Because CLEAT focuses on guiding experience collection, the comparisons that isolate its contribution are controlled exactly. The w/o guidance variant keeps the same per-history budget, paired comparisons, and inference rule and differs only in the value tilt, and it lowers scores by 1.97 to 8.33 points (Table 3). Menu-GRPO also uses the same per-history budget. We will document the schedule in Appendix B and each baseline's sampling unit and budget in Appendix C.2.


We thank the reviewer again for the careful reading and thoughtful feedback. We will clarify the rationale for value-guided collection, the controller’s sequential operation, and the training-budget comparisons. We will also improve the organization and wording to make the learning process and experimental setup clearer and easier to follow.

Thank you for your response. I will maintain my current score. Good luck to you.

We sincerely thank you for taking the time to read our rebuttal and for your valuable feedback, which has helped us improve the paper. We will incorporate the additional experimental results and clarifications into the revised manuscript. If any concerns remain, we would be happy to address them during the discussion period.

Reviewer_fbYWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CLEAT, a framework for learning when and what an interactive agent should clarify. It defines residual query value as the expected utility gained by asking a targeted question before continuing, relative to letting the current task policy continue on its own. Paired rollouts provide supervision for both a clarification-value controller and the task policy. The controller guides subsequent data collection, while renewed supervision tracks changes in the policy. The authors evaluate CLEAT across four benchmarks and ten settings, reporting improvements in task performance and clarification quality. Ablations examine controller updates, value-guided collection, and controller use at inference.

Critical strengths

A worthwhile problem with a coherent learning formulation. The paper makes a useful distinction between obtaining information and obtaining information that improves the current agent’s decisions. Defining clarification value relative to the policy that will use the answer provides a clear rationale for updating value estimates as the policy learns. This gives the method a concrete learning objective and a consistent motivation.

Solid experimental design and meaningful ablations. The evaluation covers several types of interaction tasks and separately examines the effects of updating the controller, guiding training-time collection, and using the controller at inference. The menu-augmented comparisons also help distinguish the learning procedure’s contribution from benefits introduced by the structured interface alone. Overall, the experiments provide a reasonably complete empirical assessment and support my positive view of the paper.

Critical weaknesses

None that I currently consider sufficient to warrant rejection. My remaining reservations concern the rationale for separating the controller from the task policy and the dependence on fixed clarification interfaces. These are clarification requests rather than established methodological flaws.

Questions for authors
  1. Rationale for a separate controller. I understand that the controller and task policy are co-trained using shared interaction outcomes. However, could the authors explain the advantages of maintaining them as separate modules rather than sharing a representation and optimizing them jointly within a single model? Is the separation important for stability, efficiency, or learning effectiveness? I would also appreciate clarification on whether differences in their update objectives or schedules can create a mismatch between value estimation and policy behavior.

  2. Role of the fixed clarification interfaces. The fixed candidate constructor, question realizer, and QUERY/PASS interface impose considerable structure on the agent’s clarification behavior. The menu-augmented comparisons address part of this concern, but which structural choices are essential to the method, and which primarily serve as experimental controls? How do these choices limit the scope of the learned clarification behavior, particularly when useful questions fall outside the supplied candidate set?

Overall, I am inclined toward acceptance given the coherent approach, solid evaluation, and completeness of the work. The authors’ explanations would help resolve these remaining questions; they may partly reflect gaps in my understanding of the implementation.

2 comments
AuthorsOpenReview ↗

Response to Reviewer fbYW

We thank the reviewer for the positive assessment of our policy-relative formulation and experiments, and for the thoughtful questions on controller separation and the fixed clarification interfaces.


For Q1: Controller separation and consistency with the task policy

The controller and the policy share the same backbone and have separate trainable parameters. The controller reads frozen backbone features with LoRA disabled and adds about 0.67M parameters. We keep it separate so that value learning stays orthogonal to policy optimization. The controller influences policy training only through rollout allocation, its predictions are never used as reward, and the sampling correction preserves the expected unclipped task-return gradient (Proposition 1). It therefore acts as a plug-in that can in principle accompany other policy-gradient methods. Joint optimization over a shared trainable representation would give up this property, because the value loss would update the policy directly. Separation also gives stable features as the policy evolves, value fitting that needs no backpropagation through the backbone, and a controller that can be refreshed with the policy held fixed. The design thus serves modularity and learning effectiveness first, with stability and efficiency as further benefits.

The separation does not create a mismatch in objectives, because both modules learn from the same paired outcomes. For each query arm, the mean policy advantage equals the controller's empirical RQV target. The schedules differ by a one-update lag, since a controller fitted to outcomes from πt\pi_t guides πt+1\pi_{t+1}. Proposition 1 holds for any controller, so this lag cannot bias the policy update, and Proposition 2 keeps the correction weights below 1.25 regardless of the controller's predictions. Co-training renews the labels in every round, and the final refresh fits the controller to the deployed policy. Freezing the controller after bootstrap lowers mean scores by 1.27 to 6.70 points (Table 3).

We also evaluated the joint design the reviewer describes by back-propagating the value loss into the policy's LoRA parameters, with all other settings unchanged.

MethodIN3 Judg.AccAskMind AccAskOv. Acc
CLEAT (separate controller)83.95 ± 1.4248.37 ± 2.6661.27 ± 2.19
Shared representation (value loss into LoRA)80.25 ± 1.9346.12 ± 2.4159.40 ± 2.31

The jointly optimized variant remains effective but reaches lower means in all three settings, consistent with the benefit of stable features and independent updates. We will clarify this rationale and the update schedule in Section 3.5 and Appendix B.


For Q2: Essential structure and the scope of fixed clarification interfaces

The essential element of CLEAT is the paired comparison between a targeted clarification and native continuation from the same history. The algorithm also needs a finite candidate set, built from the visible history and held fixed within each comparison, which supports menu selection, coverage sampling, and computable inclusion probabilities for the sampling correction. The remaining choices are implementation details or experimental controls. The specific constructor, prompt, and menu size can be replaced. On Travel, substituting the predefined schema with the frozen proposer changes Pass-1 from 63.72% to 62.21%. We freeze the constructor and question realizer to isolate improvements in target selection and answer use, and Table 4 shows that the gains come from the learning procedure when these interfaces are shared across methods.

The menu limits which questions the controller can select, but not which questions the agent can ask. If a useful question is missing from the menu, the policy remains free to ask it once the controller passes control. The controller does not suppress such questions either, because RQV measures each menu question against what the policy would do on its own, so the controller steps in only when a menu question is expected to do better. Measured against the benchmarks' annotated required points, which the constructor never sees, the initial menu covers 76.8% on IN3, 66.5% on AskMind, and 71.2% on AskOverconfidence (judged by Qwen3-30B-A3B). One trade-off is that questions outside the menu are learned from task returns alone, without value guidance. As discussed in Section 6, an adaptive constructor could bring further targets into value-guided selection, and a learned realizer could extend optimization to question wording. We will clarify the role of the fixed interfaces in Section 3.1 and Appendix A.2.


We thank the reviewer again for the thoughtful feedback. We will clarify the rationale for controller separation and how its updates track the evolving task policy. We will explain the role of the fixed clarification interfaces and the scope of the learned behavior more clearly.

AuthorsOpenReview ↗

Thank You and Follow-up on Our Response

Dear Reviewer,

Thank you again for your thoughtful feedback. We have provided detailed responses and additional experiments addressing your questions.

In particular, we implemented the joint design you described by back-propagating the value loss into the policy's LoRA parameters, and the separate controller achieves higher means in all three settings (e.g., 83.95 vs. 80.25 on IN3 judgment accuracy). The separation makes the controller a lightweight plug-in with only about 0.67M parameters, and Proposition 1 shows that the expected unclipped task-return gradient is preserved for any controller, so the controller and the policy remain consistent. The essential element of CLEAT is the paired comparison between a targeted clarification and native continuation, so the constructor, prompt, and menu size can all be replaced. Finally, the menu only defines the controller's candidates, and the agent remains free to ask any other question on its own.

We would greatly appreciate it if you could take a look and let us know whether these results address your concerns. We would be glad to clarify any remaining questions during the discussion period.

Best regards,

The Authors

Reviewer_ysT7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces CLEAT (CLarification through valuE-guided Agent co-Training), a framework that jointly trains an LLM task policy and a lightweight (~0.67M parameter) clarification-value controller to decide when and what clarifying questions to ask a user when instructions are underspecified or contain false premises. Motivation: Prior work often guides clarification using static information gain (IG) or fixed Value-of-Information estimates. However, an informative answer does not always change the agent's task decision, and as the agent's policy improves during training, both its ability to proceed without asking (PASS) and its ability to use a user's answer (QUERY) shift at different rates—making the incremental benefit of a question policy-relative and non-stationary.

Implementation: CLEAT defines Residual Query Value (RQV) as the expected task-utility gain of asking about a candidate target (supplied by a fixed constructor and phrased by a fixed question realizer) versus letting the current policy proceed natively via PASS. At each clarification-eligible history, CLEAT collects 5 paired continuations from the same state and simulated user goal: 2 PASS rollouts, 2 rollouts for a controller-prioritized target, and 1 rollout for a without-replacement coverage target. The mean utility difference between QUERY and PASS rollouts trains the controller via MSE, while PASS-referenced advantages (with a leave-one-out baseline for PASS arms) and Horvitz–Thompson inclusion weights update the LoRA task policy. At inference, the controller triggers the top-scoring query if its predicted RQV is positive and hands off to PASS otherwise.

Main Learnings:

  • Across four benchmarks (10 settings across UserGym, τ²-Bench, IN3, and AskBench) on Qwen3-4B, as well as scaling runs at 1.7B and 8B, CLEAT consistently outperforms prompting, uncertainty/VoI thresholding, information-based RL, outcome-based training, SFT, and GRPO baselines.
  • Ablations show that online co-training of the controller outperforms freezing it after bootstrap, RQV-guided rollout collection outperforms IG-guided collection (which actually degrades multi-turn tool use on τ²-Bench Airline and Telecom under shared turn budgets), and the trained policy retains substantial gains even when the controller is removed at test time.
Critical strengths

*Measures clarification utility against native continuation (PASS) under the current policy rather than static information gain, preventing over-asking on shared turn budgets. *Uses a shared PASS baseline across arms (and leave-one-out for PASS rollouts) so the mean policy advantage equals the controller's RQV regression target without group standard-deviation distortion. *Pairs the value-tilted draw with a without-replacement coverage draw to bound Horvitz–Thompson weights at 1.25 (Proposition 2). The mathematical and algorithmic design fits together cleanly from first principles rather than relying on ad-hoc reward shaping.

Critical weaknesses
  • relies on a frozen target constructor and fixed question realizer whose tokens are masked from the policy loss (Appendix A.2, lines 710–720).
  • Line 711 states the constructor uses predefined task-interface fields where available and a frozen model otherwise. Yet Appendices B, C, and E never specify which benchmarks use predefined fields, or whether Table 1 baselines also see those fields.
  • At inference, choosing PASS permanently disables the controller for the rest of the episode (Section 3.5, line 304). In tool-use tasks like τ²-Bench, an agent often needs to run a lookup tool (PASS) first before discovering what to clarify.
Questions for authors
  • Which of the ten benchmark settings use predefined information fields from the task interface for the candidate target set versus the frozen-model target-construction prompt in Appendix E? Where predefined fields are used, are those same fields visible in the prompts of the baselines in Table 1?

  • How is the fixed pool of 2,000 clarification-eligible histories in Appendix B (lines 1080–1082) constructed, are paired rollouts branched only at the initial user turn, or also from intermediate multi-turn states, and did the online RL baselines in Table 1 (GRPO, InfoPO, IG-Reward, IGPO) receive the same budget of 10,000 environment continuations?

3 comments
AuthorsOpenReview ↗

Response to Reviewer ysT7 (Part 1/2)

We thank the reviewer for recognizing CLEAT’s policy-relative formulation of clarification utility, its principled algorithmic design, and the empirical benefits of online co-training, and for raising important questions about our experimental setup and the scope of controller intervention.


For W1: Fixed clarification components and the scope of policy learning

CLEAT learns whether to ask, which information to request, and how to act on the answer. The policy loss covers the semantic choice between PASSPASS and QUERY(z)QUERY(z) and all subsequent agent-generated tokens (Section 3.4, Appendix B). Only the realizer's question text and the user's replies are masked, because the task policy does not generate them. The choice of zz still receives gradient through the semantic-action term, and questions the policy asks on its own remain agent-generated tokens and are trained. Keeping the constructor and realizer fixed attributes improvements to clarification selection and answer use rather than to changes in candidate generation or question wording. To test whether these fixed components alone explain the gains, Menu-Vanilla and Menu-GRPO (Appendix D.1) use the same frozen constructor and realizer as CLEAT and let the policy select menu actions without controller scores (Table 4, reproduced below).

Qwen3-4B, mean over three seeds (%).

MethodTravel Pass-1IN3 RecoverIN3 Judg.AccAskOv. AccAskOv. Cov
Vanilla26.7454.2665.7457.2440.98
Menu-Vanilla28.3555.1366.1857.7841.75
GRPO42.2653.6866.0558.6743.50
Menu-GRPO44.8358.4770.0359.1744.58
w/o guidance56.1764.4978.9557.2847.70
CLEAT (full)63.7267.5883.9561.2751.49

With the identical menu and question template, CLEAT outperforms Menu-GRPO on all five metrics, with gains of 18.89 points on Travel and 13.92 points on IN3 judgment accuracy. The improvements therefore come from paired-outcome learning and value-guided co-training rather than from the shared interface. As discussed in Section 6, the fixed components define the set of clarifications available to the controller, and learning target construction jointly with the policy is a natural extension.


For W2 and Q1: Benchmark-specific target construction and baseline information access

Only UserGym Travel uses predefined information fields, taken from the 24-field schema of the official dataset. The other nine settings use the frozen proposer. The field names are not privileged information, since they also appear in the environment's observations available to the Table 1 baselines. At each history, the constructor selects up to six relevant, unresolved fields from the visible context, with the same rule in training and inference. In all ten settings, target construction uses only the visible history and public interface information, never hidden user goals, rollout outcomes, or rewards. To measure the dependence on predefined fields, we replaced them on Travel with the frozen proposer in both training and inference.

MethodTravel Pass-1 (%, mean ± std)
Vanilla26.74 ± 1.49
GRPO42.26 ± 0.94
SFT (strongest baseline)50.48 ± 1.76
CLEAT with predefined fields63.72 ± 0.48
CLEAT w/o predefined fields62.21 ± 0.69

Without predefined fields, CLEAT reaches 62.21% and remains 11.73 points above the strongest baseline, so the gain on Travel does not depend on the predefined schema. We will specify the construction rule for each benchmark and the baselines' field access in Appendices A.2, B, C.2, and E, and add this comparison to Appendix D.1.

AuthorsOpenReview ↗

Response to Reviewer ysT7 (Part 2/2)

For W3: PASS semantics and clarification after tool use

PASSPASS hands the remaining interaction to the task policy and does not correspond to a single tool call. After PASSPASS, the policy keeps its own ability to ask, so it can perform a lookup first and then request clarification based on the result. PASSPASS continuations during training include such interactions, and all subsequent policy actions are trained on the full task return. This handoff is the native-continuation reference Vπ(h)V^{\pi}(h) in RQV: each label compares asking one targeted question first against handing control over immediately, so stopping at the first PASSPASS keeps inference consistent with the controller's supervision.

We also evaluated a re-entrant variant that matches the reviewer's scenario. The controller is consulted again at every later clarification-eligible turn, including after tool observations, and the trained policy and controller are used without retraining.

MethodAirlineRetailTelecom
GRPO24.33 ± 3.2327.50 ± 4.3324.17 ± 2.89
CLEAT (controller stops after PASS)35.00 ± 2.0032.50 ± 1.2924.94 ± 1.77
CLEAT (re-entrant controller)34.33 ± 2.3132.83 ± 1.7626.11 ± 2.04

The re-entrant variant performs comparably on Airline and Retail and reaches a higher mean on Telecom. In Airline and Retail, the user knows the needed information from the start, such as which order to change, so the opening questions already cover it and later questions only compete with tool calls for the turn budget. Telecom involves step-by-step troubleshooting, where what to ask depends on earlier diagnostic results, so later consultation helps. We will state the PASSPASS semantics consistently in Section 3.5 and Appendix E and add this variant to Appendix D.


For Q2: Training-history construction, branching points, and rollout budgets

The pool consists of starting histories from the training split, and all continuations are generated afresh under the current policy. Most starting histories are initial user requests, and a small number are nonterminal histories after the first or second interaction round, so paired rollouts branch from both initial and early intermediate states. Each branch then runs until the episode ends. The schedule follows Appendix B, with a controller bootstrap, 120 co-training rounds of 16 histories each, and a final controller refresh, using five continuations per history. All training baselines use the same task pool, backbone, user simulator, and turn limits as CLEAT. Each RL baseline keeps the sampling procedure of its original method, and we aligned the total number of training rollouts and policy updates with CLEAT's schedule. IG-Reward, for instance, keeps the Qwen3-4B task agent and trains a separate Qwen3-1.7B clarifier with DAPO on individual clarification decisions, so its rollouts are shorter than CLEAT's full-episode continuations. The comparisons that isolate CLEAT's contribution are controlled exactly. Menu-GRPO and the w/o guidance variant use the same per-history budget as CLEAT (see our response to W1). We will document the schedule in Appendix B and each baseline's sampling unit and budget in Appendix C.2.


We thank the reviewer again for the valuable feedback. We will incorporate these clarifications into the revised paper to better explain the clarification interface, experimental protocol, baseline comparisons, and scope of controller intervention.

AuthorsOpenReview ↗

Thank You and Follow-up on Our Response

Dear Reviewer,

Thank you again for your constructive feedback. We have provided detailed responses and additional experiments addressing your main concerns.

In particular, with the identical menu and question template, CLEAT outperforms Menu-GRPO on all five metrics (e.g., +18.89 points on Travel), so the fixed constructor and realizer do not explain the gains. Replacing the predefined fields on Travel with the frozen proposer still gives 62.21%, 11.73 points above the strongest baseline. We also clarified that the policy can still ask after PASSPASS, and a re-entrant controller consulted after tool observations performs comparably on Airline and Retail with a higher mean on Telecom. Finally, we detailed the training-history construction and rollout budgets for all baselines.

We would greatly appreciate it if you could take a look and let us know whether these results address your concerns. We would be glad to clarify any remaining questions during the discussion period.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.