acceptodds
Under review as a conference paper at ICLR 2027

Intent-aware Off-policy Evaluation For Shared-Realizer Dialogue Controllers

Abstract

Asking for clarification can rescue an ambiguous request, but each extra question costs the user a turn. We study how to compare dialogue controllers that make this choice using logged conversations, without treating a user simulator as ground truth. The controllers share an utterance realizer, and their decisions can use a belief over user intent. Our estimator, IM-DR, weights a compact state built from that belief and conversation progress, then corrects a user model’s continuation value with logged rewards. We show how errors in the user model, density ratio, intent posterior and state abstraction enter the resulting bias. For the unnormalised estimator, we also establish a population double robustness identity. An oracle variance comparison holds with fixed independently fitted nuisance functions and bounded or clipped weights. On a synthetic dialogue testbed with Monte Carlo value references from 40,000 rollouts and 50 independent seeds, IM-DR has 0.23× the MSE of per decision importance sampling and 0.50× that of sequential DR using the same user model. Paired comparisons clustered by seed favour it over all four baselines. Its MSE advantage persists under a separate synthetic policy shift, but coverage of nominal 95% intervals falls to 0.760. Whether these results carry over to human dialogue requires future research.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
off-policy evaluation, doubly robust estimation, dialogue systems, reinforcement learning, intent uncertainty, user simulation, marginalized importance sampling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nv4pOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper works on the off policy evaluation of dialogue controllers that choose high-level acts on top of a fixed utterance realizer shared by all controllers, where with known logging propensities can reweight at the level of acts to identify the target value. The paper then proposed IM-DR, which is a marginalized doubly robust estimator with two components: Occupancy weights and A Monte Carlo Q control variate.

Critical strengths
  • This paper is generally well motivated, with clear definition of when act-level reweighting is valid and what it excludes.
  • The empirical sections and findings are carefully designed and presented; exploratory findings and confirming findings are separated, with good experiment designs like seed usage and intervals paired and clustered by seeds, as well as how estimators variants are chosen.
  • Experiment details are properlly documented, with hyper parameters and seeds and code provided
Critical weaknesses
  • The novelty are very limited, with the intent component not really supported. Given that the author claims IM-DR is generic and standard marginalized DR estimator applied to particular state, the contribution has to come from the choice of state obviously which it self is built around the intent belief. However the experiment results did not show that intetnt feature really helps, based on Table 6, using only the turn index and the number of questions asked also works about as well as the full state. The “intent-aware” part seems to show no measurable effect.

  • The paper is motivated by user simulators being unreliable, but later in the paper the density ratios are fitted on rollouts from the user model. Under this setting when the model is wrong, the weights and the control variate are also wrong with it and as a result, the estimator loses the robustness that DR is supposed to provide. Also around the logging propensities issue, there are also works on averaging the cumulative importance ratio within each abstract state estimation which then no longer need a user model.

  • The testing set up seems to favor the method making the experiment unfair, e.g. the policies are functions of the same entropy that the abstract state is built on, and Q-estimate critics can see the true intent. Nevertheless, it seems simply running the role-play simulator already ranks the policies better than IM-DR does.

  • Most of the theory contribution are decorative with no optimality proof or connection to the main method on prediction or explaining the performance.

Questions for authors

See weakness.

  • What is the distribution of the fitted weights in the primary run? What is the MSE value for the variants used?

1 comment
AuthorsOpenReview ↗

Reviewer nv4p: Intent Ablations, Coupled Model Errors, Logged-Ratio Comparison and Weight Diagnostics

We apologize for our delayed response. Thank you for your questions about the representation, coupled nuisances, simulator ranking and fitted weights. Appendix H now provides matched fresh-seed comparisons and a primary-weight replay addressing these points.

Intent features and novelty. IM-DR instantiates marginalized DR with an intent/progress representation. Its contribution is the shared-realizer formulation and the analysis connecting user-model, ratio, posterior and abstraction errors to evaluation bias, together with the conditional variance analysis. The new ablations directly examine the role of the representation.

In the new 50-seed, three-task check, full/progress-only/posterior-only MSEs are 0.0925/0.0914/0.0913 for primary targets and 0.1087/0.1029/0.1010 under shift (all MSEs in 10−210^{-2} units). Full minus progress-only is +0.0011 [−0.0012, +0.0035] and +0.0058 [−0.0013, +0.0128], respectively. Under shift, posterior-only improves on full by 0.0076 [0.0018, 0.0135]. We report these findings in Appendix H.1: the comparisons do not resolve an incremental benefit from intent over progress alone, and the preferred representation can depend on the target family.

Shared user model and logged-ratio comparator. The fitted weights and Q share a user model, so misspecification can affect both. The bias analysis accounts for this interaction; the population DR identity applies when either nuisance satisfies the stated correctness conditions. Following your suggestion, we added a comparator that averages cumulative act-importance ratios within observed joint abstract-state/action cells, clips at 20, and uses the same Q as IM-DR. Its weights require no user-model rollouts. Its primary/shift MSEs are 0.1281/0.4501, versus 0.0925/0.1087 for full IM-DR. The paired differences are −0.0357 [−0.0470, −0.0253] and −0.3414 [−0.4167, −0.2757]. This exploratory comparison examines the suggested weight construction with matched Q values.

Logged intent, ranking and the role of the theory. To examine access to true intent, the existing posterior-sampled-Q check replaces logged intent with samples from the stored posterior. Its MSE is 0.0866 versus 0.0876 for logged-intent Q, with paired difference −0.0010 [−0.0093, +0.0072]; the observed difference is unresolved. Regarding ranking, the role-play simulator performs better for primary policies in the matched new check: τ is 0.998 versus 0.869 for IM-DR, while value MSE is 12.2973 versus 0.0925. This distinguishes ranking performance from value-estimation accuracy. The theory explains the interaction of estimation errors and gives a conditional variance bound under fixed or independently fitted nuisances; it is not an optimality result.

Primary weights and variant MSEs. Replaying seeds 200–249 reproduces primary MSE 0.0773, τ 0.884, and coverage 0.950 at the reported precision. Table 13 reports all new variant MSEs; Table 15 gives the primary replay's weight diagnostics:

TargetMeanp50p90p99Mean cell maximumESS/steps
P10.9420.9581.0041.2701.4250.991
P21.0060.9771.3112.0092.1500.928
P30.9180.9490.9901.0341.0810.991
P40.8800.9570.9961.1431.2000.962

Entries average per-cell statistics over 150 task–seed cells per target. No replayed weight reaches the clipping threshold 20.

We have uploaded a revised PDF incorporating these changes. The updated code uses the same anonymous artifact link. Should you have any remaining concerns, we would be happy to discuss them further.

Reviewer_VYVJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to evaluate clarification strategies from logged conversations, motivated by the cost of human studies, bias in user simulators, and the high variance of traditional importance sampling. To address these issues, the authors propose IM-DR, which uses summaries of the intent posterior and conversation progress to construct a compact state. Specifically, for controllers that share an utterance realizer, IM-DR combines occupancy weights with logged-reward residuals to correct continuation values predicted by a user model. Experiments cover three synthetic tasks, four controllers, and 50 random seeds, using large-scale Monte Carlo rollouts as value references. IM-DR reduces mean squared error by approximately 77% compared with per-decision importance sampling and 50% compared with sequential DR using the same user model. Its accuracy advantage persists under a synthetic policy-family shift, but empirical coverage of nominal 95% intervals falls to 76%. These results support its effectiveness in the synthetic testbed, while its applicability to human dialogue and the reliability of its uncertainty estimates remain open. In summary, its main contribution is applying marginalized doubly robust evaluation to this setting and analyzing how user-model, density-ratio, intent-posterior, and state-abstraction errors affect the estimate.

Critical strengths
  1. Research Background: This paper focuses on offline evaluation of when to clarify and when to act, balancing task performance against interaction costs, which makes the research practically relevant.
  2. Theoretical analysis: This paper clearly distinguishes errors in the user model, density ratios, intent posterior, and state abstraction, and establishes double robustness and conditional variance bounds.
  3. Method: IM-DR has a clear structure: it constructs a compact state from summaries of the intent posterior and conversation progress, then combines occupancy weights, user-model predictions, and residual corrections from logged conversations into an easy-to-follow evaluation procedure.
  4. Experiments: The experiments cover multiple synthetic tasks, 50 random seeds, and independent confirmatory tests. Paired comparisons with seed-clustered confidence intervals strengthen the evidence for the reported performance gains.
Critical weaknesses
  1. In lines 78–79, the authors use four reward components: task quality, user satisfaction, conversation length, and unnecessary questions. These components intuitively align with the paper’s goal of balancing task performance against interaction costs. However, the choice of weights is not sufficiently justified, and no sensitivity analysis is provided, making it difficult to assess how policy rankings and estimation performance depend on the chosen weights.
  2. In lines 166–168, the authors discretize posterior entropy into 8 bins and the MAP joint-intent probability into 4 bins, and cap the number of questions asked at 3. However, these choices are not sufficiently justified. Although Tables 5 and 6 compare different feature combinations, they do not examine the effects of the number of bins or the cap on question counts. A sensitivity analysis would help assess whether estimation performance depends on these particular state-discretization settings.
  3. In Table 12, interval coverage for the constant-asking and front-loaded-asking controllers is only 56.7% and 74.0%, respectively, well below the nominal 95%. Although the authors acknowledge this limitation, the reported coverage and effective sample size are insufficient to determine whether the shortfall mainly arises from estimation bias or underestimated uncertainty. Reporting bias, empirical standard deviation, and average interval width, alongside a comparison of different interval construction methods, would help explain the drop in coverage.

If these concerns are adequately addressed, I would consider raising my score.

Questions for authors

I have no specific questions.

2 comments
AuthorsOpenReview ↗

Reviewer VYVJ: Reward and Discretisation Sensitivity, Bias and Interval Coverage

We apologize for our delayed response. Thank you for the concrete requests concerning reward weights, discretisation, and interval coverage. We completed the requested sensitivity checks and a fixed-fit versus refitted-ratio bootstrap comparison, each with 50 fresh seeds across all three tasks. Appendix H reports the protocols and tables separately from the original confirmatory run.

Reward weights (Appendix H.2, Table 14). The default weights express the intended trade-off between task outcomes and interaction costs. To assess sensitivity to this choice, we separately halved/doubled the turn and unnecessary-question penalties, keeping quality and satisfaction weights at 1. Each setting has its own 40,000-rollout reference. MSEs are in units of 10−210^{-2}.

SettingIM-DR MSESequential DR MSEIM-DR ranking τ
Default0.08220.15430.862
Half turn cost0.07860.14860.849
Double turn cost0.09020.16740.900
Half unnecessary-question cost0.08330.15470.844
Double unnecessary-question cost0.08130.15410.918

The paired MSE differences favour IM-DR over sequential DR in all five settings. Ranking accuracy varies with the utility, while the reference ordering remains P4 > P3 > P1 > P2 on all three tasks. These results provide a local sensitivity check around the default weights.

Discretisation (Appendix H.1, Table 13). We vary entropy bins 4/8/16, MAP bins 2/4/8, and question cap 1/3/5 one component at a time. Across these bin/cap variants, primary MSE is 0.0916–0.0935, versus 0.0925 for the default. Under shift, cap 5 increases MSE to 0.1192, versus 0.1087 for the default; the paired full-minus-cap-5 difference is −0.0105 [−0.0135, −0.0076]. Cap 5 slightly improves primary MSE. The comparison therefore shows modest primary sensitivity and a more noticeable dependence on the question cap under shift.

Coverage, bias, SD and interval width (Appendix H.3, Table 16). We now report each diagnostic separately for every task/controller. The comparison uses 1,000 fixed-fit bootstrap resamples and 200 conversation-level resamples that refit the ratio classifier. Aggregate coverage is 81.3% fixed-fit versus 80.0% refit, so refitting does not restore nominal coverage.

For constant asking on task A, bias is +0.0384, empirical across-seed error SD is 0.0241, and mean fixed/refit widths are 0.0790/0.0764 reward units; coverage is 48%/50%. Bias is comparable to the fixed interval's mean half-width. Across tasks, constant-asking coverage is 65.3%/64.0%; front-loaded coverage is 79.3% for both. These diagnostics indicate that persistent estimation error contributes to undercoverage, and that accounting for ratio-fit variability alone is insufficient. Both methods hold the user model, Q draws, target rollouts and Monte-Carlo reference fixed. The new comparison is reported separately from the original shift experiment.

We have uploaded a revised PDF incorporating these changes. Should you have any remaining concerns, we would be happy to discuss them further.

Thank you for the additional experiments. The equal-budget comparison addresses my concern about retries. With three attempts per profile, CoverSim produced conclusive tests for 23 of 25 profiles on the new grid, compared with 15 for the original simulator. The independent judge also found coverage gains for all three simulators, which reduces my concern that the improvements depend on the judge used during optimization.

I see value in TestTOD’s formulation of validated coverage as a testing objective. My remaining concern is how much CoverSim’s specific feedback design contributes to the gains. An ablation comparing feedback on specific profile violations with a general profile-consistency signal, using the same GEPA setup, would help isolate that contribution. On balance, I maintain my rating of 2.

Reviewer_HBMWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how to evaluate dialogue systems with only two actions - Ask or Act. The authors aim to estimate its performance from previously collected conversations. They artificially create datasets from permutation of limited users intents and uses four different policies for this dialogue systems. Their method, called IM-DR, combines an imperfect user simulator with real logged rewards. It summarizes each conversation using features such as uncertainty about the user’s intent and dialogue progress, estimates how likely different state-action pairs are under the new and old controllers, and then uses logged outcomes to correct errors in the simulator’s predictions. The main contribution is therefore an off-policy evaluation framework for comparing existing dialogue controllers.

Critical strengths

This paper introduces a dialogue-state abstraction for marginalized off-policy evaluation. The method represents each history using features designed to capture intent uncertainty and dialogue progress, including posterior entropy, MAP intent confidence, turn index, and the number of clarification questions asked. Therefore, the paper proposes a way to transform high-dimensional dialogue histories into a compact and interpretable state space, in which behavior and target policies can be compared through their state-action occupancy distributions. This makes density-ratio estimation more tractable and can reduce the variance caused by cumulative importance weights over long dialogue trajectories.

Critical weaknesses

The task formulation is highly synthetic and closely aligned with the proposed estimator. The finite intent space, entropy-based ASK/ACT policies, handcrafted reward, shared realizer, and access to true logged intents make the setting convenient for controlled OPE experiments, but substantially limit its external validity. Applying the framework to another dialogue domain would require rebuilding the intent ontology, posterior model, state abstraction, reward function, and user model. It is therefore unclear whether either the formulation or the reported gains would transfer to open-ended language interactions.

The proposed method evaluates only the high-level decision of whether to ask a question or act. However, an equally important component of a clarification policy is how the question is formulated which can strongly affect the user’s response and downstream task success. By assuming a shared, fixed realizer, the paper holds question generation constant and does not evaluate differences in question quality. This substantially narrows the scope and practical significance of the method.

Questions for authors
  1. The proposed framework evaluates only whether the controller should ASK or ACT, while holding the utterance realizer fixed. How could the framework be extended to policies that differ in which clarification question they ask or how the question is phrased? If the realizers differ across policies, does the action-level identification result still hold?

  2. What evidence supports the sufficiency of the proposed abstract state? Since the main policies are themselves functions of posterior entropy, which is also a central feature of the abstraction, could the reported advantage partly result from the alignment between the policy construction and the state representation?

1 comment
AuthorsOpenReview ↗

Reviewer HBMW: Shared-Realizer Scope, Question Quality and State Sufficiency

We apologize for our delayed response. Thank you for your questions about the realizer, state abstraction and applicability. We have clarified these points and added experiments in Appendix H, summarised in Section 7.1.

Shared realizer and question quality. Act-level identification requires the same conditional utterance distribution for logging and target controllers. If the controllers choose different questions or phrasings within an act, act-only reweighting generally no longer identifies the target value. One possible extension is to treat the pair (act, utterance) as the action and use the ratio

π′(a∣h)g′(y∣h,a)πb(a∣h)gb(y∣h,a)\frac{\pi'(a\mid h)g'(y\mid h,a)}{\pi_b(a\mid h)g_b(y\mid h,a)},

provided these probabilities can be evaluated and the joint actions have logging support. Section 4 and Appendix H.1 now explain this possible extension. The current experiments evaluate when to ask with a shared realizer; evaluating question quality and transfer to natural-language dialogue remains future work.

State sufficiency and policy alignment. The primary policies share entropy with the representation, so this alignment is relevant to interpreting the results. To examine the representation directly, we ran 50 fresh seeds across all three tasks for both primary and shifted target families, sharing continuation draws across state variants. All MSEs below are in units of 10−210^{-2}.

RepresentationPrimary MSEShift MSE
Full0.09250.1087
Progress only0.09140.1029
Posterior only0.09130.1010

The paired full-minus-progress differences are +0.0011 [−0.0012, +0.0035] and +0.0058 [−0.0013, +0.0128], respectively. These comparisons do not resolve an advantage from adding intent features. Under shift, posterior-only has lower MSE than full, with a paired difference of +0.0076 [+0.0018, +0.0135]. Appendix H.1 reports these results and clarifies that the state is an interpretable design choice whose sufficiency is an assumption, rather than an empirical conclusion.

We have uploaded a revised PDF incorporating these changes. Should you have any remaining concerns, we would be happy to discuss them further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.