- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper works on the off policy evaluation of dialogue controllers that choose high-level acts on top of a fixed utterance realizer shared by all controllers, where with known logging propensities can reweight at the level of acts to identify the target value. The paper then proposed IM-DR, which is a marginalized doubly robust estimator with two components: Occupancy weights and A Monte Carlo Q control variate.
Critical strengths
- This paper is generally well motivated, with clear definition of when act-level reweighting is valid and what it excludes.
- The empirical sections and findings are carefully designed and presented; exploratory findings and confirming findings are separated, with good experiment designs like seed usage and intervals paired and clustered by seeds, as well as how estimators variants are chosen.
- Experiment details are properlly documented, with hyper parameters and seeds and code provided
Critical weaknesses
-
The novelty are very limited, with the intent component not really supported. Given that the author claims IM-DR is generic and standard marginalized DR estimator applied to particular state, the contribution has to come from the choice of state obviously which it self is built around the intent belief. However the experiment results did not show that intetnt feature really helps, based on Table 6, using only the turn index and the number of questions asked also works about as well as the full state. The “intent-aware” part seems to show no measurable effect.
-
The paper is motivated by user simulators being unreliable, but later in the paper the density ratios are fitted on rollouts from the user model. Under this setting when the model is wrong, the weights and the control variate are also wrong with it and as a result, the estimator loses the robustness that DR is supposed to provide. Also around the logging propensities issue, there are also works on averaging the cumulative importance ratio within each abstract state estimation which then no longer need a user model.
-
The testing set up seems to favor the method making the experiment unfair, e.g. the policies are functions of the same entropy that the abstract state is built on, and Q-estimate critics can see the true intent. Nevertheless, it seems simply running the role-play simulator already ranks the policies better than IM-DR does.
-
Most of the theory contribution are decorative with no optimality proof or connection to the main method on prediction or explaining the performance.
Questions for authors
See weakness.
-
What is the distribution of the fitted weights in the primary run? What is the MSE value for the variants used?
Reviewer nv4p: Intent Ablations, Coupled Model Errors, Logged-Ratio Comparison and Weight Diagnostics
We apologize for our delayed response. Thank you for your questions about the representation, coupled nuisances, simulator ranking and fitted weights. Appendix H now provides matched fresh-seed comparisons and a primary-weight replay addressing these points.
Intent features and novelty. IM-DR instantiates marginalized DR with an intent/progress representation. Its contribution is the shared-realizer formulation and the analysis connecting user-model, ratio, posterior and abstraction errors to evaluation bias, together with the conditional variance analysis. The new ablations directly examine the role of the representation.
In the new 50-seed, three-task check, full/progress-only/posterior-only MSEs are 0.0925/0.0914/0.0913 for primary targets and 0.1087/0.1029/0.1010 under shift (all MSEs in 10−2 units). Full minus progress-only is +0.0011 [−0.0012, +0.0035] and +0.0058 [−0.0013, +0.0128], respectively. Under shift, posterior-only improves on full by 0.0076 [0.0018, 0.0135]. We report these findings in Appendix H.1: the comparisons do not resolve an incremental benefit from intent over progress alone, and the preferred representation can depend on the target family.
Shared user model and logged-ratio comparator. The fitted weights and Q share a user model, so misspecification can affect both. The bias analysis accounts for this interaction; the population DR identity applies when either nuisance satisfies the stated correctness conditions. Following your suggestion, we added a comparator that averages cumulative act-importance ratios within observed joint abstract-state/action cells, clips at 20, and uses the same Q as IM-DR. Its weights require no user-model rollouts. Its primary/shift MSEs are 0.1281/0.4501, versus 0.0925/0.1087 for full IM-DR. The paired differences are −0.0357 [−0.0470, −0.0253] and −0.3414 [−0.4167, −0.2757]. This exploratory comparison examines the suggested weight construction with matched Q values.
Logged intent, ranking and the role of the theory. To examine access to true intent, the existing posterior-sampled-Q check replaces logged intent with samples from the stored posterior. Its MSE is 0.0866 versus 0.0876 for logged-intent Q, with paired difference −0.0010 [−0.0093, +0.0072]; the observed difference is unresolved. Regarding ranking, the role-play simulator performs better for primary policies in the matched new check: τ is 0.998 versus 0.869 for IM-DR, while value MSE is 12.2973 versus 0.0925. This distinguishes ranking performance from value-estimation accuracy. The theory explains the interaction of estimation errors and gives a conditional variance bound under fixed or independently fitted nuisances; it is not an optimality result.
Primary weights and variant MSEs. Replaying seeds 200–249 reproduces primary MSE 0.0773, τ 0.884, and coverage 0.950 at the reported precision. Table 13 reports all new variant MSEs; Table 15 gives the primary replay's weight diagnostics:
Entries average per-cell statistics over 150 task–seed cells per target. No replayed weight reaches the clipping threshold 20.
We have uploaded a revised PDF incorporating these changes. The updated code uses the same anonymous artifact link. Should you have any remaining concerns, we would be happy to discuss them further.