- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Evaluating a player’s performance can require many games because the observed results are noisy. This paper studies how uncertainty in heuristic predictions can help reduce this noise in AIVAT estimates. The proposed method gives more weight to estimates with lower predicted uncertainty. Experiments on poker and Goofspiel report lower standard errors. In particular, the authors estimate that the poker evaluation needs 43% fewer samples than uniform averaging with the same Gaussian process model. The paper also shows that choosing heuristics after seeing the evaluation data can lead to opposite conclusions about the same players. This finding highlights the need to fix the heuristic before using the data to evaluate performance.
Critical strengths
The paper addresses a useful problem: evaluating game-playing agents with fewer samples. It proposes a clear way to use uncertainty and reports lower standard errors in poker and Goofspiel. It also gives concrete examples showing how reusing evaluation data can produce misleading results, offering practical guidance for reliable evaluation.
Critical weaknesses
(i) The link between uncertainty in the heuristic and the actual variation across games is not fully justified. This leaves the statistical basis of the proposed weights unclear to me.
(ii) Although the paper acknowledges possible bias, its claim that Gaussian processes avoid this bias needs stronger support. Lower standard errors alone do not guarantee better accuracy.
(iii) I think the claim in lines 284–288 that Gaussian processes give unbiased weighted estimates appears incorrect without further assumptions. The following example also raises a concern about applying Proposition 1 to the proposed weights.
Consider the MIVAT special case with one chance event leading to three states, each with probability , followed by deterministic payoffs
Suppose the GP model is fixed before evaluation and has posterior mean and covariance
for the heuristic outputs at these states. This is a valid Gaussian distribution on a finite set of states. Using the posterior mean as the heuristic, the MIVAT estimate at state is
\qquad p=(1/3,1/3,1/3)^\top.$$ Let $e_i$ be the $i$th coordinate vector. The correction coefficient is $c_i=p-e_i$. Applying the paper's uncertainty propagation and weighting rules gives $$s_i^2=c_i^\top\Sigma c_i, \qquad (s_1^2,s_2^2,s_3^2)=(4,4,2), \qquad (w_1,w_2,w_3)=(1/4,1/4,1/2).$$ For independent game histories $Z_1,\ldots,Z_T$, the weighted estimate therefore satisfies, almost surely, $$\hat{\mu}_{T} = \frac{\sum_{t=1}^{T} w_{Z_t}\hat{v}(Z_t)}{\sum_{t=1}^{T} w_{Z_t}} \longrightarrow \frac{E[w_Z\hat{v}(Z)]}{E[w_Z]} = \frac{(1/2)\cdot 3}{1/4+1/4+1/2} = \frac{3}{2}.$$ Thus, the estimate converges to $1.5$, although the true expected payoff is $1$. The bias does not disappear as the sample size increases. The weighting can also increase sampling variance. With two independent games, the weighted estimate takes values $0$, $2$, and $3$ with probabilities $4/9$, $4/9$, and $1/9$, respectively. Hence, $$\mathbb{E}[\hat\mu_2]=\frac{11}{9}, \qquad \operatorname{Var}(\hat\mu_2)=\frac{104}{81}>1.$$ By comparison, uniform averaging gives $$\mathbb{E} \left[\frac{\hat v(Z_1)+\hat v(Z_2)}{2}\right]=1, \qquad \operatorname{Var} \left[ \frac{\hat v(Z_1)+\hat v(Z_2)}{2} \right]=1.$$ Could the authors explain which stated assumption rules out this example? In particular, why does the GP property cited in lines 284–288 guarantee unbiasedness when evaluating a fixed game? Why does Proposition 1 apply when the weights use uncertainty in the heuristic, rather than the sampling variance across games? If the intended guarantee instead averages over a Bayesian model of the game's payoffs, please state this explicitly and explain how it supports the experimental conclusions.Questions for authors
As shown above in the Weekness
3 comments
Thank you for your response and for clarifying that the estimator can be biased for a fixed game. However, my main concerns remain. It is still unclear why uncertainty from the GP can be used as the variance across games, and therefore why Proposition 1 applies to the proposed weights. The explanation based on a Gaussian prior also needs a clear connection to the experimental setting. Moreover, lower reported standard errors and a small estimated bias do not provide enough evidence of better accuracy. So, I will keep my original score.
There still seems to be some misunderstanding about the main result (demonstrated better accuracy)
Thank you for your continued engagement!
It seems to us that there is still some misunderstanding regarding our accuracy results, as indicated by the reviewer writing "Moreover, lower reported standard errors and a small estimated bias do not provide enough evidence of better accuracy.". Accuracy can be quantified by the mean squared error of the estimator (which equals mathematically the variance plus the squared bias); in our paper, our estimator is shown to have higher accuracy. For poker, one can see this in Table 1: our 75^2 + 3^2 < 85^2 < 88^2. Analogously with the identical mathematical comparison, Table 5 shows that our technique has higher accuracy in the Goofspiel variants as well. We will spell this math out explicitly in the text for both tables in the final version of the paper so the reader does not have to do the math themselves; we take responsibility for not spelling it out explicitly in the submitted version. In summary, it seems that either the reviewer is misunderstanding this or we are missing something about what the reviewer is trying to say (e.g., perhaps the reviewer uses "accuracy" to mean something else than mean squared error of the estimator). Can you please help clarify? Or, if our response above helps clarify this, please consider increasing your overall score accordingly.
Thank you again for the discussion!
Thank you for your review.
The link we propose is that the heuristic value function can have different confidence in its estimates (e.g., for poker, folded hands, showdown hands, big pots). In our poker experiments, we found that the hands that reached later stages received less weight, and the weights did not depend much on the pot size.
The mean squared error is the variance plus the squared bias, so lowering the variance at the cost of small bias usually helps the estimator achieve higher accuracy. In our main experimental results on poker and Goofspiel, we demonstrated empirical evidence that our technique increases the accuracy.
See also our answer to your next question.
Proposition 1 applies when the heuristic uncertainty reflects the actual variances of the estimates, which does not hold in the reviewer's example. In the example, the model is most confident where it is most wrong.
Proposition 2 in the paper predicts the reviewer's example: an asymptotic bias of 1/2. The zero-covariance property in Lines 284-288 holds under the Gaussian prior, i.e., on average over the model of the game's payoffs. For a fixed game, the estimator can still be biased, which is precisely the tradeoff we stated in our abstract. Proposition 2 at least gives the asymptotic bias, and we estimate and report it for every IVW result, in which we found the bias to be small in our main results.
We will sharpen the wording in the paper. In particular, we will tone down the remark after Proposition 2.
We hope the rebuttal adequately addresses your concerns.