- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper systematically studies the agent handover problem where a strong agent needs to figure out the best strategy when collaborating with a weak agent, because naively output the best action under the strong agent's policy is no longer the best move when the weak agent is not able to pick up the subsequent strategy on its own.
The paper provides a detailed theoretical analysis around handoff regret, and apply the proposed method in chess and navigation.
Critical strengths
- The paper presents theoretical contribution, and also have a discussion over prior work (Expector) under the theoretical framework of this work.
- The experiment is solid, with sufficient controls that well explain the source of gains. For example, the authors increase the compute for the baseline pair that does not use HAD's extra search and show that HAD still outperforms the pair.
Critical weaknesses
- The proposed HAD method is on-par with Expector, but this issue is partly mitigated by the fact that the theory in this paper helps explaining the why it works. I also appreciate the fact that the authors are honest about this.
- The method relies on opponent modeling, and in evaluations the authors use the same Maia model and Stockfish engine that the opponent pair actually uses. This is a strong assumption, and I would like to know how the system will perform when the opponent modeling is noisy.
Questions for authors
- Maybe I have missed it, but do you have any results on the robustness to opponent modeling?
Thank you for the review and for the direct question. Yes: the submission reports robustness to partner-model error on random MDPs (Appendix G, Table 14), and we have now run it in chess. With deliberately wrong partner and opponent models, the gain over the reference team holds in every condition.
W2 / Q1 (robustness to opponent and partner modelling).
New runs with the settings of Section 5.1: Maia-2 juniors truly at 1500, strict alternation, the 800 held-out opening pairs, HAD-4 against the reference team R (raw engine plus the same junior). The senior is given a wrong model of its own junior or of the opposing junior: Maia-2 at the wrong rating, or mixed with a uniform random legal move.
HAD-4 beats R under every wrong model, with every interval above 50%. Compared with the true-rating model in the same runs, the win share is 0.9 to 1.5 points lower, and no single difference is distinguishable from zero (Table 6 in Appendix D gives each with its interval). The last column is the mean total-variation distance between the assumed and the true move distribution at the positions where that junior moved. A rating error changes which moves the model ranks first; the uniform mixture keeps the ranking and flattens the probabilities.
The true-rating row is the paper's configuration, run alongside the wrong models on different machines and software builds from the submission; the submitted runs give 56.9 ± 2.2 on the same openings. In that environment we also reran all nine pre-registered comparisons: the pre-registered criterion holds again, with HAD-4 beating matched compute head to head at 57.3 ± 2.3 (submitted: 57.2 ± 2.3; Table 5).
Two further facts bear on the question. First, the senior's models were approximate in the submitted runs as well. It uses the top 4 moves of its own junior (0.83 of Maia-2's mass on average), the top 2 of the opposing junior, and a depth-10 reply for an opposing senior that searches at depth 12. In the head-to-head runs of Table 1 the opposing senior is in fact OA, the matched-compute engine or Expector, and HAD-4 scores 55.9 ± 2.3, 57.2 ± 2.3 and 50.7 ± 2.0 against them. Second, the theory has the same form: in Theorem 12 the value lost to a partner-model error ϵ (worst-case total variation) is at most linear in ϵ.
Table 6 and its description are in Appendix D of the revised PDF (in blue), with one sentence in Section 5.1.
W1 (on par with Expector). Correct, and this is what the paper claims: with full partner support and a two-step horizon the correction is Expector (Section 4), and head to head the two are level (50.7 ± 2.0%; 49.3 ± 1.9 in the rerun). What the paper adds to Expector is the account of why such a rule works. The decomposition places the partner's loss in chess in decision error at the states it is handed, which extra search cannot change: an engine given HAD-4's search budget gains nothing as a teammate (50.7 ± 1.9% against R), while HAD-4 beats it head to head (57.2 ± 2.3%, pre-registered). The analysis also gives explicit error terms for oracle error, partner-model error and truncated support (Theorem 12, Proposition 21), and states the correction for any value oracle and partner model, with scale and depth as parameters (Definition 11).
If the mismatch table answers your question, we would appreciate it if you would reconsider your rating in that light. We are glad to answer further questions.