- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper focuses on three-player perfect-information games where the game objective mismatches the cardinal score. When the optimization objectives are win probability or rank credit, the aligned value can be coarse; searching with few simulations over the aligned value cannot reliably decide root actions. Cardinal values like the raw score can provide more information, but can mismatch the aligned value. To handle this, this paper proposes to carry both aligned values and cardinal values through one search. The value outputs share the same network body, so one forward pass can directly output both values. In a single PUCT search guided by the aligned value, the method independently performs max^n backup for both aligned and cardinal value vectors. Then, the method decides whether to use aligned readout or cardinal readout based on whether the aligned margin, i.e., the gap between the top two aligned action values, clears a calibrated threshold. The experiments use exactly solved Tally and reduced Chinese Checkers with an exactly solvable endgame band, demonstrating that under 64 simulations, gated readout performs better than aligned readout and cardinal search.
Critical strengths
The empirical analysis is quite comprehensive, and the evaluation is rigorous. This includes the tournament credit, the regret calculated by the exact max^n solver for Tally, and also the analysis of the decision quality in Chinese checkers solvable endgames. Figure 2 presents the regret of the two readouts, giving the gated readout a direct empirical motivation.
Critical weaknesses
The novelty of the dual-backup search and the margin gate remains unclear. Implicit minimax backups (Lanctot et al., 2014) already propose to maintain and backup two evaluation signals within a single MCTS tree, but this work lacks a discussion or comparison with it. KataGo also studies using win signal and score together; this work compares only with a KataGo-style fixed linear blend, which may not reflect KataGo's actual score utility.
The comparisons cannot fully isolate the source of performance gain. The effectiveness of tree guidance is supported by showing that always-cardinal readout performs better than cardinal search. However, the two methods differ in both the value head guiding the tree and the root action selection (value argmax vs. most-visited), and Appendix F suggests the selection rule alone can affect regret. The main test-time baselines also use PUCT most-visited readout, without comparing with Gumbel MuZero's sequential halving and corresponding root action selection under the same simulation budget.
Questions for authors
- Please clarify the advantages over prior works like the two mentioned above. Are their approaches not working in the target scenarios? Such as implicit minimax backups or KataGo’s actual score utility beyond the fixed linear blend.
- Is it possible to do more test-time comparisons to isolate the performance gain? Such as adding one to use cardinal search with argmax cardinal value readout, compared with the current always-cardinal readout.
- Can Gumbel MuZero's sequential halving and root action selection serve as test-time comparison baselines? This can help confirm whether the proposed method performs better than the existing few-simulation search.
- In the appendix, the experiments show that the gain declines with training. Does this reflect that cardinal values provide a target that is easier to learn than aligned values? So, on the other hand, is it simply an intermediate auxiliary before the aligned values learn?
8 comments
Comment 2. Separating tree guidance from final action selection
W2 / Q2. The always-cardinal readout and cardinal search differ in both guidance and final action selection.
The original comparison changed both factors. We added the requested cardinal-argmax control: cardinal-guided search with its tree unchanged and its final action chosen by cardinal value instead of visit count. This separates guidance with final action selection fixed, and final action selection with guidance fixed. At 64 simulations:
| Comparison | Tally rank | Tally win | CC rank | CC win |
|---|---|---|---|---|
| Guidance, final action selection fixed: always-cardinal minus cardinal-argmax | +.002 [-.002,.007] | +.006 [.002,.010] | -.001 [-.018,.016] | -.022 [-.055,.011] |
| Final action selection, guidance fixed: cardinal-argmax minus cardinal search | +.007 [.001,.012] | +.008 [.001,.014] | +.106 [.087,.124] | +.206 [.169,.242] |
Final action selection accounts for most of the original difference; aligned guidance adds a resolved gain on Tally win. Selective switching contributes separately on Tally: Gated exceeds the always-cardinal readout on the same tree by +.011 (rank) and +.007 (win). On CC, the gate fires on about 99% of moves, and the difference from cardinal-argmax search is unresolved (−.002, −.026).
Manuscript locations: §5.4, Table 1 and Appendix J.
Comment 3. Gumbel search and the gate on a mean-backup tree
W2 / Q3. Comparison with Gumbel sequential halving and its root selection at the same budget.
As requested, we compared the submitted Gated method with Gumbel searches using mean and replacement backups. These controls follow the official mctx selection equations, including root sequential halving, mixed-value completion and the survivor-restricted recommendation. GM denotes mean-backup Gumbel with no root noise; noise scale 1 is tested in Appendix K.
| Gated at 64 simulations minus | Tally rank | Tally win | CC rank | CC win |
|---|---|---|---|---|
| GM, 64 | -.013 [-.020,-.006] | -.026 [-.033,-.019] | +.078 [.055,.100] | +.207 [.168,.246] |
| GM, 128 | -.018 [-.026,-.010] | -.026 [-.033,-.019] | +.032 [.009,.053] | +.145 [.110,.181] |
| Gumbel, replacement backup, 64 | +.005 [-.003,.013] | +.008 [.0004,.015] | +.147 [.124,.169] | +.281 [.247,.316] |
With the 300-game networks, Gated at 64 simulations exceeds GM at both 64 and 128 simulations on both CC objectives; GM is stronger on Tally.
In an additional experiment, we applied the same final-action rule to GM itself. GM+gate keeps GM's mean aligned backup and tree, independently backs up the cardinal vector by replacement, and uses the same threshold , retaining GM's own recommendation when the gate does not fire. All configurations below use 64 simulations.
| Comparison | Tally rank | Tally win | CC rank | CC win |
|---|---|---|---|---|
| GM+gate − GM | -.003 [-.009,.002] | -.006 [-.011,-.002] | +.097 [.077,.116] | +.269 [.227,.307] |
| GM+gate − Gated | +.010 [.003,.016] | +.019 [.013,.027] | +.019 [.004,.034] | +.061 [.034,.090] |
GM+gate exceeds the original Gated configuration in all four settings. On Tally, its point estimates are below GM. On CC, its improvement over GM shows that the proposed final-action construction can also improve mean-backup Gumbel search.
Manuscript locations: §5.3–§5.4, Tables 2, 13 and 23, and Appendix L, Tables 30–31.
Dear Reviewer emP9,
Thank you for the careful review and the suggested controls. We ran each of them and report the results below, including the settings in which a control is stronger than our method. All comparisons concern test-time search in Tally and reduced Chinese Checkers (CC), with fixed networks and opponents trained using replacement backup.
Comment 1. Implicit minimax backups and KataGo
W1 / Q1. Novelty relative to implicit minimax backups (IMB) and KataGo's score utility.
The difference lies in where the second value is used. IMB and KataGo's score utility change the values that guide tree selection, whereas our method leaves the tree unchanged and uses the cardinal value only for final action selection, when the aligned margin is below the threshold.
The rows for IMB and the KataGo-style utility describe the controls we implemented (Appendices K and L).
The final-action rule follows from a measurement (§3, Figure 2). On solved Tally positions, the aligned readout has its largest regret at small, nonzero aligned margins, where the cardinal readout is more often better. The gate therefore uses the cardinal readout in this range and retains the aligned readout elsewhere.
To test whether the two prior approaches work in our setting, we added IMB PUCT with either head as its heuristic (weight 0.5) and a three-player adaptation of KataGo's nonlinear score utility. The latter uses a root-centered arctangent of the predicted lead over the strongest opponent, with weight 0.2 on our [0,1] payoff scale. Appendix L specifies the adaptation, which evaluates the utility at the predicted mean lead.
Gated denotes our original replacement-backup search with the gate. Unless stated, test networks are trained on 300 self-play games. Entries compare tournament credit against the same two copies of a fixed, 300-game win-trained opponent; intervals are paired, pointwise 95% bootstrap intervals over tables or openings. Bold marks intervals excluding zero.
Both approaches work in our setting: cardinal-heuristic IMB improves mean-backup PUCT on CC, and the KataGo-style utility improves it on both games. On CC, Gated at 64 simulations further exceeds all three controls at both 64 and 128 simulations. On Tally rank, Gated exceeds cardinal-heuristic IMB by +.008 [.002,.014] at 64 simulations, while the KataGo-style utility is higher on Tally win. At 128 simulations, all three controls exceed Gated at 64 on Tally win.
Manuscript locations: §3, Figure 2, §5.3, Tables 2 and 13, §6 and Appendices K–L.