acceptodds
Under review as a conference paper at ICLR 2027

When Search Teaches Style: Causal Internalization of Tactical Priors in AlphaZero

69 views

Abstract

AlphaZero is normally evaluated as one agent: a policy-value network fused with Monte Carlo tree search. That fusion hides a causal question. When self-play search is given a useful prior, does the network absorb the induced behavior, or does the behavior stay rented from search at test time? We answer with Cross-Phase Prior Intervention (CPI), which switches a root-level tactical prior on and off independently during training and during evaluation, separating the prior's online effect from the learned residual it leaves in the weights. The endpoint is deliberately narrow: how often a network discharges a forced defensive obligation when no search-time guidance is available. On a sealed one-shot final test in 9x9 Gomoku and 19x19 Go, deleting the prior still leaves a large residual— response rises from 13.8% to 26.3% in Gomoku and from 0.6% to 33.8% in Go-and soft reweighting teaches as well as hard action restriction, so pruning legal actions is not the mechanism. The same cross bounds the claim: re-enabling the prior restores nearly 100% response, leaving dependence gaps of 73.7 and 65.8 points. A matched-position evaluation localizes the residual to trained geometry-absent at the shared initialization, emerging over training, worth +17.3 points on in-distribution defenses but only +1.3 on structurally novel ones. Search is therefore best read as a training-time behavioral curriculum whose lessons are real, partial, and geometry-bound, and online competence and internalized competence are different estimands that a diagonal ablation cannot tell apart.

Primary area:
learning theory
Keywords:
reinforcement learning, AlphaZero, Monte Carlo tree search, self-play, search-guided learning, policy learning, causal intervention, curriculum learning

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_tEjFOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Search-augmented neural networks like AlphaZero learn from their search policies through MCTS. This paper aims to measure the extent to which specific tactical priors get absorbed into the network. They experiment on AlphaZero in two game settings: Go and Gomoku, introducing a hard coded tactical prior into AlphaZero's training-time search for both games. Their training includes unguided, soft guided, and forced guided settings, which vary the strength of the priors. After training, the authors test how much of the behavioural effect associated with each prior remains when search is removed, finding a partial effect.

Critical strengths
  1. the cross-phase experimental design (mostly) isolates the absorption of a specific tactical prior into the policy network, which is good
  2. It's nice to see a Go paper in a sea of chess papers; Go is an under-utilized domain
Critical weaknesses
  1. incredibly difficult to read, LLM like prose
  2. their Go agent is extremely weak, hard to know whether lessons generalize to strong game playing agents that people care about
  3. only one tactical prior per game
  4. obvious result, questionable relevance and impact to either game playing AI or the policy distillation literature
Questions for authors
  1. what does this paper establish beyond the fact that policy distillation w/search works?
  2. why not turn search all the way off for your prior-off condition, which still uses 32 MCTS rollouts?
Reviewer_GkUhOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

AlphaZero has two main components in its inference core: the prediction network and the algorithmic MCTS rollout. When learned properly, the prediction model itself of AlphaZero is expected to possess an accurate estimation(prior) of the strategic action. Here, the paper draws attention to the unexpectedly-trained scenario, where the fallacy of the value network is patched by the MCTS algorithm in inference time. In this scenario, the paper claims that end-to-end winning rate overlooks this, as those discrepancies between parts and self-healing will not be debugged. Therefore the paper claims the need of better component-wise evaluation metric to better train and verify the soundness of each architecture piece. The paper thus introduces Cross Phase Prior Intervention (CPI), an intervention evaluation method to compare the model’s prior observed in train time and compare it with that observed in evaluation time. They conduct a diagonal ablation in the training curricula axis and prior-on axis.

Critical strengths

The author clearly expanded their argument on the ambiguity of the evaluation metric for AlphaZero. It clearly distinguishes the exquisite logical difference between a properly trained scenario and the opposite. The opposite scenario they have come up with is convincing, and seems to be questionable to all end-to-end evaluation metrics to a multi-module architecture.

Critical weaknesses
  1. The explanation is poorly designed. Other than the very start of the introductory part, the remaining writings are not reader-friendly. The sentences do not coherently connect to one another, even considering the reviewer’s inexperience with the context. The manuscript should guide the reader to grasp how the method is designed by explaining each part carefully along with their designing justifications, but this is mostly absent in this paper. Instead, the paper abruptly emphasizes without ever explaining the target in prior, which means the author is assuming the reader already knows what they are emphasizing. In short, the paper suffers from the curse of knowledge. I carefully doubt that those parts are solely written by a language model. It needs more proofreading and polishing from a human reader point of view.

  2. Some necessary explanations are missing. I could not pick up how each curriculum (UNGUIDED, SOFT-GUIDED, FORCE-GUIDED) are different from each other. If we assume it is author-curated, then what are the objective justifications?

  3. The methodology details and experiment settings are mixed. The paper shares section 3 to articulate both the details of evaluation metric, and the testing experiment settings, making the reader hard to distinguish the two. It would be better to split the method from the experiment settings, and spare chapters each.

  4. The contribution is still slim. Although AlphaZero paper doesn’t report the prior-on-off ablations, its prior work (AlphaGo) reports a performance ablation with the combination of components. I believe this kind of component-wise ablation is widely used, and not novel in itself. Therefore, the novelty at its minimum is assumed to be on the curriculum axis, but it is not well articulated.

Questions for authors
  1. Limited novelty beyond existing work. The paper's primary contribution appears to be the CPI framework, which separates the learned effects of search guidance from its inference-time effects. However, prior work on Expert Iteration, search-based policy improvement, and amortized planning has already investigated the transfer of search-derived knowledge to neural policies and the limitations of this transfer. While CPI offers a more controlled quantitative analysis, it remains unclear what fundamentally new scientific insight is established beyond these existing findings.

  2. Insufficient justification of the significance of separating search and network contributions. The paper argues that evaluating the combined search–network agent can obscure the extent to which tactical competence has been internalized. However, AlphaZero-style systems are generally designed to operate with both search and neural networks, making end-to-end performance a valid evaluation target. While the proposed decomposition is useful for understanding the learning process, the paper does not sufficiently establish why this distinction is essential beyond the specific goal of measuring internalization. The broader scientific and practical implications of CPI should be more clearly motivated.

  3. Incomplete validation of network-level internalization. Although the paper evaluates learned residuals under Prior-Off, MCTS remains active during evaluation. Consequently, the measured improvements represent the performance of the learned network coupled with unguided MCTS, rather than the standalone tactical decision-making ability of the network. The reported network tactical mass provides supporting evidence of changes in the learned policy, but does not directly establish whether the network independently selects correct tactical actions. A policy-only evaluation without MCTS would provide stronger evidence for network-level internalization.

Reviewer_1XQROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates whether a neural network can learn useful priors during training. The authors study this question using AlphaZero, a framework that trains a policy–value network through Monte Carlo Tree Search (MCTS)-guided self-play. Specifically, they train AlphaZero agents on 9×9 Gomoku and 19×19 Go with two types of priors. The first, Soft-On, reweights the action probabilities to favor important actions. The second, Force-On, directly assigns zero probability to unimportant actions in a forcing set. The authors propose Cross-Phase Prior Intervention (CPI), together with several causal verification checks, to investigate whether networks trained with these priors retain the knowledge when the priors are removed during evaluation. The results show that tactical priors can be partially learned. However, these improvements remain greatly below the near-perfect performance achieved when Force-On is enabled during evaluation. Furthermore, the benefits of guided training are largely concentrated on in-distribution positions, with limited generalization to structurally out-of-distribution positions.

Critical strengths
  1. The paper studys whether a neural network can learn useful prior, which is a meaningful and important question in deep learning.

  2. The experimental is rigorous and the conclusion is clear.

Critical weaknesses
  1. The methodological novelty is limited. The audit is good but no concrete solution is proposed to improve the incomplete learning or poor OOD generalization.

  2. The reasons of incomplete prior learning are not sufficiently investigated. Although the experiments show that neural networks fail to fully learn the tactical priors, the underlying mechanisms remain unclear. The authors discuss several possible explanations in Appendix G, but these hypotheses are not empirically validated.

  3. The uncertainty of experiments is not reported. The experiments in Tab.1 is trained on three seeds, while the experiments in Fig. 2 is trained on one seed. Repeating the experiments on more seeds and reporting std could help to reveal the uncertainty.

Questions for authors

I recommend authors to check 'Primary Area' of this paper. The authors chose 'learning theory', but this is an audit paper rather than theory paper.

Reviewer_thf4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether tactical guidance used in AlphaZero self-play leaves behavior that survives removal of that guidance. Cross-Phase Prior Intervention independently varies training and evaluation priors, comparing unguided, softly reweighted, and action-restricted search. Experiments in Gomoku and Go find improved defensive response after removing the prior, substantial remaining dependence on it, and limited transfer to novel Gomoku geometries. The contribution is an evaluation design and empirical diagnosis of retained behavior; the underlying learning algorithm remains standard AlphaZero.

Critical strengths
  1. The crossed design answers a concrete attribution question that a guided-versus-unguided end-to-end comparison cannot answer. Holding evaluation fixed while changing the training prior separates retained benefit from immediate enforcement. The five final-test cells suffice for the stated contrasts; running all nine is unnecessary. The reported shared-initialization checks, zero intervention calls under Prior-Off, and independently implemented set-valued oracle are sensible controls.

  2. The study contains substantial work for this diagnostic question: two games, three curricula, multiple checkpoints, and a declared three-seed final evaluation. Table 1 reports Prior-Off defensive response rising from 13.8% to 26.3% in Gomoku and 0.6% to 33.8% in Go. Soft guidance also improves response, showing that hard pruning is unnecessary for a residual. Figure 2 tests structural transfer directly, while near-perfect Force-On performance is appropriately treated as a positive control. The authors deserve credit for reporting partial learning and a failed transfer case.

Critical weaknesses
  1. Prior-Off still uses 32 MCTS simulations (Section 3.2; Table 3). The residual therefore measures changed weights acting through ordinary search, not the network’s unaided action accuracy. Policy and value changes can both contribute. Tactical-mass averages do not establish the near-argmax reordering asserted in Section 4.2; direct policy accuracy and action-rank measurements would help. Likewise, soft guidance’s success shows pruning is unnecessary, not that pruning contributes nothing under Force-Guided. The teacher–student gap in Section 4.4 does not exclude imitation. Appendix G correctly leaves target-versus-trajectory mediation unresolved; the main text should respect that boundary.

  2. Appendix E withholds uncertainty estimates despite the resampling plan in Section 3.4. Three seeds limit precision, but per-seed effects, ranges, and defensive-endpoint counts should still be reported; Table 1’s n_cell is not the endpoint denominator. Checkpoint-0 spread cannot substitute for variability after training. Consequently, similar pooled means do not establish that soft and hard curricula teach equally well. The initialization check also needs explanation: Figure 2 reports 4, 7, and 6 correct responses from allegedly identical initial weights on identical positions under noise-free Prior-Off. Any remaining stochasticity or tie-breaking should be specified.

  3. The structural-generalization claim needs a clearer account of training exposure. Appendix D describes disjoint template families, but does not explain how these families enter self-play training or how their absence from the actual replay buffers was established. Holding out evaluation templates alone does not establish unseen training geometry. Figure 2 is also one Gomoku seed, as Section 4.3 acknowledges, so it supports a failure on that suite rather than a general boundary across games. The further assertion that guidance makes learning possible, rather than accelerating eventual learning, is not established by 40 iterations.

Questions for authors
  1. What are the existing checkpoints’ greedy-policy defense accuracy and oracle-action ranks?

  2. Can you report per-seed effects and endpoint counts, and explain the checkpoint-0 discrepancies?

  3. How are template families linked to self-play, and how was structural-OOD exclusion from replay verified?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.