Action-Conditioned Predictive Learning with Causal Self-Attention: Identification, Initialization, and Planning

We study whether causal self-attention trained on action-conditioned outcomes can learn predictive models that support near-optimal decisions without optimal-action supervision. For a one-sparse logistic model on fixed binary history features, we characterize identification by positive feature-disagreement probabilities. Under this condition, population gradient flow and gradient descent from uniform attention converge, with prediction bounds uniform over all feature combinations. A logarithmic-size design attains the identification lower bound, with finite-sample guarantees from independent exploratory episodes. For a complementary multi-head model, we show that uniform-attention initialization yields tangent features of degree at most two and makes population training stationary for pure higher-order parity targets. A target-independent nonzero score initialization yields positive expected kernel eigenvalues, with interaction-order expansions and explicit bounds at every nonzero scale. These target-eigenvalue guarantees extend to finite-width population and empirical training, including bounded perturbations of paired initialization. A separate finite reachable-set analysis treats general full-support outcome distributions under representation and coverage conditions. Exact planning with the Bayes predictive model is Bayes-optimal, and uniform predictive error yields explicit value-suboptimality bounds. Controlled experiments examine sparse prediction, kernel stability, coverage, and higher-order learning, and corroborate the predicted initialization spectrum.