When Predictions Become Inputs: Hidden Feedback Errors in Latent World Models

In autoregressive latent world models, a prediction describes a task state and becomes input to the next prediction. We show that these roles can diverge: changing feedback improves future prediction while its current nonlinear task readout stays fixed. Our oracle diagnostic guides a predicted token toward its observed encoding, preserving the readout, original ReLU activation region, earlier history and future actions. Free rollout tests usefulness; norm-matched donor guidance tests source sensitivity. Across Transformer and GRU PushT models, stronger readouts and matched coordinate training retain terminal block-position MSE reductions of 9.3% and 8.8% for coordinate-teacher and physical-label objectives. On 256 fresh recipients, corrections preserving two pose readouts retain reductions of 6.4% and 6.7% under a reserved evaluator. Benefits also survive joint pose and qualified velocity constraints. Recursive Transformer training lowers free- rollout MSE by 23.1% and 30.9% under the preserved readout. Prospective tests do not confirm incremental prediction of training response or improved action selection; native DINO-WM PushT and PointMaze do not confirm correction over free rollout. These findings establish feedback insufficiency relative to specified task measurements and separate it from diagnostic value and decision quality.