A Linear Map Goes a Long Way: Learned Costs for Planning with Frozen World Models

Planning with a visual world model requires a cost that compares predicted outcomes with a goal. Learned costs outperform raw latent distance, but it remains unclear how much of this benefit requires a flexible scoring network. We keep the encoder and dynamics fixed and train three costs to predict the elapsed time between logged observations under identical conditions: a linear map, a shared nonlinear map, and the joint scorer of prior work. In TwoRoom navigation, the linear cost raises planning success from 18% to 96%, and neither flexible cost adds a clear gain. The result holds for a second family of world models, and across rooms the linear cost also outperforms a cost built from decoded positions. Whitening the latent space or training the same map on shuffled targets recovers almost none of the improvement, indicating that the temporal targets are essential to it. In PushT block pushing, the flexible costs fit elapsed time better, even on episodes excluded from cost training, yet the joint scorer plans worse than the linear cost. The linear cost raises success from 36% to 44%, while the joint scorer reaches only 29%, and this gap persists after tuning each family and after symmetrizing the joint scorer. A comparably trained linear cost is therefore the baseline that learned planning costs should beat, and temporal loss alone is not a reliable criterion for choosing among them.