Learning Nonlinguistic Deliberation Through Latent Critiques

Planners often use a learned scorer to select among candidate actions, then discard its assessment. We ask whether this assessment can instead guide revision, and what the critic should communicate to make revision useful. We introduce ReCoursive, a recurrent planner for end-to-end driving that deliberates without language. Its critic returns latent critiques, the continuous representations from which it predicts scores, to guide the proposer’s subsequent revisions. The loop is trained end to end with trajectory and assessment supervision at each step, without reasoning traces or demonstrated corrections. On NAVSIMv2, critic feedback adds 2.83 EPDMS over recurrence alone at matched depth and proposal count. Additional revision steps improve planning with latent critiques but not with predicted scores alone; critiques also need assessment supervision and must be refreshed as candidates change. ReCoursive achieves the best results among compared methods on NAVSIMv2 and, in zero-shot transfer, on HUGSIM and among non-VLM methods on KITScenes LongTail. These results suggest that nonlinguistic deliberation benefits from a learned interface between assessment and generation rather than one restricted to scores.