Rich Outcomes, Selective Constraints: Goal Overspecification in Offline Goal-Conditioned Learning

Reward-free offline goal-conditioned learning obtains generality by treating achieved outcomes as reusable supervision, enabling policies to learn from offline experience without task-specific rewards. Yet a complete achieved outcome can contain more information than a downstream task requires. We show that conflating what an outcome describes with what a task demands creates goal overspecification: full-state hindsight objectives can require reproducing properties that the current success criterion leaves unconstrained. This tension is particularly important for general-purpose learning, where such properties should be retained because they may define other tasks. We therefore separate outcome information from task constraints with Goal-Set Hindsight Relabeling (GS-HER). GS-HER pairs a reference outcome with a query that defines a task-dependent equivalence class and makes the goal-conditioning representation invariant to unqueried outcome properties, while preserving the base learner’s hindsight sampling, supervision, target construction, and optimization objective. Across five heterogeneous offline goal-conditioned learners and OGBench manipulation tasks, replacing full-state semantics with the oracle task projection yields higher mean success in all $25/25$ backbone–task pairs; GS-HER recovers much of this gap, increasing average success from $24.4\%$ to $56.5\%$, compared with $60.0\%$ for the oracle projection. In contrast, gains largely disappear on PointMaze, where outcome and task semantics are already aligned. A single GS-HER checkpoint can additionally support multiple object- and robot-centric success predicates from the query family represented during training by changing only the inference-time query. Our results suggest a simple principle for general-purpose robot learning: retain outcome information broadly, but impose task constraints selectively.