Loss-Level Connectivity Across Optimization Trajectories

The training data and optimization process jointly shape how deep neural networks navigate their loss landscapes. This interaction motivates studying not only the geometry of optimization, but also how that geometry relates to the model’s evolving behavior on the training data. Yet, existing studies have largely focused on loss-landscape geometry around converged solutions, where learning from the training data is largely complete. Mode connectivity exemplifies this line of work by asking whether independently trained solutions can be connected through paths of low loss. This perspective naturally raises a question during training: how is this geometry organized while the model is still learning the structure of the training data? However, extending this perspective to intermediate training stages is less straightforward, because the objective itself changes continuously as the network continues to learn from the data. To separate this changing objective value from the geometry being examined, we study connectivity between independently trained states at matched objective levels, which we refer to as loss-level connectivity. In the settings we examine, matched-loss states can be connected by fitted nonlinear paths that approximately preserve the endpoint-defined objective level across multiple stages of training. We then analyze example-wise residuals along these paths. These residuals distinguish controlled subsets of training examples even after matching their endpoint losses, and vary systematically under label-corruption and training interventions. These findings show that path-based measurements can contain information that is not captured by scalar endpoint loss alone.