Input Relational Structure Shapes Training Dynamics in Neural Networks

Input representations are known to influence machine learning models, yet their effects are typically evaluated only after training has converged. We ask whether the training trajectory of learned representations carries a signature of the relational structure encoded in the inputs, and whether this signature predicts robustness. Using molecular property prediction, we compare models operating on graph-, sequence-, and vector-based encodings, and track the linear extractability of $32$ chemical properties from their representations across training checkpoints. All three encodings support substantial extraction already at initialization, but differ in how they reorganize it: sequence models largely preserve their initial extractability profile (Spearman $\rho =0.90$ between initial and peak extractability), whereas graph and vector models reorganize substantially ($\rho =0.62$ and $0.68$), each on different property subsets. To determine whether these representational dynamics are related to robustness, we introduce controlled corruption across all input relational structures and repeat training. Although corruption has little effect on overall task performance, it alters representations in an encoding-specific way: disruption is diffuse and strongest for vectors, intermediate for sequences, and selective for graphs, concentrated on properties that depend on the removed components. Finally, the evolution of extractability during uncorrupted training correlates with corruption sensitivity ($\rho =0.72$), whereas geometric summaries of the same trajectories do not, suggesting representation dynamics as a complementary lens on robustness.