Connectivity, Distractors, and the Speed of Learning

In deep learning, neural network architectures are typically fixed at the start of training, allocating each unit a predetermined set of inputs. Since that set is fixed independently of what the unit learns, some of its input connections may end up being useless, carrying no information about the unit's target. We study how these distractor connections affect the speed of learning. Drawing from theory in the linear setting, we characterize how updating the weights of such connections slows learning, then investigate the extent to which this phenomenon affects neural networks. To measure this, we construct a problem in which good connectivity is known in advance, allowing good and bad connectivity patterns to be compared directly. We find that bad connectivity slows learning and is especially costly in non-stationary problems, where learning speed determines performance. Scaling network width in this regime cannot solve the problem, and beyond a point worsens rather than improves performance. Learning connectivity has been pursued mainly for parsimony; our results suggest that it could also lead to faster learning, further motivating the use of methods that learn connectivity.