Directed Residual Information: Learning from Redundant Hypothesis Representations

A new hypothesis can make different predictions without improving learning. It may repeat what existing hypotheses already predict, or vary in a way unrelated to the remaining error. We ask when adding a hypothesis helps a learner explain its target. For a linear learner with squared loss and a penalty on the new coefficient, the answer depends on the error left unexplained, the new prediction direction, and its agreement with that error. We call this relationship directed residual information. Classical regression identities make it exact for the stated objective; the contribution is an explicit account of these failure modes and of what can be known before evaluating a proposal. A controlled experiment separates useful from irrelevant directions while holding their novelty nearly fixed. On 11 public binary datasets, 2,086 generated proposals test whether the measured relationship predicts improvement on separate test data. The resulting score ranks improvement better than novelty alone by mean within-dataset Spearman $.308$ (95% CI $[.190,.431]$), but its difference from partial $R^2$ is unresolved ($-.010$ $[-.033,.004]$). Because the current representation is fixed within each split, this ranking test does not isolate the contribution of remaining target error. Frozen replay separately finds no support for one attempt to turn past measurements into a selection rule. The account explains evaluated proposals under stated objectives; choosing unseen proposals remains a prediction problem.