- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper presents a theory of feature learning during training for gated deep networks trained on data generated via a Oblique Decision Tree.
They use the model-data framework from Banerjee et al 2025, which is both complex enough to represent interesting training scenarios and simple enough to derive analytical feature predictions.
The theory is divided into single path (width=1) and multipath neural networks. For single paths, they derive how the NTK features change during training, and provide intuition with a work out example. For multipath, the main result is that a L2 penalty on the gates can reduce the error. They further identify two effects: a hierarchy-seeking effect that can promote movement toward task-relevant internal features, and a self-opposing effect that can amplify movement along spurious directions.
They also show that they can predict solution from random initialization, using a heuristic score that give different weights to any possible solution.
Experiments with ReLU networks suggest that the phenomena studied in the paper also apply to them.
Critical strengths
The paper is clearly written, easy to follow, and well organized.
Being able to predict which solution training will reach is very interesting, and, as far as I'm aware, novel.
The fact that the theory predicts an intervention (l2 penalty) that will help reduce test error is also important.
Critical weaknesses
The theory is limited as the theorems mainly focus on a width 1 network (single path). However, by analogy with other deep learning theory literature that studies two layer neural networks (i.e., depth 1), this might not be an issue.
Real data often doesn’t come from ODT, hence this data regime might question the impact of the results.
There are no real data experiments.
Questions for authors
Could the author find real data that have an approximate ODT structure and see whether their results extend to this setting?
Thank you for the review.
The ODT is just a test-bed for evaluating hypothesis. We do not claim it is powerful enough to model real data, but that decision trees are powerful data models is almost folklore and we just borrow its reputation. The decision tree labelling function is already significantly more complex than typical labelling functions studied in deep learning theory.