- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper "THE SUPPORT VECTOR MACHINE TREE" merges the SVM concept into the decision tree in two components. The decision function is linear as in the CART approach but with an L1 regularization on the weight and the leaf functions are also SVMs, if separation is needed. Thus, the hope is to have only few features accessed during the tree part and much fewer support vectors active in the leafs during inference and training.
Whereas the general idea is sound and worth exploring, literature analysis, theory, and experiments need much more expansion before this algorithm is ready to be published. Please keep working on it!
Critical strengths
- The algorithm idea is reasonable and sufficiently well presented.
- Scaling to 4M examples.
- Potential savings in inference and training.
- joint optimization of tree and leaf SVMs.
- By combining the power of decision trees and SVMs, there is potential to be superior to Random Forest and gradient boosting trees.
Critical weaknesses
- The state of the art is omitting significant parts like "A hybrid SVM based decision tree", CART, gradient-boosted trees, random forests, and there are probably many more missing.
- No statistical significance is reported.
- The references are not appropriate. I would have expected to also see decision trees in the mix. Not even, the algorithm from "Alternating optimization of decision trees" is visible in the comparison. You cannot say, this is not an SVM and then just compare against SVMs.
- The arguments on speed up need measurement (and not just FLOPS). It also needs to account for additional hyperparameters tuning.
- The model definition does not truly apply because it gets later on modified with L1 instead of L0 loss.
- There are several bold and incorrect statements without evidence.
Other issues:
- "While they are widely praised for their generalization ability," - This requires citation and I am not sure that is still the case.
- "linear on N at inference time" This is incorrect. It's linear in the number of support vectors.
- "support vector (SVs), which are a (usually large) subset of the training instances" - This needs evidence and I don't think, it is always true.
- "we find that, with hyperparameters determined by cross-validation" - This sounds like overfitting of hyper parameters without validation dataset.
- "existing fast approximate approaches" - It's missing decision trees and random forests
- https://www.sciencedirect.com/science/article/abs/pii/S0031320310003067 needs to be compared against. "A hybrid SVM based decision tree", M. Arun Kumar, M. Gopal, Pattern Recognition, Volume 43, Issue 12, 2010
- some other literature on hybrid SVM extensions might be missing like relative margin machines
- missing comparison to ordinary decision trees, gradient boosting trees, and random forest preferably on Kaggle competitions
- the paper claims, the SVM is in the nodes and there is a reduced problem but then advertises that tree and leaf nodes are jointly optimized. This is contradictory. Also, if every node is an SVM, you basically solve more and not less SVM problems.
- The tree is not a normal tree because the node decisions are linear classifications. This feels a bit like a boosting tree. A respective literature analysis on the tree side is missing.
- "the entire SVM tree model can be written as a sum of a basis function over each training instance" is incorrect AND unproven.
- Appendix B says, an approximation is used. However, it is not investigated how this interferes with the original objective function.
- It is not clear to me where equation 3 comes from. Why did the decision tree part get replaced by gj(x)
- Figure 8 highlights the low number of inference SVs but is ignoring that because of the in-between decisions, there are still more support vectors that define the decision boundaries.
- The initialization procedure is not clear and it is not clear, how a local minimum is avoided.
- train-test-validation procedure is completely undocumented.
- I am missing time measurements for training and testing to confirm the efficiency claims.
Questions for authors
- How was the splitting into training, testing, and validation data?
- How do the results of reference algorithms compare to the results in the reference literature?
- How would the algorithm work for regression or multi-class?
- What were the criteria for selecting the benchmark datasets?
- What are the respective leading values on Kaggle?
- How do you avoid local minima with the iterative approach?
- What are "random median splits"?