- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper views the problem of learning an accurate and robust classifier via the lens of sensitivity field (vector field calculus).
Assuming the data is part of a manifold, the robustness problem can be translated as the change on the classifier to be agnostic to outside the data manifold and only parallel to it, i.e. gradients of the classifier (vector field of interest) should be essentially parallel to the data manifold and insensitive to change in other directions. The authour formalizes this property as the tangency condition.
In general the classifier can be defined as the path integral of this vector field of interest. Moreover, standard classification training enforces this vector field to be naturally a gradient of the classifier leading to a conservative field.
The key insight of the paper is that, more often than not, the tangency condition to ensure robustness can't be simultaneously satisfied by a conservative field our standard gradient vector field induced by conventional classification training. The authors refer this to the geometric obstruction.
The paper provides theorems, and examples to exhibit this perspective. Additionally, they subscribe to a general vector field path integral definition of learning the classifier, and show that its implementation (by learning the non-conservative vector field directly) indeed improves robustness empirically.
Critical strengths
- The biggest strength of this paper is the novel perspective of viewing a robust classifier problem by the lens of vector field calculus, and deriving structuaral limitations of standard classification training.
- The paper also provides empirical evidence on image classification tasks : a) that indeed the learned robust vector fields directly are not conservative b) alignment of gradient field induced by standard training is worse than alignment of the learned vector fields, bolstering the theoretical argument.
Critical weaknesses
The weaknesses are outweighed by the strengths of this paper, however I would like the authors to address the following questions:
-
To address the incompatibility of the tangency and conservative field condition, we need to utilize the path integral definition of the classifier (hypothesis). However, as a practical approach this would be computationally infeasible. As the authors themselves have pointed out even for CIFAR 10 the computations can scale up quickly. Moreover, choosing an optimal path for the integrals is also a choice which needs further investigation.
-
The main issue arises from the use of single value loss functions, which lead to gradient vector fields that are conservative, an alternative way to mitigate that condition could be, by using a multi objective loss? It would be great, if authors could add some discussion and relations to existing multi objective classification loss on how they can by pass this conservative constraint.
Questions for authors
Please refer to the weakness section.
3 comments
I thank the authors for the response. I urge the authors to have a brief discussion in the limitations and future works pertaining to these questions.
Thank you again for your time and for your encouraging words and review. We have added the discussion to Section 6 of the paper, which will be available for your review as soon as the second revision of the paper is ready.
Response
This is an important and broad question, and we cannot address it fully within a single paper. What we can say at this point is that, regarding time complexity, a less accurate approximation of the path integral, such as a single-step trapezoidal rule, can be effective.
Regarding memory complexity, a promising direction is to use a Helmholtz decomposition to represent the sensitivity field as a sum of conservative, divergence-free, and harmonic components. The exact form of this decomposition depends on the domain and the boundary conditions. Nevertheless, it could provide a useful way to separate the components of the field and to investigate which of them are necessary for the representation.
Another possible direction is a method that resembles purification, as described in the general response. Such a method could avoid explicitly computing the full path integral in the manner used in our experiments.
The computational complexity and the choice of paths remain open questions. We do not claim that the current implementation is a practical solution to these problems.
This could be a promising direction, although we have not approached the issue from this perspective.
According to our framework, changing the loss function alone is not necessarily enough to bypass the obstruction, because the obstruction originates in the hypothesis rather than in the loss: a differentiable scalar hypothesis imposes a conservativeness constraint on its sensitivity field. It might nevertheless be possible to derive a Pareto front for the robustness and accuracy objectives based on the presented framework. For example, accuracy may require the sensitivity field to contain components that are incompatible with the desired tangency condition, while improving robustness may require diminishing those components. This would lead to a trade-off between accuracy and robustness.
However, this idea is still too preliminary to be added to the paper, and the mathematical machinery required to investigate it properly would likely deserve a separate paper.