- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The work studies data augmentation for Bayesian Neural Networks trained with variational inference, as an alternative to equivariant architectures. Working within exponential family variational distributions, the authors derive conditions under which augmented training keeps an invariant subspace of natural parameters corresponding to equivariant predictive maps. They also derive bounds on the equivariance error under finite sampling. They introduce three symmetrization strategies (geometric averaging, projection, and orbit expansion) designed to place or keep the model within that invariant subspace. Experiments on FashionMNIST under augmentation show that orbit expansion outperforms the baseline in both equivariance and classification accuracy.
Critical strengths
- The theoretical framework is rigorous and well constructed, deriving conditions under which augmented training preserves the invariant subspace of equivariant parameters.
- The connection to deep ensembles being exactly equivariant is well motivated, framing Bayesian neural networks as a cheaper alternative is compelling.
- Orbit expansion outperforms the baseline in both classification accuracy and equivariance metrics across all trigger timings, demonstrating that the framework can yield practical improvements when the symmetrization strategy is well chosen.
Critical weaknesses
-
While the paper is primarily theoretical, the experiments should validate that the theoretical framework translates to practice. This validation is only partially convincing. Orbit expansion consistently outperforms the baseline, supporting the theory, but geometric averaging and projection do not improve accuracy over the baseline. For a paper arguing that symmetrization strategies grounded in the theoretical framework are beneficial, this is a concern that deserves explanation. Are these methods failing due to theoretical reasons, or implementation choices?
-
The experimental validation is limited to FashionMNIST with symmetry, which is the simplest possible setting. For a theory paper, this is understandable, but the theoretical contributions are general enough that at least one additional setting would significantly strengthen the empirical validation. It currently remains unclear whether the theoretical guarantees translate to more complex symmetries or domains.
-
The restriction to finite groups is a significant theoretical limitation that the paper acknowledges but leaves for future work. Given that many practically relevant symmetries are continuous, such as arbitrary rotations in medical imaging or 3D symmetries in molecular property prediction, this limits the scope of the current contribution. Some discussion of the conceptual obstacles, or lack thereof, to extending the framework would strengthen the paper.
Questions for authors
- The best results are obtained when orbit expansion is triggered at epoch 100. Why was a higher trigger epoch not explored, and would further delaying the intervention improve results?
- Are there other symmetrization methods that could fall within this theoretical framework and potentially outperform orbit expansion?
- The experiments are limited to FashionMNIST. How difficult would it be to apply this framework to other domains, and do the authors expect similar gains in, for example, medical imaging or molecular property prediction?
- Some forms of augmentation, such as adding noise, do not correspond to invertible group transformations and therefore fall outside this framework. How significant is this restriction in practice, and are there plans to extend the framework to cover such cases?
- The closure condition required by Theorem 1 restricts the usable variational families and transformations. How difficult would it be to verify or satisfy this condition in new settings, and are there practical guidelines for doing so?
- Geometric averaging and projection do not outperform the baseline in accuracy. Is this expected from the theory, or is it a limitation of the experimental setup? Are there settings where these methods would be expected to perform better?
Prompted by the reviewers question, we have revisited this issue and have identified two likely factors.
First, when evaluating the strategies, one needs to take into account the nature of the equivariant space HG. They differ between the approaches, since they depend on the representation used — geometric averaging uses ρch, whereas projection and orbit expansion uses ρgcnn (see Appendix H). We have added a remark about this. Now, the spaces have distinct natures — the filters in HGch must be individually rotation invariant, reducing their expressivity compared to the filters in HGgcnn which are of arbitrary shapes but related to each other. This makes it plausible that approaches based on HGch fare worse.
Secondly, we realized that projecting / averaging takes away energy from the weights, and hence change their scaling of the weights. In the updated experiments, we therefore rescale the weights after projecting. With this modification, they fare much better - they then achieve comparable or better accuracy compared to the baseline. For more details on this, see the global comment about the updated experiments. We do however not reach the performance of orbit expansion. It hence seems that the projection operation alters the weights more drastically than the orbit insertion, and the training hence starts in a region of the parameter space further from optimal, from which it cannot recover. See also the discussion in Appendix K.
In the updated experiments section, we also test our approach on histological data. See the global comment for more information.
While we agree that we leave the full details of the infinite group case to future work, we have outlined concrete steps in this direction in Appendix A.
We have in fact explored later triggers in Appendix K, and triggering at epoch 150 gave no gain over epoch 100. With a fixed budget of 500 epochs, a later expansion trades off Stage-2 training of the full-width network against longer Stage-1 training of the smaller network, and an interior optimum is therefore to be expected.
The theoretical framework allows for any symmetrization strategy to be applied — the central insight is that as soon that we have made sure that η∈HG, the training dynamics that we consider will not move away from HG. We do not rule out that there are symmetrization techniques that lead to better results, in particular for other groups, and we do not claim that the methods we have proposed are optimal.
We actually tried one more strategy in (unpublished) experiments — simply initializing with η=0. This lead however to very bad results, so we did not include them in the manuscript.
We want to emphasize that the symmetrization strategy of Nabarro et. al. that we consider in the updated experimental section is not an example of a strategy of enforcing η∈HG. That strategy rather applies symmetrization directly to the function x↦Fη(x). Through the compatibility assumption, this can be reinterpreted to be a symmetrization of the distribution q↦1/∣G∣∑g∈GTg#q, but this transformation in general yields a distribution that is not in the exponential family any more — a Gaussian will typically be converted to a multimodal distribution, for instance.
When the augmentations considered do not form a group, the mathematical formalization will become very different. We do not expect our results to easily generalize.
For any given exponential family, the applicability can be checked explicitly by verifying that (11) and (12) hold for the transformation considered.
We note that Gaussian families, that are very popular in practice, are typically included. More generally, closure holds for permutation actions (e.g., in G-CNNs and under rotations, reflections, and channel permutations) when using product exponential families, such as Gaussian or Laplace. For general orthogonal actions, suitable choices include isotropic or full-covariance Gaussian families. In contrast, such as families with coordinates-dependent structure, such as fixed sign patterns or positivity constraints, that need to be avoided. We have discussed this in Appendix J, where we have also tested training an exponential family that does not fulfill the closedness condition. As expected, it does not have the same equivariance behavior as when the closedness condition is satisfied.