- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
This work investigates how compression audits can be sensitive to transformations of the reference features that do not change the behavior of the original model. They design such transformations that leave the behavior of the reference model the same and show that, after compression using either PCA or Random Projections, the results of the compression audit change. Evaluating on different pretrained models, the authors showcase how the success of individual compression methods is dependent on coordinate transformations that do not affect the initial model behavior or reduce the information in the reference uncompressed features. In addition to the observations of the compression dependence on non-behavior-changing transformations, they investigate ways to achieve invariance in such transformations, like feature whitening.
Critical strengths
- The authors make a substantial effort to study the dependency of compression audits to transformations that leave the original feature information and model behavior unchanged. Such results that study the behavior of compression audits to such non behavioral changing transformations can be of interest to the community.
Critical weaknesses
- One of the main weaknesses of this paper is the writing quality and lack of clarity. The paper's presentation is very disorganized and does not allow the reader to understand and evaluate the author's contributions. Additionally, prior work or datasets used are referred to without proper citations, with the citations appearing much later in the paper. Such an example is the reference to Pets, which is used as one of the main results in the abstract and the introduction and is first cited in Section 5.
- There is limited novelty, with the main contribution of the work being an extensive experimental study of techniques proposed in previous works.
Questions for authors
- What is the main experimental observation that establishes the novelty of this work and hasn't been explored in previous works