acceptodds
Under review as a conference paper at ICLR 2027

Jacobian-Guided Anisotropic Noise Reshaping for Enhancing Representation Utility under Local Differential Privacy

Abstract

While Local Differential Privacy (LDP) serves as a foundational primitive for distributed data collection, its stringent randomization requirements often lead to severe degradation in data representation utility. This degradation stems from the task-agnostic nature of conventional LDP mechanisms, which perturb all dimensions without accounting for their relative importance to the downstream objective. To address this issue, we propose a novel approach that mitigates noise in task-relevant subspaces of the data representation. Our method identifies task-critical subspaces via the Jacobian of a pre-trained downstream model, selectively attenuates noise along these directions, and reshapes the isotropic noise of standard LDP mechanisms into an anisotropic distribution. The resulting mechanism preserves the privacy guarantee of the underlying LDP randomizer while heterogeneously modulating the impact of noise across task directions, thereby substantially enhancing data utility. The approach is applicable to both linear and nonlinear models and can be seamlessly integrated with existing LDP mechanisms. Extensive experiments on CIFAR-10-C under brightness corruption at the highest severity level demonstrate that integrating our approach improves classification accuracy by approximately 8 % for Laplace and 20 % for PrivUnit variants at .

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Privacy-Preserving Machine Learning and Inference, Representation Geometry, Jacobian-Based Subspace Identification, Local Differential Privacy, Privacy–Utility Trade-off, Task-Aware Noise Shaping, Anisotropic Perturbation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DDkHOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper utilizes the Jacobian matrix to reduce the amount of injected noise in downstream tasks under LDP, presenting a novel approach.

Critical strengths

Meeting the representation of high-dimensional data in LDP is a research hotspot

Using the Jacobian matrix to inject task-related noise is a novel approach

Critical weaknesses

The starting point of identifying the main relevant directions in tasks is not novel. I have seen it in covariance matrices, learning importance, and principal component analysis

Some descriptions are exaggerated or subjective

Questions for authors

see w

4 comments
AuthorsOpenReview ↗

Response to Reviewer's Concerns

We are pleased to hear that the reviewer found our work interesting, and we sincerely appreciate their careful evaluation and constructive feedback.

We would like to address the reviewer's concerns and provide our detailed responses to each point below.

Weakness 1. Clarification of Our Contributions

We understand the reviewer's concern regarding the extent of our methodological novelty relative to prior work.

We acknowledge the important contributions of existing task-aware approaches, particularly the analytical encoder–decoder framework for linear tasks. Nevertheless, we would like to emphasize that there remains meaningful room for improvement under restrictive privacy budgets, even for linear task models. We believe that achieving consistent utility improvements under strong privacy constraints, without relying on substantial latent-dimensionality reduction or noise-dependent decoder shrinkage, remains an important research challenge.

Our main contribution is the introduction of a new task-aware approach that leverages global active and inactive subspaces to improve representation utility under LDP, with applicability to both linear and nonlinear downstream task models.

Beyond this methodological contribution, our approach offers several additional advantages:

  1. Our experimental results demonstrate improved utility under restrictive privacy budgets without relying on substantial latent-dimensionality reduction or noise-dependent decoder shrinkage, including in linear regression settings where existing analytical task-aware approaches exhibit limited utility improvements. We provide a detailed analysis of these phenomena in our response to Question 1 of Reviewer oJD3, explaining how decoder shrinkage and latent-dimensionality reduction can arise in existing analytical task-aware approaches under restrictive privacy budgets.

  2. Our approach naturally accommodates nonlinear task models, as further supported by additional experiments involving MLPs and Vision Transformers (Appendix A.11, page 51 of the revised manuscript).

To further clarify these contributions and distinguish our approach from prior work, we have expanded the Related Work section (pages 2–3, lines 94–111) of our revised manuscript.

Weakness 2. Overstated or Unsupported Claims in the Introduction

We thank the reviewer for pointing out that some claims in the Introduction were not sufficiently supported by the presented evidence. We agree that the scope of our claims should accurately reflect our theoretical results and experimental findings.

To address this concern, we have carefully revised the Introduction to ensure that our claims are appropriately supported by the evidence presented in the manuscript. Specifically, we now explicitly identify the approximate mechanisms that our approach outperforms in the evaluated settings, as well as the existing advanced LDP mechanisms to which our approach can be applied.

These revisions are reflected in the Introduction (page 2, lines 81–89) of our revised manuscript. We hope that these changes adequately address the reviewer's concern and improve the precision and clarity of our claims.


We sincerely appreciate the reviewer's constructive feedback, which has helped us improve the clarity, accuracy, and presentation of our work. We hope that our additional explanations, experimental results, and revisions adequately address the reviewer's concerns regarding methodological novelty and the scope of our claims.

We believe that our proposed approach makes a meaningful contribution to improving representation utility under LDP for both linear and nonlinear task models. We hope that these clarifications and revisions further strengthen the reviewer's positive assessment of our work.

Reviewer_DDkHOpenReview ↗

I am pleased to maintain the positive score.

I am pleased to maintain the positive score.

AuthorsOpenReview ↗

Appreciation for the Reviewer's Constructive Feedback

We sincerely thank the reviewer for their positive assessment and continued support of our work. We greatly appreciate their thoughtful feedback, which has helped us improve the clarity and presentation of the manuscript.

AuthorsOpenReview ↗

Clarification of Novelty and Contributions

We thank the reviewer for the constructive feedback and for recognizing the novelty of using the Jacobian to guide task-aware noise allocation.

We agree that identifying important directions to improve utility is not a new concept, as prior studies have explored covariance-based, relevance-based, and principal-component-based approaches. We have therefore clarified that our contribution lies not in the general idea of identifying important directions, but in exploiting the Jacobian of a downstream model to identify task-sensitive subspaces and guide noise allocation under LDP.

We have also revised the manuscript to remove subjective or overly broad expressions and improve the precision and clarity of our descriptions.

These revisions have been incorporated into the revised manuscript and are highlighted in the Introduction (page 2, lines 61–80).

We sincerely thank the reviewer once again for the valuable feedback.

Reviewer_oJD3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper analyzes task-relevant subspaces by studying the Jacobian matrix and then optimizes the noise along these directions. The method is applicable to both linear and nonlinear downstream models. By designing corresponding preprocessing and postprocessing steps, the method can be directly combined with existing LDP perturbation mechanisms. Moreover, the transformations are determined entirely by public information and therefore do not introduce privacy issues. The experiments also demonstrate the effectiveness of the method on both linear and nonlinear downstream tasks.

Critical strengths
  1. The paper uses public information to obtain information about the model and the transformations. It also adopts a task-aware approach rather than focusing on the error between the original and perturbed data. This can achieve better performance than considering only data distortion, and the experiments validate this idea.
  2. The method is applicable not only to linear models but also to nonlinear models. It can also be combined with existing LDP perturbation mechanisms, which greatly improves its compatibility.
Critical weaknesses
  1. The paper uses a large amount of public data and it is unclear whether this is reasonable in real-world scenarios. Meanwhile, previous work (Nasr, M., Mahloujifar, S., Tang, X., Mittal, P., & Houmansadr, A. (2023, July). Effectively using public data in privacy preserving machine learning. In International Conference on Machine Learning (pp. 25718-25732). PMLR.) on the same datasets uses only 4% of the original training set as public data. The paper does not evaluate the effectiveness of the method by reducing the amount of public data, so it remains unclear whether the method can maintain its advantages when public data are limited.
  2. There are still some unclear points in the connection between the theoretical analysis and the actual experiments. The theory looks for the optimal transformation under a fixed noise scale, but in the experiments, taking the Laplace mechanism as an example, the author sets the clipping threshold based on the transformed public data, and this threshold further affects the noise scale. In other words, the transformation and the noise scale are actually interrelated.
Questions for authors

The author mentioned that the optimization goal of Task-Aware is MSE, which aligns with the task. However, PA also makes use of downstream task information. Can the author further explain why, in the linear regression experiments with a smaller privacy budget, Task-Aware still clearly outperforms several methods that include PA?

5 comments
AuthorsOpenReview ↗

Weakness 1. Amount of public data.

We sincerely thank the reviewer for recognizing the key strengths of our work, particularly its task-aware approach, applicability to both linear and nonlinear models, and compatibility with existing LDP mechanisms. We also appreciate the constructive comments and questions, which we address below.

W1. Amount of public data.

We thank the reviewer for raising this important concern. We agree that evaluating the effectiveness of our method with limited public data is crucial for its practical applicability.

In the revised manuscript, we clarify how the hyperparameters are selected in the Evaluation section (page 7, lines 357-360). We also discuss the impact of the amount of public data (page 10, lines 510-514) and provide additional experimental results in Appendix A.10.8.

Specifically, we evaluate PA using a reduced number of public samples on CIFAR-10-C (brightness corruption, severity level 5), where using N=500N=500 and N=50N=50 samples corresponds to using only 1.25% and 0.125% of the public training dataset, respectively. The average classification accuracy remains largely unchanged across the evaluated privacy budgets and LDP mechanisms, even with fewer public samples. However, the standard deviation tends to increase as the number of public samples decreases, indicating higher performance variability. Moreover, at ϵ=10\epsilon=10, PrivUnit2+PA and PrivUnitG+PA exhibit lower accuracy when only N=50N=50 public samples (0.125%) are used. These findings suggest that PA can maintain comparable average classification accuracy even with limited public data, although using more public samples improves performance stability and accuracy in certain settings, consistent with the observations of [1].

[1] Milad Nasr, Saeed Mahloujifar, Xinyu Tang, Prateek Mittal, and Amir Houmansadr. Effectively Using Public Data in Privacy Preserving Machine Learning. In Proceedings of the 40th International Conference on Machine Learning (ICML), volume 202, pages 25718–25732, 2023.

AuthorsOpenReview ↗

Weakness 2. Relationship between theoretical analysis and experimental implementation.

W2. Relationship between theoretical analysis and experimental implementation.

We thank the reviewer for pointing out this important distinction. We agree that the transformation and noise scale are interrelated in our experimental implementation, whereas our theoretical analysis assumes a fixed noise variance.

To clarify this distinction, we have added Remark 2 in the Appendix (page 20, line 1026–1046) of the revised manuscript. Specifically, our theoretical analysis minimizes the noise-induced component of the first-order task distortion under a fixed noise variance. However, in our implementation of the Laplace mechanism, the clipping threshold ρ\rho is determined from the transformed public representations and therefore depends on the transformation L\boldsymbol{L}. As a result, the noise variance also depends on L\boldsymbol{L}, as σ2(L)=8ρ(L)2/ϵ2\sigma^2(\boldsymbol{L}) = 8\rho(\boldsymbol{L})^2/\epsilon^2.

To further assess the impact of this distinction, we performed numerical optimization accounting for the transformation-dependent noise variance. Compared with our analytical solution, the numerically optimized configurations provided classification accuracy improvements of 0.0004,0.0007,0.0013,0.0014,0.0019,0.0024,0.0033,0.0038,0.0049,0.0055,0.0084,0.0125\\{0.0004, 0.0007, 0.0013, 0.0014, 0.0019, 0.0024, 0.0033, 0.0038, 0.0049, 0.0055, 0.0084, 0.0125\\} at privacy budgets ϵ∈0.5,1.0,1.5,2.0,2.5,3.0,3.5,4.0,4.5,5.0,7.5,10.0\epsilon \in \\{0.5, 1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0, 7.5, 10.0\\}, respectively, as reported in the Evaluation section (page 10, lines 498–503). We clarify that our analytical solution characterizes the optimal transformation under a fixed noise variance, rather than jointly optimizing the transformation and noise scale. We have also explicitly acknowledged this limitation in the Limitations section (page 10) and identified joint optimization of the transformation and clipping threshold as an important direction for future work.

AuthorsOpenReview ↗

Question 1. Comparison with Task-Aware (TA) [2] under Smaller Privacy Budgets

Q1. Comparison with Task-Aware (TA) [2] under Smaller Privacy Budgets

We thank the reviewer for this insightful question. Although both TA [2] and our PA use downstream task information, they differ in how task-relevant information is preserved under restrictive privacy budgets.

In our experiments, we observed that the analytical TA [2] consistently produced reconstructed representations close to the public-data mean when ϵ≤10\epsilon \leq 10. We believe that this behavior, which limits the preservation of input-specific information, may be explained by two aspects of the analytical TA [2]: decoder shrinkage and latent-dimensionality reduction.

First, the analytical TA [2] may exhibit a mean-collapse phenomenon due to decoder shrinkage.

As shown in Proposition 4.1 of [2], the optimal decoder for the linear setting with MSE loss is given by

D∗=E⊤(EE⊤+σw2I)−1D^* = E^\top(EE^\top+\sigma_w^2I)^{-1},

where σw2\sigma_w^2 denotes the noise variance.

Given the singular value decomposition E=Udiag⁡(si)V⊤E=U\operatorname{diag}(s_i)V^\top, the decoder gain along each singular direction is gi(D)=si/(si2+σw2)g_i^{(D)}=s_i/(s_i^2+\sigma_w^2), yielding an effective encoder–decoder reconstruction gain of αi=si2/(si2+σw2)\alpha_i=s_i^2/(s_i^2+\sigma_w^2).

For a fixed encoder, decreasing the privacy budget ϵ\epsilon increases the required noise variance, thereby reducing the effective reconstruction gain along each singular direction. When the noise dominates the encoded signal, (e.g. when ϵ≤10\epsilon \le 10), the decoder strongly attenuates input-specific information, causing the reconstructed representations to concentrate around the data mean. This phenomenon can substantially limit the information available for downstream prediction.

Second, this effect may be further compounded by a substantial reduction in the effective latent dimensionality.

As described in Proposition 4.6 of [2], the analytical solution may assign zero scaling factors to certain latent directions under restrictive privacy budgets, subject to the assumptions of the theoretical analysis. In our experiments, we observed that the effective latent dimensionality was reduced to a single dimension (Z′=1Z'=1 following notations used in [2]). Z′Z' can also be computed using Equation (11) in [2]. As a result, the encoder retained only a one-dimensional projection of the original representation, discarding variations along the remaining directions.

Although the decoder subsequently maps this one-dimensional noisy representation back to the original representation space, the discarded information cannot be recovered through reconstruction. Furthermore, the reconstructed representation is shifted by the mean estimated from public data. Since the decoder also strongly attenuates the remaining noisy signal when ϵ≤10\epsilon \le 10, the resulting representations tend to remain close to the public-data mean, exhibiting limited input-dependent variation.

Together, these two effects provide a plausible explanation for the limited utility of the analytical TA [2] under restrictive privacy budgets observed in our experiments. Although the cited theoretical results are derived for linear tasks with MSE loss, they also provide insight into how dimensionality reduction and noise-dependent decoder shrinkage can affect downstream utility.

In contrast, our PA approach preserves the representation dimensionality while redistributing privacy noise according to the task-relevant directions identified from the downstream model's Jacobian. Rather than compressing the representation into a lower-dimensional latent space and reconstructing it through a noise-dependent decoder, PA applies an anisotropic transformation to allocate relatively less noise to task-sensitive directions. This avoids the same dimensionality bottleneck and decoder-induced shrinkage, allowing more task-relevant variations to be retained under the evaluated privacy constraints.

Therefore, the apparent RMSE advantage of TA [2] under restrictive privacy budgets is largely attributed to its tendency to reconstruct representations close to the public-data mean, regardless of the original inputs, rather than to its ability to preserve input-specific information.

We hope that these explanations adequately address the reviewer's concern.

We have clarified these distinctions in the revised manuscript. Specifically, we highlight the methodological differences between TA [2] and PA in the Related Work section (pages 2-3, line 94–111). We also provide an explicit discussion of the observed performance differences under restrictive privacy budgets in the Evaluation section (pages 7-8, line 377–398).

[2] Jiangnan Cheng, Ao Tang, and Sandeep Chinchali. Task-aware Privacy Preservation for Multidimensional Data. In Proceedings of the 39th International Conference on Machine Learning (ICML), volume 162, pages 3835–3851, 2022.

Reviewer_4c6LOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a Jacobian-guided wrapper for LDP representation release. Public model Jacobians determine a task-aligned basis and noise scales, implemented through paired transformations and clipping. The paper provides constrained first-order optimality results and evaluates regression and classification utility.

Critical strengths
  1. The method provides an explicit public-model-based construction compatible with multiple LDP mechanisms, preserving their privacy guarantees under stated assumptions.
  2. The paper derives optimal basis and scale choices for a constrained first-order noise objective and reports substantial utility gains across several tasks and mechanisms.
Critical weaknesses
  1. The incremental advantage over existing task-aware and anisotropic methods is insufficiently established. Cheng et al. [1] already study task-aware encoding and decoding, while [2] covers weighted-error noise allocation and scale–perturb–rescale mechanisms. In Table 4, Laplace+PA performs similarly to Task-Aware, whereas the largest gains use PrivUnit. Matched comparisons are needed to isolate the proposed geometry’s contribution.
  2. The nonlinear evaluation uses a 10-unit first hidden layer for 64-dimensional inputs, creating a large global null space. A first-layer row-space projection baseline is missing. Moreover, removing only one side of the paired transformations changes the signal itself, limiting the component ablations. [1] Cheng et al. Task-aware Privacy Preservation for Multi-dimensional Data. ICML, 2022. [2] Muthukrishnan and Kalyani. Differential Privacy With Higher Utility by Exploiting Coordinate-Wise Disparity. TIFS, 2025.
Questions for authors
  1. Can you compare with Task-Aware using the same public data, feature representation, downstream model, and base randomizer, with valid privacy calibration?
  2. Can you compare against privatizing the first-layer row-space projection and evaluate models without a narrow first-layer bottleneck?
  3. Can you provide matched rotation-only ablations that remove scaling from both transformations and report noiseless performance?
6 comments
AuthorsOpenReview ↗

Weakness. Advantages over Prior Work

We sincerely thank the reviewer for their careful reading of our manuscript and for their thoughtful and constructive feedback.

To address the reviewer's concerns, we provide detailed responses to the identified weaknesses and questions below.

Weakness. Advantages over Prior Work

One of the main contributions of prior work [1] is the development of analytical encoders and decoders designed to improve utility for linear tasks under local differential privacy (LDP). These analytical approaches can achieve theoretically near-optimal utility under relatively weak privacy constraints (i.e., large values of ϵ\epsilon).

Beyond these analytical approaches, [1] also provides a learning-based framework applicable to more general settings. As demonstrated by the earlier results in [2], which adopts the same framework, this learning-based approach can achieve strong performance under relatively weak privacy constraints (i.e., large values of ϵ\epsilon).

While acknowledging these contributions, we identify several important research challenges that motivate our work:

  1. Stable utility under more restrictive privacy budgets: Developing a new task-aware approach that achieves improved and stable utility for linear tasks under ϵ\epsilon-LDP, particularly when ϵ≤10\epsilon \leq 10. We believe that the development of task-aware approach effective under restrictive privacy budgets remains an open research challenge, even for linear task models.

  2. Exploiting nonlinear model geometry: Developing a new task-aware approach that uses the geometric structure of nonlinear downstream models to improve the utility of privatized representations.

  3. Robustness to distribution shift: Developing a new task-aware approach that mitigates potential projection errors arising from distribution shifts between public and private data.

  4. Formal privacy guarantees: Developing a new task-aware approach with rigorous and explicitly established ϵ\epsilon-LDP guarantees.

For these motivations, our proposed approach (PA) offers several advantages over prior work [1], as detailed below.

Regarding Point 1 (Stable Utility under More Restrictive Privacy Budgets):

This advantage is supported by the experimental results presented in Tables 1 and 12 of our revised manuscript. Under ϵ\epsilon-LDP with privacy budgets of ϵ≤10\epsilon \leq 10, our PA achieves more stable utility than the analytical Task-Aware method (A-TA) in [1]. Specifically, Table 1 shows that the RMSE of A-TA remains nearly unchanged as ϵ\epsilon increases from 0.5 to 10, indicating limited utility improvements despite the relaxation of privacy constraints. This observation is consistent with our finding that A-TA tends to produce predictions close to the public-data mean, thereby limiting its ability to preserve input-specific information. In contrast, our PA does not exhibit this behavior and maintains consistently improved RMSE across the evaluated privacy budgets. Furthermore, Table 12 shows that our PA achieves higher classification accuracy than A-TA, providing additional empirical evidence of its effectiveness under restrictive privacy budgets.

We provide a detailed explanation of this behavior in our response to Question 1 of Reviewer oJD3. Briefly, the combination of substantial latent-dimensionality reduction and noise-dependent decoder shrinkage in A-TA may cause reconstructed representations to concentrate around the public-data mean under restrictive privacy budgets, thereby limiting downstream utility. We have also explicitly discussed the observed performance differences under restrictive privacy budgets in the Evaluation section (pages 7-8, lines 377–398) of our revised manuscript.

[1] Jiangnan Cheng, Ao Tang, and Sandeep Chinchali. Task-aware Privacy Preservation for Multidimensional Data. In Proceedings of the 39th International Conference on Machine Learning (ICML), volume 162, pages 3835–3851, 2022.

AuthorsOpenReview ↗

Weakness. Advantages over Prior Work

Regarding Point 2 (Using nonlinear model geometry):

To improve the privacy–utility trade-off of representations under LDP, our PA identifies global active and inactive subspaces using the Jacobian of the nonlinear downstream task model and leverages these subspaces to redistribute initially isotropic noise.

For linear models, the Jacobian is constant, and the resulting subspace decomposition is equivalent to that obtained through conventional matrix decomposition. In contrast, for nonlinear models, the Jacobian varies with the input. By accounting for this input-dependent variation, our PA identifies global subspaces that capture the task-relevant sensitivity structure of the nonlinear mapping [2]. Thus, unlike analytical approaches restricted to linear task models, our PA naturally extends to nonlinear settings.

To address the reviewer's concern, we provide additional experimental results in Appendix A.11 (page 51) of our revised manuscript, demonstrating that the effectiveness of PA extends beyond nonlinear multilayer perceptrons (MLPs) to more complex architectures, including Vision Transformers (ViTs).

Regarding Point 3 (Robustness to distribution shift):

This advantage is reflected in the scale-factor design introduced in Equation (6) on page 5 of our revised manuscript. In Corollary 3 (page 23), we establish a theoretical relationship between potential projection errors arising from distribution shifts and Tr⁡(Λ)\operatorname{Tr}(\boldsymbol{\Lambda}). Motivated by this relationship, our PA imposes an upper bound on each coordinate-wise scaling factor λi\lambda_i, thereby limiting its contribution to Tr⁡(Λ)\operatorname{Tr}(\boldsymbol{\Lambda}) and helping to mitigate potential projection errors under distribution shifts.

We further elaborate on the motivation for this design in Remark 1 following Proposition 1, while Corollary 3 provides the corresponding theoretical analysis of projection errors arising from distribution shifts. We have also explicitly clarified this aspect of our approach in the Related Work section (page 3, lines 109-111) and the Proposed Approach section (page 5, lines 259–260) of the revised manuscript.

Regarding Point 4 (Formal privacy guarantees):

Our PA uses global sensitivity, rather than empirical sensitivity in [1], together with appropriately calibrated noise to establish formal ϵ\epsilon-LDP guarantees. By calibrating the noise based on global sensitivity, we ensure that the privacy guarantees hold independently of the specific data samples, both in the theoretical formulation and in its implementation. All experimental results reported in our manuscript were also obtained using this global-sensitivity-based noise calibration, ensuring consistency between our theoretical privacy guarantees and experimental evaluations.

We have further clarified these distinctions in the Related Work section of our revised manuscript. Specifically, we provide a detailed comparison of the methodological differences between TA [1] and PA (pages 2-3, lines 94-111). We also explicitly highlight that, unlike our task-aware approach, the method proposed in [3] is task-agnostic (page 3, lines 114–115).

[1] Jiangnan Cheng, Ao Tang, and Sandeep Chinchali. Task-aware Privacy Preservation for Multidimensional Data. In Proceedings of the 39th International Conference on Machine Learning (ICML), volume 162, pages 3835–3851, 2022.

[2] Paul G. Constantine, Eric Dow, and Qiqi Wang. Active Subspace Methods in Theory and Practice: Applications to Kriging Surfaces. SIAM Journal on Scientific Computing, 36(4):A1500–A1524, 2014.

[3] Gokularam Muthukrishnan and Sheetal Kalyani. Differential Privacy With Higher Utility by Exploiting Coordinate-Wise Disparity: Laplace Mechanism Can Beat Gaussian in High Dimensions. IEEE Transactions on Information Forensics and Security, 20:2836–2851, 2025

AuthorsOpenReview ↗

Question 1. Comparison with Prior Work

Q1. Comparison with Prior Work [1]

We sincerely thank the reviewer for this valuable suggestion. We have been considering a more comprehensive experimental comparison with prior work involving nonlinear task models, and we appreciate the opportunity to further investigate this aspect.

Experimental setup and implementation details.

We provide an additional performance comparison on CIFAR-10-C (brightness corruption, severity level 5) using 64-dimensional representations extracted by ResNet20 in Appendix A.10.6 (pages 36–37) of our revised manuscript. We employ the standard Laplace mechanism in the representation space, as it is the primary mechanism considered in both papers. The downstream classifier is a ReLU-based MLP with an architecture of 64–10–32–10. All results are reported as the mean ± standard deviation over 20 random seeds.

To ensure a consistent comparison under the intended ϵ\epsilon-LDP guarantees, we revisited the sensitivity calculation and noise calibration in the official codebase of [1]. During this process, we identified aspects of the implementation that required modification to correctly recompute the sensitivity and calibrate the noise in accordance with the intended privacy guarantees. We therefore made the corresponding modifications to codesign/utils.py before conducting our experiments.

Experimental results and observations.

The performance trends of the learning-based method in [1] and [2] under ϵ≤10\epsilon \leq 10 were not straightforward to interpret. To better understand their behavior, we extended our evaluation to a broader range of privacy budgets, including values up to ϵ=20\epsilon=20.

Overall, our PA outperforms both learning-based variants under restrictive privacy budgets (ϵ≤10\epsilon \leq 10). Although both PA and the learning-based method exhibit limited utility under stronger privacy constraints (ϵ≤5\epsilon \leq 5), the learning-based method shows marginal improvements in classification accuracy as the privacy budget increases from ϵ=10\epsilon=10 to ϵ=20\epsilon=20.

Additional utility improvements through post-processing.

We further investigated whether the utility of PA could be improved by incorporating a denoising filter as a post-processing step. Our additional experiments demonstrate that applying such a filter can further improve classification accuracy without compromising the ϵ\epsilon-LDP guarantees of the underlying mechanism.

The classification accuracy of the resulting Advanced PA with the Laplace mechanism is summarized below (mean ± standard deviation over 20 random seeds).

Mechanismϵ=2\epsilon=2ϵ=5\epsilon=5ϵ=7.5\epsilon=7.5ϵ=10\epsilon=10
Advanced PA + Laplace (Ours)0.122 ± 0.0050.181 ± 0.0060.243 ± 0.0060.307 ± 0.006

Beyond denoising, optimizing the scaling factors according to the public data distribution may provide further utility improvements. Similarly, incorporating projection-based clipping could offer additional opportunities to enhance the privacy–utility trade-off.

While these additional optimization strategies show promising potential, we have not yet established theoretical guarantees for their utility improvements. Therefore, we have not incorporated these further extensions into the submitted manuscript and leave their more systematic investigation for future work.

[1] Jiangnan Cheng, Ao Tang, and Sandeep Chinchali. Task-aware Privacy Preservation for Multidimensional Data. In Proceedings of the 39th International Conference on Machine Learning (ICML), volume 162, pages 3835–3851, 2022.

[2] Paul G. Constantine, Eric Dow, and Qiqi Wang. Active Subspace Methods in Theory and Practice: Applications to Kriging Surfaces. SIAM Journal on Scientific Computing, 36(4):A1500–A1524, 2014.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Additional Evidence Following the Author–Reviewer Discussion

Dear Area Chair,

We sincerely thank you for your time and consideration, as well as the reviewers for their valuable and constructive feedback on our manuscript.

Below, we provide a summary of the key revisions made during the Author–Reviewer Discussion period.

1. Clarification of Methodological Contributions and Comparison with Prior Work

Reviewer 4c6L (Weakness 1); Reviewer oJD3 (Question 1); Reviewer DDkH (Weakness 1)

Our primary contribution lies in developing a Jacobian-guided anisotropic noise reshaping approach that exploits the global geometry of downstream task models to improve representation utility under LDP. We have clarified that our methodological contribution does not lie in the general concept of identifying task-relevant directions.

We summarize the key differences between the analytical Task-Aware (A-TA) approach of Cheng et al. (2022) and our proposed approach (PA).

AspectA-TAPA
Applicable task modelsLinear modelsLinear and nonlinear models
Subspace ConsiderationRow space of linear modelsActive subspace of linear or nonlinear models via their Jacobian
Behavior under restrictive privacy budgets (e.g., ε ≤ 10)Tendency toward mean-concentrated reconstructions due to decoder shrinkage and reduced effective latent dimensionality, observed in both the authors' official implementation and our independent implementationRetains input-dependent task-relevant variations and demonstrates utility improvements as ε increases in the evaluated settings
Distribution shiftDoes not explicitly address errors arising from public–private distribution shiftsEmploys a bounded scaling rule to limit the amplification of distribution-shift-induced first-order risk discrepancies
Compatibility with base privacy mechanismsSpecialized to the Laplace mechanismPre- and post-processing wrapper compatible with Laplace, AGM, PrivUnit2, and PrivUnitG

We provide a detailed discussion of the observed behavior of A-TA under restrictive privacy budgets in our response to Question 1 of Reviewer oJD3.

Based on the observation, we claim that our PA significantly contributes to task-aware approaches for both linear and nonlinear downstream models.

2. Additional Experiments

We have substantially expanded our experimental evaluation as follows:

  • Comparison with learning-based Task-Aware methods (Appendix A.10.6): We have demonstrated that PA achieves higher classification accuracy than the evaluated learning-based Task-Aware methods under restrictive privacy budgets (ε ≤ 10). (Reviewer 4c6L, Question 1)

  • Additional nonlinear models (Appendix A.11): We have extended our evaluation to Vision Transformers, providing further evidence of the applicability of PA to more complex nonlinear architectures. (Reviewer 4c6L, Weakness 1)

  • Architectures without narrow first-layer bottlenecks (Appendix A.10.11): We have evaluated wider MLP architectures and demonstrated that PA provides utility improvements beyond those attributable to narrow first-layer bottlenecks, including improvements over a first-layer row-space projection baseline. (Reviewer 4c6L, Question 2 and Weakness 2)

  • Matched ablation studies (Table 3): We have conducted additional rotation-only and noiseless ablation studies, demonstrating that rotation alone yields limited utility improvements and clarifying the importance of the combined pre- and post-processing transformations. (Reviewer 4c6L, Question 3 and Weakness 2)

  • Limited public data (Appendix A.10.8): We have demonstrated that PA maintains largely comparable average classification accuracy using only 0.125% or 1.25% of the public training data, although smaller sample sizes may lead to increased performance variability. (Reviewer oJD3, Weakness 1)

3. Clarification of Theoretical Scope

Reviewer oJD3 (Weakness 2)

We have clarified that our PA minimizes the noise-induced component of a first-order task distortion surrogate under a fixed noise variance. We additionally conducted numerical optimization accounting for transformation-dependent noise variance and reported the results in the revised manuscript (page 10, lines 497–503).

4. Refinement of Expressions in the Introduction

Reviewer DDkH (Weakness 2)

We have refined potentially subjective or overly broad expressions and clarified claims to better reflect the scope of the supporting evidence, while preserving the intended technical meaning.

5. Reviewer Feedback and Final Considerations

We appreciate that Reviewer DDkH maintained a positive assessment following our rebuttal. We also hope that our rebuttal has satisfactorily addressed the concerns raised by Reviewer DDkH, Reviewers oJD3 and 4c6L.

We hope that the additional evidence provided will be positively considered in the final assessment.

Thank you once again for your time and consideration.

Sincerely,

The Authors of Submission 10918

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.