- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper discusses user privacy in cloud-based LLM inference and proposes an information-theoretic approach. The method keeps semantic information locally and uses a mapper to replace the original prompt while preserving its structure. After the cloud LLM generates a response, a local recoverer reconstructs the final output. Experiments show a better privacy–utility trade-off than the baselines and robustness against various attacks.
Critical strengths
S1: Unlike previous semantics-preserving approaches, the authors separate text into structure and semantics and explain the effectiveness of their approach from an information-theoretic perspective.
S2: The method uses a learned cross-domain semantic mapping instead of direct content replacement to protect user privacy while preserving utility on certain tasks.
S3: The authors discuss multiple attack scenarios and show that the method provides some level of protection against different attacks.
Critical weaknesses
W1: The paper provides extensive information-theoretic proofs, but their connection to the actual mapper training and recoverer design is unclear.
W2: The method seems to require the LLM to reason correctly with preserved structure but replaced semantics, and the recoverer to reconstruct the final output correctly. This is a strong assumption. The method may fail on semantics-dependent tasks such as coding and math, but the authors do not evaluate such tasks.
W3: Table 2 uses three metrics defined in the paper, along with Downstream QA Utility and Recovered Text Utility, to demonstrate a better privacy–utility trade-off. Although the authors evaluate attack success rates in Section 6.2, evaluating privacy and downstream utility using attack success rates would provide more convincing evidence of the claimed trade-off.
W4: Table 2 uses Qwen2.5-14B as the base model for both the mapper and recoverer, which may require too much local computing power for typical API users. Moreover, Figure 4 in Appendix F.2 shows that performance strongly depends on the model sizes of the mapper and recoverer. This suggests that the method may be largely limited by the capabilities of its base models.
Questions for authors
Q1: Does the method work on semantics-dependent tasks such as coding and math?
Q2: If users can only run small models locally, would the method be limited by the base model capabilities and fail to achieve the performance reported in Table 2?
Q3: Since the method adds two local inference steps, have the authors measured the additional inference latency?