- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper introduces a private method for federated topic discovery using bounded semantic compression. In this regard, the method uses LLMs to generate topic-relevant information for users, which are further bounded using a fixed and shared vocabulary in form of binary vectors. These vectors are securely aggregated after which the privacy noise is added and then sent to the server in the form of noisy population frequencies. The server can group the released noisy information into topics, in which the experiments show improvement for the recovery compared to several other methods.
Critical strengths
- The method does not depend on a trusted server setting, which enhances the privacy for each users data on edge devices.
- The experiments shows evidences of improvement when using semantic representations compared to other methods presented in Table 1.
- The topic recovery can still work when they are not directly named, rather implied. Table 3 shows the improvement in such case which I found interesting.
Critical weaknesses
-
The major weakness is in the evaluations. The paper reports metrics [email protected] and CS (section 5.1), however, I couldn’t find them to be established metrics used in literature. The literature in this field is quite diverse in terms of evaluation metrics, but for establishing new metrics the authors need to justfy their cause while also reporting established metrics for direct and faithful comparison.
-
The method shows mixed performance in Table 2, often underperforming when semantic representations are coupled with other existing methods. This shows the weakness for the end-to-end method when comparing with existing baselines. This is important because it reduces the overall impact of the method proposed.
-
The method does not show direct comparison with any semantic-based methods. I’m interested to know whether the method can improve over similar methods using semantic representation or not. The direct comparison of utility and topic recovery with Urania [1].
Due to these reasons, I believe the method itself has potential but the experiments and evaluations require revisions for the paper to be accepted.
References:
[1] Liu, D., Cohen, E., Ghazi, B., Kairouz, P., Kamath, P., Knop, A., ... & Zhang, C. (2025). Urania: Differentially private insights into ai use. arXiv preprint arXiv:2506.04681.
Questions for authors
Besides the issues addressed in the weakness section, I have the following questions to the authors.
-
The paper uses a secure aggregation protocol to combine the binary votes from every user (equation 1, line 207 & figure 1). What is the exact procedure of this aggregation? Does this protocol require the users to share raw binary vectors to a single central process or with other users?
-
The paper mentions usage of a shared vocabulary for constructing the binary vector. How is this shared vocabulary constrcuted, and if changing the vocabulary sizes appear to improve or reduce the performance?
-
Have there been any experiment for empirical evaluation of privacy in terms of attack methods?
I am willing to revise my score based on the authors’ response to the weaknesses and questions.
5 comments
Rebuttal to Reviewer zUKh [2/2]
W3. "No direct comparison with semantic-based methods, e.g. Urania." Summary: we added Urania and an encoder-retrieval baseline, and semantic compression beats both.Urania already uses an LLM, so this comparison isolates aggregation. The paper already includes semantic representations beyond lexical ones: SPLADE (learned sparse expansion), encoder-based extraction, and a one-pass LLM readout (Table 3). We add two comparisons.
- Urania. We applied Urania's private clustering and aggregation (our reimplementation; we will add the adaptation details to the appendix) to the same Table 2 client reports, with the same ε, δ and scoring. [email protected] on Yelp / GoEmotions / LMSYS: Urania .78 / .40 / .24 vs ours .94 / .60 / .72. Urania clusters records on a trusted server, so a term needs support within its own cluster, whereas our histogram pools all users; our trust model is also stronger, because no server sees individual data.
- Encoder + vocabulary retrieval (requested by Reviewer bVyP). Each document is embedded and its K nearest public-vocabulary words are retrieved (Table 3 setting for GoEmotions and code, Table 2 setting for Yelp and LMSYS). [email protected], semantic vs retrieval: GoEmotions .70 vs .14, Yelp .94 vs .83, LMSYS .72 vs .57, and code with absent labels ([email protected]) .83 vs .66.
Q1. "What is the secure-aggregation procedure? Do users share raw binary vectors?" Summary: no; users never reveal their vectors to the server or to each other. The protocol (Bonawitz et al., 2017, with distributed noise) has three steps:
- Each client i computes vᵢ ∈ {0,1}^d locally and adds a local integer noise share (Skellam).
- It masks the result with pairwise one-time pads, agreed with other clients via Diffie–Hellman key exchange. The masks are secret-shared, so they can be removed if a client drops out.
- The server receives only masked vectors. The masks cancel in the sum, so it learns only the noisy aggregate Σᵢvᵢ + noise (mod 2¹⁶).
Noise is therefore added before the server sees anything. The privacy accounting for this discrete, finite-field implementation under dropout is in the response to Reviewer KEnK (Q2). We will add the full details to the appendix.
Q2. "How is the shared vocabulary constructed; does its size matter?" Summary: it is a public word list fixed before any user data is seen, and results are stable from 20k words upward, whichever of two public corpora it is built from. The text vocabulary consists of the most frequent content words of a public corpus (alphabetic, longer than two characters, stop words removed). The paper names WikiText-103 as this corpus, but the default 20k vocabulary was built from 20 Newsgroups; WikiText-103 was used for the IDF weights of the code baseline. We will correct this in Section 5.1 and Appendix A. Building the vocabulary from WikiText-103 instead gives similar results (table below), so the choice of corpus does not affect our conclusions. Code uses 25k identifiers from the public Qwen2.5 tokenizer.
New sweep over corpus and size ([email protected], semantic / raw; Table 2 setting, same seeds as the re-organised table above):
| Vocabulary | GoEmotions | Yelp | LMSYS |
|---|---|---|---|
| 20 Newsgroups, 5k | .26 / .13 | .50 / .50 | .74 / .46 |
| 20 Newsgroups, 10k | .44 / .13 | .72 / .72 | .73 / .49 |
| 20 Newsgroups, 20k (default) | .60 / .12 | .94 / .83 | .72 / .48 |
| 20 Newsgroups, 50k | .63 / .10 | .94 / .83 | .72 / .48 |
| WikiText-103, 20k | .58 / .12 | 1.00 / .83 | .72 / .49 |
| WikiText-103, 50k | .62 / .11 | 1.00 / .73 | .72 / .48 |
| WikiText-103, 100k | .61 / .10 | 1.00 / .78 | .72 / .48 |
At 20k words or more, results are stable for both source corpora, and the semantic advantage holds at every size ≥ 20k. Small vocabularies (5k–10k) hurt semantic compression, because they lack the concept-level words it emits; on Yelp it then only ties raw. LMSYS is unaffected by size, because its intent labels are covered already at 5k. Vocabulary size has no privacy cost, since the noise scale depends only on K.
Q3. "Any empirical privacy evaluation with attacks?" Summary: yes; a canary membership-inference attack gives results consistent with the guarantee. We ran an optimal likelihood-ratio canary test against the calibrated Gaussian release (worst-case canary holding K unique words) over 300k trials per arm, sweeping the decision threshold. Across three independent audits, the median empirical ε (point estimate) is 0.30, 0.64 and 2.17 at target ε = 0.5, 1 and 3, so every calibrated run stays below its target. A deliberately under-noised control (σ/2) at ε = 1 reaches 1.59 > 1, which confirms the attack has power. These values correct the earlier version of this comment (0.93 at ε = 1), whose audit was run at a mis-set noise level. The formal guarantee (App. K) holds for any local representation, because sensitivity depends only on K. We will add a section on this to the revised version.
Response to the authors
Thank you to the authors for such detailed explanations to the weaknesses and questions I have raised. While most of my concerns are addressed, I still have a question about the secure aggregation protocol.
The equation 1 at page 4, and algorithm 1 shows the aggregation as averaging the binary vectors across clients. This is also true in the middle box in the algorithm 1, which shows raw aggregated across clients and then the noise is added. My question originally was, how do you achieve this aggregation without sharing the binary vectors between the clients? My understanding is that each client has their own binary vectors, and to aggregate them across clients you need to share them across clients. if that's the case, it doesn't hold that the clients are not sharing unperturbed information among each other, and therefore weakens the privacy.
Rebuttal to Reviewer zUKh [1/2]
We thank the reviewer for the detailed review and for the willingness to revise the score. We take the presentation score seriously. The revision will define STR and CS in Section 5.1 with their relation to established metrics, add a more comprehensive explanation of Table 2, and describe the secure-aggregation framework in more detail in the appendix.
As suggested in the official rebuttal guide, we have aimed for a compact response. We give numbers where appropriate and will add more extensive results to the revised version. We address the reviewer's concerns by weakness (W) and question (Q). For clarity, we occasionally split the response to a weakness into parts (e.g. W1a and W1b for weakness 1). We also occasionally paraphrase the Question or Weakness for compactness.
W1. "[email protected] and CS are not established metrics." Summary: STR and CS are semantic versions of established metrics; we add standard metrics below. STR is label recall, the usual utility measure for private heavy-hitter discovery (Zhu et al., 2020), computed with semantic instead of exact matching. We use semantic matching because exact matching would count frustration as a miss for the label frustrated, which penalises a representation for normalising word forms. CS is cluster–label alignment with Hungarian matching, as in standard unsupervised clustering accuracy. The main paper also reports LLM-judged usefulness of the released words (Figure 2c), and Table 4 reports Macro-F1 and ROC-AUC on Sentiment140, showing utility beyond clustering metrics. Nevertheless, we add established metrics below (semantic vs best lexical or raw baseline):
NPMI measures how often topic words co-occur in the source text, so it penalises descriptors that do not appear there (App. J); semantic compression still wins on it. We report F1 only on the controlled population (App. G): on the real corpora the reference labels are not exhaustive, so valid new topics would count as false positives.
W2. "Mixed performance in Table 2." Summary: within every certified mechanism, semantic compression improves SOTA methods considerably. The mixed picture comes from comparing mechanisms with different trust assumptions. The submitted Table 2 mixed mechanisms with different trust models and, in two rows (FedLDA, FedNMF), without a matched end-to-end guarantee. For example, on GoEmotions, FedLDA+LLM (.89) scored higher than SecAgg+LLM (.60), but FedLDA's local randomization is not accounted at the user level (App. J). We do not claim that our aggregator is the strongest. We claim that semantic compression helps regardless of which aggregator is used. To test this directly, we re-organised Table 2 along the privacy axis, with every certified mechanism calibrated exactly to (1, 10⁻⁶) under its own accountant (so some baseline cells differ from the submitted table) and FedNMF replaced by a zCDP-accounted DP-NMF:
Raw → semantic; text columns in the Table 2 setting, code in the Table 3 setting; means over 10 paired DP seeds. Selected rows are shown; the full table with all certified mechanisms (including Laplace, the exponential mechanism, TrieHH and local-DP OUE/RAPPOR) will be in the camera-ready version. Releases use the first 100 released words and, for GoEmotions, the label-free prompt reports of App. A.