acceptodds
Under review as a conference paper at ICLR 2027

Private Federated Topic Discovery using On-Device Language Models

Abstract

Discovering recurring topics in user-generated text can guide improvements to AI and edge systems, but centralizing this text creates substantial privacy risks. We study distributed and privacy-preserving topic discovery, focusing on how user text is represented locally before private aggregation. We propose bounded semantic compression as a local representation method, whereby an on-device language model compresses each user's data into a bounded set of semantic terms. This addresses two limitations of existing the existing literature. First, it preserves more topic-relevant information under a fixed contribution bound, improving recovery after privacy noise is added. Second, it can name topics that are implied by the text but never stated directly, removing the requirement that a topic descriptor appear in the original text. We combine this representation with distributed private aggregation, providing user-level differential privacy for the released population topics without revealing individual user contributions. Across natural-language, synthetic, and source-code datasets, semantic compression improves topic recovery across private aggregation mechanisms, with the largest gains for multi-document users and topics that are implied rather than named.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Privacy, Topic Discovery, On-device LLM

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_zUKhOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces a private method for federated topic discovery using bounded semantic compression. In this regard, the method uses LLMs to generate topic-relevant information for users, which are further bounded using a fixed and shared vocabulary in form of binary vectors. These vectors are securely aggregated after which the privacy noise is added and then sent to the server in the form of noisy population frequencies. The server can group the released noisy information into topics, in which the experiments show improvement for the recovery compared to several other methods.

Critical strengths
  • The method does not depend on a trusted server setting, which enhances the privacy for each users data on edge devices.
  • The experiments shows evidences of improvement when using semantic representations compared to other methods presented in Table 1.
  • The topic recovery can still work when they are not directly named, rather implied. Table 3 shows the improvement in such case which I found interesting.
Critical weaknesses
  • The major weakness is in the evaluations. The paper reports metrics [email protected] and CS (section 5.1), however, I couldn’t find them to be established metrics used in literature. The literature in this field is quite diverse in terms of evaluation metrics, but for establishing new metrics the authors need to justfy their cause while also reporting established metrics for direct and faithful comparison.

  • The method shows mixed performance in Table 2, often underperforming when semantic representations are coupled with other existing methods. This shows the weakness for the end-to-end method when comparing with existing baselines. This is important because it reduces the overall impact of the method proposed.

  • The method does not show direct comparison with any semantic-based methods. I’m interested to know whether the method can improve over similar methods using semantic representation or not. The direct comparison of utility and topic recovery with Urania [1].

Due to these reasons, I believe the method itself has potential but the experiments and evaluations require revisions for the paper to be accepted.

References:

[1] Liu, D., Cohen, E., Ghazi, B., Kairouz, P., Kamath, P., Knop, A., ... & Zhang, C. (2025). Urania: Differentially private insights into ai use. arXiv preprint arXiv:2506.04681.

Questions for authors

Besides the issues addressed in the weakness section, I have the following questions to the authors.

  • The paper uses a secure aggregation protocol to combine the binary votes from every user (equation 1, line 207 & figure 1). What is the exact procedure of this aggregation? Does this protocol require the users to share raw binary vectors to a single central process or with other users?

  • The paper mentions usage of a shared vocabulary for constructing the binary vector. How is this shared vocabulary constrcuted, and if changing the vocabulary sizes appear to improve or reduce the performance?

  • Have there been any experiment for empirical evaluation of privacy in terms of attack methods?

I am willing to revise my score based on the authors’ response to the weaknesses and questions.

5 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer zUKh [1/2]

We thank the reviewer for the detailed review and for the willingness to revise the score. We take the presentation score seriously. The revision will define STR and CS in Section 5.1 with their relation to established metrics, add a more comprehensive explanation of Table 2, and describe the secure-aggregation framework in more detail in the appendix.

As suggested in the official rebuttal guide, we have aimed for a compact response. We give numbers where appropriate and will add more extensive results to the revised version. We address the reviewer's concerns by weakness (W) and question (Q). For clarity, we occasionally split the response to a weakness into parts (e.g. W1a and W1b for weakness 1). We also occasionally paraphrase the Question or Weakness for compactness.

W1. "[email protected] and CS are not established metrics." Summary: STR and CS are semantic versions of established metrics; we add standard metrics below. STR is label recall, the usual utility measure for private heavy-hitter discovery (Zhu et al., 2020), computed with semantic instead of exact matching. We use semantic matching because exact matching would count frustration as a miss for the label frustrated, which penalises a representation for normalising word forms. CS is cluster–label alignment with Hungarian matching, as in standard unsupervised clustering accuracy. The main paper also reports LLM-judged usefulness of the released words (Figure 2c), and Table 4 reports Macro-F1 and ROC-AUC on Sentiment140, showing utility beyond clustering metrics. Nevertheless, we add established metrics below (semantic vs best lexical or raw baseline):

MetricSemanticLexical / raw
Heavy-hitter P / R / F1 (Zhu et al.), controlled population, B = 50, vs raw clipping.999 / .702 / .812.505 / .134 / .206
Exact-label recall (string match, no embeddings): GoEmotions / LMSYS / synthetic.56 / .60 / .44.04 / .32 / .00
Within-topic NPMI, GoEmotions, 10 seeds.075 ± .017.005 (raw), .004 (TF-IDF)

NPMI measures how often topic words co-occur in the source text, so it penalises descriptors that do not appear there (App. J); semantic compression still wins on it. We report F1 only on the controlled population (App. G): on the real corpora the reference labels are not exhaustive, so valid new topics would count as false positives.

W2. "Mixed performance in Table 2." Summary: within every certified mechanism, semantic compression improves SOTA methods considerably. The mixed picture comes from comparing mechanisms with different trust assumptions. The submitted Table 2 mixed mechanisms with different trust models and, in two rows (FedLDA, FedNMF), without a matched end-to-end guarantee. For example, on GoEmotions, FedLDA+LLM (.89) scored higher than SecAgg+LLM (.60), but FedLDA's local randomization is not accounted at the user level (App. J). We do not claim that our aggregator is the strongest. We claim that semantic compression helps regardless of which aggregator is used. To test this directly, we re-organised Table 2 along the privacy axis, with every certified mechanism calibrated exactly to (1, 10⁻⁶) under its own accountant (so some baseline cells differ from the submitted table) and FedNMF replaced by a zCDP-accounted DP-NMF:

Trust modelMechanismGoEmotionsYelpLMSYSCode, label absent ([email protected])
DistributedSecAgg-Gauss (ours).12 → .60.83 → .94.48 → .72.39 → .83
DP-NMF (SecAgg + Gaussian, zCDP).04 → .59.83 → .94.49 → .65.13 → .43
Central curatorDPSU (weighted).12 → .68.83 → .94.52 → .71.12 → .64
No matched guaranteeFedLDA (authors' code).33 → .89.83 → .94.48 → .76n/a

Raw → semantic; text columns in the Table 2 setting, code in the Table 3 setting; means over 10 paired DP seeds. Selected rows are shown; the full table with all certified mechanisms (including Laplace, the exponential mechanism, TrieHH and local-DP OUE/RAPPOR) will be in the camera-ready version. Releases use the first 100 released words and, for GoEmotions, the label-free prompt reports of App. A.

AuthorsOpenReview ↗

Rebuttal to Reviewer zUKh [2/2]

W3. "No direct comparison with semantic-based methods, e.g. Urania." Summary: we added Urania and an encoder-retrieval baseline, and semantic compression beats both.Urania already uses an LLM, so this comparison isolates aggregation. The paper already includes semantic representations beyond lexical ones: SPLADE (learned sparse expansion), encoder-based extraction, and a one-pass LLM readout (Table 3). We add two comparisons.

  • Urania. We applied Urania's private clustering and aggregation (our reimplementation; we will add the adaptation details to the appendix) to the same Table 2 client reports, with the same ε, δ and scoring. [email protected] on Yelp / GoEmotions / LMSYS: Urania .78 / .40 / .24 vs ours .94 / .60 / .72. Urania clusters records on a trusted server, so a term needs support within its own cluster, whereas our histogram pools all users; our trust model is also stronger, because no server sees individual data.
  • Encoder + vocabulary retrieval (requested by Reviewer bVyP). Each document is embedded and its K nearest public-vocabulary words are retrieved (Table 3 setting for GoEmotions and code, Table 2 setting for Yelp and LMSYS). [email protected], semantic vs retrieval: GoEmotions .70 vs .14, Yelp .94 vs .83, LMSYS .72 vs .57, and code with absent labels ([email protected]) .83 vs .66.

Q1. "What is the secure-aggregation procedure? Do users share raw binary vectors?" Summary: no; users never reveal their vectors to the server or to each other. The protocol (Bonawitz et al., 2017, with distributed noise) has three steps:

  1. Each client i computes vᵢ ∈ {0,1}^d locally and adds a local integer noise share (Skellam).
  2. It masks the result with pairwise one-time pads, agreed with other clients via Diffie–Hellman key exchange. The masks are secret-shared, so they can be removed if a client drops out.
  3. The server receives only masked vectors. The masks cancel in the sum, so it learns only the noisy aggregate Σᵢvᵢ + noise (mod 2¹⁶).

Noise is therefore added before the server sees anything. The privacy accounting for this discrete, finite-field implementation under dropout is in the response to Reviewer KEnK (Q2). We will add the full details to the appendix.

Q2. "How is the shared vocabulary constructed; does its size matter?" Summary: it is a public word list fixed before any user data is seen, and results are stable from 20k words upward, whichever of two public corpora it is built from. The text vocabulary consists of the most frequent content words of a public corpus (alphabetic, longer than two characters, stop words removed). The paper names WikiText-103 as this corpus, but the default 20k vocabulary was built from 20 Newsgroups; WikiText-103 was used for the IDF weights of the code baseline. We will correct this in Section 5.1 and Appendix A. Building the vocabulary from WikiText-103 instead gives similar results (table below), so the choice of corpus does not affect our conclusions. Code uses 25k identifiers from the public Qwen2.5 tokenizer.

New sweep over corpus and size ([email protected], semantic / raw; Table 2 setting, same seeds as the re-organised table above):

VocabularyGoEmotionsYelpLMSYS
20 Newsgroups, 5k.26 / .13.50 / .50.74 / .46
20 Newsgroups, 10k.44 / .13.72 / .72.73 / .49
20 Newsgroups, 20k (default).60 / .12.94 / .83.72 / .48
20 Newsgroups, 50k.63 / .10.94 / .83.72 / .48
WikiText-103, 20k.58 / .121.00 / .83.72 / .49
WikiText-103, 50k.62 / .111.00 / .73.72 / .48
WikiText-103, 100k.61 / .101.00 / .78.72 / .48

At 20k words or more, results are stable for both source corpora, and the semantic advantage holds at every size ≥ 20k. Small vocabularies (5k–10k) hurt semantic compression, because they lack the concept-level words it emits; on Yelp it then only ties raw. LMSYS is unaffected by size, because its intent labels are covered already at 5k. Vocabulary size has no privacy cost, since the noise scale depends only on K.

Q3. "Any empirical privacy evaluation with attacks?" Summary: yes; a canary membership-inference attack gives results consistent with the guarantee. We ran an optimal likelihood-ratio canary test against the calibrated Gaussian release (worst-case canary holding K unique words) over 300k trials per arm, sweeping the decision threshold. Across three independent audits, the median empirical ε (point estimate) is 0.30, 0.64 and 2.17 at target ε = 0.5, 1 and 3, so every calibrated run stays below its target. A deliberately under-noised control (σ/2) at ε = 1 reaches 1.59 > 1, which confirms the attack has power. These values correct the earlier version of this comment (0.93 at ε = 1), whose audit was run at a mis-set noise level. The formal guarantee (App. K) holds for any local representation, because sensitivity depends only on K. We will add a section on this to the revised version.

Reviewer_zUKhOpenReview ↗

Response to the authors

Thank you to the authors for such detailed explanations to the weaknesses and questions I have raised. While most of my concerns are addressed, I still have a question about the secure aggregation protocol.

The equation 1 at page 4, and algorithm 1 shows the aggregation as averaging the binary vectors viv_i across clients. This is also true in the middle box in the algorithm 1, which shows raw viv_i aggregated across clients and then the noise is added. My question originally was, how do you achieve this aggregation without sharing the binary vectors between the clients? My understanding is that each client has their own binary vectors, and to aggregate them across clients you need to share them across clients. if that's the case, it doesn't hold that the clients are not sharing unperturbed information among each other, and therefore weakens the privacy.

Reviewer_KEnKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a privacy-preserving framework for federated topic discovery using on-device language models. The main idea is bounded semantic compression, where an LLM summarizes each user's local documents into a limited set of semantic descriptors before private aggregation. These descriptors are aggregated using a differentially private histogram with secure aggregation, providing user-level privacy for the released topics. The authors evaluate the approach on natural-language, synthetic, and source-code datasets, comparing it against lexical representations and several private discovery methods. The results show improvements in topic recovery, particularly for multi-document users and topics that are implied rather than explicitly mentioned in the text.

Critical strengths
  1. Relevant problem and practical approach: The paper addresses an important limitation of private federated analytics by focusing on how user data is represented before aggregation. Bounded semantic compression provides a useful way to preserve topic information under a fixed user contribution limit. The user-level DP formulation and compatibility with secure aggregation strengthen the practical motivation.
  2. Broad experimental evaluation: The authors evaluate multiple datasets, private aggregation mechanisms, and representation methods. The comparisons under matched contribution bounds are particularly useful. The experiments show improvements in recovering implicit topics and preserving topic information as user histories grow. The additional analyses of privacy budgets, downstream utility, and local computation provide further support for the proposed approach.
Critical weaknesses
  1. Limitations of the topic recovery evaluation: The main evaluation relies heavily on semantic topic recovery (STR), which considers a topic recovered when any released word exceeds a fixed embedding-similarity threshold with a reference label. This does not necessarily establish that the method recovers coherent or representative population-level topics. Although the paper includes cluster alignment, LLM-judged coherence, and synthetic grounding experiments, the conclusions remain dependent on the selected evaluation methods. Furthermore, the reported uncertainty primarily reflects different DP noise seeds while holding user data and LLM-generated representations fixed. This limits the assessment of robustness to different user samples and generated descriptors.
  2. Limited validation of the distributed privacy implementation: The formal privacy analysis establishes user-level DP for an ideal Gaussian mechanism, while the distributed experiments use finite-field aggregation with Skellam noise and dropout simulations. The paper provides useful numerical checks, but does not clearly establish the same end-to-end privacy guarantee for the evaluated finite-precision implementation. Moreover, local inference is primarily benchmarked on a relatively capable MacBook rather than resource-constrained mobile devices. These limitations leave uncertainty about the privacy guarantees and practicality of the complete federated deployment.
Questions for authors
  1. How sensitive are the topic recovery improvements to the choice of embedding model and similarity threshold? Would the conclusions remain consistent under alternative semantic evaluation methods or independently sampled user populations?
  2. Can the authors clarify how the stated user-level (ϵ,δ)(\epsilon,\delta)-DP guarantee applies to the finite-field distributed implementation using Skellam noise, particularly under client dropout? Does the reported privacy budget account for these implementation details?
  3. How representative are the multi-document experiments of actual federated user histories?
4 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer KEnK [1/2]

We thank the reviewer for the thorough assessment of the evaluation and of the privacy implementation.

As suggested in the official rebuttal guide, we have aimed for a compact response. We give numbers where appropriate and will add more extensive results to the revised version. We address the reviewer's concerns by weakness (W) and question (Q). For clarity, we occasionally split the response to a weakness into parts (e.g. W1a and W1b for weakness 1). We also occasionally paraphrase the Question or Weakness for compactness.

W1a. "STR counts a topic as recovered if any released word exceeds a threshold; this does not establish coherent or representative topics." We agree that STR measures recovery, not coherence, so we evaluate the other properties separately. We would like to point out that the main paper also reports cluster alignment (CS) and LLM-judged word usefulness (Figure 2c), and the appendix reports further metrics, including LLM-judged coherence. We restate these results and add new ones below:

  • Coherence (GoEmotions). Label-blind LLM judges rate semantic topics 3.87, vs 2.82–2.84 for raw and TF-IDF (Table 23, App. J). Within-topic NPMI is .075 vs .005 (raw) and .004 (TF-IDF), even though NPMI is biased against canonicalisation (App. J).
  • Representativeness / grounding (controlled population, App. G). Recall on present topics is .70 at an absent-topic FPR of .0004 (B = 50), and heavy-hitter F1 is .81 vs .21 for raw clipping (Tables 20–21). A preliminary human study also finds the released words grounded in their source documents (response to Reviewer bVyP, W2c).
  • Partition quality. Document–topic NMI is .70 vs .62 on the controlled population; on GoEmotions, ARI/NMI is .079 / .150 vs .012 / .097 (TF-IDF).

W1b / Q1. "How sensitive are the improvements to the embedding model and threshold? Would the conclusions hold under other evaluation methods?" We re-scored all releases on five datasets under five encoders: MiniLM, mpnet, bge-base, e5-base and gte-base. A fixed τ is not comparable across encoders, because under e5 and gte, 100 random words already match every label at τ = .7. We therefore calibrate τ per encoder and dataset, as the value at which random words match ≤ 5% of labels. Releases use the Table 3 setting for GoEmotions, synthetic and code, and the Table 2 setting for Yelp and LMSYS. At the calibrated threshold, semantic compression is better than the best lexical baseline (raw or TF-IDF) in 23 of 25 encoder × dataset cells, with 2 ties and no losses. Encoder retrieval is compared separately in our response to Reviewer bVyP (Q2). A subset of the results is shown below:

STR at calibrated τ: semantic / best lexical baselineGoEmotionsYelpSyntheticCode
MiniLM.71 / .16.94 / .83.62 / .26.94 / .48
bge-base.68 / .31.94 / .83.61 / .22.94 / .78
gte-base.68 / .30.94 / .83.56 / .24.94 / .56
Exact label match, no embeddings.56 / .04.83 / .83.44 / .00.11 / .00
AuthorsOpenReview ↗

Rebuttal to Reviewer KEnK [2/2]

W1c / Q1. "Uncertainty reflects only DP noise; robustness to user samples and generated descriptors?" We added both sources of variation (the release settings of the paper tables, 10 DP seeds per condition).

  • Generated descriptors. We regenerated every client report three times with sampling (temperature 0.7, top-p 0.95). Individual reports change substantially (token Jaccard to greedy decoding is .40–.55), but the spread across generations is small (sd ≤ .006). Sampling can shift the level (synthetic .46 → .50), and the advantage over the best baseline is unchanged. Generation variance is an order of magnitude below DP-noise variance, because aggregation over users averages out per-report randomness.
GreedySampled (mean ± sd over 3 generations)Best lexical baseline (raw / TF-IDF)
GoEmotions (Table 2 setting).596.595 ± .002.115
Synthetic (Table 3 setting).464.503 ± .006.100
Code, [email protected] (Table 3 setting).833.837 ± .006.389
  • User samples. We ran 20 bootstrap resamples and 5 disjoint half-splits (both halves evaluated), each with fresh DP noise. Semantic compression beats the best lexical baseline in 100% of resamples on every dataset. Half-split means (each half has n/2 users): GoEmotions .59 vs .04, Yelp .94 vs .83, LMSYS .61 vs .51, synthetic .47 vs .09, code .73 vs .34.

W2a / Q2. "How does the user-level (ε, δ) guarantee apply to the finite-field Skellam implementation under dropout?" It applies. The accounting below covers the deployed mechanism; we will add it to the appendix as a proposition with proof.

  • Discrete noise. The server observes Σᵢvᵢ + Z mod 2¹⁶, where Z ~ Skellam(λ) per coordinate (variance 2λ) is the sum of per-client shares. An exact privacy-loss-distribution accountant, run over the K = 8 coordinates one user can shift, shows that λ = 71.16 (variance 142.3) achieves (1, 10⁻⁶). The analytic Gaussian needs variance 142.8. The paper's configuration attains ε = 0.998. A closed-form RDP certificate (Agarwal et al., 2021) needs 14.5% more variance, and utility at that level is unchanged.
  • Finite field. Reducing mod 2¹⁶ with centre-lift is post-processing of the noisy integer sum, so privacy is unaffected. No wraparound occurred in any run: the largest count is 6,932 votes (LMSYS, 100k users), far below 2¹⁵ ≈ 32.8k. An adversarial worst case at 100k users would need 18 bits.
  • Dropout. We thank the reviewer for this question. Table 12 recalibrated the noise for each dropout arm to its retained cohort, which a deployment cannot do. We now fix each client's share before the round, sized for a tolerated dropout fraction d_max = 0.3, and abort the round beyond it. By Poisson closure, the noise retained from surviving clients is at least λ for any realised d ≤ d_max, so the guarantee holds without knowing who drops out. The table uses the threshold release of Eq. 3, whose central reference is .596; Table 12 used exact top-100, with a central reference of .696.
[email protected] (threshold release)Centrald = 0d = .1d = .3
GoEmotions.596.593.593.593
LMSYS.724.719.724.719
  • Collusion. Tolerating 10% colluding clients costs 16.7% more noise variance than the dropout-robust calibration, with STR unchanged.

W2b. "Local inference benchmarked on a MacBook rather than mobile devices." Table 11 gives per-client work independent of hardware (TFLOPs and KV-cache size), so the cost can be projected to any device. The smallest compressor, Qwen2.5-1.5B, needs 1.07 GB peak memory, which fits in current phones, and retains 82% of the 8B model's recovery. Reports are computed once per round and can run in the background while charging.

Q3. "How representative are the multi-document experiments of real federated user histories?"

  • Yelp: genuine accounts with persistent IDs; the earliest 20 reviews of 5,000 accounts with ≥ 20 reviews, on a user-disjoint split. Selecting accounts with ≥ 20 reviews favours heavier users. Real histories vary in length, which our nested m = 5/10/20 prefixes partly reflect.
  • Sentiment140 (Table 4): genuine accounts, 10 tweets each.
  • LMSYS: no user IDs, so its histories are random groupings, as stated in the paper.

On genuine accounts, the advantage grows with history length; at m = 20, Yelp gives .76 vs .66 for TF-IDF (Table 1). The advantage also persists when reviews are shuffled across accounts (Yelp, m = 20: .77 vs .51 for TF-IDF), so it is not an artefact of how histories are grouped. C1 concerns how much text must pass through a fixed contribution bound, which both settings test.

Thank you for the detailed responses and additional experiments. I appreciate the authors' efforts to address my concerns, particularly through the evaluation across different embedding models, robustness analyses, and clarification of the distributed privacy guarantees. These additions strengthen the paper. However, I still have some concerns about the generalizability of the topic-quality evaluation and the practical feasibility of deployment on resource-constrained devices. The preliminary human evaluation is helpful but remains limited in scope.

Overall, while the rebuttal addresses several of my concerns, I believe some limitations remain. Therefore, I will maintain my original rating.

AuthorsOpenReview ↗

Response to Reviewer

We thank the reviewer for engaging with our response and for noting that the additional experiments strengthen the paper. We would like to clarify the two remaining points and ask which specific aspects remain open.

Topic-quality evaluation. We evaluate topic quality with a broad set of metrics:

  • semantic topic recovery (STR) and exact-label recall without embeddings,
  • heavy-hitter precision, recall and F1,
  • cluster alignment (CS) and NMI/ARI,
  • within-topic NPMI and LLM-judged coherence,
  • LLM-judged word usefulness,
  • the absent-topic false-positive rate,
  • downstream Macro-F1 and ROC-AUC,
  • a preliminary human grounding study.

These are measured on real user accounts with multi-document histories (Yelp), real chat data (LMSYS), sentiment analysis on genuine accounts (Sentiment140), and source code, across different client cohorts and history lengths, nine private aggregation mechanisms, resampled user populations, and regenerated LLM reports.

Across all metrics and settings, the results point show that our framework improves performance. This goes beyond the evaluation used in the methods we compare against: private heavy-hitter and topic-discovery methods report recall-type metrics, and Urania, the closest LLM-based approach, relies on lexical, semantic-similarity and LLM-based metrics. Human evaluation is not standard in this line of work. We added the study as a further check, not as the main evidence. If the reviewer has a specific aspect of topic quality, metric, or setting in mind that these evaluations leave open, we would be grateful to know it.

Deployment on resource-constrained devices. Our contribution is the local representation and its effect on utility under user-level DP. For deployment, we report per-client compute independently of hardware (FLOPs and KV-cache size, Table 11), show that a 1.5B-parameter compressor needs 1.07 GB of peak memory, which fits current phones, while retaining 82% of the recovery of the 8B model, and give a formally accounted distributed implementation with finite-field secure aggregation and client dropout (App. K). Building and benchmarking a full on-phone inference pipeline is an engineering effort beyond the scope of this work. Methods in private federated analytics are typically evaluated in simulation, and our deployment analysis already goes further than is common.

We would appreciate it if the reviewer could point to the specific limitations behind the current rating, so that we can address them in the remaining discussion period or in the final version.

Reviewer_bVyPOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies federated topic discovery without centralizing users’ raw text. The authors propose bounded semantic compression, using an on-device language model to summarize each user’s local history into at most KK semantic terms. These bounded contributions are privately aggregated with Gaussian noise, and the released terms are grouped into topics using kk-means, providing user-level differential privacy for the final population-level topics. Experiments on natural-language, source-code, and synthetic datasets show improved topic recovery, particularly for multi-document users and topics that are implied rather than explicitly named.

Critical strengths
  1. The paper addresses a practical challenge: users’ text histories keep growing, while each user’s contribution remains bounded. Studying how to retain more information under a fixed contribution bound is well motivated.
  2. The experiments are extensive and provide a clear body of evidence. The multi-document, implicit-topic, and cross-mechanism experiments support the effectiveness of bounded semantic compression. The user-level sensitivity analysis and the use of DP post-processing are also clearly explained.
  3. The paper is well organized around two challenges: preserving information under a bounded user contribution and discovering topics implied rather than explicitly named in the text. The methods, experiments, and analysis are clearly connected to these goals.
Critical weaknesses
  1. The overall pipeline combines prompt engineering with existing language models, contribution clipping, a private histogram, and embedding-based clustering. The technical advances over prior work are not sufficiently clear.
  2. The paper lacks a baseline based on an on-device contextual encoder with public-vocabulary retrieval. STR may also favor canonical names, and there is no blinded human grounding evaluation of generated terms.
  3. The upstream steps of FedLDA and FedNMF do not have fully matched end-to-end user-level privacy accounting. The proposed method is also constrained by a fixed public vocabulary, a single-round setting, and on-device measurements conducted on a relatively powerful machine.
Questions for authors
  1. Which results in Table 2 have end-to-end user-level privacy guarantees matched to the proposed method?
  2. Can the authors report an on-device contextual-encoder plus public-vocabulary-retrieval baseline under the same users, KK, vocabulary, aggregation, and noise?
  3. Can the authors conduct a blinded human grounding evaluation of generated terms on natural data?
4 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer bVyP [1/2]

We thank the reviewer for the precise questions; each one led to a new experiment.

As suggested in the official rebuttal guide, we have aimed for a compact response. We give numbers where appropriate and will add more extensive results to the revised version. We address the reviewer's concerns by weakness (W) and question (Q). For clarity, we occasionally split the response to a weakness into parts (e.g. W1a and W1b for weakness 1). We also occasionally paraphrase the Question or Weakness for compactness.

W1. "The pipeline combines prompt engineering, existing LMs, clipping, a private histogram and clustering; the technical advance is unclear." Our contribution is not a new aggregator. Prior work focuses on how user data is privately aggregated, whereas we argue that the local representation is equally important, a question largely unaddressed in the literature. We show experimentally and theoretically why a better representation improves utility at the same privacy level, enabling high-utility federated topic discovery under user-level DP without a trusted server. The answer lies in the semantic capabilities of on-device LLMs. We conceptualise the local representation through two properties (detailed in the paper):

  • (C1) Concentration under a bounded representation. Under a fixed per-user bound, compression maps related content onto shared coordinates, which raises population support (Eq. 6). Of the topics both methods recover without noise, more survive privacy noise with semantic compression: 91% vs 79% on Yelp and 63% vs 29% on LMSYS, vs TF-IDF (Table 9).
  • (C2) Abstraction. Extractive representations are capped by literal availability (Eq. 8), with a closed-form support threshold for recovery (Eq. 10). On code whose functionality name never appears, recovery is .83 vs .32 for same-model extraction (Table 3).

The novelty lies in (1) the conceptual shift from aggregation to local representation; (2) characterising local representation through concentration (C1) and abstraction (C2), and showing how these yield high utility under user-level DP; and (3) showing that the representation is a drop-in step that improves existing private discovery mechanisms.

W2a / Q2. "Missing baseline: on-device contextual encoder + public-vocabulary retrieval, under the same users, K, vocabulary, aggregation and noise." We added it exactly as specified. Each document is embedded with a sentence encoder (MiniLM, mpnet or bge-small), and its K nearest public-vocabulary words are retrieved, so retrieval is not restricted to source words. We also tested near-synonym-filtered and MMR variants. Users, K, vocabulary, Gaussian/SecAgg release and the 10 seeds are unchanged, with the Table 3 setting for GoEmotions and code and the Table 2 setting for Yelp and LMSYS.

[email protected]GoEmotionsYelpLMSYSCode, label absent ([email protected])
Encoder + retrieval (best variant).137.833.571.656
Semantic compression.696.944.724.833

The ordering holds when the releases are re-scored, at a calibrated threshold, with an encoder that neither method uses (gte-small). Retrieval returns the subject rather than the implied attribute; its top GoEmotions words are name, thanks and crosspost. Its votes also disperse over many more distinct words (5–9k), so fewer coordinates clear the DP threshold. Since retrieval is itself non-extractive, this is consistent with the gain on natural data coming from abstracting to the implied concept.

W2b. "STR may favour canonical names." We added metrics that involve no single embedding threshold:

  • Exact-label recall (string match, no embeddings): GoEmotions .56 vs .04, LMSYS .60 vs .32, synthetic .44 vs .00, code .11 vs .00.
  • Threshold-free and calibrated scoring: under five encoders, both the area under the STR(τ) curve and STR at a null-calibrated threshold favour semantic compression over the best lexical baseline.
  • Established metrics: heavy-hitter F1 .81 vs .21 (vs raw clipping) and NPMI .075 vs .005 (details in the response to Reviewer zUKh, W1).

Canonical names are also not released when unsupported. In the absent-topic control, a topic held by no user is matched at a rate of only .0004 at B = 50, with precision .999 (Tables 20–21).

AuthorsOpenReview ↗

Rebuttal to Reviewer bVyP [2/2]

W2c / Q3. "No blinded human grounding evaluation." Due to the limited time in the rebuttal period, it is difficult for us to establish and execute a comprehensive human-grounded study of the released vocabulary. However, we were able to obtain preliminary results from four human evaluators (non-authors) on the key released terms of the Code and Yelp datasets. We will adjust the Ethics Statement accordingly to note that we included a small human study. For 25 documents per dataset, each evaluator saw the document together with up to two words per method from its own client report that survived the release, plus a control word from another document. The words were shuffled and the method was not shown, and evaluators labelled each word supported, partial or unsupported. A document counts as covered if at least one of these words is judged supported. Agreement is high (Fleiss' κ = .81), and the control word was judged supported in only 2–6% of cases.

Covered (95% CI)SemanticTF-IDFRaw
Code.94 [.85, 1.00].52 [.34, .70].29 [.13, .47]
Yelp.80 [.66, .91].61 [.44, .77].41 [.23, .61]

On code, semantic compression beats both baselines (p ≤ .001). On Yelp it beats raw text (p = .002); against TF-IDF the gain has the same direction for all four evaluators but is not significant (p = .09).

We also note that related work uses LLM judges as proxies for human judgement. For this, we have a dedicated section in the appendix (App. I), and Figure 2c reports how LLM judges rate the usefulness of the released GoEmotions terms across methods. Semantic compression shrinks the release from 238 to 51 words while keeping about the same number of informative words, so 96% vs 22% of released words are informative. These results align with our preliminary human study, which also finds that our method outperforms the baselines.

Our population-level evidence, in which the compressor never sees the reference taxonomy, points the same way:

  • Absent-topic false-positive rate .0004 at B = 50 and precision .999 (Tables 20–21).
  • Out-of-taxonomy topics planted into 1% of accounts are named in 68% of trials, with zero false recoveries at 0% (Table 22).

W3a / Q1. "Which Table 2 results have end-to-end user-level guarantees matched to the proposed method?" Grouped by guarantee (we will add this as a column to Table 2):

  • Matched user-level (1, 10⁻⁶): SecAgg-Gauss, Laplace, DPSU (trusted curator), the exponential mechanism, and TrieHH (central DP via sampling and secure aggregation).
  • Local DP: RAPPOR/OUE.
  • Not matched: FedLDA (no whole-history accounting) and FedNMF (no formal DP).

Table 2 is intended as a within-row raw → +LLM comparison. For a fully matched answer, we built a new table in which every certified mechanism is calibrated exactly to (1, 10⁻⁶) under its own accountant, and FedNMF is replaced by a zCDP-accounted DP-NMF with total budget (1, 10⁻⁶); FedLDA is shown separately as unaccounted. Semantic representation significantly improves performance, for example DP-NMF on GoEmotions rises from .04 to .59.

W3b. "Fixed public vocabulary." The vocabulary is public and can be swapped at no privacy cost. In a new sweep from 5k to 100k words over two public corpora, results are stable at 20k words or more (Yelp .94–1.00, LMSYS .72 at every size ≥ 20k), and the semantic advantage holds at every size ≥ 20k (table in the response to Reviewer zUKh, Q2).

W3c. "Single-round setting." The release composes cleanly. In a three-round adaptive pipeline (a histogram, then two refinement rounds), each round runs at ε ≈ 0.94 (δ = 10⁻⁶). Exact Gaussian composition gives a total of ε = 1.70 at δ = 10⁻⁶, versus 2.10 for the standard zCDP conversion and (2.81, 3×10⁻⁶) for basic composition. This shows that multi-round operation is feasible with tight accounting; we do not claim that additional rounds improve recovery.

W3d. "On-device measurements on a powerful machine." Table 11 reports per-client work independently of hardware (e.g. a median 2.44 TFLOPs per document for the 8B model), so the cost can be projected to any device. The smallest compressor, Qwen2.5-1.5B, needs 1.07 GB peak memory, which fits in current phones, and retains 82% of the 8B model's recovery (.615 vs .752). Reports are computed once per analytics round and can run in the background while the device charges, as is standard for federated workloads.

Thanks the authors for providing detailed rebuttals and conducting supplementary experiments. The authors addressed my concerns, particularly regarding privacy accounting (W3a) and the inclusion of retrieval baselines (W2a). However, some reservations remain concerning the overall contribution of the paper and the rigor of the methodology. In light of this, I will maintain my current score.

AuthorsOpenReview ↗

Response to Reviewer

We thank the reviewer for the careful engagement and are glad that the concerns on privacy accounting (W3a) and the retrieval baseline (W2a) are resolved. We would like to briefly address the two remaining reservations.

Contribution. Prior work on private federated discovery improves how user data is aggregated. We show that the local representation, i.e., what is aggregated, is equally important and largely unaddressed in the current literature. We characterize it through concentration (C1) and semantic abstraction (C2), derive when each improves recovery under user-level DP, and show that it is compatible with, and improves, existing private mechanisms for topic discovery.

Rigor. The method rests on a formal user-level guarantee that also covers the deployed implementation (discrete noise, finite-field secure aggregation, dropout; App. K), an empirical privacy audit, a comparison with every mechanism calibrated to the same budget, paired DP seeds with corrected significance tests, robustness to user samples, LLM generations, scoring encoders and vocabulary, and established metrics beyond STR.

In the initial review, the reviewer found the paper clear and easy to follow, reported gaining a substantially deeper understanding of the problem, and described the soundness concerns as minor or fixable. As the concerns raised have now been addressed, could the reviewer pin down which aspect of the contribution or which specific step of the method they still consider insufficient? We would be glad to address it in the final version.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank all three reviewers for their careful and constructive reviews, and we are glad that each of them reports learning something interesting or useful from the paper. The reviewers find the problem practical and well motivated (bVyP, KEnK), and highlight that focusing on how user data is represented before aggregation addresses an important limitation of private federated analytics (KEnK). They describe the experiments as extensive and broad, with a clear body of evidence across datasets, mechanisms and matched contribution bounds (bVyP, KEnK). They also value that the method needs no trusted server (zUKh), and find the recovery of implied, unnamed topics interesting (zUKh; also noted by bVyP and KEnK). The reviews raised four themes, and we address each with new experiments with details in corresponding reviewer rebuttal:

1. Evaluation beyond STR (bVyP W2, KEnK W1/Q1, zUKh W1). We added established metrics (heavy-hitter precision/recall/F1, exact-label recall without embeddings, and NPMI), re-scored all releases under five embedding encoders with calibrated thresholds, and tested robustness to resampled user populations and regenerated LLM reports. We also ran a preliminary blinded human study of whether released words are grounded in their source documents. Every evaluation points the same way as STR: semantic compression recovers more topics, and its released words are more often judged supported by the source text.

2. Privacy matching and the distributed implementation (bVyP Q1/W3, KEnK Q2/W2, zUKh W2/Q1/Q3). We re-organised Table 2 by trust model, with every certified mechanism calibrated to the same user-level budget (1, 10⁻⁶). Semantic representation helps under every trust model. We also give formal privacy accounting for the deployed implementation (discrete Skellam noise, finite-field secure aggregation and client dropout), which attains the stated guarantee, and an empirical membership-inference audit that is consistent with it.

3. Stronger and semantic baselines (bVyP Q2, zUKh W3). We added Urania and an on-device encoder with public-vocabulary retrieval, run under the same users, contribution bound, vocabulary and noise. Against retrieval, this indicates that the gain comes from abstraction. Nevertheless, we also discuss the importance of aggregation for distributed and user-level DP.

4. Practicality (bVyP W3, KEnK W2/Q3, zUKh Q2). Results are stable across public vocabularies of 20k words or more built from two different corpora, the release composes cleanly over multiple rounds, and a 1.5B-parameter compressor fits in about 1 GB of memory while retaining most of the recovery of the 8B model.

More detailed responses to each review is given below accordingly.

We have uploaded a revised PDF with the changes requested. Changes are shown in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.