- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes CALLIOPE for discovering topic hierarchies in text data under differential privacy. An LLM summarizes each document and extracts keywords. The keyword counts are released with private selection, and an LLM proposes sub-topics from the released keywords. Documents are then assigned to the sub-topics, and the process is repeated within each branch. The authors compare against CLIO (non-private) and URANIA (private) on six datasets, using intrinsic and extrinsic clustering metrics and LLM-judge based metrics.
Critical strengths
- Using only the keyword selection as the private step is a nice idea, and the privacy guarantee follows easily from this.
- The method degrades to a coarser taxonomy as gets smaller, instead of breaking down like DP clustering.
Critical weaknesses
- The paper presents avoiding embeddings as the main novelty, but URANIA also does text-based discovery. It uses embeddings for the clustering, but the keyword part is a DP histogram over text that an LLM decodes. Private discovery over discrete vocabularies also exists in DPSU, TrieHH and private LDA, and none of these are discussed. So what remains is using an LLM to build the hierarchy top-down, which is a small step beyond URANIA. This matters because the contribution then rests on the experiments. What would help is clearly stating what new insight this paper brings, that we didn't know beforehand.
- Apart from Clio and Urania, no other baselines are evaluated. There are no baselines that use a foundation model directly. For example, the authors say that CALLIOPE approaches zero-shot prompting at small , but they never evaluate prompting an LLM for a taxonomy without any private data. A non-private text-based method like TopicGPT would also be a useful reference. This matters because the document assignment is a non-private LLM call, and the only data-dependent information is the keyword list. Without these baselines I cannot tell how much of the quality comes from the DP signal and how much from the LLM’s prior knowledge.
Questions for authors
-
I don't mind that the paper is more on the engineering side, but what is the new insight that we gain from it? From what I can tell, the hierarchy is discovered by calling an LLM on a privately selected keyword list. Is there something that we learn beyond the fact that this works, for example about what can be recovered privately, or about how the budget should be spent across levels?
-
Can the authors discuss how this relates to DPSU, FedLDA and other private text-based discovery methods (e.g. TrieHH), and say how they are different? It is fine if these methods only learn higher-level clusters or just terms. But it would help to include them as baselines or at least compare what they find to the CALLIOPE hierarchy.
-
Can the authors add baselines that use a foundation model directly? For example, prompting an LLM for a taxonomy of the dataset without any private data, and a non-private text-based method like TopicGPT as a reference. How much of the quality of CALLIOPE at small comes from the private signal, and how much from the LLM's prior knowledge?
-
Is the in the figures the total budget or the budget per level? What are the contribution bound , and the split of across levels? Was URANIA run with a comparable total budget?
-
How many human raters and items were used in Table 1? Why are the gains in Table 1 so much smaller than with the LLM judges in the appendix?
-
One of the key challenges Urania highlights is transfer to user-level DP. This paper also does record-level DP. Can the authors comment how their approach could transfer to user-level DP?
4 comments
Response to Reviewer msxj (2)
- Is the in the figures the total budget or the budget per level? What are the contribution bound , and the split of across levels? Was URANIA run with a comparable total budget?
The privacy parameters for Calliope are the total budget. The budget was split evenly between levels, and for each level we split the budget evenly between the two queries: keyword discovery and size estimation. Urania was run with double the budget (we allocated full budget to each part of Urania). This avoids the claim that Urania performed worse because of the budget allocation. As for sensitivity, we ran experiments in the record-level privacy regime, so for each level the keyword sensitivity is 5 and for size sensitivity is 1. If anything, giving Urania 2x the budget should only favor Urania.
- How many human raters and items were used in Table 1? Why are the gains in Table 1 so much smaller than with the LLM judges in the appendix?
For human evaluation, 295 raters were recruited via Prolific. The raters were prescreened for English fluency, age over 18, and history on the platform. For each algorithm comparison, 10 independent runs generated 10 questions each, for a total of 100 questions. Since there were 15 different algorithm pair comparisons, this created a bank of 1500 questions. A rater would be given 5 of the 1500 questions to evaluate. We allowed raters to rate a second time, which 5 raters did (the available slots went quickly). For each question, the rater was to select from 1 of 3 choices: A is better, B is better, or they are both equal. The positions of the algorithm output for A and B were randomized.
Note that because a single rater rates a question, we are unable to ascertain uncertainty or agreement between human raters for all of these evaluations. The ratings cost approximately 500 USD and attempts to answer the question of agreement would require at least 3 human ratings per question for an additional 1000 USD. We are certainly interested in the question of confidence or agreement between human raters but decided to focus our research funds on evaluating different datasets and epsilon values. In a previous experiment, we asked 30 questions (on the Emotions Dataset between Calliope and Urania) to 3 separate humans to test agreement. Two out of 30 were unanimous, 23 out of 30 had a 2-out-of-3 majority, while 5 had a complete disagreement (1 for A, 1 for B, 1 Tie). This validates the difference in Table 1 compared to the LLM judges in the appendix, because the same type of noise is not observed in the LLM judges.
Prolific has an ethics protocol that all the raters agree to. No PII was collected from the raters although we received coarsened metadata about them (country, age, employment status, etc.). We provided an informed consent disclosure outlining the task and they had the right to withdraw at any time without penalty. We screened the questions to ensure that no sexual or violent text was present in the question, either in the dataset or hallucinated from the cluster names. Without revealing our institutional requirements, we are exempt from requiring an IRB for these types of benign behavioral research without PII. We have added an ethics statement for transparency on our utilization of human raters.
- One of the key challenges Urania highlights is transfer to user-level DP. This paper also does record-level DP. Can the authors comment how their approach could transfer to user-level DP?
The CALLIOPE algorithm can be directly used with user-level DP by using a contribution bound on the number of keywords contributed by a user. We evaluate all algorithms at the record level and we consider evaluating the user-level DP variant as an interesting future direction.
We hope our responses allay the concerns you raised. Please let us know if you have additional issues during the discussion period. Thank you for your time in reviewing our work.
Response to Authors
I thank the authors for the revision. The paper is clearly better organized, and adding TopicGPT and discussing related methods (DPSU, TrieHH, private LDA) both help. Some points from my initial review are still open, and I list them below together with a small number of requests.
Not yet addressed (from my initial review)
- Zero-shot baseline (my third question): The rebuttal mentions a keyword-free variant, but it is not in the revision. Without it, the reader cannot tell how much of CALLIOPE's quality comes from the private signal and how much from the LLM's prior.
- Budget allocation across levels (my first question): The budget is split evenly, and footnote 1 notes that this could be optimized. No alternative split is evaluated.
Requested ablations (there is no need to re-run everything, I understand the time constraint and how much effort it takes to run all these experiments. A single ε, e.g. ε = 1, on two or three datasets, evaluated with one or two metrics of the authors' choice, would be enough)
- LLM prior vs. private signal (from my initial review): compare a zero-shot taxonomy with private CALLIOPE and non-private CALLIOPE.
- Budget split across levels (from my initial review): compare the even split with allocations that put more budget on the top level or on the deepest levels.
- Keyword extractor (new): replace the LLM keyword extraction with a classical extractor (e.g. TF-IDF or KeyBERT) to test whether the LLM is needed in the abstraction step.
Presentation and layout
- Figures 2 and 3 use 3×3 grids for seven datasets, so CLINC-OOS sits alone in the last row and each figure takes about half a page. A 2×4 layout with the legend in the eighth slot, or a merged figure with a few representative datasets in the main text and the full grids in the appendix, would free space for the ablations above.
- Figure 11 still has "To be added" placeholders for the WildChat judges.
Minor
- Please state the contribution bound c (and the sensitivities given in the rebuttal) in §4.
- Please add a short version of the rebuttal's DPSU/TrieHH/LDA discussion to the related-work section.
- Reproducibility: I appreciate the effort made here: the pseudocode, the verbatim prompts, the datasets, models and hyperparameters, and the reproducibility and ethics statements are all helpful. Since the method chains many LLM calls, I would also appreciate it if the final version included the code (anonymized now, or released upon acceptance).
Overall the paper is in a better state now, and if my remaining concerns and questions are addressed I would be willing to raise my score.
As an addition I would encourage the authors to also think about adding further ablations, which they think would improve the paper. What I listed above is enough for me to change the score, but in general for future readers I think there could be more details about the individual method components which should be tested experimentally.
We thank the reviewer for the thoughtful follow-up and for constructive suggestions that have substantially strengthened the paper. We have updated the manuscript to address all of the remaining points and requested ablations. Below we summarize where each addition appears in the revised paper.
- LLM Prior vs. Private Signal — Zero-Shot Baseline (Appendix F, Figures 13–20): We added Appendix F ("Impact of Keyword-Driven Synthesis"), which compares Calliope against a data-oblivious variant, Calliope-without-keywords. We evaluated it on the same suite of parameters as comparisons between Urania and Calliope. Calliope outperforms Calliope-without-data on almost all configurations with largest gains on fine-grained intent and open-ended conversational corpora
- Privacy Budget Allocation Across Levels and Queries (Appendix H, Table 1, Figures 22–23; Footnote 1 in Section 2.2): We updated Footnote 1 in Section 2.2 to reference Appendix H ("Alternative Splits of the Privacy Budget"), where we evaluate seven alternative splits of the same total privacy budget against the default even split. We observed that none of the seven alternatives consistently improves over the even split.
- Classical Keyword Extractor vs. LLM Keyword Extraction (Appendix G, Figure 21): We added Appendix G ("Impact of the Keyword Extractor"), which replaces the LLM keyword extraction with an LLM-free classical extractor. Evaluated on Banking77 and Emotion, LLM-extracted keywords are statistically significantly preferred on almost all configurations with one tie.
We also updated figures style and added accidentally missing comparisons for WildChat for other LLM judges.
Finally, we added a note about contribution bound and query sensitivities to Section 4 ("Experimental Settings", p. 6) and added a discussion of DPSU, TrieHH, and Private LDA to Section 5 ("Related Work", p. 9).
With regards to the reproducibility request: we are working on creating a version that can be shared. During this rebuttal phase, we focused effort on the additional experiments and could not prioritize the engineering effort to create an approachable open source implementation. We will try to include it in the final version.
Response to Reviewer msxj
We thank reviewer sxjR for their review, including their concerns listed in "Critical Weaknesses" and "Questions For Authors." We hope you find the below responses satisfactory:
We view our main contribution as proposing a DP algorithm for creating named clusters that beats the state of the art, which we demonstrate empirically. While the individual techniques that form the building blocks of our algorithm were known, the specific way we combine them is novel and provides improved performance.
Thank you for the suggestion. While we had considered these three algorithms, they are solving slightly different problems than CALLIOPE:
We have now added measurements for TopicGPT in our updated manuscript. We did not include this in the first version because we only compared against approaches that could be modified to preserve privacy; TopicGPT goes through the dataset sequentially, which makes it hard to modify to satisfy differential privacy.
We are running some experiments and hope to add some numbers for versions of Calliope that don’t use keywords.