acceptodds
Under review as a conference paper at ICLR 2027

CALLIOPE: Private Insights into AI Use, without Embeddings

14 views

Abstract

We propose _CALLIOPE_, a top-down framework for discovering topic hierarchies in large corpora of unstructured data (e.g., LLM conversations), while supporting differential privacy (DP). Unlike traditional and even recent LLM-augmented approaches that ultimately rely on clustering vector embeddings, our framework leverages LLM capabilities directly for the entire pipeline. Experiments demonstrate that this approach provides better utility over existing (embedding-based) methods in the DP setting, especially in the high privacy regime. We evaluate the quality of _CALLIOPE_ outputs using a variety of traditionally studied clustering metrics, as well as some novel utility metrics that we introduce.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Differential Privacy, Clustering, LLM, Topic Hierarchies

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_msxjOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes CALLIOPE for discovering topic hierarchies in text data under differential privacy. An LLM summarizes each document and extracts keywords. The keyword counts are released with private selection, and an LLM proposes sub-topics from the released keywords. Documents are then assigned to the sub-topics, and the process is repeated within each branch. The authors compare against CLIO (non-private) and URANIA (private) on six datasets, using intrinsic and extrinsic clustering metrics and LLM-judge based metrics.

Critical strengths
  1. Using only the keyword selection as the private step is a nice idea, and the privacy guarantee follows easily from this.
  2. The method degrades to a coarser taxonomy as ϵ\epsilon gets smaller, instead of breaking down like DP clustering.
Critical weaknesses
  1. The paper presents avoiding embeddings as the main novelty, but URANIA also does text-based discovery. It uses embeddings for the clustering, but the keyword part is a DP histogram over text that an LLM decodes. Private discovery over discrete vocabularies also exists in DPSU, TrieHH and private LDA, and none of these are discussed. So what remains is using an LLM to build the hierarchy top-down, which is a small step beyond URANIA. This matters because the contribution then rests on the experiments. What would help is clearly stating what new insight this paper brings, that we didn't know beforehand.
  2. Apart from Clio and Urania, no other baselines are evaluated. There are no baselines that use a foundation model directly. For example, the authors say that CALLIOPE approaches zero-shot prompting at small ϵ\epsilon, but they never evaluate prompting an LLM for a taxonomy without any private data. A non-private text-based method like TopicGPT would also be a useful reference. This matters because the document assignment is a non-private LLM call, and the only data-dependent information is the keyword list. Without these baselines I cannot tell how much of the quality comes from the DP signal and how much from the LLM’s prior knowledge.
Questions for authors
  1. I don't mind that the paper is more on the engineering side, but what is the new insight that we gain from it? From what I can tell, the hierarchy is discovered by calling an LLM on a privately selected keyword list. Is there something that we learn beyond the fact that this works, for example about what can be recovered privately, or about how the budget should be spent across levels?

  2. Can the authors discuss how this relates to DPSU, FedLDA and other private text-based discovery methods (e.g. TrieHH), and say how they are different? It is fine if these methods only learn higher-level clusters or just terms. But it would help to include them as baselines or at least compare what they find to the CALLIOPE hierarchy.

  3. Can the authors add baselines that use a foundation model directly? For example, prompting an LLM for a taxonomy of the dataset without any private data, and a non-private text-based method like TopicGPT as a reference. How much of the quality of CALLIOPE at small ϵ\epsilon comes from the private signal, and how much from the LLM's prior knowledge?

  4. Is the ϵ\epsilon in the figures the total budget or the budget per level? What are the contribution bound cc, mkeym_{key} and the split of (ϵ,δ)(\epsilon, \delta) across levels? Was URANIA run with a comparable total budget?

  5. How many human raters and items were used in Table 1? Why are the gains in Table 1 so much smaller than with the LLM judges in the appendix?

  6. One of the key challenges Urania highlights is transfer to user-level DP. This paper also does record-level DP. Can the authors comment how their approach could transfer to user-level DP?

4 comments
AuthorsOpenReview ↗

Response to Reviewer msxj

We thank reviewer sxjR for their review, including their concerns listed in "Critical Weaknesses" and "Questions For Authors." We hope you find the below responses satisfactory:

  1. The paper presents avoiding embeddings as the main novelty, but URANIA also does text-based discovery. It uses embeddings for the clustering, but the keyword part is a DP histogram over text that an LLM decodes. Private discovery over discrete vocabularies also exists in DPSU, TrieHH and private LDA, and none of these are discussed. So what remains is using an LLM to build the hierarchy top-down, which is a small step beyond URANIA. This matters because the contribution then rests on the experiments. What would help is clearly stating what new insight this paper brings, that we didn't know beforehand.
  1. I don't mind that the paper is more on the engineering side, but what is the new insight that we gain from it? From what I can tell, the hierarchy is discovered by calling an LLM on a privately selected keyword list. Is there something that we learn beyond the fact that this works, for example about what can be recovered privately, or about how the budget should be spent across levels?

We view our main contribution as proposing a DP algorithm for creating named clusters that beats the state of the art, which we demonstrate empirically. While the individual techniques that form the building blocks of our algorithm were known, the specific way we combine them is novel and provides improved performance.

  1. Can the authors discuss how this relates to DPSU, FedLDA and other private text-based discovery methods (e.g. TrieHH), and say how they are different? It is fine if these methods only learn higher-level clusters or just terms. But it would help to include them as baselines or at least compare what they find to the CALLIOPE hierarchy.

Thank you for the suggestion. While we had considered these three algorithms, they are solving slightly different problems than CALLIOPE:

  • DPSU is an algorithm for solving the “private partition selection” problem, i.e., finding common elements in the dataset where each user holds a collection of objects. One of the subroutines of CALLIOPE is a simple private partition selection algorithm, so DPSU could replace this subroutine; however, this is only one small piece of CALLIOPE.
  • TrieHH is an algorithm for finding common prefixes in a dataset of strings (each user holds a string). Calliope is aiming to discover semantically similar strings, not exact matches.
  • Finally, private versions of LDA are solving somewhat similar problems: given a collection of strings, we want to discover topics within them. However, there are several important differences. First of all, private LDA algorithms use different neighboring relations: they protect the presence of one term in the dataset vs. the presence of one record / user. In addition, LDA is not a clustering algorithm, in that it assigns multiple topics to each record and does not provide a partition. Finally, LDA operates on the exact text as-is (instead of clustering only one aspect of each data point, like URANIA, CLIO, and CALLIOPE) and uses real terms from the strings.
  1. Apart from Clio and Urania, no other baselines are evaluated. There are no baselines that use a foundation model directly. For example, the authors say that CALLIOPE approaches zero-shot prompting at small , but they never evaluate prompting an LLM for a taxonomy without any private data. A non-private text-based method like TopicGPT would also be a useful reference. This matters because the document assignment is a non-private LLM call, and the only data-dependent information is the keyword list. Without these baselines I cannot tell how much of the quality comes from the DP signal and how much from the LLM’s prior knowledge.
  1. Can the authors add baselines that use a foundation model directly? For example, prompting an LLM for a taxonomy of the dataset without any private data, and a non-private text-based method like TopicGPT as a reference. How much of the quality of CALLIOPE at small comes from the private signal, and how much from the LLM's prior knowledge?

We have now added measurements for TopicGPT in our updated manuscript. We did not include this in the first version because we only compared against approaches that could be modified to preserve privacy; TopicGPT goes through the dataset sequentially, which makes it hard to modify to satisfy differential privacy.

We are running some experiments and hope to add some numbers for versions of Calliope that don’t use keywords.

AuthorsOpenReview ↗

Response to Reviewer msxj (2)

  1. Is the ε\varepsilon in the figures the total budget or the budget per level? What are the contribution bound , and the split of across levels? Was URANIA run with a comparable total budget?

The privacy parameters for Calliope are the total budget. The budget was split evenly between levels, and for each level we split the budget evenly between the two queries: keyword discovery and size estimation. Urania was run with double the budget (we allocated full budget to each part of Urania). This avoids the claim that Urania performed worse because of the budget allocation. As for sensitivity, we ran experiments in the record-level privacy regime, so for each level the keyword sensitivity is 5 and for size sensitivity is 1. If anything, giving Urania 2x the budget should only favor Urania.

  1. How many human raters and items were used in Table 1? Why are the gains in Table 1 so much smaller than with the LLM judges in the appendix?

For human evaluation, 295 raters were recruited via Prolific. The raters were prescreened for English fluency, age over 18, and history on the platform. For each algorithm comparison, 10 independent runs generated 10 questions each, for a total of 100 questions. Since there were 15 different algorithm pair comparisons, this created a bank of 1500 questions. A rater would be given 5 of the 1500 questions to evaluate. We allowed raters to rate a second time, which 5 raters did (the available slots went quickly). For each question, the rater was to select from 1 of 3 choices: A is better, B is better, or they are both equal. The positions of the algorithm output for A and B were randomized.

Note that because a single rater rates a question, we are unable to ascertain uncertainty or agreement between human raters for all of these evaluations. The ratings cost approximately 500 USD and attempts to answer the question of agreement would require at least 3 human ratings per question for an additional 1000 USD. We are certainly interested in the question of confidence or agreement between human raters but decided to focus our research funds on evaluating different datasets and epsilon values. In a previous experiment, we asked 30 questions (on the Emotions Dataset between Calliope and Urania) to 3 separate humans to test agreement. Two out of 30 were unanimous, 23 out of 30 had a 2-out-of-3 majority, while 5 had a complete disagreement (1 for A, 1 for B, 1 Tie). This validates the difference in Table 1 compared to the LLM judges in the appendix, because the same type of noise is not observed in the LLM judges.

Prolific has an ethics protocol that all the raters agree to. No PII was collected from the raters although we received coarsened metadata about them (country, age, employment status, etc.). We provided an informed consent disclosure outlining the task and they had the right to withdraw at any time without penalty. We screened the questions to ensure that no sexual or violent text was present in the question, either in the dataset or hallucinated from the cluster names. Without revealing our institutional requirements, we are exempt from requiring an IRB for these types of benign behavioral research without PII. We have added an ethics statement for transparency on our utilization of human raters.

  1. One of the key challenges Urania highlights is transfer to user-level DP. This paper also does record-level DP. Can the authors comment how their approach could transfer to user-level DP?

The CALLIOPE algorithm can be directly used with user-level DP by using a contribution bound on the number of keywords contributed by a user. We evaluate all algorithms at the record level and we consider evaluating the user-level DP variant as an interesting future direction.

We hope our responses allay the concerns you raised. Please let us know if you have additional issues during the discussion period. Thank you for your time in reviewing our work.

Reviewer_msxjOpenReview ↗

Response to Authors

I thank the authors for the revision. The paper is clearly better organized, and adding TopicGPT and discussing related methods (DPSU, TrieHH, private LDA) both help. Some points from my initial review are still open, and I list them below together with a small number of requests.

Not yet addressed (from my initial review)

  • Zero-shot baseline (my third question): The rebuttal mentions a keyword-free variant, but it is not in the revision. Without it, the reader cannot tell how much of CALLIOPE's quality comes from the private signal and how much from the LLM's prior.
  • Budget allocation across levels (my first question): The budget is split evenly, and footnote 1 notes that this could be optimized. No alternative split is evaluated.

Requested ablations (there is no need to re-run everything, I understand the time constraint and how much effort it takes to run all these experiments. A single ε, e.g. ε = 1, on two or three datasets, evaluated with one or two metrics of the authors' choice, would be enough)

  1. LLM prior vs. private signal (from my initial review): compare a zero-shot taxonomy with private CALLIOPE and non-private CALLIOPE.
  2. Budget split across levels (from my initial review): compare the even split with allocations that put more budget on the top level or on the deepest levels.
  3. Keyword extractor (new): replace the LLM keyword extraction with a classical extractor (e.g. TF-IDF or KeyBERT) to test whether the LLM is needed in the abstraction step.

Presentation and layout

  • Figures 2 and 3 use 3×3 grids for seven datasets, so CLINC-OOS sits alone in the last row and each figure takes about half a page. A 2×4 layout with the legend in the eighth slot, or a merged figure with a few representative datasets in the main text and the full grids in the appendix, would free space for the ablations above.
  • Figure 11 still has "To be added" placeholders for the WildChat judges.

Minor

  • Please state the contribution bound c (and the sensitivities given in the rebuttal) in §4.
  • Please add a short version of the rebuttal's DPSU/TrieHH/LDA discussion to the related-work section.
  • Reproducibility: I appreciate the effort made here: the pseudocode, the verbatim prompts, the datasets, models and hyperparameters, and the reproducibility and ethics statements are all helpful. Since the method chains many LLM calls, I would also appreciate it if the final version included the code (anonymized now, or released upon acceptance).

Overall the paper is in a better state now, and if my remaining concerns and questions are addressed I would be willing to raise my score.

As an addition I would encourage the authors to also think about adding further ablations, which they think would improve the paper. What I listed above is enough for me to change the score, but in general for future readers I think there could be more details about the individual method components which should be tested experimentally.

We thank the reviewer for the thoughtful follow-up and for constructive suggestions that have substantially strengthened the paper. We have updated the manuscript to address all of the remaining points and requested ablations. Below we summarize where each addition appears in the revised paper.

 

  1. LLM Prior vs. Private Signal — Zero-Shot Baseline (Appendix F, Figures 13–20): We added Appendix F ("Impact of Keyword-Driven Synthesis"), which compares Calliope against a data-oblivious variant, Calliope-without-keywords. We evaluated it on the same suite of parameters as comparisons between Urania and Calliope. Calliope outperforms Calliope-without-data on almost all configurations with largest gains on fine-grained intent and open-ended conversational corpora
  2. Privacy Budget Allocation Across Levels and Queries (Appendix H, Table 1, Figures 22–23; Footnote 1 in Section 2.2): We updated Footnote 1 in Section 2.2 to reference Appendix H ("Alternative Splits of the Privacy Budget"), where we evaluate seven alternative splits of the same total privacy budget against the default even split. We observed that none of the seven alternatives consistently improves over the even split. 
  3. Classical Keyword Extractor vs. LLM Keyword Extraction (Appendix G, Figure 21): We added Appendix G ("Impact of the Keyword Extractor"), which replaces the LLM keyword extraction with an LLM-free classical extractor. Evaluated on Banking77 and Emotion, LLM-extracted keywords are statistically significantly preferred on almost all configurations with one tie.

 

We also updated figures style and added accidentally missing comparisons for WildChat for other LLM judges.

Finally, we added a note about contribution bound and query sensitivities to Section 4 ("Experimental Settings", p. 6) and added a discussion of DPSU, TrieHH, and Private LDA to Section 5 ("Related Work", p. 9).

With regards to the reproducibility request: we are working on creating a version that can be shared. During this rebuttal phase, we focused effort on the additional experiments and could not prioritize the engineering effort to create an approachable open source implementation. We will try to include it in the final version.

Reviewer_PipYOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

CALLIOPE is a top-down framework for discovering topic hierarchies without embedding-based clustering. It uses an LLM to summarize documents and extract keywords, synthesizes topic schemas from aggregated keyword frequencies, and assigns document summaries to those topics recursively. Its private variant applies differentially private partition selection to keyword histograms before schema synthesis. The paper provides a document-level privacy composition argument and evaluates the framework against CLIO and URANIA on six datasets using intrinsic clustering metrics, agreement with reference labels, semantic evaluation tasks, multiple LLM judges, and human preferences. The results suggest that CALLIOPE produces more informative cluster descriptions than the private embedding-based baseline, particularly under restrictive privacy budgets, at the cost of additional LLM inference.

Critical strengths
  1. Synthesizing schemas from private keyword histograms makes the privacy boundary explicit and allows subsequent schema generation to use the post-processing property. The recursive design is modular and avoids the additional complexity of private geometric clustering. This is a meaningful systems contribution even though the underlying privacy primitives are established.
  2. Guess-Cluster and the side-by-side comparisons assess whether topic descriptions meaningfully characterize their assigned documents, which conventional clustering metrics do not capture. The use of multiple judge models and human comparisons strengthens the evidence beyond a single-model evaluation. The generally favorable comparisons against URANIA suggest practical value for private corpus analysis.
Critical weaknesses
  1. The paper explicitly notes that, with sufficiently restrictive privacy, CALLIOPE can fall back to generating a taxonomy without meaningful dataset information. However, it does not report a corresponding data-independent taxonomy baseline followed by the same assignment procedure. On familiar domains such as banking, movie reviews, and common assistant intents, a plausible taxonomy could perform well without discovering much from the private corpus. This matters especially for the central claim of strong utility in the high-privacy regime. A zero-keyword baseline, keyword ablations, and measurements of emerging-topic coverage would help establish what private aggregation contributes.
  2. Theorem 2.2 composes privacy budgets across recursion levels, but the experimental section does not clearly specify whether each plotted CALLIOPE ε is the total hierarchy budget or a per-level budget, nor how ε and δ are allocated across depths. Appendix C also notes that keyword percentages require a noisy estimate of the partition size, without specifying its mechanism and budget. URANIA is explicitly reported as consuming 2ε, so identical legend labels already denote different budgets. In addition, the methods vary depth and target cluster count differently, making matched-granularity comparisons important. These reporting gaps limit how confidently the observed gains can be interpreted as a privacy–utility advantage.
Questions for authors
  1. What performance does CALLIOPE achieve when schema synthesis receives no corpus-derived keywords, while retaining the same assignment procedure and comparable branching and depth? How much of the high-privacy advantage remains relative to this data-independent baseline?
  2. For every reported CALLIOPE configuration, is ε the total budget or the budget at each level? Please specify the per-level ε and δ allocation, contribution limit, and any additional privacy cost for estimating the denominators used in keyword percentages. Can the main comparisons be reported against total composed privacy budgets?
  3. How were configurations selected for the side-by-side comparisons and human evaluation? Please clarify the human-rating protocol, sample counts, uncertainty, and agreement, and whether the preference for CALLIOPE over URANIA persists at comparable realized cluster counts. How often does the rejection-sampling procedure in Appendix B.3 exclude candidate comparisons?
3 comments
AuthorsOpenReview ↗

Response to Reviewer PipY

We thank reviewer PipY for their review, for not only the positive rating but also their concerns listed in "Critical Weaknesses" and "Questions For Authors." We hope you find the below responses satisfactory:

  1. The paper explicitly notes that, with sufficiently restrictive privacy, CALLIOPE can fall back to generating a taxonomy without meaningful dataset information. However, it does not report a corresponding data-independent taxonomy baseline followed by the same assignment procedure. On familiar domains such as banking, movie reviews, and common assistant intents, a plausible taxonomy could perform well without discovering much from the private corpus. This matters especially for the central claim of strong utility in the high-privacy regime. A zero-keyword baseline, keyword ablations, and measurements of emerging-topic coverage would help establish what private aggregation contributes.
  1. What performance does CALLIOPE achieve when schema synthesis receives no corpus-derived keywords, while retaining the same assignment procedure and comparable branching and depth? How much of the high-privacy advantage remains relative to this data-independent baseline?

We agree with this point. We are running the experiments, and we will try to finish them by the end of the rebuttal period.

  1. Theorem 2.2 composes privacy budgets across recursion levels, but the experimental section does not clearly specify whether each plotted CALLIOPE ε is the total hierarchy budget or a per-level budget, nor how ε and δ are allocated across depths. Appendix C also notes that keyword percentages require a noisy estimate of the partition size, without specifying its mechanism and budget. URANIA is explicitly reported as consuming 2ε, so identical legend labels already denote different budgets. In addition, the methods vary depth and target cluster count differently, making matched-granularity comparisons important. These reporting gaps limit how confidently the observed gains can be interpreted as a privacy–utility advantage.
  1. For every reported CALLIOPE configuration, is ε the total budget or the budget at each level? Please specify the per-level ε and δ allocation, contribution limit, and any additional privacy cost for estimating the denominators used in keyword percentages. Can the main comparisons be reported against total composed privacy budgets?

All epsilons for CALLIOPE are for the total budget, which we split evenly between levels; at each level, half of the budget is spent estimating node size. As for Urania, indeed, we used separate budgets of epsilon for both clustering and naming (so the total budget consumed by Urania is 2ε versus ε used by CALLIOPE). If anything, this gives Urania the upper hand in comparison to CALLIOPE since Urania is allowed double the budget.

AuthorsOpenReview ↗

Response to Reviewer PipY (2)

  1. How were configurations selected for the side-by-side comparisons and human evaluation? Please clarify the human-rating protocol, sample counts, uncertainty, and agreement, and whether the preference for CALLIOPE over URANIA persists at comparable realized cluster counts. How often does the rejection-sampling procedure in Appendix B.3 exclude candidate comparisons?

For human evaluation, 295 raters were recruited via Prolific. The raters were prescreened for English fluency, age over 18, and history on the platform. For each algorithm comparison, 10 independent runs generated 10 questions each, for a total of 100 questions. Since there were 15 different algorithm pair comparisons, this created a bank of 1500 questions. A rater would be given 5 of the 1500 questions to evaluate. We allowed raters to rate a second time, which 5 raters did (the available slots went quickly). For each question, the rater was to select from 1 of 3 choices: A is better, B is better, or they are both equal. The positions of the algorithm output for A and B were randomized.

Controlling for comparable realized cluster counts is difficult: Calliope takes in a depth and Urania takes in the number of clusters. Even over our hyperparameter selection, matching realized cluster counts is difficult. Instead, we picked the best hyperparameter that optimizes the “Guess Odd One Out” score and used that for comparisons.

Note that because a single rater rates a question, we are unable to ascertain uncertainty or agreement between human raters for all of these evaluations. The ratings cost approximately 500 USD and attempts to answer the question of agreement would require at least 3 human ratings per question for an additional 1000 USD. We are certainly interested in the question of confidence or agreement between human raters but decided to focus our research funds on evaluating different datasets and epsilon values. In a previous experiment, we asked 30 questions (on the Emotions Dataset between Calliope and Urania) to 3 separate humans to test agreement. Two out of 30 were unanimous, 23 out of 30 had a 2-out-of-3 majority, while 5 had a complete disagreement (1 for A, 1 for B, 1 Tie). This validates the difference in Table 1 compared to the LLM judges in the appendix, because the same type of noise is not observed in the LLM judges.

The rejection sampling procedure in B.3 occurs infrequently (<0.5%) in instances where non-trivial clustering occurs. It only meaningfully occurs when two algorithms have very coarse clusterings on a small dataset, so that the assignment step might make two identical clusters in both algorithms. When the hyperparameters involve large epsilon, large depth, or a large number of clusters, most often no rejection occurs.

Prolific has an ethics protocol that all the raters agree to. No PII was collected from the raters although we received coarsened metadata about them (country, age, employment status, etc.). We provided an informed consent disclosure outlining the task and they had the right to withdraw at any time without penalty. We screened the questions to ensure that no sexual or violent text was present in the question, either in the dataset or hallucinated from the cluster names. Without revealing our institutional requirements, we are exempt from requiring an IRB for these types of benign behavioral research without PII. We have added an ethics statement for transparency on our utilization of human raters.

We hope our responses allay the concerns you raised. Please let us know if you have additional issues during the discussion period. Thank you for your time in reviewing our work.

As a follow up, we added a comparison of Calliope against a data-oblivious variant, Calliope-without-keywords (Appendix F, "Impact of Keyword-Driven Synthesis"). Calliope outperforms Calliope-without-data on almost all configurations with largest gains on fine-grained intent and open-ended conversational corpora. This strongly suggests that calliope adds utility beyond the llm's prior ability to guess categories.

Reviewer_sxjROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes CALLIOPE, a framework leveraging LLMs for building topic hierarchies over sensitive text corpora without document embedding or geometric clustering. CALLIOPE recursively applies three LLM primitives: Abstract, Synthesis, and Assignment. To protect data privacy, CALLIOPE introduces privacy enhancements in the synthesis step, specifically applying DP partition selection to the keyword frequency histogram. The experiments compare CALLIOPE with CLIO and URANIA on different datasets.

Critical strengths
  1. The paper topic is important and timely. Privacy analysis of conversational corpora is important. The CALLIOPE framework is straightforward to understand, with three basic LLM-based primitives.
  2. An interesting design introduces privacy enhancements in the synthesis step by applying DP keyword partition selection to the keyword frequency histogram, and then conducts non-private LLM classification in the assignment stage.
  3. All prompts (App. C–D) and cost comparisons (Table 26, Sec. 6) are given. This provides transparency of the proposed method.
Critical weaknesses
  1. Novelty and contributions are limited. URANIA already combines DP keyword histograms and LLM keyword extraction to generate cluster description. In URANIA, DP keyword extraction is used at the very end to name clusters. CALLIOPE proposes a new design to use DP keyword partition selection upstream to drive top-down topic synthesis, and then uses non-private LLM classification to partition the documents into sub-trees. This is useful, but the novelty is limited.
  2. Configuration selections in experiments are different, which may not be apple-to-apple comparisons. For example, Table 1 uses d = 3/k = 80 for Banking77 vs. d = 2/k = 15 for CLINC; Tables 2–5 use "depth 3 vs. k = 50" at ε = 10 but "depth 2 vs. k = 100" at ε = 1. Need to explain how configurations were chosen.
  3. Because Assign must prompt an LLM for every single document at every depth level, CALLIOPE consumes many more tokens than URANIA (2.3x ~ 4.7x).
Questions for authors
  1. Can you evaluate on a real AI-usage conversation corpus, or another corpus whose taxonomy the LLM is unlikely to have memorised?
  2. Can the authors provide more details about human raters? For example, number of raters, protocol, number of judgements per cell, agreement, and ethics review.
6 comments
AuthorsOpenReview ↗

Response to Reviewer sxjR

We thank reviewer sxjR for their review, including their concerns listed in "Critical Weaknesses" and "Questions For Authors." We hope you find the below responses satisfactory:

  1. Novelty and contributions are limited. URANIA already combines DP keyword histograms and LLM keyword extraction to generate cluster description. In URANIA, DP keyword extraction is used at the very end to name clusters. CALLIOPE proposes a new design to use DP keyword partition selection upstream to drive top-down topic synthesis, and then uses non-private LLM classification to partition the documents into sub-trees. This is useful, but the novelty is limited.

Urania and Calliope are different in their approach to taxonomy generation, in that Urania creates the clusters first, and then uses keywords to name and describe them, whereas Calliope uses keywords to create clusters. So we believe that the two approaches are fundamentally different.

  1. Configuration selections in experiments are different, which may not be apple-to-apple comparisons. For example, Table 1 uses d = 3/k = 80 for Banking77 vs. d = 2/k = 15 for CLINC; Tables 2–5 use "depth 3 vs. k = 50" at ε = 10 but "depth 2 vs. k = 100" at ε = 1. Need to explain how configurations were chosen.

For each (dataset, algorithm, privacy parameters) triple, we ran the algorithm over a small sweep of the defining hyperparameter of each algorithm. Then we used “Guess Odd One Out” as the metric to pick the best-performing hyperparameter. For Urania and Clio, we used cluster sizes of {500, 1000, 2000} to infer the hyperparameter number of clusters kk. For Calliope, we varied the depth dd, which was 1 to 4 on Massive, 1 to 3 on IMDB/Emotion, and 1 to 2 on Reuters. As an example, for running Urania on CLINC-OOS, the target cluster sizes of {500, 1000, 2000} created 3 values for kk of {7, 15, 30}. For ε=0.1,k=7\varepsilon=0.1, k=7 had the best “Guess Odd One Out” score, hence its presence in the CALLIOPE vs. URANIA side-by-side comparison of Table 1 for CLINC-OOS at ε\varepsilon=0.1.

We described this in Experimental Settings, although we did not explicitly call out using “Guess Odd One Out” as the selecting criterion. We will add this clarifying explanation to the text.

  1. Because Assign must prompt an LLM for every single document at every depth level, CALLIOPE consumes many more tokens than URANIA (2.3x ~ 4.7x).

In our experiments, we use an LLM for the Assign step. But note that the Assign step is a much simpler task than extracting summaries or keywords. While we did not do this yet in our experiments, we expect that the Assign step could be made significantly more efficient using recently popular classification (a.k.a. decision) models such as Jev, Open AI Decisions API, Gemini Decision API, etc. We believe this can be done without much drop in quality.

AuthorsOpenReview ↗

Response to Reviewer sxjR (2)

  1. Can you evaluate on a real AI-usage conversation corpus, or another corpus whose taxonomy the LLM is unlikely to have memorised?

We added measurements on WildChat, a real AI-usage conversation dataset, to the paper.

  1. Can the authors provide more details about human raters? For example, number of raters, protocol, number of judgements per cell, agreement, and ethics review.

For human evaluation, 295 raters were recruited via Prolific. The raters were prescreened for English fluency, age over 18, and history on the platform. For each algorithm comparison, 10 independent runs generated 10 questions each, for a total of 100 questions. Since there were 15 different algorithm pair comparisons, this created a bank of 1500 questions. A rater would be given 5 of the 1500 questions to evaluate. We allowed raters to rate a second time, which 5 raters did (the available slots went quickly). For each question, the rater was to select from 1 of 3 choices: A is better, B is better, or they are both equal. The positions of the algorithm output for A and B were randomized.

Note that because a single rater rates a question, we are unable to ascertain uncertainty or agreement between human raters for all of these evaluations. The ratings cost approximately 500 USD and attempts to answer the question of agreement would require at least 3 human ratings per question for an additional 1000 USD. We are certainly interested in the question of confidence or agreement between human raters but decided to focus our research funds on evaluating different datasets and epsilon values. In a previous experiment, we asked 30 questions (on the Emotions Dataset between Calliope and Urania) to 3 separate humans to test agreement. Two out of 30 were unanimous, 23 out of 30 had a 2-out-of-3 majority, while 5 had a complete disagreement (1 for A, 1 for B, 1 Tie). This validates the difference in Table 1 compared to the LLM judges in the appendix, because the same type of noise is not observed in the LLM judges.

Prolific has an ethics protocol that all the raters agree to. No PII was collected from the raters, although we received coarsened metadata about them (country, age, employment status, etc.). We provided an informed consent disclosure outlining the task, and they had the right to withdraw at any time without penalty. We screened the questions to ensure that no sexual or violent text was present in the question, either in the dataset or hallucinated from the cluster names. Without revealing our institutional requirements, we are exempt from requiring an IRB for these types of benign behavioral research without PII. We have added an ethics statement for transparency on our utilization of human raters.

We hope our responses allay the concerns you raised. Please let us know if you have additional issues during the discussion period. Thank you for your time in reviewing our work.

As a follow up, we added a comparison of Calliope against a data-oblivious variant, Calliope-without-keywords (Appendix F, "Impact of Keyword-Driven Synthesis"). Calliope outperforms Calliope-without-data on almost all configurations with largest gains on fine-grained intent and open-ended conversational corpora. This strongly suggests that calliope adds utility beyond the llm's prior ability to guess categories.

Comments on the paper

1 comment
Area_Chair_Mx5AOpenReview ↗

Dear Reviewers

Since the author-reviewer discussion ends soon, I encourage the reviewer to review the authors' rebuttals and let them know whether their responses are satisfactory and whether any concerns remain, if you haven't done so.

Thank you for your time and effort in reviewing for ICLR 2027.

Best wishes,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.