acceptodds
Under review as a conference paper at ICLR 2027

RAQUEL: Robust Evaluation of Machine Unlearning Through Queries over Aligned Databases

Abstract

To address privacy, ethical, and reliability concerns, machine unlearning has emerged as an essential technique for selectively removing information from Large Language Models. However, existing evaluation benchmarks (e.g., TOFU, WMDP, and MUSE) rely on a narrow range of query forms and often fail to test unlearning robustness to semantic paraphrases, complex compositional, and aggregation questions. To tackle this challenge, we introduce RAQUEL, a framework that synthesizes diverse paraphrased, compositional, and aggregation queries by projecting forget/retain knowledge into aligned relational databases. RAQUEL systematically generates diverse database queries and executes them on the aligned databases to determine whether the query should be answered with the forget or retain knowledge. We construct 9,450 QA pairs from TOFU, WMDP, and MUSE to evaluate five unlearning methods on Llama-3.1-8B and Qwen3-8B-Base. Our analysis shows that existing methods often struggle to answer complex questions involving retained knowledge, a degradation that is largely obscured by the original retain questions in existing benchmarks. We further find that the relative ranking of unlearning methods changes when evaluated on RAQUEL-generated datasets, suggesting that evaluation results are sensitive to query diversity and complexity. In addition, training with RAQUEL-generated data improves the robustness of learned knowledge to novel query forms, while subsequent unlearning again changes the relative ordering of methods. We release our code, checkpoints, and datasets.

Primary area:
datasets and benchmarks
Keywords:
Machine unlearning

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_8MWAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RAQUEL, a framework that constructs diverse questions through relational databases to evaluate machine unlearning in large language models. The method first converts knowledge from the original forget and retain sets into a relational database with provenance records, then constructs a second database by deleting facts supported exclusively by the forget set. SQL queries are generated and executed on both databases. Questions are labeled “affected” or “unaffected” depending on whether the results change, and are subsequently converted into natural-language question–answer pairs.

Using TOFU, WMDP, and MUSE, the authors construct 9,450 answer-balanced evaluation examples covering factual, join, filtering, Boolean, and aggregation questions. Experiments across two 8B models, five unlearning methods, and three training seeds show that high accuracy on native retain questions can conceal capability degradation on new question forms, and that different evaluation questions can change the relative ranking of methods.

The authors also train models on a subset of RAQUEL questions and observe improved performance on held-out questions. However, subsequent unlearning can still impair retention on new questions. The main contributions are an evaluation construction pipeline grounded in provenance tracking and query execution, and an empirical analysis of limitations in existing unlearning evaluations.

Critical strengths
  1. Provenance tracking and execution on two databases provide an inspectable basis for evaluation construction.

RAQUEL records whether facts originate from the forget or retain set and protects facts also supported by retained sources. Comparing query results before and after deletion allows the authors to trace why a question is labeled affected.

This design is more transparent than relying solely on an LLM to determine whether a question belongs to the forget or retain set, and facilitates queries combining multiple facts. The appendix’s examination of redundantly stored facts and their effects on labels further demonstrates attention to construction consistency.

  1. The experiments provide meaningful new evidence of damage to retained capabilities.

The paper compares performance before and after unlearning and also uses a model trained only on the retain set as a reference. For example, in the WMDP–Llama setting, the retain-reference model achieves 26.6% accuracy on unaffected questions, whereas IDK+GD achieves 13.1%, despite retaining 97.5% accuracy on native retain questions.

The refusal analysis also shows that some methods rarely refuse native retain questions but frequently refuse new unaffected questions. This supports the need to broaden retention evaluation and provides practical value in distinguishing the side effects of unlearning strategies.

Critical weaknesses
  1. Unresolved scope and closed-world assumptions create a gap between database answers and the semantics of natural-language questions.

SQL execution verifies results within a given database, but does not automatically establish that the database fully represents the real-world knowledge asked about. RAQUEL’s generation prompts explicitly require questions not to mention databases, records, or sources, so some natural-language questions have a broader scope than the database actually covers.

For example, Table 11 assigns an answer of 3 to the question asking how many distinct people are associated with Ofcom. This number reflects the extracted database contents, but the question itself is not restricted to those news articles or source records. Similarly, the absence of a relationship in the database does not establish that the relationship does not exist in the natural-language sense. The vague phrase “associated with” may also encompass relationships not encoded in the database.

Moreover, deleting a piece of knowledge does not make the underlying fact false, nor does it necessarily imply that an appropriate answer should change from Yes to No. Thus, affected/unaffected labels clearly characterize database changes, but their validity as unlearning evaluation labels requires further justification. The existing small-scale manual inspection is insufficient to resolve these systematic issues.

  1. The scoring of affected questions and limited pre-unlearning performance constrain interpretations of unlearning quality and method rankings.

The paper treats a lower match rate against original answers on affected questions as better, but this counts successful forgetting, excessive refusal, reasoning failures, and general capability degradation alike as “improvements.” This also does not fully align with the objective stated in Section 3.1: making the unlearned model approximate the retain-only retrained model.

For example, in the TOFU–Llama setting in Table 4, the retain-reference model has a 27.5% match rate on affected questions, while several unlearning methods achieve only 5.1%–10.3%. These methods fall substantially below the reference model, which alone does not establish better unlearning.

Furthermore, in some main settings, the original and retain-reference models perform almost identically on affected questions. For WMDP–Llama, their scores are 27.8% and 27.7%, respectively, indicating that this aggregate metric barely distinguishes whether the model was trained on the forget set. The authors acknowledge this issue and attempt to mitigate it through additional training, which is constructive. Nevertheless, a more direct evaluation is needed to determine whether previously answerable target knowledge has been removed, while separating this effect from capability degradation.

Questions for authors

The following questions concern the validity of the current conclusions, and the answers may change my assessment.

  1. Question scope and answer validity: How do you ensure that answers to aggregation, negation, and relationship questions do not depend on an unstated database closed-world assumption? Could you separately audit whether the database answer is correct and whether the original sources sufficiently support the natural-language answer, and report results with explicitly restricted source scope?
  2. Unlearning metrics: Why should affected accuracy be minimized without qualification, rather than evaluated by proximity to the retain-only retrained model? Could you report results on the subset answered correctly before unlearning, distinguishing refusals, incorrect answers, partially retained answers, and behavior consistent with the retraining reference?
  3. Held-out set independence: When splitting the data into two halves by record in Section 4, are instances sharing the same underlying SQL query, template, or paraphrase family kept on the same side? If not, how do you distinguish generalization to new wording from generalization to new fact combinations or query structures?
  4. Ranking and scoring robustness: Do method rankings still change when the degree of forgetting is matched? Could you report forgetting–retention trade-off curves and specify the significance-testing procedure? For the many Boolean and counting questions, do deterministic scoring and a manually calibrated LLM judge yield consistent conclusions?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 8MWA

We thank the reviewer for the careful reading and for questions that helped us state the scope of RAQUEL's labels more precisely. The revised paper marks all changes in blue, and the section and table numbers below refer to it.

Answer to W1

We agree that RAQUEL's answers are defined within the source corpus, and we thank the reviewer for pointing out that the paper did not state this scope explicitly. It is now stated in Sections 2.4 and 6. This scope matches our setting, since every model is fine-tuned on the source QA pairs (Section 3.1) and RAQUEL asks about the facts in that corpus. For TOFU the scope is exact, because its authors are fictitious.

The record provenance shows how much depends on absence (Appendix D.6, Table 21). In 77.8% of the released records, every row of the answer is traced to source facts. The remaining 22.2% are absence answers (No, zero, or an empty result), which hold only within the corpus, as do count totals such as the Ofcom example. Our conclusions do not rest on them. Lookup questions, whose answers are values stated in the sources, show the same findings. M_orig answers only 11.8 to 21.6% of unaffected lookup questions, and the losses of retained knowledge persist on lookup questions alone in 12 of 20 settings (Section 3.3, Table 6). In addition, every model is compared with M_ret on the same questions, so a scope mismatch affects both equally. On the phrase "associated with", Appendix B notes that most junction tables record only that two entities are associated, which limits what the questions can ask but not whether their labels are correct.

We also agree that deleting a fact does not make it false. For this reason the main score on affected questions is alignment with the original answer, read against M_ret (Section 3.1), and not agreement with the post-deletion answer. The affected label states only that the answer depends on a forget-set fact.

Answer to W2

We agree that affected alignment should be judged by its proximity to the retain model M_ret, not minimized without qualification, and we thank the reviewer for this point. The revised Section 3.1 now states this reading explicitly, and Tables 4 and 5 no longer bold the lowest value in the forget-side columns. Table 4 reports M_ret as the reference, and Section 3.3 notes that 29 of 30 endpoints end at or below M_ret and that M_orig exceeds M_ret by only 2.3 to 12.5 points on affected lookup questions, which bounds what this measure can reveal. Under the proximity criterion, the TOFU results the reviewer cites (5.1 to 16.8% against 27.5% for M_ret) indicate over-forgetting or capability loss, not better unlearning.

Two analyses give the more direct evaluation the reviewer asks for. First, after training with RAQUEL questions, M_orig exceeds M_ret on held-out affected questions by 8.3 to 15.0 points (Section 4), so the measure separates the two models. There, most methods fall far below M_ret, for example 0.4 to 23.9% against 46.6% on WMDP, and only GA+GD on TOFU approaches it (50.4% against 53.0%). Second, RAQUEL provides a post-deletion answer for every affected question by executing its query on the nullified database, which we use as an additional diagnostic (Table 18, Appendix D.3). Among the answers that no longer match the original, only SAUL gives the post-deletion answer often (29.3%), while the other methods stay below 10%, so most of their changed answers are errors or refusals.

AuthorsOpenReview ↗

Response to Reviewer 8MWA

Answer to Question 1

The two checks the reviewer asks for are separate in RAQUEL. Whether the database answer is correct is settled by execution, which is deterministic (Section 2.4). Whether the sources support the natural-language answer was checked by hand for 100 sampled records (Section 3.2). For each record, we read every source QA pair behind the answer and required that a value equals the source value, that a count equals the number of qualifying source statements, and that a Yes or No matches what the sources establish. 99 of the 100 records passed (all 50 affected and 49 of 50 unaffected).

We also report results with restricted scope (Table 21, Appendix D.6). On the 77.8% of records whose answers are fully traced to source facts, M_orig (Llama-3.1-8B, Table 4 setting) answers 24.0%, 21.1%, and 7.3% of unaffected questions on TOFU, WMDP, and MUSE. The finding that fine-tuned models rarely answer new questions therefore holds without absence-based answers. We thank the reviewer for this suggestion and will enlarge the human evaluation.

Answer to Question 2

Affected alignment should be evaluated by proximity to M_ret, as explained in our answer to W2 and now stated in Section 3.1. The paper and the new analyses already separate several of the cases the reviewer lists. Refusals are reported in Table 8 and Section 3.3, where IDK+GD refuses about half of the affected questions on TOFU. Behavior consistent with the retraining reference is measured by the distance to M_ret and by the post-deletion answer (Table 18). We thank the reviewer for the suggestions and will add this breakdown, with refusals, incorrect answers, and answers matching the post-deletion reference.

Answer to Question 3

The two halves share no forgotten fact and no question, and unaffected questions share no entity, but query templates are deliberately shared (Appendix A.2). Affected questions are assigned by the forgotten facts their answers depend on, and unaffected questions that name the same entity stay on the same side. A leakage check confirms that the halves share no forgotten fact, no entity among unaffected questions, and no question text. Templates are balanced across the halves, because grouping by template would place whole question types on one side.

The held-out half therefore measures generalization to new facts and entities under familiar question forms, not to new query structures, as Appendix A.2 states. Two controls separate knowledge from familiarity with the form. M_ret is trained on the same unaffected train half, so it knows the question forms equally well, and M_orig still exceeds it on held-out affected questions by 8.3 to 15.0 points (Section 4). Table 9 further reports the results by question family. We thank the reviewer for this question.

Answer to Question 4

Yes, the rankings on RAQUEL still differ when the methods are compared under the same forgetting criterion, and the revised paper now reports a forgetting and retention trade-off figure (Appendix D.7, Figure 3). For every method we select the checkpoint with the highest retain ROUGE-L among those whose forget ROUGE-L is at or below that of M_ret (Section 3.1), so all methods in Table 4 meet the same criterion. Figure 3 places each method in the plane of affected and unaffected RAQUEL alignment, at its selected checkpoint and after five epochs, with M_orig and M_ret as references. On TOFU, the methods lower affected alignment without lowering unaffected alignment. On WMDP and MUSE, both fall together, and unlearning beyond the selected checkpoint lowers unaffected alignment further for GA+GD and NPO+GD (from 23.0% to 19.4% and from 21.5% to 15.5% on WMDP). We thank the reviewer for suggesting this analysis.

For significance, we use a two-sided two-proportion z-test at the 5% level against M_ret over the questions of each column, which Section 3.1 now states. For Boolean and count questions, deterministic scoring agrees with the judge (Appendix D.6). On the answers of M_orig and M_ret (Llama-3.1-8B) that state an explicit Yes, No, or number, exact matching agrees with the judge in 96.9% of 28,134 cases (97.7% for Boolean and 96.2% for count questions). The judge is still needed for the remaining answers, which are phrased in free text. It follows a fixed rubric and never sees the model identity (Appendix E). Appendix A.1 additionally reports the answer distribution of the aggregation questions.

AuthorsOpenReview ↗

Gentle Reminder Before Tomorrow's Deadline (Nov 18)

Dear Reviewer 8MWA,

With the discussion period closing tomorrow (November 18), we hope you might have a brief moment to look over our response above. Please let us know if any further clarification would be helpful before the deadline.

Thank you for your time and help in improving our work!

Reviewer_XnwBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces RAQUEL, a framework that synthesizes diverse paraphrased, compositional, and aggregation queries using diverse SQL query forms over aligned relational database.

Critical strengths

The paper proposed a traceable way to probe the target LLMs for forget/retain queries using relational database. The finding that the model trained on the original QA answers just 12-22% of the new questions on the same facts, is interesting.

Critical weaknesses

Major Weakness

  1. Need for relational database for unlearning is not properly motivated.
  2. Ablation against simpler baselines such as LLM-generated paraphrases or multi-hop questions without a DB is missing.
  3. Comparison against other types such as KG-based or multi-hop based probes is missing.
  4. The benchmark considers the stated fact only in the 3 benchmarks. Inferred relations or attributes beyond what is given in the benchmark is missing as these inferred knowledge can indirectly reduce the unlearning ability of these models.

Minor Weakness:

  1. Since the RAQUEL showed TOFU, MUSE undercounted in retain set, a similar experiment on an evaluation set, that is independent derived from these benchmarks in Table 1 with different variations of the same questions (say via KG probe or Multi-hop probe) should be evaluated on the RAQUEL trained unlearning method, to be fair. Does the ranking changes when we do the same on the RAQUEL like we did on the three benchmarks.
  2. The seed data taken from these 3 benchmarks how does it make it unbiased in evaluating against the model?
  3. It would be great to show a running example in the main paper like “Harry Potter” or “Zo Hassani” to understand how the C_\pi is collected for removing an entity or a relation and the C_\pi is converted into a new QA pair.
  4. Since Qwen3.8-27B is used for RAQUEL, there could be a bias in the evaluated Qwen3-8B-Base.
Questions for authors
  1. What is the benefits of using relational database over other structures say knowledge graphs? Why is C_\pi is better created with database operations?
  2. What is the bottleneck/requirements with all these storage? Can we improve the storage efficiency compared to the unlearning improvement gained.
  3. Does the database reflect the internal knowledge state of the target LLM? Are there any inferred relations or attributes beyond what is given in the benchmark or C_\pi, even though the model may have inferred from the written facts?
  4. How is the train/eval test split for the Section 4? Are there any effort to make sure there is a strong overlap between entities, relations or an attribute so that they can be measured?
  5. It will be useful to categorize the performance based on the question types such as yes/no, counts, etc to see the effect as different question types.
  6. Since reranking is one of key focus in the RAQUEL compared to others, is RAQUEL stable under different seed (assuming 3 runs as stated in the paper), hyperparameter changes, and early-stopping choices? Are the bold numbers in Table 4 statistically-significant?
  7. What happens when you add new QA pairs? How difficult it is to update the entire pipeline to affected/unaffected sets?
4 comments
AuthorsOpenReview ↗

Response to Reviewer XnwB

We thank the reviewer for the thorough review and the many concrete questions, which we answer one by one below. The revised paper marks all changes in blue, and the section and table numbers below refer to it.

Answer to Major W1

The relational database is needed for two purposes that unstructured QA cannot serve, as described in Sections 1 and 2. First, it makes forgetting an explicit operation. Each fact records the forget or retain QA pairs that support it (Section 2.1), and nullification deletes the facts supported only by the forget set (Section 2.2). Second, it makes the knowledge executable. Running one SQL query on both databases yields the answer and the affected or unaffected label without manual or LLM labeling (Section 2.4). SQL also expresses joins, filters, counts, and Boolean checks, which gives the compositional and aggregation questions that existing benchmarks lack (Table 1). We thank the reviewer for this comment and now state the motivation explicitly in the introduction.

Answer to Major W2

Both baselines are now available, and RAQUEL compares favorably with each. The paraphrase baseline is already in Tables 4 and 5, where F_p and R_p are LLM-generated paraphrases of the native questions (Section 3.1). They are far easier than RAQUEL questions. M_orig answers 65.2 to 87.7% of paraphrased retain questions but only 16.0 to 31.1% of unaffected RAQUEL questions (Section 3.3), and on MUSE paraphrases rank IDK+GD first while RAQUEL ranks it fourth or fifth.

For multi-hop questions without a database, we thank the reviewer for the suggestion and ran a pilot study on MUSE, in which the same LLM writes compositional and aggregation questions directly from the source QA pairs, together with their answers and labels (Table 19, Appendix D.4). Its labels were correct in 94.9% of cases, against 99.0% for RAQUEL in our human evaluation (Section 3.2). Only 61.5% of its questions were semantically distinct, and they covered 32 source QA pairs, against 61 for RAQUEL. Without a database, nothing verifies the labels or controls which facts are covered, which makes the generation hard to scale.

Answer to Major W3

RAQUEL covers what KG-based and multi-hop probes test. A relational database subsumes a knowledge graph, since entity tables are nodes and junction tables are edges (Section 2.1, Appendix B), and a join is a multi-hop question, as in the Join example of Table 3. SQL additionally provides counts, filters, and Boolean checks.

Existing multi-hop probes such as MQuAKE and RippleEdits rely on an existing knowledge graph like Wikidata. Unlearning benchmarks, however, are based on unstructured text. TOFU consists of QA pairs about fictitious authors, and MUSE is raw news text. RAQUEL builds the structure from the benchmark itself, which is what enables such probes on these datasets. We thank the reviewer for this point and now make the relation explicit in Section 5.

Answer to Major W4

RAQUEL's compositional questions do cover relations that the benchmark never states directly. In the Join example of Table 3, no TOFU QA pair connects the book "Cheness" to "Westerns literature". One pair states (Cheness, Golden Spur Award) and another states (Golden Spur Award, Westerns literature). RAQUEL derives the relation by a join and asks about it.

RAQUEL is limited to knowledge derivable from the stated facts by joins and aggregation. Inferences that need outside knowledge are excluded, since no source QA pair would support them and their label could not be verified. In fact, this aligns with the central goal of evaluating unlearning, which is to test the knowledge from the source data.

AuthorsOpenReview ↗

Response to Reviewer XnwB (cont'd)

Answer to Minor W1

Section 4 is designed as this test, and the ranking changes again. We split the RAQUEL questions into two halves that share no forgotten fact (affected) and no entity (unaffected), train on one, and evaluate on the other, so the test questions are new to the RAQUEL-trained models (Appendix A.2). In Table 5, on TOFU and WMDP, GA+GD is last and fourth on native retain questions but keeps the most held-out unaffected alignment, while the native-retain leaders (IDK+GD and NPO+GD) drop to fourth and third.

Answer to Minor W2

Deriving RAQUEL from the same benchmark data is intentional and does not favor any model. Since RAQUEL and native questions ask about the same facts, a difference in results comes from the question form, not from different knowledge. Three choices keep the evaluation fair. First, no evaluated model takes part in construction, and labels and answers come from SQL execution (Section 2.4). Second, no model in Section 3 is trained on RAQUEL questions. Third, every unlearned model is compared with the retain model M_ret on the same questions (Section 3.1), so any property of the question set affects both equally.

Answer to Minor W3

The paper traces such examples in Table 3 and in Appendix C, and we thank the reviewer for the suggestion. Table 3 follows two records from their source QA pairs through the database rows and the SQL query to the generated question and answer. Table 13 does the same for one record of every query family, including affected ones. For example, its Boolean record is traced to forget pair #3608, which states that Hsiao Yun-Hwa received the Leadership Literature Luminary award. Nullification deletes this fact, a row of the award_person junction table, and records the deletion in C_π. A query that reads this row therefore returns Yes on the aligned database and No on the nullified one, so the question "Is there any link between Leadership Literature Luminary and Hsiao Yun-Hwa?" is labeled affected. Table 14 pairs further affected and unaffected records from all three benchmarks.

Answer to Minor W4

This risk is limited for three reasons. First, Qwen3.8-27B does not decide labels or answers, which come from SQL execution. It only proposes schemas and templates, extracts facts, and words questions (Section 2). Second, the judge sees only the question, the reference answer, and the model's answer, never the model's identity (Appendix E). Third, the findings do not depend on the Qwen model. The same patterns hold on Llama-3.1-8B in every setting of Table 4, and Section 4 uses Llama-3.1-8B only.

AuthorsOpenReview ↗

Response to Reviewer XnwB (cont'd)

Answer to Question 1

The relational database offers three benefits over a knowledge graph. First, SQL covers joins, filters, counts, and Boolean checks in one language (the query families of Table 2), whereas triple-based probes mainly cover path queries. Second, tables hold typed attributes such as dates and integers, and relations with qualifiers (Table 12). Third, execution is deterministic, which gives exact answers and labels.

C_π is easier to create with database operations because nullification becomes a set of standard operations (deleting a relation row, emptying a cell, deleting an entity row) that are logged one by one (Section 2.2). Each logged change then seeds a query that must be affected (Section 2.3). A graph store with a comparable query language could implement the same idea, and SQL is the most mature choice.

Answer to Question 2

Storage is not a bottleneck. The databases are small, holding 7,891 (TOFU), 2,746 (WMDP), and 1,159 (MUSE) facts (Table 2) in at most 89 tables (Table 12). The nullified database is a clone that differs only by the 415 to 1,193 nullified facts, so it could be stored as C_π alone. The databases are needed only once, to build the dataset, and evaluating a model requires only the released QA files. The actual cost is the one-time LLM inference during construction.

Answer to Question 3

No, the database represents the knowledge stated in the benchmark's forget and retain data, not the internal state of a model (Section 2.1). It is built once and reused for all models. The gap between the two is one of our findings. M_orig is trained on all source facts but answers only 16.0 to 31.1% of unaffected RAQUEL questions (Section 3.3).

The database contains only facts stated in the source QA pairs. Relations that follow from combining them are covered by join and aggregation queries (see Major W4), and other inferences are not included, as Section 6 now states.

Answer to Question 4

Each suite's affected and unaffected questions are split in half, and the halves share no specific fact (Appendix A.2). Forgotten facts are split evenly, and an affected question joins the half containing all the facts its answer depends on (questions spanning both halves are dropped). Unaffected questions that mention the same entity go to the same half.

The overlap the reviewer asks about is kept at two levels. Both halves share the same schema (the same relations and attributes) with balanced question types, and the facts asked in the test half remain in the training data through the source QA pairs. The test half thus measures whether a model answers new questions about facts it learned. We thank the reviewer for this question and have added these details to the paper.

Answer to Question 5

We thank the reviewer for this suggestion. This breakdown is reported in Appendix A and now summarized in Section 3.3. Table 6 reports the results of Table 4 by question family (lookup, Yes/No, and count), Table 7 averages five question families over the three benchmarks, Table 8 gives refusal rates, and Table 9 breaks down the RAQUEL-trained models of Section 4. For example, before unlearning M_orig answers on average 36.6% of unaffected single-fact lookup questions, 14.0% of join questions, and 6.5% of filter questions (Table 7). The losses of retained knowledge also persist on lookup questions alone (12 of 20 settings, Section 3.3).

Answer to Question 6

RAQUEL's rankings are stable across seeds. Tables 4 and 5 report means and standard deviations over three training seeds. The deviations are at most 1.1 points in Table 4 and 1.7 in Table 5, while the methods spread over 12 to 15 points of unaffected alignment on WMDP and MUSE (Section 3.3).

Bold marks the highest mean in the retain-side columns (Table 4 caption) and is not a significance test. The losses reported in Section 3.3 (14 of 20 settings) are significant under a two-sided two-proportion z-test at the 5% level against M_ret, which Section 3.1 now states.

We did not vary hyperparameters or early stopping, since all methods share one LoRA configuration and one early-stopping rule (Section 3.1). We thank the reviewer for raising this and will add a sensitivity analysis.

Answer to Question 7

Adding QA pairs requires rerunning the pipeline, but most of its cost is not repeated. Facts are extracted per QA pair (Section 2.1) and LLM responses are cached, so calls whose inputs are unchanged are reused. Materialization, nullification, and execution are deterministic and inexpensive. Re-executing the existing queries on the updated databases refreshes their labels automatically (Section 2.4), for instance when a new retain pair now supports a fact that was previously forget-only.

AuthorsOpenReview ↗

Gentle Reminder Before Tomorrow's Deadline (Nov 18)

Dear Reviewer XnwB,

With the discussion period closing tomorrow (November 18), we hope you might have a brief moment to look over our response above. Please let us know if any further clarification would be helpful before the deadline.

Thank you for your time and help in improving our work!

Reviewer_j8CROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents RAQUEL, a framework for generating more diverse questions to evaluate LLM unlearning. It uses relational databases and SQL queries to generate questions and determine whether their answers should be affected by unlearning. The authors build 9,450 QA pairs from TOFU, WMDP, and MUSE and test five unlearning methods on two LLMs. The results show that existing benchmarks can miss losses in retained knowledge, and the relative performance of unlearning methods changes on the new questions. They also show that training with RAQUEL questions improves generalization to unseen question forms.

Critical strengths

The paper uses SQL to generate questions that combine multiple facts or ask for aggregate information. By running each query on both the original and nullified databases, the authors can determine whether the answer changes after forgetting. I find this a practical way to generate a large number of test questions without manually labeling each one.

Critical weaknesses

On affected questions, the evaluation treats any answer that differs from the original as evidence of forgetting. This means that an expected answer after deletion and an unrelated incorrect answer are treated the same way. The metric therefore does not clearly distinguish forgetting from general errors. I would like the authors to provide a clearer criterion for distinguishing these two cases.

Questions for authors

Could the authors compare RAQUEL with a simpler approach that uses an LLM to generate compositional and aggregation questions directly? I am particularly interested in how the two approaches compare in question quality, diversity, and affected/unaffected label accuracy.

3 comments
AuthorsOpenReview ↗

Response to Reviewer j8CR

We thank the reviewer for the thoughtful comments, which helped us show two strengths of RAQUEL more clearly. The revised paper marks all changes in blue, and the section and table numbers below refer to it.

Answer to W1

RAQUEL can distinguish forgetting from general errors, because executing each query on the nullified database yields an explicit post-deletion answer. We thank the reviewer for raising this point, which highlights a strength of RAQUEL that we did not emphasize. We agree that an answer differing from the original does not by itself show that a model behaves as if the forget set had been removed. Existing benchmarks such as TOFU, WMDP, and MUSE provide no reference for this, whereas RAQUEL does.

Using this reference, we ran an additional evaluation. For the affected questions on which an unlearned model no longer gives the original answer, we measured how often it gives the post-deletion answer (in %, with the counts in Table 18, Appendix D.3).

MethodTOFUWMDPMUSEPooled
IDK+GD2.6413.620.596.35
GA+GD1.499.052.734.37
NPO+GD0.1423.842.469.23
RMU0.140.510.850.47
SAUL25.6838.4322.9029.28

The post-deletion answer separates the two cases the reviewer describes. SAUL gives it most often (29.28%), while every other method stays below 10%. Most answers that no longer match the original are therefore unrelated errors or refusals rather than the post-deletion answer. The revised paper defines this criterion in Section 2.4 and reports the results in Section 3.3 and Appendix D.3.

Answer to Q1

The difference between RAQUEL and direct LLM generation is how the answer and the affected or unaffected label of each question are obtained. In direct generation, the LLM decides both by itself, and nothing verifies them or controls which facts the questions cover. In RAQUEL, the LLM only proposes query templates and words the questions. The answer and the label come from executing each SQL query on two databases, one holding all facts and one with the forget-set facts removed (Section 2.4), and queries are planned around every removed fact (Section 2.3).

We thank the reviewer for suggesting this comparison, which we ran as a pilot study on MUSE (Table 19, Appendix D.4). For direct generation, we gave the source QA pairs to the same LLM and asked it to write compositional and aggregation questions with answers and labels.

RAQUELDirect LLM generation
Label accuracy99.0%94.9%
Semantically distinct questions100%61.5%
Source QA pairs covered6132

On label accuracy, RAQUEL reaches 99.0% in our human evaluation of 100 records (Section 3.2), against 94.9% for direct generation. On diversity, 38.5% of the directly generated questions are semantically equivalent to another one, while RAQUEL's are all distinct and draw on about twice as many source QA pairs.

AuthorsOpenReview ↗

Gentle Reminder Before Tomorrow's Deadline (Nov 18)

Dear Reviewer j8CR,

With the discussion period closing tomorrow (November 18), we hope you might have a brief moment to look over our response above. Please let us know if any further clarification would be helpful before the deadline.

Thank you for your time and help in improving our work!

Thank you to the authors for their detailed rebuttal. Taking into account the concerns raised by other reviewers, I believe the paper could still benefit from further improvements in its writing and presentation. Therefore, I will maintain my original score.

Reviewer_xr26OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a framework to evaluate MU using questions generated fro the forget and retain sets called RAQUEL. This first extracts entities, and relationships from the original QA pairs and constructs a relational database, one from all data (both forget and retain) and another only from the retain set. SQL queries are generated using a LLM for these databases. This queries, when executed create questions which can either differ among the two databases (classified as forget) or remain unaffected (classified as retain).

The authors evaluate five unlearning methods on Llama-3.1-8B and Qwen3-8B-Base using datasets derived from TOFU, WMDP, and MUSE. They find that models perform well on the original benchmark questions but poorly on RAQUEL-generated questions, particularly those involving retained knowledge. The relative ranking of unlearning methods also changes under RAQUEL evaluation. Additionally, the authors train models with RAQUEL-generated questions and show improvements in generalization, although substantial gaps remain between seen and held-out questions.

Critical strengths
  • The database framework provides a systematic way to generate new questions using multiple facts.
  • The paper is well written.
Critical weaknesses
  • The most important weakness is that RAQUEL does not address the identified problem. The unaffected utility on held out sets in Table 5 still remains quite low. This is also consequential because the gap between the heldout and seen utility is so high. Thus this does not solve the main utility issue, rather introduced more problems as noted below.
  • Comparing Tables 4 and 5, NPO+GD's native retain alignment on MUSE drops from 97.4% to 10.4%, while GA+GD drops from 92.1% to 58.1%. On TOFU, IDK+GD's native forget alignment increases from 0.0% to 79.8%, and NPO+GD's increases from 2.4% to 53.9%. So native retain alignment worsens, and native forget performance worsens when RAQUEL retain alignment improves. An ideal fix would increase retain alignment from RAQUEL generated questions without severe degradation of anything else. The cause of these failures should be investigated properly.
  • The authors change both the original model training data and the unlearning data between Tables 4 and 5. This creates two factors which changed and no conclusion can be reliably drawn to questions like - Does RAQUEL based unlearning improve forget or retention ? For this, we need experiments without changing the finetuned model and then unlearned with RAQUEL generated questions, as opposed to without such questions (which is Table 4).
  • The authors finetune models on the WMDP dataset. This is not the standard setting for WMDP. Since the paper draws conclusions about benchmarks, it is crucial that the actual benchmark settings are used.
  • The paper relies on LLM-judged semantic alignment rather than the established metrics of TOFU, WMDP, and MUSE. In particular, WMDP multiple-choice accuracy and MUSE VerbMem/KnowMem/Privacy are not reported. All standard metrics must be included for reliable conclusion. - This becomes especially problematic since the native retain performance is unusually good after unlearning. It is a well known fact that most unlearning methods degrade model utility substantially.

Due to such experimental flaws, I am recommending rejection. However, I am open to changing my score if evidence and experiments are provided.

Questions for authors

Please see Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer xr26

We thank the reviewer for the detailed review and for the concrete experiments suggested. The revised paper marks all changes in blue.

Answer to W1

RAQUEL is an evaluation framework, and the low held-out utility in Table 5 is the problem it exposes. Section 4 tests whether RAQUEL can evaluate models that know their facts robustly. After training with RAQUEL questions, M_orig answers 61.3 to 69.0% of held-out affected questions, against 15.4 to 30.1% before. Thus, RAQUEL can tell a model that knows the forget set from one that does not. Before unlearning, M_orig and M_ret answer 42.3 to 53.3% of held-out unaffected questions. The low values in Table 5 appear after unlearning, where every method loses 12 to 79% of M_ret's alignment (Section 4). They are therefore a finding about the unlearning methods. The gap between seen and held-out questions is expected, because seen questions are part of the training data, as native questions are in Table 4. Our new experiment (see W3) further shows that unlearning with RAQUEL questions improves held-out unaffected alignment in 14 of 15 settings. We thank the reviewer for this comment and have clarified the role of Section 4 in the revised paper.

Answer to W2

These failures come from the change of the original model between Tables 4 and 5, not from unlearning with RAQUEL questions. We ran the controlled experiment suggested in W3 (Table 16, Appendix D.1). When M_orig is kept as in Table 4 and only the unlearning data is extended with RAQUEL questions, the failures the reviewer cites do not occur. On MUSE, native retain alignment stays at 93.2% for both NPO+GD and GA+GD, instead of 10.4% and 58.1% in Table 5. On TOFU, native forget alignment is 0.3% for IDK+GD and 1.0% for NPO+GD, instead of 79.8% and 53.9%.

Meanwhile, held-out unaffected alignment improves in 14 of 15 settings, which is the outcome the reviewer describes as ideal. The failures in Table 5 thus arise because its M_orig is also trained on RAQUEL questions and knows the forget set more robustly. It answers 61.3 to 69.0% of held-out affected questions, against 15.4 to 30.1% in Table 4 (Section 4). We believe this makes the knowledge harder to remove under the same unlearning budget. We thank the reviewer for asking for this investigation, which is now part of Section 4.

Answer to W3

With the fine-tuned model fixed, unlearning with RAQUEL questions improves both retention and forgetting. We thank the reviewer for suggesting this experiment, which is now reported in Section 4 and Table 16 (Appendix D.1). M_orig is the model of Table 4, and each method is unlearned without and with RAQUEL questions in its forget and retain sets, then evaluated on the same held-out RAQUEL questions (Llama-3.1-8B). The excerpt below shows held-out unaffected alignment (%), without → with RAQUEL questions.

MethodTOFUWMDPMUSE
IDK+GD20.08 → 28.8212.34 → 35.816.80 → 34.42
GA+GD19.28 → 33.4920.29 → 40.5911.62 → 12.03
NPO+GD19.66 → 34.1019.09 → 36.1912.01 → 27.62
RMU19.04 → 16.5410.24 → 17.2915.18 → 16.13
SAUL20.17 → 42.2320.77 → 28.522.18 → 11.84

For retention, held-out unaffected alignment improves in 14 of 15 settings, by 12.5 points on average, while native retain alignment changes by less than 2 points in 13 of 15. For forgetting, held-out affected alignment decreases in 11 of 15 settings, and native forget alignment stays at or below 4.4% in all settings. The main exceptions are on MUSE, where held-out affected alignment rises for IDK+GD (by 7.0 points) and SAUL (by 8.2 points). Table 16 reports all four measures.

Answer to W4

We agree that fine-tuning on WMDP QA differs from the standard WMDP setting, and we thank the reviewer for pointing this out. We chose this setting deliberately. Fine-tuning ensures that the forget facts are known before unlearning and provides a retain-only reference model M_ret, neither of which is available for pretrained knowledge. The revised Section 3.1 labels it explicitly as a QA-adapted setting.

To test RAQUEL in the standard setting, we ran an additional experiment on off-the-shelf Zephyr-7B-beta without any fine-tuning (Table 20, Appendix D.5). We applied RMU with the official WMDP-Cyber forget corpus, with and without RAQUEL questions built from the unlearning corpora, and evaluated on the official WMDP multiple-choice questions and MMLU. Both runs share the same initial model, seed, and hyperparameters, and the RAQUEL run replaces half of the corpus examples with RAQUEL questions.

SettingWMDP-Cyber accuracy ↓MMLU accuracy ↑
Original model43.1358.88
RMU, corpus only25.5251.15
RMU, corpus + RAQUEL26.1057.98

RAQUEL thus applies to the standard pretrained-knowledge setting. With RAQUEL questions, RMU preserves much more utility (MMLU rises by 6.8 points) while forgetting only slightly less (WMDP-Cyber rises by 0.58 points)

AuthorsOpenReview ↗

Response to Reviewer xr26 (cont'd)

Answer to W5

We agree that the established metrics should be reported, and the revised paper now reports them for the Llama-3.1-8B checkpoints of Table 4 (Table 17, Appendix D.2). They cover TOFU's ROUGE-L, Model Utility, Forget Quality, answer probability, and truth ratio, WMDP multiple-choice and MMLU accuracy, and MUSE's VerbMem, KnowMem, and PrivLeak. We thank the reviewer for this request.

We reported semantic alignment because each benchmark's own metrics are specific to its answer format, whereas the paper compares native, paraphrased, and RAQUEL questions across three benchmarks and needs one measure for all of them (now stated in Section 3.1). It agrees with the established metrics where both exist. On TOFU, Retain ROUGE-L is 96.1 to 99.3% for all methods, matching native retain alignment of 98.1 to 99.0%.

The reviewer is right that unlearning degrades utility, and this is the central claim of our paper. Native retain performance looks unusually good because the methods regularize on exactly these questions (Section 3.1), so they cannot reveal the degradation. The new metrics confirm that it is there. On MUSE, retain KnowMem falls from 34.0 to between 11.7 and 24.4 for four of the five methods (Table 17).

What RAQUEL adds is a measure of the retained facts themselves. On WMDP with Llama-3.1-8B, the three measures compare as follows.

ModelNative retain ↑MMLU ↑RAQUEL unaffected ↑
M_ret98.858.726.6
IDK+GD97.555.713.1
GA+GD96.557.822.8
NPO+GD98.358.121.3
RMU97.654.810.7
SAUL96.558.022.9

Native retain questions report almost no loss (at most 2.3 points). MMLU and RAQUEL identify the same two methods as most damaging (IDK+GD and RMU), but MMLU shows a loss of 3 to 4 points where RAQUEL shows 51 to 60% of M_ret's alignment. RAQUEL also detects a loss of 14 to 20% for the other three methods, which MMLU does not. MMLU tests general knowledge, whereas RAQUEL asks new questions about the facts the model was required to keep.

AuthorsOpenReview ↗

Gentle Reminder Before Tomorrow's Deadline (Nov 18)

Dear Reviewer xr26,

With the discussion period closing tomorrow (November 18), we hope you might have a brief moment to look over our response above. Please let us know if any further clarification would be helpful before the deadline.

Thank you for your time and help in improving our work!

Comments on the paper

2 comments

Dear Reviewer,

Thank you for your efforts in evaluating this submission. The authors have uploaded their reply. Please take your responsibility to review the authors' response, revisit your evaluation,and update your score and recommendation as appropriate. Please add comments on whether the authors have sufficiently addressed your raised points. If you have further questions,please discuss them with the authors.

Thanks,

Area Chair

AuthorsOpenReview ↗

Summary of Our Responses and Revisions

Dear Reviewers and Area Chair,

Thank you for your time and your constructive reviews!

Below is a summary of the main concerns, our responses, and where they appear in the revised paper that we uploaded (changes are in blue in the paper).

ConcernResponse and resultRevision
Tables 4 and 5 change two factors at once (xr26 W1–W3)We ran the controlled experiment the reviewer suggested, which is having the same M_orig, and unlearning with and without RAQUEL questions. On held-out RAQUEL questions, adding them improves retention (unaffected alignment rises) in 14 of 15 settings and improves forgetting (affected alignment falls) in 11 of 15.§4, Table 16
Need for a relational database over direct LLM generation (j8CR Q1; XnwB Major W1–W3/Q1)We compared RAQUEL with letting the same LLM write the questions, answers, and labels directly (pilot on MUSE). Direct generation assigns the correct label in 94.9% of cases, against 99.0% for RAQUEL in our human evaluation, and only 61.5% of its questions are semantically distinct, against 100% for RAQUEL.§1, §5, Table 19
Scoring of affected questions mixes forgetting with general errors (j8CR W1; 8MWA W2/Q2)Affected alignment is now judged by how close it is to the retain model M_ret, not by how low it is. We also added a new reference that measures answer each query returns after the forget facts are deleted. Among answers that no longer match the original, 29.3% match this post-deletion answer for SAUL and under 10% for every other method (pooled over the three benchmarks), so most of these answers are errors or refusals.§2.4, §3.1, Table 18
Non-standard WMDP setting and metrics (xr26 W4/W5)We now report each benchmark's own metrics (TOFU, WMDP, and MUSE). We also ran the standard WMDP setting, RMU on off-the-shelf Zephyr-7B-beta. Using RAQUEL questions in the unlearning data keeps MMLU at 57.98 instead of 51.15.§3.1, Tables 17 and 20
Closed-world scope of the answers (8MWA W1/Q1; XnwB Major W4/Q3)The paper now states that answers are defined within the source corpus. Specifically, RAQUEL provides answers to be traced to facts stated in the sources§2.4, §6, Table 21
Robustness of rankings and scoring (8MWA Q3/Q4; XnwB Minor W1, Q4–Q6)We now name the significance test (two-sided two-proportion z-test against M_ret, 5% level) and added a figure of the forgetting–retention trade-off for each method.§3.1, Figure 3, Appendix D.6

For more details, please refer to our individual responses.

We hope this summary is helpful to the reviewers and the Area Chair in the final assessment of our work.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.