acceptodds
Under review as a conference paper at ICLR 2027

Before The Attacker Acts: Evaluation Floors for LLM Deanonymisation

Abstract

Deanonymisation scores for language-model attackers come with no reference level, so a reader cannot tell how much of a score is inference. We give two levels. Below is a floor: the best score of null attackers that read no post and claim no more than the task’s declared volume v (the seed, random guessing at v, the modal value, abstention), fixed before the attacker runs, with a matched volume contrast added afterwards; every lift is read under one declared paired t-test. Above is a labelled-evidence ceiling: the recall of an attacker merging accounts only on specifics the answer key records, which at the planned number of personas gives a pre-run screen, saying from the key alone whether any such attacker could produce a detectable lift. Under SynthPAI’s own scorer a constant guess fitted out of sample scores 0.275 against GPT-4’s 0.780; all 18 models clear it overall, but after Holm correction over 144 model-by-attribute cells 65 clear, 1 is at or below it and 78 are undecided, none on age or income. On one of our corpora the screen finds that no attacker merging only on labelled evidence could be told from random on its 20 personas, and the agent’s cross-account recall of 0.121 sits below the 0.137 random guessing at v = 3 handles per seed is expected to reach, though the agent claimed one handle in all. Report the null row, the declared test and the pre-run screen beside the score.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
deanonymization, privacy, llm agents, evaluation baselines, cross-platform account linkage, attribute inference, measurement validity, reporting standards

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_iEnFOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes an evaluation framework for LLM-based deanonymization attacks. The authors argue that raw attack performance is misleading without appropriate reference levels. On this basis, the framework defines a floor and a conditional ceiling, as well as a pre-run statistical screening procedure, to assess whether a benchmark can detect improvements over trivial baselines. Experiments on synthetic benchmarks and existing attack systems reveal inflation in existing results with limited statistical distinguishability.

Critical strengths

S1. The paper focuses on a relevant and real concern that existing deanonymization performance may be inflated due to scoring conventions and dataset characteristics.

S2. The labeled-eveidence ceiling and screening procedure offer a potentially useful way to identify benchmark limitations.

Critical weaknesses

W1. Clarity and organization are a major problem of this paper. The paper is extremely difficult to follow. Core concepts, including "persona", "null attacker", "claim volume", "labeled-evidence ceiling" and many more, are undefined at all. The abstract and introduction are dominated by numbers and these specialized, but undefined terminologies, rather than a clear statement of the research question and contributions.

W2. The figures are clearly AI-generated and poorly organized. Figures 6 to 10 pack multiple metrics, baselines, intervals, and annotations into dense plots with overlapping or very small labels. It would be much clearer to present them as conventional tables. A substantial restructuring and simplified figures are needed.

W3. Many experiment setups are missing. The Q1/Q2 distinction also appears hard to follow. What constitutes a successful attack in each setting? What information does the attacker have in each dataset and baseline? How are ground-truth identities and attributes constructed?

W4. The principal agent audit relies on a small dataset of only 20 synthetic personas, with a single run per condition. Larger-scale benchmarks, repeated independent runs, and evaluation of different attack policies are needed to support the claims made in this paper.

W5. Novelty and the definition of the floor need stronger justification. Based on the definition in the paper, the floor is essentially the maximum over a manually selected, non-exhaustive family of trivial baselines at a pre-specified claim volume. The ceiling is a count of ground-truth links satisfying an evidence-matching rule. Technically, both are closely connected to established practice in existing benchmark evaluation, entity resolution, and power analysis. It is unclear which components are genuinely novel, and what conclusions are unique contributions from this paper, instead of simpler baseline comparisons.

Questions for authors

See weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer iEnF (1/2)

Thank you for the detailed review. Line numbers refer to the submitted PDF.

W1. Core concepts are undefined; abstract and introduction are dominated by numbers.

Most of these terms are defined, but too briefly and not always at first use. Their meanings, with where the submission defines each:

TermMeaningRef
personaone fabricated person in the corpus, with accounts (handles) on up to six platformsApp. E
seedthe one post, with its handle, that the attacker is givenSec. 4
null attackera procedure that never reads a post's contentl. 145–148
claim volume vhow many handles an attacker may claim for one seed; v = 3 herel. 141–142
floor F, liftthe best score of any null in the family; the attacker's score minus FEq. 1
labelled-evidence ceilingthe best recall of an attacker that merges two accounts only when the answer key records at least k shared specificsSec. 7, Eq. 3
corridorceiling minus floor: the room a result hasl. 178–182
screena test on the answer key alone of whether a lift as large as the corridor is detectable at n personasSec. 7

We put numbers in the abstract and introduction because this is a measurement paper and we wanted each claim to carry the value that supports it. We accept that this buried the research question, and the revised introduction states it first: how much of a reported deanonymisation score would a procedure that reads nothing also obtain, and can the benchmark tell the two apart?

W2. Figures 6 to 10 are dense; tables would be clearer.

Figures 6–10 are tables with an interval drawn beside each row. They are generated by scripts from the stored run reports, and we chose the format so that every score sits on the same line as its null and its interval, which is the paper's argument. We agree that they are too dense. In the revision every result figure has a conventional table with the same numbers, and the figure points to it. For Figures 5, 6, 8 and 10 these are the appendix tables of the submission (Tables 1, 8 and 10–16). For Figures 7, 9 and 11 the revision adds three new appendix tables, generated from the same files as the figures and checked equal to what each figure draws. The audit's scores are now also a table in the main text (Section 6).

W3. Experiment setups are missing. What is a successful attack, what does the attacker see, how is ground truth built?

Q1 (seeded)Q2 (seedless)
Attacker is givenone post and its handleconstraints over attributes
Attacker can readall posts, handles and threads of the corpus, before anonymisation (Fig. 3)same
Withheldthe handle-to-person map and the generation-side fields (Fig. 3)same
Must returnthe author's other posts across platforms and handles, and values for 25 attributesthe matching people, ranked
Scored byidentification, exact ownership, linkage F1, cross-account recall, attribute accuracy, complement F1mean average precision
Successa lift over F whose declared-test interval excludes zerosame

Ground truth: the identities are fabricated, so the handle-to-person map and each persona's attributes are exact by construction (Fig. 3; App. E). Leak labels come from a per-post detector on the audited corpora (Sec. 4). Every metric is defined with its denominator in Table 4 (App. B), the corpora in App. E, and the comparison attackers and what each reads in App. I. The revision has this table in Section 4.

AuthorsOpenReview ↗

Response to Reviewer iEnF (2/2)

W4. The audit uses 20 synthetic personas and a single run per condition.

Both are stated as limits in Section 8, and the first is one of the paper's findings: the pre-run screen shows that 20 personas of that corpus cannot separate a rule-following attacker from random (Sec. 7). The audit is listed as "a worked example" among the contributions, and Section 8 says that "the external evidence here is therefore the SynthPAI re-analysis and the published systems rather than the audit". The evidence beyond 20 personas is:

EvidenceScaleRef
SynthPAI re-analysis18 models, 144 model-by-attribute cells, 700 human-verified labelsSec. 5
Population floor45,222 census records; 1,000 sampler personasSec. 5; App. C
Audit across corpora4 corpora, 80 personas pooledFig. 7
Screen20 corpora (up to 1,000 personas), 540 cellsTable 8; App. H
Other attack policies6 published systems, 4 baselines, 5 single-tool ablationsFigs. 6, 8, 9; App. I

On single runs: each cell is one temperature-0 transcript, so we have no estimate of run-to-run variance, and Section 8 says so. The declared test takes its variance across personas, which is the unit the conclusions are about.

W5. The floor is a maximum over a hand-picked, non-exhaustive family; the ceiling is a count. What is new?

On non-exhaustive: yes, and Section 3 says so (l. 150–151: the family is "exhibited rather than exhausted", so "every lift is an upper bound"). The direction is safe. Adding a null can only raise F and lower a lift. A verdict of "does not clear the floor" therefore survives any larger family, and that is the audit's verdict on the seeded task.

On novelty, Section 2 names the origin of each part. What is borrowed and what is added:

ComponentPrior practiceWhat we addConclusion that needs it
FloorExcess over one naive guess (Anonymeter); pool chance in closed-set linkageA null family limited to a declared claim volume, for tasks with no candidate poolRecall below random at v, above random at matched volume
Harness creditFiltered metrics in link predictionComplement scoring for a seed-union harnessIdentification 1.000 and linkage F1 0.562 conceded
TestA test chosen per resultOne declared test for every lift and contrast, with a declared correction family65 of 144 SynthPAI cells clear; none on age or income
CeilingPair completeness in entity resolutionThe same count on labelled evidence under the attacker's own declared rule, with no model callOnly 10 of 121 pairs directly corroborated on SeqBench-20
ScreenPower analysis with an assumed effect sizeThe key and the rule fix the largest lift, so no effect size is assumed; three verdictsSeqBench-20 unresolvable; four corpora need 42–60 personas

We claim three things: agentic deanonymisation results are being reported without these checks, the task needs these specific adaptations for a floor to be defined at all, and applying them changes what the published numbers say.

Dear Reviewer iEnF,

We greatly appreciate the time and effort you’ve invested in reviewing our work. As we are now two days away from the discussion deadline, we wanted to reach out regarding our rebuttal, which we submitted earlier.

If possible, we would be grateful for any feedback you might be able to provide, as this would give us the opportunity to engage in a productive discussion before the deadline. We’re eager to address any remaining questions or concerns you may have about our paper.

We understand that you likely have a busy schedule, and we truly appreciate your valuable insights and expertise in this process.

Thank you for your time and consideration.

Thanks for the additional explanations. However, my concerns regarding the presentation and clarity remain. I encourage the authors to fully revise the paper. I'll keep my current rating.

Reviewer_HiysOpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Deanonymisation scores come with no reference level, so a reader cannot tell how much of a reported attack is inference. The paper supplies two anchors. Below, a floor: the maximum score of a family of null attackers that read no post and claim no more than the declared volume (seed, random at v, modal value, abstention, budget-matched), with every lift read under one declared paired t-test. Above, a labelled-evidence ceiling: the recall of an attacker that merges only on specifics the answer key records, countable from the key with no model call, which gives a pre-run screen for whether any such attacker could produce a detectable lift at the planned sample size. The payoff is substantial. The seed-union harness concedes identification of 1.000 to a null that reads nothing. On SynthPAI a constant guess scores 0.275 against GPT-4's 0.780, and after Holm correction over 144 cells only 65 clear the floor, 78 are undecided, and none clears it on age or income. Their own audited agent sits below random's expectation on cross-account recall and 0.180 below the modal null on attribute accuracy; only the seedless task clears. Published systems are re-scored against the same floors, and a four-point reporting standard (Box 1) packages the whole instrument.

Critical strengths
  • The metrology catches real concessions that published benchmarks are making right now. The seed-union result is the standout: any harness that scores the seed handle's own timeline hands every method, including a null that reads nothing, an identification of 1.000 and a linkage floor of 0.562 on their corpus. This is not a hypothetical; it is a structural property of how these evaluations are built, and the complement-scoring fix is given. The SynthPAI re-analysis is the second: no model, GPT-4 included, is shown to beat a fitted constant on age or income once the test is declared and corrected. Either result alone would change how this literature reports scores.
Critical weaknesses
  • The empirical anchor is unverifiable on two levels. The leak labels that the ceiling counts come from a post-hoc detector with no validation sample on ten of eleven corpora, and the ceiling is maximally sensitive to them: swapping SeqBench-20's detector labels for rich-v9's in-run labels moves the ceiling from 0.178 to 0.726 and flips the screen from unresolvable to resolvable. So the ceiling is only as good as labels nobody has checked. Underneath that, the population is synthetic, generated by a companion submission reviewers cannot see, and the paper's own Appendix D shows it does not transfer: sampler personas are about five times more distinguishable than census records at the same size, so floors measured here bound nothing about real populations. The instrument is sound; its calibration is faith.
Questions for authors
  • The ceiling depends entirely on leak labels that have no validation sample on ten of eleven corpora, and it moved 4.1x when the labelling method changed. Can you validate the post-hoc detector's precision and recall on a human-checked sample, and report ceilings and screen verdicts under that label uncertainty? If the labels are noisy, which of the paper's verdicts survive?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Hiys (1/2)

Thank you for the careful and fair reading. We agree that the detector's leak labels are unvalidated and that floors do not transfer to real populations. We would put the conclusion more narrowly than "its calibration is faith": most verdicts use no leak label, the rest are stated as conditional on the key, and our second reply measures how far they move. Your question is which verdicts survive if the leak labels are noisy.

Most of them, because most use no leak label. The paper has two kinds of answer key.

Answer keySourceStatus
Handle-to-person map; each persona's attributesThe generator created themExact
Leak labels (which post reveals which attribute)A post hoc detector on the audited corporaNot validated. Section 4: "every recall on them is a recall against the detector's key" (l. 218–220)
VerdictUses leak labels?If the labels are noisy
Seed-union concession: identification 1.000, linkage F1 0.562 (Sec. 3)NoUnchanged
Agent against F on linkage F1, cross-account recall, complement F1; matched-volume contrast (Sec. 6)NoUnchanged
Attribute accuracy 0.180 below the modal null; Q2 lift of +0.462 (Sec. 6)NoUnchanged
Published systems against the same floors (Fig. 6)NoUnchanged
Population floor on UCI Adult and sampler personas (Sec. 5)NoUnchanged
SynthPAI floor re-analysis (Sec. 5)SynthPAI's human-verified labelsNot our labels
SynthPAI ceiling (Fig. 11)SynthPAI's human-reviewed labelsNot our labels
Ceiling, corridor and screen on the audited corpora (Sec. 7)Yes, detectorConditional on the key
Screen on the eleven released corpora (Fig. 10, right)Yes, the generator's planner labelsConditional on the key
"4 of 9 merges break the premise" (Sec. 7)Yes, detectorConditional on the key

So the two results you call the standout ones, the seed-union concession and the SynthPAI re-analysis, do not depend on our labels at all. You write that "either result alone would change how this literature reports scores"; both hold whatever the detector's error rate.

The ceiling verdicts are conditional, and the paper says so. A spurious label can only raise the ceiling and a missed label can only lower it (Table 9, row S10: the bound holds "if every label is genuine"). The 0.178 to 0.726 comparison is our own control, run to show this sensitivity: it uses the same generator, seed and size with 633 in-run labels in place of the detector's 158, which suggests the detector misses labels and that SeqBench-20's "unresolvable" is a verdict on that key. We wrote it that way: "the verdict is about this corpus, not the generator" (l. 465), and the ceiling "bounds what the labels can license" (l. 068). What we have not done is validate the detector against a human-checked sample (Section 8 says we ran no human study), and every verdict that depends on it is stated as conditional on the key.

AuthorsOpenReview ↗

Response to Reviewer Hiys (2/2)

Label noise, measured. Since submission we ran the check you ask for in its cheapest form. On SeqBench-20 we removed or added labels at random and recomputed the ceiling and the screen (100 draws per row, k = 2, seed-adjacent reading, no model call).

LabelsCeiling, mean [min, max]Corridor, meanStill "unresolvable"Became resolvable
Detector's key0.178+0.041n/an/a
10% removed0.166 [0.134, 0.178]+0.02999% of draws0 of 100 draws
20% removed0.155 [0.115, 0.178]+0.01889%0
30% removed0.142 [0.102, 0.178]+0.00563%0
10% added0.194 [0.178, 0.243]+0.057100%0
20% added0.214 [0.178, 0.285]+0.07798%2
30% added0.234 [0.178, 0.311]+0.09792%8

The corpus became resolvable in 10 of the 600 draws. No draw with labels removed raised the ceiling and none with labels added lowered it. Every verdict that changed under removal became "dead" (no room left above the floor). The sweep covers random errors only. A detector that misses labels systematically is a larger change: the in-run key in our first reply has four times as many labels (633 against 158) and does flip the verdict.

Synthetic population and the companion submission. What this paper needs from the corpora is in App. E and App. J (sizes, seeds, schema, labeller and hash of each audited corpus). The results on SynthPAI, UCI Adult and the published systems' own data do not use our generator, and Section 8 names them as the external evidence for that reason.

"Floors measured here bound nothing about real populations." We agree, and it is the claim of Section 5 and of requirement R2 in Box 1: a floor is a property of a population, a vocabulary and a scorer, and "transfers nowhere without them". App. D reports the five-fold gap for that reason. What we offer for reuse is the procedure (the null family, the declared test, the screen), to be recomputed on each benchmark, as we did on SynthPAI.

"Its calibration is faith." That is fair for the ceiling on our corpora. We think it is too strong for the instrument as a whole. Each part, and what it rests on:

PartWhat it rests onRef
FloorsKeys that are exact by construction, or human-verified on SynthPAI. No leak label of ours entersSec. 3, 5, 6
Floor and ceiling on a benchmark we did not buildSynthPAI's 700 human-verified labels: our port of its scorer reproduces GPT-4's Table 4 to the printed decimal, and no label-keyed attacker exceeds the ceiling on any seedSec. 5; Fig. 11
Population floor45,222 real census recordsSec. 5
Ceiling and screen on our corporaThe detector's labels, unvalidatedSec. 7

The last row is the open one, and we agree it is the weakest input. Its effect under random error is measured above, and a detector that misses labels can only make the ceiling too low (Table 9, row S10). Its precision and recall against a human-checked sample remain open; Section 7 of the revision states which results depend on its labels, beside the sweep.

Dear Reviewer Hiys,

We greatly appreciate the time and effort you’ve invested in reviewing our work. As we are now two days away from the discussion deadline, we wanted to reach out regarding our rebuttal, which we submitted earlier.

If possible, we would be grateful for any feedback you might be able to provide, as this would give us the opportunity to engage in a productive discussion before the deadline. We’re eager to address any remaining questions or concerns you may have about our paper.

We understand that you likely have a busy schedule, and we truly appreciate your valuable insights and expertise in this process.

Thank you for your time and consideration.

The rebuttal defenses that the detector's validity is not related to many of the paper's claims, while leaving the question on the validity without any human label comparison. I think the paper does some meaningful investigation of floor and ceiling patterns in deanonymization tasks, and uses that investiagtion to point out problems in existing benchmarks (SynthPAI, Inferlink) and gives new insight on the scores of those benchmarks. However as the other reviewers state, the paper overall lacks a lot of clarity in their terms, and does not open-source evidence enough for reproduction and verification of their claims. I decide to lower my score.

Reviewer_SVUnOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that deanonymisation scores are not interpretable without reference points, concretely a "floor" (a baseline while doing no inference) and a "ceiling" (the most an attacker could score from fully labeled content). These reference points are independent of any proposed attack method and can be determined once per benchmark.

Critical strengths

None

Critical weaknesses
  • The writing is extremely unclear and obscures the underlying arguments and experiments
  • To my understanding, the contribution is rather basic and partially flawed: including a sanity check "floor" baseline in results is a widely known and adopted practice, while the instantiation of the "ceiling" seems flawed, as acknowledged by Section 7 which states that none of the tested attackers satisfy the necessary assumptions for it to apply.
Questions for authors
  • Given that the investigated attackers don't satisfy the assumptions for the ceiling to apply, what evidence supports its practical usefulness for evaluating LLM-based deanonymisation?
  • What concrete insight does the proposed floor-and-ceiling framework provide that could not be obtained through conventional baseline comparisons and dataset annotation analysis? In particular, do any empirical conclusions depend specifically on the proposed formalization rather than on the aforementioned existing techniques?
3 comments
AuthorsOpenReview ↗

Response to Reviewer SVUn

Thank you for the review. The general response states what each section is for.

W. A sanity-check "floor" baseline is widely known practice.

We agree that the idea is old, and Section 2 says where ours comes from: Eq. 1 is Anonymeter's (Giomi et al., 2023), and partial-input baselines come from benchmark criticism. Two things made us write the paper anyway. First, in the literature we address the practice is absent: none of the five agentic attack evaluations we cite reports a null attacker (Thees et al., 2027; Lermen et al., 2026; Ko et al., 2026; Du et al., 2026; Peng et al., 2026; l. 083–087). Second, in this task the usual baseline is not well defined. Classical identity linkage works inside a closed candidate set (Shu et al., 2017, Definition 1), where the floor is one over the pool; agentic evaluations have no pool (l. 054–055). And recall-type metrics reward how much an attacker claims (l. 057–058, 164–166). Two procedures that read nothing score 1.000 (l. 055–057):

Null attackerReads a post?Scores 1.000 on
Claims nothingNoIdentification, under a seed-union harness
Merges everythingNoCross-account recall

What we add is what makes the floor well defined here: a family of nulls limited to a declared claim volume, the maximum over that family, one declared test, and scoring of the complement where a harness grants credit.

Q2. What does the framework show that conventional baseline comparison and annotation analysis would not? Do any conclusions depend on the formalisation?

Yes. Each row needs a specific part of it.

ConclusionWhy a conventional baseline misses itRef
The harness gives identification 1.000 and linkage F1 0.562 to a procedure that reads nothingA baseline is a weaker attacker scored by the same harness, so it receives the same credit and the comparison hides itSec. 3; Fig. 12
Our agent's cross-account recall (0.121) is below random at the declared volume (0.137) but above random at its own volume (+0.032 pooled over four corpora, p = 0.039)The sign depends on how much the null may claim; without a declared volume either could be reportedSec. 6; Fig. 7
On SynthPAI all 18 models beat a constant overall, but none is shown to beat it on age or incomeIt needs a constant fitted out of sample under the benchmark's own scorer, and one correction over all 144 cellsSec. 5
SynthPAI's scorer credits content-free answers ("18-99" scores 1.000 on age)Found only by maximising over a family of nulls instead of picking oneSec. 5
k-anonymity reports 1 throughout while the population floor falls 87-foldNeeds the population floor of Eq. 2Fig. 4
SeqBench-20 cannot separate a rule-following attacker from random at 20 personasNeeds floor and ceiling together, before the runSec. 7; Fig. 10

Q1. No tested attacker satisfies the ceiling's assumptions. What supports its usefulness?

That sentence in Section 7 is a result of the check, and we kept it prominent on purpose. The ceiling is the most an attacker can reach by merging only on evidence the answer key records, under the merge rule its own prompt declares. An attacker that uses other evidence can exceed it (Proposition 1 is titled "a counting device"; the introduction says it "bounds what the labels can license", l. 068). It is useful in three ways.

UseWhenWhat the paper shows
A statement about the benchmarkBefore the run, from the key aloneOn SeqBench-20 only 10 of 121 true account pairs are directly corroborated by two specifics in the key, so the room above the floor is 0.041 and no rule-following attacker is detectable at 20 personas
A diagnostic of the attackerAfter the runAn attacker that exceeds the ceiling, or merges outside the admissible set, used evidence the key does not record or broke its own rule. The stylometric linker has 17 of 20 correct merges outside the premise and reaches 0.369; our agent has 4 of 9. Without the ceiling these merges look like ordinary successes
A bound that holdsOn human-reviewed labelsOn SynthPAI no attacker keyed on the labels exceeds it on any seed (Fig. 11)

We agree that the word "ceiling" invites a stronger reading than we intend, and the revised introduction states the condition where the ceiling is introduced.

Dear Reviewer SVUn,

We greatly appreciate the time and effort you’ve invested in reviewing our work. As we are now two days away from the discussion deadline, we wanted to reach out regarding our rebuttal, which we submitted earlier.

If possible, we would be grateful for any feedback you might be able to provide, as this would give us the opportunity to engage in a productive discussion before the deadline. We’re eager to address any remaining questions or concerns you may have about our paper.

We understand that you likely have a busy schedule, and we truly appreciate your valuable insights and expertise in this process.

Thank you for your time and consideration.

In my view the rebuttal is written in the same wordy but at the same time dense language as the original paper and while a minority of items has been clarified, my core concerns around presentation & clarity remain and I will keep my score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to all reviewers (2/2): the audit, the ceiling, and what the revision changes

Table G3: Sections 4 and 6 apply the floors.

QuestionFindingTypeRef
What does the harness concede?A null that reads nothing gets identification 1.000 and linkage F1 0.562, by arithmeticTSec. 3; Fig. 12
Does our own agent clear its floors?On the seeded task, none of the three metrics with headroom does. On the seedless task it does (+0.462)EFig. 5; Table 1 (App. B)
Does that hold across corpora?Mean lift over four corpora is −0.002 [−0.034, +0.030]EFig. 7; App. G
Published systems on the same floors?LUAR-MUD lifts complement F1 by +0.622 on SeqBench-n50 but falls below the seed null on a sparse sample; the classical rung of Lermen et al. clears its null in 1 of 12 cellsEFigs. 6, 8; App. I

Table G4: Section 7 gives the ceiling and the pre-run screen.

QuestionFindingTypeRef
What can labelled evidence reach under the attacker's own merge rule?A count on the answer key, no model callTEq. 3; Prop. 1; Table 9
Can the corpus detect such an attacker at n personas?SeqBench-20: no (t = 1.10 against 2.09). Four released corpora: not at 20 personas, yes at 42–60EFig. 10; Table 8
How much does it depend on the labels?In-run labels in place of detector labels move the ceiling from 0.178 to 0.726ESec. 7, l. 445–448
Do real attackers stay under it?Ours does not: 4 of the agent's 9 distinct correct merges use evidence outside the rule. On SynthPAI no label-keyed attacker exceeds itESec. 7; Fig. 11

How the sections support the title.

SectionWhat it establishes
3The lower level, and how to test against it
5The lower level must be recomputed for each population, vocabulary and scorer
4, 6Applied to our own agent and to six published systems, most seeded scores turn out to be conceded
7The upper level. The two together decide, before any run, whether a corpus of a given size can separate an attacker from a null

What the revised PDF changes. Presentation, plus one check. Every change is in red and the main text is now 10 pages. Section, figure and line numbers in these replies still refer to the submitted PDF.

ChangeWhere in the revision
The research question in plain words, the contributions with their sections, and a table of termsSection 1
The experimental setup as a table: what the attacker is given, reads and returns, and what counts as successSection 4
The condition under which the ceiling bounds an attacker, stated where the ceiling is introducedSection 1
The label-noise sweep, and which results use no leak label of oursSection 7
Statements made more than once cut to oneSections 1, 3, 4, 6 and 7; appendix
The appendix folded from fourteen sections into sevenAppendix
Every result figure has a plain table with the same numbers, three of them newFigure captions; appendix
The audit as a table: the agent's scores beside the best procedure that reads no post (was in Appendix B)Section 6, Table 4
The diagram of why the seed floor is 1.000 (was in Appendix A)Section 3, Figure 3
The SynthPAI and InferLink re-analyses as a table (were prose only): per attribute, how many of the 18 models clear a constant guessSection 5, Table 3
The screen's verdict counts over all 540 cells (were in Appendix E)Section 7
To make room, the tool-ablation and LUAR figures sit beside their plain tablesAppendix D.3 and F

No claim changes, and the label-noise sweep is the only new experiment. Table 3 prints per-attribute and per-cell values of analyses the submission reported in prose.

AuthorsOpenReview ↗

General response to all reviewers (1/2): what each section is for

We thank the three reviewers. Two of the three found the paper very hard to follow, and the third needed "substantial effort in places"; we take that as the main problem. Here we state what the title claims, what each section is for, and where each claim is supported. Section, figure, table and line numbers are those of the submitted PDF.

What the title claims: A deanonymisation score can only be read against two reference levels.

Words in the titleClaimWhere
FloorThe lower level: what a procedure scores that reads no postSec. 3
CeilingThe upper level: what the evidence recorded in the answer key lets a rule-following attacker reachSec. 7
Before the attacker actsBoth are computed from the task definition, the scorer and the answer key, with no model callSec. 3, 7
for evaluatingThe gap between them, at the planned sample size, says in advance whether the experiment can show anythingSec. 7

Table G1: Section 3 defines the floor. T = theory or definition, D = design, E = empirical.

QuestionFindingTypeRef
What is a floor?The best score among null attackers that read no post and claim at most v handlesTEq. 1; Def. 1 (App. A)
Why restrict what a null may claim?There is no closed candidate pool. A null claiming nothing scores identification 1.000 under a seed-union harness; a null merging everything scores cross-account recall 1.000T, El. 054–060; Sec. 3
How is a lift over the floor read?One paired t-test declared in advance, Holm-adjusted over a declared familyDSec. 3
What does the population itself concede?The share of distinct quasi-identifier classes, F_idTEq. 2; Prop. 2 (App. L)
What should a paper report?Four requirements, R1–R4DBox 1

Table G2: Section 5 shows that a floor does not transfer.

QuestionFindingTypeRef
Does it transfer across populations?k-anonymity stays 1 while F_id falls 87-fold as the census grows; sampler personas give 0.788 where census records give 0.153EFig. 4; App. C, D
Across vocabularies?The attribute floor runs from 0.489 to 1.000 across one generalisation latticeESec. 5
Across scorers, on a benchmark we did not build?Under SynthPAI's scorer a fitted constant scores 0.275 (GPT-4: 0.780). Of 144 model-by-attribute cells, 65 clear it (none on age or income) and 78 are undecidedESec. 5

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.