acceptodds
Under review as a conference paper at ICLR 2027

Ancestral Membership Auditing: Tracing Source Use Through Synthetic Descendants

Abstract

Synthetic training data can replace original records while retaining information about the sources that generated them. We study ancestral membership auditing: determining whether synthetic descendants of a known source were used for downstream fine-tuning when the source itself is excluded from training and the actual descendants remain hidden. We identify the Sibling Transfer Effect (STE), where independently regenerated examples from member sources exhibit more favorable average changes in answer likelihood than those from nonmember sources. We audit this effect using source-grounded question-answer probes and source-level likelihood readouts with base or reference calibration. Across eight model families on Enron and BillSum, five-reference calibration achieves mean AUCs of 0.9240 and 0.9477 with matched generators and 0.8328 and 0.8504 under generator mismatch. Removing exact matches to training records preserves detectability, and the strongest signal occurs among non-exact siblings sharing source evidence. STE can also appear through relative retention, where member probes deteriorate less even when their absolute likelihood decreases. These results show that synthetic transformation can change training records without necessarily erasing an auditable footprint of source participation.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
membership inference, synthetic data, privacy auditing, language models, fine-tuning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_RyW2OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ancestral membership auditing, which asks whether a source document contributed to downstream fine-tuning indirectly through synthetic examples generated from that source, even when the source itself and the actual synthetic training examples are unavailable to the auditor. The authors identify the Sibling Transfer Effect (STE): independently regenerated examples from a member source tend to exhibit more favorable changes in answer likelihood than examples generated from nonmember sources.
The proposed audit generates source-grounded QA "siblings" and scores them using the target model, optionally calibrated against the pretrained base or separately fine-tuned reference models. The strongest variant, Multi5, uses five reference models.

Experiments cover Enron and BillSum, eight target-model families, and three synthetic-data generators. Multi5 obtains mean AUCs of 0.9240/0.9477 with matched generators and 0.8328/0.8504 under generator mismatch. Importantly, removing exact matches to the hidden training examples barely changes performance, while shuffling source-sibling relationships reduces AUC to approximately 0.5. The signal is also substantially stronger when regenerated siblings share evidence with the hidden descendants.

Critical strengths
  1. Strong controls supporting the claimed mechanism. The most convincing aspect is that the authors go beyond showing a membership-classification signal. Removing audit QA that exactly matches the target training records produces almost no degradation, while randomly shuffling sibling families across sources reduces performance to chance. This provides good evidence that the reported effect is not simply memorization or accidental regeneration of training records.

  2. Broad evaluation across models and generator mismatch. The effect is demonstrated across eight model families, two domains, and both matched and mismatched provider/auditor generators. The cross-generator AUCs remain substantially above chance, which makes the result considerably more interesting than an effect that depends on reproducing the original generation pipeline.

Critical weaknesses
  1. The experimental setting remains substantially more controlled than realistic synthetic-data training pipelines. The study considers only one synthetic transformation, n=1n=1, with source-grounded QA fine-tuning on two text domains. It does not establish whether the signal remains detectable after longer transformation chains, substantial mixing with unrelated training data, or other synthetic transformations. These conditions could significantly dilute the source-specific signal and therefore affect the practical significance of ancestral auditing.

  2. Robustness to training stochasticity is insufficiently established. The paper reports one training realization per setting. Given that the proposed signal is defined through relatively small likelihood differences between members and nonmembers, it is important to know how much the reported AUC and the advantage of reference calibration vary across independent fine-tuning runs. The current source-level bootstrap quantifies uncertainty conditional on the trained models, but does not capture training-run variability. The authors explicitly acknowledge the single-realization limitation.

A secondary concern is that the comparison between direct-source scoring and sibling scoring is not fully controlled: the paper itself notes that the two approaches differ in probe format, token coverage, and query budget. Therefore, this experiment shows that the implemented sibling audit performs better, but does not cleanly establish that siblings are intrinsically more informative than directly probing the source.

Questions for authors
  1. Training variability: Have you repeated target and/or reference fine-tuning with multiple random seeds? If so, how stable are STE and the Multi5 versus MSG improvements across independently trained models? This would materially affect my confidence that the measured signal is not specific to the particular training realizations used here.

  2. Training-data dilution: Do you have results where synthetic descendants from the audited sources form only a small fraction of a larger mixed fine-tuning dataset? The current setup makes the source descendants a relatively direct component of the adaptation data, and it is unclear how quickly the audit signal disappears when they are diluted by unrelated examples.

  3. Direct-source comparison: Since Direct-S and sibling MSG use different formats and query/token budgets, can the authors provide a budget-matched comparison? This would clarify whether the improvement is specifically due to probing regenerated siblings rather than increased or differently structured query coverage.

8 comments

(1/2) Response to Reviewer RyW2

We thank the reviewer for their careful review and valuable feedback.

We have uploaded a revised PDF with changes marked in blue.

For Q1 + W2. Robustness across independent fine-tuning runs

MSG and the Multi5 advantage persist across five training seeds, eight students, and both domains. Seeds 42–46 control LoRA initialization, dropout and data order. Each new seed retrains the target and all five references; seed 42 reuses the original run. Sources, QA, memberships, reference cohorts and hyperparameters stay fixed, including the original five-epoch schedule and final-checkpoint rule. Qwen is provider and auditor, with N = 1,000, K = 4, L = 16.

The table reports AUC mean ± sample SD across five fine-tuning runs:

StudentEnron MSGEnron Multi5BillSum MSGBillSum Multi5
DeepSeek0.9330 ± 0.00200.9547 ± 0.00110.9398 ± 0.00030.9702 ± 0.0015
Qwen30.9030 ± 0.00320.9515 ± 0.00250.9345 ± 0.00150.9707 ± 0.0016
SmolLM20.9009 ± 0.00300.9482 ± 0.00280.9004 ± 0.00210.9660 ± 0.0019
OLMo20.9068 ± 0.00210.9488 ± 0.00130.9223 ± 0.00180.9712 ± 0.0027
RedPajama0.9173 ± 0.00280.9538 ± 0.00270.9371 ± 0.00160.9736 ± 0.0017
OpenLLaMA0.9242 ± 0.00200.9625 ± 0.00160.9487 ± 0.00200.9761 ± 0.0016
Falcon0.9181 ± 0.00160.9552 ± 0.00410.9394 ± 0.00140.9715 ± 0.0011
Pythia0.8863 ± 0.00600.9522 ± 0.00340.9035 ± 0.00190.9681 ± 0.0009

Multi5 improves AUC in all 80 paired evaluations, covering two domains, eight student families and five training seeds. Each evaluation uses a target and five references. AUC gains range from 0.0180 to 0.0717, with Multi5 AUC SD 0.0009–0.0041. TPR at 5% and 1% FPR also improves in all 80 comparisons, although it varies more across runs.

These experiments measure training randomness, not source or QA selection. Source bootstrap remains separate: it measures uncertainty conditional on trained models, whereas retraining measures run variability.

Revision: Section 4.9 and Table 4, p. 10; full runs in Appendix I.1, pp. 35–38, Tables A40–A43.

For Q2 + W1. Dilution by unrelated training data

Dilution weakens MSG but does not erase the signal; matched Multi5 stays close to performance without background. On both domains, we use DeepSeek with Qwen as provider and auditor. We retain the original 1,000 evaluation sources, four training QA from each of 500 members, and 16 audit QA per source. Thus, each target keeps the same 2,000 source QA while added background reduces their share. Each reference keeps its cohort, disjoint from evaluation sources, and receives the same background as the target, so calibration also accounts for adaptation to that background. The original five-epoch schedule and final checkpoint remain fixed: source QA are seen equally often, while total compute increases.

The UltraChat sweep reports AUC:

UltraChat records addedSource record shareEnron MSGEnron Multi5BillSum MSGBillSum Multi5
0100%0.93240.95510.93940.9711
2,00050%0.89700.94570.91530.9697
8,00020%0.86490.94090.90120.9681
18,00010%0.86160.94530.88850.9683
38,0005%0.83050.93950.86230.9657

At 5% source records, response tokens from source QA, including EOS, account for only 0.373% / 0.762% on Enron/BillSum. Multi5 loses just 0.0157 / 0.0054 AUC compared with training without background and retains 77.2% / 88.0% TPR at 5% FPR.

Other backgrounds agree: 13,828 human-written Dolly records yield Multi5 AUC 0.9558 / 0.9692; 9,000 Dolly plus 9,000 UltraChat yield 0.9508 / 0.9707. A broader mixture of 3,600 records each from Dolly, UltraChat, Alpaca, SQuAD v1 and OASST1 yields 0.9484 / 0.9721 AUC, 77.8% / 91.6% TPR at 5% FPR, and source token shares 1.620% / 3.270%. All use references retrained on the same background as their target.

Revision: Section 4.9 and Figure 7, p. 10; all settings in Appendix I.2, pp. 36–39, Tables A44–A45.

(2/2) Response to Reviewer RyW2

For W1. Longer transformation chains

A new BillSum pilot retains signal through five successive stages of QA generation. With Qwen generation and a DeepSeek student, n = 1 generates QA from the source. Each later stage receives only the preceding QA, not the source, and rewrites it under a fixed prompt.

At each depth, a fresh target trains for five epochs on 2,000 QA from the same 500 members. The 1,000 evaluation sources, K = 4, L = 16, original audit QA and five references stay fixed. Only targets are retrained, starting from the pretrained base rather than continuing from the preceding target. Keeping probes and references fixed isolates the effect of changing the descendants used for target training and tests whether the original audit still detects participation. Repeated QA remain; there is no deduplication, semantic filtering or additional generation to improve a stage.

DepthMSG AUCMulti5 AUCMulti5 TPR at 5% FPR
10.93940.971191.2%
20.93510.966386.8%
30.93430.966490.0%
40.93140.964490.2%
50.93130.966088.6%

At n = 5, Multi5 AUC is only 0.0051 below n = 1. This tests a controlled chain of QA rewrites, not every synthetic format or a fixed amount of information loss per stage. Semantic preservation is not certified; other transformation types remain future work.

Revision: Appendix I.3, pp. 39–40, Table A46; the introduction distinguishes this pilot from the primary study at n = 1.

For Q3. Direct-S comparison with matched budgets

The sibling advantage survives matched query and token budgets across all eight students and both domains. We reuse targets and pretrained bases with Qwen as provider and auditor, without new training or fine-tuned references. Both MSG and the direct controls use base calibration, so reference training does not confound this comparison.

Each method uses 16 probes per source per model, matching the main experiment's L=16 budget. Each direct probe matches its paired sibling QA in input tokens excluding padding and scored content tokens, excluding EOS. We test raw source windows and source continuations in the Instruction/Response template. Fixed anchors and adjustments at token boundaries select windows, without labels or scores.

Values below average metrics computed separately for the eight models with Qwen as both provider and auditor, not pooled predictions. They do not average the three matched-generator conditions as the original main table does:

DatasetProbeAUCTPR at 5% FPR
EnronMatched raw source windows0.726625.45%
EnronMatched templated source windows0.646417.78%
EnronSibling MSG0.910353.68%
BillSumMatched raw source windows0.703122.63%
BillSumMatched templated source windows0.650815.60%
BillSumSibling MSG0.928258.93%

MSG exceeds both controls on AUC and TPR at 5% FPR in every model and domain. More scoring queries or tokens alone therefore do not explain the advantage. The cost of QA generation and coverage of distinct facts are not matched.

Revision: Section 4.4, p. 5; Appendix E.1, pp. 22–25, Tables A19–A20. The original Direct-S results using full sources in Table 2 remain unchanged.

Thank you for the detailed response and substantial additional experiments. The new results address most of my main concerns, particularly regarding training-run variability, dilution with unrelated data, and the budget-matched direct-source comparison. The consistency across five seeds and the strong performance even at 5% source-record share are especially reassuring.

The longer-chain experiment is also useful, although it remains a controlled QA-to-QA rewriting pipeline. Thus, my main remaining reservation concerns generalization to substantially different synthetic transformations or pipelines, where the information preserved from the original source may differ considerably. In addition, the seed analysis isolates training randomness while keeping the sources and generated QA fixed, so robustness to regeneration or source/QA sampling variability is still less clear.

The rebuttal has strengthened the paper and addressed the majority of my concerns. I will read the revised manuscript and these new experiments more carefully and reconsider my score accordingly.

Reviewer_qrsLOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a method for detecting whether source documents contributed to LLM training through hidden synthetic QA pairs. The authors generate new QA pairs from the same documents and compare answer likelihoods under the target and reference models, revealing the Sibling Transfer Effect (STE). Experiments on two datasets and eight model families show strong AUC results, even under generator mismatch and after removing exact matches with training examples.

Critical strengths
  1. The paper addresses an important problem of tracing source documents used in LLM training through synthetic data. Given the growing use of synthetic data, this problem is becoming increasingly relevant.
  2. The authors propose an original approach to detecting source participation through independently generated QA pairs. The idea is supported by experiments across different models and datasets, along with a wide range of additional analyses that help better understand the nature of STE.
Critical weaknesses
  1. The paper lacks concrete examples of source documents and generated QA pairs, which would make the proposed approach easier to understand. It also provides limited analysis of how the results depend on which facts are selected for QA generation and how semantically similar the training and audit QA pairs are.
Questions for authors
  1. How would the method perform on non-member documents that contain the same facts as the training sources? Could it distinguish the use of a specific document from the use of the information it contains?
  2. The paper examines the effect of exact QA matches, but it would be useful to see a similar experiment with semantically close examples. How would AUC change as increasingly similar QA pairs are removed?
  3. Could the authors provide a representative example of a source document and its generated QA pairs, along with a more detailed description of the selection procedure? How sensitive are the results to the quality and diversity of the generated questions?
  4. TPR at 1% FPR differs noticeably between Enron and BillSum. What characteristics of these datasets might explain this difference?
  5. The results show that using generator A for training and B for auditing can produce a different AUC than the reverse setup. What might explain this asymmetry between the provider and auditor generators?
  6. In Figure 5(c, f), all Target checkpoints are evaluated against Reference Models trained for five epochs. How might the results change if the Target and Reference Models were compared after the same number of training epochs?
6 comments

(1/2) Response to Reviewer qrsL

We thank the reviewer for their careful review and valuable feedback.

We have uploaded a revised PDF with changes marked in blue.

For Q1 + Q2. Shared facts and semantic proximity

Our audit aggregates evidence from multiple facts and details, rather than testing whether the model knows one shared fact. The main setting uses 16 QA probes per source. Two documents that share some facts may still differ in other details covered by these probes. Partial fact overlap therefore does not necessarily make them indistinguishable, although our inspection of nonmembers with high scores found that shared facts can contribute to false positives.

To examine how strongly the audit depends on overlap with training QA, we added a progressive removal experiment on BillSum with DeepSeek as the student and Qwen as provider and auditor. Multi5 AUC is 0.9711 under natural regeneration and 0.9703 after exact training-QA matches are removed. Thus, most of the discrimination survives without identical training records.

We then remove each source's most similar 25%, 50%, or 75% of remaining QA, ranked by maximum joint question–answer embedding cosine against all 2,000 training QA. Random removal matches the retained count per source, with AUC averaged over 100 repetitions. All 1,000 sources remain.

Further QA removalMost-similar removal: AUCCount-matched random removal: AUC
25%0.90510.9635
50%0.73930.9471
75%0.59820.8990

Removing semantically close probes weakens the signal substantially more than removing the same number of random probes. At 75% removal, the additional AUC loss is 0.3008. Together with the exact-match result, this supports our central finding: source information can transfer to different QA expressions, while semantic proximity carries an important part of the auditing signal.

This is an oracle diagnostic because practical auditors cannot inspect training QA. It establishes sensitivity to semantic proximity, but does not establish unique attribution between documents containing the same facts. Our results support evidence of source participation, not a guarantee of unique document identification.

For Q5. Why is provider–auditor transfer asymmetric?

Reversing generators changes both learning and probing. In A→B, targets train on QA generated by A; B generates audit QA and QA used to train references. References are matched to the auditor's generator. Reversing the roles changes fact selection and wording, and transfer need not be symmetric. MSG also shows asymmetry without fine-tuned references.

Larger generators do not consistently yield better auditing. With matched generators, Qwen3-4B has Multi5 AUC 0.9529 / 0.9716 on Enron/BillSum, versus 0.9321 / 0.9558 for Granite-4.1-8B. Qwen→Granite gives 0.8785 / 0.9041; the reverse gives 0.8494 / 0.8893. These averages across eight students are descriptive, not a size ablation. Within a student family, reversing generators changes QA used to train references, not reference architecture or parameter count. See Section 3.3, p. 4, and Appendix C.6, pp. 19–21.

For Q6. What happens when reference and target epochs match?

Strong calibrated signal appears early when epochs are aligned. We combined saved DeepSeek checkpoints with Qwen as provider, using Qwen and Phi auditors to represent matched and mismatched generators. All five references use the indicated target epoch. Enron AUCs are:

EpochQwen auditor: MSGQwen auditor: Multi5Phi auditor: MSGPhi auditor: Multi5
10.70760.89440.64440.7468
20.85930.94560.77010.8399
30.91740.95340.81860.8600
40.93030.95400.81960.8549
50.93240.95510.81500.8536

Multi5 exceeds MSG at every epoch. The corresponding BillSum aligned results are:

EpochQwen auditor: Multi5Phi auditor: Multi5
10.88470.7006
20.96200.7983
30.96860.8413
40.97130.8383
50.97110.8334

We also completed all 100 cells of the 5×5 target/reference epoch grids, across both domains and auditors. The diagonal aligns epochs; column five holds final references fixed. For the same target at epoch 1, replacing references from epoch 5 with those from epoch 1 changes AUC with Qwen auditing from 0.5919 to 0.8944 on Enron and 0.6139 to 0.8847 on BillSum.

Thus, trajectories with fixed references and aligned epochs answer different questions. Epoch 5 remains the prespecified primary checkpoint for both targets and references; no checkpoint was selected from these results. No retraining was needed.

Revision: The original analysis with fixed references remains in Section 4.5, p. 6, and Figure 5, p. 8. Appendix G.4, pp. 28–31, adds Tables A28–A29 and Figures A5–A7 with aligned results and full AUC/TPR grids. These are four DeepSeek conditions, not averages across eight models.

(2/2) Response to Reviewer qrsL

For W1 + Q3. Example, selection, and QA sensitivity

Example. An Enron email states, with omissions marked:

Because the accident investigation is still underway, no one can yet know what caused it [...]

Over 75% of pipelines are intra-state pipelines

Two actual generated audit QA are:

  • Q: What is the current status of the investigation into the cause of the Bellingham accident? A: The accident investigation is still underway, and no one can yet know what caused it.
  • Q: What percentage of pipelines are intra-state pipelines? A: Over 75%

These probe distinct explicit facts. The source was selected randomly with a fixed seed, not by membership or attack score. Students receive QA, not source bodies. The source excerpt and both QA are also included in Appendix A.2, p. 15.

The numbers were fixed to implement the main experiment's training and audit budgets, not to maximize audit performance. We select five blocks of four QA: one block provides K=4 training descendants, and four blocks provide L=16 audit probes. Generating six candidate blocks and requesting six QA per call provides a fixed surplus to accommodate validity filtering and exact deduplication. This surplus helps meet the required counts; it is not used to search for QA that yield better auditing results. Retained QA are selected using frozen rules before role assignment, without using membership labels or audit scores. Appendix A.2, p. 15, now gives the detailed procedure:

  1. Generate QA grounded in source evidence IDs. Each block allows at most four attempts and stops once at least six valid candidates have accumulated. Check format, privacy, evidence IDs, refusals/templates and serialization, then remove exact duplicates of raw QA within each block. Require at least four valid QA for an eligible block.
  2. Take the first 1,000 sources in frozen order with at least five common eligible block IDs across all three generators. Fixed priority selects five blocks. Within each block, select four QA in frozen priority, preferring pairs not already selected in earlier blocks and filling any shortfall with repeated pairs in that priority order.
  3. A deterministic rule for each source assigns one training block and four audit blocks, giving K = 4, L = 16. This assignment occurs after QA selection. Labels and scores never guide selection. Natural train–audit recurrence is retained.

QA sensitivity. With DeepSeek and Enron, Qwen as provider and auditor, and all models fixed, we select eight of the original sixteen audit QA per source. Using L=8 allows different selections to be compared at the same query budget; it does not change the main experiment's L=16 setting. The table compares selection policies within this fixed eight-probe budget:

Selection, all L = 8Multi5 AUC
Random0.9282
Higher automatic support0.9117
More diverse0.9395
Less diverse0.9313
Matched random0.9373

We score QA support using Phi-4 and measure diversity with QA embedding distance. Diversity comparisons approximately match support scores and answer length; both random selection policies use 100 repetitions.

STE remains detectable across the tested QA selection policies. With the same trained models and L=8 audit QA per source, Multi5 AUC ranges from 0.9117 to 0.9395. Selecting QA with higher support scores is 0.0165 below random, showing that higher automatic support does not necessarily imply stronger membership evidence. Selecting more diverse QA is 0.0022 above matched random, a small observed gain. These results connect the sensitivity analysis to our central finding: regenerated siblings carry evidence of source participation across different probe selections, rather than only under one particular selection policy. This comparison concerns audit QA selection with fixed trained models, not the causal effect of training QA quality.

For Q4. Why does performance at low FPR differ?

Enron nonmembers with high scores often contain recurring notifications and shared short facts. Across eight students and three settings with matched generators, the top 1% have mean cosine similarity to the nearest training QA of 0.7619, versus 0.5485 for other nonmembers. BillSum gives 0.7110 / 0.5782. The association persists after length matching and in the top 5%.

BillSum also shares definitions and clauses, but its nonmembers with high scores have longer answers and span more distinct sources. These observed differences are consistent with its easier separation at low FPR.

Multi5 TPR at 1% FPR averages 30.29% on Enron and 50.31% on BillSum. Their 95% intervals from source bootstrap are 19.21%–47.11% and 34.94%–62.72%, from 10,000 resamples conditional on trained models. With 500 nonmembers, only five false positives reach 1% FPR, making this threshold sensitive to a few examples.

Thank you for the detailed responses and additional experiments, which address most of my concerns. I particularly appreciate the new analysis of semantic proximity. However, I would still like to see a quantitative assessment of how factual overlap between member and non-member source documents affects membership scores, to better understand the extent of this potential limitation.

After carefully reviewing the authors' responses and revisiting the paper, I have decided to raise my rating to 4.

Reviewer_2zm4OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary
  • The paper studies ancestral membership auditing which determines whether a synthetic data generated from a source document contributed to the data used to finetune a model even when the the synthetic samples are not available to the auditor.
  • The authors observe the Sibling Transfer Effect which is models which are finetuned on the member document data show greater improvements (compaared to the pretrained model) at answering questions wrt the ground-truth answers generated from a member document compared to a non-member document even if those QA pairs are different from those used to train the model.
  • They propose an auditing method based on this observation, comparing the likelihood of independent QA pairs as predicted by the model being audited compared to those predicted by a reference model.
  • The paper evaluates eight model families on Enron emails and BillSum documents, using three synthetic-data generators and finds that Multi5 which is their strongest method compares the target model's average answer negative log-likelihood (NLL) against the average NLL across five reference models.
Critical strengths
  • Given the increasing use of sythetic data for model finetuning, ancestral membership auditing is an important and relevant problem.
  • The paper does a comprehensive evaluation across 8 model families, multiple synthetic generators including cases where the auditor and provider use different generators.
  • The paper shows that even after removing exact matches with training examples from the independent QA pairs, the auditing method works, showing membership inference still works.
Critical weaknesses
  • The paper acknowledges that their MSG score is equivalent to the UserInf score. Even though the synthetic-data provenance setting might be new, it is not clear if the paper shows a new method or simply adapts an existing method to a new setting.
  • The evaluation of the method is based on a toy setting where models are finetuned only on synthetic data. So it is unclear if the method will work in realistic settings where the finetuning is done on a mix of real and synthetic data.
  • The strongest method requires access to 5 different reference models and thus may not be applicable to API models.
Questions for authors
  • How does the auditing performance change when only a small subset of the fine-tuning data is taken from the source document?
3 comments

(1/2) Response to Reviewer 2zm4

We thank the reviewer for their careful review and valuable feedback.

We have uploaded a revised PDF with changes marked in blue.

For Q1 + W2. Does auditing survive background dilution?

Yes, even when source QA supply less than 1% of supervised tokens. We added mixed training experiments on Enron and BillSum, using DeepSeek as the student and Qwen as both provider and auditor. To compare directly with the original main experiment, we retain its frozen source cohorts and QA budgets: 1,000 evaluation sources with 500 members, four training QA per member (K=4), and 16 audit QA per source (L=16). Each target therefore receives the same 2,000 training QA from member sources; added background data dilute their share without reducing their number.

Each reference retains its original auxiliary source cohort, disjoint from all evaluation sources, and receives the same background records as the target. Matching the background allows the references to capture general adaptation to the added data, helping distinguish these shared changes from evidence associated with participation of the audited sources. The target and all five references retain the five-epoch schedule fixed in the original main experiment, using the final checkpoint throughout. This keeps the number of training passes over source QA unchanged across dilution settings, while total training compute increases with the added background data.

All tested settings are below. Values are AUC; source record share is 2,000 / (2,000 + background records). Multi5 uses references trained on the same background as the target.

Background addedSource record shareEnron MSGEnron Multi5BillSum MSGBillSum Multi5
None100%0.93240.95510.93940.9711
Dolly: 2,00050%0.92230.95400.93810.9706
Dolly: 8,00020%0.91500.95030.93520.9718
Dolly: 13,82812.64%0.91260.95580.92950.9692
UltraChat: 2,00050%0.89700.94570.91530.9697
UltraChat: 8,00020%0.86490.94090.90120.9681
UltraChat: 18,00010%0.86160.94530.88850.9683
UltraChat: 38,0005%0.83050.93950.86230.9657
Dolly + UltraChat: 1,000 + 1,00050%0.90440.94420.92950.9725
Dolly + UltraChat: 4,000 + 4,00020%0.89060.95370.91250.9708
Dolly + UltraChat: 9,000 + 9,00010%0.88320.95080.91450.9707
Dolly + UltraChat + Alpaca + SQuAD v1 + OASST1: 3,600 each10%0.89570.94840.91620.9721

The mixture contains five datasets: Dolly, UltraChat, Alpaca, SQuAD v1, and OASST1. Its source token shares are 1.620% / 3.270% on Enron/BillSum, with Multi5 TPR at 5% FPR of 77.8% / 91.6%. Dolly also tests human-written background, extending beyond fine-tuning on synthetic data alone.

The strongest dilution is 38,000 UltraChat records. The share of supervised tokens counts response plus EOS, excluding prompts:

Source token share = source tokens / (source tokens + background tokens).

Background contributes 8,376,670 supervised tokens versus 31,360 / 64,359 source tokens. Thus, 5% of records means only 0.373% / 0.762% of supervised tokens. Multi5 retains 0.9395 / 0.9657 AUC and 77.2% / 88.0% TPR at 5% FPR, only 0.0157 / 0.0054 AUC below training without background.

Even without fine-tuned references, MSG retains AUCs of 0.8305 and 0.8623 on Enron and BillSum under the strongest dilution. This shows that the signal survives substantial background dilution without additional reference training; Multi5 further improves discrimination.

Revision: Section 4.9 and Figure 7, p. 10; full settings and token shares in Appendix I.2, pp. 36–39, Tables A44–A45.

(2/2) Response to Reviewer 2zm4

For W1. How does MSG relate to UserInf, and what is new?

The two reported scores use different length normalization. Let dⱼ = mean NLL(Base, zⱼ) − mean NLL(Target, zⱼ) be mean NLL gain per answer token and nⱼ the answer length, excluding EOS. For the fixed L = 16 audit siblings:

MSG(s) = (d₁ + d₂ + ⋯ + d₁₆) / 16.

UserInf(s) = (n₁d₁ + n₂d₂ + ⋯ + n₁₆d₁₆) / 16.

Reported UserInf preserves the log-likelihood ratio for the full answer, weighting longer answers more. MSG normalizes each QA before equal sibling averaging. If UserInf adopts the same normalization and aggregation, it equals MSG; that equivalent version is not a separate baseline.

The central challenge is that the source itself is absent from fine-tuning, while its actual training descendants are hidden. We address this by independently regenerating sibling probes from the candidate source and aggregating their likelihood gains, with references matched to the auditor's generator. The key insight is that training on one set of descendants can leave a detectable footprint on other examples derived from the same source. The signal survives removal of exact training-QA matches but disappears when source–sibling correspondence is shuffled. This enables auditing of hidden synthetic ancestry without recovering the actual training records.

Revision: Section 3.2, p. 4, clarifies this distinction; Appendix B.1, p. 16, gives both formulas.

For W3. Do five references prevent auditing through APIs?

References are local models trained by the auditor, not five private target models. They use auxiliary sources disjoint from evaluation sources and the auditor's generator. They add local training and scoring costs, not target queries: each source still uses 16 QA probes.

Without fine-tuned references, MSG already achieves 0.8739 / 0.8841 mean AUC with matched generators on Enron/BillSum; Multi5 reaches 0.9240 / 0.9477, averaged over eight students and three conditions with matched generators.

A concrete application is auditing a fine-tuned open model served through an API. For example, Fireworks documents both deployment of fine-tuned LoRA models and access to prompt-token log probabilities. When the pretrained base is available and the endpoint exposes conditional likelihoods for supplied answers, the auditor can query the remote target while training and evaluating references locally, without accessing target weights or gradients. The five references therefore add local computation, not a requirement for five private target models. APIs that return only generated text remain outside our current likelihood-based evaluation.

Sections 3.1 and 3.3, pp. 3–4, and Appendix B.4, p. 17, specify access and costs.

Dear Reviewer,

Thank you for your time and thoughtful review.

As the discussion period is nearing its end, we would appreciate your thoughts on our responses, additional experiments, and updated manuscript. Please let us know if any concerns remain or if further clarification would be helpful. We welcome the opportunity to discuss them with you.

Best regards, Authors of Submission 31479

Comments on the paper

1 comment
AuthorsOpenReview ↗

# General Response

For clarity and simplicity, we refer to Reviewers 2zm4, qrsL, and RyW2 as R1, R2, and R3, respectively, throughout this response.

We sincerely thank all reviewers for their thoughtful and constructive feedback. We are encouraged by R2’s and R3’s positive feedback on our additional experiments and clarifications, and appreciate their recognition of the strengthened evaluation and clearer contribution.

We have carefully considered each reviewer comment and provided detailed point-by-point responses, supported by additional experiments and analyses. In our revised manuscript, we have incorporated the new experimental results, additional discussions, and relevant clarifications to further strengthen our work. Below, we summarize our core contributions, the experimental additions, and the main clarifications made during the discussion period.


Core Contributions of Our Work

  1. Ancestral membership auditing: We study whether a source contributed to downstream training through synthetic descendants, even when the source itself was not a training record.
  2. The Sibling Transfer Effect: Independently regenerated siblings reveal source participation beyond exact recurrence of training records, including under generator mismatch.
  3. Systematic evaluation: We evaluate ancestral membership auditing across eight student model families and two domains. We further characterize generator mismatch, evidence overlap, relative retention, and auditing-resource sensitivity, and extend the evaluation to three additional synthetic transformations.

Updates of experimental results during Rebuttal

  • Main Text, Section 4.9, page 10: Added training-seed robustness results (R3) and background-dilution experiments (R1, R3). Full results are provided in Appendices I.1–I.2.
  • Main Text, Section 4.9 and Table 4, page 10: Added experiments with compressed summaries, explanatory prose, and knowledge lists (R3). Detailed protocols and audit-budget controls are provided in Appendix J, page 42.
  • Main Text, Section 4.4, and Appendix E.1: Added budget-matched Direct-S comparisons across all eight model families and both domains (R3).
  • Appendices I.3–I.4, pages 39–41: Added longer transformation chains and separate experiments varying member assignments, QA regeneration, and QA selection (R3).
  • Appendix G.4: Added the full target/reference epoch comparison grid (R2).

Updates of in-depth discussions during Rebuttal

  • Main Text, Section 3.2, and Appendix B: Clarified the relation between MSG and UserInf and their different normalization conventions (R1).
  • Appendix A.2, page 15: Added a representative source document and its generated QA pairs, together with the selection procedure (R2).
  • Responses to reviewers: Clarified reference-model and likelihood-access requirements (R1), and provided quantitative analyses of semantic proximity, QA quality and diversity, and factual overlap (R2).

Although we have not yet received follow-up feedback from R1, background dilution was a concern shared by R1 and R3. R3 described the strong performance at 5% source-record share as “especially reassuring” and subsequently noted that the extension beyond QA made the contribution clearer. R2 described the factual-overlap analysis as “a valuable addition to the evaluation.”

We believe these additional experiments and clarifications address the main concerns raised during review and strengthen the evidence and clarity of our contributions. All revisions are highlighted in blue in the updated manuscript for ease of reference.

We look forward to the reviewers’ favorable consideration of our revised manuscript and additional evidence. We sincerely appreciate their valuable feedback and welcome any further questions or discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.