- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposed a method that makes fake time-series data that looks like a private dataset without leaking private information. It doesn't train a new model on the private data. It reuses a public model in a loop: the public model makes fake samples, each private sample "votes" for the fake sample it looks most like, noise is added to the votes to protect privacy, and the winners are kept and slightly changed. After a few rounds, the fake data looks more like the real data. On five types of data (traffic, energy, activity, heart signals and crypto), the authors report better results than other privacy-safe methods, about 3.4× higher classification accuracy than DP-SGD at ε = 1, and sometimes better quality than models trained directly on the real data.
Critical strengths
S1. A practical idea. Making private fake data using only a public model, with no training on private data, is attractive for hospitals and other regulated settings. It is also cheap to run.
S2. A broad evaluation. Five domains, public models that match the private data to different degrees, and four kinds of downstream tasks are more than most papers in this area test.
Critical weaknesses
W1. The privacy proof's own assumption is broken. Theorem 2 assumes the "synthetic population sizes" are "fixed independently of the private dataset." But Appendix F.3 sets the number of fake samples to sf × |S_priv,train|, meaning it depends on the size of the private dataset. So the theorem doesn't cover the method as actually run.
W2. Private data is used outside the privacy budget.
- Appendix F.2 says the model settings for traffic, activity and crypto were chosen using "a 6-iteration PE audit on the activity training split." The activity private set is UCI-HAR, and the same sentence explicitly calls the ECG audit "public," so the activity audit appears to use private data.
- Appendix F.3 says the ECG settings were retuned "by the C-FID of PETS runs at ε = 1," which means picking settings based on quality scores computed on the private data.
- Data scaling uses a "shared public+private percentile clip and min–max scaling" (Section 4), which also reads private data.
None of these uses is counted in the privacy budget, so the real ε is higher than reported.
W3. The classification headline looks broken. The paper says the classification test uses each method's generated labels. Yet every baseline that generates samples without labels (dp-GAN, GS-WGAN, Pub2Priv, TimeDiff, TimeGAN) scores exactly 20.6% at every privacy level, including the non-private ones. That looks like these methods have no meaningful labels, so the classifier always predicts one class. If so, the "3.4× better than DP-SGD" claim is a comparison against a broken setup.
W4. Results are cherry-picked. The paper reports the best round for each metric and the best of three variants for each cell. Using the final output (what a user would actually get), MMD wins drop from 12/12 to 7/12, and wins in the mismatched-domain settings drop from 6/7 to 3/7. Choosing the best round using private-data scores is also another privacy leak (see W2).
W5. The method is graded with the tool it optimizes. PETS picks samples by similarity in the TS2Vec embedding space, and its main quality score (C-FID) is computed in that same space, so it naturally looks good there. This likely explains why PETS beats non-private models trained on the real data by up to 6.3× in C-FID.
Questions for authors
- How is the number of fake samples chosen without knowing the private dataset size? Please restate Theorem 2 for the method as actually run, or rerun with a fixed number.
- Was the "activity training split" in F.2 private UCI-HAR data? How do you account for tuning on private data (F.2, F.3), the shared public+private scaling, and picking the best round per metric?
- How do label-free methods get labels for the classification test? Why do they all score exactly 20.6%, even without privacy?
- Can you report all results on the final output only, with one fixed variant chosen in advance, and C-FID in a different embedding than the one used for voting?
- Why is GS-WGAN's C-FID the same at every privacy level?