- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces dataset-property leakage problem in aggregated demonstration task conditions of WAM. Experiments show that averaging demonstration features suppresses individual trajectory noise while amplifying broader collection-level distribution. This effect could enable adaptive adversaries to reconstruct private training-set compositions potentially.
Critical strengths
-
Introduces a novel privacy attack problem in vector from released robotic task representations, showing that task conditions could cause an dataset properties leakage.
-
Experiment shows that signal amplification under aggregation and demonstrates empirical vulnerabilities against adaptive, retrained attackers across multiple estimators across robot datasets.
-
A differential privacy guarantee with extensive closed-loop robot rollouts and matched-state causal interventions, verifying that the protected condition retains functional task control.
Critical weaknesses
-
While the paper conducts offline representation real-robot trajectories from BridgeData V2, the closed-loop policy rollouts are executed in simulation environments.
-
Evaluating closed-loop execution on a physical manipulator could better understand such a property leakage problem.
Questions for authors
Overall, I think this is a good paper. But I have some questions that I need the author to answer.
-
Robustness to Reference Anchor Quality: The public semantic ball relies on a reference anchor estimated from public data. How sensitive is policy success if there is substantial visual or domain mismatch between the public reference split and the target deployment environment?
-
Threat Model for Closed-Loop Action Observers: Table 12 demonstrates that while OpenVLA's fused prefix exposes the collection environment (AUC 0.977), the predicted action output drops to near-chance (AUC 0.535). Does this imply that black-box observers monitoring only executed robot joint actions cannot reconstruct collection composition unless intermediate states are exposed?
-
What is the runtime latency introduced by per-demonstration clipping, calibrated noise injection, and radial projection during online condition compilation?
I will further adjust my rating based on the author's response.
3 comments
Reply to Reviewer v3vQ [2/2]
Q1 — Substantial reference mismatch
We tested smaller references, nominal-only and randomized-only references, and half/double projection radii while keeping protected target collections fixed. The privacy grid additionally includes a different-task reference.
| Reference/radius settings | Tasks | Split seeds | Strongest pooled mean AUC range | Largest seed/attacker AUC |
|---|---|---|---|---|
| 9 | 4 | 3 | 0.519–0.522 | 0.553 |
The range summarizes the strongest attacker-family mean AUC in each setting, equally averaging tasks and seeds. The final column retains the largest individual task/seed/attacker value.
New control experiment. We vary the reference or radius while preserving the protected target collections, policy weights, and paired initial states.
| Tasks | Mechanism seeds | Paired simulator seeds | Episodes / setting | New full-horizon episodes |
|---|---|---|---|---|
| 4 | 3 | 10 | 120 | 720 |
Changes below are relative to the original private release. Intervals use crossed-cluster bootstrap resampling.
| Reference / radius | Success (%) | Change and paired 95% interval (pp) |
|---|---|---|
| Original private release | 86.67 | Reference |
| 20 reference episodes | 85.00 | -1.67 [-10.00, +5.00] |
| 80 reference episodes | 85.00 | -1.67 [-8.33, +5.83] |
| Nominal-only reference | 88.33 | +1.67 [-10.00, +14.17] |
| Randomized-only reference | 85.83 | -0.83 [-6.67, +5.00] |
| Half projection radius | 82.50 | -4.17 [-12.50, +2.50] |
| Double projection radius | 84.17 | -2.50 [-9.17, +3.33] |
All intervals include zero; this does not establish equivalence or non-inferiority. The domain-only settings also change reference size.
Equal-size reference comparison on real data. We use equally sized references drawn from both collection environments, non-kitchen only, or toy-kitchen only. Protected targets, data roles, and sampling seeds are unchanged; release parameters are recalibrated and attackers retrained.
| Ref. traj. / task | Target | Seeds | Retrained attackers | ||
|---|---|---|---|---|---|
| 20 | 20 | 1 | 3 | 5 |
| Real-data task | Balanced reference | Non-kitchen only | Toy-kitchen only | Largest seed/attacker AUC |
|---|---|---|---|---|
| Drawer PnP | 0.516 | 0.517 | 0.521 | 0.537 |
| Many skills | 0.511 | 0.509 | 0.510 | 0.535 |
| Stack blocks | 0.524 | 0.518 | 0.513 | 0.541 |
These comparisons measure reference sensitivity within the evaluated tasks and collection environments. They do not establish control robustness to arbitrary changes in camera, lighting, embodiment, or task. DP holds for fixed target-independent parameters, but does not itself guarantee that an unsuitable reference preserves task execution.
Q2 — Does weak action-output leakage imply action-only privacy?
No. Appendix H, Table 12 and Fig. 7 show that the observed leakage depends on the exposed interface.
| Exposed summary | Mean task AUC | Minimum task AUC |
|---|---|---|
| Fused prefix | 0.977 | 0.961 |
| Predicted action | 0.535 | 0.505 |
The predicted-action result concerns the evaluated summary. It does not cover complete executed joint-action histories, longer state-action sequences, repeated queries, or adaptive interactions.
Q3 — Condition-compilation overhead
We added a CPU microbenchmark of clipping/averaging, noise generation, projection, and complete compilation, using precomputed synthetic features.
| Measurement setting | Value |
|---|---|
| Input | Synthetic precomputed feature arrays; |
| Implementation | float64 NumPy 1.26.4 |
| CPU / threads | Intel(R) Xeon(R) Gold 6438Y+ / 1 |
| Warm-up calls per operation | 200 |
| Measured calls per operation | 2,000 |
| Timing unit | ms |
| Aggregate size | Clipping + mean | Gaussian noise | Projection | Total median / |
|---|---|---|---|---|
| 20 | 0.899 | 0.179 | 0.107 | 1.218 / 1.247 |
| 40 | 1.607 | 0.180 | 0.107 | 1.954 / 1.998 |
| 100 | 4.098 | 0.180 | 0.108 | 4.400 / 4.437 |
This cost is paid once per released aggregate, rather than at every robot action. Clipping/averaging costs , and noise generation/projection . The microbenchmark uses fixed synthetic and Gaussian-noise parameters and excludes visual encoding, reference calibration, policy inference, and network transfer. It measures release arithmetic, without claiming end-to-end robot latency or a hard real-time guarantee.
Ref:
[1] Gaussian Differential Privacy. Journal of the Royal Statistical Society: Series B (Statistical Methodology).
[2] The Algorithmic Foundations of Differential Privacy. Foundations and trends in theoretical computer science.
Thank you again for your valuable feedback. Please let us know if any concerns remain.
Thank you for the authors' response. I believe it has addressed my concerns well. After also considering the other reviewers' comments, I have decided to raise my score.
Reply to Reviewer v3vQ [1/2]
Thank you for your time in reviewing our paper and for your constructive suggestions. We have carefully revised the manuscript and addressed all the raised concerns in our point-by-point response below.
W1 / W2 — Evaluation on real robots
The original Section 5.4, Fig. 7, and Appendix H, Table 12 use BridgeData V2 trajectories collected by real robots. The fused prefix exposes collection information more strongly than the predicted-action summary; the label and session controls do not reproduce that signal. The original results are summarized below. We have now added protection testing on the same real-observation features.
New real-data protection experiments. We apply the release to analysis aggregates of OpenVLA fused-prefix states extracted from real-robot BridgeData V2 trajectories. Reference, attacker-training, and evaluation roles are disjoint by collection session. All comparisons use the same target aggregates and matched raw baseline.
The Raw and reference columns report the strongest three-seed mean AUC across attackers. Reference counts are per task; every reference column is a protected release. The final column reports the largest individual protected seed/attacker AUC across all reference choices. Balanced references draw equally from both environments. Reserving reference data reduces the original attacker-training pool, so the raw comparator is recomputed on the matched partition. These results support leakage reduction in real observations across the tested references. The aggregate is an offline analysis adapter, rather than a condition executed by OpenVLA's native controller.
This establishes that the measured protection extends to real robot observations. Physical closed-loop execution remains untested. We distinguish these two forms of evidence rather than equating offline real-data testing with operating a physical manipulator.