- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work proposed a method for generating a seemingly meaningless dataset that causes models trained on it to exhibit some pre-specified behavior, without any mention of the latter in the dataset.
Critical strengths
1、 The paper is honest about how hard it is and backs up its method with good baselines and lots of ablations (diversity constraint, best vs. last checkpoint, etc.). 2、The idea is broad and clean: it treats generating synthetic data as an optimization problem and shows that several earlier tricks (leave-one-out rewards, distillation, and reinforcement learning) are really just different ways to train this one optimizer.
Critical weaknesses
- The question is interesting, but the value of this work and the relationship between the central question and your methods are unclear.
- The main text narrates several discarded attempts yet establishes only qualitatively ("overfitting," "cold-start," "reward noise") why each fails; adding a quantitative diagnostic for each, or moving them to an appendix, would make the final design's justification convincing.
Questions for authors
N/A
5 comments
Continuing on Weakness 2
As a follow-up, we have run extensive additional experiments that quantitatively motivate our final attempt and validate the overfitting claim. Under the same settings, we evaluate our final attempt (WARMUP+PREFIX) and the non-reverting attempt (NO-REVERT) on two measures: the success rate of the best data generator checkpoint across training, and that of the final checkpoint.
| Method | Checkpoint | Animal preference (owl) | Evil persona | Complex fact | Animal (digit string) |
|---|---|---|---|---|---|
| NO-REVERT | |||||
| MG+RL | Best | 98.8 ± 0.6 | 75.0 ± 12.5 | 35.7 ± 2.8 | 2.0 ± 0.9 |
| Last | 98.8 ± 0.6 | 9.0 ± 1.3 | 32.1 ± 1.0 | 0.3 ± 0.1 | |
| MG+KD | Best | 69.3 ± 29.1 | 87.4 ± 9.4 | 72.0 ± 8.6 | 37.9 ± 26.6 |
| Last | 37.2 ± 31.4 | 43.5 ± 28.2 | 69.3 ± 9.5 | 0.3 ± 0.1 | |
| WARMUP+PREFIX | |||||
| MG+RL | Best | 98.5 ± 1.3 | 83.9 ± 1.6 | 58.1 ± 0.8 | 13.8 ± 12.8 |
| Last | 98.5 ± 1.3 | 82.2 ± 1.6 | 43.5 ± 4.8 | 13.4 ± 13.0 | |
| MG+KD | Best | 99.3 ± 0.3 | 94.8 ± 2.1 | 85.5 ± 4.1 | 99.7 ± 0.2 |
| Last | 98.2 ± 0.6 | 94.8 ± 2.1 | 85.1 ± 4.2 | 98.8 ± 0.5 |
In terms of the best checkpoints, the warmup+prefix attempt outperforms the no-revert attempt with both MG+RL and MG+KD. This quantitatively supports the advantage of warmup+prefix over no-revert. Furthermore, there is almost no gap between the best and last checkpoint success rates for the warmup+prefix attempt, while the no-revert attempt typically exhibits a large gap between the best and last checkpoints. This result validates the “overfitting” claim we made for Attempt #1 in Section 4.
Thank you for your time and effort in reviewing our paper! As the discussion period is ending soon, we would greatly appreciate any updated comments or questions about our rebuttal.
We thank the reviewer for their positive review and constructive feedback, including positive comments on the thoroughness of our experimental design and presentation. We address your concerns below. Please do not hesitate to comment if there are additional questions we can answer.
Weakness 1: the value of this work and the relationship between the central question and your methods are unclear.
Our central question investigates the “limits of synthetic data’s ability to steer model behavior.” Our proposed algorithms address this question by optimizing synthetic data generation against its effect on downstream student behavior. Our experiments allows us therefore to evaluate whether the performance of previous methods reflect limits of synthetic data in general or specific limits of these prior methods.
Across subliminal learning targets of varying difficulties (settings 1-4), our metagradient methods achieve significantly higher success rates than existing baselines, including subliminal learning, subliminal steering vectors, vanilla RL (Figure 1). As described in contribution #3 of the introduction, these results show that prior work substantially underestimate to what extent synthetic data can steer model behavior.
Regarding the value of our contributions: we introduce methods for steering with synthetic data that far exceed those of prior work (Figure 1: up to 5x higher success rates on the evil persona and complex facts tasks), highlight the value of our empirical contributions to both answering our central question and for advancing subliminal learning research in general. Given the significant threat of subliminal poisoning in AI systems, we argue that our work also has strong implications for AI safety. Finally, we hope our detailed experiments, including ablations on how various design choices affect the performance of our metagradient methods, can provide guidance for future work on both synthetic data generation and metagradient optimization.
Weakness 2: reasons for discarding attempts are only established qualitatively
In section 5.3, we provide quantitative evidence of the issues of attempts #1 (overfitting) and attempt #2 (cold-start). Specifically, Figure 2 compares our three approaches on animal preference with digit strings, empirically showing overfitting in attempt #1, the cold-start problem in attempt #2, and how attempt #3 addresses both.
Attempt #1 (overfitting): In Figure 2a (left-most panel), MG+KD achieves nearly 100% success rate in-training (blue, dotted line), but out-of-training success only rises to about 40% before falling back to 0% (blue, solid line) . Similarly, MG+RL reaches in-training success of about 40% (red dotted line), but out-of-training stays at 0% during the entire run (red, solid line). This generalization gap provides quantitative evidence of the overfitting that can occur when using the persistent-student approach (attempt #1).
Attempt #2 (cold start / uninformative reward signal): Figure 2b (middle panel) shows that resetting the student results in in-training and out-of-training success rates that stay at 0% during the entire run. In contrast, in-training success increases for attempt #1. These results are consistent with the cold-start problem described in the paper. The student being reset is not trained for enough steps to acquire the target behavior, therefore limiting the amount of signal provided back to the teacher. In turn, the teacher does not learn to produce samples that induce the target behavior, resulting in no progress.