- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper argues that strategies for improving training efficiency come with security costs. It compares Shapley-based data selection and pre-train-then-finetune against utility-matched baselines on small vision benchmarks, and zero-RL math LLMs against an instruct model, measuring data privacy, model robustness and resilience, and proprietary security. The efficient variants are generally more vulnerable, which the authors relate to sharper loss landscapes, worse calibration and more compressed representations.
Critical strengths
- The question is timely and relevant, and the study covers several training stages, two domains and a wide range of attacks.
- Matching clean accuracy between the efficient and baseline models is a reasonable way to separate security effects from utility. The paper also goes beyond reporting attack numbers by looking at loss sharpness, calibration and representation structure, which gives some intuition for the observed trends.
Critical weaknesses
- The claim of being the "first systematic study" feels too strong given the scale of the evaluation types and experiments. The vision experiments use ResNet-18/Swin-Tiny on small subsets of CIFAR-10, TinyImageNet and PubFig83, with about 36-48% test accuracy and nearly 100% training accuracy, which is far from foundation-model training. The LLM part compares only three existing checkpoints. The paper frames its contribution around foundation-model development, but the evidence is far from that setting.
- Several measurements are not convincing. Many membership-inference TPRs at 0.1% FPR are around 0.1%. The models are standard non-robust ones, so under ε = 8/255 PGD they break within a few steps anyway (0.11-3.85 correctly classified iterates out of 40, apart from TinyImageNet), which makes the differences hard to interpret. For the LLMs, all models are already easy to jailbreak (91.9% ASR for the instruct baseline), and the comparison does not separate the effect of zero-RL from other differences in data and training pipeline. The MIA conclusion also depends on which attack is chosen.
- The claimed theoretical intuition (App. A) consists of post-hoc interpretations based on existing results, without formal links between efficiency choices and vulnerability or interventions to test them.
Questions for authors
Address the weakness issues raised in Critical weaknesses session.
6 comments
W2a — "many membership-inference TPRs at 0.1% FPR are around 0.1%"
The quantity our claim rests on is the relative leakage between utility-matched pairs, and this ratio is far from random even where absolute values are small. At 0.1% FPR, random guessing gives 0.1% TPR. The High vs. Low values are 1.51% vs. 0.32% on PubFig83 (15x vs. 3x random), 0.29% vs. 0.08% on CIFAR-10 with Swin, 0.22% vs. 0.08% on TinyImageNet, and 0.16% vs. 0.12% on CIFAR-10, with seed standard deviations of 0.01 to 0.06 (Appendix Table 7), so the orderings are consistent and well separated. We additionally evaluated a logit-scaled confidence attack of the same family in our re-evaluation pipeline, and it preserves the ordering in every setting: 0.35% (High) vs. 0.17% (Low) on the paper's CIFAR-10 models, 0.34% vs. 0.14% across three retraining seeds, and 1.69% vs. 0.38% on PubFig83. The ordering our conclusion rests on is therefore stable across attacks of this family, datasets, and retraining.
W2b — "under ε = 8/255 PGD they break within a few steps anyway ... which makes the differences hard to interpret"
We agree that white-box PGD at ε=8/255 saturates for these standard models. In our re-evaluation, robust accuracy at this ε is near zero for both compared models, as expected for models without adversarial training, so differences at this operating point are hard to interpret. We therefore evaluate robustness under a black-box threat model, which is non-saturated and matches how a deployed model is actually exposed, through query access only. The attacker first extracts a surrogate with the paper's model-stealing protocol (1,000 disjoint queries), then crafts 40-step PGD examples on the surrogate, and submits them to the target. We report the target's robust accuracy, the fraction of test images it still classifies correctly under these transferred examples (mean over 3 seeds, the two models at matched clean accuracy):
| Robust accuracy under black-box attack | ε=2/255 | ε=4/255 | ε=8/255 |
|---|---|---|---|
| High | 33.6 | 20.3 | 6.4 |
| Low | 38.4 | 25.9 | 10.1 |
The efficient High-Shapley model is consistently more vulnerable, losing 4 to 6 points more robust accuracy at every ε, with the same ordering on the paper's original models. The mechanism is the extraction result itself: the surrogate replicates the High model more faithfully (agreement 0.85 vs. 0.58), so adversarial examples cross both decision boundaries together more often, while the Low baseline is protected mainly by being harder to copy. The representation simplicity that makes the efficient model easier to steal therefore also makes it easier to evade without gradient access, which connects the robustness and extraction dimensions of our study.
W2c — "all models are already easy to jailbreak (91.9% ASR for the instruct baseline)"
We agree that none of the evaluated models is safe in absolute terms. They are math-specialized models without dedicated safety training, so a high eventual attack success rate is expected for all of them. The quantity that still separates the models is attack cost. AutoDAN needs on average 4.32 search iterations to jailbreak the Instruct model but only 0.53 (SimpleRL-Zoo) and 0.41 (PRIME), where a value below 1 means most prompts already succeed within the initial population. The zero-RL models therefore require 8 to 10 times fewer search iterations. This gap is if anything understated, because the averages count only successful prompts and the Instruct model's failed prompts are excluded from its average, and the ASR ordering is consistent as well (100% and 99% vs. 91.9%). In deployment, attack cost determines how many queries a filter or rate limiter can intercept before the attack succeeds, so it is the security-relevant measurement in this regime.
W2d — "the comparison does not separate the effect of zero-RL from other differences in data and training pipeline"
We agree that the LLM comparison is between complete alignment pipelines within the same model family, and we will make this explicit in the revision. First, utility is in fact closely comparable: the three models differ by at most 2.4 points in the seven-benchmark average (Table 5), while the security gaps are far larger, for example a roughly tenfold difference in the attack steps AutoDAN needs and a far larger accuracy drop after identical fine-tuning (Tables 4 and 5). A utility gap this small cannot explain security gaps of that size. Second, as stated in Sec. 3.1, our claim is at the pipeline level: the zero-RL pipelines as released carry a security cost, and we do not attribute this cost to any single omitted component. This matches how these recipes are used in practice, since practitioners adopt a released pipeline as a whole. Accordingly, Sec. 6 discusses the missing KL regularization only as a plausible explanation, not as an established cause. Third, SimpleRL-Zoo and PRIME were developed independently, with different training data, reward designs, and implementations. A factor specific to one pipeline would not affect the other, yet both show the same vulnerability pattern relative to Instruct across all three evaluations. Their shared efficiency-oriented alignment recipe is therefore the most plausible source of the vulnerability.
W2e — "the MIA conclusion also depends on which attack is chosen"
The two attacks measure different leakage channels, and the difference between them has a concrete mechanical cause rather than being a free choice. The answer-likelihood attack scores membership by the probability the model assigns to the static reference solution text. After reward optimization, a zero-RL model solves training problems with its own reasoning paths and phrasings, so its probability on the reference text drops even for problems it has memorized through training. This attack therefore structurally understates leakage for RL-trained models, and comparing an RL model to an SFT-trained Instruct model on it is biased in the Instruct model's direction. The self-confidence attack avoids this by scoring the model's own generated solution, and there the zero-RL models leak more. We note this is not explained by generic overconfidence either, because uniform overconfidence shifts member and non-member scores together, while the attack measures the separation between the two groups. The Min-k% results (Appendix) agree with this picture. We will revise Sec. 6 to state the methodological point explicitly: reference-text attacks are not comparable across training paradigms, so MIA conclusions for RL-trained models should rest on generation-based attacks and on the suite jointly.
W3 — "post-hoc interpretations based on existing results, without formal links ... or interventions to test them"
We agree on both counts. We will no longer label Appendix A as theory: the contribution will be renamed "mechanistic analysis," and its interpretations will be marked as hypotheses grounded in prior results. In addition, we ran the interventional test the reviewer asks for: retraining the High-Shapley model with Sharpness-Aware Minimization, with everything else unchanged, reduces sharpness at matched accuracy and lowers MIA [email protected]%FPR from 0.30% to 0.18%, reaching the Low-baseline level (0.17%), while model stealing and feature dimensionality remain unchanged. The intervention thus supports the sharpness link for privacy and calibration and separates it from the representation-driven extraction risk. The full table and setup are in our response to Reviewer 1 (W3).
Thank you for the careful review.
W1a — "the claim of being the 'first systematic study' feels too strong"
Our intended claim is that this is the first study to connect the efficiency and security question across training stages and domains under matched utility. Prior work examines individual mechanisms in isolation, for example data selection and privacy, or compression and robustness, and Sec. 2.2 reviews why these fragmented results do not compose into a picture of the training pipeline. The systematic part of our contribution is the common protocol that spans the pipeline: the same utility-matched evaluation is applied to data selection, pre-training, and alignment, in both vision and language, and probes each model's privacy, robustness, extractability, jailbreak resistance, and adaptation stability. To our knowledge, no prior work provides this combination. We will make the wording precise in the revision, claiming the first cross-stage and cross-domain analysis at matched utility, and add a Limitations paragraph clarifying the scope: the vision experiments are controlled studies designed for exact utility matching, the LLM analysis covers publicly released models at two scales (7B and 32B), and extending the study to frontier-scale training is left as an open direction.
W1b — "about 36-48% test accuracy and nearly 100% training accuracy, which is far from foundation-model training"
The low accuracy is a consequence of the matching protocol: disjoint Shapley subsets are calibrated until training and test accuracy coincide, which caps the subset size. Within each pair the two models therefore have identical generalization gaps, so the vulnerability differences cannot come from different degrees of overfitting. The appendix also covers a high-accuracy regime: full-data training (94.8% accuracy) vs. High-Shapley subsets of 40,000 (93.4%) and 20,000 (88.5%) shows the same monotone increase in leakage (0.02%, 0.04%, 0.07% MIA TPR, Appendix Table 9). On the framing: the studied strategies come from foundation-model practice, but isolating their security effect requires controlled training, so our evidence is two-level by design, controlled vision studies plus released 7B and 32B models showing the same pattern.
W1c — "the LLM part compares only three existing checkpoints"
The number of models follows from the same control that governs the vision experiments: compared models must reach comparable clean utility, otherwise security differences are confounded by capability. We evaluated a broader set of publicly released aligned math models as candidates, including zero-RL models built on other base families and at other capability levels, and only these three satisfy the requirement, with a baseline gap of at most 2.4 points on the seven-benchmark average (Table 5). The comparison is small because the inclusion criterion is strict, not because the evaluation is. Within the three, the two efficient pipelines were developed independently with different data and reward designs, so the shared vulnerability pattern is a replication across pipelines rather than a single observation, and the appendix repeats the comparison at 32B with the same result.