- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces a pipeline that automatically discovers datasets causing surprising generalization, which the authors call DSSEs (datasets with surprising side effects). They produce 6,575 of them with a two-stage pipeline fed by four generation methods (DE, DPG, PT, LLS). A DSSE is defined by three criteria: elicitation of a target behavior after finetuning, naturalness of the data, and surprisingness of the side effect to an auditor. Only the evolutionary search (DE) optimizes all three jointly, through a fitness that combines elicitation with a soft minimum over LLM judges for naturalness and surprisingness; DPG optimizes a contrastive elicitation surrogate alone, and PT/LLS use the other two criteria as filters rather than objectives. A candidate is counted as a DSSE as soon as its elicitation exceeds the untuned model’s base rate, with no threshold on naturalness or surprisingness.
Stage 1 searches against cheap proxy judges (gemma4-31b) and stage 2 re-scores with held-out frontier judges (claude-opus-4.8, gpt-5.6), which limits Goodharting of the proxies. DE initializes its population with a mix of specs on ordinary unrelated topics and specs written by an LLM to elicit the target behavior. PT starts from a standard SFT dataset and regenerates completions with a teacher that holds the behavior in its system prompt.
Critical strengths
- Valuable research question. Prior work has produced DSSEs in the dozens, and a larger collection is the right thing to build if theories of generalization are to be tested at scale.
- Introduces a way to generate DSSEs automatically, with an array of four methods rather than a single recipe.
- Evaluates transfer to other models: two families (Llama-3.2, Qwen3) and sizes from 1B to 14B, i.e. up to 8x the 1.7B model used in the search loop.
- Compares LLM auditors against human auditors on the same datasets, which is what makes the surprisingness metric credible rather than circular.
- Reports a diversity check (Table 7) rather than assuming the datasets are distinct, and flags PT-noterse as the exception.
Critical weaknesses
The headline count of 6,575 DSSEs is an overclaim. It counts every dataset whose side effect is statistically above base rate, with no bar on naturalness or surprisingness, which are two of the paper’s own three defining criteria. Applying them: 2,443 datasets reach multiple-choice surprisingness of 2/3 or 3/3 (Figure 13c), and of those roughly 1,400 also transfer to all five held-out models (64% and 53% respectively). A naturalness bar at the Tulu-3 level would cut further, plausibly to the low hundreds. The paper also notes that elicitation rates are “often modest in absolute terms”, and Figure 11 shows a large share of the collection sitting just above the floor. The count that matters for the stated purpose, DSSEs comparable to the published ones, is likely an order of magnitude below 6,575. I would rather see the collection released with explicit tiers than with a single headline number.
No reference DSSEs are scored on all three criteria. Only naturalness is reported for the insecure-code data of Betley et al. (2025b) and for Tulu-3, as dashed lines in Figures 4 and 12. Without the surprisingness and elicitation of the known DSSEs on the same scales, a reader cannot tell whether the generated datasets reach the level of the examples that motivated the paper, which is exactly the comparison the introduction sets up.
Questions for authors
- Would you consider reporting the collection in tiers rather than as 6,575? E.g. a top tier filtered on surprisingness, transfer, and naturalness at the level of the published DSSEs. The permissive count is defensible as a release, but it reads as the paper’s main result.
- The elicitation floor is the untuned model’s base rate, so it controls for the model but not for finetuning itself. Did you measure the behavior rate after finetuning on a matched control dataset (e.g. a length-matched Tulu-3 slice) with the same recipe? Some of the increase may come from LoRA SFT on any data rather than from the dataset’s content.
- [The most important] Could you place the published DSSEs on both axes of Figure 4 (and Figure 12), not just the naturalness axis? Currently, the insecure-code baseline appears only as a horizontal line, so there is no way to position prior work in the surprisingness and elicitation directions.
- Could you add a 2D embedding (t-SNE/UMAP) of the datasets, or diversity metrics beyond near-duplicate row counts? Counting near-identical rows seems too weak a test: two datasets can share a topic, a persona, and a register without sharing a single row.
- Table 7 is pairwise. Within one method-behavior group (e.g. cutting_instructions from DE), what fraction of a dataset’s rows have a near-duplicate in the union of all other datasets in that group? For PT-noterse this looks severe: the acting_distressed and threatening_user pools hold 328 and 251 surviving answers, and each dataset samples 150 rows from them, so two datasets drawn from the 251-answer pool must share about 60% of their rows by construction. These datasets are near-copies, not 256 distinct DSSEs.
- All three multiple-choice auditors miss about 20% of the collection, and the open-ended auditor gets no_match on 19%. Is this subset not the most valuable part of the release? It would be worth characterizing separately: which methods and behaviors it comes from, and whether its elicitation and transfer differ from the rest.
- A further criterion for a useful DSSE: the dataset should not degrade instruction-following or general capability when the side effect has no reason to touch them. Did you measure anything like this? A dataset that induces the behavior by making the model broadly worse (the DPG example in Figure 5, whose responses sometimes do not answer the prompt, looks like a candidate) is less interesting as a test case for side-effect prediction.
- For transfer, more families would be more informative than more sizes within two families, and the largest target is 14B. Do you have results on a 32B+ model, or on a third family?
- Only 53% of the most surprising datasets transfer to all five models, against 84% for the least surprising ones. Transfer looks like a natural additional tier for the top DSSEs. Do you have a view on whether the surprisingness-transfer tradeoff is intrinsic, or an artifact of the auditors finding the non-transferring datasets hard for the same reason they are narrow?
- Did you try more than 8 behaviors? Given that the per-behavior yield is in the hundreds, 6,575 datasets over 100 behaviors would be a more useful benchmark for side-effect prediction than 6,575 over 8, since the multiple-choice format is only as hard as its option set. What limited the number of behaviors: the cost of writing the evals and rubrics, or something else?
Dear reviewers,
We’re about halfway through the discussion period, and the authors have put considerable time into responding to your feedback. Please read their rebuttal and leave a brief comment letting the authors know whether it addressed your questions or concerns and what, if anything, remains open.
Thank you.