acceptodds
Under review as a conference paper at ICLR 2027

Mass Producing Training Data with Surprising Side Effects

18 views

Abstract

Training data can have unexpected side effects on language model behavior outside the training domain: insecure code data can induce broad misalignment, and outdated bird names can induce a 19th-century persona. Understanding such generalization is hard because only a handful of isolated examples exist, each constructed by hand or discovered by accident. We propose automatically discovering such datasets with surprising side effects (DSSEs). We define three objectives for DSSEs: they should (1) elicit a target behavior, (2) read like ordinary post-training data, and (3) be surprising to human auditors reading the data. We operationalize each with LLM judges, and search for high-scoring datasets via evolution, reinforcement learning, and subliminal learning approaches. After re-scoring and filtering with held-out metrics, our pipeline produces a collection of 6,575 DSSEs across 8 behaviors, many of which generalize across model families and sizes. We find our automated metrics track ML researchers' judgments, and frontier models outperform humans at predicting side effects. These datasets form a benchmark for studying generalization and lay the groundwork for predicting training side effects before model deployment.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM generalization, finetuning side effects, alignment, safety

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 4.00 over 3 reviews, above 99% of the others.
Reviewer_ZHeiOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces a pipeline that automatically discovers datasets causing surprising generalization, which the authors call DSSEs (datasets with surprising side effects). They produce 6,575 of them with a two-stage pipeline fed by four generation methods (DE, DPG, PT, LLS). A DSSE is defined by three criteria: elicitation of a target behavior after finetuning, naturalness of the data, and surprisingness of the side effect to an auditor. Only the evolutionary search (DE) optimizes all three jointly, through a fitness that combines elicitation with a soft minimum over LLM judges for naturalness and surprisingness; DPG optimizes a contrastive elicitation surrogate alone, and PT/LLS use the other two criteria as filters rather than objectives. A candidate is counted as a DSSE as soon as its elicitation exceeds the untuned model’s base rate, with no threshold on naturalness or surprisingness.

Stage 1 searches against cheap proxy judges (gemma4-31b) and stage 2 re-scores with held-out frontier judges (claude-opus-4.8, gpt-5.6), which limits Goodharting of the proxies. DE initializes its population with a mix of specs on ordinary unrelated topics and specs written by an LLM to elicit the target behavior. PT starts from a standard SFT dataset and regenerates completions with a teacher that holds the behavior in its system prompt.

Critical strengths
  • Valuable research question. Prior work has produced DSSEs in the dozens, and a larger collection is the right thing to build if theories of generalization are to be tested at scale.
  • Introduces a way to generate DSSEs automatically, with an array of four methods rather than a single recipe.
  • Evaluates transfer to other models: two families (Llama-3.2, Qwen3) and sizes from 1B to 14B, i.e. up to 8x the 1.7B model used in the search loop.
  • Compares LLM auditors against human auditors on the same datasets, which is what makes the surprisingness metric credible rather than circular.
  • Reports a diversity check (Table 7) rather than assuming the datasets are distinct, and flags PT-noterse as the exception.
Critical weaknesses

The headline count of 6,575 DSSEs is an overclaim. It counts every dataset whose side effect is statistically above base rate, with no bar on naturalness or surprisingness, which are two of the paper’s own three defining criteria. Applying them: 2,443 datasets reach multiple-choice surprisingness of 2/3 or 3/3 (Figure 13c), and of those roughly 1,400 also transfer to all five held-out models (64% and 53% respectively). A naturalness bar at the Tulu-3 level would cut further, plausibly to the low hundreds. The paper also notes that elicitation rates are “often modest in absolute terms”, and Figure 11 shows a large share of the collection sitting just above the floor. The count that matters for the stated purpose, DSSEs comparable to the published ones, is likely an order of magnitude below 6,575. I would rather see the collection released with explicit tiers than with a single headline number.

No reference DSSEs are scored on all three criteria. Only naturalness is reported for the insecure-code data of Betley et al. (2025b) and for Tulu-3, as dashed lines in Figures 4 and 12. Without the surprisingness and elicitation of the known DSSEs on the same scales, a reader cannot tell whether the generated datasets reach the level of the examples that motivated the paper, which is exactly the comparison the introduction sets up.

Questions for authors
  1. Would you consider reporting the collection in tiers rather than as 6,575? E.g. a top tier filtered on surprisingness, transfer, and naturalness at the level of the published DSSEs. The permissive count is defensible as a release, but it reads as the paper’s main result.
  2. The elicitation floor is the untuned model’s base rate, so it controls for the model but not for finetuning itself. Did you measure the behavior rate after finetuning on a matched control dataset (e.g. a length-matched Tulu-3 slice) with the same recipe? Some of the increase may come from LoRA SFT on any data rather than from the dataset’s content.
  3. [The most important] Could you place the published DSSEs on both axes of Figure 4 (and Figure 12), not just the naturalness axis? Currently, the insecure-code baseline appears only as a horizontal line, so there is no way to position prior work in the surprisingness and elicitation directions.
  4. Could you add a 2D embedding (t-SNE/UMAP) of the datasets, or diversity metrics beyond near-duplicate row counts? Counting near-identical rows seems too weak a test: two datasets can share a topic, a persona, and a register without sharing a single row.
  5. Table 7 is pairwise. Within one method-behavior group (e.g. cutting_instructions from DE), what fraction of a dataset’s rows have a near-duplicate in the union of all other datasets in that group? For PT-noterse this looks severe: the acting_distressed and threatening_user pools hold 328 and 251 surviving answers, and each dataset samples 150 rows from them, so two datasets drawn from the 251-answer pool must share about 60% of their rows by construction. These datasets are near-copies, not 256 distinct DSSEs.
  6. All three multiple-choice auditors miss about 20% of the collection, and the open-ended auditor gets no_match on 19%. Is this subset not the most valuable part of the release? It would be worth characterizing separately: which methods and behaviors it comes from, and whether its elicitation and transfer differ from the rest.
  7. A further criterion for a useful DSSE: the dataset should not degrade instruction-following or general capability when the side effect has no reason to touch them. Did you measure anything like this? A dataset that induces the behavior by making the model broadly worse (the DPG example in Figure 5, whose responses sometimes do not answer the prompt, looks like a candidate) is less interesting as a test case for side-effect prediction.
  8. For transfer, more families would be more informative than more sizes within two families, and the largest target is 14B. Do you have results on a 32B+ model, or on a third family?
  9. Only 53% of the most surprising datasets transfer to all five models, against 84% for the least surprising ones. Transfer looks like a natural additional tier for the top DSSEs. Do you have a view on whether the surprisingness-transfer tradeoff is intrinsic, or an artifact of the auditors finding the non-transferring datasets hard for the same reason they are narrow?
  10. Did you try more than 8 behaviors? Given that the per-behavior yield is in the hundreds, 6,575 datasets over 100 behaviors would be a more useful benchmark for side-effect prediction than 6,575 over 8, since the multiple-choice format is only as hard as its option set. What limited the number of behaviors: the cost of writing the evals and rubrics, or something else?
Reviewer_xsjDOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies a post-training phenomenon in which seemingly ordinary finetuning datasets induce out-of-domain behavioral changes that are difficult to anticipate from inspecting the training data alone. The authors formalize such datasets as Datasets with Surprising Side Effects (DSSEs), requiring three properties: elicitation of a target behavior after finetuning, naturalness as plausible post-training data, and surprisingness to an auditor who only inspects the dataset. They introduce a two-stage pipeline in which several automated search procedures generate candidate datasets, followed by evaluation using held-out judges that were not used during search. From 9,119 candidates, the pipeline retains 6,575 DSSEs spanning eight target behaviors and releases them as DSSE-1. The paper further studies cross-model transfer, human agreement with automated metrics, and the potential use of DSSE-1 as a benchmark for predicting training-data side effects.

Critical strengths
  1. The paper turns isolated surprising-training-effect examples into a scalable empirical object. Prior work has reported phenomena such as emergent misalignment, unexpected generalization, and subliminal learning, but often through a small number of examples. The automated DSSE pipeline makes it possible to study such effects systematically across thousands of datasets and multiple target behaviors.
  2. The elicitation–naturalness–surprisingness formulation is well motivated. Requiring only elicitation would make the task trivial, since the target behavior could simply be stated directly in the training data. Adding naturalness and surprisingness creates a more meaningful setting in which apparently normal post-training data can induce effects that are not obvious from surface inspection.
  3. The two-stage held-out evaluation is a strong design choice. Since LLM-based judges are part of the search objective, reward hacking or evaluator exploitation is a natural concern. Re-evaluating candidates with held-out judges that were not accessible during search reduces the likelihood that the final results are purely artifacts of in-loop Goodharting.
Critical weaknesses

1. The current setting is closer to target-conditioned construction than open-world side-effect discovery.

The target behavior, behavioral rubric, and evaluation prompts are known before dataset search begins. In contrast, a realistic auditing setting starts from an arbitrary dataset and asks what unknown side effects it may induce. The paper's current setup is valuable, but the distinction matters for interpreting the practical scope of the contribution.

2. DSSE-1 is proposed as a prediction benchmark, but its benchmark utility is not yet demonstrated through a standard train/test prediction experiment.

The human and frontier-LLM auditing experiments show that predicting the target behavior from the dataset alone is difficult but possible. However, the paper does not exploit the scale of DSSE-1 to train even a simple predictor on one subset and evaluate it on held-out DSSEs. This leaves open whether the collection actually supports generalizable side-effect prediction rather than merely serving as a large set of difficult synthetic examples. Stronger evaluation should include at least ordinary held-out datasets and, ideally, held-out generation methods, behaviors, source corpora, or target-model families.

3. There remains a substantial external-validity gap between synthetic DSSEs and real-world training-data auditing.

All retained datasets are synthetically generated and optimized toward known target behaviors. A predictor could therefore exploit generator-specific, source-specific, or behavior-specific shortcuts rather than learning a general relationship between training data and post-training behavioral effects. The paper already acknowledges that prediction performance on synthetic DSSEs may not transfer to in-the-wild cases. I therefore view DSSE-1 as a promising testbed, but not yet as a validated benchmark for real-world training-data auditing.

Questions for authors
  1. Can the authors report the full elicitation effect-size distribution for the 6,575 retained DSSEs, including the number with increases greater than 5, 10, and 20 percentage points?
  2. Among high-effect datasets, how many simultaneously satisfy strong naturalness, surprisingness, and cross-model transfer? A joint “high-quality DSSE” statistic would make the headline result much easier to interpret.
  3. Do the authors agree that the current experiments primarily establish DSSE constructibility / searchability rather than prevalence under natural post-training data distributions? If so, can claims about rarity or the size of the relevant data region be scoped accordingly?
  4. Since the target behavior, rubric, and evaluation prompts are known in advance, would “target-conditioned DSSE construction” be a more precise description than open-ended “discovery”?
  5. Since DSSE-1 is positioned as a side-effect prediction benchmark, can the authors train a simple predictor and report held-out performance, ideally including held-out generation-method and held-out-behavior splits?
  6. How would the authors distinguish a predictor that learns genuine training-effect relationships from one that exploits generation-pipeline artifacts, source-corpus cues, or behavior-specific stylistic shortcuts?
  7. What additional evidence would be required, in the authors' view, before performance on DSSE-1 could be interpreted as evidence for auditing naturally occurring post-training datasets?
Reviewer_YemHOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a workflow to generate datasets with surprising side effects (DSSEs). The DSSEs are datasets that can induce unexpected behavior when an LLM is fine-tuned on them. A system of approaches is designed to identify these datasets. First, a set of candidates of DSSE is identified through evolutionary search on specifications and dataset policy gradient to tune the LLM that generates the dataset based on the specifications. Then, the candidates are scored and curated based on surprisingness and naturalness. Numerically, the identified datasets show consistent side effects for 5 different models.

Critical strengths

The problem studied is very interesting and the results show that the DSSEs are transferrable across different model architectures and sizes for 5 small-scale models. The proposed approaches also reveal a pathway to find datasets that can elicit unwanted behaviors, which can have significant impact for the community. The descriptions of the experiment procedures are very detailed.

Critical weaknesses

None

Questions for authors

Currently, the diversity of the dataset is evaluated only after it is generated. Is it possible to incorporate diversity also as a dataset design objective? This could encourage the search to discover datasets that differ from those already collected, allowing the collection to expand incrementally in both size and diversity.

Comments on the paper

1 comment

Dear reviewers,

We’re about halfway through the discussion period, and the authors have put considerable time into responding to your feedback. Please read their rebuttal and leave a brief comment letting the authors know whether it addressed your questions or concerns and what, if anything, remains open.

Thank you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.