acceptodds
Under review as a conference paper at ICLR 2027

Books, Copyright, and the Future of Foundation Models: An Empirical Study of Fiction in Pretraining

20 views

Abstract

The use of books to train large language models has prompted numerous lawsuits against commercial AI providers, yet the reasons such data choices matter remain largely opaque outside industry labs. We ask a simple question: what role does fiction play during pretraining? To answer it, we pretrain 1B-parameter models from scratch under controlled token budgets on data mixes that include web text alone, web text with public-domain books, web text with contemporary fiction, and web text with both. Our results show that fiction substantially improves downstream model performance on creative writing tasks, with contemporary fiction yielding gains beyond those from public-domain corpora. We then examine what makes fictional text distinct from other pretraining sources, and examine whether synthetic data can recover its benefits. We find that models we trained on our synthetic data recipe achieve performance comparable to models trained on contemporary fiction, while making it harder to establish provenance through membership inference and extraction attacks. Together, these experiments offer the largest empirical study to date of copyrighted fiction in model pretraining, contributing evidence to debates over data governance, creators' rights, and how particular data domains shape AI capabilities.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
foundation models, books, pretraining, fiction

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 4.00 over 3 reviews, above 99% of the others.
Reviewer_RhCpOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the contribution of fiction to language-model pretraining through controlled experiments with 1B-parameter models trained on 185B tokens. It compares a web-only baseline with mixtures containing contemporary Books3 fiction, public-domain Institutional Books fiction, and synthetic fiction generated from Books3. Contemporary fiction improves automated creative-writing performance and substantially outperforms the selected public-domain corpus, while offering less consistent benefits for other capabilities. Synthetic fiction preserves or improves several writing scores while reducing the effectiveness of tested membership-inference and extraction attacks against the original books. The contribution is empirical evidence connecting fiction-source selection, downstream utility, and source detectability.

Critical strengths
  1. The controlled experiments establish useful differences between fiction sources. Unlike comparisons of unrelated pretrained models, the study holds major training choices fixed while changing the fiction component. Books3 achieves EQ-Bench win rates of 78.6 ± 3.9% against NoBooks and 91.1 ± 4.5% against Institutional Books.

  2. The paper jointly examines utility and detectability after synthetic transformation. Synthetic achieves a 79.7 ± 6.8% EQ-Bench win rate against Books3, while the tested attacks become less effective at detecting or reproducing the original text.

Critical weaknesses
  1. The strongest creative-writing conclusions lack adequate human validation. Table 9 reports a 57% human win rate for Books3 against NoBooks over 50 examples, with one annotation per example and no reported uncertainty. Human–judge agreement is 46% for both LitBench comparisons. Agreement in the direction of aggregate preferences provides some corroboration, but does not establish reliable measurement of the quality differences.

  2. More importantly, the human study does not evaluate Synthetic, despite its particularly large automated gains. This leaves uncertainty about whether its advantage translates into reader preference. A small blinded comparison using existing Synthetic and Books3 outputs would directly validate a central result without expanding the paper’s scope.

  3. The provenance conclusions extend beyond what the reported experiment establishes. Figure 3 demonstrates reduced detectability under four membership-inference methods and one extraction procedure. It does not establish that provenance auditing generally becomes ineffective. The original books are sources of synthetic-data generation rather than literal training documents, so this distinction is important when interpreting attack failure.

  4. §4.2 discusses results across context lengths without specifying the lengths or their aggregation in Figure 3a. Its expectation that the extraction gap will increase at larger scales is also untested.

Questions for authors
  1. Targeted validation: Can you provide a blinded human comparison of Synthetic versus Books3 using a modest sample of existing generations, with win/loss/tie counts and uncertainty estimates? This would strengthen the central utility claim.

  2. Auditing clarification and control: What context lengths and passage-to-document aggregation procedures produced Figure 3a, and how were the member/non-member books separated from synthetic-generation inputs? Reporting the same attacks on the existing NoBooks model would help assess whether the candidate sets are distinguishable without the added Books3 exposure.

  3. Training-data clarification: Appendix G.2 reports approximately 450,000 generated chapters totaling 6B tokens, implying about 13,333 tokens per chapter, whereas §4.1 targets 1,000–2,000 words. Could you reconcile these figures and clarify whether the final 5B synthetic training tokens contain repeated generations? This matters for interpreting the matched-budget comparison.

4 comments
AuthorsOpenReview ↗

Author Response to Reviewer RhCp

We are grateful that the reviewer appreciated how our controlled experimental setup could help establish differences in quality between sources of fiction. We are also glad the reviewer valued how our work brings light to both the utility of synthetic data and detectability of the original source material for that synthetic data. Below we address all the points in your review, and remain available to address any remaining questions you may have before the end of the response period.

 

Appendix J: Books3 against NoBooks LitBench evaluation

We note that the LitBench reward model is an established human-validated measure from prior work [1], rather than a new LLM-as-a-judge method we have constructed for this work. In [1], the reward model was validated on a held-out set of 2,480 human-labeled story comparisons and reached 78% agreement with human preferences. We use this reward model for our experiments, adding our own human annotations in the appendix only for completeness.

We add a second human annotator so that every example now has two independent annotations. We report inter-annotator agreement and 95% confidence intervals. On LitBench, the two annotators agree on 64% of examples. The human win rate for Books3 vs No Books is 57% for the first annotator and 61% for the second. The preference for Books3 is consistent across both annotators. The human and judge agreement is 46% and 47% for two annotators. We have updated our paper to reflect these results (Appendix J and Table 9).

[1] LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative Writing (Fein et al., EACL 2026)

 

Synthetic vs Books3 outputs judge validation

We use the LitBench reward model for these experiments as well, which is an established human-validated measure from prior work [1], rather than a new LLM-as-a-judge method we propose. In [1], the reward model was validated on a held-out set of 2,480 human-labeled story comparisons and reached 78% agreement with human preferences. We use this reward model for our experiments, adding our own human annotations in the appendix only for completeness. We additionally conduct a blind human comparison of Synthetic vs Book3 generations for 50 examples on LitBench, with two independent annotations per example. Across both annotators, Synthetic wins 35, Books3 wins 42, and 23 are ties. This gives the synthetic model a win rate of 46.5% (95% CI 36-58%). These results suggest the synthetic model is statistically comparable to Books3 in human reader preference. The revised manuscript reflects these results (Appendix J and Table 9).

[1] LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative Writing (Fein et al., EACL 2026).

 

The provenance conclusions extend beyond what the reported experiment establishes. Figure 3 demonstrates reduced detectability under four membership-inference methods and one extraction procedure. It does not establish that provenance auditing generally becomes ineffective. The original books are sources of synthetic-data generation rather than literal training documents, so this distinction is important when interpreting attack failure.

We show that it is possible to generate high-quality synthetic data by conditioning on Books3 that can give similar performance improvements to training on the original source material. We demonstrate that some of the most prominent membership inference and data extraction attacks available are not successful at detecting use of the original copyrighted material as a source for this synthetic data in the resulting model. We emphasize that the membership inference and data extraction attacks that we use in the paper represent some of the most widely-used attacks in the field. For membership inference: the loss-based [1], Min-K% [2], Min-K%++ [3], and zLib [4] attacks have been widely used to establish provenance for pre-training data [5, 6, 7]. For data extraction: the method that we use in the paper has been extensively used by prior works for extracting pre-training data [4, 8], including from production models [9] and for copyrighted books [10, 11].

[1] “Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting” (Yeom et al., IEEE CSF 2018).
[2] “Detecting Pretraining Data From Large Language Models” (Shi et al., ICLR 2024).
[3] “Improved baseline for detecting pre-training data from large language models” (Zhang et al., ICLR 2025).
[4] “Extracting Training Data From Large Language Models” (Carlini et al., USENIX 2021).
[5] “RECALL: Membership Inference via Relative Conditional Log-Likelihoods” (Xie et al., EMNLP 2024).
[6] “CON-RECALL: Detecting Pre-training Data in LLMs via Contrastive Decoding” (Wang et al., COLING 2025).
[7] “Hubble: A Model Suite to Advance the Study of LLM Memorization” (Wei et al., ICLR 2026).
[8] “Quantifying Memorization Across Neural Language Models” (Carlini et al., ICLR 2023)

AuthorsOpenReview ↗

Continuation of Author Response to Reviewer RhCp

[9] “Scalable Extraction of Training Data from (Production) Language Models” (Nasr et al., ICLR 2025).
[10] “Extracting memorized pieces of (copyrighted) books from open-weight language models” (Cooper et al., CoLM 2026).
[11] “Extracting books from production language models” (Cooper et al., 2026).

 

Auditing clarification and control: What context lengths and passage-to-document aggregation procedures produced Figure 3a, and how were the member/non-member books separated from synthetic-generation inputs? Reporting the same attacks on the existing NoBooks model would help assess whether the candidate sets are distinguishable without the added Books3 exposure.

We’ll try to answer each of the reviewer’s subquestions here one at a time:

how were the member/non-member books separated from synthetic-generation inputs

The reviewer's questions helped us identify a possible data leak in the nonmember set used specifically for evaluating the synthetic-data model where some candidate nonmember books may have served as conditioning inputs during synthetic-story generation. This possibility does not affect the Books3 or NoBooks evaluations. To eliminate it, we replaced the nonmember set with external BookMIA nonmembers and reran all attacks (across all models and seeds) at context lengths 128, 256, and 512. We omit 1,024 because BookMIA does not contain enough continuous nonmember passages of that length for a matched evaluation. The revised results support the same conclusions as before. We also add this discussion in Appendix K.1.

What context lengths and passage-to-document aggregation procedures produced Figure 3a

Figure 3a reports passage-level AUROC at a context length of 512 tokens. We have also now added additional sequence length details in Appendix K in the revised manuscript and thank the reviewer for bringing this to our attention. We do not perform any passage-to-document aggregation. All member passages are samples from different documents.

Reporting the same attacks on the existing NoBooks model would help assess whether the candidate sets are distinguishable without the added Books3 exposure.

We run the membership inference and extraction attacks on the existing 185B-token NoBooks model using three training seeds. Mean AUROC values were:

Attack128256512
Loss0.5070.5000.505
Min-K0.4890.4800.486
Min-K++0.5280.5050.497
Zlib0.4070.4130.431

All four attacks on the NoBooks model perform close to chance across context lengths (Appendix K). At 512 tokens, the Books3-exposed model achieves AUROCs of 0.703 (Loss), 0.685 (Min-K), 0.655 (Min-K++), and 0.705 (Zlib), whereas the synthetic-exposed model achieves 0.515 (Loss), 0.510 (Min-K), 0.552 (Min-K++), and 0.455 (Zlib). Thus, the strong attack performance is specific to the Books3-exposed model; the synthetic-exposed and NoBooks models remain close to chance.

In addition, we run our training data extraction attack (measured via BLEU) on the NoBooks model as baseline. We find that Books3 data is no more extractable from the synthetic model than from the No Books model, which was never trained on Books3. We have updated the paper to reflect these results (Appendix K.2).

ModelMean BLEURate BLEU ≥ 0.75
Books30.0433 ± 0.00012.27e-04 ± 1e-05
No Books0.0388 ± 0.00007.67e-05 ± 2e-05
Synthetic0.0388 ± 0.00006.33e-05 ± 1e-05

 

§4.2 discusses results across context lengths without specifying the lengths or their aggregation in Figure 3a. Its expectation that the extraction gap will increase at larger scales is also untested.

It is well-established in memorization literature that memorization rate rises with model scale [1], hence our expectation that the extraction gap will get larger. On the sequence length, Fig 3a now uses 512 token sequences however we also experiment with lengths of 128 and 256. We have added them under Appendix K.1.

[1] “Quantifying Memorization Across Neural Language Models” (Carlini et al., ICLR 2023).

 

Appendix G.2 reports approximately 450,000 generated chapters totaling 6B tokens, implying about 13,333 tokens per chapter, whereas §4.1 targets 1,000–2,000 words.

Appendix G.2 contained a typo which is now fixed in the updated manuscript. Thank you for bringing it to our attention. The correct number of generated chapters is ~4500k. We first generated ~6B tokens of data from such synthetic chapters. We then remove any chapters containing at least a 50-word overlap with Books3 data (L 357-358). This procedure removes ~24M tokens. From the remaining ~5.98B tokens, we randomly sample 5B tokens for pretraining and discard the rest.

I thank the authors for their comprehensive revision. I particularly appreciate the additional human evaluation and the NoBooks control results. These additions address several of my concerns, and I have updated my score accordingly.

AuthorsOpenReview ↗

Response to Reviewer Comment

Thank you for taking the time to engage with our work and give us feedback!

Reviewer_fbXfOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studied what role fiction books play in pretraining. It's obvious that books definitely play an important role as frontier labs are buying books to train their models. But the reason behind such data selection is unclear in academia. This paper answered the question by pretraining 1B models from scratch with different data mixers of web data, public domain fictions, and contemporary fictions (mostly still copyright protected). The results show pretraining on contemporary fictions helps with various downstream tasks but not public domain fictions. The paper also created synthetic data based on contemporary fictions, and showed that synthetic books can bring the same gain, but less susceptible to membership inference, raising questions in pretraining data composition and provenance.

Critical strengths
  1. The role of (fiction) books in model pretraining is a timely question that is interesting for the perspective of model pretraining techniques and also copyright litigation. I'm so glad this paper answered the question properly. The experiment setting in the paper is well designed: (1) every condition has the same token budget, and each condition only swap 5B tokens and everything else stay identical; (2) the web data is cleaned so the effect of fiction books is not diluted by fictions from the web;
  2. The paper looks into fictions as its own data source, instead of treating books as one single domain. And in section 3.3, it provided explanations of why contemporary is better by analyzing across 62 features, such as coreference chains and theory of mind. This could be useful for pretraining data selection. And it gives controlled evidence for why frontier labs want to buy contemporary books, even though public domain books are widely available.
  3. The writing of the paper is very good. I can tell it's not some AI slop and I enjoyed reading the paper!
Critical weaknesses
  1. Section 4 asked whether synthetic data can bring similar gains. But in the experiment the synthetic books are generated by Gemma 12B conditioned on books from Books3. So it's unclear whether the the gain comes from (1) features carried over from high quality human books in Books3 such as plots, or (2) the writing ability of Gemma itself. I think this also matters for copyright since we want to know if conditioning on copyrighted books is necessary to get such gains.
Questions for authors

I don't have questions for author.

3 comments
AuthorsOpenReview ↗

Author Response to Reviewer fbXf

Thank you for your thoughtful review and for championing our work. We are honored that you found our work timely and interesting, and appreciated both its scientific contributions in better understanding model pretraining, and its broader societal relevance with respect to copyright litigation. We are also glad you highlighted our experimental setup as being well-designed and controlled, and our paper well-written and enjoyable to read. Below we address all the points in your review, and remain available to address any remaining questions you may have before the end of the response period.

 

Unclear whether synthetic data gains come from the writing ability of Gemma itself.

This is unlikely. In the author response period, we started a model run, where we pretrain a 1B model from scratch on 180B tokens of non-fiction data (like our other models) and 5B tokens from unconditioned synthetic fiction generations of the same Gemma model. “Unconditioned” here means the prompts contain no information from Books3. The unconditioned prompts are described in Appendix H.1 in the revised manuscript. The pretraining of this model has just completed (requiring ~1500 H200 hours after data generation). Previously, we saw that models trained on Books3 and Books3- conditioned synthetic data showed notable gains on narrative coherence tasks like LAMBADA (L 181 & Figure 2, L 371 & Figure 13). Here, we find that the model trained with unconditioned synthetic data performs worse than those trained with Books3 or conditioned synthetic data, and performs similarly to the NoBooks model (Figure 14 in the revised manuscript). We have updated our paper with more details on these results (Appendix H). We have also started post-training the unconditioned synthetic model, and will add a comment on the post-trained model evaluation if training finishes before the response period ends, as well as update the manuscript accordingly. These findings are in line with past work, that has shown that it is difficult to generate diverse synthetic data that is useful to improve model performance. For instance, training on synthetic text has been shown to reduce linguistic diversity in model outputs [1] with risks of model collapse [2], including synthetic stories in pretraining could give negative gains [3], and hurt model training even when using more advanced models such as Mistral 7B [4].

[1] “The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Text” (Guo et al., NAACL 2024).
[2] “AI models collapse when trained on recursively generated data” (Shumailov et al., Nature 2024).
[3] “BERTtime Stories: Investigating the Role of Synthetic Story Data in Language Pre-training” (Theodoropoulos et al., 2024).
[4] “Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls” (Kang et al., EMNLP 2025)

Reviewer_fbXfOpenReview ↗

Response to authors' rebuttal

Thanks for the explanation and additional experiments under such short time! This has addressed my concern and shown that high-quality conditioned synthetic data is required. I will keep the score as 4.

AuthorsOpenReview ↗

Response to Reviewer Comment

Thank you again for your thoughtful engagement throughout the reviewing process! The post-training experiments on the unconditioned synthetic model have just concluded, and we wanted to get back to you with the latest results before the response period concludes. We have updated the draft with these evaluations, as well as a detailed discussion on their implications (Section 4, Appendix H.2). In brief, on instruction-following and non-creative writing benchmarks, the model trained on unconditioned synthetic data achieves parity with the model trained on Books3, but the model trained on synthetic data conditioned on real books outperforms both. On creative writing, the model trained on unconditioned synthetic data outperforms the Books3 model, but the model trained on synthetic data conditioned on real books outperforms both in EQ Bench. On LitBench, the unconditioned synthetic model and conditioned synthetic model perform comparably (-1.72 vs -1.74), both outperforming the Books3 model. We have updated the paper to reflect these results and discussed their implications in detail (Appendix H.2 & Table 8). We additionally conduct human evaluations (Appendix J & Table 10). Here we observe that humans show some (but non-significant) preference for generations from the Books3 model over the unconditioned synthetic model, but similar preferences towards generations from the Books3 model and the conditioned synthetic model. These findings suggest that synthetic data overall is a high-quality data source for model pretraining, with the settings conditioned on real books achieving highest performance. We have expanded the draft to include this discussion in detail (Section 4, Appendix H.2). Thank you again for this suggestion, and helping strengthen our paper!

Reviewer_pBSBOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper measures the benefit of pretraining language models on fictional texts (books), which is an overlooked genre in the latest fully open LLMs. It is an interesting research question and the paper does a good job in empirically measuring the impact from multiple angles. One immediate problem with training on such data is copyright violation -- while legally questionable, the authors propose to synthetically rewrite the copyrighted books, showing promising downstream performance.

Critical strengths
  • Arguably, the (pre)training data mixture is the most important component of LLM development, but it's often overlooked at prestigious conferences. It's thus great to see a well-written paper that focus solely on pretraining data.
  • I enjoyed reading the paper, it is clearly structured and easy to follow. All the experiments have clear motivation and the results are properly discussed and put into context.
  • The last experiment around synthetic fiction is a very nice addition to the paper! The fact that it's possible to effectively train on a copyrighted corpus without it being detectable is indeed concerning.
Critical weaknesses
  • The pretraining data composition is rather arbitrary. It is also not clear if some of the results are not just caused by not cleaning/preprocessing the book data -- section 4 (synthetic fiction) mentions issues with boilerplate, but this doesn't seem to be addressed during standard pretraining. For example, isn't the bad performance of public-domain books just a result of poor cleaning of the corpus?
Questions for authors
  • One possible benefit of books is their length, compared to much shorter (on average) web documents. I could see that training on such long and coherent texts might substantially boost the ability of LLMs to deal with long contexts. However, both your 4096-token pretraining and the short-context synthetic generation do not allow to test that. Do you find this to be an important missing piece when studying the effect of pretraining on books?
4 comments
AuthorsOpenReview ↗

Author response to Reviewer pBSB

Thank you for your thoughtful review. We are so glad you appreciated our work addressing a novel, interesting, and overlooked research question. We are also glad that you found our experimental design clear, our paper well-written, and valued our work in shedding light on the effect of pretraining data decisions in the LLM development pipeline. Below we address all the points in your review, and we are available to address any remaining questions you may have before the end of the response period.

Pretraining data composition is arbitrary.

Our pretraining data mix is organized around the following principles:

  1. We target Chinchilla-optimal scaling [1], ensuring that the pretraining corpora contain at least 20B tokens for the 1B models we train from scratch in this work. This served as a lower bound on the training budget.
  2. The proportion of data that is fiction should be broadly in line with publicly-available information about pretraining recipes.
  3. The subset of the datamix that is not fiction should be representative of common domains used in pretraining recipes.
  4. The control NoBooks model should not contain fiction data, so as to avoid diluting the observed effect of fiction in pretraining.

Below is how we operationalized these principles:

  • For (1), all the pretraining data mixes we use are at least 185B tokens (L 97-100).
  • For (2), we use 5B tokens of book fiction, which is 2.7% of the data. This was necessary as 5B tokens was close to the total amount of tokens we could identify as fiction from the Books3 and Institutional books datasets. However, this ended up being comparable in magnitude to the book-source proportions in several published pretraining recipes: approximately 2.2% in RedPajama-V1 [3], 4.5% in LLaMA-1 [4], 3% in MPT-7B [5], 2.3% in Amber [6], and 4.2% in BTLM [7], with the caveat that fiction is not differentiated from other books in the publicly-released information about these recipes. The one recipe where fiction is reported separately comes from a document disclosed in the court proceedings of Kadrey vs. Meta [8], which lists roughly 70B fiction tokens in a target mixture of about 10T tokens. Our 2.7% fiction proportion is comparable.
  • For (3), we build on top of WebOrganizer [2] which is sampled from a performant pre-training corpus (DCLM), and spans multiple common internet domains (see Figure 1(a); L 108-120).
  • For (4), we then filter out all documents tagged as creative-writing to have a controlled setup where the only fiction in the pretraining corpus is present via our added data (L 091-095).

We thank the reviewer for the opportunity to better explain the principles behind our decisions. We will add the following text to the manuscript in the final version:

Our pretraining data mix is organized around the following principles:
1. We target Chinchilla-optimal scaling [1], ensuring that the pretraining corpora contain at least 20B tokens for the 1B models we train. This serves as a lower bound on the training budget.
2. The proportion of data that is fiction should be broadly in line with publicly-available information about pretraining recipes.
3. The subset of the datamix that is not fiction should be representative of common domains used in  pretraining recipes.
4. The control NoBooks model should not contain fiction data, so as to avoid diluting the observed effect of fiction in pretraining.

All pretraining data mixes we use are at least 185 B tokens. We use 5B tokens of book fiction, which is 2.7% of the data. This was necessary as 5B tokens were close to the total amount of tokens we could identify as fiction from the Books3 and Institutional books datasets. This is comparable in magnitude to the book-source proportions in several published pretraining recipes: ~2.2% in RedPajama-V1 [3], 4.5% in LLaMA-1 [4], 3% in MPT-7B [5], 2.3% in Amber [6], and 4.2% in BTLM [7], with the caveat that fiction is not differentiated from other books in the publicly-released information about these recipes. The one recipe where fiction is reported separately comes from Kadrey vs. Meta [8], which lists roughly 70B fiction tokens in a target mixture of about 10T tokens. Our 2.7% fiction proportion is comparable.  We also build on top of WebOrganizer [2], which is sampled from a performant pre-training corpus (DCLM), and spans multiple common internet domains (see Figure 1(a)). We then filter out all documents tagged as creative writing to have a controlled setup where the only fiction in the pretraining corpus is present via our added data.

[1] “Training Compute-Optimal Large Language Models” (Hoffmann et al., NeurIPS 2026).
[2] “Organize the Web: Constructing Domains Enhances Pre-Training Data Curation” (Wettig et al., ICML 2025).
[3] “RedPajama: an Open Dataset for Training Large Language Models” (Weber et al., NeurIPS 2024).
[4] “LLaMA: Open and Efficient Foundation Language Models” (Touvron et al., 2023).

AuthorsOpenReview ↗

Continuation of Author Response to Reviewer pBSB

[5] “Introducing MPT-7B: A New Standard for Open-Source, Commercially Usable LLMs” (Databricks 2023).
[6] “LLM360: Towards Fully Transparent Open-Source LLMs” (Liu et el., CoLM 2024).
[7] “BTLM-3B-8K: 7B Parameter Performance in a 3B Parameter Model” (Dey et al., 2023).
[8] Kadrey et al. v. Meta Platforms, Inc., No 3:23-cv-03417-VC (N.D. Cal.). Exhibit, Dkt. No. 391-14, filed Jan 14 2025. Available at https://storage.courtlistener.com/recap/gov.uscourts.cand.415175/gov.uscourts.cand.415175.391.14.pdf#page=18

 

It is also not clear if some of the results are not just caused by not cleaning/preprocessing the book data

The institutional books corpus had already undergone substantial preprocessing before our experiments. Institutional Books 1.0 includes dedicated automated OCR post-processing that uses predicted line types together with positional, lexical, and punctuation information to reconstruct text and remove page numbers and running heads [1]. Thus, the results presented here should be interpreted as being downstream of a considerable high-quality corpus-specific preprocessing effort. Of course, automated postprocessing is not perfect and we find evidence that OCR errors are still present in the resulting corpus (L 261-268). This is an issue prevalent in pretraining data curation: for example, [2] identify boilerplate, formatting artifacts, and duplication in several widely-used pretraining corpora. As we mentioned in L 268, such artifacts could be a contributing factor, though the corpus has already undergone considerable preprocessing and has been presented as a potential high-quality source of training data for large language models[1].

[1] “Institutional Books 1.0: A 242B token dataset from Harvard Library’s collections, refined for accuracy and usability” (Cargnelutti et al., 2025).
[2] “What’s In My Big Data?” (Elazar et al., ICLR 2024).

 

One possible benefit of books is their length, compared to much shorter (on average) web documents. I could see that training on such long and coherent texts might substantially boost the ability of LLMs to deal with long contexts.

We do not study pipelines for long-context extension as our scope is limited to investigating the effect of fiction in pretraining data recipes. We have added the following text to the limitations section (L 1246-1250) “Fourth, we do not study pipelines for long-context extension, which is typically performed during midtraining. Our scope is limited to investigating the effect of fiction in \emph{pretraining data recipes}, and we view studying the value of books in midtraining for long context extension as a valuable direction for future research.”

Our rationale is as follows. We decided on 4096-token pretraining for our pretraining recipe, as it is standard and used in several model training recipes such as Llama 2, OLMo2, Qwen 2/3, and Nemotron 4. Context length is typically then extended during a “midtraining phase” for some models [1], which comes later in the model development cycle, as larger context lengths can often lead to unstable training as observed in [2]. In this work we focus only on the pre-training phase, and view long context extension recipes using books as a valuable future research direction.

Even so, we do see some related signals. First, on LAMBADA [3], the model trained on Books3 performs better than others (Figure 2). This suggests that books may help the model use longer context more effectively. Second, our writing evaluations use longer generations (on average around 550 words, and up to 1600 words) and long-range coherence is one of the judged criteria for UniEval [4] and human evaluation. In both cases, we observe that the Books3 model performs the best (Table 1 & Appendix J). This suggests that training on books helps the model generate more coherent long text. We expect that long-context extension pipelines would enhance these benefits.

[1] “Effective long-context scaling of foundation models. “ (Xiong et al., NAACL 2024).
[2] “SkyLadder: Better and Faster Pretraining via Context Window Scheduling” (Zhu et al., NeurIPS 2025).
[3] “The LAMBADA dataset: Word prediction requiring a broad discourse context” (Paperno et al., ACL 2016).
[4] “Towards a Unified Multi-Dimensional Evaluator for Text Generation” (Zhong et al., EMNLP 2022).

Thank you for taking your time and responding to my review in detail. I don't seen any reason to change my opinion that this paper should be clearly accepted :)

AuthorsOpenReview ↗

Response to Reviewer Comment

We sincerely thank the reviewer for their thoughtful and encouraging engagement with this study, and for highlighting both the importance of studying pretraining data and the broader societal relevance of this work. Thank you for taking the time!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers

We thank the reviewers for their valuable feedback. We are delighted that reviewers recognize our work as addressing a timely and interesting open question (Reviewer pBSB, Reviewer fbXf) of both scientific and societal importance (Reviewer fbXf). We are glad that they found our controlled experimental setup well designed (Reviewer pBSB, Reviewer fbXf, Reviewer RhCp), our writing clear and easy to follow (Reviewer pBSB, Reviewer fbXf), and that they enjoyed reading our paper (Reviewer pBSB, Reviewer fbXf)! We are also grateful to Reviewers pBSB and fbXf for highlighting the novelty and importance of our study and awarding the paper the highest possible rating. Below, we address all reviewer comments, and will incorporate all feedback in the final version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.