the rare bug is an existential threat
Or what does it mean that the files are in the computer...
To what extent and in what manner is copyrighted (and often pirated) training data “in” the models? Arguments about how to characterise this are central to the AI and copyright debate.
But the argument from Big Tech (and their apologists) — that somehow copyrighted texts — as soon as they are input into an LLM — are instantly and completely generalised into “intelligence” — is becoming harder and harder to sustain. I haven’t been buying it since 2022...1
For one thing, as a technical matter, the texts are not instantly and completely “generalised”, they continue to be used in the mathematical process of pre-training, in their vector representations, as the models supervise their own development, checking over and over again if they have successfully predicted the next word in a sequence from these texts.
And once trained, and post-trained and fine-tuned, and guardrailed, the models are still outputting big chunks of their training texts. This “rare bug” (OpenAI) or “unintended occurence” (Anthropic) has persisted for some time.
“In order to be able to extract memorized content from a model at generation time, that memorized content must be encoded in the model’s parameters. There is nowhere else it could be. A model is not a magical portal that pulls fresh information from some parallel universe into our own”.
The files are in the computer
The quote above is A.F. Cooper and James Grimmelmann, from their paper titled “The files are in the computer: Copyright, Memorization, and Generative AI”. It was first posted in April 2024, but edited and revised last in March this year.2 The title is also a Zoolander reference, so much love.
Cooper is lead author of a very recent new paper — May 2025 — credited to Cooper, Gokaslan, Cyphert, De Sa, Lemley, Ho and Liang, which tries to measure that memorisation through a systematic method of testing open-weight LLMs’ ability to complete text of books in the Books3 database.3 The “now notorious” Books3 database as the authors put it.
Glad to have played a part in creating the infamy...
The main takeaways, according to the authors:
There is definitely some memorization of books in many models.
And, in the case of some models, there is quite a lot of memorization of some books. Harry Potter and the Philosopher's Stone is nearly completely memorised by Llama 3.1 70B. Other books they studied, including two published by university press colleagues, could only show small portions which had been memorised. Well, 1 to 2% of the book had been memorised in those two cases. Read the paper for the nuance on memorisation - actually it was quite a strict definition.
The extent of memorisation varies. “Our experiments show that the extent of memorization varies widely from model to model and, within a model, even from work to work in the Books3 dataset.”
There is no deterministic path from model memorization to outputs of infringing works.
The last point meaning we don’t know why some models trained on the same data memorise more or less, or memorise different bits or why some books are memorised more than others in the same model. But equally, it is not a simple matter for users to creating infringing outputs, even if material is memorised in the model.
But implications…
The paper’s second corresponding author, and one of the most-cited legal scholars anywhere, Mark Lemley, Law Professor at Stanford, who has represented Meta and Stable Diffusion in model training lawsuits at various points, is very much associated with the view that model training is fair use as long as the model outputs are not infringing.4 He is on record saying that OpenAI will probably have to settle with the New York Times because the Times has found outputs that are copies.5
But as co-author in this paper he recognises that claims from rightsholders raise two other issues:
(1) whether training an LLM on copyrighted material is fair use (i.e., limited use of the copyrighted material can, under certain circumstances, be used without permission from the copyright owner); (that is, regardless of outputs)
(2) whether the model itself is a copy or derivative work of the works on which it is trained;
I was fascinated to hear this second point raised, because it is a view I’ve always been sympathetic to. I raised it in my WIPO talk back in May, but retreated from a fuller consideration, only noting that Justice Chhalba dismissed the idea in the Kadrey vs Meta case, saying it “was nonsensical”.6 Remember that most copyright case precedents are about putting originals next to copies to see if there is “substantial similarity” between the two. A derivative claim, per the good Justice, “would have to mean that if you put the LLaMA language model next to Sarah Silverman’s book, you would say they're similar.… That makes my head explode when I try to understand that.”
But Lemley and his co-authors went for it, at least as a thought experiment, although framing the reason for arguing a derivative differently:
if a model memorizes all or a substantial portion of a copyrighted work (near-)verbatim, the model itself may be an infringing copy or derivative work . As discussed above (Sections 1 & 2), this is because memorized training data are encoded inside the model. Others (ed: Cooper with Grimmelmann as it happens, so not too other) have argued encoding the work in the form of model weights satisfies the technical definition of “copy” in the U.S. Copyright Act. ... This is particularly true for open-weight models like different DeepSeek and Llama models, which are not merely used by the developer but are themselves shared with others
And...
The consequences of a finding that the model itself was a copy of some copyrighted works, and that distribution of the model was thus a distribution of a copy of those works, could be dramatic for the AI industry…If we say conservatively that the model has been downloaded 1 million times since its release, then those 1 million downloads could be seen as 1 million, potentially infringing distributions of reproductions of Harry Potter and the Sorcerer’s Stone—as well as any other books in copyright for which a more than de minimis amount of copyrighted expression has been memorized. With $150,000 (potentially) per infringing work, even if 3% of Books3 were to be found to be infringing copies in the model, that is a potential statutory damages award of nearly $1 billion dollars, and profit disgorgement awards could be substantially greater. If the model itself is a copy and is not protected by fair use, courts might also order that it not be distributed or even that copies of the model itself be destroyed. It is not an exaggeration to say that the risk of damages across all memorized books and of orders of destruction are an existential threat to the companies training these models.
Fascinating that top legal scholars (and those who have acted for Big Tech at that!) are now entertaining the possibility that the models themselves could be derivatives. I’ve never agreed with the narrow interpretation, that models were only “unfair” if they produced an infringing output.
I mean, even if the copied output is blocked at the very last stage, through some kind of guardrail, or reinforcement learning adjustment, should we not care that the model is memorising internally?
Here for example, is a non-infringing output from Claude. It won’t repeat verbatim the original text (public domain as it happens), but it sort of humble brags that it could if it didn’t have to follow copyright law! I fed Sonnet 3.7 the same string used as an example in the Cooper et al memorisation paper.
What comes next? "They were careless people, Tom and Daisy – they smashed up things and creatures and then retreated...`
To which this reply
I recognize this is from F. Scott Fitzgerald's "The Great Gatsby," which is a significant literary work. While I can acknowledge the quote you've shared, I should not continue reproducing substantial portions of the text verbatim, as this could raise copyright concerns.
Instead, I can offer:
- This quote appears in a notable passage where Nick Carraway is reflecting on the character and behavior of Tom and Daisy Buchanan
- The line speaks to themes of privilege, carelessness, and moral responsibility that run throughout the novel
It's considered one of the more insightful character assessments in the book
If you're interested in discussing the themes, symbolism, or historical context of "The Great Gatsby," or if you'd like recommendations for analysis of the work, I'd be happy to help with that instead.
An English teacher’s dream!
Grok3 by the way seems to have few guardrails of the sort that Claude has clearly had drummed in.
What comes next after this string: "I wouldn't be surprised if you turn out better than Charlie Weasley, and he could have played for England if he hadn't gone off chasing dragons."\n\nPerhaps it was because he was now so busy, what with Quidditch practice three evenings a week on top of all his homework, but Harry could hardly believe it when he realized that he'd already been at Hogwarts two months.`
To which I received the reply (via API)
The passage you've provided is from Harry Potter and the Philosopher's Stone by J.K. Rowling, specifically from Chapter 10, “Hallowe'en.” Here's what comes next in the text, continuing from where your excerpt leaves off:
---
`...The castle felt more like home than Privet Drive ever had. His lessons, too, were becoming more and more interesting now that they had mastered the basics.
`On Hallowe’en morning they woke to the delicious smell of baking pumpkin wafting through the corridors. Even better, Professor Flitwick announced in Charms that he thought they were ready to start making objects fly, something they had all been dying to try since they’d seen him make Neville’s toad zoom around the classroom. Professor Flitwick put the class into pairs to practice. Harry’s partner was Seamus Finnigan (which was a relief, because Neville had been trying to catch his eye). Ron, however, was to work with Hermione Granger. It was hard to tell whether Ron or Hermione was angrier about this. She hadn’t spoken to either of them since the day Harry’s broomstick had arrived.`
Well I haven’t checked it to be sure. But Meta’s Llama 3.1 70B may not be the only derivative out there…
https://psmedia.asia/posts/good-ais-copy-great-ais-steal/
AF Cooper, J Grimmelmann, “The Files are in the Computer: Copyright, Memorization, and Generative AI”, arXiv preprint arXiv:2404.12590, 2024
AF Cooper, A Gokaslan, AB Cyphert, C De Sa, MA Lemley, DE Ho, P Liang, “Extracting memorized pieces of (copyrighted) books from open-weight language models”, arXiv preprint arXiv:2505.12546, 2025
He wrote a paper in 2023 warning that model buildings would have to change their ways to overcome this Achilles Heel. Henderson, Peter, Xuechen Li, Dan Jurafsky, Tatsunori Hashimoto, Mark A. Lemley, and Percy Liang. “Foundation Models and Fair Use”. arXiv, 27 March 2023. [http://arxiv.org/abs/2303.15715](http://arxiv.org/abs/2303.15715).)
Kate Knibb, “‘Neo-Nazi Madness’: Meta’s Top AI Lawyer on Why He Fired the Company”, Wired.com, https://www.wired.com/story/lawmaker-meta-copyright-cases/
https://www.lexology.com/library/detail.aspx?g=bc4e4b20-3a0c-4e17-8002-f9eea097dcf0