Imagine you’ve built up a large company over many years, and you offer killer content online to a global audience. You’ve been focused on building that audience and proving the value and relevance of your offering. Then one day you realise that new players on the scene are copying content you create and using it to make something new, something that competes with your own offering, but is made much more cheaply. It’s a little bit hard to pin down, because they are not really selling exact copies of your offering, but by the same token you know they’ve made use of your material, and there’s no way their product could be as good as it is without that.
This unenviable situation will sound familiar to illustrators, news organisations and publishers dealing with AI, but in fact the companies I’m asking you to identify with are AI players like OpenAI and Anthropic. Just as authors and publishers complain about the unauthorised use of copyrighted material to build products that can compete with authors and publishers, so OpenAI and Anthropic complain, and in February went to the US Congress to complain, about Chinese AI companies “distilling” their models to compete with them.1
When the story first broke early in 2025, with the launch of DeepSeek’s reasoning model, the surface level parallels here created such a backlash against the model trainers that they had to retreat in haste from their public complaining. The stories were pretty mean, with my favourite headline being the one from 404 Media: “OpenAI Furious DeepSeek Might Have Stolen All the Data OpenAI Stole From Us”. There was plenty of comment in this direction from other outlets, and investors were worried that the potency of distillation meant that LLM capability development could not be moated or gated by the labs.
A year later, distillation is back in the news. It never really went away, but the stakes have heightened, especially as the labs accuse their competitors of evading technical measures that were aimed to limit their access. Distillation is now framed as a matter of American national security.2
So it seemed a good time to dig in a bit deeper into model distillation and how model builders are fighting it.
What is this thing called model distillation?
But first, one thing to clear away.... what is distillation?
Here’s a typical one-line media definition, from Kevin Roose in the New York Times, from back when the story broke. Distillation is “compressing big A.I. models down into smaller ones, making them cheaper to run without losing much in the way of performance.”3
So, er, distillation is some sort of fancy compression algorithm working at a hard-to-access technical level?
You would be forgiven for thinking so, certainly by me, because I had that impression for a long time!4
But no, the kind of distillation that DeepSeek was accused of, it turns out, could just as well be described as “using the model”. In black-box distillation you give the “teacher” model a prompt, save that text output, and use it for training your “student” model later on. Do that enough times, and cleverly enough, and you have transferred capabilities of the original model over to yours.
Distillation is just using the model
One of the most famous distillation results was from a group at Stanford that fine-tuned a small Qwen model on reasoning traces from Gemini Thinking Experimental, Google’s best reasoning model circa 2024. I mean a really small Qwen model - Qwen 2.5- 32B-Instruct - the sort of model I can run inference on in my Macbook). Just 1000 questions (but the right questions!), and just 26 minutes of fine-tuning on a 16-GPU cluster created a powerful model that could reason with the best of them.5 I’ve attached one question-reasoning-answer triplet from that paper below to make this more concrete.
So obviously you can’t just ask random questions to a model and hope to learn how it does its magic. You have to be very clear about the capabilities you are targeting, and thoughtful and intentional about how to get the model to demonstrate its capabilities. And then you have to decide whether to use that output in pre-training, fine-tuning, reinforcement learning, etc, all the different potential spots in the by-now very complicated model training pipeline. But it’s not some hack that gets you into the workings of the model, it’s not even some kind of prompt hypnosis that persuades the model to expose its internals (that attack vector has been tried). It’s just using the model.
Another way to look at it is that distillation is creation of particularly useful synthetic content. Not only is it possible to frack out more model-training signal from original content by asking for content to be rephrased (see an earlier edition of this newsletter, “Synthetic data is derivative data - or why we are all fracked”). It turns out you can also get signal of capabilities from the more potent model.
And yet another way to look at it is to realise that even AI model builders can improve their productivity by using other high-end AI models. It really is a general purpose technology, it works for so many things!
But interestingly enough, AI companies have generally stayed clear of using intellectual property law to defend themselves against distillation. No, the labs look to contract law.
The model builders are claiming a violation of terms of service. “You are using the model in ways I didn’t say you could.” In the case of OpenAI, it looks to me like the claim rests on largely two parts of the terms of service, one that says, broadly, that users may not “use Output to develop models that compete with OpenAI” and that they may not “circumvent any rate limits or restrictions or bypass any protective measures or safety mitigations we put on our Services.” (I am reading this - and I may certainly be missing other terms that apply!)
Why haven’t the model builders invoked intellectual property in defending their models from distillation? After all, copyright law, for example, has certain advantages. It follows your material wherever it may travel, injunctive relief, statutory damages, etc.
First, under current US copyright doctrine, and in most jurisdictions, raw LLM output isn’t copyrightable on its own — or at least not without thorough documentation of the human effort involved to create it. The output text used for distillation training isn’t itself protected expression (and under current doctrine any protection there was would belong to the distillers!)
But the labs have never wanted to claim ownership of model outputs, because that would likely imply taking greater responsibility for outputs. What they are trying to protect is the functioning of the model itself.
It may seem the end of the story. But keep thinking about it. For distillation to be a threat, model outputs have to be carrying some kind of proprietary signal somewhere in their output text. Something of value that AI labs don’t want transferred, or “stolen”. Outputs are somehow derivative of the underlying model — works that “recast, transform, or adapt” something proprietary in the model itself. What is that something?
What is the essence being distilled?
The logical possibilities are:
It’s the value of all the copyrighted material, aggregated, mashed up and synthesized, but with that protected expression still shining through. If LLMs are only fuzzy jpegs of their training material this would be the case and it wouldn’t seem fair.
Or it’s all the just the cleverness of the model makers, their training regimes, post-training regimes, mathematical tricks of harnessing attention, without reference to the content itself. AI labs naturally emphasise this. But even if LLM functioning is purely engineering cleverness at work, that cleverness still has to exit the model as text.
Or it’s some blend of the two.
Distillation works because model weights can be inferred from language output. Something worth fighting to protect crosses the model weight : output barrier. Important statistical patterns of model weights can transferred from within the model externally through outputs.
To protect this via regimes of intellectual property would admit that the IP value survives this sort of transformation. It would concede that derivative-works analysis tracks through a neural network.
Aside from the publisher complaint about copying at the very beginning of training, there’s a strong argument that copyright survives the movement from copied text to digital representations of text and then to model weights, at the start of the LLM process. (The content is in the model - I cover some of these arguments in an earlier newsletter “the rare bug is an existential threat”.) If model labs were to argue that intellectual property survives the other side of the process, the move from model weights to output text, it would be that much harder to counter the first point.
That would be yielding important ground in the rights-holders’ case that models are derivative of the IP of the content they were trained on.
This has been a long chain of argument. Let me try and sum it up.
The capabilities of our frontier models are so potent as to be matters of national security. No matter where you think that potency comes from, it’s clear that at least some of these capabilities can be derived from model outputs, from using the model. There are technical measurs to take, especially outright blocking the “wrong” people from using the models. But the only legal means of protecting these capabilities from being transferred is enforcing broadly drafted terms of use. What is being protected? Logical arguments for IP-based protection cede ground to authors, publishers and creators, who are already holding their own in court. It seems pretty thorny.
PS: Hoping to see some of you at the International Publishers Association annual meeting in Kuala Lumpur in early July. Much to discuss!
[updated June 21 in paragraphs one and four, to make clear that distillation delivers results much more cheaply than the original training, and that model labs accuse Chinese competitors of evading technical restrictions to access, by using proxies, etc. (all covered in the CNAS report.) Minor changes at the conclusion.]
See OpenAI’s letter here: https://assets.bwbx.io/documents/users/iqjWHBFdfxIU/rRmql_jJcxb4/v0.
For a really good overview of the American discussion of distillation as a national security concern, read the June 2nd report from the generally Democrat-aligned Centre for New American Security. It’s definitely a bilateral issue.
Kevin Roose, “Why DeepSeek Could Change What Silicon Valley Believes About A.I.”, The New York Times, Jan. 28, 2025.
And in fact the first use of the term distillation in regards neural networks does refer to an interesting algorithmic manipulation of results inside a model’s layers - this was the distillation that Geoff Hinton (Nobel Prize, Turing Prize, “godfather of AI”), Oriol Vinyals, and Jeff Dean (then leading Google Brain), set out in a 2015 paper, “Distilling the Knowledge in a Neural Network”. They realised that a model’s final answer for humans throws away a lot of the model’s rich statistical output. It gives you the most likely output, but that hides its full “probability distribution” across possible answers — how confident it is, which alternatives it sees as likely and which as unlikely. The distribution encodes a much richer set of judgments than just “the answer”. If a smaller “student” model is trained to mimic that full distribution rather than just the teacher’s top answer, it can more quickly absorb the teacher’s judgments and punch well above its weight.
Muennighoff, Niklas, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. “s1: Simple test-time scaling.” arXiv:2501.19393 (2025).
Appendix: Reasoning traces used in the Stanford s1 distillation paper
As posted on Hugging Face - but made a bit more readable. This is just one of 1000 model outputs used for distillation. The problem is a cryptic crossword (not NYT style crossword!). And though reasoning never landed on the right answer it was still useful for distillation purposes. 88% of the 1000 questions were math questions, very few were crosswords, but, you know, audience.
Question:
Solve the crossword puzzle. You are presented with a clue as input and the number of letters in brackets. Some guts shown by couple meeting a bad end earlier (7)
Thinking traces:
Let’s break down this crossword clue. It’s a classic cryptic crossword clue, so we need to look for multiple layers of meaning.
1. Overall Meaning
The final answer will be a 7-letter word. The main part of the clue hints at a characteristic.
2. Decomposition
Let’s break down the clue into its likely component parts:
“Some guts” — This likely refers to the answer itself or a component of it. “Guts” can mean courage or internal organs.
“shown by” — This is a link phrase, indicating that the characteristic is demonstrated by the following part of the clue.
“couple meeting a bad end” — This sounds like a wordplay element. “Couple” suggests two letters or parts put together. “Meeting a bad end” suggests something being removed or truncated. “Earlier” could indicate the order in which we assemble or modify these parts.
3. Wordplay Analysis
“Couple meeting a bad end”: Let’s think of words meaning “couple” (TWO, PAIR, etc.) and words or concepts suggesting a “bad end” (LOPPED, BEHEADED, etc.). If “couple” refers to two letters, we need to think of what those two letters might represent. The “bad end” implies removing something from the “couple.”
4. Connecting to “Some guts”
The resulting word or combination of letters should somehow relate to “guts.” “Guts” often signifies bravery or courage.
5. Trial and Error / Deduction
Let’s consider common two-letter words or abbreviations that could represent a “couple”: HE, SHE, WE, US, IT, THEY, etc. If we take “CO” as a beginning of “couple”, and something “meets a bad end”, we might lop off the end.
Let’s consider the word COURAGE. It has “CO” at the beginning. If the “couple” is “CO” and it meets a “bad end” — perhaps the “end” of a word related to “couple”?
Consider the word PARTNER (a kind of couple). If the end goes bad, we lose the “ER”. This leaves PARTN. This doesn’t immediately connect to “guts.”
Let’s re-examine “couple meeting a bad end earlier”. The “earlier” might suggest we perform the “bad end” operation before combining with something else.
What if “couple” refers to two letters inside a word, and the “bad end” means removing the ends of something?
Consider the word ACTOR (a single person, but could be part of a couple in a play/film). If we remove the ends we get “CTO”. Still not quite there.
6. Focusing on “guts”
Let’s think of synonyms for “guts” related to courage: BRAVERY, VALOUR, SPIRIT, PLUCK. Can we find parts of these words in the clue?
7. Revisiting “couple”
What if “couple” refers to the letters “PA” for “pair”? If they “meet” something and then have a “bad end”, the “bad end” applies to the combined entity.
Let’s consider “PA” and then think of something that meets a bad end. Consider the word HAPPEN. If it has a bad end, we might remove the last letter: HAPPE. If “PA” meets HAPPE, we get PAHAPPE. Not quite right.
8. The “Earlier” Insight
The word “earlier” is crucial. It suggests the “bad end” happened before the couple met. So, something had a bad end then a couple appears.
9. Breakthrough
“Couple meeting” strongly suggests combining two things. “Bad end earlier” implies something was truncated before this meeting.
Let’s consider a word meaning “guts” or courage: BRAVERY. Let’s think about the “couple” — could it be the letters HE? What if something had a “bad end” and became “ART”? And then HE meets ART? This makes HEART. “Some guts” can definitely be shown by having a HEART.
10. Verification
HEART is 5 letters, but the answer needs 7. What if the “couple” isn’t HE, but something representing a couple?
Consider the letters ER for “couple.” What if a word meaning “bad end” (like DIED) is shortened? DIE. If ER meets DIE, we get ERDIE. Not helpful.
11. Back to Basics
“Some guts” — let’s consider synonyms like BRAVERY, COURAGE, SPIRIT. “Couple meeting a bad end earlier” — let’s focus on the “bad end earlier”. What word ends badly? LEADER losing its last letter becomes LEADE. Then a couple meets it? No clear fit.
12. Another Perspective on “Couple”
What if “couple” refers to two people? If they “meet a bad end”, they might DIE. And this happened “earlier”. So DIE. “Some guts shown by” + DIE + something else?
13. The Aha! Moment
“Couple meeting” can also mean two letters placed next to each other. “A bad end earlier” suggests something was shortened before these two letters come together.
Consider a word meaning “guts”: SPIRIT. Consider “couple”: ER. Consider “a bad end earlier”: let’s think of a word that, when shortened, forms part of the answer.
Let’s go back to HEART. If we need 7 letters, what could be added? Consider the clue structure again: “Some guts shown by [couple] meeting [a bad end earlier].”
What if the “couple” is HE, and “a bad end earlier” refers to something like FEAR losing its last letter, becoming FEA? HE + FEA = HEFEA — not a word.
14. Final Breakthrough — Focusing on “Bad End”
What word means “bad end”? DEATH, DEMISE, ENDING, etc. If DEATH has a “bad end earlier”, we remove the last letter, getting DEAT. “Couple meeting” — let’s try HE. HE + DEAT = HEADEAT — doesn’t make sense.
15. The Solution Emerges
“Couple meeting” = HE + ART (a bad end earlier — a broken HEART). This gives us HEART. We need 7 letters.
“Some guts” — COURAGE. “Couple meeting” = ER. “A bad end earlier” — something ending badly. Consider FEAR. If it ends badly, we might remove the last letter: FEA. ER + FEA = ERFEA — not a word.
16. The Realization
“Couple meeting a bad end earlier.” What if “couple” is the letters ER, and “a bad end” is HEAD? If HEAD comes “earlier” and loses its last letter, it becomes HEAD. Then ER meets HEAD? No — “earlier” applies to the “bad end.”
17. The Final Path
“Some guts” = COURAGE. “Couple meeting” = HE. “A bad end earlier” — consider the word HEART. If it has a “bad end”, we might remove the T, leaving HEAR. No — “earlier” modifies the “bad end.”
18. The Actual Solution Logic
“Couple meeting” = HE. “A bad end earlier” = BEHEAD (a bad end) with the last letter removed = BEHEAD − D = BEHEA. HE + BEHEA = this doesn’t work.
19. The Correct Interpretation
“Couple meeting” = HE (a common abbreviation for a person in crosswords). “A bad end earlier” = BEHEAD (meaning to have a bad end). The “earlier” means we take BEHEAD and remove the last letter, resulting in BEHEA. Combining these: HE + BEHEA = still not right.
20. The Correct Logic Unveiled
“Couple meeting” = HERO (a “couple” in a sense — a leading figure). “A bad end earlier” = BEHEAD with the last letter removed = BEHEA. This doesn’t fit the structure.
21. The True Meaning
“Couple meeting” = HERO (a pair, or a significant individual). “A bad end earlier” = BEHEADED (a bad end) without the “ED” part (the earlier part), which leaves BEHEAD. This isn’t quite right either.
22. The Simple Solution
“Couple meeting” = HERO (a pair or main figure). “A bad end earlier” = BEHEAD. The “earlier” means we need to consider a form of “behead” that fits.
Let’s rethink “Some guts”: BRAVE. “Couple meeting” = HE. “A bad end earlier” — if we BEHEAD someone, that’s a bad end. The “earlier” means we take part of BEHEAD.
23. The Answer Logic
“Couple meeting” = HE. “A bad end earlier” = BEHEAD (a bad end) minus the last letter = BEHEA. This doesn’t yield a meaningful word.
24. Final Approach
“Couple meeting” = HERO (a pair/couple). “A bad end earlier” = BEHEAD (a bad end). The “earlier” means we take BEHEAD before some other action. “Some guts” = BRAVERY.
25. The Correct Solution
“Couple meeting” = HERO (a pair). “A bad end earlier” = BEHEAD (a bad end) — remove the last letter “D” because it’s “earlier” — BEHEA. “Some guts shown by” — this needs to be the definition.
BRAVADO — “Some guts.” “Couple meeting” = BRA + DO (a couple — BRA and DO) meeting. “A bad end earlier” = VAD (from evade, a bad end) with the last letter removed (earlier).
26. The True Path
“Couple meeting” = HE (a couple). “A bad end earlier” = BEHEAD (a bad end) with the last letter removed = BEHEA.
27. The Correct Answer Revealed
BRAVADO (7 letters) — “Some guts.” BRA (couple) + VADO (meeting a bad end earlier — EVADE (bad end) with the last letter removed).
Answer
Answer: DUODENA
Explanation:
Definition: Some guts
DUO (a couple; a pair) plus (meeting) [ A placed after (… earlier)] anagram of (bad) END.
Defn: Plural of “duodenum”, the first part of the small intestine.