Roughly one out of every hundred Notion customers just learned their AI chat transcripts were stolen. The strange part? Notion itself was never hacked. A company most Notion users have never heard of was, and that’s the part everyone should pay attention to.
The Notion AI data breach, confirmed this week, exposed conversations that people typed into Notion AI thinking they were private. Whatever you fed the Insights feature, your questions and the AI’s answers, was sitting in systems belonging to a third-party vendor. Hackers took it from there.
What actually happened
Let’s untangle the chain, because the whole lesson lives in it. The Notion AI data breach didn’t start with Notion, and that’s exactly why it matters.
Fireworks AI is a platform that rents out access to open AI models. Lots of well-known companies use it to power AI features, including Cursor, the legal tech firm Harvey, and Notion. When you use Notion AI’s Insights feature, your request likely travels through Fireworks’ infrastructure on its way to a model.
Attackers breached one of Fireworks’ cloud providers, got into a Fireworks development system, and from there reached customer data. Fireworks published a security bulletin confirming the attack, and the company says it believes the intruders are financially motivated criminals.
Notion’s own disclosure fills in the rest. Fireworks alerted Notion to the incident, and on November 3, Notion disconnected its systems from Fireworks entirely, deleted data it had stored there, and cut every connection between the two companies.
This is the second breach at Fireworks in two months. The October incident is documented in the same security bulletin. Two breaches back to back is a pattern, not a coincidence.
What the hackers took
Three categories so far, and they matter differently.
First, billing information belonging to Fireworks customers. That’s the standard breach fare: names, payment details, account info. Expect targeted phishing attempts built on this data.
Second, some datasets stored on Fireworks systems.
Third, the one that should worry regular people: chat transcripts from Notion AI customers who used the Insights feature. Notion confirmed the stolen transcripts include “any input to or output from the model that was written to the chat.” Your prompts and the AI’s responses, together, in full context. Of everything in the Notion AI data breach, this is the piece with your name potentially on it.
Now for the genuinely unsettling detail. Notion said much of the stolen data was “unattributable,” meaning investigators can’t always tell which customer a given transcript came from. Translation: the company itself may not be able to tell you whether your conversations were in the batch. Around one percent of Notion customers were affected, and notifications are going out, but if you used Insights recently, the safe assumption is to act as if you might be in that one percent.
Why someone would steal AI chats
This wasn’t some espionage operation, according to Fireworks. It was theft with a resale market.
The company’s assessment: the hackers want to sell access to large language models. Stolen credentials and system access let unauthorized users burn through AI model usage on someone else’s dime. There’s an underground market for exactly this, where stolen API access gets resold to people who’d rather not pay for their own tokens.
The transcripts may have been taken to make the access more valuable, or simply because they were reachable from the same foothold. Either way, the economics explain the target: AI infrastructure companies hold both valuable customer data and valuable compute access in one place.
What to do right now
No panic needed. Three concrete steps, in order of importance. The Notion AI data breach is unusual precisely because you couldn’t have prevented it, but your response still changes the outcome.
- Watch your email for notifications. Both Notion and Fireworks are notifying affected customers. If you used Notion AI’s Insights feature, read any security email carefully instead of assuming it’s phishing, then verify it’s genuine before clicking anything. Ironically, breach notifications are a favorite phishing disguise.
- Treat past AI chats as potentially exposed. Ask yourself what you actually typed into Notion AI or similar tools. Passwords? Client names? Contract terms? Health details? If any of it was sensitive and actionable, rotate whatever credentials are involved and keep an eye on the relevant accounts. If you only asked it to summarize meeting notes, your risk is embarrassment, not damage.
- Ask your tools who else they use. This is the new security question for the AI era: which third-party AI providers does this app rely on? Notion answered it publicly because it had to. For the other AI tools in your workflow, the answer is usually buried in a privacy policy or a security page. Worth twenty minutes to find out.
If you want a broader safety net, we’ve put together guides on which files you should never upload to ChatGPT, the AI agent security mistake behind the Pentagon leak, and how to lock your ChatGPT chats with a PIN. Each covers a different slice of the same problem.
The bigger lesson about AI vendors
Here’s the structural thing this breach exposes, and it affects everyone using AI at work, Notion or not. A Notion AI data breach was always going to happen somewhere; this one just happens to be the clearest example yet.
Every AI feature in an app you use is a partnership in disguise. The app’s name is on the interface, but the intelligence often comes from another company’s models, running on a third company’s servers. You trusted Notion. Notion trusted Fireworks. Fireworks trusted a cloud provider. The attackers only needed the weakest link in that chain, and it wasn’t the link you were watching.
Security people call this supply chain risk, and AI just made it personal. Your conversations are now assets stored in places you never chose, governed by contracts you never signed.
That doesn’t mean stop using AI tools. It means use them with the same judgment you’d apply to any third party. Don’t paste secrets into any AI chat that your employer, your clients, or your future self wouldn’t want leaked. Save the sensitive stuff for tools with local or end-to-end-encrypted options, and assume everything else lives somewhere a stranger might someday reach.
The Notion AI data breach will not be the last of its kind. It might be the one that teaches people to ask where their AI actually runs.