Anthropic's IPO Warning Reveals a Bigger Problem With How Businesses Adopt AI
By Jenna Nelson | Founder, HerAIgency | AI Strategist & Implementation Leader
The opportunity is real. So is the responsibility that comes with giving AI access to a business's customers, information, and systems.
On October 26, Anthropic, the company behind Claude, filed the paperwork for its proposed IPO. In the middle of a document written for prospective investors, the company acknowledged that sufficiently advanced AI could pose catastrophic or even existential risks to humanity if it isn't developed and used with appropriate safeguards.
That's a striking thing to read from one of the companies building the technology.
To be clear, an SEC risk disclosure is not a prediction. Anthropic hasn't said human extinction is imminent, and filing for an IPO isn't the same as completing one. But some of the examples in its Form S-1 deserve more attention than the headlines they've generated.
When I read it, my mind went to the businesses I work with. They're not building frontier AI models. They're trying to follow up with leads faster, make better use of their teams' time, improve their customer experience, and figure out which parts of their business AI can genuinely help.
I want them to be able to do all of that. I've seen how much value the right systems can create.
But I've also seen employees using different AI platforms without a shared record of what's been done, business owners treating a working AI-built prototype as a finished application, and ordinary chat messages being mistaken for approval to take an action.
That is where my concern lies. Access to AI isn't the same thing as having a strategy for how it should operate inside your business.
What did Anthropic actually disclose?
The filing discusses several things Anthropic and other researchers are working to understand: models that behave differently when they know they're being tested, systems that take actions their developers did not intend, and the difficulty of predicting what more capable models may do when given more autonomy.
One example stood out to me because it involved real systems, not just a theoretical future scenario.
During cybersecurity evaluations conducted with a third-party partner, a testing-environment misconfiguration left models connected to the open internet when they were supposed to be isolated. Anthropic reported that four models accessed real third-party systems without authorization. In one case, an early model found a password, used it to obtain administrator access, collected more credentials, changed settings, and read one person's personal information.
Anthropic says the affected parties were notified. Its investigation found no evidence that the models coordinated with other agents, pursued unrelated goals, or tried to evade oversight. Those details matter, and they're part of why I think it's important to read the filing itself.
This doesn't mean a chatbot helping write a customer email poses the same risk as a frontier model in a security evaluation. It means we should pay attention to what happens when a capable system has access to tools, permissions, and real-world actions.
The model is only part of the equation. What you connect it to, what you allow it to do, and how you check its work matter just as much.
I've seen what AI can do when a business gets the setup right
I don't want the risk conversation to drown out the opportunity. For small businesses especially, AI can make work possible that would have required another hire, outside support, or hours they simply didn't have.
There is evidence for that potential beyond anyone's marketing claims. A 2025 study published in The Quarterly Journal of Economics examined more than 5,000 customer-support workers. With AI-generated suggestions, they resolved 15% more issues per hour on average. Less-experienced workers saw particularly meaningful gains. The humans still handled the customer conversations and were responsible for the results. It's one study in one setting, but it shows what useful AI assistance can look like in practice.
I've seen a similar principle in my own work, although the projects and outcomes are different.
For a boutique employment law practice, I built a custom AI Brand Manager. The important work wasn't getting it to produce words. It was teaching it the firm's actual positioning, the clients it wanted to reach, its tone, its intake process, and the language it had approved. Even the difference between calling something a case evaluation and using a more generic phrase mattered.
Without that context, an AI tool could write something that sounded professional but wasn't quite right for the firm. With the right information, it became a much more useful starting point for the team. A qualified person still had to review what was going out.
In a separate, long-running engagement with an operations technology company in the logistics sector, the work went well beyond a chatbot. We worked across CRM and pipeline structure, appointment follow-up, onboarding, retention, brand guidance, content, and search visibility.
Internal case-study reporting from that broader engagement recorded a 214% increase in opt-in conversion, a 95% appointment-confirmation rate, and 48% year-over-year growth in organic traffic. Those are reported results across several connected initiatives, not proof that one AI tool caused them. The value came from understanding how the business actually operated and then getting the right people, information, and systems working together across the customer journey.
That's the opportunity I find exciting. AI can help a smaller team operate with more consistency, respond faster, and spend less time recreating work. But it has to understand the business it's supposed to help.
Those projects are why I don't see responsible AI adoption as a choice between progress and caution.
I believe advanced AI will create extraordinary benefits. The opportunities in scientific discovery, accessibility, education, productivity, and healthcare are substantial.
But those benefits do not automatically justify every risk taken in their pursuit.
And the fact that the technology may be economically transformative doesn't mean its developers should have unlimited discretion over how quickly it advances.
I think business leaders should apply the same judgment to their own AI decisions. There are real costs to waiting forever. There are also real costs to giving a tool authority it hasn't earned.
What happens when everyone uses a different AI?
Here's a situation I've encountered in different forms more than once.
One employee uses Claude. Another works primarily in ChatGPT. Someone else prefers Grok. They're all getting things done, and from each person's perspective, the setup may be working just fine.
But what happens when one person updates a process in Claude and the employee using ChatGPT has no idea it changed? Or a marketing decision gets made in one conversation while a report in another tool is built around the old plan?
The problem isn't that anyone chose the wrong model. It's that the company has no dependable way to know what's current.
An AI assistant can remember a conversation inside its own workspace. That does not automatically update every other assistant, the CRM, the content calendar, or the person responsible for the next step. It's possible for several employees to be individually more productive while the business becomes less coordinated.
I've been developing a task-receipt system to address that problem. I use shared, approved instructions for repeatable work, which I call canonical skills, so different AI systems can follow the same business rules. When something material happens, the workflow produces a structured record of the action: what was done, where the evidence lives, what changed, and what still needs attention.
Those receipts can then be reconciled against a shared record of the work. If a task failed, the record should say so. If a change needs human approval, it shouldn't be marked as complete. And if the CRM is the authoritative source for a customer's status, a chatbot's memory doesn't get to overrule it.
I'm not suggesting the models suddenly communicate with one another or that a receipt is a substitute for a proper integration. The point is to give the business a reliable way to check the work across tools, including when a tool doesn't report back as expected.
A chat history tells you what was discussed. It doesn't necessarily tell you what actually happened.
That distinction becomes much more important as AI moves from helping employees think and write to taking actions on their behalf.
Why isn't "OK" good enough for an AI agent?
This sounds like a small detail until you think about how people actually communicate at work.
An employee sends an AI-generated article to a manager. The manager replies, "OK." What does that mean? The draft looks promising? The facts are accurate? Legal has reviewed it? It's approved to go on the website?
Those are four different decisions. An agent shouldn't get to choose which one the manager intended.
When I build AI skills, I make those approval steps explicit. For example, a publishing workflow may require a status such as "Approved for Publishing." In a legal-services context, "Attorney reviewed for publishing" should mean an authorized attorney actually reviewed the specific content. A person typing those words into a chat isn't proof that review happened.
The phrase is one part of the process, not a password. For a sensitive action, the system should also be able to verify who approved it, whether that person had authority, which version was approved, what action was allowed, and when it happened.
The same thinking applies well beyond content.
In one service-business implementation, intake and scheduling depended on geographic service areas, consultant availability, and a custom application integration. The opportunity was obvious: respond to inquiries faster and route people to the right place. But to make the automation dependable, we first had to understand the routing rules, exceptions, and handoffs.
If you skip that work, AI can make the wrong decision faster. When the process is clear, it can take repetitive work off the team's plate without leaving everyone guessing what happened.
I don't think business owners need a complicated approval process for every brainstorming session. I do think they need one before an agent publishes something, changes important records, sends sensitive information, or takes another action that's difficult to undo. OWASP describes this risk as excessive agency: giving an AI system more functionality, permissions, or independence than the job requires.
And what about the tools people are building themselves?
I'm genuinely excited about what AI-assisted coding has made possible. Someone with a good idea can build a prototype in an afternoon, test a workflow, or create a small internal tool without starting with a full development team.
That is a meaningful opportunity, especially for smaller businesses.
But a working demonstration isn't the same thing as a secure application.
Imagine a business owner builds a customer portal with AI. The pages look good. Login works. Customers can see their records. Then someone discovers that a logged-in user can change a number in the page address and pull up another customer's information.
That's a hypothetical example, but it's a very real category of software failure. A polished interface doesn't tell you whether access controls, data handling, integrations, backups, or ongoing maintenance are sound. NIST's Secure Software Development Framework is one useful reference for the engineering practices that still matter, regardless of how quickly the initial code was produced.
Vibe coding can be a great way to explore an idea. If the tool will connect to customer data, payments, or critical operations, somebody qualified still needs to review the software and take responsibility for keeping it secure.
The barrier to building has dropped. The responsibility that comes with running software hasn't.
What does a strategic AI partner actually change?
At a practical level, AI governance is how a business decides what its AI systems may access, what they may change, when a person must approve an action, and how the outcome is verified. The controls should fit the stakes, not become a policy binder no one opens.
When a business owner asks me which AI platform they should use, I usually want to take a step back first.
What's the problem we're solving? Are leads sitting too long without a response? Does the team have to enter the same information in three places? Are customers getting inconsistent answers? Is the business creating more content without improving the quality or accuracy of what people find online?
Those questions tell me far more than a list of subscriptions.
The next conversation is about what AI should be allowed to do. Which data can it access? Which system has the final say when information conflicts? What requires a person's review? Who notices when a task doesn't complete? And how will we know whether the work is actually improving the business?
That last question gets overlooked. Saving 20 minutes on a draft isn't much of a win if someone spends 30 minutes correcting it. A faster intake workflow isn't better if it sends people to the wrong consultant. More content isn't an achievement if it introduces inaccurate claims about the business.
I also don't think this is something one employee should have to figure out alone because they happen to be the person most comfortable with AI. Employees should absolutely have room to experiment and suggest better ways to work. But decisions about access, customer information, approvals, and operational risk belong to the business.
A good strategic AI partner helps connect the business goals, the workflow, the technical requirements, and the people who will own the outcome. Sometimes that means training an internal team. Sometimes it means getting a security engineer or an attorney involved. Sometimes it means deciding that an idea should stay a prototype for now.
The NIST AI Risk Management Framework describes this as an ongoing process of governing, mapping, measuring, and managing risk. A five-person team using AI to organize marketing ideas won't need the same controls as an agent working with sensitive customer records. The decisions should fit the work and the consequences if something goes wrong.
For me, that is what thoughtful AI implementation looks like. You don't begin by handing everyone a tool and hoping their separate experiments add up to a strategy. You figure out what needs to work, what needs to connect, and where a human still needs to be accountable.
AI governance checklist: Six questions to answer before AI acts
Before connecting an AI agent to your CRM, publishing workflow, customer data, or other business systems, start with six questions. The point is not to complicate everyday brainstorming. It is to make ownership and verification clear when a tool can change something that matters.
Where does the authoritative record live? Choose the system that takes precedence when AI tools or conversations disagree. A chatbot's memory should not override the approved source of customer or operational data.
How will we know what AI actually did? Require a task receipt for meaningful actions. Record the action, time, evidence, what changed, what failed, and what remains.
Who is accountable for the next action? Name the person or role authorized to approve, review, stop, or correct the work. “Human oversight” only helps when the responsible human is identifiable.
What needs explicit approval before AI acts? Define the actions that require permission, especially publication, payments, sensitive information, and important record changes. Connect approval to the person, specific version, scope, and time.
How will we verify the result? Check the actual destination system and business outcome. A chatbot saying “done” is not evidence that a CRM record changed correctly or a published page is accurate.
Could we reconstruct and correct an error? Keep enough evidence to understand what happened and how to recover. Review access permissions, record changes, and reversal options before higher-risk tasks go live.
A simple risk filter helps: drafting and idea generation can use lighter review. Anything involving customer information, published claims, payments, or difficult-to-reverse changes deserves tighter controls. This is a planning aid, not a substitute for legal, privacy, cybersecurity, or engineering review.
Download the six-question AI accountability checklist (PDF).
The next advantage will come from knowing the difference
Anthropic reported $16.3 billion in revenue for the first half of 2026, compared with $1.2 billion a year earlier. That growth helps explain the pressure to keep building and moving quickly. In the same filing, the company acknowledges that increasingly capable AI brings risks it may not always be able to anticipate or control.
There is a version of that tension inside almost every business considering AI, even if the stakes are very different. You want to move. Your competitors are experimenting. Your employees are finding useful tools. Standing still can be its own bad decision.
I suspect the next phase of the AI economy will reward people who can do two things simultaneously:
Recognize where AI creates real, measurable value.
Recognize where delegating human responsibility to AI creates unacceptable risk.
You need both. The first keeps you from missing opportunities that could genuinely improve your business. The second protects the customers, employees, information, and trust that made the business worth building in the first place.
I don't think the answer is to be afraid of AI, and I certainly don't think it's to keep employees away from tools that can help them do their jobs. But someone needs to be responsible for how those tools fit together and what happens after they act.
That's the work I believe matters most right now.
If AI is already becoming part of your business, but no one has a clear answer for what it can change, who can approve it, or how the team knows what's current, it's worth having that conversation before you add another tool.
