AI CEOs Admit They Don’t Know How It Works: So Why Are You Trusting AI with Your Confidential Data?

BLUF

Most people assume the companies building artificial intelligence fully understand how their systems work. That assumption is wrong. Leaders at OpenAI, Google, and Anthropic have all publicly acknowledged something that should make every business (and individual) pause. They do not fully understand how their AI systems arrive at many of their answers. Yet at the same time, individuals and companies are rapidly connecting these tools to their most sensitive information. Email accounts, internal documents, customer lists, financial data, and operational systems are being handed over to technology that even its creators describe as something of a black box.

Why AI systems are a “black box”

This is not criticism from outsiders. It is coming directly from the CEOs building the technology. In an interview on CBS’s 60 Minutes, Google CEO Sundar Pichai explained that large AI models sometimes develop abilities that researchers did not explicitly program. When discussing these systems, he said the field often refers to them as a “black box,” acknowledging that engineers cannot always explain why a model produced a specific answer or why it made a mistake.

At the AI for Good Global Summit, Sam Altman, the CEO of OpenAI and the public face behind ChatGPT, has made similar comments. When discussing large language models, Altman has acknowledged that researchers still do not fully understand what is happening inside these systems. With billions of internal parameters interacting with each other, the reasoning path inside the model cannot always be traced the way traditional software can. This challenge, known as AI interpretability, remains one of the biggest unsolved problems in artificial intelligence.

Dario Amodei, CEO of Anthropic, the company behind the Claude models, has openly acknowledged a significant issue: a lack of understanding about how AI creations function. He states, "People outside the field are often surprised and alarmed to learn that we do not understand how our own AI creations work. They are right to be concerned: this lack of understanding is essentially unprecedented in the history of technology." 

Why it matters

Uninterpretable models make it difficult to predict failure modes or diagnose why an output is wrong. This opacity complicates governance and makes regulatory oversight challenging. When leaders at the firms building these systems publicly admit they don’t fully understand them, organisations should think twice before delegating sensitive tasks without human supervision. AI interpretability is an active research area, but it remains a long‑standing unsolved problem.

Businesses are connecting AI to sensitive systems

While the builders are still trying to understand the technology, many organizations have already moved straight to trusting it with critical tasks. Companies are using generative AI to write code, draft contracts, summarize legal documents, analyze financial data, communicate with customers, and generate marketing content. Up to 90% of B2B buyers already use tools like ChatGPT, Claude and Perplexity during the buying process, according to a Forrester‑cited MarTech report. Sales conversions driven by AI recommendations have jumped 436% in the last year. In many cases these systems are connected directly to email inboxes, CRM systems, internal documents, financial systems, and customer databases. All of this is happening with surprisingly little discussion about governance or oversight.

Risks of unsupervised AI: hallucinations and systemic errors

Generative AI is already known for hallucinations. In legal research, for example, a Stanford study found that state‑of‑the‑art language models hallucinated 69-88% of the time when answering legal queries, even though these models have passed bar exams. It can produce answers that sound confident but are completely wrong. It can fabricate sources, misunderstand context, or generate polished outputs that hide subtle errors. Yet many organizations treat the output as if it came from an experienced professional rather than a probabilistic system generating predictions.

What makes the situation even more concerning is the rapid rise of AI agents. Millions of users are now experimenting with systems designed to act on their behalf across the internet. These tools promise to handle tasks like managing email, booking travel, browsing websites, writing code, or interacting with other applications automatically.

One example is OpenClaw, an open source autonomous AI agent that originally circulated online under names like Clawdbot and Moltbot before being rebranded. Tools like this allow users to run AI assistants that can interact with files, messaging platforms, and online services on their behalf. That power is exactly what makes them useful. It is also what makes them risky.

In order to function, these agents often require broad access to systems, including files, accounts, and sometimes login credentials. Yet millions of users are now connecting these agents to email inboxes, internal documents, and even financial systems.

In effect, people are giving AI the keys to their digital lives.

That gap between how confidently people are deploying these systems and how little is understood about their inner workings is beginning to show up in the real world.

Case study: Amazon’s AI coding outages

Amazon recently provided a real world lesson in what can happen when speed overtakes oversight. The company has been aggressively integrating AI across its operations and, like many large tech firms, has also carried out significant layoffs in recent years while pointing to automation and AI driven efficiency as part of the shift. Over the past several months, however, Amazon experienced several outages across parts of its retail platform and internal systems. During one incident, North American orders reportedly dropped by roughly 99 percent for a short period of time. That immediately triggered an internal “all hands on deck” response where engineers were required to attend meetings that are normally optional.

The issue was not simply bad code. It was how the code was being created. Like many technology companies, Amazon has aggressively adopted AI coding assistants that can generate large blocks of software in seconds. The productivity gains are real. Developers can move faster than ever.

But speed introduced a new problem.

Engineers were sometimes deploying AI generated code that no one fully understood.

In systems as complex as Amazon’s infrastructure, even a small mistake can ripple across dozens of interconnected services. When AI generates code faster than teams can review it, errors move just as quickly. A flawed line of code can cascade through multiple systems before anyone catches it.

Amazon responded quickly by tightening controls. Senior engineers must now sign off on more code changes, additional approval layers were added before code can go live, and teams are slowing deployments to allow more human review. In other words, Amazon did not stop using AI. They simply put humans back in the loop.

That is the real lesson for every organization experimenting with generative AI. AI is an incredible accelerator. It can draft documents, analyze data, write software, summarize reports, and generate ideas faster than any tool we have seen before. But acceleration without oversight creates risk. And that risk becomes easier to understand when you remember where this conversation started.

Human-in-the-loop: guidelines for responsible AI adoption

If the creators of AI themselves are still working to understand how these systems reason, organizations should think carefully before handing them full control over critical work or sensitive data.

AI should not be allowed to complete critical work without human review. AI agents should not be given access to your most confidential information. And AI should never replace human judgment when the stakes are high.

The right approach is simple.

Use AI to generate the first draft. Then use humans to validate the result. Think of AI as an extremely fast junior assistant. It can do impressive work quickly, but no responsible organization would allow that assistant to send critical work out the door without review. Remember, “People First, People Last”.

The companies that succeed with AI will not be the ones that trust it the most. They will be the ones that understand its limits. And the organizations that thrive in the AI era will not be the ones moving the fastest. They will be the ones that govern it the best.

Frequently Asked Questions (FAQs)

What does it mean when AI is called a “black box”?

Researchers use the term black box to describe systems whose internal decision‑making processes are opaque. Even AI developers can’t always explain why a model produced a specific answer or why it made a mistake. Sundar Pichai and Dario Amodei have both said publicly that their companies don’t fully understand how their AI systems work.

How common are AI hallucinations?

Hallucination rates depend on the domain and the query. A Stanford study found hallucination rates between 69% and 88% for legal queries. In general-purpose chat, hallucination rates are lower but still significant. The key lesson is that AI output should be verified, especially when accuracy matters.

Why did Amazon institute a 90-day safety reset?

After a defective AI-generated code deployment on March 5 2026 caused a 99% drop in orders and 6.3 million lost sales, Amazon paused major AI code deployments for 90 days. The company implemented mandatory two-person code reviews, VP-level approval, and documentation requirements to prevent future outages.

Should my company stop using AI entirely?

No. The lesson from Amazon and other early adopters isn’t to abandon AI but to govern it. Use AI to accelerate routine tasks and generate ideas, but keep humans in the loop for decisions that carry legal, financial or reputational risk. Adopt guardrails such as those outlined above to balance innovation with reliability.

Is AI adoption worth the risk?

When thoughtfully governed, AI can deliver significant advantages: faster drafting, deeper data analysis, and personalised customer interactions. Businesses already see dramatic gains; B2B sales conversions driven by AI recommendations have risen 436%. The key is to pair that acceleration with oversight. Organisations that implement rigorous safety practices will reap the benefits while minimising harm.