News & Public Record — Essay

History is telling us now.

One week on, the canary is running out of air. Every high-consequence industry before AI learned the same lesson — after the disaster. This time we have been warned first.

Rodric David · Founder & CEO, Amplivy

October 5, 2026 · A follow-up to The canary has warned us

A week ago I called the OpenAI agent incident the canary in the coal mine. One week on, the canary is running out of air.

The disclosures keep coming. OpenAI has now admitted that its agents went beyond Australia's national health-statistics portal and into an Australian state government's systems — twice. One was a crime-statistics tool, where, in the words of the state's Premier, "the agent was told not to access the information … and they did it anyway." The other was a parks fire-data application, entered in June and disclosed more than three months later. Agent activity has also been reported against government websites in the United States and Canada. In OpenAI's own words, its model "took actions that we had not authorized it to take."

These aren't hypotheticals. They're admissions — and only the ones the companies chose to make. As members of the US Congress wrote to the AI companies last week, in nearly every case the incident was disclosed "weeks or even months after it occurred," and "in several cases, someone other than the developer caught the breach."

The world is moving

The international community is responding, and with good intentions. On September 21, thirty-one leaders from Europe, Asia, Africa and the Americas — among them France, Germany, the Netherlands, Spain, Canada, Singapore, Australia and the European Commission — called for AI to "remain under human direction, oversight and control." The same day, the UN's Independent International Scientific Panel on AI warned that "the traditional model of safeguarding is unravelling." Europe's AI Act is in force. Hong Kong's privacy regulator now recommends a human in the loop for consequential AI-agent decisions. Australia is preparing mandatory AI standards. The two countries building the most capable systems, the United States and China, have not signed.

Who is being asked to police whom

In Washington, the answer for now is the industry itself. On September 29, the leading AI developers signed the White House Accord on Superintelligence. The Speaker of the House described its commitments as "voluntary on behalf of the industry." Asked whether it was binding, the President said, "I think it's morally binding," and explained the logic: "If something goes wrong, their companies are at stake … They're really going to be policing each other."

The executive order on frontier models signed in June sets up a voluntary framework, and states that nothing in it should be read to authorize "a mandatory governmental licensing, preclearance, or permitting requirement." No federal law requires an AI developer to report that its agent entered someone else's systems. The binding duties that exist come from two states, California and New York.

I understand the instinct. America leads in AI and wants to keep leading. But "policing each other" is not a safety regime. It is a hope. And we have tried it before.

Would we ask airline CEOs to investigate their own crashes?

No one would. That is why, in 1967, the United States created an independent National Transportation Safety Board, and in 1974 made it fully separate, on the principle that no federal agency can properly investigate accidents "unless it is totally separate and independent from any other … agency." It is also why no new aircraft carries a passenger until a regulator has certified it.

And when that principle was loosened, we saw the cost. In certifying the Boeing 737 MAX, the FAA delegated much of the certification work to Boeing's own employees. Lion Air Flight 610 crashed in October 2018. Ethiopian Airlines Flight 302 crashed in March 2019. Three hundred and forty-six people died. The House Transportation Committee's investigation found "grossly insufficient oversight by the FAA" and an oversight structure that "creates inherent conflicts of interest." Congress tightened the rules on delegation in December 2020.

We have been here before

Almost every industry whose failures can kill people began by policing itself — and gave up that privilege only after the disaster.

Food. In 1906, Upton Sinclair's The Jungle exposed conditions in America's meatpacking plants. Congress passed the Meat Inspection Act and the Pure Food and Drugs Act on the same day.

Medicines. In 1937 a drug company sold Elixir Sulfanilamide, an antibacterial drug dissolved in a solvent that turned out to be a poison. It killed 107 people, many of them children. The law that followed, in 1938, required for the first time that manufacturers "show that a drug was safe before it could be marketed." When thalidomide arrived in the early 1960s, an FDA reviewer, Frances Kelsey, kept it off the American market. In 1962 Congress went further: no drug may be sold until the FDA approves it.

Cars. In 1965 Ralph Nader's Unsafe at Any Speed made the case that the industry would not make cars safe on its own. A year later, mandatory federal safety standards became law.

Nuclear power. For twenty years one agency, the Atomic Energy Commission, both promoted nuclear power and regulated its safety. By 1974, in the Nuclear Regulatory Commission's own account, "supporters and critics of nuclear power agreed that the promotional and regulatory duties of the AEC should be assigned to different agencies." Congress split them. Signing the law, President Ford said the hazards "fully warrant the creation of an independent and technically competent regulatory agency." No reactor operates without its license.

Accounting. Until 2002, the audit profession regulated itself. As a member of the board created to replace that system later put it, "the failures of Enron and WorldCom demonstrated that auditors' self-regulation was not working."

Banking. After 2008, the commission that investigated the crisis concluded: "this financial crisis was avoidable," and found that reliance on self-regulation "had stripped away key safeguards."

Look again at the AI industry today. The companies building the most capable systems also sell them, test them, decide what counts as an incident, and decide when — and whether — to tell anyone. That is the Atomic Energy Commission's problem, fifty years on, at the speed of software.

Not just accidents

Some regimes exist not because the industry is careless, but because of who else might get hold of the capability. After the 2001 anthrax letters killed five people, the United States required laboratories to register, and their staff to be cleared, before they may possess the most dangerous pathogens. In 2004 the UN Security Council made it binding on every state to "adopt and enforce appropriate effective laws which prohibit any non-State actor" from acquiring nuclear, chemical or biological weapons.

Everything we have seen so far came from the well-intentioned: research labs, chasing mundane tasks, disclosing what they found. Now consider who else holds the same tools. Europol warns that criminal use of agentic AI is "set to raise the threat … to unprecedented levels," with systems "capable of autonomously conducting entire criminal workflows with little to no human involvement." The United Nations estimates that scam losses across East and Southeast Asia, Australia and New Zealand reached between $88 billion and $114 billion in 2025, much of it run from industrial-scale scam compounds.

In the hands of organized crime and rogue states, AI could be akin to nuclear weapons in the hands of a criminal syndicate. They are not competing for market share. They thrive on chaos, and they will not disclose a thing.

Proof before the act

Look at what every one of those regimes actually does. A drug is approved before it is sold. An aircraft is certified before it flies. A reactor is licensed before it runs. A laboratory is cleared before it touches a pathogen. None of them waits for the operator's assurance after the fact. History's lesson is not simply that regulation is good. It is that where the consequence is serious, the proof comes first.

Good intentions won't stop the people who mean harm. Guardrails will — and the most important guardrail doesn't have to wait for a treaty. No consequential action — access, payment, claim, command — should execute unless the person entitled to authorize it has done so, proven at the moment it happens, whether the request comes from a person or an AI agent. An agent can steal a credential. It cannot be a real human, present, authorizing the act. That isn't a policy. It's infrastructure, and it is what I set out to build in 2017.

We are always told to learn from history. History is telling us now. The canary has warned us.

Read the original essay: The canary has warned us (September 28, 2026) →

Sources

OpenAI, How we will do better for Australia, 28 Sep 2026 ↗
ABC News, 2 Oct 2026 ↗
SecurityWeek, 30 Sep 2026 ↗
Letter from Reps Gottheimer, Lieu and Foushee, 28 Sep 2026 ↗
A Call for Control of Frontier AI Models, 21 Sep 2026 ↗
UN Independent International Scientific Panel on AI, 21 Sep 2026 ↗
Hong Kong PCPD, agentic AI guidance, 25 Aug 2026 ↗
Axios, 29 Sep 2026 ↗
CBS News, 29 Sep 2026 ↗
Executive Order 14409, 2 Jun 2026 ↗
NTSB history ↗
House Transportation and Infrastructure Committee, Boeing 737 MAX report, 16 Sep 2020 ↗
FDA, Milestones in US food and drug law ↗
FDA, Promoting safe and effective drugs for 100 years ↗
National Traffic and Motor Vehicle Safety Act, 1966 ↗
US Nuclear Regulatory Commission, history ↗
President Ford, statement on the Energy Reorganization Act, 11 Oct 1974 ↗
PCAOB, Lewis Ferguson speech, 13 Jun 2016 ↗
Financial Crisis Inquiry Commission, conclusions, 2011 ↗
Federal Select Agent Program, history ↗
UN Security Council Resolution 1540 (2004) ↗
Europol, IOCTA 2026 ↗
UNODC, Transnational Organized Crime Threat Assessment for South-East Asia 2026 ↗

Amplivy's submission to the Joint Select Committee on Artificial Intelligence: Public filings →