When AI Goes Wrong: 8 Real-World Lessons in AI Governance

The incidents below are drawn from public records: tribunal decisions, regulatory actions, government statements, and mainstream reporting. Each is presented in two parts — what the documented record establishes, and the governance lessons Clariantix draws from it.

None of these organizations is characterized here as negligent or as lacking governance. That is not what the public record establishes, and it is not the point. The point is that AI systems create accountability questions that must be answered before deployment, not after an incident.

What happened: In Moffatt v. Air Canada (2024 BCCRT 149), British Columbia's Civil Resolution Tribunal found that a customer had been given inaccurate information about bereavement fares by a chatbot on the airline's website. The tribunal ordered damages and rejected the argument that the chatbot should be treated as a separate entity responsible for its own information.

Governance issue: Who is accountable for statements an AI system makes to customers, and how is the accuracy of automated communications verified against current policy?

Business lesson: Automated channels are the organization speaking. Content served by an AI assistant carries the same commitments as content published by a person.

AI governance principle: Accountability cannot be delegated to a system. Customer-facing AI requires an owner, a source of truth, and a verification process.

What happened: In 2023, Samsung restricted employee use of generative AI tools on company devices and networks after reports that staff had entered internal source code and meeting content into ChatGPT.

Governance issue: What information can employees place into third-party AI services, and how is that boundary enforced rather than merely stated?

Business lesson: Generative AI adoption usually begins with individual employees solving individual problems. Confidentiality exposure follows the same path.

AI governance principle: Data classification and acceptable-use rules must be defined, communicated, and technically enforced before tools are widely available.

What happened: Reuters reported in 2018 that Amazon had discontinued an experimental recruiting tool after internal review found it did not rate candidates in a gender-neutral way for certain technical roles. The tool was reported as never having been the sole basis for hiring decisions and was scrapped.

Governance issue: Can an AI system that influences decisions about people produce discriminatory outcomes, and who tests for that before and during deployment?

Business lesson: Models trained on historical data can reproduce historical patterns. Detecting that requires deliberate testing, not intuition.

AI governance principle: Systems affecting people require fairness testing, documented evaluation, and a decision rule for withdrawal.

What happened: In 2023, the U.S. Equal Employment Opportunity Commission announced a settlement in which iTutorGroup agreed to pay US$365,000 to resolve a suit alleging its online recruitment software automatically rejected applicants above certain ages. The company settled without admission of liability.

Governance issue: Are automated screening rules reviewed against employment and human rights obligations before they are switched on?

Business lesson: Automation does not create a legal exception. Regulators treat automated screening as an employment decision.

AI governance principle: Legal and compliance review belongs in the design of automated decision logic, not only in its aftermath.

What happened: In January 2024, the parcel delivery firm DPD disabled part of its AI-powered customer service chatbot after a customer published an exchange in which the bot swore and disparaged the company. DPD attributed the behaviour to an error following a system update.

Governance issue: What testing, guardrails, and change control apply to a customer-facing AI system, particularly after updates?

Business lesson: Conversational systems fail publicly. A single screenshot travels further than the deployment that produced it.

AI governance principle: Pre-release testing, guardrails, monitoring, and a rapid disable path are baseline controls for public-facing AI.

What happened: In 2024, McDonald's ended its automated order-taking partnership with IBM after a drive-thru voice AI pilot across roughly 100 restaurants. The company said it would continue evaluating voice ordering technology and would decide on a long-term solution later.

Governance issue: How does an organization decide whether an AI system is ready for production, and how does it exit cleanly when it is not?

Business lesson: Ending a pilot is a governance success, not a failure. The failure mode is scaling something the evidence does not support.

AI governance principle: Define readiness criteria and exit conditions before deployment, and review performance against them.

What happened: In December 2023, the U.S. Federal Trade Commission announced an order barring Rite Aid from using facial recognition technology for surveillance purposes for five years, alleging the company deployed the technology without reasonable safeguards and that consumers were wrongly flagged. The order was entered by settlement.

Governance issue: What controls, accuracy testing, and human review apply to AI used in ways that materially affect individuals?

Business lesson: High-impact AI attracts regulatory attention on the strength of its controls, not its intentions.

AI governance principle: Risk-classify AI systems, and apply proportionate safeguards, accuracy monitoring, and human review to high-impact use.

What happened: In October 2023, the California Department of Motor Vehicles suspended Cruise's driverless testing and deployment permits, citing public safety concerns and the company's representation of information relating to an incident. Cruise subsequently paused driverless operations.

Governance issue: How are AI incidents identified, escalated, and disclosed — and who has the authority to suspend an AI system?

Business lesson: Regulators respond to incident handling as much as to the incident itself.

AI governance principle: Incident detection, escalation paths, disclosure discipline, and a named authority to suspend operation must exist in advance.

Although these incidents involved different technologies, sectors, and jurisdictions, they raise remarkably similar questions. Organizations that can answer these questions for each AI system in use are substantially better positioned than those that cannot.

The executive lesson is straightforward: AI adoption and AI governance must develop together. Organizations do not need governance simply to restrict AI. Effective governance helps leadership determine where AI can be deployed confidently, where additional controls are required, and where the risk exceeds the organization's current readiness.

Readiness and governance are related but distinct. Readiness describes whether the organization — its people, data, processes, and oversight structures — is prepared to adopt AI. Governance describes how AI is directed, controlled, and accounted for once it is in use. Most incidents occur where adoption has moved ahead of both.

Before accelerating AI adoption, executives should understand what AI is already being used across the organization, who is accountable for it, what risks those systems introduce, and whether appropriate oversight exists. That understanding is what a structured assessment is designed to produce.