The Question
Your legal team just flagged an incident. A customer submitted a screenshot of your AI chatbot telling them they qualify for a promotional refund policy that does not exist. The customer is now threatening a consumer protection complaint, citing the chatbot's confident, detailed, and entirely fabricated explanation of a program your company never offered. Your head of AI says this is a known limitation of large language models. Your general counsel says that is not a defense.
This is the governance gap that almost every enterprise AI program has: a technical team that understands hallucination as a model characteristic, and a business leadership team that has not yet classified it as a risk category requiring the same treatment as fraud, system outages, or data breaches. Hallucination is on your AI team's radar. It is not on your risk register. There is no incident response procedure for it. There is no escalation path when a hallucination causes customer harm.
The question is not whether your AI systems hallucinate — they do, and they will continue to under current LLM architectures. The question is whether your enterprise has governance structures in place to contain, detect, and respond to the business consequences when they occur.
Hallucination is not a bug your vendor will fix — it is an inherent characteristic of current LLM architectures that your governance program must account for.
Why This Matters Now
The Air Canada case established the precedent that practitioners in AI governance now cite most frequently. In February 2024, a British Columbia Civil Resolution Tribunal ruled that Air Canada was liable for misleading information provided by its AI chatbot to a passenger seeking bereavement fare discounts. The chatbot told the passenger he could apply for a bereavement fare retroactively after completing his trip — a policy that did not exist. Air Canada's defense that the chatbot was "a separate legal entity that is responsible for its own actions" was rejected. The tribunal awarded damages and fees.
The significance is not the dollar amount — it was modest. The significance is the legal logic: an enterprise deploying an AI system to interact with customers is responsible for what that system tells those customers, even when the content is fabricated by the model. The AI's confident, plausible-sounding output was treated the same as a representation made by a human employee.
Since that ruling, additional cases and regulatory guidance have reinforced the pattern. The FTC's guidance on AI-generated deception applies the same consumer protection standards to AI output that it applies to human advertising claims. CFPB guidance confirmed that incorrect AI-generated information provided to consumers in financial services contexts can constitute a violation regardless of whether the error was intentional. Several class actions filed in 2024 and 2025 cite AI-generated incorrect information as the basis for consumer fraud claims.
The regulatory trajectory is clear: regulators are not treating AI hallucination as a technical limitation that earns enterprises a liability exemption. They are treating incorrect AI output delivered to customers as a business communication subject to the same legal standards as any other communication.
What the CURVE™ Data Shows
The 2026 Stackcurve AI Governance CURVE™ Report evaluated vendor solutions for AI output validation, hallucination monitoring, and production model observability. The evaluation covered purpose-built AI governance platforms and AI observability tools with hallucination detection capabilities.
Credal AI ranked among the Leaders for enterprise-grade output validation with policy-based guardrails and audit logging for high-stakes AI applications. Guardrails AI (open-source with enterprise support) provides structured output validation and input/output filtering with strong developer integration. Lakera scored well for real-time prompt injection detection and output policy enforcement in production API deployments. Patronus AI stood out for automated evaluation and red-teaming workflows that stress-test hallucination rates before and after model updates. Weights & Biases Weave provides model tracing and evaluation infrastructure that teams use to measure hallucination rates across prompt variations and model versions.
For enterprises relying on RAG (retrieval-augmented generation) architectures to reduce hallucination, Vectara and Cohere both embed citation-grounding mechanisms that reduce but do not eliminate fabricated output — an important distinction for governance teams setting output validation thresholds.
The full vendor rankings are in the 2026 Stackcurve AI Governance CURVE™ Report — free to download.
The Gap Most Buyers Miss
Most enterprises that have taken AI governance seriously have implemented input guardrails. They filter what users can submit to AI systems. They have policies about acceptable use. What they have not done is build the governance layer around AI output — what the model produces — and the downstream business processes that depend on it.
The risk register gap
Hallucination does not appear on most enterprise risk registers as a named risk category. It is implied inside "AI adoption risk" or "technology risk" at a level of abstraction that means no specific control owner, no defined tolerance, and no reporting mechanism. Until hallucination is a named risk with a risk owner, impact categories, and a defined acceptable frequency, it will not receive governance resources proportional to the actual exposure.
The incident classification gap
What happens at your organization when an AI system produces incorrect output that causes customer harm? At most enterprises, the answer is: it gets handled by whoever noticed it, in whatever way seems appropriate at the time. There is no incident severity classification for hallucination events. There is no distinction between a hallucination that went unnoticed versus one that influenced a material business decision versus one that was delivered to a customer and caused measurable harm.
Incident classification for hallucination events should follow the same logic as other error classifications: severity based on the nature of the harm (customer-facing vs. internal), the decision type (reversible vs. irreversible), and the scale (single instance vs. systematic pattern). Without classification, there is no aggregation, and without aggregation, there is no visibility into whether hallucination is a minor quality issue or a material risk trend.
The output validation gap
Not all AI applications carry the same hallucination risk. An AI tool that summarizes internal meeting notes for convenience has a different risk profile than an AI system that generates customer-facing policy explanations, medical information, or financial guidance. High-stakes AI applications — those where incorrect output could cause legal liability, regulatory violation, customer harm, or material business loss — need output validation requirements that are documented, enforced, and audited.
Output validation requirements should specify: what categories of claims require grounding in source documents, what mandatory human review thresholds apply before AI output reaches certain audiences, and what audit logging is required for AI outputs used in decisions that could later be subject to legal or regulatory scrutiny.
Questions Your Buying Team Should Be Asking
1. What is the vendor's documented hallucination rate for our specific use case, and under what testing conditions was it measured?
Vendors will cite benchmark scores that do not reflect production conditions with your data, your prompts, and your users. Require task-specific evaluation on representative samples from your actual use case. A hallucination rate on a general benchmark tells you almost nothing about the rate your employees or customers will experience.
2. Does the vendor notify us of model updates that could change hallucination rates, and what is the process?
Model providers update models — sometimes with improvements, sometimes with regressions in specific areas. Enterprise governance requires that you know when the model underlying your AI application changes, and that you have a validation process before updated models go to production in high-stakes applications. Ask specifically: does the vendor provide advance notice of model updates, and is rollback available?
3. What output validation controls does the platform provide, and can we configure them by application risk tier?
Different AI applications in your portfolio carry different risk levels. You need the ability to configure validation controls — citation requirements, confidence thresholds, mandatory human review triggers — at the application level, not as a single platform-wide setting.
4. What audit logging is available for AI outputs used in regulated or high-stakes decisions?
When an AI-assisted decision is later challenged — by a customer, a regulator, or in litigation — you need to be able to reconstruct what the AI produced, what data it was based on, and what human review occurred. Ask specifically what the log retention period is, what data is captured, and whether logs are tamper-evident.
5. What is the vendor's position on liability for AI-generated incorrect information delivered to our customers?
Most AI vendor agreements include broad disclaimers of liability for model output. That is a standard contractual position, and it is unlikely to change. But the question forces the conversation about where liability actually sits — with the enterprise deploying the system — and what the vendor is and is not responsible for in the event of a hallucination-caused customer harm incident.
The Stackcurve Take
Hallucination governance is a risk management discipline, not a technology configuration. The enterprises that get this right are not the ones that found a vendor claiming to eliminate hallucination — that vendor does not exist. They are the ones that classified hallucination as a business risk category, assigned it a risk owner, defined application-tier output validation requirements, built an incident classification and response process, and created audit logging infrastructure that lets them reconstruct AI-assisted decisions when challenged.
The Air Canada case will not be the last. The regulatory and legal direction is consistent: enterprises are responsible for what their AI systems tell people. The governance infrastructure to manage that responsibility is available, implementable, and increasingly expected by regulators reviewing AI deployments in financial services, healthcare, and consumer-facing industries.
The 2026 Stackcurve AI Governance CURVE™ Report covers AI output validation and hallucination governance platforms in depth, including vendor evaluation criteria, reference architecture, and governance program maturity benchmarks. Download it free →
Stackcurve Advisory Briefs are independent research. No vendor pays for placement, tier assignment, or editorial influence. The CURVE™ methodology is disclosed in full at stackcurve.net/research/methodology.