The Question
A consumer files a complaint with the CFPB. They were denied a mortgage and want to know why. Your underwriting platform uses a gradient-boosted model with features drawn from credit bureau data, bank transaction history, and a third-party risk score. Your compliance team can produce an adverse action notice. What they cannot produce is a clear, legally defensible explanation of why this specific applicant was denied — one that satisfies the CFPB examiner, would survive a fair lending challenge, and can be explained to a non-technical audience in plain language.
Your AI vendor says the model is accurate. Your data science team says the SHAP values show the top contributing features. Your compliance team is asking what SHAP values are and whether they constitute a sufficient legal explanation. The examiner's deadline is two weeks out.
This scenario is no longer hypothetical for financial services firms, insurers, healthcare organizations, or employers using AI in regulated decisions. The right to an explanation for automated decisions is enshrined in GDPR, operationalized in the EU AI Act, addressed in CFPB guidance, and increasingly expected in state-level AI regulation. Enterprises that deployed AI models without building explainability infrastructure are discovering that the question "why did your model decide that?" does not have a satisfying technical answer they can hand to a regulator.
Explainability is not a model property you can add after deployment — it is an architecture decision that must be made before the model is built, and regulatory requirements are making that decision increasingly constrained.
Why This Matters Now
GDPR Article 22, in force since 2018, establishes that individuals have the right not to be subject to decisions based solely on automated processing that produce legal or similarly significant effects, and the right to obtain "meaningful information about the logic involved." The phrase "meaningful information about the logic involved" has generated substantial regulatory interpretation and legal dispute, but the direction of travel is consistent: vague statistical summaries are not sufficient.
The EU AI Act, which reached full enforcement milestones in 2025 and 2026, applies to high-risk AI systems — including AI used in employment, education, credit access, insurance, and essential services — requirements for "sufficient transparency" to enable human oversight, technical documentation of the model's capabilities and limitations, and instructions for use that allow deployers to understand what the model can and cannot explain about its outputs.
The CFPB's 2024 guidance on adverse action notices for AI credit models is the clearest US regulatory statement on this issue. It confirmed that lenders cannot satisfy adverse action notice requirements by citing the AI system itself as the reason for denial. The specific factors that drove the specific decision for the specific applicant must be identifiable, accurate, and communicable in the notice. This is a functional requirement: if your model cannot produce those specific factors for individual decisions, it does not meet the standard.
In employment, the EEOC's guidance on algorithmic hiring tools and NYC Local Law 144 both create implicit explainability pressures: if you cannot explain why the AI ranked or screened candidates as it did, you cannot demonstrate absence of discriminatory impact. Several 2024 and 2025 fair lending enforcement actions named AI model opacity as an aggravating factor in examining disparate impact findings.
What the CURVE™ Data Shows
The 2026 Stackcurve AI Governance CURVE™ Report evaluated vendors providing explainability infrastructure for enterprise AI, covering both model-native interpretability tools and post-hoc explanation methods used in regulated industries.
Fiddler AI ranked as a Leader for production-grade model explainability and monitoring, with SHAP-based explanations surfaced in a compliance-ready format and integration with adverse action notice workflows. Truera (now part of a broader ML observability stack) provides root cause analysis for AI decisions with financial services compliance use cases. Arthur AI includes explainability monitoring alongside bias and drift detection, relevant for organizations that need a unified compliance monitoring layer. DataRobot has invested heavily in regulated industry explainability, with built-in support for producing feature contribution explanations that meet CFPB adverse action documentation standards. For organizations building on open-source tooling, SHAP (SHapley Additive exPlanations) and LIME (Local Interpretable Model-agnostic Explanations) remain the most widely deployed post-hoc explanation methods, with InterpretML providing an accessible open-source interface across both.
For inherently interpretable models in regulated contexts — where post-hoc methods are considered legally insufficient — Reliant AI and Zest AI build credit models on architectures designed for explainability from the ground up, trading some predictive performance for explanation quality.
The full vendor rankings are in the 2026 Stackcurve AI Governance CURVE™ Report — free to download.
The Gap Most Buyers Miss
Post-hoc explanations are approximations, not ground truth
The most important technical caveat in AI explainability is one that most vendor materials understate: post-hoc explanation methods like SHAP and LIME do not reveal what the model actually computed. They approximate the model's behavior in the neighborhood of a specific input and express that approximation as feature contributions. For well-behaved, lower-complexity models, this approximation is often reliable and useful. For deep learning models, neural networks, and very high-dimensional gradient-boosted models, the approximation can be unstable, inconsistent across similar inputs, and potentially misleading.
The regulatory implication is significant: a governance program that relies on SHAP values for adverse action notices is relying on an approximation whose accuracy it likely has not validated. If a regulator or plaintiff tests whether the SHAP-generated explanation is accurate — by varying the input features in the way the explanation implies should matter — and the model does not behave as the explanation predicted, the explanation is not legally defensible.
The three tiers of explainability requirement
Governance programs should distinguish three levels of explanation that different regulatory requirements demand:
Global explanations describe how the model works overall — which features are generally most important, what the model's general decision boundaries look like. This satisfies documentation requirements but does not satisfy individual rights-to-explanation.
Local explanations describe why the model made this specific decision for this specific individual — which features drove the outcome and in what direction. This is what CFPB adverse action notices require and what GDPR Article 22 individual rights requests implicate.
Counterfactual explanations describe what would have had to be different for the outcome to change — "if your debt-to-income ratio were 5 points lower, you would have qualified." This is the most actionable form for consumers and is increasingly what regulators consider "meaningful" under GDPR.
Most deployed AI systems can produce global explanations. Fewer can reliably produce local explanations at scale. Counterfactual explanations at production volume are an active area of development, and few enterprise platforms deliver them consistently.
Model architecture constrains explainability — and that constraint is pre-deployment
An LLM or a deep neural network is not inherently explainable. A logistic regression or a shallow decision tree is. Once a model architecture is chosen and a system is deployed, the maximum explainability level is largely fixed by that architecture. Post-hoc methods can approximate explanations for black-box models, but they cannot retroactively make a black-box model interpretable in the way a transparent model is interpretable.
This means that for regulated use cases — credit, employment, insurance, healthcare — the explainability architecture decision is a pre-deployment governance gate, not a post-deployment compliance add-on. Organizations that are now struggling to explain deployed AI decisions are, in significant part, living with architecture decisions made without considering regulatory explainability requirements.
Questions Your Buying Team Should Be Asking
1. What type of explanation can this system produce for individual decisions — global feature importance, local SHAP values, or counterfactual explanations — and which does the vendor claim meets the applicable regulatory standard in our jurisdiction?
Require the vendor to be specific about the explanation type and explicit about the regulatory claim. If they claim CFPB adverse action compliance, ask them to walk through the exact output the system produces for a sample adverse decision and how that output maps to the regulatory requirement.
2. Has the accuracy of the explanation method been validated against the model's actual behavior, and can you provide that validation documentation?
This question tests whether the vendor is treating explanations as a compliance feature or has genuinely validated that the approximation method reliably reflects model behavior for your use case. If the answer is "our explanations are approximate by design," that is an honest answer — and a governance risk you need to document.
3. For regulated use cases, does the vendor support inherently interpretable model architectures, and what is the performance tradeoff compared to black-box alternatives?
Some regulated contexts — particularly where individual rights-to-explanation are legally binding — may effectively require interpretable model architectures rather than post-hoc approximations. Ask the vendor what architectures they support, what the predictive performance difference is in comparable evaluations, and whether they have deployed interpretable models in compliance contexts.
4. How does the system handle explanation requests at scale — for example, if 10,000 adverse action notices are generated in a single batch processing run?
Production explainability must work at the volume and latency of the underlying model. Explanation generation that works in a demo environment but adds 30 seconds per decision at scale is not a production solution.
5. What is the process when an individual disputes the explanation provided for a decision affecting them, and what audit trail exists to support that review?
The right-to-explanation under GDPR is not one-way. Individuals can dispute the explanation and request human review. The enterprise needs a process and an audit trail. Ask the vendor how their platform supports this workflow, including what documentation is retained, for how long, and in what format.
The Stackcurve Take
AI explainability governance is fundamentally a risk stratification problem. Not every AI application requires the same level of explanation capability. A content recommendation algorithm and a mortgage underwriting model are governed by entirely different legal regimes and require entirely different explainability infrastructure. Enterprises that try to apply a single explainability standard across all AI use cases — or that apply no standard at all — will either over-invest in low-risk applications or under-invest in high-risk ones.
The governance program that works builds an AI use case inventory, maps each use case to applicable regulatory explainability requirements, assesses current model architecture against those requirements, and creates a remediation roadmap for the gaps. For organizations that have already deployed non-interpretable models in regulated contexts, that roadmap may include model replacement — an expensive outcome that was avoidable with earlier governance involvement.
The 2026 Stackcurve AI Governance CURVE™ Report covers AI explainability platforms and interpretable model architectures for regulated industries, including vendor evaluation criteria mapped to CFPB, GDPR Article 22, and EU AI Act transparency requirements. Download it free →
Stackcurve Advisory Briefs are independent research. No vendor pays for placement, tier assignment, or editorial influence. The CURVE™ methodology is disclosed in full at stackcurve.net/research/methodology.