Most enterprises discover their explainability problem when a regulator asks a question they cannot answer. The question is rarely abstract. It is specific: why did this model recommend denying this loan application, flagging this claim for fraud review, or declining this insurance policy? If your answer is "our model is accurate," you have failed the regulatory test. Accuracy is not explainability. The ability to explain a specific decision to a specific person affected by it is a distinct technical and governance requirement that most enterprise AI programs address far too late.
Explainability is not a feature you add to an existing model. It is an architectural choice made during model design that determines which techniques are available to you, what your operational overhead will be in production, and whether your system can comply with regulations that require decision-level explanation. Organizations that build first and add explainability later typically discover that their chosen model architecture is fundamentally incompatible with regulatory explanation requirements, requiring a complete rebuild.
Three Levels of Explainability: What Regulators Actually Require
Explainability requirements exist at three distinct levels, and conflating them leads to technical solutions that satisfy none of them. Understanding which level a regulation or business requirement specifies is the first step in designing an appropriate architecture.
Regulatory Explainability Requirements by Sector
Different regulatory frameworks impose different explainability requirements with different technical implications. The table below summarizes the requirements most relevant to enterprise AI programs. Organizations operating across multiple jurisdictions face potentially inconsistent requirements that must be reconciled at the architecture level.
The Architecture Decision: Inherently Interpretable vs Post-Hoc Explanation
The most consequential explainability decision is made before any model code is written: do you choose an inherently interpretable model architecture, or do you build a complex model and apply post-hoc explanation methods? This is not a purely technical choice. It has operational, regulatory, and accuracy implications that must be understood before commitment.
| Approach | Best For | Limitations | Regulatory Acceptance |
|---|---|---|---|
| Logistic Regression with Constraints Inherently interpretable. Coefficients are the explanation. |
Binary classification, adverse action, regulated credit decisions where full transparency is required | Lower accuracy on complex non-linear relationships. Requires significant feature engineering to compete with tree-based models. | HIGH — widely accepted, audit-ready |
| Gradient Boosted Trees with Monotone Constraints Post-hoc explanation via SHAP. Monotone constraints enforce directional consistency. |
Credit risk, fraud detection, churn prediction where accuracy matters and SHAP explanations are accepted | SHAP explanations approximate feature contributions and may be inconsistent for individual predictions near decision boundaries. | HIGH — widely accepted in US financial services with SR 11-7 validation |
| Deep Neural Networks with SHAP or LIME Post-hoc approximation. Explanation is separate from the model. |
Image classification, NLP tasks, complex pattern recognition where no other approach achieves required accuracy | Post-hoc methods approximate explanations and may be unfaithful to the model's actual decision logic. LIME explanations can be unstable. Does not satisfy regulatory requirements that require the explanation to reflect the model's actual reasoning. | VARIES — increasing regulatory scrutiny in high-risk use cases |
| Counterfactual Explanation Systems Generates "nearest possible different outcome" explanations independently of model type. |
Cases where actionable recourse is required: what could the applicant change to get a different result? | Computational overhead at scale. Must ensure generated counterfactuals are actionable for the individual, not just mathematically proximate. | EMERGING — increasingly required for GDPR Article 22 compliance |
The model with the best accuracy on your test set is not necessarily the right model for your use case. If you cannot explain its decisions to a regulator, a customer, or a jury, accuracy is irrelevant. Architecture choice must include explainability requirements from the beginning.
The Implementation Checklist for Enterprise Explainability
Based on our work with regulated enterprise AI programs across financial services, healthcare, and insurance, these are the implementation requirements that are most commonly missed. Each one represents a gap we have found in enterprise AI programs that believed they had addressed explainability.
- Explanation generation at inference time, not batch: Explanations must be generated when the decision is made and stored alongside the decision record. Batch-generating explanations retrospectively is operationally fragile and may not reflect the actual model state at the time of the decision.
- Human-readable translation layer: Raw SHAP values are not adequate for adverse action notices or individual rights responses. A translation layer that converts feature contributions into plain language with regulatory-compliant framing is required for customer-facing explanation systems.
- Explanation consistency monitoring: Monitor whether explanation characteristics drift over time independently of model performance. If the distribution of top adverse action reason codes shifts significantly, this is a signal that the model's decision logic has changed, potentially from data drift or from upstream feature pipeline changes.
- Override documentation: When a human reviewer overrides an AI recommendation, document both the original AI output with its explanation and the human's override rationale. This creates the accountability trail required for model validation and regulatory audit.
- Explanation audit trail: Maintain an immutable record of every explanation generated, including the model version, the input features, the output, and the explanation components. Retention periods vary by regulation but financial services typically require five years minimum.
- Proxy variable testing: Run regular tests for protected characteristic proxy effects in the features that drive explanations. A feature that correlates with a protected characteristic and appears in a majority of adverse action explanations is a regulatory red flag even if it is facially neutral.
For the complete model governance framework that includes explainability architecture, model lifecycle governance, and regulatory alignment, our Enterprise AI Governance Handbook covers all 56 pages of what enterprises in regulated industries need in place. You can also explore how explainability integrates into our AI governance advisory service and how it connects to the broader AI risk management framework that high-performing enterprises use.
