To explain a machine learning model well, begin with the stakeholder’s decision—not with the explanation tool. A data scientist may need global behaviour, residuals and stability tests. An operational user needs to know how to use an output and when to escalate. A person affected by a decision needs understandable reasons, limitations and a route to challenge it. An auditor needs traceable evidence that the explanation is faithful and governed.
These are not different versions of the truth. They are different views of the same verified evidence. Good explainability preserves accuracy while changing depth, language and presentation. It never turns correlation into causation, conceals uncertainty or implies that a colourful feature chart proves a model is fair.
Key takeaways
- Define the audience, decision and consequence before selecting an explanation method.
- Separate how the model generally behaves from why one particular output occurred.
- Test explanations for fidelity, stability and usefulness; plausible is not the same as faithful.
- Pair reasons with limitations, human oversight and a route for correction or challenge.
- Maintain one evidence base so that audience-specific explanations remain consistent.
What does it mean to explain a machine learning model?
An explanation is information that helps a defined person understand an output, process or system well enough to take an appropriate action. Depending on the purpose, it may describe influential features, rules, typical behaviour, uncertainty, data limits, an individual prediction or a change that could alter the result.
The NIST Four Principles of Explainable AI provide a useful quality test: a system should provide reasons, make them meaningful to the user, ensure that the explanation correctly reflects the process, and recognise its knowledge limits. The principles show why one generic dashboard cannot serve every audience.
Three distinctions prevent many communication errors:
- Global versus local: global analysis describes overall model behaviour; local analysis focuses on one output.
- Intrinsic versus post-hoc: an interpretable model exposes understandable structure directly; a post-hoc method approximates or attributes behaviour after prediction.
- Association versus causation: a feature’s contribution to a prediction does not prove that changing it will cause the predicted outcome to change in the real world.
Match the explanation to the stakeholder
| Stakeholder | Decision they need to make | Useful explanation | Evidence to retain |
|---|---|---|---|
| Affected person or customer | Understand, respond to or challenge an outcome | Plain-language reasons, important factors, limits and recourse | Case data, output, explanation version and review route |
| Operational user | Use the output safely | Prediction meaning, confidence limits, warning signs and escalation | User guidance, thresholds, exceptions and overrides |
| Manager or executive | Approve, resource or stop the system | Purpose, benefit, material risks, performance range and control ownership | Validation summary, impact assessment and monitoring decisions |
| Model developer | Diagnose and improve behaviour | Global effects, local attributions, residuals, slices and stability tests | Code, data lineage, model version and experiments |
| Validator or auditor | Challenge evidence and governance | Method assumptions, fidelity, robustness, limitations and independent tests | Reproducible artefacts, approvals, issues and remediation |
| Regulator or oversight body | Assess compliance and accountability | System purpose, affected groups, instructions, human oversight and traceability | Technical documentation, logs and accountable decisions |
The EPW CLEAR explanation framework
CLEAR is an original EPW framework for planning an audience-specific explanation without losing technical integrity. Complete it before choosing SHAP, LIME, a decision tree, counterfactuals or another technique.
C — Context and consequence
State what the model does, where it sits in the process and what happens after its output. Is it recommending a document, prioritising a maintenance inspection or influencing access to a service? Record whether a human decides, how quickly action occurs and what a wrong result could cost.
Consequence determines depth. A low-impact recommendation may need simple disclosure and controls. A consequential decision affecting rights, safety or livelihood requires stronger validation, documentation, human oversight and routes for contestability.
L — Listener and decision
Name the listener and the action the explanation must support. “The business” is too broad. A service agent deciding whether to escalate and a director deciding whether to renew a system need different information. Include the person affected by the decision rather than treating explanation as an internal technical exercise.
Ask what the listener already knows, what terminology is safe, whether they need an individual reason or system overview, and what misunderstanding would be harmful. Accessibility, language and time available are part of explanation design.
E — Evidence and method
Choose evidence appropriate to the question. Global feature effects, partial dependence and performance by subgroup can support system-level review. Local attribution or a counterfactual may help examine one prediction. Examples and prototypes can support user research, but they do not replace fidelity tests.
Post-hoc methods have assumptions. LIME fits a local surrogate around a prediction, so results can depend on sampling and neighbourhood choices. SHAP allocates contributions using a Shapley-value framework, but correlated features and background data affect interpretation. Explain the method’s limits and compare more than one view when the decision is material.
A — Accuracy and limits
An explanation must be accurate about what it claims. If a chart says a feature contributed to a model score, do not translate that into “this factor caused the outcome”. Test whether small irrelevant changes produce unstable explanations, whether an attribution agrees with model behaviour and whether the explanation remains useful across representative cases.
Report uncertainty and knowledge limits. A model may be outside its validated population, missing critical data or encountering a new operating condition. An honest explanation sometimes says that the system does not have enough evidence and that a person must review the case.
R — Response and recourse
End with what the stakeholder can do. An operational user may verify a source or escalate. An affected person may correct information, ask for human review or challenge the outcome. A model owner may pause a segment, investigate drift or roll back a version.
The joint ICO and Alan Turing Institute guidance distinguishes process-based and outcome-based explanations and focuses on people affected by AI-assisted decisions. Its practical lesson is that reasons, responsibility and recourse belong together.
How to choose an explanation method
| Question | Possible method | Best use | Important caution |
|---|---|---|---|
| How does the model behave overall? | Interpretable model, feature effects, permutation importance, PDP, ICE or ALE | Development, validation and governance | Aggregate behaviour can hide subgroup or local failures |
| Why did this prediction receive this score? | Local attribution, local surrogate or case comparison | Case review and diagnosis | Attribution may be unstable or sensitive to background data |
| What change is associated with a different result? | Counterfactual explanation | Recourse design and scenario analysis | A mathematically feasible change may be impossible, unfair or non-causal |
| What visual evidence influenced a deep model? | Saliency, activation or concept methods | Technical inspection | A persuasive heat map is not proof of faithful reasoning |
| Can a user apply the explanation correctly? | Comprehension and task-based user testing | Interface and communication design | User confidence can increase even when understanding is wrong |
Prefer a simpler intrinsically interpretable model when its performance and constraints satisfy the problem. Complexity should earn its place through material benefit. Where a post-hoc explanation is necessary, preserve the difference between explaining the model and explaining the real-world event.
Worked example: a service-priority model
Imagine a model that prioritises service cases for specialist review. It uses case type, reported impact, previous contact and document completeness. A local attribution indicates which recorded inputs most influenced one score. The same evidence can be communicated in several ways.
| Audience | Appropriate explanation | What not to say | Next action |
|---|---|---|---|
| Customer | “Your case was prioritised mainly because the recorded impact was high and required documents were complete.” | “Those factors caused your underlying problem.” | Offer correction and human review if data is wrong |
| Service adviser | Show priority, influential recorded factors, uncertainty and escalation triggers | Present the score as a mandatory decision | Verify the record and override with a reason where policy permits |
| Manager | Show queue outcomes, error rates, subgroup checks, overrides and capacity impact | Use one favourable example as proof of system value | Review thresholds and resource effects |
| Validator | Reproduce local attribution; test correlated inputs, perturbations and alternative methods | Accept the chart because it looks intuitive | Record stability, fidelity and limitations |
This design gives each audience what it needs while preserving traceability. The customer explanation is concise but includes recourse. The adviser sees operational limits. The manager sees system-level outcomes. The validator receives enough detail to challenge the method.
Test whether an explanation is good enough
Evaluate the explanation as a product in its own right. At minimum, test five properties:
- Fidelity: does the explanation accurately reflect the model behaviour it claims to describe?
- Stability: do similar cases receive reasonably consistent explanations, and are differences justifiable?
- Comprehension: can the intended audience interpret the explanation correctly?
- Usefulness: does it help the stakeholder make the required decision or take the right action?
- Coverage: is an explanation available for the cases and conditions where it is required?
Use task-based questions rather than asking only whether users “like” the explanation. Can they identify when to escalate? Can they distinguish a model factor from a causal claim? Do they know how to correct data? Measure incorrect confidence as well as reported trust.
For systems used in the EU, the EU Artificial Intelligence Act establishes risk-based obligations, including transparency and information requirements in specified contexts. Explainability work should be mapped to the system’s actual classification, role and jurisdiction with qualified legal advice; a generic feature-importance chart is not a compliance programme.
Common explanation mistakes
- Starting with a tool: method choice comes after audience and decision requirements.
- Confusing importance with cause: prediction contribution does not establish real-world causality.
- Explaining one case as if it described the whole model: local and global questions require different evidence.
- Using plausibility as validation: an intuitive story can still be unfaithful or unstable.
- Hiding uncertainty: confident presentation can encourage automation bias.
- Omitting recourse: reasons without a correction or review route leave affected people powerless.
- Creating inconsistent narratives: leadership, user and audit explanations must trace to the same evidence.
Model explanation checklist
- Is the model’s purpose, context and consequence clear?
- Is each stakeholder and required decision named?
- Does the explanation distinguish global, local, intrinsic and post-hoc evidence?
- Are assumptions, uncertainty and knowledge limits explicit?
- Has fidelity and stability been tested on representative cases?
- Can intended users interpret the explanation correctly and act safely?
- Is there a documented route to correct data, escalate or challenge an outcome?
- Can every explanation be reproduced from the model, data and method version?
Develop practical explainability capability
EPW’s five-day Explainable AI and Model Interpretability Techniques course covers stakeholder requirements, interpretable models, global analysis, LIME, SHAP, counterfactuals, deep-model techniques, fidelity and stability testing, model cards, governance and audience-specific communication.
Professionals can also explore EPW’s Artificial Intelligence and Machine Learning Courses and the AI and machine learning article hub. To apply CLEAR to a real model, review the course outline, available dates and locations, or request tailored course details.
Sources and references
- National Institute of Standards and Technology, Four Principles of Explainable Artificial Intelligence, NISTIR 8312, 2021.
- Information Commissioner’s Office and The Alan Turing Institute, Explaining decisions made with AI, accessed 1 September 2026.
- European Union, Regulation (EU) 2024/1689: Artificial Intelligence Act, current consolidated text accessed 1 September 2026.
- Ribeiro, Singh and Guestrin, “Why Should I Trust You?”: Explaining the Predictions of Any Classifier, 2016.
- Lundberg and Lee, A Unified Approach to Interpreting Model Predictions, 2017.
- EPW Training, Explainable AI and Model Interpretability Techniques Course, accessed 1 September 2026.
