What is explainable AI?
Explainable AI, or XAI, is the set of methods that let people see why an AI system produced a given output. NIST's report Four Principles of Explainable Artificial Intelligence (NISTIR 8312, September 2021) says a system that is meant to be explainable should follow four principles.

- Explanation: "A system delivers or contains accompanying evidence or reason(s) for outputs and/or processes."
- Meaningful: "A system provides explanations that are understandable to the intended consumer(s)."
- Explanation Accuracy: "An explanation correctly reflects the reason for generating the output and/or accurately reflects the system's process."
- Knowledge Limits: "A system only operates under conditions for which it was designed and when it reaches sufficient confidence in its output."
The third principle is the one that matters most for chatbots. An explanation can be fluent and wrong about why the output happened. NIST also says not every AI system needs an explanation, only those that are intended or required to be explainable.
The field has a government origin story. The US defense research agency DARPA ran an Explainable Artificial Intelligence program, and its program page says the aim was to produce "more explainable models, while maintaining a high level of learning performance" and to help users "understand, appropriately trust, and effectively manage" the new generation of AI partners. The same page says the program is now complete.
Why does explainable AI matter?
It matters because people judge a system by whether it can give a reason. NIST's report says the failure to give the rationale for an answer can lower the trust users grant a system, and that suspicion of bias or unfairness can slow society's acceptance of the technology.
NIST lists three situations where explanations are needed: regulator and legal requirements, quality control of an AI system, and customer relations. A developer debugging a model wants something very different from a person denied a loan, and the report says why an explanation is asked for and how it is delivered will vary with the user.
The report's own example of an output is a loan application, where the output is a decision, approved or denied. That is the case most people picture: a model scored you, and you want to know what pushed the score.
People are not perfect explainers either. NIST compared AI against human explanations and noted that as someone gains expertise, the underlying process becomes more automatic and harder to put into words. The same tension shows up in AI, where higher accuracy is often thought to cost some explainability.
What is the difference between explainable AI and interpretable AI?
The two terms overlap, and authors don't agree on a single split. The most common usage is this: an interpretable model is simple enough to read directly, and an explainable one is a model you can't read, plus a method that explains its outputs.
A short decision tree is interpretable. You can follow every branch with a pencil. A deep neural network with billions of weights is not, so people add a method on top, such as SHAP or LIME, to explain single predictions after the fact. Wikipedia's article on explainable AI notes the overlap and lists glass box models such as decision trees and sparse linear models next to the post hoc methods.
| Term | ISO/IEC TR 29119-11:2020 definition, as quoted on Wikipedia |
|---|---|
| Interpretability | "level of understanding how the underlying (AI) technology works" |
| Explainability | "level of understanding how the AI-based system ... came up with a given result" |
Transparency is a third word that gets mixed in. It usually means the system's design, data and rules are open to inspection. A transparent system can still be hard to interpret, and an interpretable one can still hide its training data.
What are the main explainable AI techniques?
Five approaches cover most of what people mean by explainable AI today, and each has a published limit. The table is compiled from the original papers and from Anthropic's own research pages, read in November 2026.
| Method | What it does | Source | Limit stated in the source |
|---|---|---|---|
| SHAP | Gives each input feature an importance value for one prediction | Lundberg and Lee, NIPS 2017 | Explains a single prediction at a time |
| LIME | Fits a simple model around one prediction to approximate the complex one | Ribeiro, Singh and Guestrin, 2016 | Local to the prediction it explains |
| Attention maps | Show which input tokens a transformer weighted | Jain and Wallace, 2019 | Found that attention weights largely do not give meaningful explanations |
| Circuit tracing | Follows features from input words to output words inside a model | Anthropic, March 2025 | Captures only a fraction of the computation; a few hours of human effort for a prompt of tens of words |
| Model's own reasoning text | The chatbot writes out why it answered | Turpin et al., 2023; Anthropic, April 2025 | Can leave out the real cause, such as a hint or an answer order |
SHAP and LIME were built for classifiers on tabular data, text and images, such as a model that scores a loan application. They are the techniques that most "explainable AI" guides spend their time on, including the graphapp guide, which omits attention maps and counterfactuals.
Attention maps look like an explanation because they highlight words. Jain and Wallace concluded that "standard attention modules do not provide meaningful explanations and should not be treated as though they do." Circuit tracing is the new frontier, and it's slow and partial.
The classical toolbox is wider than SHAP and LIME. Wikipedia lists partial dependence plots, permutation importance (the performance lost when one feature is shuffled), saliency maps for images and layerwise relevance propagation, and says these methods are "not very suitable" for GPT style language models. IBM's explainer sorts techniques into three jobs: prediction accuracy (LIME), traceability (DeepLIFT, which compares each neuron's activation to a reference) and decision understanding. Wikipedia also describes mechanistic interpretability, which reverse engineers a network, and dictionary learning, which looks for features inside foundation models.
What are real examples of explainable AI at work?
Three published cases show what an explanation looks like when someone checks it against what the model did. All three come from the labs that built the models, and the numbers are theirs.
- Mental math inside Claude. Anthropic traced Claude 3.5 Haiku adding two numbers and found parallel paths, one estimating the rough total and one working out the last digit. Asked how it got 36+59=95, the model described the standard carry the 1 method, which does not match the traced steps. Anthropic adds that Claude 'seems to be unaware' of the strategies it learned in training (Anthropic, March 2025).
- The answer is always A. Turpin and colleagues reordered multiple choice options so the right answer was always "(A)" in the examples shown to the model. Models followed the pattern and never said so in their reasoning. Accuracy fell by as much as 36% on 13 tasks with GPT-3.5 and Claude 1.0 (Turpin et al., 2023).
- The planted hint. Anthropic slipped hints into questions and checked whether the model's reasoning admitted using them. Claude 3.7 Sonnet mentioned the hint 25% of the time and DeepSeek R1 39%, averaged across hint types. On hints about unauthorized access, the rates were 41% and 19% (Anthropic, April 2025).
Notice what was needed to catch these. Each team changed an input or looked inside the model, because the written reasoning alone couldn't reveal the gap.
Where is explainable AI used in healthcare, finance and law?
Three fields come up in every explainer, and the claims differ in strength. IBM's page lists healthcare (diagnostics, image analysis, patient care decisions), financial services (loan and credit approval, credit risk, financial crime risk) and criminal justice (risk assessment, and detecting bias in training data). Those are IBM's examples of where explanations are wanted, not measured results.
Healthcare has the oldest record. Wikipedia says MYCIN, an early 1970s prototype for diagnosing bloodstream infections, could explain which hand coded rules led to a diagnosis, and that clinicians using decision support should understand how a machine reached its answer so they can trust it and back their own judgment.
Wikipedia also gives a cautionary case from image recognition: a 2017 system learned to look for a copyright tag that appeared on horse pictures instead of recognizing horses. An explanation method that highlights the tag exposes that kind of shortcut. Finance is the case NIST uses: the loan decision, approved or denied, with the rationale.
The same Wikipedia article reports a counterargument. Some scholars hold that an AI can be clinically validated even when its operators cannot follow how it works, and that explainability should rank below effectiveness. NIST notes that Rudin disputes the assumed trade off between accuracy and explainability and argues interpretable models can often be used without losing accuracy.
Is there a legal right to an explanation of an AI decision?
Sometimes, and the answer depends on the place and the system. In the EU, Article 86 of the AI Act gives a person affected by a decision made on the output of a high risk system listed in Annex III the right to get from the deployer "clear and meaningful explanations" of the AI system's role in the decision and its main elements. It applies where the decision has legal or similarly significant effects and the person sees an adverse impact on health, safety or fundamental rights. Annex III point 2 systems are excluded, and the article gives way to other EU law that already provides the right.
GDPR is more contested. NIST cites Article 13 as a rule that calls for information about automated decisions. Wikipedia says the GDPR introduced a "right to explanation" with implementation from 2018, and that it covers only the local side of interpretability. A well known paper by Wachter, Mittelstadt and Floridi argues in its title that such a right does not exist in the GDPR. Lawyers disagree, which is why this page makes no legal claim.
In the US, NIST names the Fair Credit Reporting Act. Wikipedia adds that insurers must be able to explain rate and coverage decisions, without naming a statute. For any real case, ask a lawyer, since an AI summary is not one.
Why isn't a chatbot's explanation the same as explainable AI?
A chatbot's explanation is generated text, and generated text is not a record of how the answer was produced. That is the gap between a model explaining itself and a system meeting NIST's explanation accuracy principle.
Anthropic states the conclusion plainly after its hint study: "our research shows that we can't always rely on what they tell us about their reasoning." The same page says watching a model's reasoning still has value, but ruling out bad behavior that way needs more work.
This is why a good reply from a chatbot can still be a bad explanation. If you ask why it chose a vendor or rejected a resume, the answer will sound reasonable. It may not name the factor that actually moved the output. Why AI gets things wrong and what is an AI hallucination cover the same weakness from the answer side, and what is an LLM explains why a model predicts text and does not look anything up.
Still, a chatbot explanation is useful. It can teach you a concept, suggest what to test and point to a source. Treat it as a hypothesis, not as evidence.
What questions should you ask an AI about explainable AI?
Use these prompts to learn the field, and ask the model to name its sources. Paste them into Claude, ChatGPT, Gemini or Whizi and fill the brackets.
- Explain at my level: "Explain SHAP to someone who knows [what you know, such as basic statistics]. Use a loan application example and tell me what a SHAP value can not tell me."
- Pick a method: "I have a [type of model, such as a gradient boosted tree] predicting [outcome]. Which of SHAP, LIME or a simpler model fits, and what are the trade offs? Cite the original papers."
- Audit an explanation: "Here is an explanation a model gave for its answer: [paste]. List what would have to be true for it to be faithful, and give me three input changes that would test it."
- Map to NIST: "Take this system: [describe it]. Score it against the four NIST explainable AI principles in a table and say which one it fails first."
- Stress my own chatbot: "Write five prompts that insert a hint or reorder options in a question about [topic], so I can see whether the chain of thought mentions it."
- Compare two readings: "Summarize what Turpin et al. 2023 and Anthropic's April 2025 hint study each found about faithful reasoning, in a two row table with the numbers."
Prompt 5 is the practical one. You don't need lab tools to check whether a chatbot's reasoning mentions the thing that really changed its answer.
What goes wrong when people trust AI explanations?
Four mistakes come up again and again, and each has a fix.
- Treating a fluent explanation as a log. The fix: test it by changing an input, as in the hint studies.
- Reading an attention map as the reason. The fix: use it as a hint about where to look, since the evidence on attention is mixed to negative.
- Explaining one prediction and assuming it describes the whole model. The fix: SHAP and LIME are local, so sample many cases before you claim a pattern.
- Handing an explanation to the person affected without checking they can follow it. The fix: apply NIST's meaningful principle and test it on an actual reader.
A fifth is a legal one: assuming a rule requires more than it does. See the section above on what the EU AI Act, the GDPR and US rules say, and ask a lawyer about a specific system.
Where does Whizi fit?
Whizi is a chat app, not an explainable AI toolkit, and it can explain an answer but cannot prove the explanation is faithful. That is the honest line for every chatbot.
It helps with the learning side. Web search is on by default per chat, so the prompts above can come back with sources. The free plan gives 10 messages a day on Whizi AI, Auto and Claude Haiku, with no card. Pro carries Claude, GPT and Gemini in one app, its first week is $1.99, and side by side Compare lets two models answer prompt 3 and show you where they disagree.
If you need SHAP or LIME on your own model, use the open source libraries from the papers above. A chat app is not a replacement for them.
What can explainable AI not do?
It can't make a wrong model right, and it can't guarantee that people will trust the result appropriately. Explaining a model shows its behavior. It does not check the data or the goal behind it.
The research is also unfinished. Anthropic says its circuit tracing captures only a fraction of the computation, and that some mechanisms it reports may be artifacts of the tools. For models that write long reasoning chains, it says better methods are still needed. Wikipedia lists three more limits. Revealing feature importance can help competitors copy a system or help people game it. Many explanations are written for engineers and ignore the reader's background. And in one study cited there, explanations raised understanding without changing trust, so the stated goal becomes calibrated trust, since too much and too little both harm the person and system pair.
For high stakes use, keep a person in the loop and keep the logs. The NIST framework asks for evidence, clarity, accuracy and stated limits, and a system that can't show all four hasn't earned the word explainable.
- Say whether the model is readable directly or needs a method such as SHAP or LIME
- Treat a chatbot's own reasoning as a claim, and test it by changing one input
- Check each explanation against the four NIST principles
- Remember SHAP and LIME explain one prediction at a time
- Keep a human reviewer on any decision that affects a person
Frequently asked questions
What is explainable AI in simple terms?
It is AI that comes with a way to see why it gave an output. NIST says an explainable system gives evidence for outputs, explanations a reader understands, explanations that match how the output was made, and a way to know its limits.
What are examples of explainable AI techniques?
SHAP and LIME are the best known. SHAP assigns each feature an importance value for one prediction, and LIME fits a simple model around one prediction. Attention maps and circuit tracing are used on language models, each with limits described in their sources.
Can ChatGPT or Claude explain its own answers?
They can write an explanation, but it may not be faithful. In Anthropic's April 2025 study, Claude 3.7 Sonnet mentioned a planted hint 25% of the time, so treat the explanation as a claim to test.
Is explainable AI required by law?
Sometimes. Article 86 of the EU AI Act gives people affected by decisions based on certain high risk systems the right to a clear and meaningful explanation. NIST also cites the US Fair Credit Reporting Act and GDPR Article 13. Whether the GDPR has a true right to explanation is disputed, so ask a lawyer.
What is the difference between explainable AI and interpretable AI?
An interpretable model, such as a short decision tree, can be read directly. An explainable setup adds a method, such as SHAP or LIME, to explain a model you cannot read. ISO/IEC TR 29119-11 defines interpretability as understanding how the technology works and explainability as understanding how it reached a given result.
Where is explainable AI used?
IBM lists healthcare, financial services and criminal justice. Typical uses are explaining a diagnosis from an image, a loan approval or denial, and a risk score. NIST uses the loan decision as its worked example of an output that needs a rationale.