In this post, we’ll explore explainability in AI-based decision-making and how it contributes to the overall understandability of automated decisions. This is part of our series on what’s new in SMARTS™ Washington.
What is Explainability?
Explainability, broadly speaking, is the ability for humans to describe why an AI system reached a specific outcome (prediction, recommendation, decision). While non-academics tend to interchange explainability with interpretability, they are actually different. Interpretability is the ability for humans to describe how an AI system reached a specific outcome. Both are in response to the “black box” or opaque nature of the machine learning models in an AI system. Lack of transparency leads to mistrust and risk of adverse AI bias.
Explainable AI (XAI) is a discipline that explores methods for explainability. Descriptions can be global (explaining the model in general) or local (explaining why a specific set of input data led to a certain outcome). Here are a couple examples:
- SHapley Addictive exPlanations (SHAP): SHAP is a method that is based on game theory and assigns an importance value to each feature in a model. The greater the magnitude, the more important the feature. Negative values indicate a negative impact. SHAP can be used on any machine learning model (linear regressions, decision trees, random forests, etc.) and can provide both global and local explanations.
- Local Interpretable Model-agnostic Explanations (LIME): LIME is method that creates a good approximate “glass box” model for a given outcome in the original model. In a nutshell, LIME creates synthetic input data, runs it through the original model, and uses the outcomes to train the new model. LIME can be used on any machine learning model but can only provide local explanations. However, as the name suggests, LIME offers both explainability and interpretability.
Explainability Metrics in SMARTS™
Decision management platforms like SMARTS™ enable business analysts to manage decision logic for automated decisions in one place. This logic can be explicit (business rules) or implicit (machine learning models). Most of the time, data scientists develop the machine learning models, and they have their own tools for explainability (and interpretability). SMARTS™ provides different methods for operationalizing machine learning once the models have been finalized.
However, there may be cases when business analysts want to create their own models to complement data science ones. SMARTS™ BluePen enables business analysts to explore data and build an executable model. In addition, SMARTS™ BluePen’s Interactive Tree Model tool enables business analysts to interactively build decision trees in a completely transparent manner. With SMARTS™ Washington, we added explainability (and fairness) metrics. Business analysts can now run explainability algorithms such as SHAP and LIME on any BluePen model. After selecting the algorithm, business analysts simply need to select the model(s), inputs, and target. SMARTS™ will run the algorithm and display the results in table form. Through these explainability metrics, business analysts can better understand the models that they build.
From Explainability to Understandability
However, explainability has its limits (and why interpretable AI developed in response). Understandable AI takes both XAI and interpretable AI a step further by focusing on AI accessible, transparent, and understandable to all stakeholders. Making decisions, more broadly, understandable has always been the goal at Sparkling Logic. In our next post, we will discuss our approach to Understandable Decisions and how our new Repository Time Travel contributes to understandability.
Learn more about Sparkling Logic’s SMARTS™ Data-Powered Decision Manager.