In this post, we’ll explore bias in AI-based decision-making and how organizations can prevent it. This is part of our series on what’s new in SMARTS™ Washington.
Bias in Human Decision-Making
Bias in decision-making is nothing new. Research in psychology, neuroscience, behavioral economics, and other related fields have shown that biases enable humans to navigate an ever increasingly complex world when attention and energy are in short supply. Here are examples of cognitive biases or systematic shortcuts that we take to process information and make decisions:
- Anchoring bias: We tend to put more weight on the first piece of information that we receive. It’s why first impressions really do matter!
- Confirmation bias: We tend to prioritize new information that supports our existing beliefs. It’s why the old dog doesn’t want to learn new tricks.
- Negativity bias: We tend to place more significance on negative events than positive ones. It’s what enabled our ancestors to survive in harsh conditions.
Bias underlies not only our values and interests, but also our stereotypes and prejudices. At best, bias has enabled us to thrive and has helped shape our identities. At worst, bias prevents us from taking “good” risks and leads to decision outcomes that are morally wrong and/or illegal. For example, even after the Civil Rights Act of 1964, redlining was a common practice to deny Black Americans and other minorities access to credit. In fact, the Federal Housing Administration was largely responsible for institutionalizing and supporting this practice for mortgage loans until the 1968 Fair Housing Act (which outlawed redlining). Today, the Equal Credit Opportunity Act prohibits creditors from discriminating applicants based on race, color, religion, national origin, sex, and other factors.
From Humans to Machines
AI can help avoid adverse human biases, and one of the many reasons why AI is becoming more popular in supporting and, in some cases, replacing human decision-making. While humans are prone to superstition, AI can separate the signal from the noise. Humans get hangry, distracted, and tired (which often clouds judgment), while AI doesn’t. And while humans can only process so much information, AI can process “endless” amounts and therefore can identify patterns and connections that no one human can. Altogether, AI promises more accuracy and consistency in decision-making. That being said, AI is not free of bias. In certain cases, bias is a good thing. For example, recommendation algorithms are typically designed to place more value on recent consumption, purchases, and viewing history.
However, AI can also perpetuate adverse human biases at scale. Sampling errors and bias in the training data are a common cause for AI bias (aka machine learning bias or algorithmic bias). For example, a machine learning credit model, trained on mortgage applications pre-1968 would discriminate against non-whites and women even if the model wasn’t explicitly designed to do so. This is because majority, if not all, of the applications from women and non-white men in the training set would have been declined. Similar historical discrimination exists in schools and organizations. Therefore models for admissions and hiring are also prone to carry on that discrimination. That being said, having “perfect” training data is not enough. Adverse bias can still be introduced during modeling and review.
Better Decisions Through Fairness
Fairness is one of the emerging ways to address AI bias. Generally speaking, fairness is the absence of preferential or prejudicial treatment towards an individual or group. Several different AI researchers and tech companies have come up with various techniques and tools to measure fairness. Here are a few examples.
Disparate impact: This metric compares the positive outcomes between two groups. You calculate it by taking the ratio of the proportion of the positive outcomes of an unprivileged, minority, or monitored group to the proportion of the positive outcomes of the privileged, majority, or reference group. 1 = fair. Less than 1 indicates a bias in favor of the privileged group. In general, less than 80% is considered a violation. Statistical party difference is a similar metric but is calculating by subtracting the proportion of the privileged from the proportion of the unprivileged (0 = fair).
Equalized odds: This metrics compares the false positives and false negatives between two groups. In a nutshell, this metric assesses how likely a model makes an error in one group over another. A model that has a higher false positive rate for the privileged group is biased towards the privileged group. Conversely, a model that has a higher false negative rate for the unprivileged group is also biased against the unprivileged group. Equalized opportunity difference is a related metrics that compares true positives between two groups.
The challenge with fairness is that there’s no universal definition and, therefore, no definitive way to measure it. The same model could be fair according to one metric and unfair according to another. However, when defining fairness in light of regulations and company values, policies, and strategies, fairness metrics can be a powerful tool.
Fairness Metrics in SMARTS™
SMARTS™ is a decision management platform designed for business analysts. Our BluePen module enables domain experts to explore data by calculating statistics and correlations. They can then use those insights to build an executable model. These models can supplement any imported data science model and be integrated into a decision flow. This enables business analysts to manage the entire decision lifecycle, including any models, in one place. With SMARTS™ Vienna, we enhanced BluePen by adding Interactive Tree Model, a tool that enables business analysts to interactively build decision trees in a completely transparent manner.
And now, with SMARTS™ Washington, we further enhanced BluePen by adding fairness (and explainability) metrics. Business analysts can now calculate fairness metrics on any BluePen model. BluePen includes numerous fairness metrics, including the examples mentioned above, from which business analysts can choose from (select all that apply). After selecting the metrics, business analysts simply need to select the inputs and target and SMARTS™ will run the calculations and display the results in table form. Fairness metrics enable business analysts to apply the same rigor that data scientists do in uncovering hidden biases in the models that they build.
In our next post, we will discuss how explainability and our explainability metrics can also mitigate AI bias.
Learn more about Sparkling Logic’s SMARTS™ Data-Powered Decision Manager.