AI models, while powerful, are prone to bias, often leading to unfair outcomes for certain groups. Addressing these biases isn’t just about compliance; fair AI can enhance a company’s reputation, reduce legal risks, and improve business results. Yet, identifying and managing bias remains challenging for many organizations, especially those with limited resources.
In this guide, we’ll walk through how data scientists, artificial intelligence, and machine learning engineers can use Deeploy’s Python client & the Subgroup Harm Assessor application to audit models for bias, identify at-risk subgroups, and build more transparent, fair AI systems.
Why AI fairness Matters
Fairness in AI is essential for organizations. Unchecked bias can damage trust, harm individuals, and expose companies to legal risks. On the other hand, fair models can also drive better business results. For example, Harvard Business Review found that implementing fair credit scoring models led to over 150,000 more approved loan applications, cases that would have otherwise been rejected by biased human evaluators.
What makes an AI model fair
Defining “fair AI” is challenging because it depends on subjective factors like personal, cultural, and contextual biases. Also, fairness isn’t just a technical issue; it’s a sociotechnical problem, which requires discussion in both business and legal contexts.
From a technical standpoint, though, fairness can be viewed in terms of how bias translates into harms caused by the AI model. One of the most common types of harm relates to the “quality of service” the model provides, which means that each person based on whose data the model makes a prediction, should experience similar model performance regardless of which sensitive group they belong to.
These biases often come from human judgment embedded in the data or are amplified by the model’s architecture. To prevent ML models from causing harm, we should not only identify and measure the types of harm the application of the model can cause but also understand the biases that lead to these harms.
Measuring model fairness
To measure harm, we can use multiple metrics. While traditional performance metrics can be useful, it’s important to include specific fairness metrics. Compared to traditional metrics, fairness metrics put more emphasis on the distribution of outcomes across selected groups, for example, by evaluating rates of correctly identified (qualified) candidates between groups, rather than simply considering the performance of the model for each group.
The three primary fairness notions are demographic parity, equalized odds, and equal opportunity.
Unfortunately, we cannot simply evaluate and optimize for the 3 fairness notions at the same time, as they have been proven to be conflicting with each other. Practitioners need to carefully choose the most appropriate fairness concept and metric for each specific application.
For more information about the metrics, the differences between them, and why they are conflicting, we recommend the Fairness Handbook.
Auditing bias with Deeploy
Deeploy’s open-source Python client provides powerful tools to audit AI models and detect biases. In particular, you can use subgroup discovery algorithms to identify subgroups in which a model may behave differently, potentially leading to unfair or biased outcomes.
To follow along, read more on how to start using the Deeploy Python client here.
Detecting bias in input data
Subgroup discovery algorithms analyze input data to identify subgroups in which certain features or attributes are disproportionately represented. For instance, it might reveal that certain demographic groups (like older adults or specific ethnic groups) are underrepresented in training data, potentially leading to biased predictions. This helps pinpoint areas where the data itself could be perpetuating unfair outcomes.
deeploy_fair_lab = deepfair.DeeployFairLab(
x_test,
y_test,
y_pred,
sensitive_features
)
subgroups = deeploy_fair_lab.find_sub_groups(
quality_factor="accuracy",
depth=1,
min_quality=0.0001,
min_support=100
)
To allow developers to investigate subgroups in their data without uploading the data, we have added subgroup discovery functionalities to the Deeploy Python Client. A couple of simple lines of code shown on the right are sufficient to identify subgroups most impacted by model biases.
Before starting, ensure that you have installed Deeploy Python client with the “fair” extra dependency.
When using this technique, it’s crucial to choose the right evaluation metric. For this example, we used accuracy as a metric. While traditional ML performance metrics such as accuracy, recall, or AUROC can be used, fairness-specific metrics may be more appropriate. The Fairlearn library is one of the best sources to look for fairness metrics, which also includes practical guidelines on how to select the best metrics for your use case.
Once subgroups are identified with the above code, you can proceed with data exploration and analysis of the subgroups to identify potential reasons for the bias. For example, you can plot the distribution of the sensitive features divided into target classes to check how the distribution is different between subgroups.
Naturally, not all discrepancies are unjust, for example, in the case of hospital readmission prediction, people in an older population subgroup are more prone to be readmitted which can be a justified difference. In credit scoring applications, we can usually find that marital status is an important factor when making predictions about an individual’s suitability for a loan, but unsurprisingly, the population of married applicants tends to be older than average.
However, data distribution is not the only source of data bias, as it only corresponds to representation bias. When evaluating fairness of ML models, we should also consider predictive bias, which occurs when certain features are less informative to the model for a subgroup than for the full population. To investigate this type of data bias, one needs to utilize explainable AI techniques, as described in the following section.
Detecting model bias through explainability
Subgroup harm assessor
To identify biases in model predictions, not just those coming from input data, it’s essential to explain how the model makes decisions. The relationship between AI fairness and explainable AI can be complex, both in research and in practice. To address this, a recent project at Deeploy explored this intersection and developed an open-source tool called the Subgroup Harm Assessor.
The project was an extension of the subgroup discovery functionalities used in the Deeploy Python client showcasing how explanations can facilitate the fairness assessment process by highlighting the features that are the least informative.
To install and try out the Subgroup Harm Assessor, consult this repository.
The application is designed to automatically find and analyze underperforming subgroups in a dataset. Using selected quality metrics, it highlights where the model may not perform well for specific subgroups, such as the married population, and identifies which features are less informative for the subgroups and thus contribute the most to the underperformance.
Between the left side summarizing the model performance on the full dataset and the right side showing the performance evaluated on the data points for the selected subgroup, we can see the underperformance in terms of model uncertainty shown by the higher ratio of samples with a predicted probability around the decision boundary. When the model is unsure, it can more often make wrong predictions, resulting in the quality of service harms.
These remaining app pages help users investigate model underperformance by exploring performance and calibration details, and most importantly, highlight the model’s most and least informative features using SHAP loss explanations. By visually comparing these features between a specific subgroup and the full model, users can quickly and clearly assess performance across multiple metrics, making evaluations faster and easier to understand.
The research project discussed in this guide resulted in a demo-paper published at the well-known ECML Conference. To read more, see: “Subgroup Harm Assessor: Identifying Potential Fairness-Related Harms and Predictive Bias“.
About the author
Adam Dubowski
As a Research and Implementation Engineer, Adam works closely with customers to implement responsible AI solutions. His extensive background in research and commercial explainable AI solutions supports Deeploy's mission to drive meaningful impact through AI that is safe, transparent, and fair.
