When monitoring machine learning models, teams increasingly need visibility into custom metrics, like fairness, bias, or business-specific KPIs.
That’s why we’ve introduced custom monitoring metrics in Deeploy. This new feature lets you define and visualize your own performance indicators, tailored to your unique goals and use cases.
- Real-world alignment – Teams can track what matters most to their business or use case (e.g. fairness, sustainability, or domain-specific KPIs), not just standard accuracy or F1 scores.
- Regulatory readiness – With laws like the EU AI Act demanding accountability and transparency, custom metrics let teams prove compliance by tracking ethical, legal, or risk-related metrics.
Uploading your own custom metrics
You can now add up to four custom metrics per model deployment in Deeploy. These metrics appear in dedicated charts, alongside your standard monitoring views.
To get started:
- On the custom tab of monitoring, click add metric and provide the necessary configuration
- Use the Deeploy Python client or API to push metric data to the dedicated endpoint
- Each custom metric you define will appear in a dedicated chart within the Deeploy UI.
For the complete implementation details, see our documentation.
Practical use case: Monitor bias in credit risk scoring
Let’s say you’ve trained a credit risk model with fairness in mind, working to reduce bias. Even with careful training, biases can still emerge once the model is in production.
With custom metrics, you can keep track of these fairness concerns in an ongoing, structured way. For example:
- Subgroup accuracy – Measure how well the model performs for different subsets.
- Representation – Track how often each subgroup appears in incoming applications.
- Outcome parity – Compare approval rates across subgroups to detect disparities.
By uploading these metrics, teams can proactively monitor fairness alongside operational metrics. This gives data scientists and responsible AI teams a clear signal when intervention is needed, before small issues become big problems.
Other use cases for custom metrics
Bias and fairness are just one use case. We’ve seen customers also use custom metrics to track:
- Model running costs for machine learning and large language models
- Model latency or energy usage for sustainability
- Approval/rejection rates tied to business impact
- Metrics specific to regulatory compliance in healthcare or finance
Every challenge is unique, and this flexibility gives you the control to define what “good” looks like in your context. Whether you’re tracking bias, regulatory KPIs, or operational thresholds, custom metrics give you the control, visibility, and flexibility you need to run responsible AI at scale.
