Deeploy helps you optimise AI system and model performance, and detect issues early before problems escalate.
With comprehensive tracking and custom alerts, you’ll detect issues early and ensure your models remain compliant, accurate, and reliable.
Monitor operational robustness and AI performance with real-time tracking of metrics tailored to classification, regression, LLMs, and more. Catch performance bottlenecks, system issues, and model degradation before they affect end users.
Ensure meaningful human oversight by capturing and monitoring feedback from domain experts and end-users, or detect when model inputs deviate from expected distributions through statistical drift monitoring.
Track domain-specific performance indicators by creating custom metrics tailored to your business needs, and visualise them alongside standard metrics.
Stay ahead of issues with customisable alerts via email, Slack, or webhook, that notify your team the moment metrics cross defined thresholds.
Maintain full visibility and accountability in agentic workflows and LLM chains by tracing every step, ensuring you can explain and audit how your AI systems reach their outputs.
Protect your AI deployments with guardrails that automatically detect and replace sensitive patterns like emails, prompt injections, and malicious scripts.
Add alerts to notify teams instantly and trigger reviews when violations occur.
Build trust and transparency; add explainability to any model with built-in explainers like SHAP and saliency, or bring your own trained explainer.
Monitor organisation-wide AI governance at a glance with a unified dashboard showing compliance status, risk levels, and framework progress across all use cases and teams.
Start monitoring your AI today
Discover how Deeploy can be applied in various industries
Deeploy monitors metrics across four categories. Traffic metrics cover activity (prediction volume), errors (failed requests), and average response time.
Performance metrics are available once you supply actuals and vary by problem type: classification models get accuracy, precision, recall, and F1; regression models get RMSE, MAE, MAPE, and R-squared; and text generation models get BLEU score.
Evaluation metrics track human oversight through disagreement ratios, showing how often domain experts or end-users disagree with model outputs.
Drift metrics use Jensen-Shannon divergence to detect when live input data starts deviating from the expected baseline distribution, signaling potential model degradation before it affects downstream outcomes.
Deeploy includes a built-in evaluation mechanism that lets domain experts and end-users submit feedback on individual predictions, marking whether they agree or disagree with the model’s output. This feedback is aggregated into two monitoring metrics: the overall disagreement ratio (the share of evaluated predictions that were flagged as wrong) and disagreement per class (how those disagreements are distributed across outcome classes).
For regulatory purposes, this creates a documented feedback loop that serves as evidence of meaningful human oversight, a requirement for high-risk AI systems under the EU AI Act.
AI agents are inherently harder to oversee than single-model deployments, a single user request can trigger a chain of decisions, tool calls, and model invocations that are invisible without the right instrumentation.
Deeploy’s tracing feature gives you step-by-step visibility into exactly what happened inside an agentic workflow: every operation is captured as a named span within a trace, so you can see the full sequence of steps an agent took, where latency was introduced, and where failures occurred. This matters for governance because regulators and auditors don’t just want to know what output an AI system produced, they want to understand how it got there.
Traces are stored for 90 days and viewable directly on the Monitoring page of your deployment, giving compliance and AI teams a shared, auditable record of agent behaviour that can be referenced during reviews or investigations.
Yes. Deeploy’s custom metrics feature lets you define and visualise any domain-specific performance indicator that isn’t covered by the standard metric set.
You create a custom metric by giving it a name, choosing a graph type (line or bar), and defining axis labels and units. Data is then pushed to the metric’s dedicated endpoint from your own systems or pipeline, in batches of x/y value pairs. You can optionally link data points to specific prediction log IDs for traceability. Custom metrics appear on the same monitoring dashboard as standard metrics, giving teams a unified view of both technical model health and business-level KPIs.
Alerts in Deeploy notify your team the moment a metric crosses a defined threshold. Alert rules can be configured for any standard metric, including traffic, performance, drift, and input validation, with the exception of disagreement per class and custom metrics.
Each rule lets you set a severity level (warning or critical), a threshold (minimum, maximum, or range depending on the metric), a rolling time window for evaluation, a minimum sample size to avoid false positives on sparse data, and a cooldown period to suppress repeated notifications.
Notifications are delivered via email, in-product alerts, Slack, or any webhook you configure. Multiple rules can be stacked on the same metric with different severity levels to create a tiered alerting system.
Guardrails are regex-based filters applied to model inputs, outputs, or both at inference time, automatically detecting and replacing sensitive or unwanted patterns before they reach the model or are returned to the user.
Deeploy ships with four default patterns: email addresses, prompt injection attempts, inline HTML event handlers, and script tags. You can also define custom regex patterns to match anything specific to your use case.
Guardrails are configured at the Workspace level and applied per deployment. When a guardrail fires, the original unfiltered content is recorded in the prediction log for audit purposes, while the model and downstream systems only ever see the sanitised version. Alerts can be added to notify teams when violations occur.
Deeploy supports both standard explainers, which require no training and are applied automatically based on model type, and custom trained explainers you bring yourself.
Standard explainers include Tree SHAP for tree-based classification models (XGBoost, Scikit-learn, LightGBM, CatBoost), and Saliency and Attention for text generation models on Hugging Face.
Trained explainer frameworks include SHAP kernel, Anchors, MACE, and PDP.
Explanations can be requested through the UI test page, the Deeploy API, or the Python client.
For supported explainer types, Deeploy automatically generates visual explanations, viewable per prediction on the Predictions page, which can be used as evidence of model transparency when addressing EU AI Act controls around explainability and human oversight.
The AI governance dashboard gives compliance leads and AI program owners a single, organisation-wide view of AI governance health without having to dig into individual use cases or workspaces.
At a glance it shows the total number of use cases, active deployments, and controls in play across the organisation, alongside four key breakdowns: use cases by risk classification (unacceptable, high, limited, minimal, and unclassified), use cases by lifecycle stage (exploration through to production), overall controls progress (completed, in progress, and not started), and use cases by applied framework, showing which frameworks are in use and how many use cases under each have completed their controls.
A time-series chart also tracks how quickly new use cases are being added over the last 30 days, giving leadership a sense of how fast the AI portfolio is growing relative to governance capacity.
Together these views make it easy to spot concentration risks, for example, a large share of unclassified or high-risk use cases still in exploration, and to report on governance programme progress to regulators or the board.
The EU AI Act requires organisations to maintain ongoing oversight of high-risk AI systems, not just assess them once and move on.
Deeploy’s monitoring layer directly supports this through several interconnected mechanisms: performance metrics and drift monitoring detect degradation and distributional shifts that could signal the model is no longer fit for its approved use; human evaluation tracking documents that meaningful oversight is taking place; alerts create a response mechanism when thresholds are breached; guardrails enforce runtime safety controls; and tracing provides auditability for complex agentic systems.
All monitoring activity is tied back to the use case and its control framework, meaning that sustained compliance, not just initial sign-off, is built into the operational workflow.
