Easy Open Source ML Monitoring
Boxkite makes it easy to monitor ML models in production and integrate into your existing workflows. It aims to be simple, flexible, and reliable.
Boxkite simplifies model monitoring by capturing feature and inference distributions used in model training, then comparing them against realtime production distributions via Prometheus and Grafana:
Prometheus and Grafana are the de facto DevOps solution for storing, querying, visualising, and alerting on monitoring data. Boxkite integrates directly with this stack, so as to provide ML model observability as a lightweight extension of existing DevOps tooling.
Besides visualising data distributions, Boxkite also includes PromQL queries to calculate divergence metrics, such as KL Divergence to give you a measure of how different your production traffic is from your training data. This allows you to measure model and data drift more objectively, as well as set up alerting rules.
Demo!
Goals
Boxkite is an instrumentation library designed from ground up for tracking concept drift in HA (Highly Available) model servers. It integrates well with existing DevOps tools (ie. Grafana, Prometheus, fluentd, kubeflow, etc.), and scales horizontally to multiple replicas with no code or infrastructure change.
Fast
- 0.5 seconds to process 1 million data points (training)
- Sub millisecond p99 latency (serving)
- Supports sampling for large data sets
Correct
- Aggregates histograms from multiple server replicas (using PromQL)
- Separate counters for discrete and continuous variables (ie. categorical and numeric features)
- Initialises serving histogram bins from training data set (based on Freedman-Diaconis rule)
- Handles unseen data,
nan
,None
,inf
, and negative values
Simple
- One metric for each counter type (no confusion over which metric to choose)
- Default configuration supports both feature and inference monitoring (easy to setup)
- Small set of dependencies: prometheus, numpy, and fluentd
- Extensible metric system (support for image classification coming soon)
Non-goals
Some non-goals of this project are:
- Adversarial detection
If you are interested in alternatives, please refer to our discussions in FAQ.
Getting Started
Follow one of our tutorials to easily get started and see how Boxkite works with other tools:
- Prometheus & Grafana in Docker Compose
- Kubeflow & MLflow on Kubernetes with easy online test drive in the browser
See Installation & User Guide for how to use Boxkite in any environment.
Contributors
The following people have contributed to the original concept and code
A full list of contributors, which includes individuals that have contributed entries, can be found here.
Shameless Plug
Boxkite is a project from BasisAI, who offer an MLOps Platform called Bedrock.
Bedrock helps data scientists own the end-to-end deployment of machine learning workflows. Boxkite was originally part of the Bedrock client library, but we've spun it out into an open source project so that it's useful for everyone!