Skip to content

Easy Open Source ML Monitoring

Boxkite logo

Boxkite makes it easy to monitor ML models in production and integrate into your existing workflows. It aims to be simple, flexible, and reliable.

Boxkite simplifies model monitoring by capturing feature and inference distributions used in model training, then comparing them against realtime production distributions via Prometheus and Grafana:

Grafana dashboard

Prometheus and Grafana are the de facto DevOps solution for storing, querying, visualising, and alerting on monitoring data. Boxkite integrates directly with this stack, so as to provide ML model observability as a lightweight extension of existing DevOps tooling.

Besides visualising data distributions, Boxkite also includes PromQL queries to calculate divergence metrics, such as KL Divergence to give you a measure of how different your production traffic is from your training data. This allows you to measure model and data drift more objectively, as well as set up alerting rules.

Demo!

Goals

Boxkite is an instrumentation library designed from ground up for tracking concept drift in HA (Highly Available) model servers. It integrates well with existing DevOps tools (ie. Grafana, Prometheus, fluentd, kubeflow, etc.), and scales horizontally to multiple replicas with no code or infrastructure change.

Fast

  • 0.5 seconds to process 1 million data points (training)
  • Sub millisecond p99 latency (serving)
  • Supports sampling for large data sets

Correct

  • Aggregates histograms from multiple server replicas (using PromQL)
  • Separate counters for discrete and continuous variables (ie. categorical and numeric features)
  • Initialises serving histogram bins from training data set (based on Freedman-Diaconis rule)
  • Handles unseen data, nan, None, inf, and negative values

Simple

  • One metric for each counter type (no confusion over which metric to choose)
  • Default configuration supports both feature and inference monitoring (easy to setup)
  • Small set of dependencies: prometheus, numpy, and fluentd
  • Extensible metric system (support for image classification coming soon)

Non-goals

Some non-goals of this project are:

  • Adversarial detection

If you are interested in alternatives, please refer to our discussions in FAQ.

Getting Started

Follow one of our tutorials to easily get started and see how Boxkite works with other tools:

See Installation & User Guide for how to use Boxkite in any environment.

Contributors

The following people have contributed to the original concept and code

A full list of contributors, which includes individuals that have contributed entries, can be found here.

Shameless Plug

Boxkite is a project from BasisAI, who offer an MLOps Platform called Bedrock.

Bedrock helps data scientists own the end-to-end deployment of machine learning workflows. Boxkite was originally part of the Bedrock client library, but we've spun it out into an open source project so that it's useful for everyone!