Grafana & Prometheus
This tutorial uses docker-compose locally to demonstrate training a model, recording its training data histogram, then deploying it with a simple Flask server and then observing the model's divergence in Grafana & Prometheus.
Clone repo
git clone https://github.com/boxkite-ml/boxkite
cd boxkite/examples/grafana-prometheus
Setup
Initialise a python3
virtual environment and activate it. Then install the training dependencies.
pip install -r requirements.txt
pip install -r app/requirements.txt
Train
The training script generates a linear regression model using sklearn's diabetes dataset.
python3 train_completed.py
Running the command above creates two files in the current directory: histogram.prom
and model.pkl
.
Serve
The serving script creates a flask server that uses the trained model.
python3 app/serve_completed.py
You can test the server by sending a HTTP request using curl
.
curl localhost:5000 -H "Content-Type: application/json" \
-d "[0.03, 0.05, -0.002, -0.01, 0.04, 0.01, 0.08, -0.04, 0.005, -0.1]"
PromQL Metrics
Tip
Make sure you have generated histogram.prom
and model.pkl
in your current directory before continuing.
Start the serving container with Prometheus using docker-compose
. It automatically scrapes the flask server every 15 seconds for feature distribution metrics.
docker-compose up
Install requests
library and call metrics/load.py
to generate some load.
pip install requests
python3 metrics/load.py
Navigate to http://localhost:3000 for the Grafana GUI, then login with admin
and admin
, skip changing the admin password, then navigate to Dashboards -> Manage -> Model Metrics: