Annotating Change Points in Grafana

Change points found by analyze can be exported as Grafana annotations using the --update-grafana flag:

$ otava analyze <test or group> --update-grafana

The annotations generated by Otava get the following tags:

otava
change-point
test:<test name>
metric:<metric name>
tags configured in the tags property of the test
tags configured in the annotate property of the test
tags configured in the annotate property of the metric

Additionally, the annotate property supports variable tags:

%{TEST_NAME} - name of the test
%{METRIC_NAME} - name of the metric
%{GRAPHITE_PATH} - resolves to the path to the data in Graphite
%{GRAPHITE_PATH_COMPONENTS} - splits the path of the data in Graphite into separate components and each path component is exported as a separate tag
%{GRAPHITE_PREFIX} - resolves to the prefix of the path to the data in Graphite (the part of the path up to the metric suffix)
%{GRAPHITE_PREFIX_COMPONENTS} - similar as %{GRAPHITE_PATH_COMPONENTS} but splits the prefix of the path instead of the path

Example

tip

See otava.yaml for the full Graphite & Grafana example.

Start docker-compose with Graphite in one tab:

docker-compose -f examples/graphite/docker-compose.yaml up --force-recreate --always-recreate-deps --renew-anon-volumes --build

Run otava in another tab:

docker-compose -f examples/graphite/docker-compose.yaml run otava otava analyze my-product.test --since=-10m --update-grafana

Expected output:

time                       run    branch    version    commit      throughput    response_time    cpu_usage
-------------------------  -----  --------  ---------  --------  ------------  ---------------  -----------
2024-12-14 22:45:10 +0000                                               61160               87          0.2
2024-12-14 22:46:10 +0000                                               60160               85          0.3
2024-12-14 22:47:10 +0000                                               60960               89          0.1
                                                                 ············                   ···········
                                                                        -5.6%                       +300.0%
                                                                 ············                   ···········
2024-12-14 22:48:10 +0000                                               57123               88          0.8
2024-12-14 22:49:10 +0000                                               57980               87          0.9
2024-12-14 22:50:10 +0000                                               56950               85          0.7

Open local Grafana in your browser (use default admin/admin credentials), open Benchmarks dashboard, and see your data points and annotations on both charts.:

Example​

Example