Graphing osmo-* KPI with grafana » History » Version 3
laforge, 12/14/2022 02:59 PM
1 | 1 | laforge | {{>toc}} |
---|---|---|---|
2 | |||
3 | h1. Graphing osmo-* KPI with grafana |
||
4 | |||
5 | Ever wanted to render nice "grafana":https://grafana.com/oss/grafana/ graphs / dashboards from the various statistics/counters/KPIs exposed by the various osmocom programs? This is possible via the _statsd exporter_ which is part of the counter / rate_counter infrastructure provided by [[libosmocore:]]. |
||
6 | |||
7 | This wiki page aspires to provide an introduction on how to collect those statistics to make them available to grafana. |
||
8 | |||
9 | As usual in the Unix/Linux/GNU/FOSS world, there are many possible ways how to achieve this. We are focusing on some examples here. Usually you would only deploy _one_ of the described strategies |
||
10 | |||
11 | h2. osmocom stats reporter -> statsd_exporter -> prometheus |
||
12 | |||
13 | This is a simple configuration which looks like this: |
||
14 | |||
15 | {{graphviz_link() |
||
16 | graph G { |
||
17 | rankdir="LR"; |
||
18 | osmo [label="osmo-bsc"]; |
||
19 | bts [label="osmo-bts"]; |
||
20 | mgw [label="osmo-mgw"]; |
||
21 | statsd_exporter; |
||
22 | prometheus; |
||
23 | grafana; |
||
24 | |||
25 | osmo -- statsd_exporter [label="UDP statsd protocol\npush"]; |
||
26 | bts -- statsd_exporter [label="UDP statsd protocol\npush"]; |
||
27 | mgw -- statsd_exporter [label="UDP statsd protocol\npush"]; |
||
28 | statsd_exporter -- prometheus [label="HTTP requests\npull"]; |
||
29 | prometheus -- grafana[label="Data source"]; |
||
30 | } |
||
31 | }} |
||
32 | |||
33 | So in this configuration we have |
||
34 | * various osmocom programs (bts, bsc, mgw in this example) pushing statsd protocol UDP messages to the listening UDP port of the @statsd_exporter@ |
||
35 | * @statsd_exporter@ listening to a local HTTP port waiting for connections from prometheus |
||
36 | * @prometheus@ connecting via HTTP to @statsd_exporter@ to obtain the statistics in the prometheus-inherent _pull_ semantics |
||
37 | * @prometheus@ storing the data |
||
38 | * @prometheus@ listening to a local port, waiting for connections from grafana |
||
39 | * @grafana@ offering a HTTP web interface, using @prometheus@ as a data source, issuing _PromQL_ queries to prometheus |
||
40 | |||
41 | None of the elements shown in the above graph must run on the same physical or virtual machine. They all interact via IP based protocol. |
||
42 | |||
43 | The general recommendation would be to keep the @statsd_exporter@ close to the data-generating @osmo-*@ programs, as the protocol between is UDP based (can loose packets) and not authenticated or encrypted. So you certainly want to keep that rather local/private. The TCP based protocols between @statsd_exporter@ and @prometheus@ and also between @prometheus@ and @grafana@ can be operated via TLS/HTTPS and hence over public networks, if so desired. |
||
44 | |||
45 | h3. configuration of osmo-* |
||
46 | |||
47 | Please see the stats reporter chapter in the respective osmo-* user manual for details. |
||
48 | |||
49 | In general you would add something like the below snippet to your respective osmo-* program: |
||
50 | <pre> |
||
51 | stats interval 10 |
||
52 | stats reporter statsd |
||
53 | remote-ip 127.0.0.1 |
||
54 | remote-port 8125 |
||
55 | level global |
||
56 | no prefix |
||
57 | enable |
||
58 | </pre> |
||
59 | |||
60 | This instructs the osmo-* program to report all of its statistics every 10s via UDP to @127.0.0.1:9125@ |
||
61 | |||
62 | h3. installation / configuration of statsd_exporter |
||
63 | |||
64 | * obtain the latest release of statsd_exporter from https://github.com/prometheus/statsd_exporter/releases and install it to @/opt/statsd_exporter@ |
||
65 | * create a statsd_exporter user with @useradd -r statsd_exporter@ |
||
66 | * deploy a attachment:statsd_exporter.service systemd unit file to @/etc/systemd/system/statsd_exporter.service@ |
||
67 | ** adjust the @--web.listen-address@ and @--statsd.listen-udp@ accordng to your requirements |
||
68 | * enable + start it via @systemctl enable statsd_exporter; systemctl start statsd_exporter@ |
||
69 | 2 | laforge | * make sure by network separation and/or packet filter rule sets that only the IP addresses of your osmo-* processes can access the _statsd.isten-udp_ port, and only the IP address running your prometheus can create inbound TCP connections to the _web.listen_address_ |
70 | 1 | laforge | |
71 | h3. installation / configuration of prometheus |
||
72 | |||
73 | * install prometheus (in our example, we just used the debian 11 package via @apt install prometheus@) |
||
74 | * configure the retention period in @/etc/default/prometheus@, e..g. via @ARGS="--storage.tsdb.retention.time=1y"@ for 1 year |
||
75 | * configure it to scrape @statsd_exporter@ via a snippet like below in @/etc/prometheus/prometheus.yml@ (assuing your statsd_exporter is listening on localhost:9102 for HTTP requests, as per @web.listen-address@ configured above) |
||
76 | <pre> |
||
77 | - job_name: 'statsd_exporter' |
||
78 | scrape_interval: 10s |
||
79 | static_configs: |
||
80 | - targets: ['localhost:9102'] |
||
81 | </pre> |
||
82 | * enable + start it via @systemctl enable prometheus; systemctl start prometheus@ |
||
83 | 3 | laforge | * make sure by network separation and/or packet filter rule sets that only the IP addresses hosting your grafana processes can create inbound connections the prometheus port (default: 9090) |
84 | 1 | laforge | |
85 | h3. installation / configuration of grafana |
||
86 | |||
87 | * Install _Grafana OSS_ following the instructions at https://grafana.com/docs/grafana/latest/setup-grafana/installation/debian/ |
||
88 | * Sign into grafana vie http://localhost:3000/ using the @admin@ user/password and change the password as instructed at https://grafana.com/docs/grafana/latest/getting-started/build-first-dashboard/ |
||
89 | * add prometheus as a data source. Do this via the Web UI navigating to _Settings/Data Sources_ and adding a Prometheus source at @http://localhost:9090@ (default port of prometheus, unless you changed it in @/etc/defaults/prometheus@ / @--web.listen-address@). |
||
90 | |||
91 | |||
92 | h2. osmocom stats reporter -> collectd -> prometheus |
||
93 | |||
94 | This is a slightly more complex setup in which "collectd":https://collectd.org/ is used. This doesn't really provide any known benefits to the setup with @statsd_exporter@ described above, but is useful if you already have existing collectd-based systems running. |
||
95 | |||
96 | |||
97 | {{graphviz_link() |
||
98 | graph G { |
||
99 | rankdir="LR"; |
||
100 | osmo [label="osmo-bsc"]; |
||
101 | bts [label="osmo-bts"]; |
||
102 | mgw [label="osmo-mgw"]; |
||
103 | collectd; |
||
104 | prometheus; |
||
105 | grafana; |
||
106 | |||
107 | osmo -- collectd [label="UDP statsd protocol\npush"]; |
||
108 | bts -- collectd [label="UDP statsd protocol\npush"]; |
||
109 | mgw -- collectd [label="UDP statsd protocol\npush"]; |
||
110 | collectd -- prometheus [label="HTTP requests\npull"]; |
||
111 | prometheus -- grafana[label="Data source"]; |
||
112 | } |
||
113 | }} |
||
114 | |||
115 | So in this configuration we have |
||
116 | * various osmocom programs (bts, bsc, mgw in this example) pushing statsd protocol UDP messages to the listening UDP port of the @statsd_exporter@ |
||
117 | * @collectd@ listening to a local HTTP port waiting for connections from prometheus |
||
118 | * @prometheus@ connecting via HTTP to @collectd@ to obtain the statistics in the prometheus-inherent _pull_ semantics |
||
119 | * @prometheus@ storing the data |
||
120 | * @prometheus@ listening to a local port, waiting for connections from grafana |
||
121 | * @grafana@ offering a HTTP web interface, using @prometheus@ as a data source, issuing _PromQL_ queries to prometheus |
||
122 | |||
123 | h3. configuration of osmo-* |
||
124 | |||
125 | see above; identical to the previous case using statsd_exporter. |
||
126 | |||
127 | h3. installation / configuration of collectd |
||
128 | |||
129 | We assume you already have a @collectd@ installation, or simply installed it from your distribution package manager, such as @apt install collectd@ on Debian 11. |
||
130 | |||
131 | * make sure to enable the "Write_Prometheus":https://collectd.org/wiki/index.php/Plugin:Write_Prometheus collectd plugin. This will expose the statistic on a prometheus-compatible HTTP port. Example config: |
||
132 | <pre> |
||
133 | LoadPlugin write_prometheus |
||
134 | |||
135 | <Plugin "write_prometheus"> |
||
136 | Port "9103" |
||
137 | </Plugin> |
||
138 | </pre> |
||
139 | |||
140 | h3. installation / configuration of prometheus |
||
141 | |||
142 | * install prometheus (in our example, we just used the debian 11 package via @apt install prometheus@) |
||
143 | * configure the retention period in @/etc/default/prometheus@, e..g. via @ARGS="--storage.tsdb.retention.time=1y"@ for 1 year |
||
144 | * configure it to scrape @collectd@ via a snippet like below in @/etc/prometheus/prometheus.yml@ (assuing your collectd is listening on localhost:9103 for HTTP requests, as configured above) |
||
145 | <pre> |
||
146 | - job_name: 'collectd' |
||
147 | scrape_interval: 10s |
||
148 | static_configs: |
||
149 | - targets: ['localhost:9103'] |
||
150 | </pre> |
||
151 | * enable + start it via @systemctl enable prometheus; systemctl start prometheus@ |
||
152 | |||
153 | h3. installation / configuration of grafana |
||
154 | |||
155 | see above; identical to the previous case using statsd_exporter. |