Graphing osmo-* KPI with grafana » History » Revision 4
Revision 3 (laforge, 12/14/2022 02:59 PM) → Revision 4/5 (laforge, 12/14/2022 03:00 PM)
{{>toc}}
h1. Graphing osmo-* KPI with grafana
Ever wanted to render nice "grafana":https://grafana.com/oss/grafana/ graphs / dashboards from the various statistics/counters/KPIs exposed by the various osmocom programs? This is possible via the _statsd exporter_ which is part of the counter / rate_counter infrastructure provided by [[libosmocore:]].
This wiki page aspires to provide an introduction on how to collect those statistics to make them available to grafana.
As usual in the Unix/Linux/GNU/FOSS world, there are many possible ways how to achieve this. We are focusing on some examples here. Usually you would only deploy _one_ of the described strategies
h2. osmocom stats reporter -> statsd_exporter -> prometheus
This is a simple configuration which looks like this:
{{graphviz_link()
graph G {
rankdir="LR";
osmo [label="osmo-bsc"];
bts [label="osmo-bts"];
mgw [label="osmo-mgw"];
statsd_exporter;
prometheus;
grafana;
osmo -- statsd_exporter [label="UDP statsd protocol\npush"];
bts -- statsd_exporter [label="UDP statsd protocol\npush"];
mgw -- statsd_exporter [label="UDP statsd protocol\npush"];
statsd_exporter -- prometheus [label="HTTP requests\npull"];
prometheus -- grafana[label="Data source"];
}
}}
So in this configuration we have
* various osmocom programs (bts, bsc, mgw in this example) pushing statsd protocol UDP messages to the listening UDP port of the @statsd_exporter@
* @statsd_exporter@ listening to a local HTTP port waiting for connections from prometheus
* @prometheus@ connecting via HTTP to @statsd_exporter@ to obtain the statistics in the prometheus-inherent _pull_ semantics
* @prometheus@ storing the data
* @prometheus@ listening to a local port, waiting for connections from grafana
* @grafana@ offering a HTTP web interface, using @prometheus@ as a data source, issuing _PromQL_ queries to prometheus
None of the elements shown in the above graph must run on the same physical or virtual machine. They all interact via IP based protocol.
The general recommendation would be to keep the @statsd_exporter@ close to the data-generating @osmo-*@ programs, as the protocol between is UDP based (can loose packets) and not authenticated or encrypted. So you certainly want to keep that rather local/private. The TCP based protocols between @statsd_exporter@ and @prometheus@ and also between @prometheus@ and @grafana@ can be operated via TLS/HTTPS and hence over public networks, if so desired.
h3. configuration of osmo-*
Please see the stats reporter chapter in the respective osmo-* user manual for details.
In general you would add something like the below snippet to your respective osmo-* program:
<pre>
stats interval 10
stats reporter statsd
remote-ip 127.0.0.1
remote-port 8125
level global
no prefix
enable
</pre>
This instructs the osmo-* program to report all of its statistics every 10s via UDP to @127.0.0.1:9125@
h3. installation / configuration of statsd_exporter
* obtain the latest release of statsd_exporter from https://github.com/prometheus/statsd_exporter/releases and install it to @/opt/statsd_exporter@
* create a statsd_exporter user with @useradd -r statsd_exporter@
* deploy a attachment:statsd_exporter.service systemd unit file to @/etc/systemd/system/statsd_exporter.service@
** adjust the @--web.listen-address@ and @--statsd.listen-udp@ accordng to your requirements
* enable + start it via @systemctl enable statsd_exporter; systemctl start statsd_exporter@
* make sure by network separation and/or packet filter rule sets that only the IP addresses of your osmo-* processes can access the _statsd.isten-udp_ port, and only the IP address running your prometheus can create inbound TCP connections to the _web.listen_address_
h3. installation / configuration of prometheus
* install prometheus (in our example, we just used the debian 11 package via @apt install prometheus@)
* configure the retention period in @/etc/default/prometheus@, e..g. via @ARGS="--storage.tsdb.retention.time=1y"@ for 1 year
* configure it to scrape @statsd_exporter@ via a snippet like below in @/etc/prometheus/prometheus.yml@ (assuing your statsd_exporter is listening on localhost:9102 for HTTP requests, as per @web.listen-address@ configured above)
<pre>
- job_name: 'statsd_exporter'
scrape_interval: 10s
static_configs:
- targets: ['localhost:9102']
</pre>
* enable + start it via @systemctl enable prometheus; systemctl start prometheus@
* make sure by network separation and/or packet filter rule sets that only the IP addresses hosting your grafana processes can create inbound connections the prometheus port (default: 9090)
h3. installation / configuration of grafana
* Install _Grafana OSS_ following the instructions at https://grafana.com/docs/grafana/latest/setup-grafana/installation/debian/
* Sign into grafana vie http://localhost:3000/ using the @admin@ user/password and change the password as instructed at https://grafana.com/docs/grafana/latest/getting-started/build-first-dashboard/
* add prometheus as a data source. Do this via the Web UI navigating to _Settings/Data Sources_ and adding a Prometheus source at @http://localhost:9090@ (default port of prometheus, unless you changed it in @/etc/defaults/prometheus@ / @--web.listen-address@).
h2. osmocom stats reporter -> collectd -> prometheus
This is a slightly more complex setup in which "collectd":https://collectd.org/ is used. This doesn't really provide any known benefits to the setup with @statsd_exporter@ described above, but is useful if you already have existing collectd-based systems running.
{{graphviz_link()
graph G {
rankdir="LR";
osmo [label="osmo-bsc"];
bts [label="osmo-bts"];
mgw [label="osmo-mgw"];
collectd;
prometheus;
grafana;
osmo -- collectd [label="UDP statsd protocol\npush"];
bts -- collectd [label="UDP statsd protocol\npush"];
mgw -- collectd [label="UDP statsd protocol\npush"];
collectd -- prometheus [label="HTTP requests\npull"];
prometheus -- grafana[label="Data source"];
}
}}
So in this configuration we have
* various osmocom programs (bts, bsc, mgw in this example) pushing statsd protocol UDP messages to the listening UDP port of the @statsd_exporter@
* @collectd@ listening to a local HTTP port waiting for connections from prometheus
* @prometheus@ connecting via HTTP to @collectd@ to obtain the statistics in the prometheus-inherent _pull_ semantics
* @prometheus@ storing the data
* @prometheus@ listening to a local port, waiting for connections from grafana
* @grafana@ offering a HTTP web interface, using @prometheus@ as a data source, issuing _PromQL_ queries to prometheus
h3. configuration of osmo-*
see above; identical to the previous case using statsd_exporter.
h3. installation / configuration of collectd
We assume you already have a @collectd@ installation, or simply installed it from your distribution package manager, such as @apt install collectd@ on Debian 11.
* make sure to enable the "Write_Prometheus":https://collectd.org/wiki/index.php/Plugin:Write_Prometheus collectd plugin. This will expose the statistic on a prometheus-compatible HTTP port. Example config:
<pre>
LoadPlugin write_prometheus
<Plugin "write_prometheus">
Port "9103"
</Plugin>
</pre>
h3. installation / configuration of prometheus
* install prometheus (in our example, we just used the debian 11 package via @apt install prometheus@)
* configure the retention period in @/etc/default/prometheus@, e..g. via @ARGS="--storage.tsdb.retention.time=1y"@ for 1 year
* configure it to scrape @collectd@ via a snippet like below in @/etc/prometheus/prometheus.yml@ (assuing your collectd is listening on localhost:9103 for HTTP requests, as configured above)
<pre>
- job_name: 'collectd'
scrape_interval: 10s
static_configs:
- targets: ['localhost:9103']
</pre>
* enable + start it via @systemctl enable prometheus; systemctl start prometheus@
* make sure by network separation and/or packet filter rule sets that only the IP addresses hosting your grafana processes can create inbound connections the prometheus port (default: 9090)
h3. installation / configuration of grafana
see above; identical to the previous case using statsd_exporter.
