Project

General

Profile

Actions

Feature #5751

closed

io_uring support in libosmocore

Added by laforge over 1 year ago. Updated 26 days ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
-
Target version:
-
Start date:
11/09/2022
Due date:
% Done:

100%

Spec Reference:
Tags:

Description

Traditionally our I/O abstraction in libosmocore has been select(). In libosmocore 1.5.0 (2020) we migrated over to poll() to support more than 1024 FDs and to avoid the extreme amount of fd-set memcpy()ing involved in the venerable select interface.

Now of course both select and poll are ancient unix interfaces for non-blocking I/O, and both come at a high cost for systems under high load.

Specifically, we are getting reports from osmo-bsc users that indicate a busy BSC with 100 BTS ( 400 TRX)_is spending about 40% of its CPU cycles in the (kernel side) sock_poll, tcp_poll, do_sys_poll.

There are other interfaces such as linux aio, posix aio and epoll, but the brightest and shiniest new I/O interface on Linux is io_uring. Contrary to any of its predecessors, io_uring can, in the "worst" case, operate without any system calls at all anymore. io_uring recognizes that each syscall is associated with a rather high context switch cost.

io_uring consists of memory-mapped (between kernel and userspace process) queues for requests and completions, as well as lockless primitives to enqueue/dequeue from these.

The requests in the queue are requests like read N bytes from this file descriptor or write N bytes to that file descriptor. But io_uring can do much more (many other syscalls), though the read/write is the most relevant part to us.

we already have two io_uring users in the osmocom universe: the GTP and the UDP/RTP load generators I wrote some time ago. They manage their file descriptors internally.

This ticket is now about introducing io_uring support into libosmocore itself, in a way to enable all osmocom programs to use that shared infrastructure.

Conceptual differences

reading from a socket

Conceptually, the existing code typically works like this:

  1. register some socket file descriptor for read
  2. libosmocore includes it in the poll-set
  3. libosmocore calls poll()
  4. kernel returns from poll, indicating fd is readable
  5. libosmocore dispatches to the application call-back
  6. application allocates msgb, reads data from socket
  7. application processes data in msgb

With io_uring, this model needs to change to something like this:

  1. application tells us it wants to read from a socket
  2. libosmocore or application pre-allocate the msgb
  3. libosmocore uses liburing to add a read request to the io_uring submission queue
  4. kernel signals us at some point a completion event via io_uring / liburing
  5. libosmocore dispatches pre-filled msgb to application call-back
  6. application processes data n msgb

So as we can see, the responsibility for the actual reading transfers from application (or intermediate library like libosmo-netif / libosmo-sigtran) into library.

writing to a socket

Conceptually, the existing code typically works like this:

  1. register some socket file descriptor for read
  2. libosmocore includes it in the poll-set
  3. libosmocore calls poll()
  4. kernel returns from poll, indicating fd is writeable
  5. libosmocore dispatches to the application call-back
  6. application writes data to msgb and free's msgb.

With io_uring, this model needs to change to something like this:

  1. application tells us it wants to write to a socket, including the msgb
  2. libosmocore uses liburing to add a write request to the io_uring submission queue
  3. kernel signals us at some point a completion event via io_uring / liburing
  4. libosmocore releases the msgb with msgb_free()

Again, the actual reading/writing passes into the library, and outside the scope of the application (or intermediate library like libosmo-netif / libosmo-sigtran)


Checklist

  • sctp support in osmo_io

Related issues

Related to libosmo-sccp + libosmo-sigtran - Feature #5752: io_uring support in libosmo-sigtranResolvedjolly11/09/2022

Actions
Related to libosmo-netif - Feature #5753: io_uring support in libosmo-netifResolvedHoernchen11/09/2022

Actions
Related to OsmoMGW - Feature #5754: io_uring support in libosmo-mgcp-clientResolvedjolly11/09/2022

Actions
Related to OsmoBSC - Feature #5755: io_uring support in osmo-bscIn Progressjolly11/09/2022

Actions
Related to libosmo-abis - Bug #5756: io_uring support in libosmo-abisNew11/09/2022

Actions
Related to libosmo-abis - Feature #5766: use Linux kernel KCM for IPA header?New11/13/2022

Actions
Related to Cellular Network Infrastructure - Bug #5948: Fix socket (-write) functions in multiple projects (by moving them to a common library...)Rejected03/19/2023

Actions
Related to Core testing infrastructure - Feature #6357: run (some?) tests with io_uring backend for osmo_ioResolvedosmith02/09/2024

Actions
Related to OsmoMGW - Feature #6387: osmo_io / io_uring support for RTP/RTCPIn Progresslaforge03/02/2024

Actions
Related to libosmocore - Feature #6388: stats_reporter via osmo_io / io_uring?New03/02/2024

Actions
Related to libosmocore - Feature #6389: port VTY over to osmo_io / io_uringNew03/02/2024

Actions
Related to libosmocore - Feature #6390: port CTRL over to osmo_io / io_uringNew03/02/2024

Actions
Actions

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)