Bug #6256
closedosmo-bsc running out of file descriptors in production setups
100%
Description
We have a production setup where there are >= 200 BTs with each up to 4 TRX. In that setup, the OS default number of open file descriptors (1024) is hit, resulting in call drops.
Given that there is N+1 sockets for each N-TRX BTS (e.g. 5 for a 4TRX) plus a handful for the AoIP interface, for speaking MGCP to the MGW, CTRL/VTY, etc. it is thus completely reasonable to exceed 1024.
Now the 1024 is not a limit imposed by osmo-bsc. However, we might want to either- add a note to the user manual informing users that they should tune their system accordingly if they expect a high number of BTSs, and/or
- consider tuning the limit for osmo-bsc using
LimitNOFILE
in the systemd unit we're shipping.
- osmo-hnbgw (only one Iuh socket for each HNB, though)
- osmo-mgw (2 sockets per endpoint (RTP+RTCP), resulting in 4 sockets for each call, but single-threaded mgw might not handle 256 calls anyway?
- osmo-gbproxy (number of sockets depends on number of NS-VCs configured. But at least common setups probably only have few UDP sockets. Unlike TCP/SCTP, we don't need new sockets for each peer/connection). So probably not required
- bts, trx, msc, hlr, sgsn, ggsn all have only a low number of sockets, from what I can tell.
Updated by neels 5 months ago
- consider tuning the limit for osmo-bsc using
LimitNOFILE
in the systemd unit we're shipping.
But tune it to what number?
I think 200 4trx BTS is already pretty high, so it actually seems sane to ship with a default of 1024.
Having that value in the systemd unit file may already help admins to adjust it.
Of course that means a hassle of having a modified unit file == a failure mode for each new upgrade.
So, if there isn't much harm done, we may as well crank it up to the largest number we currently know is in use.
What number would that be; is LimitNOFILE=4096 sufficiently huge? (65536?)
Updated by neels 5 months ago
similar for hnbgw: https://gerrit.osmocom.org/c/osmo-hnbgw/+/35177
(though it does seem uncommon to have >1000 HNB per HNBGW)