Create testsuite to test osmo-bts-trx+osmo-trx under high channel load
The final aim of this task is to check if we run into CPU limitations of the Raspi CM3 of the LimeNet-micro when maximizing the channel load even of a single TRX.
Say, for example, 14 concurrent TCH/H with AMR inside on the 7 TS. It would be very interesting to see if that works, and if there is any margin on the CPU left, etc.
Quick way to test manually: Use osmo-bsc connected to osmo-bts-trx and use VTY command "bts <0-255> trx <0-255> timeslot <0-7> sub-slot <0-7> (activate|deactivate) (hr|fr|efr|amr) [<0-7>]"
"Automatic" testing: Use TTCN3, add test to BTS_Tests.ttcn:
As we even have that RTP source and sink you could even send some (random payload) RTP messages if you'd want, they just need to match in size and in terms of the RTP payload type. We don't care about the content of the RTP at all here.
Make sure the config/vty is configured for 7 timeslots at TCH/H and unlimited radio link timeout and then send the 14x CHAN ACT with the right channel mode (and if you want to add RTP, the IPA CRCX/MDCX).
#2 Updated by pespin about 1 month ago
- Status changed from New to In Progress
- % Done changed from 0 to 50
I started some work in osmo-ttcn3-hacks.git and docker-playground.git branch "pespin/bts-perf".
In there I have a passing test which uses an osmo-bsc.cfg with all channels 1..7 set to TCH/H and the TC_pespin test activates speech AMR on all of them in the BTS, waits 10 seconds, then deactivates them.
Next step is to launch the test against osmo-bts-trx+osmo-trx-lms running on my own laptop to see if there's any issue. Later, run it against same setup but running in LimeNet-micro.
#3 Updated by ipse about 1 month ago
Note, that in normal operation,
osmo-bts-trx also consumes a considerable amount of CPU. Not as much as
osmo-trx but still quite noticeable. But this only happens when
osmo-trx produces bursts for
osmo-bts-trx to process. If you just enable channels,
osmo-trx will produce idle bursts (or no bursts, depending on the version and configuration), which means
osmo-bts-trx won't have anything to process.
Also note that
osmo-bts-trx CPU usage might depend on the uplink signal quality because it involves Viterbi decoding which iterates over a signal. I personally haven't tested this but this is the theory.
I have a ttcn3 test opening 14 TCH/H channels from BSC side towards a bts-trx->LimeNet-micro all orchestrated with osmo-gsm-tester, and it seems to be working fine
In LimeNet-micro, top shows a load average of: 0.77, 0.78, 0.51
That's with osmo-bts-trx running outside of LimeNet-micro and without RTP processing though, but looks like there's enough load to keep it going anyway.
In osmo-gsm-tester prod main unit: jenkins ~/test/:
./run_locally.sh -s ttcn3_bts_tests:trx-lms-limenet-micro -l dbg -T -t highchanload_tchh
It worked fine for a while (20 mins) with no overruns or similar, while I could monitor it though VTY, top, etc.
I then tried to run stress-ng in the RPI CM3 while osmo-trx was running, and osmo-bts-trx quickly dropped the connection probably due to CLOCK IND arriving to late, despite osmo-trx-lms running with rt-prio 18. That's expected I'd say though because I did really put the whole system into a crazy stress which shouldn't happen in usual ways. I'll try do run with more realistic stress next times. that's what I used so far:
stress-ng --vm 4 --hdd 4 --cpu 4 --open 4 --sem 4 --sock 4 --io 2 --timer 2
https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/17098 bts: Introduce new module for performance testsNext steps:
- Support running osmo-bts-trx remotely by osmo-gsm-tester into RPI CM3 module (same as we do for osmo-trx-lms, run the packaged binary directly).
- Add an RTP source/sink to the TTCN3 code to make sure some data is being sent so that osmo-bts-trx does some work (first in docker setup branch pespin/bts-perf, then in real HW with osmo-gsm-tester).
The problem is that we don't have an easy way to feed the uplink in this case afaiu, only the downlink.
I have the testcase BTS_Tests_perf.TC_highchanload_tchh now running on my side. I am currently testing with faketrx and trxcon. I have implemented Sending of the IPA CRCX and now I get the IP/Port from the BTS back. I wonder if sending RTP packets possible immediately or if I have to go through the complete procedure.