Osmocom Titan TTCN3 Testsuites¶
- Table of contents
- Osmocom Titan TTCN3 Testsuites
In 2017, Osmocom started to create extensive tests suites for the Cellular Network Infrastructure elements. Those suites are written in the TTCN-3 programming language and use the compiler and executor of the Eclipse_TITAN project. There are special cases, but in general, the idea is to test one component (e.g. osmo-mgw) independently.
Development¶
Source Code / Patches¶
The source code of our test suites lives in theosmo-ttcn3-hacks repository
- web interface: https://gitea.osmocom.org/ttcn3/osmo-ttcn3-hacks
- clone:
git clone https://gitea.osmocom.org/ttcn3/osmo-ttcn3-hacks - patch review: https://gerrit.osmocom.org/#/q/project:osmo-ttcn3-hacks
We're following the Gerrit process for patch review.
Proprietary APER<->BER transcoding library for Iu tests¶
As TITAN can only generate and parse ASN.1 BER encoding, but the Iu-CS, Iu-PS and Iuh interfaces are using APER encoding, sysmocom is providing a proprietary transcoding library called libfftranscode. Debian9 amd64 packages are available from https://ftp.osmocom.org/binaries/libfftranscode/. For Arch Linux users, there is an AUR package that basically downloads the Debian package and extracts libfftranscode.so from it.
We don't like to depend on proprietary software, but given that there is no FOSS ASN.1 compiler that can parse all ASN.1 constructs of the RANAP/RUA/HNBAP specs and parse + generate both BER and APER, this is the lesser evil. Note that the dependency to this proprietary library is only required for the TTCN3 test suite and not a runtime dependency of any of the Osmocom programs itself.
Test Suites¶
Running a testsuite¶
You have multiple options on how to execute a test suite.
Running test suite in the dockerized environment¶
This is the much easier way to run the test suite. No manual configuration of Osmocom programs or the test suite are required. Both the Osmocom program (Implementation Under Test) as well as the test suite are packaged as docker containers, which are then executed next to each other using a docker network with the "right" addresses between IUT and testsuite.
Also, using this setup you will be guaranteed to run in the exact same environment as the automatically-executed tests on jenkins.osmocom.org, i.e. your results should be exactly identical, without any differences introduced by your runtime environment, whether specific library versions or intentional or inadvertent configuration differences.
Installing docker¶
Make sure you have *docker-ce* installed (not to be confused with the possibly outdated version that may be shipped in your distribution's repositories). Check that the version string contains -ce as follows:
$ docker -v Docker version 18.06.1-ce, build e68fc7a
Next, add yourself to the docker group, log off and on again, and verify that you can use docker with your user:
$ docker info
Cloning docker-playground¶
Osmocom related containers are stored in the docker-playground git repository.
$ git clone https://gitea.osmocom.org/osmocom/docker-playground $ cd docker-playground
Running jenkins.sh¶
All testsuite directories start with ttcn3 or nplab. Run the following to build/update all required containers and start a specific testsuite:
$ cd ttcn3-mgw-test $ ./jenkins.shThe
jenkins.sh script will make sure to
- create the needed docker network(s)
- run all the required containers (IUT, helpers, testsuite)
- shut them down after test suite execution
- collect the log files after execution,
/tmp/logswill contain them
Control the behavior of jenkins.sh with environment variables as described in docker-playground's README.md. This file also explains advanced features, such as the kernel module tests.
Running only a sub-set of the test cases¶
See running the testsuite outside of Docker for reference (below).
Running it natively on your machine¶
This is the most complicated bit to set-up, as you will have to run the respective Osmocom Program (Implementation Under Test) in the right configuration with all the IP addresses, port numbers, config file, etc. exactly like it's expected by the test suite. Even though the configuration files contained in osmo-ttcn3-hacks.git are expected to work 'as-is' when running tests natively, some of them may get out of sync with the respective files in docker-playground.git.
You will need a deeper understanding about how the test suite works, and what its requirement are. Oftentimes looking at how it's done in the Docker scripts is helpful.
Preparation¶
As stated above certain parts of the testsuite require libfftranscode! If the build fails this might be the issue.
While you can try to use any version of Eclipse TITAN, for the best possible experience it's a good idea to use the same version as Osmocom uses in the dockerized environment. As can be seen in the Dockerfile, this (currently) points to the eclipse-titan package from the Latest_Builds on Debian 11":.
$ apt install eclipse-titan $ git clone https://gitea.osmocom.org/ttcn3/osmo-ttcn3-hacks $ cd osmo-ttc3-hacks/deps $ make
Compile a testsuite¶
Let's compile the mgw testsuite for example:
$ cd mgw $ ./gen_links.sh $ ./regen_makefile.sh
The next command will transform the TTCN3 test data into C++ code. Do not use -j here, that won't work (and it is pretty fast anyway).
$ make compile
Note: if you observe errors during make compile, try running it from top-level as make mgw which should trigger dependency update.
Finally compile the testsuite:
$ make -j5
Run a testsuite¶
In case you're using tmux, osmo-dev contains useful scripts for starting some test suites in a dedicated tmux session. These scripts prepare the working environment by starting the IUT and its dependencies in separate windows/panes. This helps to save time a lot, e.g. for ttcn3-bsc-test one would need to run osmo-bsc, osmo-stp, and three instances of osmo-bts-omldummy. Just execute the respective script and you're ready to go. Below are the instructions for general (tmuxless) use case.
Start the component that is about to get tested with the config that the testsuite expects. Again, with mgw as example. A suitable config file should be in the same directory of osmo-ttcn3-hacks.git (otherwise one can use the one from Jenkins and replace all IPs with 127.0.0.1).
$ osmo-mgw -c osmo-mgw.cfg
Then run the testsuite:
$ cd osmo-ttcn3-hacks/mgw $ ../start-testsuite.sh ./MGCP_test MGCP_Test.cfg
Afterwards you can merge and format the logs as follows:
$ ttcn3_logmerge MGCP_test*.log > ./merged.log $ ttcn3_logformat ./merged.log > ./result.log
or
$ ../log_merge.sh MGCP_test
Depending on the component that you want to test, you may need to run multiple Osmocom programs. For OsmoMSC, you would run osmo-msc and osmo-stp for example. See the containers that jenkins.sh is running for reference.
Running only a sub-set of the test cases¶
Running whole testsuites may take a long time. If you only want to run one specific test, or a subset of tests, open up the _*Test.cfg file in the directory of the testsuite and edit the [EXECUTE] section. For example, in MGCP_Test.cfg from the mgw directory it looks like the following:
... [EXECUTE] MGCP_Test.control #MGCP_Test.TC_selftest #MGCP_Test.TC_crcx #MGCP_Test.TC_crcx_noprefix ...
The only line that is not commented out is MGCP_Test.control. This *.control line will cause all tests to be executed. So comment this line out, and then activate one of the lines of which you would like to execute the tests (see the example below). Then execute the testsuite as usually, and it will skip all commented out tests.
... [EXECUTE] #MGCP_Test.control #MGCP_Test.TC_selftest MGCP_Test.TC_crcx MGCP_Test.TC_crcx_noprefix ...
Reading the logs¶
xfailmeans, that a test is known to fail,FAILare unexpected test failures- search for
setverdict(to find the results of single tests.-> failshould be the location where a test is failing.
Further information¶
- April 2018 talk by Harald on State of the Osmocom TTCN-3 Test Suites
- December 2021 talk by Oliver on osmo-dev and running Osmocom TTCN-3 testsuites locally
Updated by osmith 7 months ago · 52 revisions
