https://projects.osmocom.org/https://projects.osmocom.org/favicon.ico?16647414092018-04-10T17:05:43ZOpen Source Mobile CommunicationsOsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=87952018-04-10T17:05:43Zlaforge
<ul></ul><p>related: <a class="external" href="https://brmlab.cz/project/gsm/shield">https://brmlab.cz/project/gsm/shield</a></p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=88112018-04-10T17:42:21Zlaforge
<ul></ul><p><a class="external" href="https://wiki.cuvoodoo.info/doku.php?id=osmotoserial">https://wiki.cuvoodoo.info/doku.php?id=osmotoserial</a> is the project I was originally referring-to.</p>
<p>We still want to build it into an enclosure, though.</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=88542018-04-12T19:31:31Zlaforge
<ul><li><strong>Priority</strong> changed from <i>Normal</i> to <i>High</i></li></ul> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=88672018-04-13T16:51:14Zmschramm
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li><li><strong>% Done</strong> changed from <i>0</i> to <i>10</i></li></ul><p>We got three osmotoserial PCBA by tsaitgaist. I got them working in a C123 and in C118 (only a C155 did not come up as expected).</p>
<p>No additional power switch is needed, as the PCBA controls the phone power itself by listening to the state of DTR signal on the USB serial (CP2104).</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=88682018-04-13T18:41:37Zmschramm
<ul></ul><p>Only on the C123 the MS-147 plug connects reasonably, but must be fixed somehow too. On at least one C118 used here, it simply falls off. Maybe we strip the phones' enclosures to achieve better contact.</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=89312018-04-17T17:20:41Zpespin
<ul></ul>List of what's needed after first evaluation:
<ul>
<li>HW:
<ul>
<li>A main unit (re-use osmo-gsm-tester main unit)</li>
<li>A motorola Cyxz phone plugged with a serial interface plugged in and a software based way to power ir on/off from the main unit.</li>
<li>a sysmobts (re-use the osmo-gsm-tester one).</li>
<li>RF network setup between motorola phone and sysmobts.</li>
</ul>
</li>
<li>SW:
<ul>
<li>tools needed to flash and control the motorola Cxyz. Provide either debian packages or a docker image with them. Stuff needed: osmocon, firmware blob, flashing tool?</li>
<li>trial borrowed from osmo-gsm-tester build jobs: osmo-bts-sysmo, those will be copied to sysmobts</li>
<li>trial borrowed from osmo-gsm-tester build jobs: osmo-bsc. It will configure the sysmobts through OML and tell it to connect via RSL to TTTCN3 test.</li>
<li>docker image containing a built and running BTS_Tests.ttcn</li>
</ul></li>
</ul>
<p>Flashing the phone doesn't seem to be required, it can be controlled directly by osmocon through serial at boot time(<a class="external" href="https://osmocom.org/projects/baseband/wiki/Osmocon">https://osmocom.org/projects/baseband/wiki/Osmocon</a>)</p>
Then, we need a jenkins job which does the following:
<ol>
<li>Get artifacts from following jobs: osmo-gsm-tester_build-osmo-bts-sysmo osmo-gsm-tester_build-osmo-msc</li>
<li>Fetch latest already built docker images</li>
<li>run a shell/python script to do all the job.</li>
<li>Use the produced junit file.</li>
</ol>
The script should do the following:
<ol>
<li>Prepare a osmo-bts-sysmo.cfg, osmo-bsc.cfg according to specific setup (ip address, etc., have a look at the one used in BTS_Tests.ttcn)</li>
<li>Start osmo-bsc using the prepared osmo-bsc.</li>
<li>scp the osmo-bts-trial trial to the sysmobts and start the osmo-bts-sysmo process.</li>
<li>Build/download the osmocom firmware.</li>
<li>Start osmocon binary, be it in a docker image or via trial.</li>
<li>Start latest Bts_Tests ttcn3 docker image, which should start the tests automatically.</li>
<li>Wait until Bts_Tests image is done</li>
<li>Tear down everything: osmocon, sysmobts processes, osmo-bsc, docker images</li>
<li>Store results in artficats and provide a junit file.</li>
</ol>
<p>Locking mechanism between osmo-gsm-tester and this new run job is not required because it's already integrated in jenkins (the main unit node only executes 1 job in parallel so far).</p>
<p>The job should actually be a matrix job, as we actually want to run the tests for:<br />- sysmobts<br />- octphy (requires osmo-bts trial for osmo-bts-octphy)<br />- sysmocell500 (requires osmo-bts trial for osmo-bts-trx)<br />- nanobts</p>
<p>Remark:<br />It may actually make sense to reuse osmo-gsm-tester to start and manage all those processes (BTS+BSC), and provide a specific test which waits for a specific event in the system to finish (for instance a unix socket receiving some stuff).<br />For instance the shell script starting osmo-gsm-tester with that mentioned test, then starting osmocon, then starting the docker image with the ttcn3 tests.</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=89362018-04-17T20:09:10Zlaforge
<ul></ul><p>On Tue, Apr 17, 2018 at 05:20:41PM +0000, pespin [REDMINE] wrote:</p>
<blockquote>
List of what's needed after first evaluation:
<ul>
<li>HW:
<ul>
<li>A main unit (re-use osmo-gsm-tester main unit)</li>
<li>A motorola Cyxz phone plugged with a serial interface plugged in and a software based way to power ir on/off from the main unit.</li>
</ul></li>
</ul>
</blockquote>
<p>let's focus on C123 or C118, and make sure it's the same model everywhere. I can bring<br />more phones from home, as needed.</p>
<blockquote>
<ul>
<li>SW:
<ul>
<li>tools needed to flash and control the motorola Cxyz. Provide either debian packages or a docker image with them. Stuff needed: osmocon, firmware blob, flashing tool?</li>
</ul></li>
</ul>
</blockquote>
there is no "flashing" involved, it's all just loaded into ram. You need the following bits<br />and pieces from osmocom-bb.git:
<ul>
<li>osmocon</li>
<li>layer1 firmware image (cross-compiled with ancient toolchain)
<ul>
<li>make sure to enable transmit in the firmware Makefile!</li>
</ul></li>
</ul>
<blockquote>
<ul>
<li>docker image containing a built and running BTS_Tests.ttcn</li>
</ul>
</blockquote>
<p>It still remains to be seen if the docker image can be executed on the APU (resource wise)</p>
<p>We clearly don't have to build the docker image on the APU itself, but we have to run it there.</p>
<blockquote>
<p>Flashing the phone doesn't seem to be required, it can be controlled directly by osmocon through serial at boot time(<a class="external" href="https://osmocom.org/projects/baseband/wiki/Osmocon">https://osmocom.org/projects/baseband/wiki/Osmocon</a>)</p>
</blockquote>
<p>ACK.</p>
<blockquote>
<ol>
<li>Fetch latest already built docker images</li>
</ol>
</blockquote>
<p>we can push those from the build hosts to our own registry (We do have a container running on<br />admin2.sysmocom.de, but it's not used/tested yet)</p>
<blockquote>
<ol>
<li>Build/download the osmocom firmware.</li>
</ol>
</blockquote>
<p>I think it's fine to build it on jenkins build slaves (I think we even have an osmocom-bb build verification job already,<br />it's just not storing the resulting builds and probably builds without TX enabled)</p>
<blockquote>
<ol>
<li>Start latest Bts_Tests ttcn3 docker image, which should start the tests automatically.</li>
</ol>
</blockquote>
<p>correct, if the phone, BTS, <abbr title="!">BSC</abbr> and 'osmocon' are all running, executing the tests works exactly<br />like in the virtualized environment with trxcon+fake_trx.</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=89492018-04-18T18:26:53Zmschramm
<ul><li><strong>% Done</strong> changed from <i>10</i> to <i>20</i></li></ul><p>A phone is now installed with an osmotoserial PCBA in the R&D test rig. However, it switches on immediately, though no terminal process is working for this serial (right now it is /dev/ttyUSB14) which is different to the behaviour seen on my local machine. I'll leave this to pespin; because of OsmoDevCon I don't think that progress on this will be made before upcoming week.</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=93672018-05-17T11:39:05Zlaforge
<ul></ul><p><a class="user active" href="https://projects.osmocom.org/users/30187">pespin</a>: What's the status here? I think this should be done still in may, so that you won't have to touch the hardware setup after your move from Berlin.</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=93682018-05-17T11:39:35Zlaforge
<ul><li><strong>Priority</strong> changed from <i>High</i> to <i>Urgent</i></li></ul> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=94292018-05-18T15:35:31Zpespin
<ul><li><strong>% Done</strong> changed from <i>20</i> to <i>30</i></li></ul><p>I finally decided to run TTCN3 binaries from inside osmo-gsm-tester test, once everything is set up.</p>
<p>Work done so far can be found in: <a class="external" href="https://git.osmocom.org/osmo-gsm-tester/log/?h=pespin/ttcn3">https://git.osmocom.org/osmo-gsm-tester/log/?h=pespin/ttcn3</a><br />I already have osmocon built (firwmare missing) and a class to use it in osmo-gsm-tester. I have a test using the osmocon class, starting all the required elements (BTS, BSC, etc.) and which calls a script to start the docker once everything is set up. When docker finishes, everything is teared down.</p>
<p>For the missing parts, here's a TODO list:</p>
<pre>
TODO:
* Build osmocom-bb's firwmare (cross-compile) in jenkins job build.
* Find a way to programatically find /dev/ttyUSBX for the motorla phone.
* Create script which fetches latest ttcn3-hacks docker image and starts osmo-gsm-tester ttcn3 test.
** status: we need to build the "ttcn3-bts-test" docker container and pull it in osmo-gsm-tester
* ansible: Add "docker" to ansible osmo-gsm-tester buildhost.
** status: Code to do it already exists in our ansible files. We nedd to apply it to osmo-gsm-tester nodes.
</pre> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=94412018-05-19T10:40:07Zlaforge
<ul></ul><p>On Fri, May 18, 2018 at 03:35:31PM +0000, pespin [REDMINE] wrote:</p>
<blockquote>
<ul>
<li>Find a way to programatically find /dev/ttyUSBX for the motorla phone.</li>
</ul>
</blockquote>
<p>if you use a sysmocom CP2102 adapter, it will have a device-unique fixed serial number,<br />which you can find in /dev/serial/by-id/</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=94642018-05-23T18:34:47Zpespin
<ul></ul><p>Current state:</p>
<p>I have most of the required elements set up and running:<br />- TTCN3 BTS tests are pulled from registry.sysmocom.de docker.<br />- jenkins job to run the tests in prod setup (+ merged in osmo-ci): <a class="external" href="https://jenkins.osmocom.org/jenkins/view/osmo-gsm-tester/job/osmo-gsm-tester_ttcn3/">https://jenkins.osmocom.org/jenkins/view/osmo-gsm-tester/job/osmo-gsm-tester_ttcn3/</a><br />- improvmenets of osmo-gsm-tester + specific ttcn3 test subdir with all required stuff to run the TTCN3 tests. Can be found in <a class="external" href="https://git.osmocom.org/osmo-gsm-tester/log/?h=pespin/ttcn3">https://git.osmocom.org/osmo-gsm-tester/log/?h=pespin/ttcn3</a><br />- osmocon with motorola phone seems to be running fine in in RnD main unit.<br />- A BTS_Tests.cfg is generated from a template with the IP addr used by the BTS set up by osmo-gsm-tester.<br />- BTS is respawned automatically when it stops (due to TTCN3 tests closing the RSL link).<br />- docker container is killed correctly if osmo-gsm-tester test times out.<br />- I can run the setup in RnD through ssh easily: <code>OSMO_GSM_TESTER_CONF=/home/pespin/osmo-gsm-tester/ttcn3 ./run_remote.sh -T -l dbg</code><br />- All TTCN3 tests are stored correctly under osmo-gsm-tester test's rundir, ready to be packer and stored as an artifact.</p>
<p>Current status:<br />There seem to be some issues with TTCN3 connecting to IPA-CTRL, I'll have to look at the logs+pcap files for more details.</p>
<p>TODO:<br />- Add some manually installed stuff in main unit with ansible<br />- Collect the TTCN3 junit log and make sure the run.tar.gz is stored as artifact.</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=94762018-05-24T16:42:52Zpespin
<ul></ul>Updated status:
<ul>
<li>All elements are in place and starting/stopping correctly in <a class="external" href="https://git.osmocom.org/osmo-gsm-tester/log/?h=pespin/ttcn3">https://git.osmocom.org/osmo-gsm-tester/log/?h=pespin/ttcn3</a></li>
<li>I fixed the TTCN3 IPA-CTRL reported issues, now some tests are already passing.</li>
<li>Required config in ansible are already in gerrit and setup correctly in <a class="external" href="https://gerrit.osmocom.org/#/c/osmo-ci/+/9274/">https://gerrit.osmocom.org/#/c/osmo-ci/+/9274/</a></li>
<li>TTCN3 junit log is collected in the correct path.</li>
</ul>
TODO:
<ul>
<li>Add motorola phone to Prod setup (with <a class="user active" href="https://projects.osmocom.org/users/72">roh</a>) so we can run the jenkins setup.</li>
<li>Investigate issue with osmocon/kernel/hw not switching the DTR correctly most of the times in the RnD main unit (with <a class="user active" href="https://projects.osmocom.org/users/128">mschramm</a>).</li>
<li>TTCN3 has an issue opening the PCU socket, but the file is there in docker (I checked with file /data/unix/pcu_sock). I added a commit in ttcn3 to print the error str associated in <a class="external" href="https://gerrit.osmocom.org/#/c/osmo-ttcn3-hacks/+/9273/">https://gerrit.osmocom.org/#/c/osmo-ttcn3-hacks/+/9273/</a>. Once it's merged, I need to push an updated ttcn3-bts-test docker image and re-try and see the reason. For now I disabled the PCU socket in the tests by using an empty string as pcu socket path in BTS_Tests.cfg.</li>
<li>Most tests still don't pass, they give some errors probably related to l1 after getting into osmocon. I still need to check into the details.</li>
</ul> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=95002018-05-25T16:02:17Zpespin
<ul></ul><p>a motorola phone has been installed in the prod setup, but it seems it has the same serial issues than the one in RnD setup (phone not power cycled when osmocon process is started and serial device is opened).</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=95062018-05-25T19:10:08Zlaforge
<ul></ul><p>On Fri, May 25, 2018 at 04:02:18PM +0000, pespin [REDMINE] wrote:</p>
<blockquote>
<p>a motorola phone has been installed in the prod setup, but it seems it has the same serial issues than the one in RnD setup (phone not power cycled when osmocon process is started and serial device is opened).</p>
</blockquote>
<p>as it has been mentioned before, it's not the phone that has the issue,<br />but it's the operating system (most likely cp210x usb-serial driver) in<br />debian stable.</p>
<p>Have you meanwhile tried to adjust DTR manually? Maybe it's just the<br />automatic DTR switch-off (high-level) on port close that's broken, and<br />the manuel setting via termios still works? In that case, a small<br />modification in osmocon might be sufficient.</p>
<p>If that also doeesn't work, I suppose we'll have to resort to building a<br />patched kernel, or at very least a patched cp210x.ko :/ It may also<br />make sense to report this to Debian upstream so they can fix it in their<br />next stable kernel update?</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=95132018-05-25T21:10:53Zpespin
<ul></ul><p>laforge wrote:</p>
<blockquote>
<p>Have you meanwhile tried to adjust DTR manually? Maybe it's just the<br />automatic DTR switch-off (high-level) on port close that's broken, and<br />the manuel setting via termios still works? In that case, a small<br />modification in osmocon might be sufficient.</p>
</blockquote>
<p>The manual setting of DTR is already being done at startup.<br />osmocon calls "osmo_serial_init" in <a class="external" href="https://git.osmocom.org/osmocom-bb/tree/src/host/osmocon/osmocon.c#n1467">https://git.osmocom.org/osmocom-bb/tree/src/host/osmocon/osmocon.c#n1467</a><br />Then osmo_serial_init sets DTR: <a class="external" href="https://git.osmocom.org/libosmocore/tree/src/serial.c#n112">https://git.osmocom.org/libosmocore/tree/src/serial.c#n112</a></p>
<p>So if I understand correctly, your idea is to try calling ioctl TIOCMBIC (clear bit) TIOCM_DTR before calling close() on the fd right? I'll give it a try on Monday.</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=95142018-05-26T07:50:05Zlaforge
<ul></ul><p>On Fri, May 25, 2018 at 09:10:53PM +0000, pespin [REDMINE] wrote:</p>
<blockquote>
<p>laforge wrote:</p>
<blockquote>
<p>Have you meanwhile tried to adjust DTR manually? Maybe it's just the<br />automatic DTR switch-off (high-level) on port close that's broken, and<br />the manuel setting via termios still works? In that case, a small<br />modification in osmocon might be sufficient.</p>
</blockquote>
<p>The manual setting of DTR is already being done at startup.</p>
</blockquote>
<p>yes, that's clear. But AFAIK the problem was not that the phone isn't powered on,<br />but it is that it's never powered off.</p>
<blockquote>
<p>So if I understand correctly, your idea is to try calling ioctl TIOCMBIC (clear bit) TIOCM_DTR before<br />calling close() on the fd right? I'll give it a try on Monday.</p>
</blockquote>
<p>I can't recite the details about the ioctl() to use, but yes, the idea is<br />to explicitly release DTR before close. Or, in case of an unclean shutdown,<br />one might also think of a DTR cycling at startup (i.e. first clear/release DTR,<br />wait a second, then set it).</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=95152018-05-26T08:39:43Zlaforge
<ul><li><strong>File</strong> <a href="/attachments/3164">cp210x-dkms_4.16-0.tar.gz</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/3164/cp210x-dkms_4.16-0.tar.gz">cp210x-dkms_4.16-0.tar.gz</a> added</li><li><strong>File</strong> <a href="/attachments/3165">cp210x-dkms_4.16-0.dsc</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/3165/cp210x-dkms_4.16-0.dsc">cp210x-dkms_4.16-0.dsc</a> added</li><li><strong>File</strong> <a href="/attachments/3166">cp210x-4.16.tar.bz2</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/3166/cp210x-4.16.tar.bz2">cp210x-4.16.tar.bz2</a> added</li><li><strong>File</strong> <a href="/attachments/3167">cp210x-dkms_4.16-0_all.deb</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/3167/cp210x-dkms_4.16-0_all.deb">cp210x-dkms_4.16-0_all.deb</a> added</li></ul><p>I created a DKMS package for current mainline cp210x which can be build on debian 9. I also created a debian package that can be installed with <code>dpkg -i</code> and which will ensure the module will be re-build using dkms on any future kernel update.</p>
<p>Please try if this fixes the problem, and if so, integrate with ansible.</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=95402018-05-28T12:43:19Zpespin
<ul></ul><p>I did several tests by clearing the DTR bits, then setting them after receiving the fd from osmo_serial_init():<br /><pre>
dnload.serial_fd.fd = osmo_serial_init(serial_dev, MODEM_BAUDRATE);
+ /* Reset ready to read/write */
+ fprintf(stderr, "Clearing DTR bit\n");
+ int v24 = TIOCM_DTR | TIOCM_RTS;
+ if (ioctl(dnload.serial_fd.fd, TIOCMBIC, &v24) < 0)
+ fprintf(stderr, "ioctl(TIOCMBIC) %d", errno);
+ usleep(5*1000*1000);
+ fprintf(stderr, "Setting DTR bit\n");
+ if (ioctl(dnload.serial_fd.fd, TIOCMBIS, &v24) < 0)
+ fprintf(stderr, "ioctl(TIOCMBIS) %d", errno);
</pre></p>
<p>I also tried using getchar() instead of usleep() to control the state changes and be able to visually check the modem.</p>
<p>First times I run it, it seemed to have to difference: Modem still prints "Charinging" in the screen, and I get nothing in the serial (osmocon blocked in select()).<br />I then re-plugged the usb connector while osmocon was running (with DTR suppousedly on), and then next time I run osmocon the fw was loaded correctly. But then next times I saw that actually the previously loaded by osmocon firmware was still running, and that the modem was not being rebooted. So it is now running the osmocom fw even when the serial session is closed, or even when we explicitly clear the DTR bit as shown by code above.</p>
<p>So it really seems kernel is not correctly setting the DTR bit on the device. I'll give a try to the dkms package and see how it goes.</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=95452018-05-28T13:56:30Zpespin
<ul></ul><p>I tested the provided kernel module, and I got same results, it is still not working.</p>
<p>I used:<br /><pre>
rmmod cp210x
dpkg -i ....deb
modprobe cp210x
</pre></p>
<p>And then checked that the out of tree module was loaded:<br /><pre>
[868127.383269] cp210x: loading out-of-tree module taints kernel.
</pre></p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=95472018-05-28T14:51:14Zpespin
<ul></ul><p>I'm testing now the motorola phone in an APU2 I have on my table, running same debian and kernel version than the one in osmo-gsm-tester RnD setup:<br /><pre>
Linux apu 4.9.0-6-amd64 #1 SMP Debian 4.9.88-1+deb9u1 (2018-05-07) x86_64 GNU/Linux
</pre></p>
<p>I connected the usb serial connector directly to one of the USB slots in the APU2 board. Everything seems to be working fine with osmocon, like in my PC.</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=95482018-05-28T15:10:06Zlaforge
<ul></ul><p>On Mon, May 28, 2018 at 02:51:14PM +0000, pespin [REDMINE] wrote:</p>
<blockquote>
<p>I'm testing now the motorola phone in an APU2 I have on my table, running same debian and kernel version than the one in osmo-gsm-tester RnD setup:</p>
<p>I connected the usb serial connector directly to one of the USB slots in the APU2 board. Everything seems to be working fine with osmocon, like in my PC.</p>
</blockquote>
<p>this is very odd.</p>
<p>Maybe there is some other process on the gsm-tester APU which open the serial port [in parallel]?</p>
<p>If there's of course another process holding the port still open, then existing 'osmocon' will<br />not release DTR.</p>
<p>'lsof' should help to find out...</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=95492018-05-28T15:27:33Zpespin
<ul></ul>I finally found how to trigger the work and non-working state for the motorola phone
<ul>
<li>plug in the RF antenna -> screen prints "charging" and osmocon cannot connect</li>
<li>unplug the RT antenna -> screen stops printing "charging" and osmocon can connect</li>
</ul> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=95752018-05-29T15:13:44Zroh
<ul></ul><p>i reworked 2 boards to use high-side switching by using AAT4626 from our lab-stock (any high side switch would do)</p>
<p>since they expect high level i removed just the power-fet, bridged ground and cut a trace to have the vbat (via diode) and vcharge switched and the pullups not switched. works nicely for me (tested with neocon and minicom)</p>
<p>the boards are connected to the -dev and -prod testers again.</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=95762018-05-29T15:58:32Zpespin
<ul></ul><p>I confirm -dev can be opened with osmocon fine all the time now.<br />For prod, it seems minicom powers on/off the device fine, but I don't know why yet, osmocon doesn't power it on. I tried updating the cp210x module to the 4.16 dkms one, like in -dev, but it didn't help with the issue.</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=96372018-05-30T14:20:29Zlaforge
<ul><li><strong>Tags</strong> set to <i>TTCN3</i></li></ul> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=96822018-05-31T09:47:42Zpespin
<ul></ul><p>I rebooted the machine and made sure to start with a clean libosmocore+osmocon setup to investigate why was the serial not working properly yet, but it turns out after using this clean test env it works fine now, like in RnD. So not sure why, but it's working now. I'm starting to run the jenkins job to get some first results and see how far can I get.</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=96842018-05-31T12:21:27Zpespin
<ul></ul><p>First jenkins run finished, report can be found in:<br /><a class="external" href="https://jenkins.osmocom.org/jenkins/view/osmo-gsm-tester/job/osmo-gsm-tester_ttcn3/21/testReport/">https://jenkins.osmocom.org/jenkins/view/osmo-gsm-tester/job/osmo-gsm-tester_ttcn3/21/testReport/</a></p>
<p>66 failures out of 84 tests.</p>
The archived artifacts should contain all the information required to find out what's failing in each case.<br />Some interesting files/paths:
<ul>
<li>osmocon log: run.2018-05-31_11-41-09/ttcn3_bts_tests:trx/ttcn3_bts_tests.py/osmocon_ttyUSB3/stdout</li>
<li>TTCN3 log: run.2018-05-31_11-41-09/ttcn3_bts_tests:trx/ttcn3_bts_tests.py/ttcn3/stdout</li>
<li>BTS_Tests.cfg used: run.2018-05-31_11-41-09/ttcn3_bts_tests:trx/ttcn3_bts_tests.py/ttcn3/BTS_Tests.cfg</li>
<li>Separate test logs + pcaps: run.2018-05-31_11-41-09/ttcn3_bts_tests:trx/ttcn3_bts_tests.py/ttcn3/logs/bts-tester/</li>
</ul> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=96852018-05-31T12:51:01Zlaforge
<ul></ul><p>On Thu, May 31, 2018 at 12:21:27PM +0000, pespin [REDMINE] wrote:</p>
<blockquote>
<p>66 failures out of 84 tests.</p>
</blockquote>
<p>this is <strong>highly</strong> unexpected. As indicated before, I was able to get almost all tests to pass,<br />except those dealing with measurement values, which is of course expected.</p>
<p>From looking at the results, I would expect at least the following to pass:</p>
<ul>
<li>TC_chan_act_*</li>
<li>TC_deact_sacch</li>
<li>TC_encr_cmd_*</li>
<li>TC_rll_*</li>
<li>TC_rll_*</li>
<li>TC_sacch_*</li>
<li>TC_si_sched_*</li>
</ul>
<p>All of the above have nothing related to measurement or statistics/rate, and should pass.</p>
<p>Even more worrying are</p>
<ul>
<li>TC_dyn_ipa_pdch_act_deact</li>
<li>TC_dyn_ipa_pdch_act_tchf_act_nack</li>
<li>TC_dyn_osmo_pdch_act_deact</li>
<li>TC_dyn_osmo_pdch_double_act</li>
<li>TC_pcu_act_req</li>
<li>TC_pcu_act_req_wrong_bts</li>
<li>TC_pcu_act_req_wrong_trx</li>
<li>TC_pcu_act_req_wrong_ts</li>
<li>TC_pcu_deact_req</li>
<li>TC_pcu_deact_req_wrong_ts</li>
</ul>
<p>which all don't even use L1CTL / the phone. They should work even without any phone present in the first<br />place, as they only use the IP based interfaces of the BTS to test.</p>
<p>So I think there's still a lot going wrong in the setup.</p>
<p>In fact, I think from the tests results you are getting, not a single one that passes is using the phone.</p>
<p>So we have</p>
<ul>
<li>tests that don't use the phone but don't pass (very odd)</li>
<li>tests that don't use the phone but pass (good)</li>
<li>tests that use the phone all don't pass</li>
</ul> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=97932018-06-07T17:55:01Zpespin
<ul></ul><p>I submited patch <a class="external" href="https://gerrit.osmocom.org/#/c/9492/">https://gerrit.osmocom.org/#/c/9492/</a> in order to change the RxLev sent through L1CTL, but it seems it's not needed so far and actually the arfcn sent by ttcn3 was not matching the one set up by osmo-gsm-tester. By configuring the arfcn correctly in the BTS_Tests.cfg.tmpl file tests now reach further and ttcn3 states it receives Assignment Commands after sending a RACH.</p>
<p>However, at some point both ends (ttcn3 and osmocon) started failing stating an error with the unix socket. ttcn3 "connection refused" and osmocon "Connection reset by peer".</p>
<p>osmocon: <code>Err from socket: Connection reset by peer</code><br />ttcn3: <code>TC_sacch_info_mod(92)@64afac245c1c: Dynamic test case error: Stand-alone receive statement failed in file L1CTL_PortType.ttcn, line 190. (Connection refused)</code></p>
<p>It seems ttcn3 test is probably re-connecting the unix socket in f_connect_reset(), and osmocon doesn't like that and exits. This behavior was not shown while Harald did the BTS tests without real hardware because in there trxcon is used instead of osmocon, so it seems osmocon needs to be fixed/improved to support this kind of scenarios.</p>
<p>About the "tests that don't use the phone but don't pass" from Harald's last post, I bet it's related to the fact that I disabled using the PCU socket (<code>BTS_Tests.mp_pcu_socket := ""</code>) due to some issues I was finding in the setup, I'll look into that later.</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=97952018-06-07T19:08:23Zfixeria
<ul></ul><p>Hi Pau,</p>
<p>I think that the problem is here:</p>
<p><a class="external" href="https://git.osmocom.org/osmocom-bb/tree/src/host/osmocon/osmocon.c#n1161">https://git.osmocom.org/osmocom-bb/tree/src/host/osmocon/osmocon.c#n1161</a></p>
<p>it exits if 'rc == 0', while trxcon doesn't exit in such cases.</p>
<p>I hope, this will help.</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=98602018-06-12T13:43:05Zpespin
<ul></ul><p>I tried removing that exit path (rc==0) and added an extra stderr log line to show what read() from handle_buffer returns. <br /><pre>
commit a607471194aab81261c9b67a41a871eb36a42e6e (HEAD -> pespin/no-exit, origin/pespin/no-exit)
Author: Pau Espin Pedrol <pespin@sysmocom.de>
Date: Tue Jun 12 14:38:28 2018 +0200
osmocon: Do not exit on read return 0
Change-Id: Ifedfbc9742792b4f8d7850dc7ce84d5b3dbc0096
diff --git a/src/host/osmocon/osmocon.c b/src/host/osmocon/osmocon.c
index 76f60374..e9427884 100644
--- a/src/host/osmocon/osmocon.c
+++ b/src/host/osmocon/osmocon.c
@@ -789,8 +789,11 @@ static int handle_buffer(int buf_used_len)
}
nbytes = read(dnload.serial_fd.fd, bufptr, buf_left);
- if (nbytes <= 0)
+ if (nbytes <= 0) {
+ fprintf(stderr, "handle_buffer() read(%d) = %d (%s)\n",
+ buf_left, nbytes, strerror(errno));
return nbytes;
+ }
if (!dnload.expect_hdlc) {
printf("got %i bytes from modem, ", nbytes);
@@ -1173,8 +1176,6 @@ static int serial_read(struct osmo_fd *fd, unsigned int flags)
while ((rc = handle_read()) > 0);
break;
}
- if (rc == 0)
- exit(2);
}
if (flags & BSC_FD_WRITE) {
</pre></p>
<p>It seems however that osmocon is not finishing in there. It seems to exit due to a SIGPIPE error. I could get this trace while running it out of osmo-gsm-tester under gdb while running the TTCN3 tests:</p>
<pre>
handle_buffer() read(1) = -1 (Resource temporarily unavailable)
handle_buffer() read(1) = -1 (Resource temporarily unavailable)
handle_buffer() read(1) = -1 (Resource temporarily unavailable)
handle_buffer() read(1) = -1 (Resource temporarily unavailable)
Program received signal SIGPIPE, Broken pipe.
0x00007ffff76da730 in __write_nocancel () at ../sysdeps/unix/syscall-template.S:84
84 ../sysdeps/unix/syscall-template.S: No such file or directory.
(gdb) bt
#0 0x00007ffff76da730 in __write_nocancel () at ../sysdeps/unix/syscall-template.S:84
#1 0x0000555555556e31 in hdlc_tool_cb (dlci=5 '\005', msg=0x55555576cec0) at /home/pespin/dev/git/osmocom-bb/src/host/osmocon/osmocon.c:769
#2 0x0000555555559376 in dispatch_rx_msg (dlci=5 '\005', msg=0x55555576cec0)
at /home/pespin/dev/git/osmocom-bb/src/host/osmocon/../../target/firmware/comm/sercomm.c:243
#3 0x000055555555949c in sercomm_drv_rx_char (ch=126 '~') at /home/pespin/dev/git/osmocom-bb/src/host/osmocon/../../target/firmware/comm/sercomm.c:286
#4 0x0000555555556fec in handle_buffer (buf_used_len=7) at /home/pespin/dev/git/osmocom-bb/src/host/osmocon/osmocon.c:804
#5 0x000055555555703b in handle_read () at /home/pespin/dev/git/osmocom-bb/src/host/osmocon/osmocon.c:816
#6 0x0000555555557d56 in serial_read (fd=0x55555575c358 <dnload+24>, flags=1) at /home/pespin/dev/git/osmocom-bb/src/host/osmocon/osmocon.c:1176
#7 0x00007ffff79a83ae in osmo_fd_disp_fds (_eset=0x7fffffffe2a0, _wset=0x7fffffffe220, _rset=0x7fffffffe1a0) at select.c:217
#8 osmo_select_main (polling=<optimized out>) at select.c:257
#9 0x00005555555588df in main (argc=10, argv=0x7fffffffe478) at /home/pespin/dev/git/osmocom-bb/src/host/osmocon/osmocon.c:1523
</pre> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=98612018-06-12T13:44:02Zpespin
<ul></ul><p>It's strange too that I see a lot of EAGAIN errors in read(), because afaik it should be driven by select(), so it should only signal READ cb if there's something to read from the device...</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=98622018-06-12T14:55:37Zpespin
<ul></ul><p><a class="external" href="https://gerrit.osmocom.org/#/c/osmocom-bb/+/9562">https://gerrit.osmocom.org/#/c/osmocom-bb/+/9562</a> seems to fix the premature exit of osmocon while running ttcn3 BTS_Tests.</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=98632018-06-12T15:18:18Zpespin
<ul><li><strong>% Done</strong> changed from <i>30</i> to <i>40</i></li></ul><p>Running with trx-b200 in RnD setup after the patch shared in last comment:</p>
<pre>
Comparing expected results /osmo-ttcn3-hacks/bts/expected-results.xml against results in junit-xml-19.log
--------------------
pass BTS_Tests.TC_chan_act_stress
pass BTS_Tests.TC_chan_act_react
pass BTS_Tests.TC_chan_deact_not_active
pass BTS_Tests.TC_chan_act_wrong_nr
pass->FAIL BTS_Tests.TC_deact_sacch
pass BTS_Tests.TC_sacch_filling
pass BTS_Tests.TC_sacch_info_mod
pass BTS_Tests.TC_sacch_multi
xfail BTS_Tests.TC_sacch_multi_chg
pass BTS_Tests.TC_rach_content
pass->FAIL BTS_Tests.TC_rach_count
pass->FAIL BTS_Tests.TC_rach_max_ta
pass->FAIL BTS_Tests.TC_meas_res_sign_tchf
pass->FAIL BTS_Tests.TC_meas_res_sign_tchh
pass BTS_Tests.TC_meas_res_sign_sdcch4
pass BTS_Tests.TC_meas_res_sign_sdcch8
pass->FAIL BTS_Tests.TC_meas_res_sign_tchh_toa256
pass->FAIL BTS_Tests.TC_conn_fail_crit
pass->FAIL BTS_Tests.TC_paging_imsi_80percent
pass->FAIL BTS_Tests.TC_paging_tmsi_80percent
pass->FAIL BTS_Tests.TC_paging_imsi_200percent
pass->FAIL BTS_Tests.TC_paging_tmsi_200percent
pass BTS_Tests.TC_rsl_protocol_error
pass BTS_Tests.TC_rsl_mand_ie_error
pass BTS_Tests.TC_rsl_ie_content_error
pass BTS_Tests.TC_si_sched_default
pass BTS_Tests.TC_si_sched_1
pass BTS_Tests.TC_si_sched_2bis
pass BTS_Tests.TC_si_sched_2ter
pass BTS_Tests.TC_si_sched_2ter_2bis
pass->FAIL BTS_Tests.TC_si_sched_2quater
pass BTS_Tests.TC_si_sched_13
pass->FAIL BTS_Tests.TC_si_sched_13_2bis_2ter_2quater
pass BTS_Tests.TC_ipa_dlcx_not_active
pass BTS_Tests.TC_ipa_crcx_twice_not_active
pass BTS_Tests.TC_ipa_crcx_mdcx_dlcx_not_active
pass BTS_Tests.TC_ipa_crcx_mdcx_mdcx_dlcx_not_active
pass BTS_Tests.TC_ipa_crcx_sdcch_not_active
pass->FAIL BTS_Tests.TC_pcu_act_req
pass->FAIL BTS_Tests.TC_pcu_act_req_wrong_ts
pass->FAIL BTS_Tests.TC_pcu_act_req_wrong_bts
pass->FAIL BTS_Tests.TC_pcu_act_req_wrong_trx
pass->FAIL BTS_Tests.TC_pcu_deact_req
pass->FAIL BTS_Tests.TC_pcu_deact_req_wrong_ts
pass->FAIL BTS_Tests.TC_pcu_ver_si13
pass->FAIL BTS_Tests.TC_pcu_data_req_wrong_bts
pass->FAIL BTS_Tests.TC_pcu_data_req_wrong_trx
pass->FAIL BTS_Tests.TC_pcu_data_req_wrong_ts
xfail BTS_Tests.TC_pcu_data_req_ts_inactive
pass->FAIL BTS_Tests.TC_pcu_data_req_pdtch
pass->FAIL BTS_Tests.TC_pcu_data_req_ptcch
pass->FAIL BTS_Tests.TC_pcu_data_req_agch
pass->FAIL BTS_Tests.TC_pcu_data_req_imm_ass_pch
pass->FAIL BTS_Tests.TC_pcu_rach_content
pass->FAIL BTS_Tests.TC_pcu_paging_from_rsl
pass->FAIL BTS_Tests.TC_dyn_osmo_pdch_act_deact
pass BTS_Tests.TC_dyn_osmo_pdch_unsol_deact
pass->FAIL BTS_Tests.TC_dyn_osmo_pdch_double_act
pass BTS_Tests.TC_dyn_osmo_pdch_tchf_act
pass BTS_Tests.TC_dyn_osmo_pdch_tchh_act
pass->FAIL BTS_Tests.TC_dyn_ipa_pdch_act_deact
pass BTS_Tests.TC_dyn_ipa_pdch_tchf_act
pass BTS_Tests.TC_dyn_ipa_pdch_tchf_act_pdch_act_nack
pass->FAIL BTS_Tests.TC_dyn_ipa_pdch_act_tchf_act_nack
pass->FAIL BTS_Tests.TC_rll_est_ind
pass BTS_Tests.TC_rll_est_req_DCCH_3
pass BTS_Tests.TC_rll_est_req_ACCH_3
pass->FAIL BTS_Tests.TC_rll_rel_ind_DCCH_0
pass->FAIL BTS_Tests.TC_rll_rel_ind_DCCH_3
pass BTS_Tests.TC_rll_rel_ind_ACCH_0
pass BTS_Tests.TC_rll_rel_ind_ACCH_3
pass->FAIL BTS_Tests.TC_rll_rel_req
pass BTS_Tests.TC_rll_unit_data_req_DCCH
pass BTS_Tests.TC_rll_unit_data_req_ACCH
pass->FAIL BTS_Tests.TC_rll_unit_data_ind_DCCH
pass BTS_Tests.TC_rll_unit_data_ind_ACCH
pass BTS_Tests.TC_chan_act_a51
pass BTS_Tests.TC_chan_act_a52
pass->FAIL BTS_Tests.TC_chan_act_a53
pass->FAIL BTS_Tests.TC_encr_cmd_a51
</pre>
<p>So around half of them are passing now.</p>
<p>The PCU ones probably fail because I disabled passing the PCU socket path, I need to investigate why it couldn't open it to enable it again.</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=98892018-06-13T15:05:48Zpespin
<ul></ul><p>junit jenkins output can already be found in <a class="external" href="https://jenkins.osmocom.org/jenkins/view/osmo-gsm-tester/job/osmo-gsm-tester_ttcn3/test_results_analyzer/">https://jenkins.osmocom.org/jenkins/view/osmo-gsm-tester/job/osmo-gsm-tester_ttcn3/test_results_analyzer/</a></p>
<p>As mentioned before, I still need to check the PCU socket issue.</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=99242018-06-15T15:41:26Zpespin
<ul></ul><p>PCU socket issue fixed (by bind mounting the directory instead of the file, otherwise if the file is recreated outside the container, the container cannot see the new file and keeps seeing the old one).</p>
<p>I modified the code to store the XML junit file as change its classname to include the osmo-gsm-tester suite name, this way if we have for instance:<br />ttcn3_bts_tests:trx-b200<br />ttcn3_bts_tests:trx-sysmo5k<br />ttcn3_bts_tests:sysmo</p>
<p>We get 3 xml files, each of them having a different class name and all of them end up being shown in Jenkins' "Test Results Analyzer".</p>
<p>All the required bits can be found in 1 commit in osmo-gsm-tester branch "pespin/ttcn3".</p>
<p>TTCN3 running with a trx-b200 can be found in: <a class="external" href="https://jenkins.osmocom.org/jenkins/view/osmo-gsm-tester/job/osmo-gsm-tester_ttcn3/test_results_analyzer/">https://jenkins.osmocom.org/jenkins/view/osmo-gsm-tester/job/osmo-gsm-tester_ttcn3/test_results_analyzer/</a></p>
<p>I need to find a way to clean the build history of the job, otherwise the old entries (without having the osmo-gsm-tester suite name) still are shown and it's a bit messy.</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=99362018-06-16T15:10:08Zlaforge
<ul></ul><p>There are still way too many tests failing, so I suspect there's still some fundamental<br />problem. The TC_rll_* and TC_sacch_* should be exactly like in the virtual setup.</p>
<p>I still believe it is best to establish a baseline with a OsmocomBB phone + BTS<br />on your desk and then compare that to the automatic test setup.</p>
<p>Regards,<br /> Harald</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=99392018-06-17T22:10:20Zpespin
<ul></ul><p>I see a lot of tests (rll and sacch) failing with this kind of message:</p>
<p>"Dynamic test case error: Error message was received from MC: The connect operation refers to test component with component reference 41, which has already terminated."</p>
<p>I can now easily run separate tests in the RnD setup through ssh, so I can debug each test in there. As far as I remember, I noticed that some tests observed to be failing in the Prod automated run (Test Results Analyzer) passed fine when running them separately in RnD.<br />I'm still not applying the Rxlev != 57 in the tests, since they seemed to work fine enough with 57 as used in the virt setup. I know half of the tests are not yet passing, but after fixing the PCU part I wanted to let them run for a while to see the success/fail patterns and see what kind of error to look at and fix for each test.</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=99532018-06-18T17:50:27Zpespin
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-5 priority-2 priority-default closed" href="/issues/3353">Bug #3353</a>: osmo-bts: TTCN3 BTS_Tests.TC_rll_est_ind fail behavior not clear</i> added</li></ul> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=99552018-06-18T18:04:21Zpespin
<ul></ul><p>Here's a full log of the "Dynamic test case error", it already happened during the first test of the full suite, so it doesn't seem to be related to state left from previous tests:<br /><pre>
MC2> smtc
Executing all items of [EXECUTE] section.
MC2> MTC@f58d536352bc: Starting external command `../ttcn3-tcpdump-start.sh BTS_Tests.TC_rll_est_ind'.
Waiting for tcpdump to start... 0
MTC@f58d536352bc: External command `../ttcn3-tcpdump-start.sh BTS_Tests.TC_rll_est_ind' was executed successfully (exit status: 0).
MTC@f58d536352bc: Test case TC_rll_est_ind started.
TC_rll_est_ind-RSL-IPA(3)@f58d536352bc: IPA: Connected
TC_rll_est_ind-RSL-IPA(3)@f58d536352bc: CCM Tx:{ msg_type := IPAC_MSGT_ID_GET (4), u := { get := { { len := 1, tag := IPAC_IDTAG_UNITNAME (1) } } } }
TC_rll_est_ind-RSL-IPA(3)@f58d536352bc: CCM Rx:{ msg_type := IPAC_MSGT_ID_RESP (5), u := { resp := { { len := 28, tag := IPAC_IDTAG_UNITNAME (1), data := "sysmoBTS-00-00-00-00-00-00" & char(0, 0, 0, 0) } } } }
TC_rll_est_ind-RSL-IPA(3)@f58d536352bc: IPA ID RESP: { { len := 28, tag := IPAC_IDTAG_UNITNAME (1), data := "sysmoBTS-00-00-00-00-00-00" & char(0, 0, 0, 0) } }
TC_rll_est_ind-RSL-IPA(3)@f58d536352bc: CCM Tx:{ msg_type := IPAC_MSGT_ID_ACK (6), u := omit }
TC_rll_est_ind-RSL-IPA(3)@f58d536352bc: CCM Rx:{ msg_type := IPAC_MSGT_ID_ACK (6), u := omit }
MTC@f58d536352bc: Setting RSL_SYSTEM_INFO_3 (3): { header := { l2_plen := { l2_plen := 18, zero_one := '01'B }, skip_indicator := 0, rr_protocol_discriminator := 6, message_type := SYSTEM_INFORMATION_TYPE_3 (27) }, payload := { si3 := { cell_id := 23, lai := { mcc_mnc := '262F42'H, lac := 42 }, ctrl_chan_desc := { msc_r99 := true, att := true, bs_ag_blks_res := 1, ccch_conf := CCHAN_DESC_1CCCH_COMBINED (1), si22ind := false, cbq3 := CBQ3_IU_MODE_NOT_SUPPORTED (0), spare := '00'B, bs_pa_mfrms := 0, t3212 := 1 }, cell_options := { dn_ind := false, pwrc := false, dtx := MS_MAY_USE_UL_DTX (0), radio_link_tout_div4 := 7 }, cell_sel_par := { cell_resel_hyst_2dB := 2, ms_txpwr_max_cch := 7, acs := '0'B, neci := true, rxlev_access_min := 0 }, rach_control := { max_retrans := RACH_MAX_RETRANS_7 (3), tx_integer := '1001'B, cell_barr_access := false, re_not_allowed := true, acc := '0000010000000000'B }, rest_octets := '2B2B2B2B'O ("++++") } } }
MTC@f58d536352bc: Setting RSL_SYSTEM_INFO_3 (3): '49061B001762F224002AC90001074740E504002B2B2B2B'O
MTC@f58d536352bc: Setting RSL_SYSTEM_INFO_2 (2): { header := { l2_plen := { l2_plen := 22, zero_one := '01'B }, skip_indicator := 0, rr_protocol_discriminator := 6, message_type := SYSTEM_INFORMATION_TYPE_2 (26) }, payload := { si2 := { bcch_freq_list := '00000000000000000000000000000000'O, ncc_permitted := '11111111'B, rach_control := { max_retrans := RACH_MAX_RETRANS_7 (3), tx_integer := '1001'B, cell_barr_access := false, re_not_allowed := true, acc := '0000010000000000'B } } } }
MTC@f58d536352bc: Setting RSL_SYSTEM_INFO_2 (2): '59061A00000000000000000000000000000000FFE50400'O
MTC@f58d536352bc: Setting RSL_SYSTEM_INFO_4 (4): { header := { l2_plen := { l2_plen := 12, zero_one := '01'B }, skip_indicator := 0, rr_protocol_discriminator := 6, message_type := SYSTEM_INFORMATION_TYPE_4 (28) }, payload := { si4 := { lai := { mcc_mnc := '262F42'H, lac := 42 }, cell_sel_par := { cell_resel_hyst_2dB := 2, ms_txpwr_max_cch := 7, acs := '0'B, neci := true, rxlev_access_min := 0 }, rach_control := { max_retrans := RACH_MAX_RETRANS_7 (3), tx_integer := '1001'B, cell_barr_access := false, re_not_allowed := true, acc := '0000010000000000'B }, cbch_chan_desc := omit, cbch_mobile_alloc := omit, rest_octets := ''O } } }
MTC@f58d536352bc: Setting RSL_SYSTEM_INFO_4 (4): '31061C62F224002A4740E50400'O
MTC@f58d536352bc: "TC_rll_est_ind": XXX Starting { sapi := 0, link_id := { c := FACCH_SDCCH (0), na := false, prio := SAPI0_PRIO_NORMAL (0), sapi := 0 }, l3 := '01020304'O, exp := true } on { u := { sdcch4 := { tag := '001'B, sub_chan := 2 } }, tn := 0 }
MTC@f58d536352bc: "TC_rll_est_ind": XXX Starting { sapi := 0, link_id := { c := FACCH_SDCCH (0), na := false, prio := SAPI0_PRIO_NORMAL (0), sapi := 0 }, l3 := ''O, exp := true } on { u := { sdcch4 := { tag := '001'B, sub_chan := 2 } }, tn := 0 }
TC_rll_est_ind-RSL(4)@f58d536352bc: Dynamic test case error: Data cannot be sent on port CLIENT_PT to component 6 because there is no connection towards component 6.
TC_rll_est_ind-RSL-IPA(3)@f58d536352bc: Dynamic test case error: Port IPA_RSL_PORT has neither connections nor mappings. Message cannot be sent on it.
MTC@f58d536352bc: "TC_rll_est_ind": XXX Starting { sapi := 3, link_id := { c := FACCH_SDCCH (0), na := false, prio := SAPI0_PRIO_NORMAL (0), sapi := 3 }, l3 := '01020304'O, exp := false } on { u := { sdcch4 := { tag := '001'B, sub_chan := 2 } }, tn := 0 }
MTC@f58d536352bc: Dynamic test case error: Error message was received from MC: The connect operation refers to test component with component reference 4, which has already terminated.
MTC@f58d536352bc: Test case TC_rll_est_ind finished. Verdict: error
MTC@f58d536352bc: Starting external command `../ttcn3-tcpdump-stop.sh BTS_Tests.TC_rll_est_ind error'.
Waiting for tcpdump to finish... 0 (prev_count=-1, count=8815)
Waiting for tcpdump to finish... 1 (prev_count=8815, count=9550)
MTC@f58d536352bc: External command `../ttcn3-tcpdump-stop.sh BTS_Tests.TC_rll_est_ind error' was executed successfully (exit status: 0).
MC@f58d536352bc: Test execution finished.
</pre></p>
<p>However, I have been running this test several times alone, and now that I switched back to running BTS_Tests.control again, it strikes.</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=103672018-07-18T09:46:36Zpespin
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Stalled</i></li></ul><p>Waiting until patches improving error detection / stopping tests at the right time are merged before continuing with looking at issues appearing in different tests, it will make my life easier:<br /><a class="external" href="https://gerrit.osmocom.org/#/c/osmo-ttcn3-hacks/+/9905/">https://gerrit.osmocom.org/#/c/osmo-ttcn3-hacks/+/9905/</a><br /><a class="external" href="https://gerrit.osmocom.org/#/c/osmo-ttcn3-hacks/+/9907/">https://gerrit.osmocom.org/#/c/osmo-ttcn3-hacks/+/9907/</a></p>
<p>Once they are merged, I need to generate a new docker file for TTCN3 tests and run the tests.</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=114372018-09-18T15:58:39Zpespin
<ul></ul><p>I had a look at current status of the test setup today and I found out most tests failed while trying to send a RESET.conf message through L1CTL to osmocon. Further investigation showed that osmocon starts and enters an infinite loop loading the firmware to RAM forever.</p>
<p>I checked several versions to make sure it was not a software regression, and it's not. I tested with an old sysroot (osmocon bin + libosmocore + osmocombb fw) that is known to have worked, and I saw same issue. Exact same binaries work fine in RnD setup.</p>
<p>Power-cycling the prod setup didn't help. It looks like a physicial or device related issue, I asked <a class="user active" href="https://projects.osmocom.org/users/72">roh</a> for support regarding the physisical HW aspects and he should look at it soon.</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=115892018-09-25T11:41:34Zpespin
<ul></ul><p>I'm running the tests now in osmo-gsm-tester RnD setup since there are issues with the motorola+osmocon in prod setup (infinite loop loading fw).</p>
<p>These are more or less the test set failing:<br /><pre>
BTS_Tests.TC_meas_res_sign_tchf fail
BTS_Tests.TC_meas_res_sign_tchh_toa256 fail
BTS_Tests.TC_paging_tmsi_200percent fail
BTS_Tests.TC_si_sched_13_2bis_2ter_2quater fail
BTS_Tests.TC_pcu_rach_content fail
BTS_Tests.TC_rll_est_ind fail
BTS_Tests.TC_rll_rel_ind_DCCH_0 fail
BTS_Tests.TC_rll_rel_ind_DCCH_3 fail
BTS_Tests.TC_rll_rel_ind_ACCH_0 fail
BTS_Tests.TC_rll_rel_ind_ACCH_3 fail
BTS_Tests.TC_rll_rel_req fail
BTS_Tests.TC_encr_cmd_a51 fail
</pre></p>
<p>I started looking at "TC_meas_res_sign_tchf", which fails this way:<br /><pre>
10:44:55.614221 131 BTS_Tests.ttcn:1299 setverdict(fail): pass -> fail reason: "Received unspecific MEAS RES { msg_disc := { msg_group := RSL_MDISC_DCHAN (4), transparent := false }, msg_type := RSL_MT_MEAS_RES (40), ies := { { iei := RSL_IE_CHAN_NR (1), body := { chan_nr := { u := { ch0 := RSL_CHAN_NR_Bm_ACCH (1) }, tn := 1 } } }, { iei := RSL_IE_MEAS_RES_NR (27), body := { meas_res_nr := 1 } }, { iei := RSL_IE_UPLINK_MEAS (25), body := { uplink_meas := { len := 3, rfu := '0'B, dtx_d := false, rxlev_f_u := 53, reserved1 := '00'B, rxlev_s_u := 53, reserved2 := '00'B, rxq_f_u := 0, rxq_s_u := 0, supp_meas_info := omit } } }, { iei := RSL_IE_BS_POWER (4), body := { bs_power := { reserved := 0, epc := false, fpc := false, power_level := 0 } } }, { iei := RSL_IE_L1_INFO (10), body := { l1_info := { ms_power_lvl := 7, fpc := false, reserved := 0, actual_ta := 0 } } }, { iei := RSL_IE_L3_INFO (11), body := { l3_info := { len := 18, payload := '0615282801C0000000000000000000000000'O } } }, { iei := RSL_IE_MS_TIMING_OFFSET (37), body := { ms_timing_offset := 63 } } } }", new component reason: "Received unspecific MEAS RES { msg_disc := { msg_group := RSL_MDISC_DCHAN (4), transparent := false }, msg_type := RSL_MT_MEAS_RES (40), ies := { { iei := RSL_IE_CHAN_NR (1), body := { chan_nr := { u := { ch0 := RSL_CHAN_NR_Bm_ACCH (1) }, tn := 1 } } }, { iei := RSL_IE_MEAS_RES_NR (27), body := { meas_res_nr := 1 } }, { iei := RSL_IE_UPLINK_MEAS (25), body := { uplink_meas := { len := 3, rfu := '0'B, dtx_d := false, rxlev_f_u := 53, reserved1 := '00'B, rxlev_s_u := 53, reserved2 := '00'B, rxq_f_u := 0, rxq_s_u := 0, supp_meas_info := omit } } }, { iei := RSL_IE_BS_POWER (4), body := { bs_power := { reserved := 0, epc := false, fpc := false, power_level := 0 } } }, { iei := RSL_IE_L1_INFO (10), body := { l1_info := { ms_power_lvl := 7, fpc := false, reserved := 0, actual_ta := 0 } } }, { iei := RSL_IE_L3_INFO (11), body := { l3_info := { len := 18, payload := '0615282801C0000000000000000000000000'O } } }, { iei := RSL_IE_MS_TIMING_OFFSET (37), body := { ms_timing_offset := 63 } } } }"
</pre></p>
<p>So it fails to match the specific measurements created at "f_TC_meas_res_periodic". It seems related to the fact that we don't use mp_bb_trxc_port in the HW setup because we don't use trxcon there (mp_bb_trxc_port = -1):<br /><pre>
if (mp_bb_trxc_port != -1) {
g_pars.l1_pars.meas_ul.full.rxlev := dbm2rxlev(-100);
f_trxc_fake_rssi(100);
g_pars.l1_pars.timing_offset_256syms := 512; /* 2 symbols */
f_trx_fake_toffs256(g_pars.l1_pars.timing_offset_256syms);
} else {
g_pars.l1_pars.timing_offset_256syms := 0; /* FIXME */
g_pars.l1_pars.meas_ul.full.rxlev := dbm2rxlev(-55); /* FIXME */
}
</pre></p>
<p>So those FIXME should be fixed, the question is what's actually th best procedure to do so. I'm sure "TC_meas_res_sign_tchh_toa256" fails due to same reason.</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=115912018-09-25T12:20:06Zlaforge
<ul></ul><p>On Tue, Sep 25, 2018 at 11:41:34AM +0000, pespin [REDMINE] wrote:</p>
<blockquote>
<p>So those FIXME should be fixed, the question is what's actually th best procedure to do so.</p>
</blockquote>
<ul>
<li>run the test in the given setup</li>
<li>record the min/max/avg values for timing and rxlev</li>
<li>enter them as a range of permitted values in the receive templates in BTS_Tests.ttcn</li>
</ul>
<p>I you want to make it less hard-coded, one could have the values exposed as module<br />paraemters in to the config file. I think it's best to enter/specify them is the following way:</p>
<ul>
<li>expected (average/mean) value</li>
<li>permitted tolerance (in percent, or absolute value)</li>
</ul>
<p>Then the template would permit the range (mean - tolerance) .. (mean + tolerance)<br />as the permitted range.</p>
<p>With some luck, the setup-specific value that needs to change is the expected value,<br />as this depends on the attenuation between transmitter and receiver.<br />The tolerance should be the same on every setup/installation, as this is only determined<br />by the overall tolerances of the various elements used.</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=115922018-09-25T13:07:26Zfixeria
<ul></ul><p>It should be noted that the (RXLEV) measurements produced by a phone running OsmocomBB firmware<br />may be inaccurate, because reading and applying the unit calibration data is not supported yet (see <a class="issue tracker-2 status-3 priority-2 priority-default closed" title="Feature: Merge reading of factory RF calibration values (Resolved)" href="https://projects.osmocom.org/issues/3582">#3582</a>).</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=115972018-09-25T15:31:43Zpespin
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-3 priority-2 priority-default closed" href="/issues/3025">Bug #3025</a>: BTS_Tests.TC_paging_tmsi_200percent fails</i> added</li></ul> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=116392018-09-27T10:36:27Zpespin
<ul><li><strong>File</strong> <a href="/attachments/3376">pcap-recorder_any(filters=_host 10.42.42.8 and port not 22_).pcap</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/3376/pcap-recorder_any(filters=_host%2010.42.42.8%20and%20port%20not%2022_).pcap">pcap-recorder_any(filters=_host 10.42.42.8 and port not 22_).pcap</a> added</li></ul><p>Regarding discussion in <a class="external" href="https://gerrit.osmocom.org/#/c/osmo-ttcn3-hacks/+/11083/2/bts/BTS_Tests.ttcn@58">https://gerrit.osmocom.org/#/c/osmo-ttcn3-hacks/+/11083/2/bts/BTS_Tests.ttcn@58</a> with test "BTS_Tests.TC_meas_res_sign_tchf" having the motorola CXX phone sending different values for ms_power_level (0 and requested 7).</p>
<p>I checked the pcap file of a complete test run and it seems both values are more a les sdistributed over time, that is, we don't get a few ms_power_level=0 at start then ms_power_level=7 starting from some point.</p>
<p>I attach the GSMTAP pcap file of the test to show it. You can find the ms_power_level values by filtering Measurement Report (gsm_a.dtap.msg_rr_type == 0x15) and then looking at "SACCH L1 Header".</p>
Some details of what I see:
<ul>
<li>It seems most Measurement Report messages come with ms_power_level=0 (59msgs vs 8msgs).</li>
<li>Usually Measurement Report messages with ms_power_level=7 come in pairs (2 consecutive).</li>
<li>Measurement Report messages with ms_power_level=7 contain:
<ul>
<li>L1 Header FPC=1: In use</li>
<li>RXLEV: -71 <= x < -70 dBm (40)</li>
<li>NO-NCELL-M: Neighbour cell information not available for serving cell (7)</li>
</ul>
</li>
<li>On the other hand, Measurement Report messages with ms_power_level=0 contain:
<ul>
<li>L1 Header FPC=0: Not in use</li>
<li>RXLEV:-88 <= x < -87 dBm (23)</li>
<li>NO-NCELL-M: No neighbour cell measurement result (0)</li>
</ul></li>
</ul> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=116402018-09-27T11:10:06Zlaforge
<ul></ul><p>On Thu, Sep 27, 2018 at 10:36:27AM +0000, pespin [REDMINE] wrote:</p>
<blockquote>
Some details of what I see:
<ul>
<li>It seems most Measurement Report messages come with ms_power_level=0 (59msgs vs 8msgs).</li>
<li>Usually Measurement Report messages with ms_power_level=7 come in pairs (2 consecutive).</li>
<li>Measurement Report messages with ms_power_level=7 contain:
<ul>
<li>L1 Header FPC=1: In use</li>
</ul></li>
</ul>
</blockquote>
<p>this is clearly wrong. We don't do fast power control (FPC) in OsmoBTS, but only normal power control.</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=116432018-09-27T14:16:53Zpespin
<ul></ul><p>It's strange because in those messages, the FPC field is taken from 1 bit which is also part of the ms_power_level field value, so I'm pretty sure it doesn't apply here or somehow wireshark is wrong. Otherwise it makes no sense to me.</p>
<p>I have been investigating how it comes the motorola phone is sometimes sending ms_power_level 7 and sometimes 0, by looking at osmocombb and L1CTL TTCN parts. In osmocombb, the interesting related variable is called "l1s.tx_power" everywhere. It seems this value is set to default value of 7 when the state is reset (l1s_reset(), l1ctl_rx_reset_req(), L1CTL_RESET_REQ). Other than that reset code path, l1s.tx_power can only be set to another value by using l1ctl_rx_param_req() which can only be called by sending an L1CTL message to it: L1CTL_PARAM_REQ. In TTCN3, that message is only sent by template ts_L1CTL_PAR_REQ, which is only used in function f_L1CTL_PARAM(), which is never used in any TTCN3 code.</p>
<p>So as a result, we have a received tx_power changing but nobody seems to ever be telling the motorola CXX phone to change it.</p>
I then did the following changes:
<ul>
<li>In TTCN3, Set ms_txpwr_max_cch=9 (to be different than the default 7 in osmocombb fw code).</li>
<li>Do following change, to force tx_power to some value (different than the default 7 in osmocombb fw code, and different than the one received by network).<br /><pre>
@@ -1363,7 +1363,10 @@ private function f_est_dchan(boolean encr_enable := false) runs on ConnHdlr {
var GsmFrameNumber fn;
var ImmediateAssignment imm_ass;
var integer ra := 23;
+ var integer ta := 0;
+ var integer tx_power := 8;
+ f_L1CTL_PARAM(L1CTL, ta, tx_power);
fn := f_L1CTL_RACH(L1CTL, ra);
</pre></li>
</ul>
<p>Result: I get same pattern as before, but with 0 and 8. Which means none of the default value nor the network values are being used, and still 0 is sometimes sent which apparently comes from nowhere.</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=116542018-09-27T18:10:19Zpespin
<ul><li><strong>Status</strong> changed from <i>Stalled</i> to <i>In Progress</i></li></ul><p>Submitted patch for FPC issue seen in wireshark: <a class="external" href="https://code.wireshark.org/review/#/c/29893/">https://code.wireshark.org/review/#/c/29893/</a></p>
<p>Regarding the different values in ms_power_level, it seems related to f_L1CTL_PARAM only controlling values for dummy meas report which are sent if no meas report message is received from upper layer on time.<br />The upper layer messages are crafted in TTCN3 function as_l1_sacch(), and the 2 octets or SACCH L1 HEADER are hardcoded to 0: "var octetstring pl := '0000'O & enc_LapdmFrameAB(lb);"</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=116552018-09-27T19:07:42Zfixeria
<ul></ul><blockquote>
<p>Regarding the different values in ms_power_level, it seems related to f_L1CTL_PARAM only<br />controlling values for dummy meas report which are sent if no meas report message<br />is received from upper layer on time.</p>
</blockquote>
<p>Actually, this is the problem I was talking about in:</p>
<p><a class="external" href="https://lists.osmocom.org/pipermail/baseband-devel/2018-March/005513.html">https://lists.osmocom.org/pipermail/baseband-devel/2018-March/005513.html</a></p>
<p>This dummy measurement report shall not "expose" the actual TA and TX power level values<br />(i.e. which are used by the hardware) as it breaks the idea of distance spoofing...</p>
<p>Instead of putting the actual values into a static template, it makes sense to cache<br />the last received measurement report and use it until the next one arrives.</p>
<p>But, in any case this is a problem of OsmocomBB, and weakly related to the issue topic.<br />I just need to find some time and finally fix this in both firmware and trxcon.</p>
<blockquote>
<p>The upper layer messages are crafted in TTCN3 function as_l1_sacch(), and the 2 octets<br />or SACCH L1 HEADER are hardcoded to 0: "var octetstring pl := '0000'O & enc_LapdmFrameAB(lb);"</p>
</blockquote>
<p>I think this is exactly the reason of Measurement Reports with different values you observed.</p>
<p>There are several possible solutions:</p>
<p>1) The simplest solution is to call f_L1CTL_PARAM(ta = 0, tx_power = 0), which isn't called anywhere yet.<br />This would make all Measurement Reports contain the same values. Not sure if this is the best solution.<br />Please note that tx_power is being set to default value by 'layer1/sync.c/l1s_reset()', so f_L1CTL_PARAM()<br />should be called <strong>after</strong> reset is done.</p>
<p>2) Fix the problem I mentioned above, so all Measurement Reports would always contain the indicated<br />parameters (tx_power=0 and ta=0), despite the actual values used by HW/FW would remain default.</p>
<p>I also think it makes sense to make the L1 SACCH header configurable in BTS_Tests.ttcn/as_l1_sacch()...</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=120142018-10-04T15:57:14Zroh
<ul></ul><p>i think i found the issue with the c118 not being reliable/loading in circles:</p>
<p>the cable-ties mounting it to the testbed were too tight and there was no backing behind the phone (while using glued-in-place rf cable).<br />this resuled in the case torqueing a bit, which was enough for the battery simulator board making bad contact via the battery pins.<br />such the phone tried to load, but failed in the process due to brown-out.</p>
<p>i changed the board (cutting the leads protruding short as possible) and made the contact pads thicker by adding some metallic pieces. i also changed the mechanics to hold the phone in a more straight position and added some padding material below before mounting it with zip-ties again.</p>
<p>it re-enumerated as ttyUSB1 /dev/serial/by-path/pci-0000:00:12.2-usb-0:5.4.3:1.0-port0</p>
<p>please test</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=121212018-10-09T16:34:44Zpespin
<ul></ul><p>After HW fix, motorola c118 phone is working fine again in the prod setup.</p>
<p>I took over looking at failing tests in <a class="external" href="https://jenkins.osmocom.org/jenkins/view/osmo-gsm-tester/job/osmo-gsm-tester_ttcn3/test_results_analyzer/">https://jenkins.osmocom.org/jenkins/view/osmo-gsm-tester/job/osmo-gsm-tester_ttcn3/test_results_analyzer/</a></p>
<p>I had a look at <strong>TC_encr_cmd_a51</strong>, which seems to be failing most of the time but it recently passed twice in a row (#650 and #649). I see a similar pattern for <strong>TC_encr_cmd_a52</strong>.</p>
<p>In the failing case (run #651 and #646):<br /><pre>
09:43:01.047017 450 BTS_Tests.ttcn:3793 Timeout T: 3 s
09:43:01.047367 450 BTS_Tests.ttcn:3794 setverdict(fail): pass -> fail reason: "Timeout waiting for DATA_IND", new component reason: "Timeout waiting for DATA_IND"
</pre></p>
<p>Looking at log files and pcap file, the failure is triggered because according to BTS GSMTAP the MS apparently is not sending the first ciphered I-frame after receiving RSL ENC CMD (with Ciphering Mode Command inside):<br /><pre>
/* Test unencrypted channel activation followed by explicit ENCR CMD later */
function f_TC_encr_cmd(charstring id) runs on ConnHdlr {
...
/* then send the RSL ENCR CMD with an actual RR CIPH MOD CMD inside */
RSL.send(ts_RSL_ENCR_CMD(g_chan_nr, link_id, g_pars.encr.alg_id, g_pars.encr.key, l3));
/* expect the L3 to arrive still unencrypted on the MS side */
f_l1_exp_lapdm(tr_LAPDm_I(link_id.sapi, cr_MT_CMD, ?, ?, ?, l3)); <--------- THIS IS RECEIVED FINE
/* configure L1 to apply ciphering */
var uint8_t alg_id := f_alg_id_to_l1ctl(g_pars.encr.alg_id);
f_L1CTL_CRYPTO_REQ(L1CTL, g_pars.chan_nr, alg_id, g_pars.encr.key);
/* send first ciphered I-frame in response and expect it on RSL */
f_data_mo(link_id, true, 1, 0, '0a0b0c0d'O, exp_sacch := true); <--------- THIS NEVER APPEARS IN BTS GSTMAP
</pre></p>
<p>Test fails actually after second round (it's run multiple times for different channels), while using <code>{ u := { lm := { tag := '0001'B, sub_chan := 1 } }, tn := 5 }</code> and as shown in wireshark: "Lm + ACCH (sub-chan 1) (3)". The first round "Bm + ACCH (1)" is fine.</p>
<p>If we look at other failing runs, like #648 and #647, both fail in a similar way, while sending an uplink message during 2nd round (Lm+ACCH) which never shows up in GSMTAP, but does not fail exactly at the same place. In this case, both fail during establishing ABM after dedicated channel has been established:<br />"f_TC_encr_cmd --> f_est_rll_mo(link_id.sapi, link_id, '23420815'O); --> f_tx_lapdm(ts_LAPDm_SABM(sapi, cr_MO_CMD, true, l3), link_id);"</p>
<p>So it seems osmocom-bb "randomly" fails to send data correctly over TCH/H+ACCH, or osmo-bts-trx/osmo-trx fail to receive it. That's why sometimes the test fails during one step, sometimes during next step, and sometimes passes.</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=121272018-10-10T10:43:39Zpespin
<ul></ul><p>Attaching archive with osmo-gsm-tester+ttcn3 run information during TC_chan_act_a51() run, which has exactly same issue as TC_encr_cmd_*() tests explained above. Inside the archive there's both a successful run and a failing run, to be able to check differences.</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=121282018-10-10T10:43:58Zpespin
<ul><li><strong>File</strong> <a href="/attachments/3396">chan_act_a51-failpass.tar.gz</a> <a class="icon-only icon-download" title="Download" href="/attachments/download/3396/chan_act_a51-failpass.tar.gz">chan_act_a51-failpass.tar.gz</a> added</li></ul> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=121472018-10-10T15:18:36Zfixeria
<ul></ul><p>Hi Pau,</p>
<blockquote>
<p>So it seems osmocom-bb "randomly" fails to send data correctly over TCH/H+ACCH,<br />or osmo-bts-trx/osmo-trx fail to receive it. That's why sometimes the test fails<br />during one step, sometimes during next step, and sometimes passes.</p>
</blockquote>
<p>I guess this is a problem of the OsmocomBB firmware. FACCH/H can be initiated at specific<br />frame numbers only, so ignoring this rule may result in misaligned FACCH/H transmission.</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=121622018-10-11T15:00:46Zpespin
<ul></ul><p>Checking at FACCH/H transmission scheduling the used FNs seems fine. According to the code, the DSP expects to receive the message on the preceding burst, and that's what it does correctly.<br />I also tried switching to expected and not preceding bursts just in case, but got same results during TTCN3 tests, that is, sometimes fails and sometimes pass.<br /><pre>
--- a/src/target/firmware/layer1/prim_tch.c
+++ b/src/target/firmware/layer1/prim_tch.c
@@ -402,17 +402,15 @@ static int l1s_tch_cmd(__unused uint8_t p1, __unused uint8_t p2, uint16_t p3)
icnt++;
/* Load FACCH data if we start a new burst */
- /* (the DSP wants the data on the CMD of the burst _preceding_ the
- * first burst) */
if (tch_f_hn) {
/* FACCH/F: B0(0...7),B1(4...11),B2(8...11,0...3) */
facch_tx_now = ((l1s.next_time.fn % 13) % 4) == 3;
} else {
/* FAACH/H: See GSM 05.02 Clause 7 Table 1of9 */
uint8_t t2_norm = l1s.next_time.t2 - tch_sub;
- facch_tx_now = (t2_norm == 23) ||
- (t2_norm == 6) ||
- (t2_norm == 15);
+ facch_tx_now = (t2_norm == 0) ||
+ (t2_norm == 8) ||
+ (t2_norm == 17);
}
</pre></p>
<p>As requested by <a class="user active" href="https://projects.osmocom.org/users/67">fixeria</a>, I also tested adding a sleep after sending L1CTL_CRYPTO_REQ to make sure the DSP is not applying encryptin in the middle of the burst block, but I got same results:<br /><pre>
--- a/bts/BTS_Tests.ttcn
+++ b/bts/BTS_Tests.ttcn
@@ -1399,6 +1399,7 @@ private function f_est_dchan(boolean encr_enable := false) runs on ConnHdlr {
if (encr_enable) {
var uint8_t alg_id := f_alg_id_to_l1ctl(g_pars.encr.alg_id);
f_L1CTL_CRYPTO_REQ(L1CTL, g_pars.chan_nr, alg_id, g_pars.encr.key);
+ f_sleep(2.0); /* make sure encryption is applied */
}
g_first_meas_res := true;
@@ -3863,7 +3864,7 @@ function f_TC_encr_cmd(charstring id) runs on ConnHdlr {
/* configure L1 to apply ciphering */
var uint8_t alg_id := f_alg_id_to_l1ctl(g_pars.encr.alg_id);
f_L1CTL_CRYPTO_REQ(L1CTL, g_pars.chan_nr, alg_id, g_pars.encr.key);
-
+ f_sleep(2.0); /* make sure encryption is applied */
/* send first ciphered I-frame in response and expect it on RSL */
f_data_mo(link_id, true, 1, 0, '0a0b0c0d'O, exp_sacch := true);
</pre></p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=122932018-10-20T19:29:52Zlaforge
<ul></ul><p>Adding <a class="user active" href="https://projects.osmocom.org/users/12">tnt</a> here to the list of followers. The plan is that he can help with investigating any test failures of tests that pass in the virtual environment but fail in the real hardware environment. He has extensive Layer0/Layer1 knowledge and should be able to help where <a class="user active" href="https://projects.osmocom.org/users/30187">pespin</a> is a bit out of his usual comfort zone.</p>
<p>The problems could likely be in the test case expectations, in OsmocomBB or in OsmoBTS.</p>
<p>It may make sense to start more tickets about individual issues/failures rather than adding more to this generic ticket.</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=139962019-04-15T17:15:15Zpespin
<ul><li><strong>% Done</strong> changed from <i>40</i> to <i>50</i></li></ul><p>I added a job in sysmocom's jenkins to push latest ttcn3-bts-test docker images to registry.sysmocom.de, which are then pulled every time a jenkins job in <a class="external" href="https://jenkins.osmocom.org/jenkins/view/osmo-gsm-tester/job/osmo-gsm-tester_ttcn3/">https://jenkins.osmocom.org/jenkins/view/osmo-gsm-tester/job/osmo-gsm-tester_ttcn3/</a> is run (see osmo-gsm-tester.git/ttcn3/jenkins-run.sh).</p>
<p>Until now I was updating the docker image manually from time to time. Now we should always be running master.</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=187472020-06-17T08:51:43Zfixeria
<ul></ul><p>According to <a class="external" href="https://jenkins.osmocom.org/jenkins/view/osmo-gsm-tester/job/osmo-gsm-tester_ttcn3/">https://jenkins.osmocom.org/jenkins/view/osmo-gsm-tester/job/osmo-gsm-tester_ttcn3/</a>, we're not doing well.<br />Most of the test cases in groups 'trial-ttcn3_bts_tests:{oc2g,sysmo}' fail due to:</p>
<pre>
"BTS_Tests.ttcn:188 : Timeout waiting for RSL bring up"
BTS_Tests.ttcn:6496 BTS_Tests control part
BTS_Tests.ttcn:721 TC_chan_deact_not_active testcase
</pre>
<p>Looks like we're using old configuration files for osmo-{bsc,bts}, where we have only one transceiver configured, while the test case expects 3 additional transceivers to show up on the A-bis interface. I could not figure out where the configuration files are. <a class="user active" href="https://projects.osmocom.org/users/30187">pespin</a> could you give me a hint?</p>
<p>Test cases in a group 'trial-ttcn3_bts_tests:trx' are not executed at all (all N/A). Also, it's really <strong>hard to read</strong> the output of test case analyzer due to lots of N/A and gaps. Can we do something to fix this?</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=187502020-06-17T10:22:19Zpespin
<ul></ul><p>fixeria wrote:</p>
<blockquote>
<p>According to <a class="external" href="https://jenkins.osmocom.org/jenkins/view/osmo-gsm-tester/job/osmo-gsm-tester_ttcn3/">https://jenkins.osmocom.org/jenkins/view/osmo-gsm-tester/job/osmo-gsm-tester_ttcn3/</a>, we're not doing well.<br />Most of the test cases in groups 'trial-ttcn3_bts_tests:{oc2g,sysmo}' fail due to:</p>
<p>[...]</p>
<p>Looks like we're using old configuration files for osmo-{bsc,bts}, where we have only one transceiver configured, while the test case expects 3 additional transceivers to show up on the A-bis interface. I could not figure out where the configuration files are. <a class="user active" href="https://projects.osmocom.org/users/30187">pespin</a> could you give me a hint?</p>
</blockquote>
<p>Well, it doesn't make sense to have more than 1 transceiver with sysmobts (and also with oc2g?). So when running with those we need to adapt the tests to run with only 1 transceiver. But I think since last RSL changes the BTS_tests always expects 3 transceivers? we should have that configurable through module parameter in ttcn3.</p>
<p>All ttcn3 related config when running in the real HW setup can be found in osmo-gsm-tester.git/sysmocom/ttcn3.</p>
<blockquote>
<p>Test cases in a group 'trial-ttcn3_bts_tests:trx' are not executed at all (all N/A). Also, it's really <strong>hard to read</strong> the output of test case analyzer due to lots of N/A and gaps. Can we do something to fix this?</p>
</blockquote>
<p>It's hard to read due to some regressions storing the junit stuff in a different manner. That was fixed and I cleaned the history so now only a few last job runs appear.<br />Test runs using osmo-bts-trx are failing due to currently osmo-bts-trx exiting due to clock issues for some yet unknown reason.</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=187522020-06-17T10:38:39Zfixeria
<ul><li><strong>Assignee</strong> changed from <i>pespin</i> to <i>fixeria</i></li></ul><blockquote>
<p>Well, it doesn't make sense to have more than 1 transceiver with sysmobts (and also with oc2g?). So when running with those we need to adapt the tests to run with only 1 transceiver. But I think since last RSL changes the BTS_tests always expects 3 transceivers? we should have that configurable through module parameter in ttcn3.</p>
</blockquote>
<p>That was my initial intention to have the number of expected transceivers configurable, but unfortunately the 'interleave' statements in TTCN-3 are not flexible enough. So yes, now we always expect all 3 transceivers to connect. I'll try to make it configurable, so assigning to myself.</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=188442020-06-21T15:08:50Zlaforge
<ul><li><strong>Priority</strong> changed from <i>Urgent</i> to <i>Immediate</i></li></ul><p><a class="user active" href="https://projects.osmocom.org/users/67">fixeria</a> breaking all of our end-to-end TTCN3 tests with osmo-gsm-tester and real BTS hardware is problematic. Please resolve this ASAP.</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=188542020-06-21T15:26:51Zfixeria
<ul></ul><p>I am working on it. Let's see if I can resolve it today.</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=188732020-06-22T11:02:15Zfixeria
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Feedback</i></li><li><strong>% Done</strong> changed from <i>50</i> to <i>80</i></li></ul><p>Please see:</p>
<p><a class="external" href="https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/18951">https://gerrit.osmocom.org/c/osmo-ttcn3-hacks/+/18951</a> BTS: refactor f_init_rsl(): make number of transceivers configurable</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=188742020-06-22T11:21:22Zfixeria
<ul></ul><p>Also, this one:</p>
<p><a class="external" href="https://gerrit.osmocom.org/c/docker-playground/+/18952">https://gerrit.osmocom.org/c/docker-playground/+/18952</a> ttcn3-bts-test/BTS_Tests.cfg: configure number of transceivers</p>
<p>so by default we will be expecting only one transceiver, unless explicitly configured.</p>
<p>I'll re-trigger "osmo-gsm-tester_ttcn3" in Jenkins as soon as both changes are merged.</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=188912020-06-23T08:38:13Zfixeria
<ul><li><strong>Assignee</strong> changed from <i>fixeria</i> to <i>pespin</i></li><li><strong>Priority</strong> changed from <i>Immediate</i> to <i>High</i></li><li><strong>% Done</strong> changed from <i>80</i> to <i>50</i></li></ul><p>Starting from build#2604, some test cases pass. At least the test suite can work with one transceiver again.</p>
<p>Unfortunately, most of them still fail due to other reasons:</p>
<pre>
FBSB Failed with non-zero return code 255
BTS_Tests.ttcn:6621 BTS_Tests control part
BTS_Tests.ttcn:6077 TC_chan_act_a51 testcase
</pre>
<pre>
Timeout waiting for L1CTL_RESET_CONF
BTS_Tests.ttcn:6572 BTS_Tests control part
BTS_Tests.ttcn:4423 TC_pcu_ptcch testcase
</pre>
<p>so assigning back to <a class="user active" href="https://projects.osmocom.org/users/30187">pespin</a>.</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=192482020-07-31T15:32:21Zfixeria
<ul></ul><p>I just fixed a serious regression in the firmware:</p>
<p><a class="external" href="https://gerrit.osmocom.org/c/osmocom-bb/+/19479">https://gerrit.osmocom.org/c/osmocom-bb/+/19479</a> firmware/layer1: refactor multi-frame task mask composition<br /><a class="external" href="https://gerrit.osmocom.org/c/osmocom-bb/+/19480">https://gerrit.osmocom.org/c/osmocom-bb/+/19480</a> firmware/layer1: fix properly apply secondary multi-frame task</p> OsmoBTS - Feature #3155: execute BTS_Tests.ttcn with real (C123) phone hardware in LTHW setuphttps://projects.osmocom.org/issues/3155?journal_id=205252020-12-04T19:30:25Zpespin
<ul><li><strong>Status</strong> changed from <i>Feedback</i> to <i>Stalled</i></li></ul>