https://projects.osmocom.org/https://projects.osmocom.org/favicon.ico?16647414092021-11-12T10:23:05ZOpen Source Mobile CommunicationsCore testing infrastructure - Bug #5301: Run TTCN3 docker tests with sanitizer enabledhttps://projects.osmocom.org/issues/5301?journal_id=229872021-11-12T10:23:05Zdaniel
<ul><li><strong>Related to</strong> <i><a class="issue tracker-1 status-3 priority-2 priority-default closed" href="/issues/5302">Bug #5302</a>: ns2: ASan heap-use-after-free in ns2_nse_notify_unblocked() when running GBProxy_Tests.TC_bvc_reset_blocked_ptp_from_sgsn</i> added</li></ul> Core testing infrastructure - Bug #5301: Run TTCN3 docker tests with sanitizer enabledhttps://projects.osmocom.org/issues/5301?journal_id=230002021-11-12T15:36:29Zosmith
<ul></ul><blockquote>
<p>Instead we could build an image that builds all the libraries from master (with sanitizer enabled) and installs those and then use that as base for osmo-*-master.</p>
</blockquote>
<p>If we do that instead of the current debian and centos tests, then we end up not testing the distro packages anymore. But this was the point, and the reason why we test both debian and centos as I understand.</p>
<p>How about we do what you described, but in addition to the tests that make use of distro packages?</p>
We could have a new TTCN3-san view, next to:
<ul>
<li><a class="external" href="https://jenkins.osmocom.org/jenkins/view/TTCN3/">https://jenkins.osmocom.org/jenkins/view/TTCN3/</a></li>
<li><a class="external" href="https://jenkins.osmocom.org/jenkins/view/TTCN3-centos/">https://jenkins.osmocom.org/jenkins/view/TTCN3-centos/</a></li>
</ul>
<p>If we only test master and not latest too, then this new view has only half as much jobs.</p>
<p><a class="user active" href="https://projects.osmocom.org/users/7">laforge</a>, what do you think?</p> Core testing infrastructure - Bug #5301: Run TTCN3 docker tests with sanitizer enabledhttps://projects.osmocom.org/issues/5301?journal_id=230082021-11-13T14:20:14Zlaforge
<ul></ul><p>On Fri, Nov 12, 2021 at 03:36:30PM +0000, <a class="email" href="mailto:redmine@lists.osmocom.org">redmine@lists.osmocom.org</a> wrote:</p>
<blockquote><blockquote>
<p>Instead we could build an image that builds all the libraries from master (with sanitizer enabled) and installs those and then use that as base for osmo-*-master.</p>
</blockquote>
<p>If we do that instead of the current debian and centos tests, then we end up not testing the distro packages anymore. But this was the point, and the reason why we test both debian and centos as I understand.</p>
</blockquote>
<p>ACK.</p>
<blockquote>
<p>How about we do what you described, but in addition to the tests that make use of distro packages?</p>
We could have a new TTCN3-san view, next to:
<ul>
<li><a class="external" href="https://jenkins.osmocom.org/jenkins/view/TTCN3/">https://jenkins.osmocom.org/jenkins/view/TTCN3/</a></li>
<li><a class="external" href="https://jenkins.osmocom.org/jenkins/view/TTCN3-centos/">https://jenkins.osmocom.org/jenkins/view/TTCN3-centos/</a></li>
</ul>
<p>If we only test master and not latest too, then this new view has only half as much jobs.</p>
</blockquote>
<p>OK. I would argue to test master once per day only, and not every commit. I can add more<br />build slaves if we run into problems there.</p> Core testing infrastructure - Bug #5301: Run TTCN3 docker tests with sanitizer enabledhttps://projects.osmocom.org/issues/5301?journal_id=230202021-11-15T16:54:18Zdaniel
<ul></ul><p>laforge wrote in <a href="#note-3">#note-3</a>:</p>
<blockquote>
<p>On Fri, Nov 12, 2021 at 03:36:30PM +0000, <a class="email" href="mailto:redmine@lists.osmocom.org">redmine@lists.osmocom.org</a> wrote:</p>
<blockquote>
<p>How about we do what you described, but in addition to the tests that make use of distro packages?</p>
We could have a new TTCN3-san view, next to:
<ul>
<li><a class="external" href="https://jenkins.osmocom.org/jenkins/view/TTCN3/">https://jenkins.osmocom.org/jenkins/view/TTCN3/</a></li>
<li><a class="external" href="https://jenkins.osmocom.org/jenkins/view/TTCN3-centos/">https://jenkins.osmocom.org/jenkins/view/TTCN3-centos/</a></li>
</ul>
<p>If we only test master and not latest too, then this new view has only half as much jobs.</p>
</blockquote>
<p>OK. I would argue to test master once per day only, and not every commit. I can add more<br />build slaves if we run into problems there.</p>
</blockquote>
<p>Yeah, that would be great. I think TTCN3 tests every commit is a bit too much anyway (with or without sanitizer). Running it once per day is fine.</p> Core testing infrastructure - Bug #5301: Run TTCN3 docker tests with sanitizer enabledhttps://projects.osmocom.org/issues/5301?journal_id=242722022-06-27T16:58:49Zlaforge
<ul></ul><p>As this topic has come up again today: One alternative might also be to create a dpkg repo (for one distro/arch only to avoid tons of extra builds) with asan enabled. something like <code>osmocom:nightly:asan</code> or the like on <a class="external" href="https://obs.osmocom.org/">https://obs.osmocom.org/</a></p> Core testing infrastructure - Bug #5301: Run TTCN3 docker tests with sanitizer enabledhttps://projects.osmocom.org/issues/5301?journal_id=243712022-07-13T09:28:47Zosmith
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>In Progress</i></li></ul> Core testing infrastructure - Bug #5301: Run TTCN3 docker tests with sanitizer enabledhttps://projects.osmocom.org/issues/5301?journal_id=243722022-07-13T09:29:20Zosmith
<ul><li><strong>Description</strong> updated (<a title="View differences" href="/journals/24372/diff?detail_id=39890">diff</a>)</li></ul> Core testing infrastructure - Bug #5301: Run TTCN3 docker tests with sanitizer enabledhttps://projects.osmocom.org/issues/5301?journal_id=245582022-08-03T13:42:07Zosmith
<ul><li><b>Checklist item</b> <input type='checkbox' class='checklist-checkbox' checked disabled> <i>initial build of osmocom:nightly:asan</i> added</li><li><b>Checklist item</b> <input type='checkbox' class='checklist-checkbox' checked disabled> <i>test running one testsuite with osmocom:nightly:asan packages locally</i> added</li><li><b>Checklist item</b> <input type='checkbox' class='checklist-checkbox' checked disabled> <i>submit all code changes to gerrit</i> added</li><li><b>Checklist item</b> <input type='checkbox' class='checklist-checkbox' disabled> <i>get patches merged</i> added</li><li><b>Checklist item</b> <input type='checkbox' class='checklist-checkbox' disabled> <i>roll it out (may need some fixups)</i> added</li><li><strong>% Done</strong> changed from <i>0</i> to <i>70</i></li></ul><p>I've created <a class="external" href="https://obs.osmocom.org/project/show/osmocom:nightly:asan">https://obs.osmocom.org/project/show/osmocom:nightly:asan</a> and adjusted the osmo-ci scripts to generate debian source packages with --enable-sanitize.</p>
<p>When trying to build them on OBS, it failed to run the regression tests:<br /><pre>
[ 237s] ==33619==ERROR: AddressSanitizer failed to allocate 0xdfff0001000 (15392894357504) bytes at address 2008fff7000 (errno: 12)
[ 237s] ==33619==ReserveShadowMemoryRange failed while trying to map 0xdfff0001000 bytes. Perhaps you're using ulimit -v
</pre></p>
<p>Using ulimit -v is hardcoded in OBS here: <a class="external" href="https://github.com/openSUSE/obs-build/blob/e82ad7023a03a81e0d48f44f0260ced7b6c0643e/build#L743-L744">https://github.com/openSUSE/obs-build/blob/e82ad7023a03a81e0d48f44f0260ced7b6c0643e/build#L743-L744</a><br />(It runs through that code path, as it prints "Memory limit set to 66248548KB" at the beginning of the log.)</p>
<p>So as a workaround, I disabled the tests. I also had to disable building of documentation, as otherwise it would run into the same error while generating VTY docs. Now all Osmocom packages build successfully with sanitize flags in that new osmocom:nightly:asan repository.</p>
<p>I went on to adjust the scripts to actually run the testsuite with these binary packages, and in a quick local test it seems to be working fine.</p>
<p>Patches submitted:<br /><a class="external" href="https://gerrit.osmocom.org/q/topic:ttcn3-asan">https://gerrit.osmocom.org/q/topic:ttcn3-asan</a></p> Core testing infrastructure - Bug #5301: Run TTCN3 docker tests with sanitizer enabledhttps://projects.osmocom.org/issues/5301?journal_id=245592022-08-03T14:01:45Zpespin
<ul></ul><p>I'd probably change the osmo-*-master Dockerfiles to build all dependency libs from master with --enable-asan enabled, I have been doing that for ages in my local copy of docker-playground.git<br />This has the benefit that during development one can for instance, easily set LIBOSMO_NETIF_BRANCH=myownbranch when testing ie. osmo-stp-master.</p> Core testing infrastructure - Bug #5301: Run TTCN3 docker tests with sanitizer enabledhttps://projects.osmocom.org/issues/5301?journal_id=245602022-08-03T14:06:59Zpespin
<ul></ul><p>I think it makes more sense to have a -nightly using nightly repos with prebuilt binaries, and have a -master with libs+binary compiled with --enabe-asan.</p> Core testing infrastructure - Bug #5301: Run TTCN3 docker tests with sanitizer enabledhttps://projects.osmocom.org/issues/5301?journal_id=245762022-08-04T14:02:10Zosmith
<ul></ul><p>Discussions in code review:</p>
<ul>
<li>is the package repository the right approach? <a class="external" href="https://gerrit.osmocom.org/c/docker-playground/+/28902/1#message-30650d8450d2d129b6ef3f447f65e4b866d2f801">https://gerrit.osmocom.org/c/docker-playground/+/28902/1#message-30650d8450d2d129b6ef3f447f65e4b866d2f801</a></li>
</ul>
<ul>
<li>when to run asan builds, really in addition to master builds? <a class="external" href="https://gerrit.osmocom.org/c/osmo-ci/+/28909/2#message-c3ce96b7775c1899a177b77437976d9a36bdd0ca">https://gerrit.osmocom.org/c/osmo-ci/+/28909/2#message-c3ce96b7775c1899a177b77437976d9a36bdd0ca</a></li>
</ul> Core testing infrastructure - Bug #5301: Run TTCN3 docker tests with sanitizer enabledhttps://projects.osmocom.org/issues/5301?journal_id=245792022-08-04T16:46:20Zdaniel
<ul></ul><p>What I care about is running the full ttcn3 test suite with recent osmo-* packages/binaries in a regular interval (say at least once a week, more would be better). I could see benefits in having (additional) asan-enabled packages that one could also install locally.<br />The packages would only be rebuilt if something in git or a dependency changes, right?</p> Core testing infrastructure - Bug #5301: Run TTCN3 docker tests with sanitizer enabledhttps://projects.osmocom.org/issues/5301?journal_id=245812022-08-05T08:27:43Zosmith
<ul></ul><p>daniel wrote in <a href="#note-12">#note-12</a>:</p>
<blockquote>
<p>What I care about is running the full ttcn3 test suite with recent osmo-* packages/binaries in a regular interval (say at least once a week, more would be better). I could see benefits in having (additional) asan-enabled packages that one could also install locally.<br />The packages would only be rebuilt if something in git or a dependency changes, right?</p>
</blockquote>
<p>They would be rebuilt every night, just like the other nightly packages.</p>
<p>We decided earlier to have all nightly Osmocom packages depend on a specific version (build date) of a meta package (e.g. osmocom-nightly). This ensures packages of different build days can't be mixed by accident, as we don't ensure ABI compatibility on master - only for latest. For that reason we need to also rebuild all nightly asan packages at once, even if they did not change.</p>
<p>FWIW the asan packages build pretty fast since they can't run tests or be built with documentation in OBS.</p> Core testing infrastructure - Bug #5301: Run TTCN3 docker tests with sanitizer enabledhttps://projects.osmocom.org/issues/5301?journal_id=246472022-08-16T09:29:42Zosmith
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Stalled</i></li></ul><p>Setting to stalled until discussed how to move forward. (Maybe in one of the next weekly meetings once everybody is back from holidays?)</p> Core testing infrastructure - Bug #5301: Run TTCN3 docker tests with sanitizer enabledhttps://projects.osmocom.org/issues/5301?journal_id=270742023-06-13T09:12:33Zlaforge
<ul></ul><p>I just updated the gerrit tickets linked above. I do think running daily/nightly asan builds is useful. Ideally we'd be able to figure out when to run them to avoid clashes with other busy times for our builders. Given that we now have grafana/prometheus, we should hopefully be able to figure out when would be a good time, or in general when to run which builds / tests (not just those expected new ones).</p> Core testing infrastructure - Bug #5301: Run TTCN3 docker tests with sanitizer enabledhttps://projects.osmocom.org/issues/5301?journal_id=276362023-08-24T10:49:53Zosmith
<ul><li><strong>Status</strong> changed from <i>Stalled</i> to <i>In Progress</i></li></ul><p>I forgot to update this issue.</p>
Last time I looked into this, building all packages in OBS with asan did not work yet - specifically libosmo-pfcp fails because:
<ul>
<li>it runs a generated binary during the build (gen__pfcp_ies_auto)</li>
<li>running asan binaries fails in our OBS (==16918==ERROR: AddressSanitizer failed to allocate 0xdfff0001000 (15392894357504) bytes at address 2008fff7000 (errno: 12)) (#6146)</li>
</ul>
<p>Having another look at it.</p> Core testing infrastructure - Bug #5301: Run TTCN3 docker tests with sanitizer enabledhttps://projects.osmocom.org/issues/5301?journal_id=296512024-03-20T09:28:03Zosmith
<ul><li><strong>Status</strong> changed from <i>In Progress</i> to <i>Stalled</i></li></ul><p>To summarize the current status: I couldn't get the asan packages building on OBS last time I tried (asan binaries could not execute, and we need to run some binaries during build for some projects even if we disable tests). Meanwhile we have replaced the server that the OBS workers are running on (#6186) and I have a better understanding of how the OBS workers are set up. So it would be worth trying again when I'm done with issues with higher priorities.</p>