Project

General

Profile

Actions

Bug #5301

open

Run TTCN3 docker tests with sanitizer enabled

Added by daniel 9 months ago. Updated 10 days ago.

Status:
In Progress
Priority:
Normal
Assignee:
Target version:
-
Start date:
11/10/2021
Due date:
% Done:

70%

Spec Reference:

Description

After updating libosmocore I noticed that the TTCN3 GbProxy tests start to fail with an ASan issue when run locally.

I think at least for the TTCN3 tests on master we should enable *San to catch hidden bugs early. Unfortunately this has a large impact on how the osmo-*-master docker images are built if we want to enable it for the libraries as well - currently we install the nightly packages and build the target from master.

Instead we could build an image that builds all the libraries from master (with sanitizer enabled) and installs those and then use that as base for osmo-*-master.

Not sure what the downsides are, any ideas?


Checklist

  • initial build of osmocom:nightly:asan
  • test running one testsuite with osmocom:nightly:asan packages locally
  • submit all code changes to gerrit
  • get patches merged
  • roll it out (may need some fixups)

Related issues

Related to libosmocore - Bug #5302: ns2: ASan heap-use-after-free in ns2_nse_notify_unblocked() when running GBProxy_Tests.TC_bvc_reset_blocked_ptp_from_sgsnResolveddaniel11/10/2021

Actions
Actions #1

Updated by daniel 9 months ago

  • Related to Bug #5302: ns2: ASan heap-use-after-free in ns2_nse_notify_unblocked() when running GBProxy_Tests.TC_bvc_reset_blocked_ptp_from_sgsn added
Actions #2

Updated by osmith 9 months ago

Instead we could build an image that builds all the libraries from master (with sanitizer enabled) and installs those and then use that as base for osmo-*-master.

If we do that instead of the current debian and centos tests, then we end up not testing the distro packages anymore. But this was the point, and the reason why we test both debian and centos as I understand.

How about we do what you described, but in addition to the tests that make use of distro packages?

We could have a new TTCN3-san view, next to:

If we only test master and not latest too, then this new view has only half as much jobs.

laforge, what do you think?

Actions #3

Updated by laforge 9 months ago

On Fri, Nov 12, 2021 at 03:36:30PM +0000, wrote:

Instead we could build an image that builds all the libraries from master (with sanitizer enabled) and installs those and then use that as base for osmo-*-master.

If we do that instead of the current debian and centos tests, then we end up not testing the distro packages anymore. But this was the point, and the reason why we test both debian and centos as I understand.

ACK.

How about we do what you described, but in addition to the tests that make use of distro packages?

We could have a new TTCN3-san view, next to:

If we only test master and not latest too, then this new view has only half as much jobs.

OK. I would argue to test master once per day only, and not every commit. I can add more
build slaves if we run into problems there.

Actions #4

Updated by daniel 9 months ago

laforge wrote in #note-3:

On Fri, Nov 12, 2021 at 03:36:30PM +0000, wrote:

How about we do what you described, but in addition to the tests that make use of distro packages?

We could have a new TTCN3-san view, next to:

If we only test master and not latest too, then this new view has only half as much jobs.

OK. I would argue to test master once per day only, and not every commit. I can add more
build slaves if we run into problems there.

Yeah, that would be great. I think TTCN3 tests every commit is a bit too much anyway (with or without sanitizer). Running it once per day is fine.

Actions #5

Updated by laforge about 2 months ago

As this topic has come up again today: One alternative might also be to create a dpkg repo (for one distro/arch only to avoid tons of extra builds) with asan enabled. something like osmocom:nightly:asan or the like on https://obs.osmocom.org/

Actions #6

Updated by osmith about 1 month ago

  • Status changed from New to In Progress
Actions #7

Updated by osmith about 1 month ago

  • Description updated (diff)
Actions #8

Updated by osmith 12 days ago

  • Checklist item initial build of osmocom:nightly:asan added
  • Checklist item test running one testsuite with osmocom:nightly:asan packages locally added
  • Checklist item submit all code changes to gerrit added
  • Checklist item get patches merged added
  • Checklist item roll it out (may need some fixups) added
  • % Done changed from 0 to 70

I've created https://obs.osmocom.org/project/show/osmocom:nightly:asan and adjusted the osmo-ci scripts to generate debian source packages with --enable-sanitize.

When trying to build them on OBS, it failed to run the regression tests:

[  237s] ==33619==ERROR: AddressSanitizer failed to allocate 0xdfff0001000 (15392894357504) bytes at address 2008fff7000 (errno: 12)
[  237s] ==33619==ReserveShadowMemoryRange failed while trying to map 0xdfff0001000 bytes. Perhaps you're using ulimit -v

Using ulimit -v is hardcoded in OBS here: https://github.com/openSUSE/obs-build/blob/e82ad7023a03a81e0d48f44f0260ced7b6c0643e/build#L743-L744
(It runs through that code path, as it prints "Memory limit set to 66248548KB" at the beginning of the log.)

So as a workaround, I disabled the tests. I also had to disable building of documentation, as otherwise it would run into the same error while generating VTY docs. Now all Osmocom packages build successfully with sanitize flags in that new osmocom:nightly:asan repository.

I went on to adjust the scripts to actually run the testsuite with these binary packages, and in a quick local test it seems to be working fine.

Patches submitted:
https://gerrit.osmocom.org/q/topic:ttcn3-asan

Actions #9

Updated by pespin 12 days ago

I'd probably change the osmo-*-master Dockerfiles to build all dependency libs from master with --enable-asan enabled, I have been doing that for ages in my local copy of docker-playground.git
This has the benefit that during development one can for instance, easily set LIBOSMO_NETIF_BRANCH=myownbranch when testing ie. osmo-stp-master.

Actions #10

Updated by pespin 12 days ago

I think it makes more sense to have a -nightly using nightly repos with prebuilt binaries, and have a -master with libs+binary compiled with --enabe-asan.

Actions #11

Updated by osmith 11 days ago

Discussions in code review:

Actions #12

Updated by daniel 11 days ago

What I care about is running the full ttcn3 test suite with recent osmo-* packages/binaries in a regular interval (say at least once a week, more would be better). I could see benefits in having (additional) asan-enabled packages that one could also install locally.
The packages would only be rebuilt if something in git or a dependency changes, right?

Actions #13

Updated by osmith 10 days ago

daniel wrote in #note-12:

What I care about is running the full ttcn3 test suite with recent osmo-* packages/binaries in a regular interval (say at least once a week, more would be better). I could see benefits in having (additional) asan-enabled packages that one could also install locally.
The packages would only be rebuilt if something in git or a dependency changes, right?

They would be rebuilt every night, just like the other nightly packages.

We decided earlier to have all nightly Osmocom packages depend on a specific version (build date) of a meta package (e.g. osmocom-nightly). This ensures packages of different build days can't be mixed by accident, as we don't ensure ABI compatibility on master - only for latest. For that reason we need to also rebuild all nightly asan packages at once, even if they did not change.

FWIW the asan packages build pretty fast since they can't run tests or be built with documentation in OBS.

Actions

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)