Feature #4212

TTCN3 docker containers: tell glibc to print segfault output to stderr

Added by pespin about 1 year ago. Updated 11 months ago.

Target version:
Start date:
Due date:
% Done:


Spec Reference:


According to links below, glibc has a feature to log crash related output to stderr instead of /dev/tty (which in the case of running udner docker gets lost afaiu). The feature can be enabled by env var: LIBC_FATAL_STDERR_=1

The idea is to enable that var in docker containers we use to run osmo-* during TTCN3 tests (and unit tests) in order to be able to easily see that the program is crashing, and hopefully also get a backtrace.

We should also enable that in osmo-gsm-tester.



#1 Updated by osmith about 1 year ago

  • Subject changed from TTCN3 docker containers: tell glibc to print segfault outputto stderr to TTCN3 docker containers: tell glibc to print segfault output to stderr

Note that right now, the osmo-* programs are configured to log to a file, e.g.

log file /data/osmo-msc.log

The osmo-* containers that run the programs are running in the background in the scripts (docker run -d), which means printing the segfault output to stderr would get lost.

But it should work when we change the configs to log to stderr, and then start the osmo-* programs with redirected output:

osmo-msc > /data/osmo-msc.log 2>&1

(A nice side-effect is, that we also get some output if the program does not start because the config can't be parsed etc.)

#2 Updated by pespin about 1 year ago

Agree with @osmith.

#3 Updated by pespin 12 months ago

I confirm we get nice output if the program fails to start (due to VTY CMD_WARNING for instance).
For instance, I run osmo-stp in a like this:

[... here all the docker run cmd line ....] \
$REPO_USER/osmo-stp-$IMAGE_SUFFIX /bin/sh -c "/usr/local/bin/osmo-stp -c /data/osmo-stp.cfg >/data/osmo-stp.output 2>&1" 

#4 Updated by pespin 12 months ago

  • Status changed from New to Feedback
  • % Done changed from 0 to 80
I have prepared patches in docker-playground.git to log processes stdout+stderr to log files which were used previously by VTY "log file" commands. As a result, we get:
  • If process fails to start due to VTY misconfiguration, we get the failure issue
  • If process crash, we get a backtrace. For instance see SGSN_Tests.TC_attach_req_id_req_ra_update which is expected to crash osmo-sgsn master (patch to fix it in gerrit):
    20191113155240771 DMM <0002> gprs_gb.c:40 MM_STATE_Gb(0)[0x56519ab58850]{Idle}: Received Event E_MM_PDU_RECEPTION
    20191113155240771 DMM <0002> gprs_gmm.c:1531 MM(262420000000026/f9ab395e) -> GMM RA UPDATE REQUEST type="RA updating" 
    20191113155240771 DMM <0002> gprs_gmm.c:1615 MM(262420000000026/f9ab395e) The MM context cannot be used, RA: 262-42-13135-0
    Assert failed mmctx->gb.llme == NULL gprs_gmm.c:1620
    backtrace() returned 15 addresses
    /usr/lib/x86_64-linux-gnu/ [0x7f6973eefd2b]
    osmo-sgsn(+0xcf6f) [0x56519a014f6f]
    osmo-sgsn(+0xd66e) [0x56519a01566e]
    osmo-sgsn(+0x8672) [0x56519a010672]
    osmo-sgsn(+0x1f77d) [0x56519a02777d]
    /usr/lib/x86_64-linux-gnu/ [0x7f6974337de3]
    /usr/lib/x86_64-linux-gnu/ [0x7f6974331c0c]
    /usr/lib/x86_64-linux-gnu/ [0x7f697433369a]
    /usr/lib/x86_64-linux-gnu/ [0x7f6974333807]
    /usr/lib/x86_64-linux-gnu/ [0x7f6973ee3a2f]
    /usr/lib/x86_64-linux-gnu/ [0x7f6973ee40a6]
    osmo-sgsn(+0x813f) [0x56519a01013f]
    /lib/x86_64-linux-gnu/ [0x7f69721e72e1]
    osmo-sgsn(+0x842a) [0x56519a01042a]
    talloc report on 'vty' (total 159756 bytes in 8441 blocks)
    [... here the full talloc report ...]
    signal 6 received
    Aborted (core dumped)

The LIBC_FATAL_STDERR_ is actually not needed afaict, I get the backtraces anyway.

remote: Update osmo-*/Dockerfile to log stdout+stderr of process to log file
remote: Update ttcn3-*/ to log stdout+stderr of process to log file

#5 Updated by pespin 11 months ago

  • Status changed from Feedback to Resolved
  • % Done changed from 80 to 100

Merged, closing.

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)