Project

General

Profile

Feature #4212

TTCN3 docker containers: tell glibc to print segfault output to stderr

Added by pespin 2 months ago. Updated 6 days ago.

Status:
Feedback
Priority:
Normal
Assignee:
sysmocom
Target version:
-
Start date:
09/18/2019
Due date:
% Done:

80%


Description

According to links below, glibc has a feature to log crash related output to stderr instead of /dev/tty (which in the case of running udner docker gets lost afaiu). The feature can be enabled by env var: LIBC_FATAL_STDERR_=1

The idea is to enable that var in docker containers we use to run osmo-* during TTCN3 tests (and unit tests) in order to be able to easily see that the program is crashing, and hopefully also get a backtrace.

We should also enable that in osmo-gsm-tester.

[1] https://stackoverflow.com/a/32057301
[2] https://stackoverflow.com/questions/32056387/catching-libc-error-messages-redirecting-from-dev-tty
[3] https://stackoverflow.com/questions/4290336/how-to-redirect-runtime-errors-to-stderr

History

#1 Updated by osmith 2 months ago

  • Subject changed from TTCN3 docker containers: tell glibc to print segfault outputto stderr to TTCN3 docker containers: tell glibc to print segfault output to stderr

Note that right now, the osmo-* programs are configured to log to a file, e.g.

log file /data/osmo-msc.log

The osmo-* containers that run the programs are running in the background in the jenkins.sh scripts (docker run -d), which means printing the segfault output to stderr would get lost.

But it should work when we change the configs to log to stderr, and then start the osmo-* programs with redirected output:

export LIBC_FATAL_STDERR_=1
osmo-msc > /data/osmo-msc.log 2>&1

(A nice side-effect is, that we also get some output if the program does not start because the config can't be parsed etc.)

#2 Updated by pespin 2 months ago

Agree with @osmith.

#3 Updated by pespin 12 days ago

I confirm we get nice output if the program fails to start (due to VTY CMD_WARNING for instance).
For instance, I run osmo-stp in a jenkins.sh like this:

[... here all the docker run cmd line ....] \
$REPO_USER/osmo-stp-$IMAGE_SUFFIX /bin/sh -c "/usr/local/bin/osmo-stp -c /data/osmo-stp.cfg >/data/osmo-stp.output 2>&1" 

#4 Updated by pespin 6 days ago

  • Status changed from New to Feedback
  • % Done changed from 0 to 80
I have prepared patches in docker-playground.git to log processes stdout+stderr to log files which were used previously by VTY "log file" commands. As a result, we get:
  • If process fails to start due to VTY misconfiguration, we get the failure issue
  • If process crash, we get a backtrace. For instance see SGSN_Tests.TC_attach_req_id_req_ra_update which is expected to crash osmo-sgsn master (patch to fix it in gerrit):
    ...
    20191113155240771 DMM <0002> gprs_gb.c:40 MM_STATE_Gb(0)[0x56519ab58850]{Idle}: Received Event E_MM_PDU_RECEPTION
    20191113155240771 DMM <0002> gprs_gmm.c:1531 MM(262420000000026/f9ab395e) -> GMM RA UPDATE REQUEST type="RA updating" 
    20191113155240771 DMM <0002> gprs_gmm.c:1615 MM(262420000000026/f9ab395e) The MM context cannot be used, RA: 262-42-13135-0
    Assert failed mmctx->gb.llme == NULL gprs_gmm.c:1620
    backtrace() returned 15 addresses
    /usr/lib/x86_64-linux-gnu/libosmocore.so.12(osmo_panic+0xcb) [0x7f6973eefd2b]
    osmo-sgsn(+0xcf6f) [0x56519a014f6f]
    osmo-sgsn(+0xd66e) [0x56519a01566e]
    osmo-sgsn(+0x8672) [0x56519a010672]
    osmo-sgsn(+0x1f77d) [0x56519a02777d]
    /usr/lib/x86_64-linux-gnu/libosmogb.so.9(bssgp_rcvmsg+0x543) [0x7f6974337de3]
    /usr/lib/x86_64-linux-gnu/libosmogb.so.9(+0x8c0c) [0x7f6974331c0c]
    /usr/lib/x86_64-linux-gnu/libosmogb.so.9(gprs_ns_rcvmsg+0x4a) [0x7f697433369a]
    /usr/lib/x86_64-linux-gnu/libosmogb.so.9(+0xa807) [0x7f6974333807]
    /usr/lib/x86_64-linux-gnu/libosmocore.so.12(+0xba2f) [0x7f6973ee3a2f]
    /usr/lib/x86_64-linux-gnu/libosmocore.so.12(osmo_select_main+0x6) [0x7f6973ee40a6]
    osmo-sgsn(+0x813f) [0x56519a01013f]
    /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1) [0x7f69721e72e1]
    osmo-sgsn(+0x842a) [0x56519a01042a]
    talloc report on 'vty' (total 159756 bytes in 8441 blocks)
    [... here the full talloc report ...]
    signal 6 received
    Aborted (core dumped)
    

The LIBC_FATAL_STDERR_ is actually not needed afaict, I get the backtraces anyway.

remote: https://gerrit.osmocom.org/c/docker-playground/+/16063 Update osmo-*/Dockerfile to log stdout+stderr of process to log file
remote: https://gerrit.osmocom.org/c/docker-playground/+/16064 Update ttcn3-*/jenkins.sh to log stdout+stderr of process to log file

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)