Cellular Network Infrastructure » Core Network (CN) » OsmoHNBGW

Starting from December 5th, we're seeing regressions in ttcn3-hnbgw-test latest:

https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-hnbgw-test-latest/535/ (+28 failures)

All affected testcases fail due to a DTE:

MTC@6005ed7d57b8: setverdict(fail): none -> fail reason: ""VTY Timeout for prompt: enable"", new component reason: ""VTY Timeout for prompt: enable"" 

We can also see a coredump file in the artifacts:

https://jenkins.osmocom.org/jenkins/view/TTCN3/job/ttcn3-hnbgw-test-latest/535/artifact/logs/hnbgw/core

I managed to reproduce the problem locally and examined the coredump in gdb:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f75debf1489 in on_success (data=<optimized out>, ci=0x562ca9a8f690) at ./src/libosmo-mgcp-client/mgcp_client_endpoint_fsm.c:539
539     ./src/libosmo-mgcp-client/mgcp_client_endpoint_fsm.c: No such file or directory.

(gdb) bt
#0  0x00007f75debf1489 in on_success (data=<optimized out>, ci=0x562ca9a8f690) at ./src/libosmo-mgcp-client/mgcp_client_endpoint_fsm.c:539
#1  osmo_mgcpc_ep_fsm_handle_ci_events (fi=<optimized out>, event=<optimized out>, data=<optimized out>) at ./src/libosmo-mgcp-client/mgcp_client_endpoint_fsm.c:957
#2  0x00007f75deaf2ef0 in _osmo_fsm_inst_dispatch (fi=0x562ca9a8be50, event=0, data=0x562ca9a9603c, file=0x7f75debf5ba3 "mgcp_client_fsm.c", line=446) at ./src/core/fsm.c:875
#3  0x00007f75deaf2ef0 in _osmo_fsm_inst_dispatch (fi=0x562ca9a95c10, event=3, data=0x562ca9a95d40, file=0x7f75debf5ba3 "mgcp_client_fsm.c", line=429) at ./src/core/fsm.c:875
#4  0x00007f75debe2817 in mgcp_client_handle_response (mgcp=0x562ca9a7d970, pending=0x562ca9a8b840, response=<optimized out>) at ./src/libosmo-mgcp-client/mgcp_client.c:246
#5  0x00007f75debe2dc4 in mgcp_client_rx (mgcp=mgcp@entry=0x562ca9a7d970, msg=msg@entry=0x562ca9a96380) at ./src/libosmo-mgcp-client/mgcp_client.c:741
#6  0x00007f75debe3da7 in mgcp_do_read (fd=0x562ca9a7ddb0) at ./src/libosmo-mgcp-client/mgcp_client.c:771
#7  0x00007f75deb0f241 in osmo_wqueue_bfd_cb (fd=0x562ca9a7ddb0, what=1) at ./src/core/write_queue.c:47
#8  0x00007f75deb00a94 in poll_disp_fds (n_fd=<optimized out>) at ./src/core/select.c:419
#9  _osmo_select_main (polling=polling@entry=0) at ./src/core/select.c:457
#10 0x00007f75deb00ba6 in osmo_select_main_ctx (polling=polling@entry=0) at ./src/core/select.c:513
#11 0x0000562ca98dc6e2 in main (argc=3, argv=0x7ffc2bf14318) at ./src/osmo-hnbgw/osmo_hnbgw_main.c:317

Looks like the problem is actually in libosmo-mgcp-client rather than in osmo-hnbgw?

ii  libosmo-mgcp-client12:amd64        1.12.1                         amd64        libosmo-mgcp-client: Osmocom's Media Gateway Control Protocol client utilities
ii  libosmocore                        1.9.2                          amd64        Open Source MObile COMmunications CORE library (metapackage)
ii  osmo-hnbgw                         1.5.0                          amd64        OsmoHNBGW: Osmocom Home Node B Gateway